How infrared sensors enhance situational awareness

Computer vision is becoming an increasingly important capability, not only in the aerial solutions industry, but many others. As such, a large portion of our efforts within AutonoSky has been dedicated to increasing our computer vision capability.

AutonoSky was founded to drive the innovation of aerial systems to assist rescue, fire and security organisations in their operations. To increase situational awareness throughout missions, we developed a streaming solution, ESP (extra sensory perception), to consume, analyse and distribute payload data in real-time. The uniqueness of our solution is the ability to incorporate our computer vision capabilities and draw insights from the payload data. These insights are relaid to operators such that appropriate actions can be taken.

The majority of the techniques and algorithms specific to computer vision (e.g. facial, object and scene detection) consider only RGB (red-green-blue) images that capture information from the visible portion of the electromagnetic spectrum. Given that the goal of computer vision is to replicate the human function of sight, it is natural that we’d consider the portion of the spectrum we as humans are comfortable with observing. The quality of these RGB images, however, are susceptible to degradation under unfavourable lighting conditions such as darkness and glares; this poses critical challenges on these algorithms and deteriorates their accuracy. You may rightly say, “We can only see what is visible though?” and “In the absence of light, everyone is blind?”, and you would be mostly correct. Fortunately, technology exists to extend our “sight” beyond that which is visible to us.

All objects emit thermal energy. The hotter the object, the more energy it emits; this is often referred to as an object’s heat signature. This energy, however, is not visible, at least not to us humans, as this energy is within the infrared band of the electromagnetic spectrum. The image below illustrates the different components of the electromagnetic spectrum and their corresponding wavelengths. For interest sake, shorter wavelengths correspond to higher energy; this is (broadly) why UV and X-rays are harmful to your body tissues.

Infrared sensors and images

In contrast to RGB camera sensors, thermal or infrared sensors detect radiation in the infrared (as opposed to the visible) portion of the electromagnetic spectrum. Infrared cameras detect thermal energy or heat radiated by objects in a scene and convert it into an electronic signal that is processed to produce an image. Such an image would yield brighter pixels for objects with higher temperatures and darker ones for objects with lower temperatures. An example of the difference between visible spectrum and infrared images is given below:

RGB image (left) and its infrared counterpart (right)

How then is all of this relevant? Well, given that computer vision algorithms analyse images (an array of pixel values), all they require as input is an image. A thermal or infrared image is no different to a standard image, it’s an array of pixels! These infrared images are able to provide us with additional information about a scene in scenarios where RGB images are compromised.

Practical Demonstration

Let us consider a scenario where visible light is not sufficient to describe a scene sufficiently and whether the utilisation of an infrared sensor can aid us in gaining a better understanding of that scene. This video shows the outputs of RGB and infrared images of a given scene.

It is evident that it is difficult to gather much information from an RGB image in low light scenarios. Infrared images, however, provide additional context to a scene. Equipping a vehicle’s payload with both an RGB and infrared/thermal camera, therefore, enables us to extend our context of a given scene and ultimately extends our capability.