Engineers have long struggled to construct cameras that can see in limited visibility environments, but Brown researchers recently developed an imaging system that can track and reconstruct moving objects obscured in these conditions.
Fog is difficult to see through because it both blocks and scrambles light, said Ning Zhang, the paper’s first author and a postdoctoral research associate in the School of Engineering. When using a camera to visualize moving objects through fog, “only a tiny fraction of that light ever touched the object, and that faint signal is buried under a bright, featureless glow.”
The newly developed imaging system couples a camera that imitates the human retina with a computer model based on the brain’s visual cortex. Cameras capture moving objects similarly to how the retina processes and encodes data in the brain.
According to Zhang, this “is the first fully brain-inspired system that both tracks and images randomly moving objects through this kind of dense scattering.”
The system’s first half is a dynamic vision sensor, also known as an event camera. While a regular camera measures how much light it receives, an event camera measures the changes in that light, said Associate Professor of Computer Science James Tomkin, who was not affiliated with the study.
“This particular design for building cameras is very good at sensing motion … in a low-power way,” Tomkin added.
Since the way that fog scatters light is “fairly steady” while an obscured object is in motion, a sensor that “responds only to change should ignore much of the fog and pick out the object,” Zhang wrote. This sensor produces a “flickering cloud of dots that looks like noise to a person.”
These dots are then interpreted using the second half of the system, which is similar to a human neural network. Zhang explained that the brain encodes visual information in short pulses, known as spikes. Together, these spikes track movement and reconstruct the object’s shape.
The system “speaks one language, spikes, from capture to final image,” Zhang wrote.
Zhang estimated that the spiking neural network sensor uses “18 times less computing energy than a comparable conventional network.”
The team used a 12 millimeter silicone slab, a water tank obscured with a milky emulsion and a fog chamber to create three testing environments. Instead of using moving physical objects, researchers projected images including handwritten digits, Japanese characters and bird silhouettes that each flew 100 randomized paths to ensure that the network was tested “on flights it had never seen,” Zhang wrote.
The researchers then measured how their visualizations compared to the actual objects using the Structural Similarity Index Measure, a “standard measure of how alike two images look,” according to Zhang. SSIM measures objects on a scale from negative one to one, where one is perfect similarity, zero is no similarity and negative one is anticorrelated to the object.
In this study, SSIM values up to 0.96 were produced, which indicated that “the shape our system rebuilt from fog-scrambled spikes is nearly identical in structure to the image we actually projected,” Zhang wrote.
According to Tomkin, that number is “not a percent per se.” Researchers have to “contextualize” the metric score to the task, and account for the difficulty level of matching any particular object. But, generally, a score close to one is “ a pretty good score,” he said.
The work is “a laboratory proof of concept, and there are real steps between here and a car or a drone,” Zhang wrote, adding that it may take several years before reaching an “early, specialized prototype.”
Tomkin believes that there is still more work to do before the results of the study can be fully applied to the real world, as objects that change shape over time — like a bird flapping its wings — make the imaging process “much more difficult.” Another limitation, Tomkin noted, is that the system only recovers silhouettes rather than full grayscale images.
Davide Scaramuzza, a professor at the University of Zurich who studies event cameras, called the new research “a promising laboratory demonstration.”
The study’s strengths lie in “combining tracking and reconstruction across several scattering conditions,” Scaramuzza wrote.
In the future, Tomkin believes that because brain-inspired imaging requires low amounts of power, it has the potential to improve the energy efficiency of current technology.
“The encouraging part is that the hardware is simple, compact and low-power, which suits battery-powered platforms,” Zhang wrote. “Likely first uses are specialized ones: drones or helmet cameras for firefighters in smoke, underwater robots in turbid water and spotting vehicles or pedestrians in dense fog.”
Laila Posner is a senior staff writer covering business and development.
