NEMESIS Keeps an Eye on Warfighters
Warfighters save time and energy in the field when guided by NEMESIS-equipped drones. © 2020 Getty Images.
HRL Laboratories’ Neuromorphic Eyes in the Sky Provides Actionable Intelligence in Real Time
HRL Laboratories, LLC, have completed successful live demonstrations of their NEMESIS program (NEurMorphic EyeS In the Sky), a revolutionary drone-based system to analyze situational awareness and provide actionable intelligence in real time.
With NEMESIS, time and human mental energy are not spent watching and analyzing a video feed constantly during a mission.
NEMESIS is a bioinspired system of fast, accurate video processing that emulates the human vision system. With NEMESIS, small drones can extract, recognize and track human movement, behaviors, and overall scenes from multi-modal sensor data, providing high-quality tactical decisions and actionable intelligence to warfighters on the ground as action is happening. By analyzing video imagery and other data with low power and high speed, NEMESIS signals are very low bandwidth, which could be sent as an audio feed to a headset or as text messages on a recipient’s receiving device “as it happens.”
For the field demonstrations, two primary tests were conducted. The first was activity recognition. This test included real-time object detection, classification, and tracking of people, vehicles, or packages. NEMESIS also indicated single or group human behaviors, such as whether the people detected were carrying or aiming weapons, dropping or picking up objects, walking alone or with others, or conversing with each other.
The NEMESIS system enables autonomous drones to recognize scenes, oversee, and direct warfighters in the field in real time. © 2020 Getty Images.
The second test was mission analytics. This involved recognizing mission phases such as unit position, building entry, and regrouping phases. NEMESIS successfully detected mission anomalies such as incorrect individual positions, incorrect unit positions, and incomplete regroupings, i.e., one number of subjects starts out on a mission and a different number returns. NEMESIS aso provided explanation of such mission anomalies via human-understandable text.
“We demonstrated the NEMESIS capabilities of providing accurate real-time situational awareness with low size, weight, and power intelligent autonomous platforms. These can enable reduced warfighter cognitive workload and more effective missions,” said HRL’s Deepak Khosla, principal investigator on the NEMESIS project. “With NEMESIS, time and human mental energy are not spent watching and analyzing a video feed constantly during a mission. The drones analyze scenes immediately, giving guidance and decisions that can allow warfighters to focus directly on their mission with greatly reduced distraction and fatigue.”
“The neuromorphic aspect of NEMESIS derives from the way the system emulates the human visual pathway for fast, accurate, and efficient video processing,” Khosla said. “The drone is making sense of what it’s looking at and explaining it so that the end user can benefit from the information. The original concept for this project was expeditionary warfare, such as a team of soldiers moving to a new position. They send this drone a couple miles ahead to find out if there are military vehicles in the area or it’s flooded or a bridge has collapsed. Then instead of interpreting a raw feed all day, they have actionable intelligence processed for their immediate use. Their commander can then use the same technology to keep track of his men in field.”
NEMESIS was integrated with small quadcopters for the live field demonstrations on November 14, 2019 at Camp Snyder in Prince William County Virginia, a 350-acre facility that includes a managed wetland. Other members of the NEMESIS team include: Leon Nguyen, Yang Chen, Fan Hung and Haden Smith.
This material is based upon work supported by the Office of Naval Research (ONR) under Contract No. N00014-15-C-0091. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Office of Naval Research (ONR).
HRL Laboratories, LLC, Malibu, California (hrl.com) is a corporate research-and-development laboratory owned by The Boeing Company and General Motors specializing in research into sensors and materials, information and systems sciences, applied electromagnetics, and microelectronics. HRL provides custom research and development and performs additional R&D contract services for its LLC member companies, the U.S. government, and other commercial companies.
Media Inquiries: media[at]hrl.com, (310) 317-5000