UVA researchers engineer AI breakthrough in human action detection technology

What if a security camera could not only capture video but understand what’s happening — distinguishing between routine activities and potentially dangerous behavior in real time? That’s the future being shaped by researchers at the University of Virginia’s School of Engineering and Applied Science with their latest breakthrough: an AI-driven intelligent video analyzer capable of detecting human actions in video footage with unprecedented precision and intelligence.

The system, called the Semantic and Motion-Aware Spatiotemporal Transformer Network (SMAST), promises a wide range of societal benefits from enhancing surveillance systems and improving public safety to enabling more advanced motion tracking in healthcare and refining how autonomous vehicles navigate through complex environments.

“This AI technology opens doors for real-time action detection in some of the most demanding environments,” said professor and chair of the Department of Electrical and Computer Engineering, Scott T. Acton, and the lead researcher on the project. “It’s the kind of advancement that can help prevent accidents, improve diagnostics and even save lives.”

AI-Driven Innovation for Complex Video Analysis

So, how does it work? At its core, SMAST is powered by artificial intelligence. The system relies on two key components to detect and understand complex human behaviors. The first is a multi-feature selective attention model, which helps the AI focus on the most important parts of a scene — like a person or object — while ignoring unnecessary details. This makes the system more accurate at identifying what’s happening, such as recognizing someone throwing a ball instead of just moving their arm.

The second key feature is a motion-aware 2D positional encoding algorithm, which helps the AI track how things move over time. Imagine watching a video where people are constantly shifting positions —this tool helps the AI remember those movements and understand how they relate to each other. By integrating these features, SMAST can accurately recognize complex actions in real time, making it more effective in high-stakes scenarios like surveillance, healthcare diagnostics, or autonomous driving.

SMAST redefines how machines detect and interpret human actions. Current systems struggle with chaotic, unedited contiguous video footage, often missing the context of events. But SMAST’s innovative design allows it to capture the dynamic relationships between people and objects with remarkable accuracy, powered by the very AI components that allow it to learn and adapt from data.

Setting New Standards in Action Detection Technology

This technological leap means the AI system can identify actions like a runner crossing a street, a doctor performing a precise procedure or even a security threat in a crowded space. SMAST has already outperformed top-tier solutions across key academic benchmarks including AVA, UCF101-24 and EPIC-Kitchens, setting new standards for accuracy and efficiency.

“The societal impact could be huge,” said Matthew Korban, a postdoctoral research associate in Acton’s lab working on the project. “We’re excited to see how this AI technology might transform industries, making video-based systems more intelligent and capable of real-time understanding.”

This research is based on the work published in the article "A Semantic and Motion-Aware Spatiotemporal Transformer Network for Action Detection" in the IEEE Transactions on Pattern Analysis and Machine Intelligence. The authors of the paper are Matthew Korban, Peter Youngs, and Scott T. Acton from the University of Virginia.

The project was supported by the National Science Foundation (NSF) under Grant 2000487 and Grant 2322993.

About UVA Engineering: As part of the top-ranked, comprehensive University of Virginia, UVA Engineering is one of the nation’s oldest and most respected engineering schools. Our mission is to make the world a better place by creating and disseminating knowledge and by preparing future engineering leaders. Outstanding students and faculty from around the world choose UVA Engineering because of our growing and internationally recognized education and research programs. UVA is the No. 1 public engineering school in the country for the percentage of women graduates, among schools with at least 75 degree earners; among the top engineering schools in the United States for the four-year graduation rate of undergraduate students; and among the top-growing public engineering schools in the country for the rate of Ph.D. enrollment growth. Our research program has grown by 95% since 2016. Learn more at engineering.virginia.edu.