App creates time-lapse videos of body for telemedicine

ITHACA, N.Y. – A new app developed by Cornell University researchers helps users record highly accurate time-lapse videos of body parts – a surprisingly difficult task and an unmet need in remote medicine and telehealth applications.

A team led by Abe Davis, assistant professor of computer science, created the app, called MeCapture. It uses augmented reality, 3D tracking and interactive visual feedback to help users consistently capture images of body parts day after day and collect medically relevant information outside of the clinic.

The app’s inspiration came during the COVID-19 pandemic. “A lot of patients were communicating with their doctors using data they were capturing on their phones,” Davis said. However, previous time-lapse apps weren’t up to the task.

“Patients might take pictures from different angles or under different lighting, and this makes it hard to compare two photos and tell if the patient is getting better or worse,” he said. “The solution is to make sure the only thing changing from one picture to the next is the patient’s condition. But this requires precise tracking, and existing on-phone trackers break down on subjects that move and bend like the body.”

Researchers identified three factors that must be replicated to yield medically useful information: the pose; the viewpoint of the camera; and the lighting conditions. They integrated multiple new functions into a camera-based app to control these variables.

They also added an automatic recapture feature that snaps the photos as soon as the camera is properly aligned, so that the act of pressing the camera shutter button doesn’t throw off the alignment..

Researchers tested MeCapture by tracking their facial hair growth and recording the life cycle of a Chia Pet. Another lab member accidentally burned his hand and one researcher tracked a foot bruise healing, all of which can be seen in their interactive 3D gallery.

The app in its current form provides a very general solution for collecting data for telemedicine, but the technology can be tailored for specific medical applications.

Beyond medicine, this type of app could record the growth of plants for field research and monitor structural health in buildings – tracking damage, such as corrosion or a crack, to see if it spreads. 

For additional information, read this Cornell Chronicle story.

Media note: Pictures can be viewed and downloaded here: https://cornell.box.com/v/MeCaptureAppDavis

-30-