Fish-ial recognition software aims to protect trout

New research focused on brook trout is using artificial intelligence to identify individual fish, with the goal of building population models that track trout health and habitat changes.

This groundbreaking use of AI, a collaboration between data scientists at the University of Virginia and the U.S. Geological Survey, will create a more efficient and accurate way to track trout by using fish-ial recognition software.

Researchers are classifying fish in both controlled and natural environments in West Virginia and Massachusetts, building a unique database that has the potential to save the taxpayer millions of dollars and advance protective measures for trout and streams. They hope to engage anglers as boots-on-the-ground citizen scientists to assist with the project, creating an interactive application where fishermen can upload images of fish and participate in protecting the health of brook trout and preserving their natural environment.

Fish biologists have been studying climate change and conservation for decades, and tracking fish is not new. Previously, however, scientists have had to use markers or injections to identify individual fish, methods that are invasive, require minor surgery, and do not work on small fish. “The new frontier is individual recognition using AI technology,” said Nathaniel Hitt, a research fish biologist with the U.S. Geological Survey.

The project originated during work at Shenandoah National Park by researchers from the U.S. Geological Survey’s Ecological Science Center in West Virginia. “We were using video sampling in stream pools to estimate the abundance of brook trout. We would take underwater video and have human observers count fish,” said Hitt. “We actually crowdsourced this to schools across the nation.”

The success of the crowdsourcing got the fish biologists thinking about how they could automate the process. With the rise of AI and computer science applications like facial recognition software, they thought, why not apply it to fish? Brook trout have unique identifying markings, making them the perfect fish species to test this theory.

Brook trout are unique in that they are the only native trout of Appalachia and have been around for millions of years. Anglers for generations have come to love the fish and are invested in protecting its future. Brook trout have ecological importance as well, according to Hitt: “They’re the canary in the coal mine for climate change.”

Ben Letcher, a research ecologist at the Conte Research Laboratory in Turners Falls, Mass., who is partnering on the project, explains: “Each state in New England has cold-water criteria, and some states use the presence of a brook trout to identify a cold-water stream. Cold-water streams get special protections, so knowing where the trout are now and where they will be in the future is important for land protection and conservation.”

To build a database of images large enough to be useful for prediction models, the researchers are capturing fish images in both controlled fisheries in West Virginia and in the wild streams of western Massachusetts, using different methods while working toward the same goal.

In Massachusetts, the team uses an electrofisher backpack to collect fish. They then place the caught fish in a bucket, anesthetize a few at a time, and then take measurements and photographs before releasing them back into the stream from which they came. In West Virginia researchers have used GoPro cameras to collect images of fish while they swim in tanks. The team then uses anesthetics to capture measurements and take additional photographs.

All of those images are then shared with data scientists at the University of Virginia who feed them into an image processing pipeline that identifies individual fish features. The team, led by Sheng Li, an assistant professor of data science at UVA, then trains the model to improve image recognition.

“It’s quite challenging,” said Li. “You see a large variation in fish appearance such as body size and other changes over time. We have had to develop multiple AI methods to improve the recognition of each individual fish.” 

The data scientists rely heavily on the images provided by the on-the-ground fish biologists and ecologists wading into streams, catching fish, and carefully and categorically photographing them.

Everyone on the project credits the project’s success to interdisciplinary collaboration. Fish experts like Hitt and Letcher work with computer and data scientists like Li and others to use new techniques to solve old, persistent problems.

Hitt believes the tools they are developing using AI could have applications toward research on any animal with spots. “We envision this transforming fish biology globally.”

But the challenge of amassing a large enough and current database of images remains, which is why the team hopes to appeal to citizen scientists and anglers to be active participants.

By using their phone to capture and upload photos of fish caught, a future interactive database has the potential to identify a specific fish, trace its tracking history, and feed up-to-date information in real time. The U.S. Geological Survey is working with fishing expedition companies to test out this new method of collecting data.

Letcher predicts a phone application could be created where an angler takes a photo of caught fish; uploads it to an open, shared database; and learns its exact identification and history. “This could be very valuable for collecting scientific information but also to engage anglers in new ways,” he said.

“Using images, we can create individual fish ID and could monitor population trajectories,” said Hitt, “but this also changes the relationship between anglers and these natural resources. It fosters a deeper sense of stewardship and connection to the streams and rivers.”

When speaking of brook trout, Letcher and his colleagues become almost reverent. “You’re taking something so ancient, so deeply rooted in the evolution of our planet, and developing a new appreciation and respect for it.”