Bird Watching in the 21st Century
Was that a catbird or a mockingbird you just saw? The answer is now just a smartphone tap away. Birdsnap—a new app developed with the help of a University of Maryland computer scientist—can identify birds from photos, using methods borrowed from facial recognition software. Simply snap a photo of a bird, and the app can tell you what species it is and all about it.
Birdsnap is the latest in a series of electronic field guides that David Jacobs, professor of computer science at UMD, has helped to develop. About 10 years ago, he and his colleague Peter Belhumeur, professor of computer science at Columbia University, were looking to connect their research on facial recognition software with other fields. They realized that the same techniques used by computers to recognize faces could also be used to identify plants and animals. Together, they approached W. John Kress, research scientist and curator in the Department of Botany at the Smithsonian’s National Museum of Natural History, to collaborate on an electronic field guide for plants. As smartphones and mobile apps became popular, they decided to build an app based on their work. The result was Leafsnap, a free iPhone app that identifies trees from pictures of their leaves, which was introduced in May 2011.
Encouraged by their success with plants, Jacobs and Belhumeur decided to try identifying animals. They chose to look at dogs, partly because, as Jacobs notes, “it’s really easy to get a lot of pictures of dogs.” The researchers released the corresponding app, Dogsnap, in 2012. However, there’s not a huge need to identify dogs from photos, according to Jacobs, so he and Belhumeur sought a more in-demand use for their techniques. Birdsnap, which debuted in Apple’s App Store on May 27, 2014, extends the methods of the earlier apps to identifying birds.
“In human face recognition, a key problem is to find the parts of the face, like the corners of the mouth and the tip of the nose, so that you can compare the nose to the nose and the mouth to the mouth,” explains Jacobs, who also has an appointment in the University of Maryland Institute for Advanced Computer Studies.
To identify a bird from a photo, the app first identifies the different parts of the bird, such as the beak, based on thousands of examples of what each body part looks like. It also considers the beak’s position relative to the other parts of the bird, such as the tail and wings. That way, the app is less likely to mistake, say, a leg for a beak. Next, each feature is compared against a database of the 500 most common bird species in North America. The algorithm calculates the likelihood of each species matching all of the features and returns the most likely candidates. One way in which Birdsnap improves on Leafsnap and Dogsnap is by incorporating available data on bird sightings from the Cornell Lab of Ornithology. Users specify the date and place where a photo was taken, and Birdsnap looks up which birds are most common in that location at that time of the year. This helps the algorithm to narrow the list of possible matches.
But Birdsnap goes beyond simple visual recognition. The app contains a wealth of information about the birds in its database. You can sort birds according to how common they are in a particular place and time or according to their evolutionary relationships. The app also shows which birds are arriving, departing or migrating through, based on the location and time of year. Clicking on a bird will show you photos, a description (sourced from Wikipedia and independently verified), and maps showing the bird’s range and likelihood of sighting in different places throughout the year, as well as play audio files of the bird’s song. Most interestingly, you can see side-by-side comparisons of similar species that highlight the most distinguishing features. Thus, Birdsnap not only identifies birds for you, it also trains you how to recognize them yourself.
Leafsnap took 10 years to develop, in part because staff at the Smithsonian had to collect and photograph thousands of leaves. By contrast, most of the photos in Birdsnap’s database were taken from appropriately tagged Flickr images. Workers hired through Amazon Mechanical Turk, an online labor market, labeled individual parts of birds to be used in identification. The websites eBird and BirdLife provided the data for the birds’ range and likelihood of sightings. By building the database from existing online resources, Jacobs and his colleagues got Birdsnap up and running much more quickly than Leafsnap.
Although Birdsnap is geared toward recreational bird watchers, Jacobs notes that the app, and others like it, could be useful for serious research. “There’s a lot of interest in studying how the distribution of species is varying over time,” Jacob says. “Are we seeing birds in parts of the country where we didn’t typically see them?” Such information could be useful in studying the effects of climate change, for instance.
Jacobs’ next steps include looking at other animals and other parts of the world (a version of Leafsnap for the United Kingdom was released May 16, 2014). Jacobs’ big dream is to create a system that automatically generates new field guides, so that “if someone wanted to create a field guide for beetles, that person could take images of different species of beetles and we could automatically generate a field guide that included those species.”
This research was supported by the National Science Foundation (NSF) under Award No. 1116631. The content of this article does not necessarily reflect the views of the NSF.
Media Relations Contact: Abby Robinson, 301-405-5845, abbyr@umd.edu
Writer: Brian Doctrow
University of Maryland
College of Computer, Mathematical, and Natural Sciences
2300 Symons Hall
College Park, Md. 20742
www.cmns.umd.edu
About the College of Computer, Mathematical, and Natural Sciences
The College of Computer, Mathematical, and Natural Sciences at the University of Maryland educates more than 7,000 future scientific leaders in its undergraduate and graduate programs each year. The college’s 10 departments and more than a dozen interdisciplinary research centers foster scientific discovery with annual sponsored research funding exceeding $150 million.
