Facebook is now using AI to describe photos to the blind

A description of what’s in an image is being provided via new object recognition technology that Facebook has been working on for months.  This means that blind or visually impaired users who access Facebook via screen readers (devices or programs that give an audio description of what’s happening on a screen) will get short descriptions of the images people are posting.

Facebook is calling the system “automatic alternative text” and it’s based on a neural network primed with billions of parameters and millions of examples.    The company has been working on automatic alt text for around ten months now, and notes that around 2 billion photos are shared each day across Facebook, Instagram, Messenger, and WhatsApp.  The AI software doesn’t actually “see” the picture, but it can compare the objects in it with its vast internal database of similar photos and make an educated guess about what’s being shown.  Part of the challenge, is in getting computers to recognize what’s most important in an image.  For each image, the AI system returns a confidence score indicating how sure it is that it can identify what’s in the picture. If this is above 80 percent, an automatically-generated caption appears.  If there’s some ambiguity about the picture then the sentence starts with “image may contain” to express that uncertainty.

This follows Twitter recently allowing users to manually add alt text to their pictures to help out any visually impaired followers.

Currently, automatic alt text for Facebook is only available in English and only in the US, UK, Canada, Australia, and New Zealand. More importantly it only works with screen readers being used on an iOS device. Facebook says that will all change with more platforms, languages and regions coming soon.

The goal of Facebook  is to connect the world, “If we’re going to achieve that goal, then we’re going to develop strategies that work for accessibility. It’s not going to happen automatically,” per Jeff Wieland, the project manager for Facebook’s accessibility initiative.

sources:  huffingtonpost.com, gizmag.com, cnet.com

Brian CecilFacebook is now using AI to describe photos to the blind