Using Machine learning to help a woman with visual impairments recognize her guide horse

Researching how we could help Monique, we investigated an existing Microsoft app named ‘Seeing AI’. This team has built a unique AI powered application that allows low sighted people to recognize photos, currency, read text, recognize colors and more. This is already close to what Monique was looking for, the colleagues in that team were super excited to hear about Monique’s specific request. After several discussions on how we could develop something, building on top of the code base was the approach to take. Leveraging the work already done in Seeing AI allowed us to deliver a new prototype, a special Seeing AI version for Monique!

One of the features we added to the prototype was a special recognition of her own blind-guided horse. Yes, not a dog but a very small and adorable horse named Dinky.

While we leveraged an open source image recognition ML model for object classification, recognizing the little horse required custom training. We wanted to apply transfer learning for classifying Dinky. And we wanted this to run on the smartphone without needing a connection or web service call delay, on other words all real time and offline. This was an important requirement so that Monique could move her phone and point towards Dinky, and know the horse is on the image without needing to wait for up to a few seconds for a response.

Customvision.ai allows for exposing the trained model as a hosted API but also for exporting into several offline platforms.

We leveraged Microsoft’s Azure Cognitive Services, more specifically CustomVision.ai which allows us to train and classify images through a web interface. These are the steps:

The iOS app is developed using Xamarin, so code samples here will be C#. But you will be able to find several samples leveraging native tooling by searching the net.

Import and Train Images
Gather as many images as you can, make sure you have different lighting and positions. In this case we have about 30 images of Dinky. Upload the images and add the tag.

Add images that you want to tag using the Negative tag. For example for Dinky we should use a few pictures of horses that don’t represent Dinky and that we want to train the model on understanding it’s not Dinky. Note at the time of our development this feature was not yet available.

Train the model
This step can take a few minutes if you have many images and tags.

Test the model
You can upload images with and without Dinky to test the result of your scoring. Continue fine tuning until you get a good result.

Export the model to CoreML
Under the Performance tab, choose Export. You can find different export options including a TensorFlow for Android use, a Dockerfile for use with IoT Edge for example and in our case the CoreML option for use in iOS. The download will be a .mlmodel file.

Compile the model
Use your favorite command line tool, move into the folder where you saved the mlmodel to and run the following command:

Import the model into your Xamarin app and scoring the image frame
You can find several samples online for integrating CoreML:

Informing Monique about where Dinky is on the image
At the time of the app development, customvision.ai did not have an object recognition option that we could export into CoreML. We still wanted to provide Monique with positioning info but all we get back from the scoring is the object classification and the accuracy percentage. So, we ended up with the following solution: we split the image frame coming through into 9 equal blocks, so 3 rows and 3 columns.
For each image we first want to score the full frame to understand if dinky is found on it.

And that’s all that is required to add some AI into your applications.

Related posts:

If you ask an existentialist He will tell you that At any moment One is always free To do what one will To be what one wants to be But if you don’t believe it It’s okay But don’t say I don’t know…

When it comes to security, there is always something to upgrade. According to past records, combining different technologies together has always proven to significantly increase the disruption as…

Masawa Minute is Masawa's bi-weekly newsletter where we share what we're reading in the areas of mental wellness, social impact, and impact investing.