Source Code: You can find source code on GitHub
AI infused mobile and web apps are now everywhere.
Personalized object detection can be one of the features that power many of the existing innovative apps.
Being able to turn the phone camera onto object locator especially without internet connection to find out where personal object (like toys, keys,...ect) is the primary objective of this platform.
I worked in a project where we needed to create personal object detection platform that supports exporting custom trained models to run in an offline mode on mobile devices (iOS (CoreML), Android (TensorFlow) and Windows (ONNX)).
The project provide several services that will allow the creation of a personal object detection workspace for each user of an app. Each project will have its own images and training data and the exported offline models (CoreML, TensorFlow and ONNX).
Project leverages Azure Custom Vision, Azure Storage, Azure Cosmos Db, Azure Functions among other Azure services to provide all needed backend to build personalized workspace for each user.
This project objective is to create a platform that will allow developers to offer personalized offline object detection models to be as part of a mobile app.
In order to successfully deploy this platform, you need to provision the following services on Azure:
- Cosmos DB: used in storing the user object detection workspace, which includes training files with regions and information about the workspace
- Azure Storage: used in storing the uploaded training images and the exported models for the 3 platforms in a container for each user
- Custom Vision Service: used in providing the object detection training backend
- API Management Service: used to securely expose the Azure Functions to the Internet through subscription based access with call rate limitation policy in place
- Application Insights: used to collect performance and health telemetry for the Azure Functions
- OPTIONAL Computer Vision Service: used in providing access to Microsoft Computer Vision pretrained service to detect objects without any custom training
Vision Object Detector project include 3 components to achieve that:
- ObjectDetectionManager: a .NET Standard Library that offers all needed services to orchestrate workspace creation, custom vision training and other management services needed.
- ObjectDetectionManager.AzureFunctions: exposes several Azure Functions of the ObjectDetectionManager library to provide HTTP access to the service beyond C#
- ObjectDetectionManager.TestClient: a simple console application with demo images and data to show how the ObjectDetectionManager library is used in a client application. Demo data is used from Azure Cognitive Services Sample on Github
Currently I'm working to:
- Complete the Azure Functions implementation and testing 🙂
- Complete other workspace management features (like extract workspace and delete workspace) to offer GDPR compiant system.
- Create asynchronous scenario using Durable Azure Functions and storage queues
- Provide better error handling especially when it relates to failures in submission to Custom Vision service
- Create a ClientSDK to the Azure Function to .NET apps can access the HTTP functions without writing HTTP calls
- Create a sample Xamarin Forms mobile client
About the project
I tried to make sure I cover all aspects and best practices while building this project, but all included architecture, code, documentation and any other artifact represent my personal opinion only. Think of it as a suggestion of how a one way things can work.
Keep in mind that this is a work-in-progress, I will continue to contribute to it when I can.
All constructive feedback is welcomed 🙂
You can always create issue, suggest an update through PR or direct message me on Twitter.