A Preliminary Study of Windows ML, the Built-in Machine Learning Platform

Artificial intelligence is very popular now. Although the limelight has been overshadowed by the blockchain recently, it is still one of the preferred directions for future technological transformation. Machine learning, the core of AI, has also evolved to the point where models can be automatically trained based on platforms, such as Azure Machine Learning Service and Google AutoML Service. This greatly reduces the difficulty of training the model, and developers can focus more on the application of the trained model.

In this context, various operating system platforms have launched built-in machine learning frameworks/running environments, such as CoreML for iOS and TensorFlow for Android. After the recent RS4 (build 1803) update, Windows also officially has a built-in machine learning platform - Windows ML .

What is Windows ML?

Windows ML is Windows' new built-in machine learning platform for natively executing pre-trained machine learning models and provides APIs that allow us to quickly integrate into applications.

Its highlights are as follows:

Support hardware acceleration

DirectX 12 compatible devices can directly use GPU acceleration to ensure that machine learning models can be executed efficiently.

native execution

It does not depend on any remote service and is not limited by any network connection, and the machine can achieve low-latency and high-performance execution.

Image processing optimization

For computer vision scenarios, video, image and camera data are preprocessed into VideoFrame format to simplify the image processing process.

Model requirements

Windows ML currently only supports the execution of ONNX format models, other formats need to be converted before use.

ONNX is a general and open machine learning model format launched by companies such as Microsoft, Facebook, and Intel, and officially supports the conversion of existing machine learning frameworks. ONNX project address

Existing model sources that support conversion:

Core ML
Scikit-Learn
XGBoost
LibSVM

The conversion tool used is WinMLTools provided by Microsoft: https://pypi.org/project/winmltools/

For the tutorial on using the conversion tool, please refer to the official documentation: https://docs.microsoft.com/en-us/windows/uwp/machine-learning/conversion-samples

code generation

After installing Windows SDK Build 17110 or newer, the model code generation tool mlgen.exe is added by default for Visual Studio 2017 projects . It can automatically generate C#/CX definition files based on the added ONNX model file and Visual Studio 2017 Preview, so that the code can be called directly.

Here is an example of the FNS-La-Muse model, which is a model that can transform images into a specific style.

The generated code is as follows:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Windows.Media;
using Windows.Storage;
using Windows.AI.MachineLearning.Preview;

// FNSLaMuse

namespace Demo
{
    public sealed class FNSLaMuseModelInput
    {
        public VideoFrame inputImage { get; set; }
    }

    public sealed class FNSLaMuseModelOutput
    {
        public VideoFrame outputImage { get; set; }
        public FNSLaMuseModelOutput()
        {
            this.outputImage = VideoFrame.CreateWithSoftwareBitmap(new Windows.Graphics.Imaging.SoftwareBitmap(Windows.Graphics.Imaging.BitmapPixelFormat.Bgra8, 720, 720));
        }
    }

    public sealed class FNSLaMuseModel
    {
        private LearningModelPreview learningModel;
        public static async Task<FNSLaMuseModel> CreateFNSLaMuseModel(StorageFile file)
        {
            LearningModelPreview learningModel = await LearningModelPreview.LoadModelFromStorageFileAsync(file);
            FNSLaMuseModel model = new FNSLaMuseModel();
            model.learningModel = learningModel;
            return model;
        }
        public async Task<FNSLaMuseModelOutput> EvaluateAsync(FNSLaMuseModelInput input) {
            FNSLaMuseModelOutput output = new FNSLaMuseModelOutput();
            LearningModelBindingPreview binding = new LearningModelBindingPreview(learningModel);
            binding.Bind("inputImage", input.inputImage);
            binding.Bind("outputImage", output.outputImage);
            LearningModelEvaluationResultPreview evalResult = await learningModel.EvaluateAsync(binding, string.Empty);
            return output;
        }
    }
}

View Code

At present, because the SDK is still in preview, the official version of Visual Studio will not automatically call the mlgen tool to generate the definition file. You need to manually execute the following commands:

mlgen -i INPUT-FILE -l LANGUAGE -n NAMESPACE [-o OUTPUT-FILE]

INPUT-FILE: ONNX model file
LANGUAGE: C++ or C#
NAMESPACE: Namespaces
OUTPUT-FILE: output path, can be default

Summarize

With Windows ML, we can realize machine learning features that were difficult to achieve before, and at the same time, without relying on external web services, many innovative experiences can be realized, not only on PCs, but also on HoloLens. Machine learning capabilities can also be used.

Finally, I will give you my open source project - Awesome WindowsML ONNX Models . In addition to the models I have verified, this project also provides a quick conversion tool for CoreML models.

At the same time, I am also developing a demo for HoloLens, and I will meet you soon