Skip to content

Tutorial: MobileNetV2 (ONNX)

This tutorial takes an ONNX image classifier, MobileNetV2 trained on ImageNet (1000 classes), and runs it on the board's CPU: from the .onnx file to a live pipeline in the No-Code Flow Editor and in the MuseBox Client.

The files are in musebox-compiler/examples/mobilenet_v2_onnx/:

File What it is
mobilenetv2-12-int8.onnx the model (quantized to 8 bits, float input)
labels.txt the 1000 ImageNet class names, one per line
preprocess.cpp, postprocess.cpp optional C++ code, see Your own C++ code
mobilenet_v2_pipeline.json a ready pipeline for the editor and the client

You need:

  • the MuseBox Compiler installed on your PC (chapter 6);
  • the AI server running on the board (chapter 3), and ssh root@<board-ip> working from the PC (deploy copies the package with it);
  • the editor (chapter 4) or the client (chapter 5).

1. Look inside the model

On your PC, Linux / macOS:

pc$ cd ~/musebox/musebox-compiler
pc$ source .venv/bin/activate

On your PC, Windows 10 / 11 (Command Prompt or PowerShell):

pc$ cd C:\Users\<your-user>\musebox\musebox-compiler
pc$ .venv\Scripts\activate

On your PC:

pc$ mbx-compile inspect-cpu examples/mobilenet_v2_onnx/mobilenetv2-12-int8.onnx

{
  "source": "onnx",
  "backend": "onnx",
  "inputs": [
    {
      "name": "input",
      "shape": [-1, 3, 224, 224],
      "data_type": "FLOAT32",
      "format": "RGB",
      "layout": "NCHW"
    }
  ],
  "outputs": [
    {
      "name": "output",
      "shape": [-1, 1000],
      "data_type": "FLOAT32"
    }
  ],
  "opset_version": 12,
  "ir_version": 7
}

What this tells you:

  • Input: one RGB image of 224 × 224 pixels, channels first (NCHW), as float numbers. -1 is the batch size: the AI server sends one image at a time. The model is quantized inside, but it takes float input, so the AI server normalizes the pixels (step 3).
  • Output: 1000 raw scores, one per ImageNet class. They are not probabilities yet: the AI server applies a softmax to them.

2. Create the package

On your PC:

pc$ mbx-compile init --model examples/mobilenet_v2_onnx/mobilenetv2-12-int8.onnx --name mobilenet_v2_onnx --task classification --labels examples/mobilenet_v2_onnx/labels.txt --num-threads 4

This creates the folder mobilenet_v2_onnx/ with the model, the labels and a manifest.json. No C++ code is needed: for a classifier the AI server resizes the frame, converts it to RGB, normalizes it, then applies the softmax and returns the 5 best classes with their names.

--num-threads 4 lets the model use the four CPU cores of the board.

3. Set the normalization

MobileNetV2 was trained on images normalized with the ImageNet mean and standard deviation. init cannot know this, and writes a normalization that only scales the pixels to 0–1. Open mobilenet_v2_onnx/manifest.json and change the normalization block to:

  "input": {
    "tensor_name": "input",
    "shape": [-1, 3, 224, 224],
    "format": "RGB",
    "data_type": "FLOAT32",
    "normalization": {
      "mean": [0.485, 0.456, 0.406],
      "std": [0.229, 0.224, 0.225]
    }
  },

The AI server computes (pixel / 255 - mean) / std for each channel (it divides by 255 first because all the values are at most 1). With the wrong normalization the model still runs, but its answers are much less accurate.

The rest of the manifest is already right:

  "postprocessing": {
    "type": "classification",
    "top_k": 5,
    "apply_softmax": true,
    "labels_file": "labels.txt",
    ...
  },
  "runtime": {
    "backend": "onnx",
    "num_threads": 4,
    ...
  }

All the fields are described in the package specification.

4. Check and package

On your PC:

pc$ mbx-compile verify mobilenet_v2_onnx/
pc$ mbx-compile package mobilenet_v2_onnx/ -o mobilenet_v2_onnx.mbx

verify prints:

[PASSED] Package 'mobilenet_v2_onnx/' is valid and ready for deployment!

5. Copy it to the board

On your PC:

pc$ mbx-compile deploy mobilenet_v2_onnx.mbx --remote root@<board-ip> --dest /root/musebox-ai-servers/ai_models

Steps 1 to 5 on the PC:

Recorded on the Mercury+ PE3 Hailo-8 (Mercury+ MP1 SoM)

This unpacks the package into /root/musebox-ai-servers/ai_models/mobilenet_v2_onnx/ on the board. Restart the AI server so that it finds the new model. If the server runs in an ssh terminal, press Ctrl+C in that terminal; if you started it in the background (chapter 3), run pkill -f musebox-ai-server on the board (pkill -f musebox_ai for the Python server). Then start it again with the commands below. Keep running the pc$ commands in a separate terminal on your PC.

On the board:

board$ cd /root/musebox-ai-servers
board$ ./musebox-ai-server-cpp/musebox-ai-server

When the server prints ... task(s) ready., it has loaded the models. Open http://<board-ip>:8080/models in the browser: it now lists mobilenet_v2_onnx.

Recorded on the Mercury+ PE3 Hailo-8 (Mercury+ MP1 SoM)

6. Build the pipeline in the editor

Start the editor (chapter 4) and build this pipeline:

  1. In the configuration node already on the canvas (add it from MuseBox Configuration → configuration if it is missing): URL ws://<board-ip>:8080, then Connect / Reconnect.
  2. Click Sync Models in the toolbar. The editor asks the board for its models and adds a node for each custom model (the built-in tasks are already in the menu).
  3. MuseBox Tasks → CPU Models → mobilenet_v2_onnx.
  4. MuseBox Input → PC webcam.
  5. MuseBox Operators → Scale, width 224, height 224: the frames travel to the board already at the model's size.
  6. MuseBox Output → Image Rendering.

Connect them: webcam → Scale → the frame / bounding box input of mobilenet_v2_onnx, and its post process output → Image Rendering. Press Play: the image shows the three most likely classes with their probabilities. For a list of the five best classes, connect the predictions / data output to MuseBox Output → Top 5.

Instead of Sync Models you can use MuseBox Tasks → Custom → Custom Model: click Fetch Models from Server and choose mobilenet_v2_onnx.

Recorded on the Mercury+ PE3 Hailo-8 (Mercury+ MP1 SoM)

7. Generate the pipeline and run it with the client

mbx-compile can write the same pipeline (configuration, webcam, Scale 224 × 224, the model, Image Rendering) for you:

On your PC:

pc$ mbx-compile pipeline mobilenet_v2_onnx --server ws://<board-ip>:8080 -o mobilenet_v2_pipeline.json

Load the file in the editor (Manage → import), or run it with the MuseBox Client (chapter 5):

On your PC, Linux / macOS:

pc$ cd ~/musebox/musebox-client/python
pc$ source .venv/bin/activate
pc$ python main.py --json ~/musebox/musebox-compiler/mobilenet_v2_pipeline.json --source 0

On your PC, Windows 10 / 11 (Command Prompt or PowerShell):

pc$ cd C:\Users\<your-user>\musebox\musebox-client\python
pc$ .venv\Scripts\activate
pc$ python main.py --json C:\Users\<your-user>\musebox\musebox-compiler\mobilenet_v2_pipeline.json --source 0

--source 0 is the PC webcam; a video file or an image works too. The client shows the video with the best classes and their probabilities.

The configuration node of the generated pipeline has Board set to CPU: that is expected for .tflite and .onnx models.

Your own C++ code

The built-in classification is enough for MobileNetV2. When a model needs something different, add your own C++ code to the package. The example folder has a template for each side:

  • preprocess.cpp prepares the image: here it resizes the frame to 224 × 224 and converts it to RGB, exactly what the AI server already does. The AI server still applies the normalization of the manifest to the image it returns.
  • postprocess.cpp turns the 1000 scores into the result: softmax, the best classes, their names.

This is the core of postprocess.cpp:

extern "C" nlohmann::json postprocess(
    const std::map<std::string, RawOutput>& raw_outputs,
    const nlohmann::json& request_msg,
    const cv::Mat& frame,
    const PreprocessMeta& meta,
    const nlohmann::json& manifest
) {
    ...
    const auto& first = raw_outputs.begin()->second;
    std::vector<float> logits;
    musebox::dequantize(first, logits);
    ...
    return {
        {"task", "classification"},
        {"custom_cpp", true},
        {"predictions", predictions}
    };
}

Add the files with --preprocess and --postprocess when you create the package:

On your PC:

pc$ mbx-compile init --model examples/mobilenet_v2_onnx/mobilenetv2-12-int8.onnx --name mobilenet_v2_onnx --task classification --labels examples/mobilenet_v2_onnx/labels.txt --num-threads 4 --preprocess examples/mobilenet_v2_onnx/preprocess.cpp --postprocess examples/mobilenet_v2_onnx/postprocess.cpp

init rewrote manifest.json: do step 3 again (the normalization), then steps 4 and 5. The board compiles the .cpp files the first time it loads the model, and with a postprocess.cpp the AI server returns exactly what your code returns.

The board uses any preprocess.cpp or postprocess.cpp it finds in the package folder, even if you did not pass it to init. To go back to the built-in processing, create the package without them, and also delete them on the board, because deploy does not remove old files:

On the board:

board$ rm -f /root/musebox-ai-servers/ai_models/mobilenet_v2_onnx/*process.*

The functions you can write and the helpers available are described in Custom pre- and post-processing.

All the commands

On your PC, Linux / macOS:

pc$ cd ~/musebox/musebox-compiler
pc$ source .venv/bin/activate
pc$ mbx-compile inspect-cpu examples/mobilenet_v2_onnx/mobilenetv2-12-int8.onnx
pc$ mbx-compile init --model examples/mobilenet_v2_onnx/mobilenetv2-12-int8.onnx --name mobilenet_v2_onnx --task classification --labels examples/mobilenet_v2_onnx/labels.txt --num-threads 4
# edit mobilenet_v2_onnx/manifest.json: ImageNet normalization (step 3)
pc$ mbx-compile verify mobilenet_v2_onnx/
pc$ mbx-compile package mobilenet_v2_onnx/ -o mobilenet_v2_onnx.mbx
pc$ mbx-compile deploy mobilenet_v2_onnx.mbx --remote root@<board-ip> --dest /root/musebox-ai-servers/ai_models
# restart the AI server on the board
pc$ mbx-compile pipeline mobilenet_v2_onnx --server ws://<board-ip>:8080 -o mobilenet_v2_pipeline.json

On your PC, Windows 10 / 11 (Command Prompt or PowerShell):

pc$ cd C:\Users\<your-user>\musebox\musebox-compiler
pc$ .venv\Scripts\activate
pc$ mbx-compile inspect-cpu examples/mobilenet_v2_onnx/mobilenetv2-12-int8.onnx
pc$ mbx-compile init --model examples/mobilenet_v2_onnx/mobilenetv2-12-int8.onnx --name mobilenet_v2_onnx --task classification --labels examples/mobilenet_v2_onnx/labels.txt --num-threads 4
# edit mobilenet_v2_onnx/manifest.json: ImageNet normalization (step 3)
pc$ mbx-compile verify mobilenet_v2_onnx/
pc$ mbx-compile package mobilenet_v2_onnx/ -o mobilenet_v2_onnx.mbx
pc$ mbx-compile deploy mobilenet_v2_onnx.mbx --remote root@<board-ip> --dest /root/musebox-ai-servers/ai_models
# restart the AI server on the board
pc$ mbx-compile pipeline mobilenet_v2_onnx --server ws://<board-ip>:8080 -o mobilenet_v2_pipeline.json