Package specification

A MuseBox package holds your model and the instructions the AI server needs to run it. You normally create it with mbx-compile init (chapter 6), which writes everything for you. This page describes what is inside, for when you want to check or change it by hand.

What a package contains

my_model/
├── manifest.json       how to run the model (required)
├── model.hef           the model: .hef (Hailo NPU), .tflite or .onnx (CPU)
├── labels.txt          class names, one per line (optional)
├── preprocess.cpp      your own image preparation (optional)
├── postprocess.cpp     your own decoding of the output (optional)
└── musebox_plugin.hpp  added by mbx-compile package when there are .cpp files

mbx-compile package turns the folder into a single .mbx file, and mbx-compile deploy copies it into /root/musebox-ai-servers/ai_models/ on the board: use them. If you copy the folder yourself (scp -r my_model root@<board-ip>:/root/musebox-ai-servers/ai_models/) and it has .cpp files, copy musebox-compiler/include/musebox_plugin.hpp into it too, because only package adds it. In either case the AI server unpacks .mbx files and loads every folder that has a manifest.json when it starts.

The .cpp files are compiled on the board the first time the model is loaded. See Custom pre- and post-processing.

manifest.json

An example, for a ResNet-50 classifier on the Hailo NPU:

{
  "format_version": "1.0",
  "model_name": "resnet_v1_50",
  "task_type": "classification",
  "backend": "hailo",
  "hef_file": "resnet_v1_50.hef",
  "input": {
    "shape": [224, 224, 3],
    "format": "RGB"
  },
  "postprocessing": {
    "top_k": 5,
    "apply_softmax": false,
    "labels_file": "labels.txt"
  }
}

The model

Field	Description
format_version	Always "1.0".
model_name	The model's name: the editor, the clients and /models show it. Letters, digits and _.
task_type	What the model does: see Tasks.
backend	Where it runs: "hailo" (NPU), "tflite" or "onnx" (CPU).
hef_file / tflite_file / onnx_file	The model file, for the chosen backend.
version, author, description	Free text, for you.

The input image

Field	Description
input.shape	The size of the model's input, for example [224, 224, 3] or [1, 3, 224, 224]. Both channel orders work.
input.format	The colour order the model expects: "RGB" (default), "BGR" or "GRAY".
input.data_type	CPU models only: "FLOAT32" (default), or "UINT8" / "INT8" for quantized models.
input.normalization	CPU models with float input only: {"mean": [...], "std": [...]}, one value per channel. Each pixel becomes (pixel - mean) / std. If all the values are at most 1, the pixel is first divided by 255.
preprocessing.type	How the frame is resized to the input: "standard" (default) stretches it, "letterbox" keeps the proportions and fills the rest with grey.

Models on the Hailo NPU receive the 8-bit pixels directly: the normalization is part of the .hef.

The result

Field	Description
postprocessing.labels_file	The class names file, default labels.txt.
postprocessing.confidence_threshold	Detection: results below this score are dropped. A client can change it per request.
postprocessing.max_detections	Detection: at most this many results per frame.
postprocessing.top_k	Classification: how many classes to return.
postprocessing.apply_softmax	Classification: true if the model outputs raw scores rather than probabilities.
preprocessing.custom_params, postprocessing.custom_params	Your own settings, read by your .cpp files.

CPU models

Field	Description
runtime.num_threads	How many CPU cores to use. 0 (default) chooses automatically.

mbx-compile also writes other fields (outputs, tensor_name, nms_threshold, the rest of runtime): they describe the model, and the AI server does not need them.

Tasks

Without a postprocess.cpp, the AI server decodes the output with its built-in post-processing, chosen by task_type. It works only when the model's output already has the format below:

task_type	The model's output must be	What the AI server returns
object_detection	One tensor with the final boxes, one row per object: [ymin, xmin, ymax, xmax, score] or [ymin, xmin, ymax, xmax, score, class], coordinates from 0 to 1 or in pixels. Hailo models compiled with NMS on the chip produce exactly this.	A box, a label and a score for each object (confidence_threshold, max_detections).
classification	One vector with a score per class, in the order of labels.txt (probabilities, or raw scores with apply_softmax: true).	The top_k classes, with label and probability.
segmentation	One tensor with a score per class for each pixel (the highest wins), or with the class number of each pixel: height × width × classes on Hailo and TFLite, classes × height × width on ONNX.	The class of each pixel (class_mask_rle, width, height) and the classes found (classes_detected). The editor and the clients draw it over the image.
feature_extraction	One vector (embedding).	The vector (embedding, dim), and every output tensor as for custom.
custom	Anything: no decoding.	The values of every output tensor (tensors: shape, dtype, values). A tensor above 65 536 values comes without values (values_omitted).

NMS (non-maximum suppression) removes the overlapping boxes of a detector. If mbx-compile inspect shows a single output that already lists the boxes (5 or 6 values each), NMS runs on the chip and the built-in object_detection works; otherwise you need a postprocess.cpp.

object_detection, classification and segmentation read only the first output tensor. Any other output (for example YOLO before NMS, SSD anchors, several heads, depth maps) needs a postprocess.cpp that decodes it; with custom you get the raw values, but nothing in the editor or the clients knows how to show them.

The segmentation mask is sent run-length encoded ("encoding": "rle"): every run of pixels of the same class is 5 bytes, the class (uint8) and the length (uint32, little-endian), row by row, in base64. A request with "mask_encoding": "raw" gets class_mask instead: one byte per pixel. Class numbers above 255 are sent as 255.

With a postprocess.cpp, the AI server returns whatever your code produces, for any task_type.