Skip to content

Custom pre- and post-processing

The AI server prepares the image and decodes the model's output by itself only for a few standard output formats (see Tasks): the final boxes of a detector with NMS on the chip, the class scores of a classifier, the per-pixel classes of a segmentation, an embedding vector. For any other model, and so for most custom models, you write the post-processing yourself. The same goes for the image, when the model needs something different from a resize, for example a crop. You add your own C++ code to the package:

File Replaces Use it for
preprocess.cpp the built-in resize and colour conversion crops, image enhancement, any other preparation of the image
postprocess.cpp the built-in decoding of the output your own output format: anchors, several heads, scores, embeddings…

You can use one, the other or both. The files are compiled on the board the first time the model is loaded: you do not need a cross-compiler on the PC.

How it works

image from the client or the camera
        │
        ▼
preprocess.cpp   (or the built-in resize, see preprocessing.type)
        │  image of the model's input size
        ▼
model on the Hailo NPU or the CPU
        │  output tensors
        ▼
postprocess.cpp  (or the built-in decoding for the task_type)
        │  JSON
        ▼
"data" field of the reply

Each file exports one function with a fixed name and signature, declared extern "C". Both functions receive the package's manifest.json, so they can read your own settings from it.

Add the files to a package

Pass them to mbx-compile init. They are copied into the package as preprocess.cpp and postprocess.cpp. In this example the model age_gender.tflite, its labels and your two files my_crop.cpp and my_decoder.cpp are in the current folder:

On your PC:

pc$ mbx-compile init --model age_gender.tflite --name age_gender --task custom --input-shape 1,62,62,3 --preprocess my_crop.cpp --postprocess my_decoder.cpp --labels labels.txt

You can also copy them into an existing package folder by hand. The names must be exactly preprocess.cpp and postprocess.cpp, next to manifest.json: the AI server uses them whenever they are there, whatever the manifest says.

Each file starts with:

#include "musebox_plugin.hpp"

mbx-compile package adds musebox_plugin.hpp to the package. It gives you OpenCV, nlohmann/json (nlohmann::json) and the MuseBox types described below. The code is compiled as C++17 and can use OpenCV, nlohmann/json and the C++ standard library; other libraries are not available.

preprocess.cpp

extern "C" cv::Mat preprocess(
    const cv::Mat& frame,
    int target_w,
    int target_h,
    const std::string& color_fmt,
    const nlohmann::json& manifest,
    PreprocessMeta& meta
);
Argument Description
frame The image to process, 8-bit BGR (OpenCV's order). It can be smaller than the image the client sent: the AI server may decode a large image at a reduced size.
target_w, target_h The model's input size, from input.shape.
color_fmt input.format in capitals: "RGB", "BGR" or "GRAY".
manifest The whole manifest.json.
meta Information for postprocess.cpp: see PreprocessMeta.

Return the image the model must receive, already in the colour order of color_fmt. The AI server then:

  • turns a one-channel image into three channels, and any other pixel type into 8 bits;
  • resizes it to target_w × target_h if it has a different size;
  • on the CPU, with a float input, applies input.normalization (see the specification). Do not normalize the image yourself.

The ResNet-50 example crops the centre of the image instead of stretching it:

// Custom center crop preserving aspect ratio on shortest dimension
float scale = std::max(static_cast<float>(target_w) / frame_w, static_cast<float>(target_h) / frame_h);
int new_w = static_cast<int>(std::round(frame_w * scale));
int new_h = static_cast<int>(std::round(frame_h * scale));

cv::Mat resized;
cv::resize(frame, resized, cv::Size(new_w, new_h));

int x = (new_w - target_w) / 2;
int y = (new_h - target_h) / 2;
cv::Mat crop = resized(cv::Rect(x, y, target_w, target_h)).clone();

if (color_fmt == "RGB") {
    cv::cvtColor(crop, crop, cv::COLOR_BGR2RGB);
}
return crop;

(frame_w and frame_h are frame.cols and frame.rows.)

postprocess.cpp

extern "C" nlohmann::json postprocess(
    const std::map<std::string, RawOutput>& raw_outputs,
    const nlohmann::json& request_msg,
    const cv::Mat& frame,
    const PreprocessMeta& meta,
    const nlohmann::json& manifest
);
Argument Description
raw_outputs The model's output tensors, by tensor name: see RawOutput.
request_msg The request as the client sent it, so a client can pass you its own options.
frame The image given to preprocess (BGR, possibly smaller than the client's image).
meta The PreprocessMeta filled by the AI server and by your preprocess.
manifest The whole manifest.json.

Whatever JSON you return becomes the data field of the reply, for any task_type:

{
  "status": "success",
  "topic": "age_gender",
  "clientId": "...",
  "data": { ...what postprocess returned... },
  "execution_time": 3.412,
  "board": "..."
}

If your function throws a C++ exception, the client receives an error reply with its message, and the AI server keeps running.

The age and gender example reads the two outputs of the model (one value for the age, two for the gender) and returns them as named fields:

float uncertainty_thresh = request_msg.value("uncertainty_threshold", 0.1f);

for (const auto& [name, out] : raw_outputs) {
    std::vector<float> values;
    musebox::dequantize(out, values);

    if (values.size() == 1) {
        // Age head: normalized scalar [0, 1] scaled by 100 to years [0, 100]
        age = std::max(0, static_cast<int>(std::round(values[0] * 100.0f)));
    } else if (values.size() == 2) {
        // Gender head: index 0 = female, index 1 = male
        ...
    }
}

return {
    {"task", "age_gender"},
    {"age", age},
    {"gender", gender},
    {"gender_confidence", std::round(gender_conf * 10000.0f) / 10000.0f}
};

The editor's Custom Model node draws bounding boxes, keypoints and classifications. To see any other result in the editor (an image, a map), add a node of your own: Advanced: your own nodes in the editor.

RawOutput

Each output tensor arrives as it leaves the model: 8 or 16 bits for quantized models, float otherwise. musebox::dequantize(out, values) turns any of them into a std::vector<float> with the real values (scale × (value - zp)), so you rarely need the fields below.

Field Description
shape The tensor's dimensions, as the model declares them.
numel() The number of values.
is_f32, is_u16, is_i8 The value type. All false means 8-bit unsigned.
data_f32(), data_u16(), data_i8(), data_u8() Pointer to the values, for the matching type.
scale, zp Quantization scale and zero point.

PreprocessMeta

The AI server fills meta before calling preprocess, and passes it to postprocess. Use it to map your results back to the client's image.

Field Filled by the AI server Description
orig_w, orig_h yes Size of the image the client sent. Return coordinates in this space.
target_w, target_h yes The model's input size.
scale_x, scale_y yes orig_w / target_w and orig_h / target_h.
scale, pad_x, pad_y, net_w, net_h no Free for your preprocess, for example the scale and the padding of a letterbox.

frame can be smaller than the client's image, so scale coordinates to orig_w × orig_h, not to frame.cols × frame.rows.

Settings and files

  • Your own settings: put them in preprocessing.custom_params or postprocessing.custom_params in manifest.json, and read them from manifest:

    const auto ptr = nlohmann::json::json_pointer("/postprocessing/custom_params/min_score");
float min_score = manifest.value(ptr, 0.5f);

  • Options per request: read them from request_msg, with a default, as the example above does with uncertainty_threshold.

  • Files of the package, such as labels.txt: the AI server adds the package folder to manifest, as package_dir, so open them from there:

    std::ifstream f(manifest.value("package_dir", std::string(".")) + "/labels.txt");

The classification examples do this, and load the labels only once into a static variable.

Compile and run

Package and copy the model as usual (chapter 6), from the folder where you ran init:

On your PC:

pc$ mbx-compile package age_gender/ -o age_gender.mbx
pc$ mbx-compile deploy age_gender.mbx --remote root@<board-ip> --dest /root/musebox-ai-servers/ai_models

Then restart the AI server on the board: press Ctrl+C in the ssh terminal where it runs (or, if it runs in the background, run pkill -f musebox-ai-server on the board, pkill -f musebox_ai for the Python server, as in chapter 3), then start it again. When it loads the model it compiles the files (about one or two minutes for each file; the server answers only when it has finished), and prints:

[PackageLoader] Auto-compiling native C++ plugin: .../postprocess.cpp -> .../postprocess.so
[PackageLoader] Successfully compiled: .../postprocess.so
Loaded custom C++ postprocess plugin from: .../postprocess.so

The .so files stay in the package folder; the next start compiles again only if a .cpp file is newer.

If the compilation fails, the AI server prints, in the terminal where it runs (or in server.log if you started it in the background), Compilation failed for ... followed by the first lines of the compiler's errors, and runs the model without your code, with the built-in processing. Always check these lines after a change. Fix the file, deploy again and restart.

If a change does not seem to take effect (for example because the board's clock is behind the PC's), delete the old .so files on the board and restart:

On the board:

board$ rm /root/musebox-ai-servers/ai_models/age_gender/*.so

Complete examples

Each tutorial ships a working preprocess.cpp and postprocess.cpp: