Skip to content

Advanced: your own nodes in the editor

The Custom Model node of the editor draws the usual results of a model: bounding boxes with label and confidence, keypoints, and the top classes of a classification. When your post-processing (custom pre- and post-processing) returns something else (an image, a map, values to show in your own way), add a node of your own to the editor that turns it into what you want to see.

How it works

What your postprocess() returns arrives in the data field of the server's response. The model node passes the whole response on its first output, predictions / data:

{"status": "success", "topic": "my_model", "data": { ...what postprocess() returned... }, "execution_time": 10.3}

Your node reads it from its input, builds what you want to see, and passes it to its output: an image, for example, which you connect to MuseBox Output → Image Rendering.

A node is a JavaScript file that registers a node type with LiteGraph, the graph library of the editor: its inputs and outputs, and onExecute(), called at every step of the flow.

Add a node

The editor folder is ~/musebox/no-code-flow-editor (C:\Users\<your-user>\musebox\no-code-flow-editor on Windows), the folder where you run serve.py.

  1. In the editor folder, create nodes/user/ and your file in it, for example nodes/user/result-image-view.js (the example below).

  2. Open index.html in the editor folder and add a line for your file after the line of custom-model.js:

    <script type="text/javascript" src="/nodes/musebox-tasks/custom-model.js"></script>
<script type="text/javascript" src="/nodes/user/result-image-view.js"></script>

  3. Reload the editor page. The node is in right-click → Add Node → My Nodes.

  4. Connect it: model node predictions / data → your node → Image Rendering.

The browser console (F12) shows the errors of your file, if any.

Example: show an image returned by the post-processing

The post-processing draws the image itself in C++ with OpenCV (a colour map, overlays, text) and returns it as a JPEG in base64, in the field image of its result:

// in postprocess(): `view` is the cv::Mat (BGR) you want to see
std::vector<unsigned char> jpeg;
cv::imencode(".jpg", view, jpeg, {cv::IMWRITE_JPEG_QUALITY, 90});
return {
    {"image", to_base64(jpeg)},      // standard base64, without the "data:" prefix
    {"width", view.cols},
    {"height", view.rows}
};

to_base64 is not part of the SDK: copy this function into postprocess.cpp, above postprocess():

static std::string to_base64(const std::vector<unsigned char>& in) {
    static const char* t = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
    std::string out;
    out.reserve((in.size() + 2) / 3 * 4);
    size_t i = 0;
    for (; i + 2 < in.size(); i += 3) {
        const unsigned v = (in[i] << 16) | (in[i + 1] << 8) | in[i + 2];
        out += t[v >> 18]; out += t[(v >> 12) & 63]; out += t[(v >> 6) & 63]; out += t[v & 63];
    }
    if (i < in.size()) {
        unsigned v = in[i] << 16;
        if (i + 1 < in.size()) v |= in[i + 1] << 8;
        out += t[v >> 18]; out += t[(v >> 12) & 63];
        out += (i + 1 < in.size()) ? t[(v >> 6) & 63] : '=';
        out += '=';
    }
    return out;
}

A JPEG is also much lighter than the raw values: a 224 × 224 image is about 15 KB, the same 50 176 values as JSON numbers about 0.5 MB per frame.

The editor node decodes the image and passes it on (nodes/user/result-image-view.js):

// Result Image View: shows an image (base64 JPEG / PNG) that a custom post-processing returns in its result.
// Input: "predictions / data" of the model node. Output: the image, for Image Rendering.
(function (global) {
    var LiteGraph = global.LiteGraph;

    function ResultImageView() {
        this.addInput("model data", "array");
        this.addOutput("image", "image");
        this.properties = { field: "image" };          // the key of the image inside "data"
        this.addWidget("text", "Field", this.properties.field, "field");
        this.image = null;                              // the last decoded image
        this.lastB64 = null;
    }
    ResultImageView.title = "Result Image View";
    ResultImageView.desc = "Shows an image (base64 JPEG / PNG) returned by a custom post-processing";

    ResultImageView.prototype.onExecute = function () {
        var response = this.getInputData(0);
        var b64 = response && response.data && response.data[this.properties.field];
        if (typeof b64 === "string" && b64 !== this.lastB64) {  // a new result: decode it
            this.lastB64 = b64;
            var img = new Image();
            var node = this;
            img.onload = function () { node.image = img; };
            img.src = (b64.indexOf("data:") === 0 ? "" : "data:image/jpeg;base64,") + b64;
        }
        if (this.image) this.setOutputData(0, this.image);
    };

    LiteGraph.registerNodeType("My Nodes/Result Image View", ResultImageView);
})(this);

The Field widget of the node chooses the key of the image in data, if your post-processing uses another name. To draw on the camera frame instead, give the node a second input for the frame and draw both on a <canvas> (the Scale node in nodes/musebox-operators.js shows how to create one).

Your application: the client does not know your node

The MuseBox Client (chapter 5) runs only the MuseBox Tasks nodes of a saved pipeline: your nodes are ignored, and nothing of what they do in the editor reaches your application by itself. For a model it does not know, the client draws only an image found in data.image, blended over the frame; any other display must be written again in your application:

  • Python client: musebox-client/python/pipeline_runner.py, _run_root_task(), which chooses how to draw each result by its topic;
  • C++ client: musebox-client/cpp/src/pipeline_runner.cpp, the same choice in PipelineRunner::run();
  • your own code: read data from the response (the messages) and draw it as you need.

Editor updates replace index.html

The editor's Update button (chapter 4) replaces the editor's own files, index.html included: after an update, add the line of step 2 again. Your files in nodes/user/ stay, and the previous index.html is kept in .musebox-previous/. A saved pipeline that uses your node opens correctly only when your file is loaded.