Skip to content

5. Running a flow from your code

The MuseBox Client runs a pipeline exported from the editor without the browser. It reads the .json file, sends the frames of a camera, video or image to the board and draws the results. Use it as a ready program, or as the starting point of your own application.

It exists in Python and in C++, and both read the same .json files.

Download

Download the client you need into the musebox folder of chapter 4: ~/musebox on Linux / macOS, C:\Users\<your-user>\musebox on Windows.

On your PC, Linux / macOS:

pc$ mkdir -p ~/musebox
pc$ curl -L https://s3.eu-central-1.wasabisys.com/musebox/SDK-RELEASES/MUSEBOX_CLIENT/PYTHON/musebox-client-python-latest.tar.gz | tar -xz -C ~/musebox
pc$ curl -L https://s3.eu-central-1.wasabisys.com/musebox/SDK-RELEASES/MUSEBOX_CLIENT/CPP/musebox-client-cpp-latest.tar.gz | tar -xz -C ~/musebox

On your PC, Windows 10 / 11 (Command Prompt or PowerShell):

pc$ cd C:\Users\<your-user>\musebox
pc$ curl.exe -LO https://s3.eu-central-1.wasabisys.com/musebox/SDK-RELEASES/MUSEBOX_CLIENT/PYTHON/musebox-client-python-latest.tar.gz
pc$ tar -xzf musebox-client-python-latest.tar.gz
pc$ curl.exe -LO https://s3.eu-central-1.wasabisys.com/musebox/SDK-RELEASES/MUSEBOX_CLIENT/CPP/musebox-client-cpp-latest.tar.gz
pc$ tar -xzf musebox-client-cpp-latest.tar.gz

They create musebox-client/python, musebox-client/cpp and musebox-common, the shared files both clients need: keep musebox-common next to musebox-client.

Download only the one you need, if you like. On Windows you can also download musebox-client-python-latest.tar.gz or musebox-client-cpp-latest.tar.gz with the browser into C:\Users\<your-user>\musebox and run only the cd and tar lines.

Python

Install it in its own Python environment (a venv), once:

On your PC, Linux / macOS:

pc$ cd ~/musebox/musebox-client/python
pc$ python3 -m venv .venv
pc$ source .venv/bin/activate
pc$ pip install -r requirements.txt

On your PC, Windows 10 / 11 (Command Prompt or PowerShell):

pc$ cd C:\Users\<your-user>\musebox\musebox-client\python
pc$ python -m venv .venv
pc$ .venv\Scripts\activate
pc$ pip install -r requirements.txt

On Debian / Ubuntu, if venv says ensurepip is not available, install it once with sudo apt install python3-venv and run the command again. If PowerShell refuses to run activate because running scripts is disabled, allow it once with Set-ExecutionPolicy -Scope CurrentUser RemoteSigned, or use Command Prompt.

The client runs a pipeline you exported from the editor (chapter 4, Manage → export). The browser saved it in your Downloads folder as <name>.json, for example face_detection_pipeline.json. In the commands of this chapter, my_pipeline.json stands for that file: copy it into the client's folder (~/musebox/musebox-client/python, C:\Users\<your-user>\musebox\musebox-client\python on Windows), or give its full path, for example --json ~/Downloads/face_detection_pipeline.json.

Make sure the AI server is running on the board, then run the pipeline with the venv active (activate it again in every new terminal, as above):

Stop the pipeline in the editor first

A camera works with one program at a time. Before you start the client, turn off the pipeline in the No-Code Flow Editor, so that it frees the camera:

  • press Stop in the editor's toolbar: the browser releases the PC webcam, and the board ends the editor's board camera stream;
  • if the pipeline is still playing, the client cannot open the PC webcam, and with the board camera in stream mode the editor and the client keep taking the stream from each other (the board runs one stream at a time).

If the client still cannot open the PC webcam, your editor is an older version that keeps it after Stop: close the editor's tab, and update the editor (chapter 4).

On your PC:

pc$ python main.py --json my_pipeline.json

A window shows the video with the results. To stop, click the window and press Q or Esc. With --no-show there is no window: press Ctrl+C in the terminal on your PC where main.py runs.

Recorded on the Mercury+ PE3 Hailo-8 (Mercury+ MP1 SoM)

The server address and the board come from the configuration node of the pipeline. Pass --url to override the address:

On your PC:

pc$ python main.py --json my_pipeline.json --url ws://<board-ip>:8080

Useful options

Option Meaning
--source 0 camera index, video file or image (default: the one in the pipeline, else the webcam 0)
--source board-camera the camera attached to the board; only a preview comes back
--source board-stream the board runs the whole pipeline on its camera and sends the results (fastest)
--save out.mp4 write the annotated video
--no-show no window (servers, scripts)
--raw-frames send uncompressed frames: faster on a wired network
--zmq, --mqtt, --rabbitmq, --ros2 use another protocol instead of WebSocket
All the options (python main.py --help) 
usage: main.py [-h] --json JSON [--source SOURCE]
               [--preview-max-side PREVIEW_MAX_SIDE] [--max-frames MAX_FRAMES]
               [--url URL] [--board BOARD] [--client-id CLIENT_ID] [--no-show]
               [--save SAVE] [--image-size IMAGE_SIZE] [--flip] [--raw-frames]
               [--zmq] [--zmq-url ZMQ_URL] [--mqtt] [--mqtt-host MQTT_HOST]
               [--mqtt-port MQTT_PORT]
               [--mqtt-request-topic MQTT_REQUEST_TOPIC]
               [--mqtt-response-topic MQTT_RESPONSE_TOPIC] [--mqtt-raw]
               [--mqtt-username MQTT_USERNAME] [--mqtt-password MQTT_PASSWORD]
               [--rabbitmq] [--rabbitmq-host RABBITMQ_HOST]
               [--rabbitmq-port RABBITMQ_PORT]
               [--rabbitmq-vhost RABBITMQ_VHOST]
               [--rabbitmq-username RABBITMQ_USERNAME]
               [--rabbitmq-password RABBITMQ_PASSWORD]
               [--rabbitmq-request-queue RABBITMQ_REQUEST_QUEUE]
               [--rabbitmq-exchange RABBITMQ_EXCHANGE] [--rabbitmq-raw]
               [--ros2] [--ros2-node-name ROS2_NODE_NAME]
               [--ros2-request-topic ROS2_REQUEST_TOPIC]
               [--ros2-response-topic ROS2_RESPONSE_TOPIC]
               [--ros2-domain-id ROS2_DOMAIN_ID]
               [--ros2-qos {default,sensor_data,reliable,best_effort}]

MuseBox Python Client - load a no-code flow JSON, query the MuseBox AI server
and run the real-time pipeline.

options:
  -h, --help            show this help message and exit
  --json JSON, -j JSON  Flow JSON exported from the no-code-flow-editor
  --source SOURCE, -s SOURCE
                        Camera index, video file, image file, 'board-camera'
                        (grab frames on the board per tick) or 'board-stream'
                        (the board runs the whole pipeline and pushes results;
                        WebSocket only) (default: from JSON or 0)
  --preview-max-side PREVIEW_MAX_SIDE
                        Board camera only: longest side of the preview JPEG
                        sent back for display (default: 320)
  --max-frames MAX_FRAMES
                        Stop after N frames (0 = run until the stream ends)
  --url URL, -u URL     Override server URL: ws://, tcp:// (ZeroMQ), mqtt://,
                        amqp:// (RabbitMQ) or ros2://domain/<id>
  --board BOARD, -b BOARD
                        Override the board profile name (default: from JSON)
  --client-id CLIENT_ID
                        Client identifier sent with every request
  --no-show             Disable the GUI window (headless)
  --save SAVE           Save the annotated video to this file (e.g.
                        output.mp4)
  --image-size IMAGE_SIZE
                        Override the resize resolution (WxH, e.g. 640x480)
  --flip, --flip-camera
                        Flip the input horizontally
  --raw-frames          Send uncompressed BGR frames (WebSocket / ZeroMQ
                        multipart): no JPEG decode on the board; best on a
                        wired LAN

ZeroMQ:
  --zmq                 Use ZeroMQ instead of WebSocket
  --zmq-url ZMQ_URL     ZeroMQ endpoint (default: tcp://<flow server
                        host>:5555)

MQTT:
  --mqtt                Use MQTT instead of WebSocket
  --mqtt-host MQTT_HOST
                        MQTT broker host (default: the flow's server host)
  --mqtt-port MQTT_PORT
  --mqtt-request-topic MQTT_REQUEST_TOPIC
  --mqtt-response-topic MQTT_RESPONSE_TOPIC
  --mqtt-raw            Send raw JPEG frames instead of JSON/base64
  --mqtt-username MQTT_USERNAME
                        MQTT broker user (default: anonymous)
  --mqtt-password MQTT_PASSWORD
                        MQTT broker password

RabbitMQ:
  --rabbitmq            Use RabbitMQ instead of WebSocket
  --rabbitmq-host RABBITMQ_HOST
                        RabbitMQ broker host (default: the flow's server host)
  --rabbitmq-port RABBITMQ_PORT
  --rabbitmq-vhost RABBITMQ_VHOST
  --rabbitmq-username RABBITMQ_USERNAME
  --rabbitmq-password RABBITMQ_PASSWORD
  --rabbitmq-request-queue RABBITMQ_REQUEST_QUEUE
  --rabbitmq-exchange RABBITMQ_EXCHANGE
  --rabbitmq-raw        Send raw JPEG frames instead of JSON/base64

ROS 2:
  --ros2                Use ROS 2 instead of WebSocket
  --ros2-node-name ROS2_NODE_NAME
  --ros2-request-topic ROS2_REQUEST_TOPIC
  --ros2-response-topic ROS2_RESPONSE_TOPIC
  --ros2-domain-id ROS2_DOMAIN_ID
  --ros2-qos {default,sensor_data,reliable,best_effort}

MQTT, RabbitMQ, ROS 2

What to install

Protocol On the board On your PC
MQTT nothing: the server has MQTT built in, and the board has its own broker (mosquitto) nothing: the Python client's requirements.txt installs paho-mqtt
RabbitMQ nothing: the server has RabbitMQ built in, but the board has no broker a RabbitMQ broker, on your PC or on another computer that the board and the PC both reach (the Python client's requirements.txt installs pika)
ROS 2 nothing: the board has ROS 2 Humble ROS 2 Humble, the same version as the board

The C++ client also needs the MQTT and RabbitMQ libraries to build, whatever protocol you use: see C++.

RabbitMQ broker. Install it and create the user the server and the client log in with (musebox / musebox, from config/rabbitmq.json). RabbitMQ's own guest user only works from the broker's computer, so the board cannot use it.

On your PC, Linux / macOS:

pc$ sudo apt install rabbitmq-server
pc$ sudo rabbitmqctl add_user musebox musebox
pc$ sudo rabbitmqctl set_permissions -p / musebox ".*" ".*" ".*"

The first line is for Debian / Ubuntu, and it also starts the broker. On macOS, with Homebrew, run instead (without sudo):

pc$ brew install rabbitmq
pc$ brew services start rabbitmq
pc$ rabbitmqctl add_user musebox musebox
pc$ rabbitmqctl set_permissions -p / musebox ".*" ".*" ".*"

On your PC, Windows 10 / 11: install Erlang and RabbitMQ with the installer from rabbitmq.com (it starts RabbitMQ as a Windows service), then open RabbitMQ Command Prompt (sbin dir) from the Start menu as administrator and run:

pc$ rabbitmqctl add_user musebox musebox
pc$ rabbitmqctl set_permissions -p / musebox ".*" ".*" ".*"

The broker listens on port 5672: allow it in the PC's firewall, so that the board can connect.

ROS 2 Humble. Follow the official guide, docs.ros.org/en/humble/Installation.html: binary packages exist for Ubuntu 22.04 and Windows 10; on macOS ROS 2 has to be built from source. The board and the PC must use the same ROS domain (domain_id in config/ros2.json, 0 by default, and the ROS_DOMAIN_ID variable on the PC, also 0 when not set) and be on the same network.

Using them

These protocols are off on the board by default. Start the server with the matching option (for example /root/musebox-ai-servers/musebox-ai-server-cpp/musebox-ai-server --mqtt --mqtt-host <broker-ip>), or set them in config/mqtt.json, config/rabbitmq.json or config/ros2.json. Then run the client with the same option.

MQTT and RabbitMQ go through a broker: the board and your PC must both reach it, so give its network address (not localhost) to the server and to the client:

On your PC:

pc$ python main.py --json my_pipeline.json --mqtt --mqtt-host <broker-ip>
pc$ python main.py --json my_pipeline.json --rabbitmq --rabbitmq-host <broker-ip>

RabbitMQ: the server and the client log in as musebox / musebox, the user created above. To use another user, set it in config/rabbitmq.json on the board and pass it to the client with --rabbitmq-username and --rabbitmq-password.

The board has its own MQTT broker (mosquitto), but it only accepts connections from the board itself. To use it from your PC, open it to the network: add these two lines at the end of /etc/mosquitto/mosquitto.conf on the board, then restart it:

listener 1883
allow_anonymous true

On the board:

board$ systemctl restart mosquitto

Then use --mqtt-host <board-ip> on the client. Both lines are needed: listener 1883 accepts connections from the network, and allow_anonymous true lets in the server and the client, which connect without a user name. Anyone on your network can then send requests to the board and read the results. On a shared network, protect the broker with a user and password instead of allow_anonymous true (mosquitto's password_file), and give them to the server (--mqtt-username, --mqtt-password, or username / password in config/mqtt.json) and to the client (--mqtt-username, --mqtt-password).

For --ros2, the client uses the ROS 2 Python packages (rclpy) of your ROS 2 installation, which a venv does not see. Load ROS 2 first and create the venv with access to them (in place of the venv of Python):

On your PC, Linux / macOS:

pc$ cd ~/musebox/musebox-client/python
pc$ source /opt/ros/humble/setup.bash
pc$ python3 -m venv --system-site-packages .venv
pc$ source .venv/bin/activate
pc$ pip install -r requirements.txt

On your PC, Windows 10 / 11 (Command Prompt only, with ROS 2 installed in C:\dev\ros2_humble as in the ROS 2 guide):

pc$ cd C:\Users\<your-user>\musebox\musebox-client\python
pc$ call C:\dev\ros2_humble\local_setup.bat
pc$ python -m venv --system-site-packages .venv
pc$ .venv\Scripts\activate
pc$ pip install -r requirements.txt

Load ROS 2 again (source /opt/ros/humble/setup.bash, or call C:\dev\ros2_humble\local_setup.bat) in every new terminal, before activating the venv.

ZeroMQ is on by default, on port 5555:

On your PC:

pc$ python main.py --json my_pipeline.json --zmq --zmq-url tcp://<board-ip>:5555

With the C++ client the options are the same. Here the server runs with --mqtt, --rabbitmq and --ros2, and the client runs the face detection pipeline over each protocol in turn:

Recorded on the Mercury+ PE3 Hailo-8 (Mercury+ MP1 SoM)

C++

The C++ client needs a C++17 compiler, CMake 3.16 or newer, OpenCV 4, Boost 1.74 or newer, and the ZeroMQ, MQTT (mosquitto) and RabbitMQ (rabbitmq-c) libraries. All of them are needed to build, even if you only use WebSocket. ROS 2 is optional: the --ros2 option is built in only when ROS 2 is loaded before cmake.

Install the libraries, once.

On your PC, Linux / macOS:

pc$ sudo apt install build-essential cmake libopencv-dev libboost-system-dev libboost-thread-dev libzmq3-dev libmosquitto-dev librabbitmq-dev   # Debian / Ubuntu
pc$ brew install cmake opencv boost zeromq mosquitto rabbitmq-c                                                                                  # macOS (Homebrew)

On your PC, Windows 10 / 11: install Visual Studio 2022 (Community, or the Build Tools) with the Desktop development with C++ workload, which includes the compiler and CMake, and Git for Windows. Then open x64 Native Tools Command Prompt for VS 2022 from the Start menu and install the libraries with vcpkg. vcpkg compiles them, so the last command takes about an hour the first time:

pc$ git clone https://github.com/microsoft/vcpkg C:\vcpkg
pc$ C:\vcpkg\bootstrap-vcpkg.bat
pc$ C:\vcpkg\vcpkg install opencv4 boost-beast boost-system boost-thread zeromq mosquitto librabbitmq nlohmann-json --triplet x64-windows

Build and run it. As for the Python client, first stop the pipeline in the editor (Stop), so that the camera is free.

On your PC, Linux / macOS:

pc$ cd ~/musebox/musebox-client/cpp
pc$ cmake -B build && cmake --build build -j
pc$ ./build/musebox-client-cpp --json my_pipeline.json

On your PC, Windows 10 / 11 (x64 Native Tools Command Prompt for VS 2022):

pc$ cd C:\Users\<your-user>\musebox\musebox-client\cpp
pc$ cmake -B build -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake
pc$ cmake --build build --config Release
pc$ build\Release\musebox-client-cpp.exe --json my_pipeline.json

my_pipeline.json is the pipeline exported from the editor, as for the Python client: copy it into the cpp folder or give its full path. For --ros2, load ROS 2 before cmake (source /opt/ros/humble/setup.bash, or call C:\dev\ros2_humble\local_setup.bat on Windows): without it the client is built without ROS 2.

The build copies the libraries' DLLs next to musebox-client-cpp.exe: run it from there, or copy the whole build\Release folder.

As with the Python client, to stop click the window and press Q or Esc. With --no-show, press Ctrl+C in the terminal on your PC where musebox-client-cpp runs.

Recorded on the Mercury+ PE3 Hailo-8 (Mercury+ MP1 SoM)

The options are those of the Python client, except --raw-frames, --max-frames and the board-camera / board-stream sources, which only the Python client has.

All the options (musebox-client-cpp --help) 
MuseBox C++ Client v1.1.4
Usage: ./build/musebox-client-cpp --json <flow.json> [options]
Options:
  --json, -j <path>         Path to flow JSON file exported by the no-code-flow-editor (required)
  --source, -s <src>        Video source: camera index (e.g. 0), video or image file path
  --url, -u <url>           Override server URL (ws://, tcp://, mqtt://, amqp://, ros2://domain/<id>)
  --client-id <id>          Client identifier sent with every request (default: 1)
  --ros2                    Use ROS 2 protocol interface instead of WebSocket
  --ros2-node-name <name>   ROS 2 client node name
  --ros2-request-topic <t>  ROS 2 request topic (default: musebox/request)
  --ros2-response-topic <t> ROS 2 response topic (default: musebox/response)
  --ros2-domain-id <id>     ROS 2 domain ID (default: 0)
  --ros2-qos <profile>      QoS profile: default, sensor_data, reliable, best_effort
  --zmq                     Use ZeroMQ protocol interface instead of WebSocket
  --zmq-url <endpoint>      ZeroMQ server endpoint (default: tcp://127.0.0.1:5555)
  --mqtt                    Use MQTT protocol interface instead of WebSocket
  --mqtt-host <host>        MQTT broker host (default: 127.0.0.1)
  --mqtt-port <port>        MQTT broker port (default: 1883)
  --mqtt-request-topic <t>  MQTT request topic (default: musebox/request)
  --mqtt-response-topic <t> MQTT response topic (default: musebox/response)
  --mqtt-raw                Publish raw JPEG binary frames over MQTT
  --mqtt-username <usr>     MQTT broker user (default: anonymous)
  --mqtt-password <pwd>     MQTT broker password
  --rabbitmq                Use RabbitMQ protocol interface instead of WebSocket
  --rabbitmq-host <host>    RabbitMQ broker host (default: 127.0.0.1)
  --rabbitmq-port <port>    RabbitMQ broker port (default: 5672)
  --rabbitmq-vhost <vhost>  RabbitMQ virtual host (default: /)
  --rabbitmq-username <usr> RabbitMQ username (default: musebox)
  --rabbitmq-password <pwd> RabbitMQ password (default: musebox)
  --rabbitmq-request-queue  RabbitMQ request queue (default: musebox_requests)
  --rabbitmq-exchange <ex>  RabbitMQ exchange name (default: '')
  --rabbitmq-raw            Publish raw JPEG binary frames over RabbitMQ
  --board, -b <board>       Override board hardware profile (e.g. MP1 ST1)
  --no-show                 Disable visual GUI window
  --save <path.mp4>         Save annotated video output file
  --image-size <WxH>        Override image resize resolution (e.g. 640x480)
  --flip, --flip-camera     Flip input frame horizontally
  --help, -h                Show this help

The messages, for your own application

Every protocol carries the same JSON.

A request:

{"topic": "FaceDetection", "clientId": "1", "board": "Mercury+ PE3 Hailo",
 "imageWidth": 640, "imageHeight": 480, "image": "<base64 JPEG>"}

The answer:

{"status": "success", "topic": "FaceDetection", "data": [...], "execution_time": 12.3, "board": "pluto-hailo"}

board in the answer is the board name set on the board in config/models.json.

or, on error:

{"status": "error", "error": "..."}

topic is the task. The tasks your board offers are listed at http://<board-ip>:8080/models.

Next: 6. Your own models