libZMP integration in 3rd-party frameworks

[josuah]

My motivation for this is replacing the DDR3 RAM chip of Linux-based cameras by using Zephyr instead of Linux (and an MCU instead of an MPU eventually) as well as shrink video systems in various metrics.

In Linux world, pipelines are present through many different frameworks. In Zephyr, because we start ex-nihilo, it is possible to use libZMP as a single pipeline framework on top of which build many things.

None of the below is currently supported out of the box, and at the same time, everything becomes doable once libZMP is merged. You would need to either be patient or reach in for how to accelerate things and contribute. :)

Note that libZMP is not my project, and I am only trying to put it in context.
The official issue is #98514

[josuah]

Programming language/frameworks integration

This is a thread to discuss various higher-level languages integration, such as Pigweed, C++, MicroPython, Rust, WebASM, Arduino...

[josuah]

Pigweed integration

A follow-up of this post

Typical example: if LTE/WiFi/Bluetooth reception goes below 512 kbit/s, signal every element of the chain (there is an ->eventfn() hook) so that a strategy can be taken: Propagating the back-pressure from the network until the network queue goes back to manageable level:

• Reduce the FPS at the image sensor level, or drop frames if not supported
• Reduce the resolution if possible dynamically, which is usually not the case so:
• Encode multiple resolutions in parallel in a pipeline "fork" and have an element dynamically select the resolution according to the downstream "back-pressure"

As we see here, there are a lot of "... or if (hardware does not support it) { insert software fallback with a different pipeline topology }".

@quic-devanshi How does the libmp pipeline integrate with Pigweed?

In turn, SDKs like Pigweed might be able to take advantage of the complete abstraction offered by libZMP to remove all these if (hardware has ...) from the application developer.

If the users asks get_frame_from_camera(format=JPEG), whichever way JPEG compression happens in the pipeline (directly from the sensor OV5640/OV2640, USB webcam, Arducam Mega with JPEG over SPI, hardware-based encoder 1 2, optimized software implementation...), it becomes possible to much more easily make it an implementation detail that pigweed users do not need to worry about, but can still control directly through libZMP APIs, (i.e. allow custom pipelines to be used through pigweed high-level APIs).

This allows to build a generic "get media" function that supports a large range of audio/video formats and integrates all available hardware acceleration and sources.

Edit: this is just one proposed way of how to try it from my personal perspective, to avoid having to maintain a 1:1 API that just maps libZMP in pigweed, there might be better ways and different opinions.

Can you elaborate more on this?
Also Google seems to have defined .proto interfaces for various multimedia usecases like video to display. I was trying to think how the libMP interfaces can augment multimedia functionality to Google defined interfaces.

Originally posted by @quic-devanshi in #98514 (comment)

@quic-devanshi

• Was there a particular point you wanted to know more about?
• Do you have a link to the Google uXR project?

Edit: nothing available on my end about uXR (micro eXtended Reality I assume), I will have to assume a generic Pigweed integration layer. Maybe you find some useful elements in the other threads below.

[josuah]

MicroPython integration

For Gstreamer, a low-level C API exists side-by-side with a Python API, this brings an opportunity to create bindings to easily prototype pipelines. This means not needing a C toolchain to reconfigure a libZMP pipeline, which allows all the hardware integration to lead to a system image further configured in Python, and was relatively interesting feature to split the complexity between "getting the hardware to work" and "making something useful out of it". Particularly helpful for debugging, organizing teamwork, prototyping stages...

[josuah]

Dynamic Pipelines: media handling

Some essential feature of media systems is the ability to receive a stream from another system, for instance network, or stored file, or Bluetooth, etc.

Multi-media means that streams will not always contain content know in advance when it is received, and the handling of the format will depend on a complex matrix of available hardware and software support.

To simplify this matrix, as well as making media handling possible in the first place, a pipeline such as libZMP becomes necessary. Such pipeline exist in any system able to handle media, often boiling down to FFmpeg and Gstreamer.

This is very well described in Gstreamer documentation about dynamic pipelines.

[josuah]

Given anything willing to support "open a file/stream/hardware-source/URL and play the content" will need such a pipeline, this makes libZMP building block for providing a high level API that decode a stream from any source, and play/record/stream it.

This does not force users to include all the pipelines, and does not force users to use dynamic pipelines as static pipelines handling a single format are also an option, so supporting a high-level API does not force to use dynamic piplines either, but modeling the API in knowledge of dynamic pipelines allows a single API to work for everything.

For instance:

/* Initialize a dynamic (or static if Kconfig says so) libZMP pipeline */
ret = mylib_init_stream(&stream);
if (ret != 0) {
    return ret;
}

Then either of these:

/* Connect to this stream and let libZMP handle everything */
ret = mylib_open_stream_from_uri(&stream, "srtp://192.168.10.2");
if (ret != 0) {
    return ret;
}

/* Try to detect the format, i.e. with magic bytes */
ret = mylib_open_stream_from_buffer_auto(&stream, buf, sizeof(buf));
if (ret != 0) {
    return ret;
}

/* Specify a pre-configured pipeline */
ret = mylib_open_stream_from_buffer_manual(&stream, MYLIB_PIPELINE_LOCAL_MICROPHONE_LEFT);
if (ret != 0) {
    return ret;
}

The last example is interesting as it shows that declaring a pipeline and using it in the code can be decoupled.

This has particularly interesting abstraction properties: a hardware integration can have a single, completely abstract endpoint such as 'local microphone'.

Deciding what device provide the local microphone, and how much pre-processing (i.e. echo cancellation) is done on this stream becomes a platform detail.

This in turns allows building applications that are completely hardware agnostic, as well as provide an even higher level of abstraction:

ret = mylib_record_audio(buf, sizeof(buf), duration, MYLIB_FORMAT_MP3);
if (ret != 0) {
    return ret;
}

At this level, the entirety of all the hardware, available acceleration, and integration is completely hidden away, and it starts to build truly hardware agnostic services.

Next part is connected to this: pipewire: local media routing.

[josuah]

Pipewire-style hardware abstraction

One particular use-case for pipeline engines is the ability to provide standard sources/sinks representing the underlying hardware as a mixing table.

Each underlying hardware has an identifier.

Each identifiers is bound to roles (such as "the local speaker", or "the local microphone").

Configuration and applications allows to match sources and sinks (connect this source and sink if the user did not enable mute).

All available sources and sinks and their capabilities are exposed on a same grid. -- qpwgraph screenshot showing a pipewire graph as example

The local hardware and applications as well as the "middleware" are all represented on a same grid allowing arbitrary "A to B" connection at various performance as system permits.

Example: screen sharing

While this seems an advanced feature reserved to only desktop users, but this is not true: MCUs are perfectly capable of piping the LVGL output back to a H.264 hardware encoder, and further pack this into an srtp:// stream sent over WiFi.

The reason why it might appear less doable on an MCU is only because so far, MCUs did not have an ecosystem to allow them handling these sort of integration.

A LVGL screen-sharing endpoint shows-up as an application source tagged as "display loopback", and applications wanting to implement screen-sharing can then select such a libZMP stream, and add an extra layer of hardware encoding as configured. Which hardware this runs on becomes an application detail, up to system integrator to select compatible hardware (i.e. video scaler like DMA2D things, H.264/MJPEG encoder...).

The key element is to flatten all the APIs on a same level, allowing routing to be done at runtime, which is what libZMP provides as a first brick.

[josuah]

Libcamera-style integration

A particular pipeline engine found on Linux is libcamera, which does not use gstreamer or pipewire, but instead integrate with them.

Libcamera is built out of the need to handle "complex cameras": video harware where the image data decoding and tuning is left for the host MCU rather than a dedicated ISP chip or directly on the sensor. This means that microprocessors, and lately, microcontrolers, inherit the duty of handling the bayer data from sensors, and pursue with all the subsequent image tuning.

Like with libcamera, libZMP can be used as a toolkit for inter-connecting such elements and provide a system a simplified, rich "camera pipeline endpoint".

For instance, video conferencing on Linux-based laptops goes through pipewire exposing a libcamera source to the web browser sink.

Reminder: raw images from "good" image sensors tend to look like this before correction (here is extreme example):

As features are provided to camera (and audio) "default pipelines" or "hardware-specific pipelines", this allows to progressively integrate better and better features, such as improved calibration for a particular sensor, or integration of fish-eye correction...

All such features being possible to integrate without requiring modification in the application in case a pipeline engine is present and provides a "Camera 0" output endpoint.

[josuah]

Audio subsystem for Zephyr and Bluetooth

By lacking a dedicated audio subsystem that unifies all the different audio driver sources, application developers are left packing everything manually and fixing bugs repeatedly on every new project.

Having an uniform source/sink strategy helps with reducing the API churn that every developer has to go through to play/record/encode/decode/transmit live media.

MCUs and DSPs are often used for audio processing, but little existing infrastructure helps building streams and wrap all the runtime decoding, or solve the synchronization and timing issues at a platform level.

A multimedia pipeline does not imply large runtime overhead. The amount of abstraction is light, and not necessarily slower than some bare metal applications. This can help building more complex multi-channel applications, and help wrapping audio encoding/decoding.

Audio over Bluetooth faces the same challenges as video over network, and scaling the sampling resolution and compression levels can help fitting the incoming data in the available bandwidth.

Doing so already requires advanced pipeline control to detect congestion and propagate it back all across the pipeline down to audio hardware re-configuration.

[josuah]

Video to display

In some cases, an embedded system needs to do this: i.e. dead angle camera on vehicles (from airplanes to cars to extra add-on for buses/trucks), or a temporary preview of a camera used for other purposes (i.e. to check if it works).

This seems like a very basic example, but:

• How many cameras are present on a Zephyr board?
• Should they both be integrated into a single LVGL GUI?
• Directly send the frames to the display?
• Convert the frames with hardware to RGB565 for the display? To L8? To L4?
• Does the camera support RGB565?
• At the correct resolution?
• Is there a hardware video converter/scaler available?
• Should de-fish-eye be performed to correct the perspective of the camera?
• Is the camera a webcam sending JPEG frames?
• Should the user be allowed to select?
• Should there be some kind of frame-rate adaptation by dropping frames from a higher FPS?

Even the most simple minimal example has a lot of complexity to solve, in hardware-specific ways, like the current Zephyr video capture sample show.

Video pipelines help absorbing all this complexity and take it away from every sample, and beyond this, away from the application developer.

[ngphibang]

My motivation for this is replacing the DDR3 RAM chip of Linux-based cameras by using Zephyr instead of Linux (and an MCU instead of an MPU eventually) as well as shrink video systems in various metrics.

I am not sure to understand this 1st phrases of the PR description. Could you elaborate more ?

[josuah]

This was more a long-term direction than an actual mid/short-term goal.

I do not personally know a lot of system that can boot Linux out of on-chip RAM. BL808 has 32/64 MBytes and can boot Linux, but most other SoCs used for multimedia have an external DDR3 RAM.

Most of the Linux-based cameras just run Gstreamer pipeline (or proprietary equivalent) as main application and that's it.

The small, high-volume cameras (i.e. CCTV) can start using MCUs instead of Linux. Costs go down, one-chip solution means low reliance on external parts (i.e. DDR3 RAMs). Replacing Linux by Zephyr, and MPUs by MCUs like STM32 N6, ESP32 P4, i.MXRT... is tempting. What is missing is the software integration layers, and libZMP makes some very ubiquitous framework to connect everything.

[josuah]

Examples, Linux systems tend to support 32-MBytes ~ 1024 MBytes of RAM:

• Ingenic T32 has in-package or external DDR3
• STM32 MP151 supports DDR2/DDR3
• Rockchip RV1106G2 (small one) uses DDR3
• i.MX95 uses DDR4/DDR5

Meanwhile, in addition to boot Zephyr it looks like:

• 1 MBytes are enough for one 1280x720 frame in black/white + H.264/JPEG compressed version
• 2 MBytes are enough for one 1280x720 frame in NV12 format + H.264/JPEG compressed version
• 4 MBytes are enough for one 1920x1080 frame in NV12 format + H.264/JPEG compressed version

With plenty of extra room for an application.