Adding metadata to a stream

[jagdspyder]

Hello,

The goal here is to be able to send some metadata (timestamps, objects found per frame) with a stream within a single pipeline or multiple pipelines over network (e.g. RTP UDP). I know this has been asked a ton of times but all the answers are deleted by the authors! Looks like a bug to me.

Within The pipeline

This is straight forward by defining a new GstMeta API and register and implement it. Then add it to GstBuffers via buffer probes or another element solely designed for this purpose.

Multiple pipelines over Network

The only solution I have heard about is to add one or more RTP header extension to RTP packets coming out of a payloader. Actually some people decided to transform their Custom GstMeta into those header extensions. But Is This really a good use-case for it?

There's more ....

Another idea is to create your own custom media type and for that you will need to write a typefinding function and an autoplugger on top of it. Not to mention a couple of elements to deal with this new type and convert it to other exiting media types. Now that new type should actually have a place where it can handle the meta data I was talking about.


That summarizes my research, Are there any other methods that I am not aware of?
I would definitely appreciate your input on this!

Thanks in advance!

[Marianna S. Buschle]

I have send metadata using the EXIF tag of JPEG for a MJPEG stream.

I also intend to do the same (attach metadata) to a MPEG-TS stream (with H264 video) which is send using RTP but haven't had the time to try it yet. From my research it should be possible.

[jagdspyder]

So we either can send our metadata over RTP header extensions or create our new media type that is capable of handling such metadata over network? Are these the only solutions available?

I mean there isn't even that much to go on when creating your own new media type. let alone the hassle to do so which may compromise the performance.

[Marianna S. Buschle]

I don't know if there are any other options.

My use case was H264 video, and I could find that KLV metadata should be supported as part of the MPEG-TS container.

And since I was also streaming MJPEG I found out I could hack the EXIF tag to add custom metadata as well.

I don't know if there are any other container formats that also support custom metadata.

Or any other formats.

Though it seems to be possible to just send KLV metadata directly as well: https://gstreamer.freedesktop.org/documentation/rtp/rtpklvpay.html?gi-language=c

[jagdspyder]

Thanks for you answer.

I decided to add my metadata as header extension on top of rtph264pay. Though originally my meta data are implemented via the GstMeta API, I attach my GstMeta when the data is entering the rtph264pay (Sink) via a data probe, then I try to inspect the same meta while leaving the rtph264pay (Src) but the meta always seem to null.

And because of the MTU of 1500 bytes, one frame (whether key or delta) might actually correspond to multiple RTP packets, hence I use gst_rtp_buffer_get_marker(&rtpbuff) to detect the the frame boundary and it is here that I shall add my meta as RTP header except that my meta are always null when I try I get them from GstBuffer.

Any idea why I can't retrieve my meta from GstBuffer after leaving the rtph264pay?

[jagdspyder]

Glad to report that muxing the klv data worked in an MPEG2 TS container.
I was even surprised when the mpeg2tsmux didn't complain when I sent it arbitrary bytes on its pad of type meta/x-klv that is not klv-encoded