raw video stream with per frame meta data over network

[Thomas Johansson]

Hello all, I'm Thomas Johansson and working on a problem involving a raw video stream with associated meta data.

I need to transfer a stream of raw images from a video4linux2 source with a set of per frame data fields over UDP, The meta data will be different for each and every frame.

Refer to the RFC3550 for RTP transport, the RFC4175 for transferring raw images (YCbCr 4:2:2) and RFC6597 (KLV data) for the associated meta data and as of the spec for RFC4175:

"This payload format requires that applications using such synchronized ancillary

   data SHOULD deliver it in separate RTP sessions that operate

   concurrently with the video session.  The normal RTP mechanisms

   SHOULD be used to synchronize the media."

I have made an attempt to build a gstreamer rtpbin solution but I'm unsure of the proper way to do it.

Question 1, in the world of gstreamer, what would be the natural way to transfer meta data and associate it with each frame ? In my solution I'm using the buffer-mode" setting on the receivers jitterbuffer to force the PTS timestamp to follow the buffers PTS on the sending side.

Question 2, I'm using the KLV rtp payloader and the appsource element, however I'm unsure of how to actually send a new packet of KLV meta data for each frame. My first attempt was to use a Identity element of the video4linux source and at every handoff transfer a packet of the KLV, however it fails and I don't know how to properly set up the transfer.

I would appreciate if someone could assist me with a proper design of the overall problem, the issue of attaching frame specific data with each and every frame over a UDP link. Also, if the KLV is the way to go, I'm having trouble using the appsource for KLV so any pointers would be good.

Thanks

/Thomas

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Tue, 2018-12-18 at 10:42 +0100, Thomas Johansson wrote:

Hi Thomas,

There are multiple ways of doing this.

How large do you expect the KLV data chunks to be? Will they be very
short or can they vary in size?

Cheers
 Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Thomas Johansson]

So the meta data will not be more than some 10 items and total meta data size
less than 128 bytes, the KLV will then add to this.
/Thomas



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Tue, 2018-12-18 at 07:32 -0600, Thomas Johansson wrote:

Hi,

> So the meta data will not be more than some 10 items and total meta
> data size less than 128 bytes, the KLV will then add to this.

Ok, so fairly small.

There are at least two options:

1) Embed the KLV data in your RTP-payloaded video stream via RTP header
extensions. This should work fine for small data chunks like yours.
It's a bit finicky to do, you'd have to write a custom element to do
this or use pad probes and the GstRTPBuffer API, but it's been done
before and should just work. Same then on the receiving end to extract
it again.

2) Send the KLV data as a separate KLV RTP stream and
synchronise/combine on the receiver side.

I would probably go for Option 1, unless you need it as a separate
stream on the receiver side anyway (e.g. to mux into an MPEG-TS stream
or such).

Cheers
 -Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel