WebRTC with frame-synchronized metadata

[Jesse Gray]

Hello,

We are interested in sending frame-synchronized data along with the video frames in a WebRTC stream (using GStreamer's webrtcbin).  For example, consider a moving camera: it would be useful to include the current coordinates of the camera along with the video frame.  Or in a computer vision scenario, we might want to send the rectangle bounding some detected object in that specific frame.  In each case we want the receiver of a WebRTC stream to be able to match up the metadata to the same video frame as intended by the sender.

The most convenient solution would be if the sender could somehow attach the metadata to the frame so it arrives at the receiver associated with the correct frame.  But, also workable is just having a mechanism to identify the frames uniquely across sender/receiver, and then transmit the metadata some other way keyed off that unique identity.

Does anyone know if there is a way to attach metadata to an image frame in such a way that it is accessible to the receiver after traveling over the WebRTC connection?

Or, is there any unique ID associated with an image frame on the sending side that is preserved over the WebRTC link to the receiver that could be used as a key to associate data with that specific frame?  For example, is there a timestamp value that is the same on both ends?

It would be great to find a solution that works between GStreamer and a browser, but a solution that relies on GStreamer's webrtcbin managing both ends of the connection is also very interesting!

Thanks very much for any ideas or suggestions!
Jesse

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mer. 2 janv. 2019 18 h 11, Jesse Gray <[hidden email]> a écrit :

Hello,

We are interested in sending frame-synchronized data along with the video frames in a WebRTC stream (using GStreamer's webrtcbin).  For example, consider a moving camera: it would be useful to include the current coordinates of the camera along with the video frame.  Or in a computer vision scenario, we might want to send the rectangle bounding some detected object in that specific frame.  In each case we want the receiver of a WebRTC stream to be able to match up the metadata to the same video frame as intended by the sender.

The most convenient solution would be if the sender could somehow attach the metadata to the frame so it arrives at the receiver associated with the correct frame.  But, also workable is just having a mechanism to identify the frames uniquely across sender/receiver, and then transmit the metadata some other way keyed off that unique identity.

Does anyone know if there is a way to attach metadata to an image frame in such a way that it is accessible to the receiver after traveling over the WebRTC connection?

Or, is there any unique ID associated with an image frame on the sending side that is preserved over the WebRTC link to the receiver that could be used as a key to associate data with that specific frame?  For example, is there a timestamp value that is the same on both ends?

It would be great to find a solution that works between GStreamer and a browser, but a solution that relies on GStreamer's webrtcbin managing both ends of the connection is also very interesting!

I'm not sure for the browser part, but GStreamer to GStreamer, we have successfully used RTP header extension. See GstRTPBuffer API, there is helpers to add such extension to RTP packets. We were using pad probes to attach the information.  One to detect new frame, and another to attach the information on all packets of this frame. All in case a packet is lost.

Thanks very much for any ideas or suggestions!
Jesse

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel