Match RTP packets with decoded image frames

[myamens]

Hello,

I am currently using GStreamer to stream video over network. In my application, I need to perfectly sync data with the video images (zero latency between both data and the corresponding images).
Since data size per frame is very small (8 bytes), I am adding this data into the first RTP packet's padding for each encoded frame just before sending it over UDP. On the receiver side, after I receive the packet I extract the data from the RTP padding and save it in a queue. Once the frame is decoded, I pop from the queue the first element which suppose to match the decoded frame.
In theory it should work fine, however in practice, the sink can drop some frames which would results mismatching between the frames and the corresponding data in the queue.

I am looking for better solutions in which I can perfectly sync between the received RTP packets and the corresponding decoded image frames.

I tried to add metadata into the RTP packets on the receiver side, and extract the meta data after decoding, however it didn't work (the metadata is removed apparently).

Looking forward for an advice!

[Mikl]

Hello,

What if you will stream video container with metadata inside?

For example MKV file with subtitles?

PS sorry for late reply

Mikl

---

From: gstreamer-devel <[hidden email]> on behalf of myamens <[hidden email]>
Sent: Monday, February 27, 2017 5:10:09 AM
To: [hidden email]
Subject: Match RTP packets with decoded image frames

 

Hello,

I am currently using GStreamer to stream video over network. In my
application, I need to perfectly sync data with the video images (zero
latency between both data and the corresponding images).
Since data size per frame is very small (8 bytes), I am adding this data
into the first RTP packet's padding for each encoded frame just before
sending it over UDP. On the receiver side, after I receive the packet I
extract the data from the RTP padding and save it in a queue. Once the frame
is decoded, I pop from the queue the first element which suppose to match
the decoded frame.
In theory it should work fine, however in practice, the sink can drop some
frames which would results mismatching between the frames and the
corresponding data in the queue.

I am looking for better solutions in which I can perfectly sync between the
received RTP packets and the corresponding decoded image frames.

I tried to add metadata into the RTP packets on the receiver side, and
extract the meta data after decoding, however it didn't work (the metadata
is removed apparently).

Looking forward for an advice!



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Match-RTP-packets-with-decoded-image-frames-tp4682031.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mercredi 21 juin 2017 à 13:26 +0000, Michael Yarochkin a écrit :
> I am looking for better solutions in which I can perfectly sync between the
> received RTP packets and the corresponding decoded image frames.
>
> I tried to add metadata into the RTP packets on the receiver side, and
> extract the meta data after decoding, however it didn't work (the metadata
> is removed apparently).

Using GstMeta should work, if not, then it's a bug. It's a type of bug
we have often found and fixed in the past. Please file a bug on
bugs.gnome.org, produce GStreamer component "unknown". Describe your
pipeline, and where you do add the GstMeta. We'll follow the chain to
find were the meta is dropped.

regards,
Nicolas

p.s. Using serialized custom event should also work.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel