Seek commands and tsdemux element

[Mario Montagud Climent-2]

Dear colleagues,

In a previous message, we discussed the performance and behavior of  
seek commands when using the tsdemux element.

We’ve run some tests when using both tsdemux and qtdemux elements, for  
MPEG2-TS and MP4 local files, respectively. These are the employed  
pipelines:

                                 +---------+   +------------+    
+---------+    +-------------+
                             |-> |  queue  |-> | h264_parser|-> |  
h264dec |->  | xvimagesink |
                                 +---------+   +------------+    
+---------+    +-------------+
+---------+    +---------+  |
| filesrc | -> | *demux  | -
+---------+    +---------+  |
                                 +---------+   +----------+    
+---------+    +---------------+
                             |-> | queue   |-> | aacparse |->  |    
faad  |->  | autoaudiosink |
                                 +---------+   +----------+    
+---------+    +---------------+

The seek command is as follows (target position = current position +  
5s, just for testing):

/* Seek to current_position + 5 seconds, just for testing */
static void send_seek_event (CustomData *data) {
   GstEvent *seek_event;

   /* We obtain the current position, needed for the seek command */
   if (!gst_element_query_position (data->pipeline, GST_FORMAT_TIME,  
&position)) {
     g_printerr ("Unable to retrieve current position.\n");
     return;
   }

   /* Seek event */
   seek_event = gst_event_new_seek (1.0, GST_FORMAT_TIME, GST_SEEK_FLAG_FLUSH ,
           GST_SEEK_TYPE_SET, position + 5000000000, GST_SEEK_TYPE_SET, -1);

   /* Send the event */
   gst_element_send_event (data->xvimagesink, seek_event);

}

When using tsdemux, there is a variable offset between the target seek  
position and the segment start field, varying between 2.34s and 2.42s  
for different seek operations (just for testing). However, when using  
qtdemux, there is a constant offset of 66.7ms (2 frames, 30 fps)  
between both values for different seek operations.

Are these values reasonable? Should the segment start field correspond  
with the exact target seek position?

The existence of these offset values can be seen as reasonable, as the  
decoder may need a key frame for starting the decoding process after  
the seek command, but the offset values are too large when using  
tsdemux (GoP length is 1s!) and constant when using qtdemux (2 frames)  
for different seek operations.

Our goal is to synchronize the playout of different, but related,  
streams (both on the same devices and across different devices) at the  
stream-level, based on relative and absolute timelines inserted into  
these streams (these values are obtained from the tsdemux element, we  
have slightly modified it).

We check the synchronization accuracy by comparing the timelines  
within the tsdemux elements (using pad probes), the stream time  
(calculated using the info from segments) and by using overlays (frame  
numbers) added during the encoding process, by using ffmpeg.

When using GoP lengths of 1s and 30fps, the synchronization accuracy  
between two mpeg2-ts streams is quite accurate, but we have also  
tested with other GoP lengths and, in these cases, it is necessary to  
add manual offsets for the seek commands to achieve an accurate  
synchronization. It is very strange and we do not understand why.

In these cases, several buffers are simultaneously pushed from the  
tsdemux to the decoder after the seek command and we lose the timeline  
references for synchronization. It does not seem to be any  
relationship between the number of pushed buffers and the offset  
between the target seek position and the segment start.
Is there any way to know how many (to-be-played) buffers are between  
the tsdemux element and the decoder?

We would appreciate it very much if you could clarify our doubts.

Thank you very much in advance!

Best regards,

Mario Montagud
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Tue, 2016-07-19 at 13:24 +0200, Mario Montagud Climent wrote:

Hi Mario,

I've written down some notes on the ideal expected behaviour of
demuxers here:
https://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/part-
seeking.txt#n115

Which might answer some questions.

Typically people either seek with FLAG_KEY_UNIT or FLAG_ACCURATE.

Cheers
 -Tim

--

Tim Müller, Centricular Ltd - http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Mario Montagud Climent-2]

Hi Tim,

These notes are very useful, and have helped in clarifying some of our  
doubts. Thank you!

The qtdemux behaves as expected when setting the FLAG_KEY_UNIT and  
FLAG_ACCURATE flags. In the latter case, as expected, the segment  
start field is equal to the target seek position. However, it does not  
happen when using tsdemux, as we’ve measured a variable offset between  
2.2 and 2.4 s, much larger than the GoP length (1s, 30fps)!

Moreover, we experienced the following issues when using MPEG2-TS:

We created the master and slave MPEG2-TSs using ffmpeg, and we added  
numeric overlays (PTS and frame numbers) to visually check the sync  
performance. We set a frame rate of 30fps, and we tested with  
different GoP lengths. When using GoP of 30 frames (1s), the result of  
the seek command was OK, but when using other (larger) GoP lengths, we  
had to add an offset (a period around 2 or 3 GoPs) to the target seek  
position to successfully seek to the targeted media stream position  
(the current playou position of the master stream). We had to infer  
that offset from the numeric overlays on the video frames of both  
streams, which is clearly not a good solution. We performed these  
tests in both Ubuntu and Android.

So, we assume that (at this moment) it is not convenient to use  
MPEG2-TS streams for the slave streams we want to synchronize, as the  
behavior and performance of the tsdemux when seeking is (still) not so  
good.

We are not testing with DASH content for the slave streams, and the  
behavior and performance of the seek position is much better.

Thank you very much for your help!

Cheers,

Mario Montagud

Tim Müller <[hidden email]> escribió:

> On Tue, 2016-07-19 at 13:24 +0200, Mario Montagud Climent wrote:
>
> Hi Mario,
>
> I've written down some notes on the ideal expected behaviour of
> demuxers here:
> https://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/part-
> seeking.txt#n115
>
> Which might answer some questions.
>
> Typically people either seek with FLAG_KEY_UNIT or FLAG_ACCURATE.
>
> Cheers
>  -Tim
>
> --
>
> Tim Müller, Centricular Ltd - http://www.centricular.com
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Fri, 2016-07-22 at 11:02 +0200, Mario Montagud Climent wrote:

Hi Mario,

What version of GStreamer / tsdemux are you using?

There were some bug-fixes related to accurate seeks in tsdemux, e.g.

https://bugzilla.gnome.org/show_bug.cgi?id=763262

If you're using a recent release and things don't work as expected,
please file a bug in bugzilla :)

Cheers
-Tim

--

Tim Müller, Centricular Ltd - http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel