How to synchronize buffer timestamp across two gstreamer pipeline(appsink,appsrc, rtspserver)

[Jake Zhang]

Greetings,

I have been working on this for a while but still have no luck to get this working. My pipeline can be simplified as below: 

pipeline1:

v4l2src -> tee -> x264enc -> appsink1  

             ->  appsink2

                 

RTSP server pipeline2: 

appsrc -> omxh264enc -> rtph264pay

Pipeline 1 have other processing unit and I will need pull buffers from appsink2 and push those buffers into appsrc of RTSP server. 

My RTSP server part of implementation is very similar to test-appsrc.c of gst-rtsp-server 1.6.4. 

The buffer I got from appsink already has their own PTS and to get RTSP pipeline2 running, I have to re-stamp the buffer PTS the exactly same way as test-appsrc.c (set ctx->timestamp=0 and increment based of buffer duration) otherwise, the RTSP server will not get running. 

I have went through the gstreamer manual and I have some basic understanding about running time/ base time and stream time 

I have tried below things:

1. set the basetime of pipeline2 to be the same as pipeline1

2. set the start time of pipeline2 to be GST_CLOCK_TIME_NONE

3. make sure the two pipeline is using the same clock. 

4. I have checked the segment event on the appsrc of pipeline2 and the value is as below, and I have tried generate a new segment to appsrc sink pad but I have a hard time know how to set the right base and start value. 

//appsrc0:src segment: rate 1 format 3, start: 0:00:00.000000000, stop: 99:99:99.999999999, time: 0:00:00.000000000 base: 0:00:00.000000000

Why I have to re-stamp the PTS of the buffer?  

pipeline2 running time = clocktime - basetime of pipeline1, so pipeline2 should be able to process buffers with original PTS, right? 

How I can  retain the original buffer PTS in pipeline2? I guess segment event is the right direction to go, but like I said I have not figure out how to play with segment to let pipeline2 happy with original PTS. 

Thanks in advance for any comments here.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

So,

Assuming you didn't do any mistake, what you have now is buffer in pipeline 2 that are really late. To be able to play those the way you do you'll have to set pipeline 1 latency on your pipeline 2 appsrc. As you have multiple branches in pipeline 1, you'll be to implement do-latency signal, so you submit on appsink early, and render late enough to be on sync whit pipeline 2 sink. Make sure appsrc is set to live mode.

Le 16 juin 2016 5:21 PM, "Jake Zhang" <[hidden email]> a écrit :

Greetings,

I have been working on this for a while but still have no luck to get this working. My pipeline can be simplified as below: 

pipeline1:

v4l2src -> tee -> x264enc -> appsink1  

             ->  appsink2

                 

RTSP server pipeline2: 

appsrc -> omxh264enc -> rtph264pay

Pipeline 1 have other processing unit and I will need pull buffers from appsink2 and push those buffers into appsrc of RTSP server. 

My RTSP server part of implementation is very similar to test-appsrc.c of gst-rtsp-server 1.6.4. 

The buffer I got from appsink already has their own PTS and to get RTSP pipeline2 running, I have to re-stamp the buffer PTS the exactly same way as test-appsrc.c (set ctx->timestamp=0 and increment based of buffer duration) otherwise, the RTSP server will not get running. 

I have went through the gstreamer manual and I have some basic understanding about running time/ base time and stream time 

I have tried below things:

1. set the basetime of pipeline2 to be the same as pipeline1

2. set the start time of pipeline2 to be GST_CLOCK_TIME_NONE

3. make sure the two pipeline is using the same clock. 

4. I have checked the segment event on the appsrc of pipeline2 and the value is as below, and I have tried generate a new segment to appsrc sink pad but I have a hard time know how to set the right base and start value. 

//appsrc0:src segment: rate 1 format 3, start: 0:00:00.000000000, stop: 99:99:99.999999999, time: 0:00:00.000000000 base: 0:00:00.000000000

Why I have to re-stamp the PTS of the buffer?  

pipeline2 running time = clocktime - basetime of pipeline1, so pipeline2 should be able to process buffers with original PTS, right? 

How I can  retain the original buffer PTS in pipeline2? I guess segment event is the right direction to go, but like I said I have not figure out how to play with segment to let pipeline2 happy with original PTS. 

Thanks in advance for any comments here.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Jake Zhang]

Thanks for quick reply Nicolas.

I am still pretty new with Gstreamer so please bear with me if my questions sounds trivial to you.

My appsrc has been in live mode and my v4l2src has been set as do-timestamp. Like I said, the pipeline is running without any issue, but I really need make sure the time-stamp of the same buffer on two pipelines are the same, as we need use PTS a frame tag to do some other processing on the RTSP client side . 

On Thu, Jun 16, 2016 at 9:27 PM, Nicolas Dufresne <[hidden email]> wrote:

So,

Assuming you didn't do any mistake, what you have now is buffer in pipeline 2 that are really late. To be able to play those the way you do you'll have to set pipeline 1 latency on your pipeline 2 appsrc. As you have multiple branches in pipeline 1, you'll be to implement do-latency signal, so you submit on appsink early,

How can I submit buffers earlier? Should I just subtract latency of pipeline1 from bastime of pipeline2 to make sure the buffer is early enough? 

and render late enough to be on sync whit pipeline 2 sink. Make sure appsrc is set to live mode.

How the gstreamer will drop a buffer? 

Reading from the documentation, I learned that the buffer will be processed if the buffer running time = pipeline running time. 

pipeline running time = clock time - basetime

buffer time = PTS -segment.start (which is zero in my case). 

So I guess, I should either change the basetime of pipeline2 to offset the pipeline1 latency or change the segment.start time? 

Thanks again.

 

Le 16 juin 2016 5:21 PM, "Jake Zhang" <[hidden email]> a écrit :

Greetings,

I have been working on this for a while but still have no luck to get this working. My pipeline can be simplified as below: 

pipeline1:

v4l2src -> tee -> x264enc -> appsink1  

             ->  appsink2

                 

RTSP server pipeline2: 

appsrc -> omxh264enc -> rtph264pay

Pipeline 1 have other processing unit and I will need pull buffers from appsink2 and push those buffers into appsrc of RTSP server. 

My RTSP server part of implementation is very similar to test-appsrc.c of gst-rtsp-server 1.6.4. 

The buffer I got from appsink already has their own PTS and to get RTSP pipeline2 running, I have to re-stamp the buffer PTS the exactly same way as test-appsrc.c (set ctx->timestamp=0 and increment based of buffer duration) otherwise, the RTSP server will not get running. 

I have went through the gstreamer manual and I have some basic understanding about running time/ base time and stream time 

I have tried below things:

1. set the basetime of pipeline2 to be the same as pipeline1

2. set the start time of pipeline2 to be GST_CLOCK_TIME_NONE

3. make sure the two pipeline is using the same clock. 

4. I have checked the segment event on the appsrc of pipeline2 and the value is as below, and I have tried generate a new segment to appsrc sink pad but I have a hard time know how to set the right base and start value. 

//appsrc0:src segment: rate 1 format 3, start: 0:00:00.000000000, stop: 99:99:99.999999999, time: 0:00:00.000000000 base: 0:00:00.000000000

Why I have to re-stamp the PTS of the buffer?  

pipeline2 running time = clocktime - basetime of pipeline1, so pipeline2 should be able to process buffers with original PTS, right? 

How I can  retain the original buffer PTS in pipeline2? I guess segment event is the right direction to go, but like I said I have not figure out how to play with segment to let pipeline2 happy with original PTS. 

Thanks in advance for any comments here.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Do, 2016-06-16 at 22:45 -0400, Jake Zhang wrote:
> Thanks for quick reply Nicolas.
>
> I am still pretty new with Gstreamer so please bear with me if my
> questions sounds trivial to you.
> My appsrc has been in live mode and my v4l2src has been set as do-
> timestamp. Like I said, the pipeline is running without any issue,
> but I really need make sure the time-stamp of the same buffer on two
> pipelines are the same, as we need use PTS a frame tag to do some
> other processing on the RTSP client side . 

If you need them to be exactly the same you need to:
a) use the same clock on both pipeline (gst_pipeline_use_clock())
b) set the same base time on both pipelines
(gst_element_set_base_time()) and set start time to GST_CLOCK_TIME_NONE
(gst_element_set_start_time())
c) configure the latency correctly, that is: query the latency (with
the latency query) on the appsink once it is PLAYING, and configure
exactly those values on the corresponding appsrc as min/max latency.

Additionally you need to ensure that the segment event that comes out
of appsrc is the same as the one that went into appsink, but in your
pipeline that is most likely the case. But better double check and if
not you need to ensure that appsrc is producing the same segment event.


If you use a single, connected pipeline, GStreamer will ensure these
things by itself already. If you use multiple, you will need to do that
yourself.

--

Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[IgalKroyter]

Hi,

although it is an old post, if someone could reply on the following questions:
I have 3 pipelines which have to be synchronized.
I have applied the above mentioned queries and saw that all pipelines share different value.
How do I modify the values taken from the main pipeline to the other two pipelines and when (before or after setting
PLAYING)
1) min Latency,max latency, live (how and when)
2) basetime and starttime (when)

Regards.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[e-mail]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mercredi 24 juillet 2019 à 09:31 -0500, IgalKroyter a écrit :

> Hi,
>
> although it is an old post I have the following questions:
> I have 3 pipelines which have to be synchronized.
> I have applied the above mentioned queries and saw that all pipelines share
> different value.
> The questions that I have are how do I modify the values taken from the main
> pipeline to the other two pipelines and when (before or after setting
> PLAYING)
> 1) min Latency,max latency, live (how and when)

You can only modify the min latency, see gst_pipeline_set_latency()

> 2) basetime and starttime (when)

You will always set starttime to -1 (GST_CLOCK_TIME_NONE) (which will
tell the pipeline to not update these) and set the basetime to that
same everywhere using gst_element_set_base_time() on you pipeline
object.

>
> Regards.
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[IgalKroyter]

Nicholas, thanks for the reply.

1) I have modified the latency of the pipeline with
gst_pipeline_set_latency() and I read it with gst_pipeline_get_latency().
both values are equal. But if I query the
pipeline(gst_query_new_latency..gst_element_query) I get: live:0,
min_latency:0 and max_latency: -1. by the way I query this from within a pad
callback every time it is invoked (just for the test). Is it normal?

2) Does it matter when I set the starttime or basetime before or after
PLAYING?
 
One more thing though you stated that the is-live cannot be set, I have noticed that there is the gst_base_src_set_live() function which can be applied on sources (I guess) so I tried to apply it on v4l2src and interaudiosrc without success. Normal behavior?

Regards.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[e-mail]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeudi 25 juillet 2019 à 00:02 -0500, IgalKroyter a écrit :
> Nicholas, thanks for the reply.
>
> 1) I have modified the latency of the pipeline with
> gst_pipeline_set_latency() and I read it with gst_pipeline_get_latency().
> both values are equal. But if I query the
> pipeline(gst_query_new_latency..gst_element_query) I get: live:0,
> min_latency:0 and max_latency: -1. by the way I query this from within a pad
> callback every time it is invoked (just for the test). Is it normal?

I'm not sure if that mechanism will impact the queried latency, also
you pipeline need to be running.

>
> 2) Does it matter when I set the starttime or basetime before or after
> PLAYING?

It has to happen in NULL state, anything else feels racy to me.

>  
> Regards.
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[pisymbol .]

On Wed, Jul 24, 2019 at 4:39 PM Nicolas Dufresne <[hidden email]> wrote:

> 2) basetime and starttime (when)

You will always set starttime to -1 (GST_CLOCK_TIME_NONE) (which will
tell the pipeline to not update these) and set the basetime to that
same everywhere using gst_element_set_base_time() on you pipeline
object.

You do this before set the pipeline tate to playing? And then all streams will use the same wall clock, is that right?

I have a similar issue where I have two pipelines (left and right camera) and I would like both of them to be aligned to the same clock.

-aps

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeu. 25 juill. 2019 19 h 10, pisymbol . <[hidden email]> a écrit :

On Wed, Jul 24, 2019 at 4:39 PM Nicolas Dufresne <[hidden email]> wrote:

> 2) basetime and starttime (when)

You will always set starttime to -1 (GST_CLOCK_TIME_NONE) (which will
tell the pipeline to not update these) and set the basetime to that
same everywhere using gst_element_set_base_time() on you pipeline
object.

You do this before set the pipeline tate to playing? And then all streams will use the same wall clock, is that right?

I have a similar issue where I have two pipelines (left and right camera) and I would like both of them to be aligned to the same clock.

We'll, don't forget to enforce a specific clock with use_clock, otherwise it may pick an element provided clock and that won't work.

-aps

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel