timecodestamper after webrtcbin -> decodebin failed ?

[Michael-2]

Hello,

I try to add a `timecodestamper` and a `timeoverlay` after a `decodebin`

which I put after a `webrtcbin`

def on_add_stream(self,element, pad):

of a `webrtcbin`.

But I always get 

** (python3:1527): CRITICAL **: 18:44:43.253: gst_video_time_code_add_frames: assertion 'gst_video_time_code_is_valid (tc)' failed

Why isn't that possible?
Or better why doesn't the webrtcbin -> decodebin offer a timestamp?

Thanks in advance, Michael

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Thibault Saunier-4]

Hello,

I am not sure why it doesn't work, but could give us a way to reproduce easily or get us a backtrace (set G_DEBUG=fatal-criticals so that your app abort when this critical happens) and a `GST_DEBUG=5` debug log file?

- Thibault

On Fri, May 8, 2020 at 2:53 PM Michael <[hidden email]> wrote:

Hello,

I try to add a `timecodestamper` and a `timeoverlay` after a `decodebin`

which I put after a `webrtcbin`

def on_add_stream(self,element, pad):

of a `webrtcbin`.

But I always get 

** (python3:1527): CRITICAL **: 18:44:43.253: gst_video_time_code_add_frames: assertion 'gst_video_time_code_is_valid (tc)' failed

Why isn't that possible?
Or better why doesn't the webrtcbin -> decodebin offer a timestamp?

Thanks in advance, Michael

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Michael-2]

Hello Thibault,

but what might be the reason and it caught my eye now because my latency is ramping up from ms to ms, is that I get `framerate=(fraction)0/1`after the decodebin which is linked to webrtcbin.

on_incoming_decodebin_pad:

video/x-raw, format=(string)I420, width=(int)640, height=(int)480, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)jpeg, colorimetry=(string)bt601, framerate=(fraction)0/1

I will deliver the rest later.

Thanks, Michael

Am Mo., 11. Mai 2020 um 17:05 Uhr schrieb Thibault Saunier <[hidden email]>:

Hello,

I am not sure why it doesn't work, but could give us a way to reproduce easily or get us a backtrace (set G_DEBUG=fatal-criticals so that your app abort when this critical happens) and a `GST_DEBUG=5` debug log file?

- Thibault

On Fri, May 8, 2020 at 2:53 PM Michael <[hidden email]> wrote:

Hello,

I try to add a `timecodestamper` and a `timeoverlay` after a `decodebin`

which I put after a `webrtcbin`

def on_add_stream(self,element, pad):

of a `webrtcbin`.

But I always get 

** (python3:1527): CRITICAL **: 18:44:43.253: gst_video_time_code_add_frames: assertion 'gst_video_time_code_is_valid (tc)' failed

Why isn't that possible?
Or better why doesn't the webrtcbin -> decodebin offer a timestamp?

Thanks in advance, Michael

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Michael-2]

also already added "framerate" as MediaConstraints but still

video/x-raw, format=(string)I420, width=(int)640, height=(int)480, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)jpeg, colorimetry=(string)bt601, framerate=(fraction)0/1

// navigator.mediaDevices.getSupportedConstraints()

const stream = await navigator.mediaDevices.getUserMedia({

"audio": true,

// https://developer.mozilla.org/en-US/docs/Web/API/MediaTrackConstraints

"video": {

frameRate: {

exact: 25,

ideal: 25

}

}

});

console.debug("md::getUserMedia sucess",stream);

//Play it

addVideoForStream(stream,true);

// we add stream later

pc.addStream(stream);

I am lost in this case, now.

Am Di., 12. Mai 2020 um 18:53 Uhr schrieb Michael <[hidden email]>:

Hello Thibault,

but what might be the reason and it caught my eye now because my latency is ramping up from ms to ms, is that I get `framerate=(fraction)0/1`after the decodebin which is linked to webrtcbin.

on_incoming_decodebin_pad:

video/x-raw, format=(string)I420, width=(int)640, height=(int)480, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)jpeg, colorimetry=(string)bt601, framerate=(fraction)0/1

I will deliver the rest later.

Thanks, Michael

Am Mo., 11. Mai 2020 um 17:05 Uhr schrieb Thibault Saunier <[hidden email]>:

Hello,

I am not sure why it doesn't work, but could give us a way to reproduce easily or get us a backtrace (set G_DEBUG=fatal-criticals so that your app abort when this critical happens) and a `GST_DEBUG=5` debug log file?

- Thibault

On Fri, May 8, 2020 at 2:53 PM Michael <[hidden email]> wrote:

Hello,

I try to add a `timecodestamper` and a `timeoverlay` after a `decodebin`

which I put after a `webrtcbin`

def on_add_stream(self,element, pad):

of a `webrtcbin`.

But I always get 

** (python3:1527): CRITICAL **: 18:44:43.253: gst_video_time_code_add_frames: assertion 'gst_video_time_code_is_valid (tc)' failed

Why isn't that possible?
Or better why doesn't the webrtcbin -> decodebin offer a timestamp?

Thanks in advance, Michael

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Fri, 2020-05-08 at 20:52 +0200, Michael wrote:

>
> I try to add a `timecodestamper` and a `timeoverlay` after a
> `decodebin`
>
> [...]
>
> which I put after a `webrtcbin`
>
> ** (python3:1527): CRITICAL **: 18:44:43.253:
> gst_video_time_code_add_frames: assertion
> 'gst_video_time_code_is_valid (tc)' failed
>
> Why isn't that possible?
> Or better why doesn't the webrtcbin -> decodebin offer a timestamp?

The problem is not about timestamps here. Note that timecodes and
timestamps are completely different things, the former acting more like
a "frame label" while the latter are used for actual synchronization.
webrtcbin (and RTP streams in general) provide timestamp information
but not timecodes. There are some RTP extensions for carrying timecodes
over RTP though, if that's something you need, but this is not
standardized for WebRTC. You could make it work between GStreamer
applications, or other custom applications, but web browsers would
simply ignore that.

For the actual problem here with timecodestamper: RTP streams generally
have no fixed framerate, so you get 0/1 as framerate. Timecodes however
only make sense with a fixed framerate, so the above always gives you
invalid timecodes.

To solve this you need to use a videorate element before
timecodestamper to enforce a fixed framerate, and after videorate a
capsfilter that specifies the framerate you want to have. Then
timecodestamper gets a fixed framerate stream of a given framerate, and
can properly generate timecodes for that.

What do you want to use the timecodes for?

--
Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Michael-2]

Hello Sebastian,

thank you for your answer. 

Why I wanted to habe timecodes is because webrtcbin is added to a videomixer and has a lot of latency as well as if it is getting visible after the mixer it feels like in slow motion.

As you might guess latency get's bigger and bigger and audio is completely out of syn and around 20-30 sec off. 

I already added a latency tracing but nothin obvious is shown there. 🤷‍♂️

So to further investigate this I wanted to see how the latency increases because.... maybe of the framerate? 

Anyways it was just a approach because I need to find out why webrtcbin in combination videomixer has such a tremendous latency and is completely off to use as a kind of call-in approach into a live broadcast with webrtc (all in a docker container, btw). 

webrtcbin -> decodebin -> queues/converter/videoscale/... -> videomixer/compositor -> ....

Any idea, here would be my pipeline?

Full -> https://drive.google.com/file/d/1g7iCNGEIhPwhkEvGFCXW9lyQYXv3keKu/view?usp=sharing

Regards, Michael

Sebastian Dröge <[hidden email]> schrieb am Sa., 16. Mai 2020, 10:33:

On Fri, 2020-05-08 at 20:52 +0200, Michael wrote:
>
> I try to add a `timecodestamper` and a `timeoverlay` after a
> `decodebin`
>
> [...]
>
> which I put after a `webrtcbin`
>
> ** (python3:1527): CRITICAL **: 18:44:43.253:
> gst_video_time_code_add_frames: assertion
> 'gst_video_time_code_is_valid (tc)' failed
>
> Why isn't that possible?
> Or better why doesn't the webrtcbin -> decodebin offer a timestamp?

The problem is not about timestamps here. Note that timecodes and
timestamps are completely different things, the former acting more like
a "frame label" while the latter are used for actual synchronization.
webrtcbin (and RTP streams in general) provide timestamp information
but not timecodes. There are some RTP extensions for carrying timecodes
over RTP though, if that's something you need, but this is not
standardized for WebRTC. You could make it work between GStreamer
applications, or other custom applications, but web browsers would
simply ignore that.

For the actual problem here with timecodestamper: RTP streams generally
have no fixed framerate, so you get 0/1 as framerate. Timecodes however
only make sense with a fixed framerate, so the above always gives you
invalid timecodes.

To solve this you need to use a videorate element before
timecodestamper to enforce a fixed framerate, and after videorate a
capsfilter that specifies the framerate you want to have. Then
timecodestamper gets a fixed framerate stream of a given framerate, and
can properly generate timecodes for that.

What do you want to use the timecodes for?

--
Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

Hi Michael,

On Thu, 2020-05-21 at 16:29 +0200, Michael wrote:
>
> Why I wanted to habe timecodes is because webrtcbin is added to a
> videomixer and has a lot of latency as well as if it is getting
> visible after the mixer it feels like in slow motion.
> As you might guess latency get's bigger and bigger and audio is
> completely out of syn and around 20-30 sec off.

As said, timecodes are not for synchronization so you're looking at the
wrong solution here :)

Also videomixer is not going to work well for live input. Please try
using the compositor element instead, which is the successor of
videomixer and correctly handles live streams.

> I already added a latency tracing but nothin obvious is shown there.
> 🤷‍♂️
>
> So to further investigate this I wanted to see how the latency
> increases because.... maybe of the framerate?

What's the sender and are you sure it's not the sender that is causing
the latency increase? Try simplifying your pipeline and directly output
to display/speakers to see if that behaves correctly. Then step by step
add everything else back in to see when things start to misbehave.

Generally this is supposed to work well but would need some further
debugging and details.

> Any idea, here would be my pipeline?
>
> Full ->
> https://drive.google.com/file/d/1g7iCNGEIhPwhkEvGFCXW9lyQYXv3keKu/view?usp=sharing

The resolution is a bit low, none of the text can be read here :)

--
Sebastian Dröge, Centricular Ltd · https://www.centricular.com



_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel