Synchronization precision

[RiccardoCagnasso]

At gstreamer conference 2017, doctor Raghavan gave a presentation about his
library that allowed to create simple synchronized. During his conference he
did highlight something that I had encountered before: the synchronized
playback is not really perfectly accurate.

There's usually, an random error that is maximum less than a frame. This
means that "sometimes", usually once in several seconds, a frame is rendered
slight after or before its counterpart. This seems to happen with the same
extent whether the two videos are played on the same computer or on two
distinct connected via ethernet (which makes sense, since the delay of
ethernet is sub-1ms).

In practical scenarios, the error is unnoticeable if any kind of bezel is
involved, e.g. any kind of video wall, but it becomes apparent on a
continuous screen, e.g. a ledwall. It's also quite easy to spot using the
"slow motion" feature (120 fps) of any smartphone.

https://youtu.be/-1EhKn8E5zY here is an example taken from a very big
videowall.

Is there any kind of active development trying to isolate/fix/mitigate this
issue? With the availability of sub 1mm pitch ledwall, there's definitely a
market for synchronized playback. It's not immediately obvious how to play a
16k video on a number of smaller, contiguous screens.

We don't have the resources nor the expertise to work on this, but we might
be able to provide some (very, very limited. small company with a different
core business) financial incentive to someone tackling this.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le lun. 1 juill. 2019 15 h 40, RiccardoCagnasso <[hidden email]> a écrit :

At gstreamer conference 2017, doctor Raghavan gave a presentation about his
library that allowed to create simple synchronized. During his conference he
did highlight something that I had encountered before: the synchronized
playback is not really perfectly accurate.

There's usually, an random error that is maximum less than a frame. This
means that "sometimes", usually once in several seconds, a frame is rendered
slight after or before its counterpart. This seems to happen with the same
extent whether the two videos are played on the same computer or on two
distinct connected via ethernet (which makes sense, since the delay of
ethernet is sub-1ms).

In practical scenarios, the error is unnoticeable if any kind of bezel is
involved, e.g. any kind of video wall, but it becomes apparent on a
continuous screen, e.g. a ledwall. It's also quite easy to spot using the
"slow motion" feature (120 fps) of any smartphone.

https://youtu.be/-1EhKn8E5zY here is an example taken from a very big
videowall.

Is there any kind of active development trying to isolate/fix/mitigate this
issue? With the availability of sub 1mm pitch ledwall, there's definitely a
market for synchronized playback. It's not immediately obvious how to play a
16k video on a number of smaller, contiguous screens.

I believe some vendors have a solution, but they kept it for themselves. I have tackled the idea, but the first thing I notice (with seperate screens) is that I would also need to sync the display clock, and that requires more hardware.

Meanwhile I've experimented with Wayland presentation timestamp, since none of our display sync have controllable and reliable frame scheduling. I was told the GStreamer network clock can achieve less the 2ms sync, so far I felt it was sufficient.

Let me know if you'd like to resurrect that wayland branch. I don't know if it makes sense with freesync or other HW protocol, it barely existed when I was looking at this.

We don't have the resources nor the expertise to work on this, but we might
be able to provide some (very, very limited. small company with a different
core business) financial incentive to someone tackling this.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[RiccardoCagnasso]

Nicolas Dufresne-5 wrote
> I believe some vendors have a solution, but they kept it for themselves. I
> have tackled the idea, but the first thing I notice (with seperate
> screens)
> is that I would also need to sync the display clock, and that requires
> more
> hardware.

Well you can sync the display clock - in theory - with nvidia quadros and
they are as cheap as 100 euro.
If you need to sync the display clock of different computers, you can use
quadro sync cards. They are sure expensive and probably a mess to configure
on linux. But not nearly as expensive as the solutions that are used to
tackle the problem in the industry now.

If we can get near-perfect sync on the same monitor, then the problem should
be solved with hardware and OS configuration. But, as far as I know, at the
moment, the problem presents itself on the same video card and even on the
same monitor.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le lundi 01 juillet 2019 à 17:44 -0500, RiccardoCagnasso a écrit :

> Nicolas Dufresne-5 wrote
> > I believe some vendors have a solution, but they kept it for themselves. I
> > have tackled the idea, but the first thing I notice (with seperate
> > screens)
> > is that I would also need to sync the display clock, and that requires
> > more
> > hardware.
>
> Well you can sync the display clock - in theory - with nvidia quadros and
> they are as cheap as 100 euro.
> If you need to sync the display clock of different computers, you can use
> quadro sync cards. They are sure expensive and probably a mess to configure
> on linux. But not nearly as expensive as the solutions that are used to
> tackle the problem in the industry now.
>
> If we can get near-perfect sync on the same monitor, then the problem should
> be solved with hardware and OS configuration. But, as far as I know, at the
> moment, the problem presents itself on the same video card and even on the
> same monitor.

This last sentence surprise me. I did quite some test, and on the same
monitor, I have never seen a de-sync. But the was not with Arun library
though. Maybe you need a bit of a debugging sessions.

>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[RiccardoCagnasso]

Nicolas Dufresne-5 wrote
>
> This last sentence surprise me. I did quite some test, and on the same
> monitor, I have never seen a de-sync. But the was not with Arun library
> though. Maybe you need a bit of a debugging sessions.

I don't use Arun library, but my own code is very similar. There might be a
bug, yes, but I don't think this is the case. For two reasons.

First the code is very simple.
Second, I routinely use a single video played on a borderless window that
spans across multiple outputs on big led walls and there's no perceivable
de-sync.
This means that the de-sync that you see across different monitors, is not
caused by the "multiple monitors".

I have somewhere a video of two synchronized playbins, one next to the
other, on my laptop screen. The video is recorded in 120 fps slow motion and
the de-sync every couple of second is quite clear.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mar. 2 juill. 2019 06 h 10, RiccardoCagnasso <[hidden email]> a écrit :

Nicolas Dufresne-5 wrote
>
> This last sentence surprise me. I did quite some test, and on the same
> monitor, I have never seen a de-sync. But the was not with Arun library
> though. Maybe you need a bit of a debugging sessions.

I don't use Arun library, but my own code is very similar. There might be a
bug, yes, but I don't think this is the case. For two reasons.

First the code is very simple.
Second, I routinely use a single video played on a borderless window that
spans across multiple outputs on big led walls and there's no perceivable
de-sync.
This means that the de-sync that you see across different monitors, is not
caused by the "multiple monitors".

I have somewhere a video of two synchronized playbins, one next to the
other, on my laptop screen. The video is recorded in 120 fps slow motion and
the de-sync every couple of second is quite clear.

At 120fps, I wonder if the normal Linux scheduler can work. Just to remind, what we do in sink is simply wait on the pipeline clock and render. Whenever you are on the edge of the vblank, even if you have perfect clock, a slight scheduling jitter will lead to the image being rendered on unpredictable vblank.

What my waylandsink experiment was about, was to use the presentation timestamp to predict the effective vblank time (on the pipeline clock) and initiate the render right after that vblank. This way the renderer could tolerate as much jitter as the vblank interval, hence would lead to a much more predictable render. To me what you observe at high framerate seems to match, but it's always hard to be certain.

P.s. Right now this would work with freesync or other tech as long as your compositor rendering is paced. If the compositor was pacing the render based on app activity, the stats would end up unstable.

--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[RiccardoCagnasso]

Nicolas Dufresne-5 wrote
> Le mar. 2 juill. 2019 06 h 10, RiccardoCagnasso &lt;

> riccardo@

> &gt; a
> écrit :
>
>> Nicolas Dufresne-5 wrote
>> >
>> > This last sentence surprise me. I did quite some test, and on the same
>> > monitor, I have never seen a de-sync. But the was not with Arun library
>> > though. Maybe you need a bit of a debugging sessions.
>>
>> I don't use Arun library, but my own code is very similar. There might be
>> a
>> bug, yes, but I don't think this is the case. For two reasons.
>>
>> First the code is very simple.
>> Second, I routinely use a single video played on a borderless window that
>> spans across multiple outputs on big led walls and there's no perceivable
>> de-sync.
>> This means that the de-sync that you see across different monitors, is
>> not
>> caused by the "multiple monitors".
>>
>> I have somewhere a video of two synchronized playbins, one next to the
>> other, on my laptop screen. The video is recorded in 120 fps slow motion
>> and
>> the de-sync every couple of second is quite clear.
>>
>
> At 120fps, I wonder if the normal Linux scheduler can work. Just to
> remind,
> what we do in sink is simply wait on the pipeline clock and render.
> Whenever you are on the edge of the vblank, even if you have perfect
> clock,
> a slight scheduling jitter will lead to the image being rendered on
> unpredictable vblank.
>
> What my waylandsink experiment was about, was to use the presentation
> timestamp to predict the effective vblank time (on the pipeline clock) and
> initiate the render right after that vblank. This way the renderer could
> tolerate as much jitter as the vblank interval, hence would lead to a much
> more predictable render. To me what you observe at high framerate seems to
> match, but it's always hard to be certain.
>
> P.s. Right now this would work with freesync or other tech as long as your
> compositor rendering is paced. If the compositor was pacing the render
> based on app activity, the stats would end up unstable.
>
>
>>
>>
>> --
>> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
>> _______________________________________________
>> gstreamer-devel mailing list
>>

> gstreamer-devel@.freedesktop

>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
> _______________________________________________
> gstreamer-devel mailing list

> gstreamer-devel@.freedesktop

> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

What I meant is that I recorded my laptop screen with a 120 fps camera (my
smartphone). Crude but effective, the desync shows up nicely.

What you are saying about vsync makes perfectly sense to me. If a screen is
60hz you have a vblank every 16 ms. If two processes write images on the
screen, without any synchronization with the vsync and with a error of about
2 ms between their own clocks, there will be a certain probability that two
matching frame will be rendered after different vblanks. Which is perfectly
consistent with what we see on camera.

I don't know much about wayland and freesync, but sure. If you can guarantee
that the frame is rendered right after the vblank, this should work just
fine. How can I help?





--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Mon, 2019-07-01 at 14:26 -0500, RiccardoCagnasso wrote:

Hi,

> At gstreamer conference 2017, doctor Raghavan gave a presentation
> about his library that allowed to create simple synchronized. During
> his conference he did highlight something that I had encountered
> before: the synchronized playback is not really perfectly accurate.
>
> There's usually, an random error that is maximum less than a frame.
> This means that "sometimes", usually once in several seconds, a frame
> is rendered slight after or before its counterpart. This seems to
> happen with the same extent whether the two videos are played on the
> same computer or on two distinct connected via ethernet (which makes
> sense, since the delay of ethernet is sub-1ms).
>
> In practical scenarios, the error is unnoticeable if any kind of
> bezel is involved, e.g. any kind of video wall, but it becomes
> apparent on a continuous screen, e.g. a ledwall. It's also quite easy
> to spot using the "slow motion" feature (120 fps) of any smartphone.
>
> https://youtu.be/-1EhKn8E5zY here is an example taken from a very big
> videowall.

For what it's worth, Marc Leeman also gave a talk about this at the
GStreamer Conference 2011:

https://gstconf.ubicast.tv/videos/displaying-sychronized-video-2

(Don't remember into how much detail it went in terms of solutions
though, just thought it might be of interest).

Cheers
 Tim
--
Tim Müller, Centricular Ltd - http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[RiccardoCagnasso]

Wait a second. If I understand correctly, would it mean that the solution
could simply be to enable vertical synchrony in glimagesink (vulkansink)?



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Marc Leeman]

For what it's worth, Marc Leeman also gave a talk about this at the
GStreamer Conference 2011:

https://gstconf.ubicast.tv/videos/displaying-sychronized-video-2

(Don't remember into how much detail it went in terms of solutions
though, just thought it might be of interest).

The demo back then was in GStreamer 0.10 and was using GstNetClock. It was sync'd  to less than one frame. I will need to set it up gain using GStreamer 1.x

That is the slight difference between 'content lock' and 'frame lock'. For getting the lock better than 'content lock', you'll need to steer the vsync of the displays.

--

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mardi 02 juillet 2019 à 11:00 -0500, RiccardoCagnasso a écrit :

> What I meant is that I recorded my laptop screen with a 120 fps camera (my
> smartphone). Crude but effective, the desync shows up nicely.
>
> What you are saying about vsync makes perfectly sense to me. If a screen is
> 60hz you have a vblank every 16 ms. If two processes write images on the
> screen, without any synchronization with the vsync and with a error of about
> 2 ms between their own clocks, there will be a certain probability that two
> matching frame will be rendered after different vblanks. Which is perfectly
> consistent with what we see on camera.
>
> I don't know much about wayland and freesync, but sure. If you can guarantee
> that the frame is rendered right after the vblank, this should work just
> fine. How can I help?

As said, I haven't worked on that in the last two years. I have
uploaded my ancient branch. Back then, Arun helped me reproduce some
corner cases. Note that seeking was broken, I didn't have time to look
at this.

https://gitlab.freedesktop.org/ndufresne/gst-plugins-bad/commits/wayland-presentation

What it does, it implements GstWlClock, that reuses the WlDisplay
thread for async clock id. All renders now happens on the WlDisplay
thread (that was removing a lot of race with the IPC). The part that
likely need work is the scheduling. Basically it takes the render time
of a buffer and choose ahead of time which vblank will be best. It is
then scheduled for render using an async clock id. There was some
corner cases I also started to tackle, like playback 120fps over 60Hz
display and vis-versa. The hit strategy was to allow rendering half a
frame before after the running time target. That allowed me reducing
the render latency by half a frame.

One thing you need to know, is that because the render and the stream
are generally out of sync, the latency will jitter with a very specific
pattern. The problem we wanted to tackled can be well observed in ones
of Arun observation here:

https://imgur.com/a/7VijX

Near the edge, the render goes back and forth on one vblank. At some
point, this can be more then one frame off, and that would be visible
with your camera test. Similarly, if you have two display with an out-
of-sync vblank, you will naturally have this side effect. You can find
more information on this issue:

https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/402

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mardi 02 juillet 2019 à 12:07 -0500, RiccardoCagnasso a écrit :
> Wait a second. If I understand correctly, would it mean that the solution
> could simply be to enable vertical synchrony in glimagesink (vulkansink)?

Vertical synchronization (waiting for the vblank to pass) would ensure
that you systematically render on the next vblank. Though, if your
system has a higher then usual load, this next vblank may drift back
and forth. It also creates gaps where the stream thread goes idea,
which may cause frame rate drops.

I believe there is another mechanism in GL, I forgot the name,
something around async render. Which you can sync with some frame
counter. I have never used that one, but the Wayland Presentation
Timestamp protocol is meant to be used to implement this feature in GL
through a compositor. That being said, only Weston implements this
protocol, I have no idea if Mesa is using it.

Nicolas

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[RiccardoCagnasso]

I took some time to read your answer, the discussion you linked and some of
the code in your branch. I think I managed to understand what you are trying
to do and what problems you are facing, at least to an extent (which is an
achievement by itself). I'm not sure that I can give any meaningful insight
about that though. Except for maybe the fact that synchronizing the
gstreamer clock with the wayland presentation layer that way doesn't sound
right to me. It feels like the process of synchronizing with vblanks shoud
be confined to the video sink and not have a larger impact on the pipeline
(and clock) as a whole. But this is probably a stupid observation since I
don't really know much about gstreamer internals, so meh.

What I also did, possibly less pointless, is to make some tests with
glimagesink. I'll link some results here because they are puzzling.

1) https://youtu.be/RFRUZrsllnU
This is my laptop, with an Intel video card, vaapidecode/vaapisink and both
videos on the same monitor. You can see random desyncs consistent with what
we discussed.

2) https://youtu.be/C2lDp-Lh8B8
This is one of my workstations with a Quadro P400. Vsync is enabled and full
composition pipeline. nvdec and glimagesink. Both videos on one monitor. The
sync is perfect as far as I can see.

3) https://youtu.be/wwvfbHGh4a0
This is the same workstation but with one video on each monitor. One video
seems exactly one frame later than the second.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le dim. 7 juill. 2019 08 h 25, RiccardoCagnasso <[hidden email]> a écrit :

I took some time to read your answer, the discussion you linked and some of
the code in your branch. I think I managed to understand what you are trying
to do and what problems you are facing, at least to an extent (which is an
achievement by itself). I'm not sure that I can give any meaningful insight
about that though. Except for maybe the fact that synchronizing the
gstreamer clock with the wayland presentation layer that way doesn't sound
right to me. It feels like the process of synchronizing with vblanks shoud
be confined to the video sink and not have a larger impact on the pipeline
(and clock) as a whole. But this is probably a stupid observation since I
don't really know much about gstreamer internals, so meh.

The slight thing you have missed is that this is doing the exact opposite. The Wayland clock is being slaved to the pipeline clock, hence have no impact on the pipeline. So it is confined. That being said, if that clock was the only one available, it would make sense to provide it, that wasn't anything I wanted to achieve, specially that I'd need an hybrid clock in order to increase precision.

What I also did, possibly less pointless, is to make some tests with
glimagesink. I'll link some results here because they are puzzling.

1) https://youtu.be/RFRUZrsllnU
This is my laptop, with an Intel video card, vaapidecode/vaapisink and both
videos on the same monitor. You can see random desyncs consistent with what
we discussed.

The desync is likely not random though, you'd need a better test.

2) https://youtu.be/C2lDp-Lh8B8
This is one of my workstations with a Quadro P400. Vsync is enabled and full
composition pipeline. nvdec and glimagesink. Both videos on one monitor. The
sync is perfect as far as I can see.

3) https://youtu.be/wwvfbHGh4a0
This is the same workstation but with one video on each monitor. One video
seems exactly one frame later than the second. 

I'm not sure if we can with this test see the difference between .5 frame delay and 1 frame delay.

--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel