Maximum AV sync possible with gstreamer 1.8.0 in L4T 24.2 at 23.976 FPS

[dawnpaul]

We are doing a gstreamer application which requires AV sync of less than +/- 3 ms between audio and video. We use jetson TX1, gstreamer 1.8.0 and L4T24.2. Should support 23.976 FPS. We use Sync meter to measure the difference between an audio beep and a flash. The observations are at UHD 23.976 the sync meter reading is around +1 ms and remain constant sometimes. For other session the offset is observed to be around 43 ms which is equivalent to one video frame. We tried to print the audio and video timestamp using gst_pad_add_probe() and compare. But could not figure out the issue.Could we achieve this avsync tolerance with gstreamer. Is there another method to debug the issue.

[Baby Octopus]

As you see, the resolution of AVSync GAP is 1/frameRate, which means you could get either 0 or +/-42ms(for 23.976), or anything worser than these

Its possible somewhere in your pipeline your wrongly reporting the latency or not reporting at all, so they get rendered as fast as possible. The issue could also be in certain element if it has certain jitters in PTS deduction

For most of the cases, such a jitter of 40ms could be acceptable!

[dawnpaul]

If it is latency report problem how could we debug which element is causing the issue. Could we do something like sense the latency and adjust the delay accordingly.
<AVSync GAP is 1/frameRate
Does this mean that the AV Sync delay would be a multiple of 1/framerate

[Sebastian Dröge-3]

On Thu, 2016-11-10 at 02:28 -0800, dawnpaul wrote:

> We are doing a gstreamer application which requires AV sync of less than +/-
> 3 ms between audio and video. We use jetson TX1, gstreamer 1.8.0 and
> L4T24.2. Should support 23.976 FPS. We use Sync meter to measure the
> difference between an audio beep and a flash. The observations are at UHD
> 23.976 the sync meter reading is around +1 ms and remain constant sometimes.
> For other session the offset is observed to be around 43 ms which is
> equivalent to one video frame. We tried to print the audio and video
> timestamp using gst_pad_add_probe() and compare. But could not figure out
> the issue.Could we achieve this avsync tolerance with gstreamer. Is there
> another method to debug the issue.

Getting such accurate synchronization between audio and video requires
all parts of the pipeline to work correctly, and to correctly report
their latency (in case of a live pipeline).

Especially also the video sink (and the API used by it) needs to be
able to correctly schedule frames (otherwise you get +/- 1 frame),
which is not easily possible with most APIs. Which sink do you use?

For audio, the driver needs to correctly and accurately report its
latency. Depending on the driver, those values reported can be rather
random.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[dawnpaul]

We are playing media from MP4 file. We used nveglglessink as video sink. We also tried fpsdisplaysink. In both case the AV sync issue has similar results. We used alsasink as audio sink and avdec_aac as audio decoder. We probed the video and audio timestamps at each stage of pipeline and observed that time difference between first audio and video  timestamp after the video decoder has delay approximately equal to one frame. The difference is in time at which the print is coming in gst-debug. The issue is more frequent in UHD than FHD. So could we confirm that issue is with video decoder since it is the most intensive task in the pipeline. Could we try to query the decoder latency dynamically and adjust the pipeline latency in a non live source. What could be the best method to figure out where is the issue. Issue appears only in some sessions. Approximately once in 5 sessions.

[Sebastian Dröge-3]

On Sun, 2016-11-13 at 22:14 -0800, dawnpaul wrote:

> We are playing media from MP4 file. We used nveglglessink as video sink. We
> also tried fpsdisplaysink. In both case the AV sync issue has similar
> results. We used alsasink as audio sink and avdec_aac as audio decoder. We
> probed the video and audio timestamps at each stage of pipeline and observed
> that time difference between first audio and video  timestamp after the
> video decoder has delay approximately equal to one frame. The difference is
> in time at which the print is coming in gst-debug. The issue is more
> frequent in UHD than FHD. So could we confirm that issue is with video
> decoder since it is the most intensive task in the pipeline. Could we try to
> query the decoder latency dynamically and adjust the pipeline latency in a
> non live source. What could be the best method to figure out where is the
> issue. Issue appears only in some sessions. Approximately once in 5
> sessions.

In non-live pipelines the latency does not matter as everything pre-
rolls to the sink first before output actually starts.

My best guess from your description is that the problem is inside the
video sink, and frames are being sent to OpenGL exactly at the time
when they should be displayed but that often causes them to only be
displayed on the next VSYNC (i.e. one frame too late then).

Solving this requires feedback from the driver to know when VSYNC is
happening and then based on that schedule the frames accordingly.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[dawnpaul]

Hi Sebastian
Could we confirm that the issue is not with the omxh265dec?. How could we dynamically change the frame scheduling in videosink after going to the playing state. Could we  use FBIO_WAITFORVSYNC to get the vsync time from driver.

[dawnpaul]

Hi sebastian,
We also need to enable the 10 bit output of the h265 decoder. Will that be possible by modifying any of the gst-omx\OpenMax IL code available . We found that a firmware file is loaded by the driver when gstreamer application is called. Could we get the source of this firmware from nvidia?. Should we need to modify the firmware file also. How the app calls the firmware. We are not able to figure-out the link
Would like to know how gstomxh265dec.c calls the OpenMax IL or the linux kernel driver.

Thanks

[Sebastian Dröge-3]

On Wed, 2016-11-16 at 06:08 -0800, dawnpaul wrote:
> Hi Sebastian
> Could we confirm that the issue is not with the omxh265dec?. How could we
> dynamically change the frame scheduling in videosink after going to the
> playing state.

I don't understand what you mean here, sorry.

> Could we  use FBIO_WAITFORVSYNC to get the vsync time from
> driver.

Possibly, you could make use of that in the videosink (modify the code)
to schedule frames based on the vsync.

If it's just synchronous waiting for the next vsync, then that's not
trivial though. You would need to calculate based on that when the next
vsync would most likely be (based on past observations), and then
schedule frames according to that.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Wed, 2016-11-16 at 06:19 -0800, dawnpaul wrote:
> Hi sebastian,
> We also need to enable the 10 bit output of the h265 decoder. Will that be
> possible by modifying any of the gst-omx\OpenMax IL code available . We
> found that a firmware file is loaded by the driver when gstreamer
> application is called. Could we get the source of this firmware from
> nvidia?. Should we need to modify the firmware file also. How the app calls
> the firmware. We are not able to figure-out the link

If the OpenMAX IL already supports 10 bit h265, all is fine and you
only need to modify gst-omx to add support for that color format.

Otherwise you need to talk to NVIDIA to add support for it to the
driver/firmware.

> Would like to know how gstomxh265dec.c calls the OpenMax IL or the linux
> kernel driver.

Take a look at the code really, that's too much to explain in a simple
mail. The important lower-level parts are in gstomxvideodec.c and
gstomx.c, gstomxh265dec.c is just adding the H265 bits on top.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel