Trouble syncing 2 mono audio IP cams into stereo

[Shervin Emami]

Hi,

I'm trying to convert mono (G.711 mulaw) streams from 2 IP cameras into a single stereo stream, but I've had a lot of trouble getting the Left & Right channel to stay in sync. Sometimes the 2 channels are perfectly in sync but sometimes they are 125ms out of sync. Here is my current pipeline (using Gstreamer 1.2.4 on Jetson TK1):

gst-launch-1.0 -e \
        interleave name=i ! \
        audioconvert ! avimux name=mux ! filesink location=out.avi \
        rtspsrc location=rtsp://camL latency=2000 name=in1 \
        in1. ! queue min-threshold-time=200000000 ! application/x-rtp, media=audio ! rtppcmudepay ! mulawdec tolerance=1000000 ! audiorate tolerance=150000000 ! audioconvert ! audio/x-raw,channels=1,channel-mask=(bitmask)0x1 ! queue ! i. \
        rtspsrc location=rtsp://camR latency=2000 name=in2 \
        in2. ! queue min-threshold-time=200000000 ! application/x-rtp, media=audio ! rtppcmudepay ! mulawdec tolerance=1000000 ! audiorate tolerance=150000000 ! audioconvert ! audio/x-raw,channels=1,channel-mask=(bitmask)0x2 ! queue ! i.

If I reduce some of the parameters I also get some dropped audio packets, but the pipeline above has mostly perfect audio but has left-right sync issues. I will pay anyone that helps me get left & right in sync properly.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Do, 2016-06-16 at 18:51 +1000, Shervin Emami wrote:
> Hi,
>
> I'm trying to convert mono (G.711 mulaw) streams from 2 IP cameras
> into a single stereo stream, but I've had a lot of trouble getting
> the Left & Right channel to stay in sync. Sometimes the 2 channels
> are perfectly in sync but sometimes they are 125ms out of sync. Here
> is my current pipeline (using Gstreamer 1.2.4 on Jetson TK1):
> [...]

The interleave element in 1.2 did not properly synchronize. You should
update to 1.8 and use the new audiointerleave element instead. That one
will synchronize the audio correctly according to its timestamps.

--

Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Shervin Emami]

Thanks Sebastian. I am using Gstreamer 1.2 because the NVIDIA Tegra K1 OpenMAX plugins (for video codecs) are provided just for that version. Do you think we are likely to be able to port NVIDIA's OMX codec drivers from 1.2 to 1.8, or port Gstreamer's audiointerleave element from 1.8 down to 1.2? Or would it would be really hard either way? We're getting to the point where we might have to give up on our 7 month project because we can't get audio and video to sync together in Gstreamer!

On Fri, Jun 17, 2016 at 9:31 PM, Sebastian Dröge <[hidden email]> wrote:

On Do, 2016-06-16 at 18:51 +1000, Shervin Emami wrote:
> Hi,
>
> I'm trying to convert mono (G.711 mulaw) streams from 2 IP cameras
> into a single stereo stream, but I've had a lot of trouble getting
> the Left & Right channel to stay in sync. Sometimes the 2 channels
> are perfectly in sync but sometimes they are 125ms out of sync. Here
> is my current pipeline (using Gstreamer 1.2.4 on Jetson TK1):
> [...]

The interleave element in 1.2 did not properly synchronize. You should
update to 1.8 and use the new audiointerleave element instead. That one
will synchronize the audio correctly according to its timestamps.

--

Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Mi, 2016-06-22 at 13:39 +1000, Shervin Emami wrote:
> Thanks Sebastian. I am using Gstreamer 1.2 because the NVIDIA Tegra
> K1 OpenMAX plugins (for video codecs) are provided just for that
> version. Do you think we are likely to be able to port NVIDIA's OMX
> codec drivers from 1.2 to 1.8, or port Gstreamer's audiointerleave
> element from 1.8 down to 1.2? Or would it would be really hard either
> way? We're getting to the point where we might have to give up on our
> 7 month project because we can't get audio and video to sync together
> in Gstreamer!

NVIDIA should really update to a recent GStreamer version, but I know
that some people are using 1.8 on the TK1 so it must be possible.

Let us know what problems you run into when trying to update.

--

Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Kammacher Tobias (kamm)]

Hi

I have used GStreamer 1.8 on both Nvidia TK1 and TX1. In both cases
the default GSt provided by Nvidia is 1.2.4.
It is possible to build the newer GStreamer and just copy the original OMX
and Nvidia plugins:
cd /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/
cp libgstnv* libnvgst* libgstomx.so ~/gst_1.6.0/out/lib/gstreamer-1.0/

Most of them worked for me on 1.8.0, except nvvidconv, which is a
proprietary plugin from Nvidia.

There are instructions for doing this also from Nvidia if you look at the
Multimedia User Guide for TX1 24.1 [1]. Also there is a script for building
newer GStreamer versions for the TX1, but I never tried this on TK1.


[1] http://developer.download.nvidia.com/embedded/L4T/r24_Release_v1.0/Docs/L4T_Tegra_X1_Multimedia_User_Guide_Release_24.1.pdf
--
Kammacher Tobias, MSc ZFH in Engineering
Research Assistant

ZHAW, Zurich University of Applied Sciences
InES, Institute of Embedded Systems
Postfach
Technikumstr. 22
CH-8401 Winterthur
--
Tel: [phone]
Web: http://ines.zhaw.ch
--
________________________________________
Von: gstreamer-devel <[hidden email]> im Auftrag von Sebastian Dröge <[hidden email]>
Gesendet: Mittwoch, 22. Juni 2016 08:41
An: Discussion of the development of and with GStreamer
Betreff: Re: Trouble syncing 2 mono audio IP cams into stereo

On Mi, 2016-06-22 at 13:39 +1000, Shervin Emami wrote:
> Thanks Sebastian. I am using Gstreamer 1.2 because the NVIDIA Tegra
> K1 OpenMAX plugins (for video codecs) are provided just for that
> version. Do you think we are likely to be able to port NVIDIA's OMX
> codec drivers from 1.2 to 1.8, or port Gstreamer's audiointerleave
> element from 1.8 down to 1.2? Or would it would be really hard either
> way? We're getting to the point where we might have to give up on our
> 7 month project because we can't get audio and video to sync together
> in Gstreamer!

NVIDIA should really update to a recent GStreamer version, but I know
that some people are using 1.8 on the TK1 so it must be possible.

Let us know what problems you run into when trying to update.

--

Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Shervin Emami]

Thanks Kammacher,

Yes I know that some people on NVIDIA's forum have been able to built latest Gstreamer for TK1, but I haven't found anyone that got nvvidconv working in a recent Gstreamer, and I need nvvidconv for the video part of my pipeline. Do you know if anyone has been able to get nvvidconv working for a recent version of Gstreamer on TK1? Because our other option is to use NVIDIA's gstreamer 1.2 for our video pipeline, and use open-source Gstreamer 1.8 for the audio pipeline at the same time, and get Gstreamer 1.8 to multiplex the video (from the Gstreamer 1.2 pipeline) with the audio into an AVI file. But obviously that's a messy solution so I'm hoping we can find a way to get Gstreamer 1.8 working on Jetson TK1 including nvvidconv, omxh264dec and omxh264enc.

On Wed, Jun 22, 2016 at 5:00 PM, Kammacher Tobias (kamm) <[hidden email]> wrote:

Hi

I have used GStreamer 1.8 on both Nvidia TK1 and TX1. In both cases
the default GSt provided by Nvidia is 1.2.4.
It is possible to build the newer GStreamer and just copy the original OMX
and Nvidia plugins:
cd /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/
cp libgstnv* libnvgst* libgstomx.so ~/gst_1.6.0/out/lib/gstreamer-1.0/

Most of them worked for me on 1.8.0, except nvvidconv, which is a
proprietary plugin from Nvidia.

There are instructions for doing this also from Nvidia if you look at the
Multimedia User Guide for TX1 24.1 [1]. Also there is a script for building
newer GStreamer versions for the TX1, but I never tried this on TK1.


[1] http://developer.download.nvidia.com/embedded/L4T/r24_Release_v1.0/Docs/L4T_Tegra_X1_Multimedia_User_Guide_Release_24.1.pdf
--
Kammacher Tobias, MSc ZFH in Engineering
Research Assistant

ZHAW, Zurich University of Applied Sciences
InES, Institute of Embedded Systems
Postfach
Technikumstr. 22
CH-8401 Winterthur
--
Tel: <a href="tel:%2B41%2058%20934%2078%2038" value="[phone]">[phone]
Web: http://ines.zhaw.ch
--
________________________________________
Von: gstreamer-devel <[hidden email]> im Auftrag von Sebastian Dröge <[hidden email]>
Gesendet: Mittwoch, 22. Juni 2016 08:41
An: Discussion of the development of and with GStreamer
Betreff: Re: Trouble syncing 2 mono audio IP cams into stereo

On Mi, 2016-06-22 at 13:39 +1000, Shervin Emami wrote:
> Thanks Sebastian. I am using Gstreamer 1.2 because the NVIDIA Tegra
> K1 OpenMAX plugins (for video codecs) are provided just for that
> version. Do you think we are likely to be able to port NVIDIA's OMX
> codec drivers from 1.2 to 1.8, or port Gstreamer's audiointerleave
> element from 1.8 down to 1.2? Or would it would be really hard either
> way? We're getting to the point where we might have to give up on our
> 7 month project because we can't get audio and video to sync together
> in Gstreamer!

NVIDIA should really update to a recent GStreamer version, but I know
that some people are using 1.8 on the TK1 so it must be possible.

Let us know what problems you run into when trying to update.

--

Sebastian Dröge, Centricular Ltd · http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Matthew Waters]

On 22/06/16 17:00, Kammacher Tobias (kamm) wrote:
> Hi
>
> I have used GStreamer 1.8 on both Nvidia TK1 and TX1. In both cases
> the default GSt provided by Nvidia is 1.2.4.
> It is possible to build the newer GStreamer and just copy the original OMX
> and Nvidia plugins:
> cd /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/
> cp libgstnv* libnvgst* libgstomx.so ~/gst_1.6.0/out/lib/gstreamer-1.0/

Yes, this works well enough without zerocopy encoding/decoding.  I
haven't used nvvidconv and don't have a TX1.

If you need zerocopy between the decoder and upstream libgstgl, you'll
need to patch gst-omx to know about libgstgl.  Fortunately, I have
already created some branches for that purpose that work with GStreamer 1.8:
https://github.com/ystreet/gst-omx-nvidia/
https://github.com/ystreet/gst-plugins-bad/tree/nvidia-1.8

The next GStreamer release (1.10) changes this slightly to more
performant implementation of the EGLImage memory.  I'm not aware of any
branch that deals with that yet.  From 1.10 onwards, something like
https://cgit.freedesktop.org/gstreamer/gst-omx/commit/?id=27d2cdd45dcf0bbd58c753e4b33058607153c4c1
would need to happen in nvidia's fork of gst-omx or someone would need
to forward port nvidia's gst-omx to upstream gst-omx.

Hope that helps
-Matt

> Most of them worked for me on 1.8.0, except nvvidconv, which is a
> proprietary plugin from Nvidia.
>
> There are instructions for doing this also from Nvidia if you look at the
> Multimedia User Guide for TX1 24.1 [1]. Also there is a script for building
> newer GStreamer versions for the TX1, but I never tried this on TK1.
>
>
> [1] http://developer.download.nvidia.com/embedded/L4T/r24_Release_v1.0/Docs/L4T_Tegra_X1_Multimedia_User_Guide_Release_24.1.pdf



_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Shervin Emami]

Thanks Matt, I'll try your branches.

I didn't see nvvidconv in your branch, are VPU decoding & encoding working for you in Gstreamer 1.8 but not nvvidconv (for color format conversions or image resize)? Is there someone I can pay to get nvvidconv working in Gstreamer 1.8 for me (using VPU hardware acceleration, but perhaps without zero-copy)? I'd pay someone to get it working (potentially using my Jetson TK1 remotely) and would let them make the result public for others, if it could be done in just a few days time.

On Wed, Jun 22, 2016 at 6:32 PM, Matthew Waters <[hidden email]> wrote:

On 22/06/16 17:00, Kammacher Tobias (kamm) wrote:
> Hi
>
> I have used GStreamer 1.8 on both Nvidia TK1 and TX1. In both cases
> the default GSt provided by Nvidia is 1.2.4.
> It is possible to build the newer GStreamer and just copy the original OMX
> and Nvidia plugins:
> cd /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/
> cp libgstnv* libnvgst* libgstomx.so ~/gst_1.6.0/out/lib/gstreamer-1.0/

Yes, this works well enough without zerocopy encoding/decoding.  I
haven't used nvvidconv and don't have a TX1.

If you need zerocopy between the decoder and upstream libgstgl, you'll
need to patch gst-omx to know about libgstgl.  Fortunately, I have
already created some branches for that purpose that work with GStreamer 1.8:
https://github.com/ystreet/gst-omx-nvidia/
https://github.com/ystreet/gst-plugins-bad/tree/nvidia-1.8

The next GStreamer release (1.10) changes this slightly to more
performant implementation of the EGLImage memory.  I'm not aware of any
branch that deals with that yet.  From 1.10 onwards, something like
https://cgit.freedesktop.org/gstreamer/gst-omx/commit/?id=27d2cdd45dcf0bbd58c753e4b33058607153c4c1
would need to happen in nvidia's fork of gst-omx or someone would need
to forward port nvidia's gst-omx to upstream gst-omx.

Hope that helps
-Matt

> Most of them worked for me on 1.8.0, except nvvidconv, which is a
> proprietary plugin from Nvidia.
>
> There are instructions for doing this also from Nvidia if you look at the
> Multimedia User Guide for TX1 24.1 [1]. Also there is a script for building
> newer GStreamer versions for the TX1, but I never tried this on TK1.
>
>
> [1] http://developer.download.nvidia.com/embedded/L4T/r24_Release_v1.0/Docs/L4T_Tegra_X1_Multimedia_User_Guide_Release_24.1.pdf

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Matthew Waters]

On 22/06/16 20:10, Shervin Emami wrote:

Thanks Matt, I'll try your branches.

I didn't see nvvidconv in your branch, are VPU decoding & encoding working for you in Gstreamer 1.8 but not nvvidconv (for color format conversions or image resize)? Is there someone I can pay to get nvvidconv working in Gstreamer 1.8 for me (using VPU hardware acceleration, but perhaps without zero-copy)? I'd pay someone to get it working (potentially using my Jetson TK1 remotely) and would let them make the result public for others, if it could be done in just a few days time.

Hi,

I've never tested the nvvidconv element on the TK1.  The omx decoders work with zerocopy between libgstgl and gst-omx with the branches/repos I provided earlier.  You have to configure libgstgl for that though.

The source for nvvidconv (as well as all it's dependant libraries) is unfortunately not available so there's little an outsider can do to forward port it to a recent version of GStreamer if it doesn't work.  You'd have to talk to Nvidia for that.

Cheers
-Matt

Cheers,

On Wed, Jun 22, 2016 at 6:32 PM, Matthew Waters <[hidden email]> wrote:

On 22/06/16 17:00, Kammacher Tobias (kamm) wrote:
> Hi
>
> I have used GStreamer 1.8 on both Nvidia TK1 and TX1. In both cases
> the default GSt provided by Nvidia is 1.2.4.
> It is possible to build the newer GStreamer and just copy the original OMX
> and Nvidia plugins:
> cd /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/
> cp libgstnv* libnvgst* libgstomx.so ~/gst_1.6.0/out/lib/gstreamer-1.0/

Yes, this works well enough without zerocopy encoding/decoding.  I
haven't used nvvidconv and don't have a TX1.

If you need zerocopy between the decoder and upstream libgstgl, you'll
need to patch gst-omx to know about libgstgl.  Fortunately, I have
already created some branches for that purpose that work with GStreamer 1.8:
https://github.com/ystreet/gst-omx-nvidia/
https://github.com/ystreet/gst-plugins-bad/tree/nvidia-1.8

The next GStreamer release (1.10) changes this slightly to more
performant implementation of the EGLImage memory.  I'm not aware of any
branch that deals with that yet.  From 1.10 onwards, something like
https://cgit.freedesktop.org/gstreamer/gst-omx/commit/?id=27d2cdd45dcf0bbd58c753e4b33058607153c4c1
would need to happen in nvidia's fork of gst-omx or someone would need
to forward port nvidia's gst-omx to upstream gst-omx.

Hope that helps
-Matt

> Most of them worked for me on 1.8.0, except nvvidconv, which is a
> proprietary plugin from Nvidia.
>
> There are instructions for doing this also from Nvidia if you look at the
> Multimedia User Guide for TX1 24.1 [1]. Also there is a script for building
> newer GStreamer versions for the TX1, but I never tried this on TK1.
>
>
> [1] http://developer.download.nvidia.com/embedded/L4T/r24_Release_v1.0/Docs/L4T_Tegra_X1_Multimedia_User_Guide_Release_24.1.pdf

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Shervin Emami]

Thanks Matt for confirming that nvvidconv can't be ported except by NVIDIA.

On Wed, Jun 22, 2016 at 9:34 PM, Matthew Waters <[hidden email]> wrote:

On 22/06/16 20:10, Shervin Emami wrote:

Thanks Matt, I'll try your branches.

I didn't see nvvidconv in your branch, are VPU decoding & encoding working for you in Gstreamer 1.8 but not nvvidconv (for color format conversions or image resize)? Is there someone I can pay to get nvvidconv working in Gstreamer 1.8 for me (using VPU hardware acceleration, but perhaps without zero-copy)? I'd pay someone to get it working (potentially using my Jetson TK1 remotely) and would let them make the result public for others, if it could be done in just a few days time.

Hi,

I've never tested the nvvidconv element on the TK1.  The omx decoders work with zerocopy between libgstgl and gst-omx with the branches/repos I provided earlier.  You have to configure libgstgl for that though.

The source for nvvidconv (as well as all it's dependant libraries) is unfortunately not available so there's little an outsider can do to forward port it to a recent version of GStreamer if it doesn't work.  You'd have to talk to Nvidia for that.

Cheers
-Matt

Cheers,

On Wed, Jun 22, 2016 at 6:32 PM, Matthew Waters <[hidden email]> wrote:

On 22/06/16 17:00, Kammacher Tobias (kamm) wrote:
> Hi
>
> I have used GStreamer 1.8 on both Nvidia TK1 and TX1. In both cases
> the default GSt provided by Nvidia is 1.2.4.
> It is possible to build the newer GStreamer and just copy the original OMX
> and Nvidia plugins:
> cd /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/
> cp libgstnv* libnvgst* libgstomx.so ~/gst_1.6.0/out/lib/gstreamer-1.0/

Yes, this works well enough without zerocopy encoding/decoding.  I
haven't used nvvidconv and don't have a TX1.

If you need zerocopy between the decoder and upstream libgstgl, you'll
need to patch gst-omx to know about libgstgl.  Fortunately, I have
already created some branches for that purpose that work with GStreamer 1.8:
https://github.com/ystreet/gst-omx-nvidia/
https://github.com/ystreet/gst-plugins-bad/tree/nvidia-1.8

The next GStreamer release (1.10) changes this slightly to more
performant implementation of the EGLImage memory.  I'm not aware of any
branch that deals with that yet.  From 1.10 onwards, something like
https://cgit.freedesktop.org/gstreamer/gst-omx/commit/?id=27d2cdd45dcf0bbd58c753e4b33058607153c4c1
would need to happen in nvidia's fork of gst-omx or someone would need
to forward port nvidia's gst-omx to upstream gst-omx.

Hope that helps
-Matt

> Most of them worked for me on 1.8.0, except nvvidconv, which is a
> proprietary plugin from Nvidia.
>
> There are instructions for doing this also from Nvidia if you look at the
> Multimedia User Guide for TX1 24.1 [1]. Also there is a script for building
> newer GStreamer versions for the TX1, but I never tried this on TK1.
>
>
> [1] http://developer.download.nvidia.com/embedded/L4T/r24_Release_v1.0/Docs/L4T_Tegra_X1_Multimedia_User_Guide_Release_24.1.pdf

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel