Video Chat using mpegtsmux

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Video Chat using mpegtsmux

GStreamer-devel mailing list
Hi, 

I am trying to implement a video chat using GStreamer on Windows 10. I intend to use gstreamer C API, but for now I am just using the command line for basic testing. In order to keep video and audio in sync, I am muxing them using mpegtsmux and then sending them via RTP. It works fine, but the latency is pretty high (I think it's about 1 second).

My sending pipeline is as follows:
 
gst-launch-1.0 mpegtsmux name=mux ! rtpmp2tpay ! udpsink host=127.0.0.1 port=5555^
 wasapisrc ! audioresample ! audioconvert ! avenc_aac ! queue ! mux.^
 ksvideosrc ! videoconvert ! x264enc tune=zerolatency ! h264parse config-interval=-1 ! queue ! mux.^
 
My receiving pipeline is as follows:

gst-launch-1.0^
 udpsrc address=127.0.0.1 port=5555 caps="application/x-rtp" ! rtpmp2tdepay ! tsdemux name=demux^
 demux. ! queue ! avdec_aac ! audioresample ! audioconvert ! wasapisink low-latency=true^
 demux. ! queue ! h264parse ! avdec_h264 ! videoconvert ! autovideosink

Questions:
1. Is this the correct approach for a video chat or should I use separate streams for audio and video?
2. If this is the correct approach, am I using the correct muxer or should I use something else such as asfmux?
3. If this is the correct approach and the correct muxer, is there something I can do in order to reduce the latency?



_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Video Chat using mpegtsmux

GStreamer-devel mailing list
Hey,

I would recommend using webRTC (webrtcbin in the GStreamer context) for
such an application :)

Best,

--
Mathieu Duponchelle · https://www.centricular.com

On 5/19/21 8:14 AM, Gregory AE40 via gstreamer-devel wrote:
Hi, 

I am trying to implement a video chat using GStreamer on Windows 10. I intend to use gstreamer C API, but for now I am just using the command line for basic testing. In order to keep video and audio in sync, I am muxing them using mpegtsmux and then sending them via RTP. It works fine, but the latency is pretty high (I think it's about 1 second).

My sending pipeline is as follows:
 
gst-launch-1.0 mpegtsmux name=mux ! rtpmp2tpay ! udpsink host=127.0.0.1 port=5555^
 wasapisrc ! audioresample ! audioconvert ! avenc_aac ! queue ! mux.^
 ksvideosrc ! videoconvert ! x264enc tune=zerolatency ! h264parse config-interval=-1 ! queue ! mux.^
 
My receiving pipeline is as follows:

gst-launch-1.0^
 udpsrc address=127.0.0.1 port=5555 caps="application/x-rtp" ! rtpmp2tdepay ! tsdemux name=demux^
 demux. ! queue ! avdec_aac ! audioresample ! audioconvert ! wasapisink low-latency=true^
 demux. ! queue ! h264parse ! avdec_h264 ! videoconvert ! autovideosink

Questions:
1. Is this the correct approach for a video chat or should I use separate streams for audio and video?
2. If this is the correct approach, am I using the correct muxer or should I use something else such as asfmux?
3. If this is the correct approach and the correct muxer, is there something I can do in order to reduce the latency?



_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Video Chat using mpegtsmux

GStreamer-devel mailing list
Hi, thank you for the prompt response.
I wasn't specific enough, the application will run on a LAN, so I can use multicast in order to send only one stream from each PC. 
With WebRTC multicast is not possible, because it requires a separate stream for each receiver.

On Wed, May 19, 2021, 15:02 Mathieu Duponchelle <[hidden email]> wrote:
Hey,

I would recommend using webRTC (webrtcbin in the GStreamer context) for
such an application :)

Best,

--
Mathieu Duponchelle · https://www.centricular.com

On 5/19/21 8:14 AM, Gregory AE40 via gstreamer-devel wrote:
Hi, 

I am trying to implement a video chat using GStreamer on Windows 10. I intend to use gstreamer C API, but for now I am just using the command line for basic testing. In order to keep video and audio in sync, I am muxing them using mpegtsmux and then sending them via RTP. It works fine, but the latency is pretty high (I think it's about 1 second).

My sending pipeline is as follows:
 
gst-launch-1.0 mpegtsmux name=mux ! rtpmp2tpay ! udpsink host=127.0.0.1 port=5555^
 wasapisrc ! audioresample ! audioconvert ! avenc_aac ! queue ! mux.^
 ksvideosrc ! videoconvert ! x264enc tune=zerolatency ! h264parse config-interval=-1 ! queue ! mux.^
 
My receiving pipeline is as follows:

gst-launch-1.0^
 udpsrc address=127.0.0.1 port=5555 caps="application/x-rtp" ! rtpmp2tdepay ! tsdemux name=demux^
 demux. ! queue ! avdec_aac ! audioresample ! audioconvert ! wasapisink low-latency=true^
 demux. ! queue ! h264parse ! avdec_h264 ! videoconvert ! autovideosink

Questions:
1. Is this the correct approach for a video chat or should I use separate streams for audio and video?
2. If this is the correct approach, am I using the correct muxer or should I use something else such as asfmux?
3. If this is the correct approach and the correct muxer, is there something I can do in order to reduce the latency?



_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Video Chat using mpegtsmux

GStreamer-devel mailing list
Le mercredi 19 mai 2021 à 18:23 +0300, Gregory AE40 via gstreamer-devel a
écrit :
> Hi, thank you for the prompt response.
> I wasn't specific enough, the application will run on a LAN, so I can use
> multicast in order to send only one stream from each PC. 
> With WebRTC multicast is not possible, because it requires a separate stream
> for each receiver.

Yet, you should still use RTP for mulicast. For your use case, just creating an
RTSP server (using gst-rtp-server library) seem sufficient no ?

>
> On Wed, May 19, 2021, 15:02 Mathieu Duponchelle <[hidden email]>
> wrote:
> >  Hey,
> >  
> >  I would recommend using webRTC (webrtcbin in the GStreamer context) for
> >  such an application :)
> >  
> >  Best,
> >  
> > _______________________________________________
> > gstreamer-devel mailing list
> > [hidden email]
> > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Video Chat using mpegtsmux

GStreamer-devel mailing list
Yes, I understand that I should use RTP, my question is whether to mux audio and video to a single stream or to send two separate streams (with RTCP for every stream for synchronization). The first option seems more fail-safe to me in terms of audio/video synchronization.  However, I tested also the second option and its latency is lower. I guess that if you suggested to use WebRTC, I should stick with the second option, because AFAIK, WebRTC sends separate streams for video and audio. 

On Wed, May 19, 2021, 20:54 Nicolas Dufresne <[hidden email]> wrote:
Le mercredi 19 mai 2021 à 18:23 +0300, Gregory AE40 via gstreamer-devel a
écrit :
> Hi, thank you for the prompt response.
> I wasn't specific enough, the application will run on a LAN, so I can use
> multicast in order to send only one stream from each PC. 
> With WebRTC multicast is not possible, because it requires a separate stream
> for each receiver.

Yet, you should still use RTP for mulicast. For your use case, just creating an
RTSP server (using gst-rtp-server library) seem sufficient no ?

>
> On Wed, May 19, 2021, 15:02 Mathieu Duponchelle <[hidden email]>
> wrote:
> >  Hey,
> >  
> >  I would recommend using webRTC (webrtcbin in the GStreamer context) for
> >  such an application :)
> >  
> >  Best,
> >  
> > _______________________________________________
> > gstreamer-devel mailing list
> > [hidden email]
> > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel



_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Video Chat using mpegtsmux

GStreamer-devel mailing list
Le mercredi 19 mai 2021 à 21:44 +0300, Gregory AE40 a écrit :
> Yes, I understand that I should use RTP, my question is whether to mux audio
> and video to a single stream or to send two separate streams (with RTCP for
> every stream for synchronization). The first option seems more fail-safe to me
> in terms of audio/video synchronization.  However, I tested also the second
> option and its latency is lower. I guess that if you suggested to use WebRTC,
> I should stick with the second option, because AFAIK, WebRTC sends separate
> streams for video and audio. 

Correct, muxing and demuxing MPEG TS requires adding some latency due to how the
timing information is stored. We added a latency property to reduce it from the
default 700ms (maximum specified) for those that have the luxury to control the
muxing parameters, which should be your case here. Yet, you still need to
introduce a small amount of latency compare to streaming audio/video
individually.

>
> On Wed, May 19, 2021, 20:54 Nicolas Dufresne <[hidden email]> wrote:
> > Le mercredi 19 mai 2021 à 18:23 +0300, Gregory AE40 via gstreamer-devel a
> > écrit :
> > > Hi, thank you for the prompt response.
> > > I wasn't specific enough, the application will run on a LAN, so I can use
> > > multicast in order to send only one stream from each PC. 
> > > With WebRTC multicast is not possible, because it requires a separate
> > > stream
> > > for each receiver.
> >
> > Yet, you should still use RTP for mulicast. For your use case, just creating
> > an
> > RTSP server (using gst-rtp-server library) seem sufficient no ?
> >
> > >
> > > On Wed, May 19, 2021, 15:02 Mathieu Duponchelle <[hidden email]>
> > > wrote:
> > > >  Hey,
> > > >  
> > > >  I would recommend using webRTC (webrtcbin in the GStreamer context) for
> > > >  such an application :)
> > > >  
> > > >  Best,
> > > >  
> > > > _______________________________________________
> > > > gstreamer-devel mailing list
> > > > [hidden email]
> > > > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> >
> >


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Video Chat using mpegtsmux

GStreamer-devel mailing list
In reply to this post by GStreamer-devel mailing list
I echo the suggestions given by Nicolas and Mathieu. But also wanted to add that, if you chose to continue using TS, there is a “latency” property in the tsdemux element that you can tweak. By default it is set to 700ms (to comply with the TS standard), which sounds like the 1s lag you are experiencing.

Michael

On 19 May 2021, at 12:44, Gregory AE40 via gstreamer-devel <[hidden email]> wrote:

Yes, I understand that I should use RTP, my question is whether to mux audio and video to a single stream or to send two separate streams (with RTCP for every stream for synchronization). The first option seems more fail-safe to me in terms of audio/video synchronization.  However, I tested also the second option and its latency is lower. I guess that if you suggested to use WebRTC, I should stick with the second option, because AFAIK, WebRTC sends separate streams for video and audio. 

On Wed, May 19, 2021, 20:54 Nicolas Dufresne <[hidden email]> wrote:
Le mercredi 19 mai 2021 à 18:23 +0300, Gregory AE40 via gstreamer-devel a
écrit :
> Hi, thank you for the prompt response.
> I wasn't specific enough, the application will run on a LAN, so I can use
> multicast in order to send only one stream from each PC. 
> With WebRTC multicast is not possible, because it requires a separate stream
> for each receiver.

Yet, you should still use RTP for mulicast. For your use case, just creating an
RTSP server (using gst-rtp-server library) seem sufficient no ?

>
> On Wed, May 19, 2021, 15:02 Mathieu Duponchelle <[hidden email]>
> wrote:
> >  Hey,
> >  
> >  I would recommend using webRTC (webrtcbin in the GStreamer context) for
> >  such an application :)
> >  
> >  Best,
> >  
> > _______________________________________________
> > gstreamer-devel mailing list
> > [hidden email]
> > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

RE: Video Chat using mpegtsmux

GStreamer-devel mailing list

Ok, thank you all for the useful information and suggestions

 

From: Michael Gruner <[hidden email]>
Sent: Wednesday, May 19, 2021 10:06 PM
To: Discussion of the development of and with GStreamer <[hidden email]>
Cc: Nicolas Dufresne <[hidden email]>; Gregory AE40 <[hidden email]>; Mathieu Duponchelle <[hidden email]>
Subject: Re: Video Chat using mpegtsmux

 

I echo the suggestions given by Nicolas and Mathieu. But also wanted to add that, if you chose to continue using TS, there is a “latency” property in the tsdemux element that you can tweak. By default it is set to 700ms (to comply with the TS standard), which sounds like the 1s lag you are experiencing.

 

Michael



On 19 May 2021, at 12:44, Gregory AE40 via gstreamer-devel <[hidden email]> wrote:

 

Yes, I understand that I should use RTP, my question is whether to mux audio and video to a single stream or to send two separate streams (with RTCP for every stream for synchronization). The first option seems more fail-safe to me in terms of audio/video synchronization.  However, I tested also the second option and its latency is lower. I guess that if you suggested to use WebRTC, I should stick with the second option, because AFAIK, WebRTC sends separate streams for video and audio. 

 

On Wed, May 19, 2021, 20:54 Nicolas Dufresne <[hidden email]> wrote:

Le mercredi 19 mai 2021 à 18:23 +0300, Gregory AE40 via gstreamer-devel a
écrit :
> Hi, thank you for the prompt response.
> I wasn't specific enough, the application will run on a LAN, so I can use
> multicast in order to send only one stream from each PC. 
> With WebRTC multicast is not possible, because it requires a separate stream
> for each receiver.

Yet, you should still use RTP for mulicast. For your use case, just creating an
RTSP server (using gst-rtp-server library) seem sufficient no ?


>
> On Wed, May 19, 2021, 15:02 Mathieu Duponchelle <[hidden email]>
> wrote:
> >  Hey,
> >  
> >  I would recommend using webRTC (webrtcbin in the GStreamer context) for
> >  such an application :)
> >  
> >  Best,
> >  
> > _______________________________________________
> > gstreamer-devel mailing list
> > [hidden email]
> > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

 


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel