Latency regression on v4l2src ! x264enc ! rtp using most recent x264enc

[quackking]

Semi newbie question here, I hope this is the correct place to post.

I am having trouble getting the most recent gstreamer H264 encoder to  
perform as well as the 0.10.12 release. I have been working with  
Ubuntu Lucid, and until a few days ago, I could capture and display  
locally from a V4L2 source (a Hauppauge WinTV HVR 950Q USB stick) with  
a pipeline I include below, and send a stream via RTP to another  
machine on my LAN. Latency as measured by comparing the image shown on  
the sending machine's xvimagesink tee to the receiving machine's  
display of the decoded image was about .25 seconds (that is, a quarter  
of a second - very low.) After I upgraded the latency has gone to  
about 2.5 seconds, ten times higher.

I then built a new machine with identical hardware to the first  
capture machine and installed Ubuntu Karmic AMD64 (these are all 64  
bit desktop installs) and was able to stream to the (same) receiving  
machine using the same pipeline with the expected quarter-second  
latency.

The Karmic install has these versions:

gstreamer0.10-x :: 0.10.25-2ubuntu1.2
gstreamer0.10-tools :: 0.10.25-2
gstreamer0.10-plugins-ugly :: 0.10.12-1
gstreamer0.10-plugins-ugly-multiverse :: 0.10.12-0ubuntu1
libx264-67 :: 1:0.svn20090621+git364d7d-0ubuntu2


This setup, using exactly the same WinTV950Q to capture and the same  
machine to receive
that I have been testing on gives very low latency - basically .25  
sec. If I reduce latency in
the receive script to 80ms (it jitters and gives me frame-drop error  
messages if I go
below that, like to 20ms)

If I go back to the Lucid machine, it has these versions:

gstreamer0.10-x :: 0.10.28-1
gstreamer0.10-tools :: 0.10.28-1
gstreamer0.10-plugins-ugly :: 0.10.14-1
gstreamer0.10-plugins-ugly-multiverse :: 0.10.14libx264-0ubuntu1
libx264-85 :: 2:0.85.1448+git1a6d32-2

Streaming from the Lucid machine I have almost 2.5 seconds of latency  
( about an order of magnitude more ) using exactly the same hardware  
capture device, the same gst-launch pipeline on capture and on  
receive.  However I do not get jitter. Also, the earlier gstreamer has  
a caps parameter that is
missing on the newer gstreamer -

  profile-level-id=(string)64001e

The receiver works with and without this parameter in the caps line.

The sender is running this:

------------------------
DEST=192.168.0.30
VOFFSET=0
AOFFSET=0

# H264 encode from the source
VELEM="v4l2src device=/dev/video0 always-copy=false"
VCAPS="video/x-raw-yuv, width=720,  
height=480,format=(fourcc)UYVY,framerate=(fraction)30000/1001"
VSOURCE="$VELEM ! $VCAPS ! queue ! tee name=t1 ! queue ! xvimagesink  
sync=false t1. ! queue ! videorate ! ffmpegcolorspace"
VENC="x264enc byte-stream=true bitrate=1500 dct8x8=true subme=6  
cabac=true threads=0 ! rtph264pay "

VRTPSINK="udpsink port=5000 host=$DEST ts-offset=$VOFFSET name=vrtpsink"
VRTCPSINK="udpsink port=5001 host=$DEST sync=false async=false name=vrtcpsink"
VRTCPSRC="udpsrc port=5005 name=vrtpsrc"
AELEM="autoaudiosrc"

ASOURCE="$AELEM ! queue ! audioresample ! audioconvert"
AENC="alawenc ! rtppcmapay"

ARTPSINK="udpsink port=5002 host=$DEST ts-offset=$AOFFSET name=artpsink"
ARTCPSINK="udpsink port=5003 host=$DEST sync=false async=false name=artcpsink"
ARTCPSRC="udpsrc port=5007 name=artpsrc"

gst-launch -v gstrtpbin name=rtpbin \
         $VSOURCE !  $VENC ! rtpbin.send_rtp_sink_0                    
                          \
         rtpbin.send_rtp_src_0 ! $VRTPSINK                              
                     \
         rtpbin.send_rtcp_src_0 ! $VRTCPSINK                            
                     \
       $VRTCPSRC ! rtpbin.recv_rtcp_sink_0                              
                     \
     $ASOURCE ! $AENC ! rtpbin.send_rtp_sink_1                          
                     \
         rtpbin.send_rtp_src_1 ! $ARTPSINK                              
                     \
         rtpbin.send_rtcp_src_1 ! $ARTCPSINK                            
                     \
       $ARTCPSRC ! rtpbin.recv_rtcp_sink_1
------------------------




The receiver is running this: (the caps line varies, of course, each  
time I start the sender, I replace the caps line in the receiver with  
the proper caps)

------------------------
DEST=192.168.0.162
LATENCY=100
VIDEO_CAPS="application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=(string)H264,sprop-parameter-sets=\"Z2QAHqwkqAtD2wFQgAAB9IAAdTAHixdQ,aO8yyLA=\",ssrc=(guint)204906444,payload=(int)96,clock-base=(guint)3249570813,seqnum-base=(guint)5734"
AUDIO_CAPS="application/x-rtp,media=(string)audio,clock-rate=(int)8000,encoding-name=(string)PCMA"
OUTSINK="xvimagesink"

gst-launch -v gstrtpbin name=rtpbin latency=$LATENCY                    
                 \
      udpsrc caps=$VIDEO_CAPS   port=5000 ! rtpbin.recv_rtp_sink_0      
                   \
        rtpbin. ! rtph264depay ! ffdec_h264 ! $OUTSINK  \
      udpsrc port=5001 ! rtpbin.recv_rtcp_sink_0                        
                 \
          rtpbin.send_rtcp_src_0 ! udpsink port=5005 host=$DEST  
sync=false async=false \
      udpsrc caps=$AUDIO_CAPS port=5002 ! rtpbin.recv_rtp_sink_1        
                 \
        rtpbin. ! rtppcmadepay ! alawdec ! audioconvert !  
audioresample ! autoaudiosink     \
      udpsrc port=5003 ! rtpbin.recv_rtcp_sink_1                        
                 \
          rtpbin.send_rtcp_src_1 ! udpsink port=5007 host=$DEST  
sync=false async=false

------------------------

There is no audio being sent, or rather, there is nothing connected to  
the audio input.

I want to use the newer releases, of course, but I don't see why th  
latency has increased, using exactly the same hardware and pipelines.

Any help or commentary is greatly appreciated!

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Wim Taymans]

On Fri, 2010-03-26 at 00:24 -0400, Mallard wrote:

Hi,

It's very likely that the new x264 encoder version uses more latency.
You can probably tune that with some properties on x264enc.

Wim

> Semi newbie question here, I hope this is the correct place to post.
>
> I am having trouble getting the most recent gstreamer H264 encoder to  
> perform as well as the 0.10.12 release. I have been working with  
> Ubuntu Lucid, and until a few days ago, I could capture and display  
> locally from a V4L2 source (a Hauppauge WinTV HVR 950Q USB stick) with  
> a pipeline I include below, and send a stream via RTP to another  
> machine on my LAN. Latency as measured by comparing the image shown on  
> the sending machine's xvimagesink tee to the receiving machine's  
> display of the decoded image was about .25 seconds (that is, a quarter  
> of a second - very low.) After I upgraded the latency has gone to  
> about 2.5 seconds, ten times higher.
>
> I then built a new machine with identical hardware to the first  
> capture machine and installed Ubuntu Karmic AMD64 (these are all 64  
> bit desktop installs) and was able to stream to the (same) receiving  
> machine using the same pipeline with the expected quarter-second  
> latency.
>
> The Karmic install has these versions:
>
> gstreamer0.10-x :: 0.10.25-2ubuntu1.2
> gstreamer0.10-tools :: 0.10.25-2
> gstreamer0.10-plugins-ugly :: 0.10.12-1
> gstreamer0.10-plugins-ugly-multiverse :: 0.10.12-0ubuntu1
> libx264-67 :: 1:0.svn20090621+git364d7d-0ubuntu2
>
>
> This setup, using exactly the same WinTV950Q to capture and the same  
> machine to receive
> that I have been testing on gives very low latency - basically .25  
> sec. If I reduce latency in
> the receive script to 80ms (it jitters and gives me frame-drop error  
> messages if I go
> below that, like to 20ms)
>
> If I go back to the Lucid machine, it has these versions:
>
> gstreamer0.10-x :: 0.10.28-1
> gstreamer0.10-tools :: 0.10.28-1
> gstreamer0.10-plugins-ugly :: 0.10.14-1
> gstreamer0.10-plugins-ugly-multiverse :: 0.10.14libx264-0ubuntu1
> libx264-85 :: 2:0.85.1448+git1a6d32-2
>
> Streaming from the Lucid machine I have almost 2.5 seconds of latency  
> ( about an order of magnitude more ) using exactly the same hardware  
> capture device, the same gst-launch pipeline on capture and on  
> receive.  However I do not get jitter. Also, the earlier gstreamer has  
> a caps parameter that is
> missing on the newer gstreamer -
>
>   profile-level-id=(string)64001e
>
> The receiver works with and without this parameter in the caps line.
>
> The sender is running this:
>
> ------------------------
> DEST=192.168.0.30
> VOFFSET=0
> AOFFSET=0
>
> # H264 encode from the source
> VELEM="v4l2src device=/dev/video0 always-copy=false"
> VCAPS="video/x-raw-yuv, width=720,  
> height=480,format=(fourcc)UYVY,framerate=(fraction)30000/1001"
> VSOURCE="$VELEM ! $VCAPS ! queue ! tee name=t1 ! queue ! xvimagesink  
> sync=false t1. ! queue ! videorate ! ffmpegcolorspace"
> VENC="x264enc byte-stream=true bitrate=1500 dct8x8=true subme=6  
> cabac=true threads=0 ! rtph264pay "
>
> VRTPSINK="udpsink port=5000 host=$DEST ts-offset=$VOFFSET name=vrtpsink"
> VRTCPSINK="udpsink port=5001 host=$DEST sync=false async=false name=vrtcpsink"
> VRTCPSRC="udpsrc port=5005 name=vrtpsrc"
> AELEM="autoaudiosrc"
>
> ASOURCE="$AELEM ! queue ! audioresample ! audioconvert"
> AENC="alawenc ! rtppcmapay"
>
> ARTPSINK="udpsink port=5002 host=$DEST ts-offset=$AOFFSET name=artpsink"
> ARTCPSINK="udpsink port=5003 host=$DEST sync=false async=false name=artcpsink"
> ARTCPSRC="udpsrc port=5007 name=artpsrc"
>
> gst-launch -v gstrtpbin name=rtpbin \
>          $VSOURCE !  $VENC ! rtpbin.send_rtp_sink_0                    
>                           \
>          rtpbin.send_rtp_src_0 ! $VRTPSINK                              
>                      \
>          rtpbin.send_rtcp_src_0 ! $VRTCPSINK                            
>                      \
>        $VRTCPSRC ! rtpbin.recv_rtcp_sink_0                              
>                      \
>      $ASOURCE ! $AENC ! rtpbin.send_rtp_sink_1                          
>                      \
>          rtpbin.send_rtp_src_1 ! $ARTPSINK                              
>                      \
>          rtpbin.send_rtcp_src_1 ! $ARTCPSINK                            
>                      \
>        $ARTCPSRC ! rtpbin.recv_rtcp_sink_1
> ------------------------
>
>
>
>
> The receiver is running this: (the caps line varies, of course, each  
> time I start the sender, I replace the caps line in the receiver with  
> the proper caps)
>
> ------------------------
> DEST=192.168.0.162
> LATENCY=100
> VIDEO_CAPS="application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=(string)H264,sprop-parameter-sets=\"Z2QAHqwkqAtD2wFQgAAB9IAAdTAHixdQ,aO8yyLA=\",ssrc=(guint)204906444,payload=(int)96,clock-base=(guint)3249570813,seqnum-base=(guint)5734"
> AUDIO_CAPS="application/x-rtp,media=(string)audio,clock-rate=(int)8000,encoding-name=(string)PCMA"
> OUTSINK="xvimagesink"
>
> gst-launch -v gstrtpbin name=rtpbin latency=$LATENCY                    
>                  \
>       udpsrc caps=$VIDEO_CAPS   port=5000 ! rtpbin.recv_rtp_sink_0      
>                    \
>         rtpbin. ! rtph264depay ! ffdec_h264 ! $OUTSINK  \
>       udpsrc port=5001 ! rtpbin.recv_rtcp_sink_0                        
>                  \
>           rtpbin.send_rtcp_src_0 ! udpsink port=5005 host=$DEST  
> sync=false async=false \
>       udpsrc caps=$AUDIO_CAPS port=5002 ! rtpbin.recv_rtp_sink_1        
>                  \
>         rtpbin. ! rtppcmadepay ! alawdec ! audioconvert !  
> audioresample ! autoaudiosink     \
>       udpsrc port=5003 ! rtpbin.recv_rtcp_sink_1                        
>                  \
>           rtpbin.send_rtcp_src_1 ! udpsink port=5007 host=$DEST  
> sync=false async=false
>
> ------------------------
>
> There is no audio being sent, or rather, there is nothing connected to  
> the audio input.
>
> I want to use the newer releases, of course, but I don't see why th  
> latency has increased, using exactly the same hardware and pipelines.
>
> Any help or commentary is greatly appreciated!
>
> ------------------------------------------------------------------------------
> Download Intel® Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[roope.jarvinen]

 

If you're interested in more details of x264 latency, here's some good reading:

http://x264dev.multimedia.cx/?p=249

>-----Original Message-----
>From: ext Mallard [mailto:[hidden email]]
>Sent: 26 March, 2010 06:24
>To: [hidden email]
>Cc: Mike Berkowitz
>Subject: [gst-devel] Latency regression on v4l2src ! x264enc !
>rtp using most recent x264enc
>
>Semi newbie question here, I hope this is the correct place to post.
>
>I am having trouble getting the most recent gstreamer H264
>encoder to perform as well as the 0.10.12 release. I have been
>working with Ubuntu Lucid, and until a few days ago, I could
>capture and display locally from a V4L2 source (a Hauppauge
>WinTV HVR 950Q USB stick) with a pipeline I include below, and
>send a stream via RTP to another machine on my LAN. Latency as
>measured by comparing the image shown on the sending machine's
>xvimagesink tee to the receiving machine's display of the
>decoded image was about .25 seconds (that is, a quarter of a
>second - very low.) After I upgraded the latency has gone to
>about 2.5 seconds, ten times higher.
>
>I then built a new machine with identical hardware to the
>first capture machine and installed Ubuntu Karmic AMD64 (these
>are all 64 bit desktop installs) and was able to stream to the
>(same) receiving machine using the same pipeline with the
>expected quarter-second latency.
>
>The Karmic install has these versions:
>
>gstreamer0.10-x :: 0.10.25-2ubuntu1.2
>gstreamer0.10-tools :: 0.10.25-2
>gstreamer0.10-plugins-ugly :: 0.10.12-1
>gstreamer0.10-plugins-ugly-multiverse :: 0.10.12-0ubuntu1
>libx264-67 :: 1:0.svn20090621+git364d7d-0ubuntu2
>
>
>This setup, using exactly the same WinTV950Q to capture and
>the same machine to receive that I have been testing on gives
>very low latency - basically .25 sec. If I reduce latency in
>the receive script to 80ms (it jitters and gives me frame-drop
>error messages if I go below that, like to 20ms)
>
>If I go back to the Lucid machine, it has these versions:
>
>gstreamer0.10-x :: 0.10.28-1
>gstreamer0.10-tools :: 0.10.28-1
>gstreamer0.10-plugins-ugly :: 0.10.14-1
>gstreamer0.10-plugins-ugly-multiverse :: 0.10.14libx264-0ubuntu1
>libx264-85 :: 2:0.85.1448+git1a6d32-2
>
>Streaming from the Lucid machine I have almost 2.5 seconds of
>latency ( about an order of magnitude more ) using exactly the
>same hardware capture device, the same gst-launch pipeline on
>capture and on receive.  However I do not get jitter. Also,
>the earlier gstreamer has a caps parameter that is missing on
>the newer gstreamer -
>
>  profile-level-id=(string)64001e
>
>The receiver works with and without this parameter in the caps line.
>
>The sender is running this:
>
>------------------------
>DEST=192.168.0.30
>VOFFSET=0
>AOFFSET=0
>
># H264 encode from the source
>VELEM="v4l2src device=/dev/video0 always-copy=false"
>VCAPS="video/x-raw-yuv, width=720,
>height=480,format=(fourcc)UYVY,framerate=(fraction)30000/1001"
>VSOURCE="$VELEM ! $VCAPS ! queue ! tee name=t1 ! queue !
>xvimagesink sync=false t1. ! queue ! videorate ! ffmpegcolorspace"
>VENC="x264enc byte-stream=true bitrate=1500 dct8x8=true
>subme=6 cabac=true threads=0 ! rtph264pay "
>
>VRTPSINK="udpsink port=5000 host=$DEST ts-offset=$VOFFSET
>name=vrtpsink"
>VRTCPSINK="udpsink port=5001 host=$DEST sync=false async=false
>name=vrtcpsink"
>VRTCPSRC="udpsrc port=5005 name=vrtpsrc"
>AELEM="autoaudiosrc"
>
>ASOURCE="$AELEM ! queue ! audioresample ! audioconvert"
>AENC="alawenc ! rtppcmapay"
>
>ARTPSINK="udpsink port=5002 host=$DEST ts-offset=$AOFFSET
>name=artpsink"
>ARTCPSINK="udpsink port=5003 host=$DEST sync=false async=false
>name=artcpsink"
>ARTCPSRC="udpsrc port=5007 name=artpsrc"
>
>gst-launch -v gstrtpbin name=rtpbin \
>         $VSOURCE !  $VENC ! rtpbin.send_rtp_sink_0            
>        
>                          \
>         rtpbin.send_rtp_src_0 ! $VRTPSINK                    
>        
>                     \
>         rtpbin.send_rtcp_src_0 ! $VRTCPSINK                  
>        
>                     \
>       $VRTCPSRC ! rtpbin.recv_rtcp_sink_0                    
>        
>                     \
>     $ASOURCE ! $AENC ! rtpbin.send_rtp_sink_1                
>        
>                     \
>         rtpbin.send_rtp_src_1 ! $ARTPSINK                    
>        
>                     \
>         rtpbin.send_rtcp_src_1 ! $ARTCPSINK                  
>        
>                     \
>       $ARTCPSRC ! rtpbin.recv_rtcp_sink_1
>------------------------
>
>
>
>
>The receiver is running this: (the caps line varies, of
>course, each time I start the sender, I replace the caps line
>in the receiver with the proper caps)
>
>------------------------
>DEST=192.168.0.162
>LATENCY=100
>VIDEO_CAPS="application/x-rtp,media=(string)video,clock-rate=(i
>nt)90000,encoding-name=(string)H264,sprop-parameter-sets=\"Z2QA
>HqwkqAtD2wFQgAAB9IAAdTAHixdQ,aO8yyLA=\",ssrc=(guint)204906444,p
>ayload=(int)96,clock-base=(guint)3249570813,seqnum-base=(guint)5734"
>AUDIO_CAPS="application/x-rtp,media=(string)audio,clock-rate=(i
>nt)8000,encoding-name=(string)PCMA"
>OUTSINK="xvimagesink"
>
>gst-launch -v gstrtpbin name=rtpbin latency=$LATENCY          
>        
>                 \
>      udpsrc caps=$VIDEO_CAPS   port=5000 !
>rtpbin.recv_rtp_sink_0      
>                   \
>        rtpbin. ! rtph264depay ! ffdec_h264 ! $OUTSINK  \
>      udpsrc port=5001 ! rtpbin.recv_rtcp_sink_0              
>        
>                 \
>          rtpbin.send_rtcp_src_0 ! udpsink port=5005
>host=$DEST sync=false async=false \
>      udpsrc caps=$AUDIO_CAPS port=5002 !
>rtpbin.recv_rtp_sink_1        
>                 \
>        rtpbin. ! rtppcmadepay ! alawdec ! audioconvert !  
>audioresample ! autoaudiosink     \
>      udpsrc port=5003 ! rtpbin.recv_rtcp_sink_1              
>        
>                 \
>          rtpbin.send_rtcp_src_1 ! udpsink port=5007
>host=$DEST sync=false async=false
>
>------------------------
>
>There is no audio being sent, or rather, there is nothing
>connected to the audio input.
>
>I want to use the newer releases, of course, but I don't see
>why th latency has increased, using exactly the same hardware
>and pipelines.
>
>Any help or commentary is greatly appreciated!
>
>---------------------------------------------------------------
>---------------
>Download Intel® Parallel Studio Eval Try the new software
>tools for yourself. Speed compiling, find bugs proactively,
>and fine-tune applications for parallel performance.
>See why Intel Parallel Studio got high marks during beta.
>http://p.sf.net/sfu/intel-sw-dev
>_______________________________________________
>gstreamer-devel mailing list
>[hidden email]
>https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[quackking]

This is pretty much what I expected to find (a change in default  
behavior) - but it raises another question. Are all the -85 args  
exposed in x264enc-0.10-14 and if not , which ones are? Is there any  
way to determine this without reading the source code?

Reading threads on the videolan.org H264 forums, it seems the most  
significant latency-related mods in recent libx264 development may be  
related to enabling single-slice-based encoding. Is this particular  
set of parameters exposed to me using the present 'ugly' and  
'ugly-multiverse' x264enc element?

Thanks for your comments!




Quoting Wim Taymans <[hidden email]>:

> On Fri, 2010-03-26 at 00:24 -0400, Mallard wrote:
>
> Hi,
>
> It's very likely that the new x264 encoder version uses more latency.
> You can probably tune that with some properties on x264enc.
>
> Wim
>


------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel