is there a problem with vp8enc/rtpvp8pay?

[Andres Gonzalez]

I am having a problem with vp8enc and would like to ask a couple of questions.

I have an application that generates a stream using rtpbin with the following pipeline:

appsrc ! queue ! timeoverlay textoverlay x264enc ! rtph264pay

This pipeline is working very well, I can use a typical gst-launch script on another local host and receive the video and audio fine.

But when I then change the x264/rtph264pay  to vp8/rtpvp8pay the corresponding gst-launch script does not process the stream. I am using the default properties for vp8, and rtpvp8pay uses the same basic properties as rtph264pay. I have verified that RTP pkts are being sent, and running the gst-launch script using GST_DEBUG=3,*rtp*:6 shows no obvious errors.

When I compare the working h264 debug output with the non-working vp8 debug output I find that all the vp8 frames are dropped:

...<rtpvp8depay0> Dropping inter-frame before intra-frame

The reason appears to be related to the dts and pts. Here is a debug output from both a working h264 stream and a non-working vp8 stream:

...<rtpvp8depay0> discont 0, seqnum 654, rtptime 282916,     pts 0:00:22.869452680, dts 0:00:22.871813393
...<rtph264depay0> discont 0, seqnum 18570, rtptime 3109586, pts 0:00:31.537955395, dts 0:00:31.536494323

The working h264 has a pts before the dts, which seems reasonable since the frame has a decode time before the presentation time. But the vp8 has the pts BEFORE the dts.  

Question #1: Is that analysis correct, that is, the dts must be younger than the pts?

Question #2: What in the server pipeline is changing the dts and pts?   Since this stream is generated with an appsrc, my application is loading the video data and my code is setting the dts and pts of a GstBuffer before it is pushed into the appsrc:

GST_BUFFER_PTS(pVideoBuffer) = tCaptureTime - m_tBaseClockTime;
GST_BUFFER_DTS(pVideoBuffer) = tCaptureTime - m_tBaseClockTime;

So my code is setting the pts and the dts to the same time, but when the client receives the RTP pkt they are different, and for the vp8 case it appears the change was incorrect. Again, the h264 case is working fine.

Question #3: Am I correct in setting the pts and the dts to the same time when I load and push the GstBuffer?

Question #4: Assuming the pts/dts are the problem, are there any properties of rtpvp8pay that I should set that will fix this?

Thanks,
-Andres

[Nicolas Dufresne-4]

I believe you forgot to mention which version of GStreamer.

Le mercredi 16 mars 2016 à 21:42 -0700, Andres Gonzalez a écrit :

> I am having a problem with vp8enc and would like to ask a couple of
> questions.
>
> I have an application that generates a stream using rtpbin with the
> following pipeline:
>
> appsrc ! queue ! timeoverlay textoverlay x264enc ! rtph264pay
>
> This pipeline is working very well, I can use a typical gst-launch
> script on
> another local host and receive the video and audio fine.
>
> But when I then change the x264/rtph264pay  to vp8/rtpvp8pay the
> corresponding gst-launch script does not process the stream. I am
> using the
> default properties for vp8, and rtpvp8pay uses the same basic
> properties as
> rtph264pay. I have verified that RTP pkts are being sent, and running
> the
> gst-launch script using GST_DEBUG=3,*rtp*:6 shows no obvious errors.

For vp8enc, you probably want to set a "deadline", otherwise libvpx
seems to wait random amount of time before producing a frame. For both
VP8 and H264, you probably want to control the keyframe distance.

>
> When I compare the working h264 debug output with the non-working vp8
> debug
> output I find that all the vp8 frames are dropped:
>
> ...<rtpvp8depay0> Dropping inter-frame before intra-frame
>
> The reason appears to be related to the dts and pts. Here is a debug
> output
> from both a working h264 stream and a non-working vp8 stream:
>
> ...<rtpvp8depay0> discont 0, seqnum 654, rtptime 282916,     pts
> 0:00:22.869452680, dts 0:00:22.871813393
> ...<rtph264depay0> discont 0, seqnum 18570, rtptime 3109586, pts
> 0:00:31.537955395, dts 0:00:31.536494323

At reception of RTP packet, the DTS is simply set to the time of
arrival according to local clock. To void issues, you should use an
rtpjitterbuffer before the depayloaders.

>
> The working h264 has a pts before the dts, which seems reasonable
> since the
> frame has a decode time before the presentation time. But the vp8 has
> the
> pts BEFORE the dts.  
>
> Question #1: Is that analysis correct, that is, the dts must be
> younger than
> the pts?

It is wrong, though most likely that this is a side effect of not using
a jitterbuffer. As VP8 does not have B-Frames, it might make sense to
bugfix the depayloader to ensure the PTS and DTS are always equal
though.

>
> Question #2: What in the server pipeline is changing the dts and
> pts?  

It's not in the server, it's at reception time (udpsrc).

> Since this stream is generated with an appsrc, my application is
> loading the
> video data and my code is setting the dts and pts of a GstBuffer
> before it
> is pushed into the appsrc:
>
> GST_BUFFER_PTS(pVideoBuffer) = tCaptureTime - m_tBaseClockTime;
> GST_BUFFER_DTS(pVideoBuffer) = tCaptureTime - m_tBaseClockTime;
>
> So my code is setting the pts and the dts to the *same* time, but
> when the
> client receives the RTP pkt they are different, and for the vp8 case
> it
> appears the change was incorrect. Again, the h264 case is working
> fine.
>
> Question #3: Am I correct in setting the pts and the dts to the
> *same* time
> when I load and push the GstBuffer?

It's raw data, you should only set the PTS, and leave the DTS to
GST_CLOCK_TIME_NONE. Though, GStreamer should behave correct if you set
both to the same thing.

>
> Question #4: Assuming the pts/dts are the problem, are there any
> properties
> of rtpvp8pay that I should set that will fix this?
>
> Thanks,
> -Andres
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble
> .com/is-there-a-problem-with-vp8enc-rtpvp8pay-tp4676403.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Do, 2016-03-17 at 08:41 -0400, Nicolas Dufresne wrote:
> 
> > Question #1: Is that analysis correct, that is, the dts must be
> > younger than
> > the pts?
> It is wrong, though most likely that this is a side effect of not using
> a jitterbuffer. As VP8 does not have B-Frames, it might make sense to
> bugfix the depayloader to ensure the PTS and DTS are always equal
> though.

This would actually break things. The DTS is always the receive time,
the PTS is whatever time the jitterbuffer calculated as the
presentation time. This is usually not the same as the DTS.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeudi 17 mars 2016 à 14:54 +0200, Sebastian Dröge a écrit :
> This would actually break things. The DTS is always the receive time,
> the PTS is whatever time the jitterbuffer calculated as the
> presentation time. This is usually not the same as the DTS.

Can you really pretend DTS can have two meanings ? After depayloaders
the DTS/PTS pair should respect what a correct stream is. Otherwise, we
can stand that muxing an RTP depayloaded stream is not supported. I'm
not saying this is the case right now, but both the jitterbuffer and
depayloader should one day learn to get this right.

Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Do, 2016-03-17 at 14:26 -0400, Nicolas Dufresne wrote:

> Le jeudi 17 mars 2016 à 14:54 +0200, Sebastian Dröge a écrit :
> >
> > This would actually break things. The DTS is always the receive
> > time,
> > the PTS is whatever time the jitterbuffer calculated as the
> > presentation time. This is usually not the same as the DTS.
> Can you really pretend DTS can have two meanings ? After depayloaders
> the DTS/PTS pair should respect what a correct stream is. Otherwise,
> we
> can stand that muxing an RTP depayloaded stream is not supported. I'm
> not saying this is the case right now, but both the jitterbuffer and
> depayloader should one day learn to get this right.

The depayloader (or parser) could probably. rtpjitterbuffer can't
really.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Andres Gonzalez]

Nicolas,
Sebastian,

Thank you for your responses.

I am using version 1.4.4

Setting the deadline property for vp8enc corrected this problem in that the client gst-launch script now will run properly and I get video and audio as expected. I have also changed my server code to not explicitly set the DTS but instead I am setting it to GST_CLOCK_TIME_NONE  as suggested.

However, there is a very bad memory leak in that vp8enc/rtpvp8pay apparently is not freeing any buffers. Monitoring memory on the application server shows that memory consumption steadily increases to exhaustion and eventually will crash my app when all 16G are used.

In my code, I can pick which encoder to use, so the only difference between the h264 and vp8 cases is the x264enc/rtph264pay  vs. the vp8enc/rtpvp8pay pipeline elements.  When using x264 there is no memory leak and I can run the application all day without problems. But when I use vp8enc/rtpvp8pay, memory is consumed like a starving beast. Latency also steadily increases with vp8 until the client is over a minute behind the other outputs on my server.

Any ideas on why vp8enc/rtpvp8pay is not freeing buffers?
Is there a property of either that deals with freeing buffers?

Thanks,
-Andres

[Sebastian Dröge-3]

On Do, 2016-03-17 at 12:33 -0700, Andres Gonzalez wrote:

> Nicolas,
> Sebastian,
>
> Thank you for your responses.
>
> I am using version 1.4.4
>
> Setting the deadline property for vp8enc corrected this problem in that the
> client gst-launch script now will run properly and I get video and audio as
> expected. I have also changed my server code to not explicitly set the DTS
> but instead I am setting it to GST_CLOCK_TIME_NONE  as suggested.
>
> However, there is a very bad memory leak in that vp8enc/rtpvp8pay apparently
> is not freeing *any* buffers. Monitoring memory on the application server
> shows that memory consumption steadily increases to exhaustion and
> eventually will crash my app when all 16G are used. 
>
> In my code, I can pick which encoder to use, so the only difference between
> the h264 and vp8 cases is the x264enc/rtph264pay  vs. the vp8enc/rtpvp8pay
> pipeline elements.  When using x264 there is no memory leak and I can run
> the application all day without problems. But when I use vp8enc/rtpvp8pay,
> memory is consumed like a starving beast. Latency also steadily increases
> with vp8 until the client is over a minute behind the other outputs on my
> server.
>
> Any ideas on why vp8enc/rtpvp8pay is not freeing buffers?
> Is there a property of either that deals with freeing buffers?

That would be a bug, so there's no point in having a property for that :)

Can you check if it's vp8enc or rtpvp8pay by only plugging vp8enc for
testing? This might also be fixed since 1.4.4 already, so ideally also
check with a newer version.

Is it a real memory leak, or just accumulating memory usage? You can
check with valgrind if it reports any big amount of leaked memory after
then application exited. In case of accumulating memory usage,
valgrind's massif tool could be helpful to locate the place.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel