rtspsrc jitterbuffer stats

[Victor Weinstein]

I would like to be able to tune my pipeline which uses an rtspsrc element to smooth out frame delivery and to add latency to prevent duplicate or missed frames.  The rtspsrc element implements buffering with configurable latency, buffer-mode, and drop-on-latency parameters.  However, in order to read back buffering, it appears that I need to disable buffering in the rtspsrc by setting latency to 0 and adding an rtpjitterbuffer element immediately after the rtspsrc, i.e.

Change
rtspsrc location=rtsp://... latency=50 ! ...
to
rtspsrc location=rtsp://... latency=0 ! rtpjitterbuffer latency=50 ! ...

I am able to read back the percent property of the rtpjitterbuffer in this way, as well as the stats property of the rtpjitterbuffer.

The stats are quite limited however, there is a recent commit that provides better statistics, see rtpjitterbuffer: Add and expose more stats and increase testing of it

The commit message states:
Add num-pushed and num-lost. Expose num-late, num-duplicates and avg-jitter.

My questions are these:

1) Does setting the latency to 0 in the rtspsrc element effectively disable buffering or is there some additional overhead associated with having two jitter buffers in the pipeline if I add the rtpjitterbuffer element immediately downstream of the rtspsrc?

2) What are the meanings of the statistics available?
  a) does num-lost = the number of packets dropped because the buffer was full or the latency was too high?
  b) does num-duplicates mean that the last frame sent was duplicated when the buffer ran dry?

The behavior I'm looking for is for the buffer to never exceed a certain depth and for the last frame to be resent if the buffer runs empty or hits its minimum depth.  I would like to know how many frames were dropped due to the buffer reaching its size limit and the number of frames that were resent due to the buffer running empty.  It seems like in order to do this, I need to use the rtpjitterbuffer element and apply the commit to my gstreamer source tree (assuming that those updated stats provide the dropped and resent information I'm looking for).  It would also be highly desirable to resend the same frame when the buffer runs empty or hits its minimum depth.

Thanks for your help, feel free to let me know if I'm barking up the wrong tree,
Victor

[Sebastian Dröge-3]

On Mon, 2016-11-21 at 10:09 -0800, Victor Weinstein wrote:
> I would like to be able to tune my pipeline which uses an rtspsrc element to
> smooth out frame delivery and to add latency to prevent duplicate or missed
> frames.  The rtspsrc element implements buffering with configurable latency,
> buffer-mode, and drop-on-latency parameters.  However, in order to read back
> buffering, it appears that I need to disable buffering in the rtspsrc by
> setting latency to 0 and adding an rtpjitterbuffer element immediately after
> the rtspsrc, i.e.

That's not a good idea as rtpjitterbuffer is used inside the rtpbin
inside rtspsrc, and not just for buffering.

>
> 1) Does setting the latency to 0 in the rtspsrc element effectively disable
> buffering or is there some additional overhead associated with having two
> jitter buffers in the pipeline if I add the rtpjitterbuffer element
> immediately downstream of the rtspsrc?

No, it still does its job. Just without waiting. What you really want
is to get access to the rtpbin and rtpjitterbuffer that are inside
rtspsrc and then implement your stuff on top of that.

> 2) What are the meanings of the statistics available?
>   a) does num-lost = the number of packets dropped because the buffer was
> full or the latency was too high?

The number of packets that were considered lost because they didn't
arrive (at all or in time).

>   b) does num-duplicates mean that the last frame sent was duplicated when
> the buffer ran dry?

The number of packets that were received multiple times from the
network.

> The behavior I'm looking for is for the buffer to never exceed a certain
> depth and for the last frame to be resent if the buffer runs empty or hits
> its minimum depth.  I would like to know how many frames were dropped due to
> the buffer reaching its size limit and the number of frames that were resent
> due to the buffer running empty.  It seems like in order to do this, I need
> to use the rtpjitterbuffer element and apply the commit to my gstreamer
> source tree (assuming that those updated stats provide the dropped and
> resent information I'm looking for).  It would also be highly desirable to
> resend the same frame when the buffer runs empty or hits its minimum depth.

Currently the rtpjitterbuffer has infinite size in packets, but will
not have more than latency amount of time stored (unless downstream
fails to consume packets in real-time).

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Victor Weinstein]

>>
>> 1) Does setting the latency to 0 in the rtspsrc element effectively disable
>>  buffering or is there some additional overhead associated with having two
>> jitter buffers in the pipeline if I add the rtpjitterbuffer element
>> immediately downstream of the rtspsrc?

> No, it still does its job. Just without waiting. What you really want
> is to get access to the rtpbin and rtpjitterbuffer that are inside
> rtspsrc and then implement your stuff on top of that.

Hi Sebastian,

I was able to do just this by connecting a "new-manager" callback after creating the rtspsrc element

video_camera::StartStream( )
{
    m_pipeline = gst_parse_launch( "rtspsrc name=videosrc...", &parsingError );  
    m_element_videosrc = gst_bin_get_by_name( GST_BIN( m_pipeline ), "videosrc" );  
    g_signal_connect( m_element_videosrc, "new-manager", G_CALLBACK(callback_rtspsrc_new_manager), this );
}

In the callback, I setup a subsequent callback for "new-jitterbuffer" signal of the rtpbin element, i.e.

  static void callback_rtspsrc_new_manager( GstRTSPSrc *rtspsrc,
    GstElement* manager, video_camera* video_camera_p )
  {
    g_signal_connect( manager, "new-jitterbuffer", G_CALLBACK(callback_rtspsrc_new_jitterbuffer),
      video_camera_p );
    return;
  }

  static void callback_rtspsrc_new_jitterbuffer( GstElement *rtpbin, GstElement *jitterbuffer,
    guint session, guint ssrc, video_camera* video_camera_p)
  {
    video_camera_p->SetJitterBuffer(jitterbuffer);
    return;
  }

This allows me to access the stats associated with the rtpjitterbuffer.  Is this how you envisioned me accessing the rtpbin and rtpjitterbuffer?  Regardless, I appreciate the suggestion to get at the rtpbin and rtpjitterbuffer associated with the rtpsrc.

>> 2) What are the meanings of the statistics available?
>>   a) does num-lost = the number of packets dropped because the buffer was
>> full or the latency was too high?

> The number of packets that were considered lost because they didn't
> arrive (at all or in time).

Ok, so that implies that num-lost = num-late + num that didn't arrive at all

>>   b) does num-duplicates mean that the last frame sent was duplicated when
>> the buffer ran dry?

> The number of packets that were received multiple times from the
> network.

>> The behavior I'm looking for is for the buffer to never exceed a certain
>> depth and for the last frame to be resent if the buffer runs empty or hits
>> its minimum depth.  I would like to know how many frames were dropped due to
>> the buffer reaching its size limit and the number of frames that were resent
>> due to the buffer running empty.  It seems like in order to do this, I need

The rtpjitterbuffer doesn't "resend" or duplicate packets, it merely accumulates and drops rtp packets as they arrive, so my desire to have frames resent when the buffer runs dry isn't supported.  It also doesn't know anything about frames or framerates.  I see where I was mistaken in my assumptions.   Perhaps there's a method downstream to resend frames using the videorate element, or similar.  I'll have to look into it.  

> Currently the rtpjitterbuffer has infinite size in packets, but will
> not have more than latency amount of time stored (unless downstream
> fails to consume packets in real-time).

So, if I understand correctly, a bursty camera that runs slow and then bursts frames will starve the buffer and then if the buffer is not sufficiently deep enough to store the burst, any rtp packets that exceed the latency specified for the buffer relative to the rtp packet at the head of the buffer will be dropped.

What is the meaning of "completely filled", then, with regard to the drop-on-latency property:
    The “drop-on-latency” property

    “drop-on-latency”          gboolean

  Drop oldest buffers when the queue is completely filled.

Similarly, the latency property seems to suggest that rtp packets expire and are discarded if left in the buffer beyond the latency.  My experience shows that the latency parameter is a fixed depth that must be accumulated before rtp packets can be released downstream.

    “latency”                  guint

  The maximum latency of the jitterbuffer. Packets will be kept in the buffer for at most this time.

  Flags: Read / Write

  Default value: 200

Can you clarify?  Perhaps I have the notion of what is considered "too late" backwards, i.e. packets that have been in the buffer longer than latency relative the most recent RTP packet in the buffer are discarded as being too late.

Thanks so much for your help.  I wanted to make sure I acknowledged your assistance and post the method I used for obtaining the rtpjitterbuffer element since it took some effort for me to figure it out and I wanted others to benefit.

Victor