FEC and retransmissions for video

[Lauri Ehrenpreis]

Hi!

I tried to set up a pipeline which does both retransmissions and FEC for video. My hope was that first pipeline will try to recover packet with FEC and if that fails it will request retransmission. As there was no working example for such setup I used gst-plugins-good/tests/examples/rtp/client-rtpaux.c and gst-plugins-good/tests/examples/rtp/server-rtpaux.c as basis and added request-fec-decoder & request-fec-encoder signal handlers there. 

After adding FEC handlers gstrtpbin came up with following pipeline on client side (attached image). So retransmission component is before jitterbuffer and FEC is after. Do I understand correctly the pipeline will first try to request retransmission if packet is lost and only after the retransmission did not deliver the packet in time for jitterbuffer, it will try use FEC to recover packet? Why it is done this way - wouldn't it be better to try FEC first?

--

Lauri

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Mathieu Duponchelle]

Hi Lauri,



On 05/14/2018 02:53 PM, Lauri Ehrenpreis wrote:
> Hi!
>
> I tried to set up a pipeline which does both retransmissions and FEC for video. My hope was that first pipeline will try to recover packet with FEC and if that fails it will request retransmission. As there was no working example for such setup I used gst-plugins-good/tests/examples/rtp/client-rtpaux.c and gst-plugins-good/tests/examples/rtp/server-rtpaux.c as basis and added request-fec-decoder & request-fec-encoder signal handlers there. 
>
> After adding FEC handlers gstrtpbin came up with following pipeline on client side (attached image). So retransmission component is before jitterbuffer and FEC is after. Do I understand correctly the pipeline will first try to request retransmission if packet is lost and only after the retransmission did not deliver the packet in time for jitterbuffer, it will try use FEC to recover packet? Why it is done this way - wouldn't it be better to try FEC first?

Yes, you are correct, FEC recovery will only be attempted once a packet is actually
considered lost.

While we could indeed reconstruct packets as soon as they are late, there is no
guarantee that the reconstructed packet would be "complete", at least with ULPFEC.

As you might already know, ULP stands for "uneven level protection", and works based
on the assumption that with most video codecs / payloaders, the most important part
of a packet is at the start, which means that a percentage of protection can be applied
at the packet level by only protecting the initial portion, possibly making exceptions
for keyframe packets for example, which could be fully protected.

This means that while we could reconstruct a packet, if retransmission has been
enabled it makes sense to ask for that packet to be retransmitted anyway.

I hope that made sense :)

--
Mathieu Duponchelle · https://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Lauri Ehrenpreis]

ULP does not always protect only part of the packets. It can also protect full packets. In fact the current implementation of gstrtpulpfecenc.c is only protecting full packets. Also the fec decored knows if it recovered full packet or only part of the pecket. 

Given this I see no reason why FEC decoder should be placed downstream from jitterbuffer. Being downstream and only working when jitterbuffer reported loss means that we are not getting any latency benefit from FEC as it only recovers packet _after_ jitterbuffer timed out the packet and reported loss. Jitterbuffer latency can be very large - 2 sec for example. This means that even if we paid for the bandwidth and sent FEC, we still get 2 sec delay. Also additional bandwidth will be wasted because RTX will anyway retransmit the packet which was already recovered by FEC. So double loss because of current implementation.

To make RTX and FEC work together I placed FEC decoder upstream from jitterbuffer (using request-aux-receiver which returns a bin containing rtprtxreceive, rtpstorage and rtpulpfecdec). Also I modified FEC decoder so that it looks for the sequence number gaps in storage on every incoming packet and tries to recover those immediately. This way FEC will reduce the latency if it recovers something and also it can work together with rtx. There are minor optimizations to be done since for example currently jitterbuffer also requests retransmit for FEC packets. These requests should be ignored on sender side. 

I think in case it's not known if FEC will be able to recover full or only partial packets, there should be a FEC decoder before jitterbuffer which only recovers full packets and another one downstream which recovers partial packets when jitterbuffer considers those lost.

--

LauriE

On Wed, May 16, 2018 at 3:43 PM, Mathieu Duponchelle <[hidden email]> wrote:

Hi Lauri,



On 05/14/2018 02:53 PM, Lauri Ehrenpreis wrote:
> Hi!
>
> I tried to set up a pipeline which does both retransmissions and FEC for video. My hope was that first pipeline will try to recover packet with FEC and if that fails it will request retransmission. As there was no working example for such setup I used gst-plugins-good/tests/examples/rtp/client-rtpaux.c and gst-plugins-good/tests/examples/rtp/server-rtpaux.c as basis and added request-fec-decoder & request-fec-encoder signal handlers there. 
>
> After adding FEC handlers gstrtpbin came up with following pipeline on client side (attached image). So retransmission component is before jitterbuffer and FEC is after. Do I understand correctly the pipeline will first try to request retransmission if packet is lost and only after the retransmission did not deliver the packet in time for jitterbuffer, it will try use FEC to recover packet? Why it is done this way - wouldn't it be better to try FEC first?

Yes, you are correct, FEC recovery will only be attempted once a packet is actually
considered lost.

While we could indeed reconstruct packets as soon as they are late, there is no
guarantee that the reconstructed packet would be "complete", at least with ULPFEC.

As you might already know, ULP stands for "uneven level protection", and works based
on the assumption that with most video codecs / payloaders, the most important part
of a packet is at the start, which means that a percentage of protection can be applied
at the packet level by only protecting the initial portion, possibly making exceptions
for keyframe packets for example, which could be fully protected.

This means that while we could reconstruct a packet, if retransmission has been
enabled it makes sense to ask for that packet to be retransmitted anyway.

I hope that made sense :)

--
Mathieu Duponchelle · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Mathieu Duponchelle]

Hey,

On 05/18/2018 07:59 AM, Lauri Ehrenpreis wrote:

ULP does not always protect only part of the packets. It can also protect full packets. In fact the current implementation of gstrtpulpfecenc.c is only protecting full packets.

Yes, I am aware of that, but the decoder isn't necessarily :)

Also the fec decored knows if it recovered full packet or only part of the pecket.

That's interesting, for context I was the one that upstreamed the ulpfec elements,
but I didn't do the implementation, how can the fec decoder determine that?

Given this I see no reason why FEC decoder should be placed downstream from jitterbuffer. Being downstream and only working when jitterbuffer reported loss means that we are not getting any latency benefit from FEC as it only recovers packet _after_ jitterbuffer timed out the packet and reported loss. Jitterbuffer latency can be very large - 2 sec for example. This means that even if we paid for the bandwidth and sent FEC, we still get 2 sec delay. Also additional bandwidth will be wasted because RTX will anyway retransmit the packet which was already recovered by FEC. So double loss because of current implementation.

I assume that this concern only exists when using faststart-min-packets, correct?

To make RTX and FEC work together I placed FEC decoder upstream from jitterbuffer (using request-aux-receiver which returns a bin containing rtprtxreceive, rtpstorage and rtpulpfecdec). Also I modified FEC decoder so that it looks for the sequence number gaps in storage on every incoming packet and tries to recover those immediately. This way FEC will reduce the latency if it recovers something and also it can work together with rtx. There are minor optimizations to be done since for example currently jitterbuffer also requests retransmit for FEC packets. These requests should be ignored on sender side. 

I think in case it's not known if FEC will be able to recover full or only partial packets, there should be a FEC decoder before jitterbuffer which only recovers full packets and another one downstream which recovers partial packets when jitterbuffer considers those lost.

That could be a nice approach, as far as I understand all the machinery is already
in place in rtpbin to support this, the only thing that would be needed is a special
mode in ulpfecdec, detecting seqnum gaps and trying to do preemptive recovery
of packets it knows it can fully recover (this hinges on my earlier question).

Alternatively, if we can indeed determine whether the recovered packets are fully
recoverable, we can probably dispense with the downstream decoder altogether,
by tagging the recovered buffers in a way that the jitter buffer can interpret to
decide whether it should send retransmission requests nevertheless, replacing
the recovered packet with the original if it arrives late or is retransmitted.

--

LauriE

On Wed, May 16, 2018 at 3:43 PM, Mathieu Duponchelle <[hidden email]> wrote:

Hi Lauri,



On 05/14/2018 02:53 PM, Lauri Ehrenpreis wrote:
> Hi!
>
> I tried to set up a pipeline which does both retransmissions and FEC for video. My hope was that first pipeline will try to recover packet with FEC and if that fails it will request retransmission. As there was no working example for such setup I used gst-plugins-good/tests/examples/rtp/client-rtpaux.c and gst-plugins-good/tests/examples/rtp/server-rtpaux.c as basis and added request-fec-decoder & request-fec-encoder signal handlers there. 
>
> After adding FEC handlers gstrtpbin came up with following pipeline on client side (attached image). So retransmission component is before jitterbuffer and FEC is after. Do I understand correctly the pipeline will first try to request retransmission if packet is lost and only after the retransmission did not deliver the packet in time for jitterbuffer, it will try use FEC to recover packet? Why it is done this way - wouldn't it be better to try FEC first?

Yes, you are correct, FEC recovery will only be attempted once a packet is actually
considered lost.

While we could indeed reconstruct packets as soon as they are late, there is no
guarantee that the reconstructed packet would be "complete", at least with ULPFEC.

As you might already know, ULP stands for "uneven level protection", and works based
on the assumption that with most video codecs / payloaders, the most important part
of a packet is at the start, which means that a percentage of protection can be applied
at the packet level by only protecting the initial portion, possibly making exceptions
for keyframe packets for example, which could be fully protected.

This means that while we could reconstruct a packet, if retransmission has been
enabled it makes sense to ask for that packet to be retransmitted anyway.

I hope that made sense :)

--
Mathieu Duponchelle · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Lauri Ehrenpreis]

On Fri, May 18, 2018 at 4:57 PM, Mathieu Duponchelle <[hidden email]> wrote:

That's interesting, for context I was the one that upstreamed the ulpfec elements,
but I didn't do the implementation, how can the fec decoder determine that?

I hope I am not mistaken since it's now only 2nd time I look at rfc5109:

8.1.  Generation of the FEC Header

...

      o Unsigned network-ordered 16-bit representation of the media
        packet length in bytes minus 12 (for the fixed RTP header),
        i.e., the sum of the lengths of all the following if present:
        the CSRC list, extension header, RTP payload, and RTP padding
        (16 bits)

So the length of original RTP packet is actually encoded into FEC packet header. Now when we run recovery on 

all the available FEC levels and the resulting packet has same length as stated in FEC header, we know the full packet got recovered. 

 

I assume that this concern only exists when using faststart-min-packets, correct?

 

My concern is following: I want to decode all frames as soon as possible since every millisecond of latency counts. Jitter doesn't 

matter to me. I set jitterbuffer latency parameter to fairly large value since in bad network case I don't want jitterbuffer to time 

out easily when waiting for missing packet (and I also have RTX). Now if the FEC recovery is only done after jitterbuffer times out then 

I get the whole jitterbuffer latency worth of additional delay added in cases where I actually had all the needed data available 

inside FEC packet. However if I run FEC and recover packet before jitterbuffer then jitterbuffer will not delay the frame at all.

--

LauriE

 

--

LauriE

On Wed, May 16, 2018 at 3:43 PM, Mathieu Duponchelle <[hidden email]> wrote:

Hi Lauri,



On 05/14/2018 02:53 PM, Lauri Ehrenpreis wrote:
> Hi!
>
> I tried to set up a pipeline which does both retransmissions and FEC for video. My hope was that first pipeline will try to recover packet with FEC and if that fails it will request retransmission. As there was no working example for such setup I used gst-plugins-good/tests/examples/rtp/client-rtpaux.c and gst-plugins-good/tests/examples/rtp/server-rtpaux.c as basis and added request-fec-decoder & request-fec-encoder signal handlers there. 
>
> After adding FEC handlers gstrtpbin came up with following pipeline on client side (attached image). So retransmission component is before jitterbuffer and FEC is after. Do I understand correctly the pipeline will first try to request retransmission if packet is lost and only after the retransmission did not deliver the packet in time for jitterbuffer, it will try use FEC to recover packet? Why it is done this way - wouldn't it be better to try FEC first?

Yes, you are correct, FEC recovery will only be attempted once a packet is actually
considered lost.

While we could indeed reconstruct packets as soon as they are late, there is no
guarantee that the reconstructed packet would be "complete", at least with ULPFEC.

As you might already know, ULP stands for "uneven level protection", and works based
on the assumption that with most video codecs / payloaders, the most important part
of a packet is at the start, which means that a percentage of protection can be applied
at the packet level by only protecting the initial portion, possibly making exceptions
for keyframe packets for example, which could be fully protected.

This means that while we could reconstruct a packet, if retransmission has been
enabled it makes sense to ask for that packet to be retransmitted anyway.

I hope that made sense :)

--
Mathieu Duponchelle · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Dark_Knight-2]

CONTENTS DELETED

The author has deleted this message.