GStreamer arch and latency

I don't really understand the underlying design, but I have a question about latency:

I notice that the rtp payloader assumes that a complete video frame is available before

it starts payloading. Would it not give better perf and lower latency if the payloader

could begin processing as the MTUs come in over the wire? 

Thanks,

Aaron

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Sa, 2016-05-14 at 12:28 -0400, Aaron Boxer wrote:
> I don't really understand the underlying design, but I have a question about latency:
>
> I notice that the rtp payloader assumes that a complete video frame is available before
> it starts payloading. Would it not give better perf and lower latency if the payloader
> could begin processing as the MTUs come in over the wire?  

The payloader is (usually) processing data that does not come over the
wire, and in many cases it also needs to be able to look at the
complete frame first before actually doing something.

For the depayloader, e.g. rtph264depay should (or could) output
individual NALs instead of complete AUs but as our decoder also work on
frame-units this wouldn't help much. It would however be beneficial to
change that in some situations, it just has to be done.
-- 
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

On Sun, May 15, 2016 at 2:04 AM, Sebastian Dröge <[hidden email]> wrote:

On Sa, 2016-05-14 at 12:28 -0400, Aaron Boxer wrote:
> I don't really understand the underlying design, but I have a question about latency:
>
> I notice that the rtp payloader assumes that a complete video frame is available before
> it starts payloading. Would it not give better perf and lower latency if the payloader
> could begin processing as the MTUs come in over the wire?  

The payloader is (usually) processing data that does not come over the
wire, and in many cases it also needs to be able to look at the
complete frame first before actually doing something.

For the depayloader, e.g. rtph264depay should (or could) output
individual NALs instead of complete AUs but as our decoder also work on
frame-units this wouldn't help much. It would however be beneficial to
change that in some situations, it just has to be done.

Thanks. Since J2K supports tiling, tiles could be processed (encoded, decoded, etc)

before the entire frame is present.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On So, 2016-05-15 at 16:09 -0400, Aaron Boxer wrote:
> 
> Thanks. Since J2K supports tiling, tiles could be processed (encoded,
> decoded, etc) before the entire frame is present.

Yes, the same goes for many video codecs where you could start decoding
individual slices before you have the whole frame.


--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

On Tue, May 17, 2016 at 3:19 AM, Sebastian Dröge <[hidden email]> wrote:

On So, 2016-05-15 at 16:09 -0400, Aaron Boxer wrote:
> 
> Thanks. Since J2K supports tiling, tiles could be processed (encoded,
> decoded, etc) before the entire frame is present.

Yes, the same goes for many video codecs where you could start decoding
individual slices before you have the whole frame.

Do you think this is a feature worth working on ?  I would be interested in taking a look.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Di, 2016-05-17 at 11:21 -0400, Aaron Boxer wrote:

>
>
> On Tue, May 17, 2016 at 3:19 AM, Sebastian Dröge <sebastian@centricul
> ar.com> wrote:
> > On So, 2016-05-15 at 16:09 -0400, Aaron Boxer wrote:
> > > 
> > > Thanks. Since J2K supports tiling, tiles could be processed
> > (encoded,
> > > decoded, etc) before the entire frame is present.
> >
> > Yes, the same goes for many video codecs where you could start
> > decoding
> > individual slices before you have the whole frame.
>
> Do you think this is a feature worth working on ?  I would be
> interested in taking a look.

It doesn't seem like the most pressing issue, but if the way it is
currently done is a performance problem for you, sure :)

See also
  https://bugzilla.gnome.org/show_bug.cgi?id=660770

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel