Reduce latency for a MJPEG over UDP multicast pipe

[STJME]

I am trying to reduce the latency for a live video feed from a camera connected to a video grabber board, compressing using MJPEG and distribute it over UDP to a computer uncompressing the video and imediately render the video to the display. Today, using Dual core Inte CPU's and Gbit network, the total latency for the above scenario is 200 ms (from the light hits the camera until the pixel illuminates).

Q: How can I tweek the latency to get it as short as possible.

Basically, I want to remove all buffers, compress/decompress with such codecs and rates so it is as quick as possible. Less focus on packet loss etc..

[Arnout Vandecappelle]

Here's a bag of tricks:

- Make sure your framegrabber is configured at a high framerate.

- If it's an analog camera that is digitized by your framegrabber, you need to make sure it doesn't do deinterleaving or other multi-frame tricks. I've had a framegrabber that introduces a 80ms latency for that...

- If the framegrabber can do jpeg compression by itself, use that. It's not really faster than doing it on the processor, but you get your compressed data as you go along (otherwise you have a full frame before the encoder and a full frame after the encoder, so a minimum delay of two frames).

- Make sure your gstreamer pipeline isn't doing any copies.

- Make sure you use v4l2src, which avoids a kernel-to-user copy in the source element.

- Configure your kernel to use jumbo packets on the gigabit link. This way you can put a full video frame in each ethernet frame and you avoid fragmentation.

- If all that isn't enough yet, you're in for some serious hacking :-). In the v4l2 driver, instead of passing a buffer to user space when you have a full frame, pass it when you have enough data. For already compressed data, it's the size of your UDP packets. For raw data, that means 8 lines (all that's needed for jpeg compression). You'll also need to hack into jpegenc to make it work that way. Therefore, you're probably better off dropping GStreamer altogether and directly ioctl'ing v4l2 data into jpeg buffers.

- On the receiver side, the story is more or less the same.

Regards,

Arnout

On Thursday 30 September 2010 09:18:00, STJME wrote:

> I am trying to reduce the latency for a live video feed from a camera

> connected to a video grabber board, compressing using MJPEG and distribute

> it over UDP to a computer uncompressing the video and imediately render the

> video to the display. Today, using Dual core Inte CPU's and Gbit network,

> the total latency for the above scenario is 200 ms (from the light hits the

> camera until the pixel illuminates).

>

> Q: How can I tweek the latency to get it as short as possible.

>

> Basically, I want to remove all buffers, compress/decompress with such

> codecs and rates so it is as quick as possible. Less focus on packet loss

> etc..

--

Arnout Vandecappelle arnout at mind be

Senior Embedded Software Architect [phone]

Essensium/Mind http://www.mind.be

G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven

LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle

GPG fingerprint: 31BB CF53 8660 6F88 345D 54CC A836 5879 20D7 CF43

------------------------------------------------------------------------------
Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2 & L3.
Spend less time writing and  rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[STJME]

Thanks alot for this substantial analyzis! We'll dig right into your suggestions and try to tweek accoringly.

Some things we cannot control. We have no control of the camera and framerate of the camera and I guess we cannot elaborte much about the actual frame grabbing. The grabber does not support HW compression for MJPEG, but it appears not to add much latency anyways as you say..

Another thing is that it appears as if the RTP protocol adds substantial delay (70ms or so). By using raw UDP we could decrease the delay. However, that is quite uggly. The best thing would be to try to tweek the RTP stack. We have tryied, but cannot see any effect. Do you know anything about that?

[Arnout Vandecappelle]

On Friday 08 October 2010 15:32:22, STJME wrote:
> Another thing is that it appears as if the RTP protocol adds substantial
> delay (70ms or so). By using raw UDP we could decrease the delay. However,
> that is quite uggly. The best thing would be to try to tweek the RTP stack.
> We have tryied, but cannot see any effect. Do you know anything about that?

 I didn't know about that...  At reception side, RTP has a jitterbuffer which
reorders packets and compensates for clock and network jitter, but I guess you
already configured that down to the minimum.  

 Do you know if this latency is caused by the sender or by the receiver?

 Regards,
 Arnout

 PS If you want a quick reply, CC me, since I don't read the list very often
:-)

--
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 [phone]
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43

------------------------------------------------------------------------------
Download new Adobe(R) Flash(R) Builder(TM) 4
The new Adobe(R) Flex(R) 4 and Flash(R) Builder(TM) 4 (formerly
Flex(R) Builder(TM)) enable the development of rich applications that run
across multiple browsers and platforms. Download your free trials today!
http://p.sf.net/sfu/adobe-dev2dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[STJME]

Yea, we have now implemented a "low latency raw UDP plugin" that we uses and really get down the latency. Around 100ms from the photon hits the CCD till the pixel illuminates on the screen. Using MJPEG and UDP multicast. We will use this transport plugin for now. We did not manage to tweek RTP to be much faster. The latency appeared to come from the receiver side.

Hopefully we'll give this low latency plug back to gstreamer community. Perhaps someone else would like to have it :-) But, it does not follow any standard or anything. Just a simple protocol to fragment the UDP packets (they are sometimes larger than the allowed UDP frame ~63k) and put them back together again on the other side.