v4l2rc MJPEG to Videomixer slow - alternative pathways?
Locked
 
|
Hi all,
I'm currently working with some UVC cameras that support outputting both raw frames (YUV) and mjpeg. Due to USB 2.0 bandwidth restrictions I'm hoping to switch over to MJPEG so I can support higher resolution video. My current pipeline takes X camera raw feeds, composites them via videomixer, and pushes them into an appsink which I read from my Python application. When I try to switch over from raw feeds to MJPEG, the stream is extremely laggy if I use more than 3 cameras; I imagine this has to do with the way MJPEG images are parsed, as well as with the timing involved with videomixer (I convert the MJPEG feeds back to raw video because that's all videomixer can support). My question is, is there a way to more smoothly stream mjpeg frames into videomixer? I've tried placing queues in different locations; sync is set to false (on appsink) and is-live and do-timestamp are set to true (on v4l2src). Otherwise, would it be possible for me to design my pipeline in a different way to support MJPEG (i.e. for X cameras, stream frames into X different appsinks and composite them manually in Python/C - though I worry reading from X appsinks in python will be slow)? Perhaps there's a way to use tees I'm not thinking of? Or I could stream the mjpeg feeds via UDP locally and simply stream the UDP locally as well? Any suggestions or thoughts on how this could work? Thanks in advance, Ashwath -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: v4l2rc MJPEG to Videomixer slow - alternative pathways?
|
|
hI Aswath,
Can you please share your pipeline, so we all members can have a better look at it and possibly find a solution for the same. -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
Sure, here is an example:
gat-launch-1.0 videomixer name=mix sink_0::xpos=640 sink_0::ypos=240 sink_1::xpos=0 sink_1::ypos=0 sink_2::xpos=0 sink_2::ypos=480 sink_3::xpos=0 sink_3::ypos=960 ! videorate ! video/x-raw,framerate=30/1 ! appsink name=app enable-last-sample=true emit-signals=true sync=false drop=true qos=true blocksize=2764800 v4l2src device=/dev/video0 do-timestamp=true ! image/jpeg,height=480,width=640,framerate=30/1 ! jpegparse ! jpegdec ! videoconvert ! video/x-raw,format=BGR ! mix.sink_0 v4l2src device=/dev/video1 do-timestamp=true ! image/jpeg,height=480,width=640,framerate=30/1 ! jpegparse ! jpegdec ! videoconvert ! video/x-raw,format=BGR ! mix.sink_1 v4l2src device=/dev/video2 do-timestamp=true ! image/jpeg,height=480,width=640,framerate=30/1 ! jpegparse ! jpegdec ! videoconvert ! video/x-raw,format=BGR ! mix.sink_2 v4l2src device=/dev/video3 do-timestamp=true ! image/jpeg,height=480,width=640,framerate=30/1 ! jpegparse ! jpegdec ! videoconvert ! video/x-raw,format=BGR ! mix.sink_3 I was reading that MJPEG has no timestamps -- is it possible that I can set the timestamp settings differently? Or perhaps my blocksize is wrong? Thanks -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
Bump - just seeing if anyone has any thoughts on how I can concatenate
multiple mjpeg feeds. This is better since mjpeg payloads should be smaller than raw frames, enabling higher throughput. Yet, for some reason, the pipeline i have above is super laggy. Any help would be so so appreciated. Ashwath -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
Hi - just wanted to add that I'm trying to run this with 4 cameras using the
above command, and it's still pretty slow. When I use export GST_DEBUG=2 I see lots of warnings that are: "v4l2src gstv4l2src.c:862:gst_v4l2src_create:<v4l2src#> lost frames detected: count = [2-15] - ts So it seems like v4l2src is somehow dropping frames. Not sure if this is because videomixer is forcing a speed it can't keep up with, or what. One other caveat -- the cameras I'm using only support 30 FPS. Still, even if I halve the resolution (to 320x240), it does nothing to affect the pipeline. Finally, my cpu usage is staying pretty low, at max 1 core (of the 8 that I have on my i7 machine) goes to 35% I'm sure there's something simple that I'm missing -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
I'm sorry, just 1 more update. Adding a leaky queue after the capture helped
get rid of the lost frames problem, but the feed is still quite laggy (not to mention I'm also losing lots of frames). I see a lot of gstreamer warnings: 0:00:06.013149788 15050 0x9250a0 WARN videodecoder gstvideodecoder.c:4003:_gst_video_decoder_error:<jpegdec2> error: Failed to decode JPEG image 0:00:06.013162819 15050 0x9250a0 WARN videodecoder gstvideodecoder.c:4005:_gst_video_decoder_error:<jpegdec2> error: Decode error #62: Unsupported JPEG process: SOF type 0xcf new pipeline: gst-launch-1.0 videomixer name=mix sink_0::xpos=0 sink_0::ypos=0 sink_1::xpos=640 sink_1::ypos=0 sink_2::xpos=0 sink_2::ypos=480 sink_3::xpos=640 sink_3::ypos=480 ! autovideosink sync=false v4l2src device=/dev/video0 do-timestamp=false ! image/jpeg,height=480,width=640,framerate=30/1 ! queue leaky=2 max-size-buffers=1 ! jpegparse ! jpegdec ! videoconvert ! mix.sink_0 v4l2src device=/dev/video1 do-timestamp=false ! image/jpeg,height=480,width=640,framerate=30/1 ! queue leaky=2 max-size-buffers=1 ! jpegparse ! jpegdec ! videoconvert ! mix.sink_1 v4l2src device=/dev/video2 do-timestamp=false ! image/jpeg,height=480,width=640,framerate=30/1 ! queue leaky=2 max-size-buffers=1 ! jpegparse ! jpegdec ! videoconvert ! mix.sink_2 v4l2src device=/dev/video3 do-timestamp=false ! image/jpeg,height=480,width=640,framerate=30/1 ! queue leaky=2 max-size-buffers=1 ! jpegparse ! jpegdec ! videoconvert ! mix.sink_3 -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: v4l2rc MJPEG to Videomixer slow - alternative pathways?
|
|
Le jeudi 28 septembre 2017 à 16:45 -0700, ashwath a écrit :
> I'm sorry, just 1 more update. Adding a leaky queue after the > capture helped > get rid of the lost frames problem, but the feed is still quite laggy > (not > to mention I'm also losing lots of frames). I see a lot of gstreamer > warnings: > > > 0:00:06.013149788 15050 0x9250a0 WARN videodecoder > gstvideodecoder.c:4003:_gst_video_decoder_error:<jpegdec2> error: > Failed to > decode JPEG image > 0:00:06.013162819 15050 0x9250a0 WARN videodecoder > gstvideodecoder.c:4005:_gst_video_decoder_error:<jpegdec2> error: > Decode > error #62: Unsupported JPEG process: SOF type 0xcf > > > new pipeline: > > gst-launch-1.0 videomixer name=mix sink_0::xpos=0 sink_0::ypos=0 Replace with videomixer with compositor, and configure latency property to something large enough (experiment to find the good value for your use case). The leaky queues won't be required anymore since compositor has it's own queue mechanism. Make sure so add downstream caps, compositor will produce a perfect framerate from the detected framerate in that filter. > sink_1::xpos=640 sink_1::ypos=0 sink_2::xpos=0 sink_2::ypos=480 > sink_3::xpos=640 sink_3::ypos=480 ! autovideosink sync=false v4l2src > device=/dev/video0 do-timestamp=false ! > image/jpeg,height=480,width=640,framerate=30/1 ! queue leaky=2 > max-size-buffers=1 ! jpegparse ! jpegdec ! videoconvert ! mix.sink_0 > v4l2src device=/dev/video1 do-timestamp=false ! > image/jpeg,height=480,width=640,framerate=30/1 ! queue leaky=2 > max-size-buffers=1 ! jpegparse ! jpegdec ! videoconvert ! mix.sink_1 > v4l2src device=/dev/video2 do-timestamp=false ! > image/jpeg,height=480,width=640,framerate=30/1 ! queue leaky=2 > max-size-buffers=1 ! jpegparse ! jpegdec ! videoconvert ! mix.sink_2 > v4l2src device=/dev/video3 do-timestamp=false ! > image/jpeg,height=480,width=640,framerate=30/1 ! queue leaky=2 > max-size-buffers=1 ! jpegparse ! jpegdec ! videoconvert ! mix.sink_3 > > > > > -- > Sent from: http://gstreamer-devel.966125.n4.nabble.com/ > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
Hi, thank you so much for the response. What do you mean by "make sure to add downstream caps?" Another piece of info I learned: I'm getting a TON of jpeg decode errors, but only with a very specific camera module, that can output up to 1280x720 pixel resolution. If I use cameras that only run at max 640x480 resolution, they get no decode errors. Is there something I can do that may be specific to the decode errors in this camera? Thank you! On Thu, Sep 28, 2017 at 6:30 PM Nicolas Dufresne <[hidden email]> wrote: Le jeudi 28 septembre 2017 à 16:45 -0700, ashwath a écrit : _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
In reply to this post by Nicolas Dufresne-5
Hi, thank you so much for the response.
What do you mean by "make sure to add downstream caps?" Another piece of info I learned: I'm getting a TON of jpeg decode errors, but only with a very specific camera module, that can output up to 1280x720 pixel resolution. If I use cameras that only run at max 640x480 resolution, they get no decode errors. Is there something I can do that may be specific to the decode errors in this camera? Thank you! -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: v4l2rc MJPEG to Videomixer slow - alternative pathways?
|
|
In reply to this post by ashwath
Le vendredi 29 septembre 2017 à 01:39 +0000, Ashwath Rajan a écrit :
> Hi, thank you so much for the response. > > What do you mean by "make sure to add downstream caps?" To add a caps filter after compisitor, something like: compositor ! video/x-raw,width=1920,height=1080,framerate=30/1 ! ... > > Another piece of info I learned: I'm getting a TON of jpeg decode > errors, but only with a very specific camera module, that can output > up to 1280x720 pixel resolution. If I use cameras that only run at > max 640x480 resolution, they get no decode errors. Is there > something I can do that may be specific to the decode errors in this > camera? You are probably over USB 2 and saturating the bandwidth. To protect your decoder, you can use this after v4l2src to drop corrupted buffers: ... ! identity drop-buffer-flags=corrupted ! ... To be fair, I didn't think about encoded data when I decide to only "mark" the corrupted rather then dropping the data. At the same time, most of the time you still get something, which in certain situation is better then nothing. > > Thank you! > > On Thu, Sep 28, 2017 at 6:30 PM Nicolas Dufresne <[hidden email]> wrote: > > Le jeudi 28 septembre 2017 à 16:45 -0700, ashwath a écrit : > > > I'm sorry, just 1 more update. Adding a leaky queue after the > > > capture helped > > > get rid of the lost frames problem, but the feed is still quite laggy > > > (not > > > to mention I'm also losing lots of frames). I see a lot of gstreamer > > > warnings: > > > > > > > > > 0:00:06.013149788 15050 0x9250a0 WARN videodecoder > > > gstvideodecoder.c:4003:_gst_video_decoder_error:<jpegdec2> error: > > > Failed to > > > decode JPEG image > > > 0:00:06.013162819 15050 0x9250a0 WARN videodecoder > > > gstvideodecoder.c:4005:_gst_video_decoder_error:<jpegdec2> error: > > > Decode > > > error #62: Unsupported JPEG process: SOF type 0xcf > > > > > > > > > new pipeline: > > > > > > gst-launch-1.0 videomixer name=mix sink_0::xpos=0 sink_0::ypos=0 > > > > Replace with videomixer with compositor, and configure latency property > > to something large enough (experiment to find the good value for your > > use case). The leaky queues won't be required anymore since compositor > > has it's own queue mechanism. Make sure so add downstream caps, > > compositor will produce a perfect framerate from the detected framerate > > in that filter. > > > > > sink_1::xpos=640 sink_1::ypos=0 sink_2::xpos=0 sink_2::ypos=480 > > > sink_3::xpos=640 sink_3::ypos=480 ! autovideosink sync=false v4l2src > > > device=/dev/video0 do-timestamp=false ! > > > image/jpeg,height=480,width=640,framerate=30/1 ! queue leaky=2 > > > max-size-buffers=1 ! jpegparse ! jpegdec ! videoconvert ! mix.sink_0 > > > v4l2src device=/dev/video1 do-timestamp=false ! > > > image/jpeg,height=480,width=640,framerate=30/1 ! queue leaky=2 > > > max-size-buffers=1 ! jpegparse ! jpegdec ! videoconvert ! mix.sink_1 > > > v4l2src device=/dev/video2 do-timestamp=false ! > > > image/jpeg,height=480,width=640,framerate=30/1 ! queue leaky=2 > > > max-size-buffers=1 ! jpegparse ! jpegdec ! videoconvert ! mix.sink_2 > > > v4l2src device=/dev/video3 do-timestamp=false ! > > > image/jpeg,height=480,width=640,framerate=30/1 ! queue leaky=2 > > > max-size-buffers=1 ! jpegparse ! jpegdec ! videoconvert ! mix.sink_3 > > > > > > > > > > > > > > > -- > > > Sent from: http://gstreamer-devel.966125.n4.nabble.com/ > > > _______________________________________________ > > > gstreamer-devel mailing list > > > [hidden email] > > > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel > > _______________________________________________ > > gstreamer-devel mailing list > > [hidden email] > > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel > > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
«
Return to GStreamer-devel
|
1 view|%1 views
| Free forum by Nabble | Edit this page |