Still can't get videomixer to do live video at full frame rate
Locked
 
|
Trying to display two genlocked camera images from two v4l2 capture devices in a single frame, ultimately to an appsink. This pipeline works but can't get full frame rate :( gst-launch v4l2src device=/dev/video0 ! queue ! \ video/x-raw-yuv,format =\(fourcc\)I420, framerate=\(fraction\)30000/1001, width=640, height=480 ! \ videomixer name=mix sink_0::alpha=1.0 sink_0::xpos=0 sink_0::ypos=0 sink_1::alpha=1.0 sink_1::xpos=0 sink_1::ypos=480 \ sink_2::zorder=3 ! xvimagesink \ v4l2src device=/dev/video1 ! queue ! video/x-raw-yuv,format=\(fourcc\)I420, framerate=\(fraction\)30000/1001, width=640, height=480 ! mix. \ videotestsrc pattern=2 ! queue ! video/x-raw-yuv,format=\(fourcc\)I420, framerate=\(fraction\)1/1, width=640, height=960 ! mix. Top shows the gst-launch command running 7-12% CPU. I can add a second xvimagesink while the above is running with: gst-launch v4l2src device=/dev/video4 ! video/x-raw-yuv,format =\(fourcc\)I420, framerate=\(fraction\)30000/1001, width=640, height=480 ! xvimagesink and it displays at full frame rate even though it shows 34-45% CPU in top while running. Xorg always stays below 10%. Occasionally the videomixer gst-launch command spits out (not often, but I saw it this time and its why I'm posting this) a warning message: WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped. Additional debug info: gstbasesink.c(2597): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: There may be a timestamping problem, or this computer is too slow. What is a timestamping problem and what can I do about it? _______________________________________________ gstreamer-devel mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
On Tue, Aug 16, 2011 at 5:03 PM, wally bkg <[hidden email]> wrote:
> > Trying to display two genlocked camera images from two v4l2 capture devices > in a single frame, ultimately to an appsink. > > This pipeline works but can't get full frame rate :( > > gst-launch v4l2src device=/dev/video0 ! queue ! \ > video/x-raw-yuv,format =\(fourcc\)I420, framerate=\(fraction\)30000/1001, > width=640, height=480 ! \ > videomixer name=mix sink_0::alpha=1.0 sink_0::xpos=0 sink_0::ypos=0 > sink_1::alpha=1.0 sink_1::xpos=0 sink_1::ypos=480 \ > sink_2::zorder=3 ! xvimagesink \ > v4l2src device=/dev/video1 ! queue ! video/x-raw-yuv,format=\(fourcc\)I420, > framerate=\(fraction\)30000/1001, width=640, height=480 ! mix. \ > videotestsrc pattern=2 ! queue ! video/x-raw-yuv,format=\(fourcc\)I420, > framerate=\(fraction\)1/1, width=640, height=960 ! mix. I'm not sure what's wrong with this pipeline, it's almost identical to one I've used without problem. > [...] > > Occasionally the videomixer gst-launch command spits out (not often, but I > saw it this time and its why I'm posting this) a warning message: > > WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A > lot of buffers are being dropped. > Additional debug info: > gstbasesink.c(2597): gst_base_sink_is_too_late (): > /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: > There may be a timestamping problem, or this computer is too slow. > > > What is a timestamping problem and what can I do about it? It's not likely a timestamping problem, but rather your system/pipeline can't keep up. You can ignore this, increase the time/frames/size of your queue's, or maybe set sync=false to xvimagesink. -Josh _______________________________________________ gstreamer-devel mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: Still can't get videomixer to do live video at full frame rate
|
|
On Wed, 2011-08-17 at 07:06 -0400, Josh Doe wrote:
> I'm not sure what's wrong with this pipeline, it's almost identical to > one I've used without problem. I haven't read the thread in detail, but still, a couple of comments: Have you tried videomixer2 yet? The old videomixer mixes streams based on buffer durations and not running time, which may lead to weird behaviour. It also doesn't handle live video well at all as far as I know. If you must use the old videomixer instead of videomixer2, try adding videorate elements before the input. Cheers -Tim _______________________________________________ gstreamer-devel mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
Videomixer2 was the first thing I tried, doesn't exist in the Ubuntu 10.04 system:
gst-inspect videomixer2 No such element or plugin 'videomixer2' I thought the documentation blurb said videomixer2 will replace videomixer, I'd assumed that had happened by now. gst-inspect videomixer Factory Details: Long name: Video mixer Class: Filter/Editor/Video Description: Mix multiple video streams Author(s): Wim Taymans <wim@fluendo.com> Rank: primary (256) Plugin Details: Name: videomixer Description: Video mixer Filename: /usr/lib/gstreamer-0.10/libgstvideomixer.so Version: 0.10.21 License: LGPL Source module: gst-plugins-good Binary package: GStreamer Good Plugins (Ubuntu) Origin URL: https://launchpad.net/distros/ubuntu/+source/gst-plugins-good0.10 Curious that adding timeoverlays appears to fix the performance problem (at least for a while): // live dual video gst-launch v4l2src device=/dev/video0 ! \ video/x-raw-yuv,format =\(fourcc\)I420, framerate=\(fraction\)30000/1001, width=640, height=480 ! timeoverlay ! \ videomixer name=mix sink_0::alpha=1.0 sink_0::xpos=0 sink_0::ypos=0 sink_1::alpha=1.0 sink_1::xpos=0 sink_1::ypos=480 sink_2::zorder=3 ! xvimagesink \ v4l2src device=/dev/video1 ! video/x-raw-yuv,format=\(fourcc\)I420, framerate=\(fraction\)30000/1001, width=640, height=480 ! timeoverlay ! mix. \ videotestsrc pattern=2 ! video/x-raw-yuv,format=\(fourcc\)I420, framerate=\(fraction\)1/1, width=640, height=960 ! mix. // third image for full speed reference, started in second xterm window: gst-launch v4l2src device=/dev/video4 ! video/x-raw-yuv,format =\(fourcc\)I420, framerate=\(fraction\)30000/1001, width=640, height=480 ! \ timeoverlay ! xvimagesink Top now shows about 57% for the stacked dual image display, and 45% for the third image gst-launch command, with Xorg at about 11%. I dislike voodoo programming in the absence of decent documentation, but this seems to be usable, its been running the whole time I've been composing this message -- spoke too soon now its clearly not maintaining the frame rate as the incrementing of the time overlay clearly shows vs. the reference frame. There are other "bar-codes" overlaid on the video tape I playback for testing that makes it easy to see if the frame rate is full or not. Normal epochs need to be a lot longer than this! I'll try playing with the sync property on the xvimagesink and anything else I can think of, but its about time to give up and go to "Plan B". |
Re: Still can't get videomixer to do live video at full frame rate
|
|
On Wed, 2011-08-17 at 06:48 -0700, wally_bkg wrote:
> Videomixer2 was the first thing I tried, doesn't exist in the Ubuntu 10.04 > system: (...) > I thought the documentation blurb said videomixer2 will replace videomixer, > I'd assumed that had happened by now. It has. Maybe you need to upgrade to a newer Ubuntu or try the GStreamer PPA for newer GStreamer packages. 10.04 is more than a year old.. Cheers -Tim _______________________________________________ gstreamer-devel mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
I tried the 10.04 PPA a few months ago and it hosed my system to the point that: gst-launch v4l2src ! xvimagesink wouldn't even work. All other multimedia stuff was pretty much foobar too. Recovering was a PITA I'm not desperate enough to try again at present. Changing the sync property on the xvimagesink to false may be the solution as this pipeline: gst-launch v4l2src device=/dev/video0 ! video/x-raw-yuv,format =\(fourcc\)I420, framerate=\(fraction\)30000/1001, width=640, height=480 ! timeoverlay ! videomixer name=mix sink_0::alpha=1.0 sink_0::xpos=0 sink_0::ypos=0 sink_1::alpha=1.0 sink_1::xpos=0 sink_1::ypos=480 sink_2::zorder=3 ! xvimagesink sync=false v4l2src device=/dev/video1 ! video/x-raw-yuv,format=\(fourcc\)I420, framerate=\(fraction\)30000/1001, width=640, height=480 ! timeoverlay ! mix. videotestsrc pattern=2 ! video/x-raw-yuv,format=\(fourcc\)I420, framerate=\(fraction\)1/1, width=640, height=960 ! mix. Has maintained full speed for a stream for well over an hour, which is almost always a long enough epoch, although the Xorg CPU usage has increased to ~25%. It has also survived the video glitches of ff/rev of the tape and the transitions of where the recording was started and stopped originally. I'm having some trouble with the C syntax to get the videomixer sink pads and set the ypos properties on them as done by the gst-launch. Any examples out there? The real test will be when I get it to terminate in my appsink instead of xvimagesink so I can delta the stacked buffer timestamps to verify full frame rate is actually achieved. |
|
|
I figured out something that seems to work, the docs say function is depreciated and should not be used, but its all I could get to work: // build up the pipeline // container blank frame link_ok=gst_element_link_many(blank, filter960, mixer, NULL); if (!link_ok) { g_warning ("Failed to link: blank, filter960, mixer\n"); } // camera R link_ok=gst_element_link_many(source, colorspace, I420filter, scale, tover, mixer, NULL); if (!link_ok) { g_warning ("Failed to link: source, colorspace, I420filter, scale, mixer\n"); } // camera L link_ok=gst_element_link_many(source1, colorspace1, I420filter1, scale1, tover1, mixer, NULL); if (!link_ok) { g_warning ("Failed to link: source1, colorspace1, I420filter1, scale1, mixer\n"); } // mixer to xvimagesink link_ok=gst_element_link(mixer, sink); if (!link_ok) { g_warning ("Failed to link: mixer, sink\n"); } // shift a camera to bottom half of container frame mixsink1=gst_element_get_pad(mixer,"sink_1"); if(mixsink1){ g_object_set (G_OBJECT (mixsink1), "ypos", 480, NULL); }else g_print("Can't get mixer sink_1 pad!"); |
|
|
In reply to this post by Tim-Philipp Müller-2
I had another system temporarily available to "clone" for another application (its the same target system I'm aiming at) so after I cloned it there was no risk to breaking it. I installed the lucid PPA gstreamer and at least its not totally broken now. However gst-inspect shows no videomixer2 and its also missing ffenc-mpeg4. In fact, if you believe the version numbers, the PPA has the same as my stock system according to gst-inspect videomexer: Filename: /usr/lib/gstreamer-0.10/libgstvideomixer.so Version: 0.10.21 I've got the C version of the pipeline running and while it looks good the overlaid, timestamps are skewed (initially by about a field (18 milliseconds) and it varies throughout the run. I use ffenc-mpeg4 to save the combined frames so I can detect dropped frames and check skew by reading the barcode on the test video. We can live with a constant skew between the frames (although its sub-optimal to have it) and we can live with a few dropped frames as long a the same frames are dropped in each camera. This doesnot seem to be happening. The target systems are not setup to build the code. Am I correct in assuming that an upgrade to the gstreamer libraries should not require a rebuild of code that uses them? (ignoring the missing pieces for now). |
Re: Still can't get videomixer to do live video at full frame rate
|
|
In reply to this post by wally_bkg
On 08/17/11 18:37, wally_bkg wrote:
> wally_bkg wrote: >> >> I'm having some trouble with the C syntax to get the videomixer sink pads >> and set the ypos properties on them as done by the gst-launch. Any >> examples out there? >> >> > > I figured out something that seems to work, the docs say function is > depreciated and should not be used, but its all I could get to work: > > // build up the pipeline > // container blank frame > link_ok=gst_element_link_many(blank, filter960, mixer, NULL); > if (!link_ok) { > g_warning ("Failed to link: blank, filter960, mixer\n"); > } > // camera R > link_ok=gst_element_link_many(source, colorspace, I420filter, scale, tover, > mixer, NULL); > if (!link_ok) { > g_warning ("Failed to link: source, colorspace, I420filter, scale, > mixer\n"); > } > // camera L > link_ok=gst_element_link_many(source1, colorspace1, I420filter1, scale1, > tover1, mixer, NULL); > if (!link_ok) { > g_warning ("Failed to link: source1, colorspace1, I420filter1, scale1, > mixer\n"); > } > // mixer to xvimagesink > link_ok=gst_element_link(mixer, sink); > if (!link_ok) { > g_warning ("Failed to link: mixer, sink\n"); > } > // shift a camera to bottom half of container frame > mixsink1=gst_element_get_pad(mixer,"sink_1"); Stefan > if(mixsink1){ > g_object_set (G_OBJECT (mixsink1), "ypos", 480, NULL); > }else > g_print("Can't get mixer sink_1 pad!"); > > > -- > View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Still-can-t-get-videomixer-to-do-live-video-at-full-frame-rate-tp3748457p3750509.html > Sent from the GStreamer-devel mailing list archive at Nabble.com. > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel _______________________________________________ gstreamer-devel mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
Thanks, but this is a bit confusing. Are you saying I should just use gst_element_get_static_pad() with the same arguments I used with gst_element_get_pad()? I though the mixer pads were "request". The documentation says the problem with gst_element_get_pad() was after getting the pad reference it wasn't clear which kind of pad it was. I'm not really clear why I need to care at this "top level" of my program as these pad links are permanent for the duration of my program and I only needed the reference to set the ypos property on it before running the pipeline. I don't see the need to free it here, what bad might happen if I don't free it before the program terminates? As a library design issue, why can't the pad_free function use object properties to free the proper pad type when called and free the user from knowing these internal details? My main complaint with gstreamer is there always seems to be multiple ways to do something and little or no guidance as to in what situation one way would be better than the other. I tend to want to use an Occam's razor in that the approach that uses fewer lines of code is certainly better in terms of the next person to inherit my code when I retire. |
Re: Still can't get videomixer to do live video at full frame rate
|
|
On 08/19/11 18:12, wally_bkg wrote:
> Stefan Kost wrote: >> On 08/17/11 18:37, wally_bkg wrote: >>> wally_bkg wrote: >>> // shift a camera to bottom half of container frame >>> mixsink1=gst_element_get_pad(mixer,"sink_1"); >> gst_element_get_static_pad >> >> Stefan >> >>> if(mixsink1){ >>> g_object_set (G_OBJECT (mixsink1), "ypos", 480, NULL); >>> }else >>> g_print("Can't get mixer sink_1 pad!"); > Thanks, but this is a bit confusing. Are you saying I should just use > gst_element_get_static_pad() with the same arguments I used with > gst_element_get_pad()? > > I though the mixer pads were "request". The documentation says the problem > with gst_element_get_pad() was after getting the pad reference it wasn't > clear which kind of pad it was. Sorry indeed those are request pads, so please use the other function. > I'm not really clear why I need to care at this "top level" of my program as > these pad links are permanent for the duration of my program and I only > needed the reference to set the ypos property on it before running the > pipeline. I don't see the need to free it here, what bad might happen if I > don't free it before the program terminates? We can't change this in 0.10. Stefan > As a library design issue, why can't the pad_free function use object > properties to free the proper pad type when called and free the user from > knowing these internal details? > > My main complaint with gstreamer is there always seems to be multiple ways > to do something and little or no guidance as to in what situation one way > would be better than the other. > > I tend to want to use an Occam's razor in that the approach that uses fewer > lines of code is certainly better in terms of the next person to inherit my > code when I retire. > > > -- > View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Still-can-t-get-videomixer-to-do-live-video-at-full-frame-rate-tp3748457p3755579.html > Sent from the GStreamer-devel mailing list archive at Nabble.com. > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel _______________________________________________ gstreamer-devel mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
OK but the documentation at: http://developer.gnome.org/gstreamer/unstable/GstElement.html#gst-element-request-pad says: gst_element_get_request_pad() This method is slow and will be deprecated in the future. New code should use gst_element_request_pad() with the requested template. I'm not trying to be difficult, but using "gst_element_request_pad() with the requested template" would add complexity to my code for dubious benefits. If I'm missing understanding something, please explain. Can I expect any real difference in using gst_element_get_pad() instead of gst_element_get_request_pad()? Other than gst_element_get_pad() not being there is some future gstreamer version? Obviously this would be a good reason for such a trivial change, although it could be moot if gst_element_get_request_pad() ends up suffering the same fate. The similarity among the names makes my head spin and my eyes water! :( I understand that in a media player that is building pipelines on the fly as the media type is discovered, such complexity is required to prevent memory leaks or worse, but once my pipeline is built and playing its unchanged for the duration of the program. I'm not freeing the pad requests or other references after I get them, and far as I can tell nothing bad happens. Having all these references hang around was certainly convenient during development to be able to do queries and g_prints of values to see what was going on without having to redo the pad request and free along with it. What is accomplished by my <500 lines of code (including whitespace, but I hate white space :) ) speaks volumes to the power of gstreamer, but the development time to write such a small program has been absurd given all the arcane knowledge of the internals that I had to pick up along the way, with great difficulty. I'd vote these functions not be removed, but merely documented as being a trade off of simplicity vs. speed/power that are really only suitable for static pipelines. Actually the issue in the title appears to be solved now, although some combinations of capture devices seem to have a frame skew (camera0 image one frame ahead or behind the camera1 image) on start up, while others don't. It may be a USB transport issue and/or differences in number of hardware buffers in the capture device. If its consistent, I suspect I can take action based on the capture card "device-name" to drop a frame on start up or not. I've more testing to do. Is there an element that would be easy to drop the first "n" frames and pass the rest? |
|
|
In reply to this post by Stefan Sauer
Actually gst_element_get_request_pad() doesn't work, I get: GStreamer-CRITICAL **: Padname sink_0 is not unique in element mixer, not adding I'd forgotten about this while I was stumbling around trying to figure out how to do it before I came up with using gst_element_get_pad(). It seems the "depreciated and to be removed" gst_element_get_pad() function does something that can be most useful and code simplifying. Namely returning a reference to a request pad that was automatically linked with: gst_element_link_many(source1, colorspace1, scale1, I420filter1, vrate1, tover1, queue1, mixer, NULL); In light of this, I think removing it from a future version would be a mistake, unless a clearly documented better way is introduced. |
Re: Still can't get videomixer to do live video at full frame rate
|
|
On 08/20/11 00:29, wally_bkg wrote:
> Stefan Kost wrote: >> On 08/19/11 18:12, wally_bkg wrote: >>> Stefan Kost wrote: >>>> On 08/17/11 18:37, wally_bkg wrote: >>>>> wally_bkg wrote: >>>>> // shift a camera to bottom half of container frame >>>>> mixsink1=gst_element_get_pad(mixer,"sink_1"); >>>> gst_element_get_static_pad >>>> >>>> Stefan >>>> >>>>> if(mixsink1){ >>>>> g_object_set (G_OBJECT (mixsink1), "ypos", 480, NULL); >>>>> }else >>>>> g_print("Can't get mixer sink_1 pad!"); >>> Thanks, but this is a bit confusing. Are you saying I should just use >>> gst_element_get_static_pad() with the same arguments I used with >>> gst_element_get_pad()? >>> >>> I though the mixer pads were "request". The documentation says the >>> problem >>> with gst_element_get_pad() was after getting the pad reference it wasn't >>> clear which kind of pad it was. >> Sorry indeed those are request pads, so please use the other function. >> > Actually gst_element_get_request_pad() doesn't work, I get: > > GStreamer-CRITICAL **: Padname sink_0 is not unique in element mixer, not > adding gst_element_get_request_pad(mixer,"sink_%d"); I made a small change to the docs to point that out. > I'd forgotten about this while I was stumbling around trying to figure out > how to do it before I came up with using gst_element_get_pad(). > > > It seems the "depreciated and to be removed" gst_element_get_pad() function > does something that can be most useful and code simplifying. The problem with hat one is that it will leak request_pads. > Namely returning a reference to a request pad that was automatically linked > with: > > gst_element_link_many(source1, colorspace1, scale1, I420filter1, vrate1, > tover1, queue1, mixer, NULL); > > > In light of this, I think removing it from a future version would be a > mistake, unless a clearly documented better way is introduced. We are improving the documentation. You need to understand that a static pad is always there, for request_pads you create a new one when getting it, someone has to drop it also at some point. Stefan > > > > -- > View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Still-can-t-get-videomixer-to-do-live-video-at-full-frame-rate-tp3748457p3756256.html > Sent from the GStreamer-devel mailing list archive at Nabble.com. > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel _______________________________________________ gstreamer-devel mailing list [hidden email] http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
«
Return to GStreamer-devel
|
1 view|%1 views
| Free forum by Nabble | Edit this page |