Get v4l2src width and height before pipeline runs?

[wally_bkg]

Up to now I've been setting caps with a specific width and height needed for our table driven image processing algorithm and using videoscale to force the pipeline to that size (640x480) at the appsink.

The algorithm's original author visited recently and his recollections and insights have allowed me to remove that restriction.  This has exposed the fact that I don't know how to get the width and height caps until the pipeline is running, at which point its effectively too late.

Is there a way to get the results of the caps negotiation before the pipeline is running?

 

[Edward Hervey]

Hi,

  Caps negotiation is *always* done before any data passing happens.
  i.e. your element's pad setcaps function will be called before any
call to the chainfunction.

  If you haven't implemented a pad setcaps function ... there's your
problem :)

    Edward

On Wed, 2011-10-26 at 13:34 -0700, wally_bkg wrote:

> Up to now I've been setting caps with a specific width and height needed for
> our table driven image processing algorithm and using videoscale to force
> the pipeline to that size (640x480) at the appsink.
>
> The algorithm's original author visited recently and his recollections and
> insights have allowed me to remove that restriction.  This has exposed the
> fact that I don't know how to get the width and height caps until the
> pipeline is running, at which point its effectively too late.
>
> Is there a way to get the results of the caps negotiation before the
> pipeline is running?
>
>  
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Get-v4l2src-width-and-height-before-pipeline-runs-tp3942212p3942212.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[wally_bkg]

Edward Hervey wrote

Hi,

  Caps negotiation is *always* done before any data passing happens.
  i.e. your element's pad setcaps function will be called before any
call to the chainfunction.

  If you haven't implemented a pad setcaps function ... there's your
problem :)

    Edward

I'm using appsink and appsrc, not a custom element.
I guess that could be the problem.  The goal is to use gstreamer
and as much of its "automation" as possible.

What I'll probably do is re-write the video capture front end to be v4l2 (libv4l2) code and dispense with the appsink.  I've already used some v4l2 code to set the norm (NTSC) and select the input (Composite) as the gst-tuner interface has not been reliable across the six types of capture devices I have to test against.

Thanks, seems you've confirmed my fears.

[Edward Hervey]

Hi,

On Thu, 2011-10-27 at 12:10 -0700, wally_bkg wrote:
> I'm using appsink and appsrc, not a custom element.
> I guess that could be the problem.  The goal is to use gstreamer
> and as much of its "automation" as possible.
>
> What I'll probably do is re-write the video capture front end to be v4l2
> (libv4l2) code and dispense with the appsink.

  Why aren't you using the v4l2src/v4l2sink elements ? If you have
issues with them, please explain it in detail and/or file a bug report.

>  I've already used some v4l2
> code to set the norm (NTSC) and select the input (Composite) as the
> gst-tuner interface has not been reliable across the six types of capture
> devices I have to test against.
>
> Thanks, seems you've confirmed my fears.
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Get-v4l2src-width-and-height-before-pipeline-runs-tp3942212p3945672.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[wally_bkg]

Edward Hervey wrote

  Why aren't you using the v4l2src/v4l2sink elements ? If you have
issues with them, please explain it in detail and/or file a bug report.

I have vrl2src elements at the start of my pipeline that ends with an appsink, this worked fine with fixed video frame sizes, but now that I've removed that restriction in the image processing algorithm the question is how do I get the frame size from v4l2 without setting the pipeline to playing?

I've had lots of issues with the gstreamer v4l2src tuner interface and never could get it to reliably set the norm and select the input across the various v4l2 capture devices on hand.  As to bug reports, the answer was always to try a newer version, but when I tried the Ubuntu 10.04 gstreamer PPA it broke all things multimedia and wasted a fair amount of time recovering.  At that point it became easier to just use v4l2 calls to set the norm and select the input, then build the pipeline.

Grossly simplified my code is

V4l2src->ffmpegcolorspace->fixed caps filter->appsink
duplicate the appsink buffer data
do some image processing on the duplicated buffer data
appsrc->xvimagesink (for realtime viewing, of the processed data)
appsrc->mp4 encoding->filesink ( original data to allow non-realtime processing afterwards)

Works quite well with fixed caps.  I'm stuck on how to get the frame height and width without fixed caps before the pipeline runs.   I have to build a bunch of tables that depend on the frame size and takes a couple of seconds to do (perhaps not the most efficient recursive code, but it only runs once).

I could work around this by pre-building tables of the possibilities (640x480, 704x480, & 720x480 so far, depending on the capture device, fixed caps was just set to 640x480) but the real issue is I'd like the v4l2 timestamps from the driver to use to synchronize A/D data from another process running on the same machine.  I asked about this earlier but got no response:

http://gstreamer-devel.966125.n4.nabble.com/Getting-V4l2-timestamps-from-v4l2src-td3915764.html

Dealing with various frame sizes is at this point only needed to test my changes (which I quickly did by changing the fixed caps  to match the different capture card possibilities) in anticipation of someday using HD video.  The timestamps to synchronize the A/D data with the video is an action item.

[Stefan Sauer]

On 10/28/2011 10:53 PM, wally_bkg wrote:

> Edward Hervey wrote:
>>   Why aren't you using the v4l2src/v4l2sink elements ? If you have
>> issues with them, please explain it in detail and/or file a bug report.
>>
> I have vrl2src elements at the start of my pipeline that ends with an
> appsink, this worked fine with fixed video frame sizes, but now that I've
> removed that restriction in the image processing algorithm the question is
> how do I get the frame size from v4l2 without setting the pipeline to
> playing?
>
> I've had lots of issues with the gstreamer v4l2src tuner interface and never
> could get it to reliably set the norm and select the input across the
> various v4l2 capture devices on hand.  As to bug reports, the answer was
> always to try a newer version, but when I tried the Ubuntu 10.04 gstreamer
> PPA it broke all things multimedia and wasted a fair amount of time
> recovering.

Unless we figure out wat broek in your setup, you will be stuck with an
old version. The v4l2 api is a bit fragile regarding to what things to
call in which order for which device/driver. In order to try a new
v4l2src, you can always just build git head and only install v4l2src, I
don't think we're using API there that requires a new core version.

Stefan

>   At that point it became easier to just use v4l2 calls to set
> the norm and select the input, then build the pipeline.
>
> Grossly simplified my code is
>
> V4l2src->ffmpegcolorspace->fixed caps filter->appsink
> duplicate the appsink buffer data
> do some image processing on the duplicated buffer data
> appsrc->xvimagesink (for realtime viewing, of the processed data)
> appsrc->mp4 encoding->filesink ( original data to allow non-realtime
> processing afterwards)
>
> Works quite well with fixed caps.  I'm stuck on how to get the frame height
> and width without fixed caps before the pipeline runs.   I have to build a
> bunch of tables that depend on the frame size and takes a couple of seconds
> to do (perhaps not the most efficient recursive code, but it only runs
> once).
>
> I could work around this by pre-building tables of the possibilities
> (640x480, 704x480, & 720x480 so far, depending on the capture device, fixed
> caps was just set to 640x480) but the real issue is I'd like the v4l2
> timestamps from the driver to use to synchronize A/D data from another
> process running on the same machine.  I asked about this earlier but got no
> response:
>
> http://gstreamer-devel.966125.n4.nabble.com/Getting-V4l2-timestamps-from-v4l2src-td3915764.html
>
> Dealing with various frame sizes is at this point only needed to test my
> changes (which I quickly did by changing the fixed caps  to match the
> different capture card possibilities) in anticipation of someday using HD
> video.  The timestamps to synchronize the A/D data with the video is an
> action item.
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Get-v4l2src-width-and-height-before-pipeline-runs-tp3942212p3949188.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Kapil Agrawal]

Just a thought, may not be realted to this specific issue of yours
Writing your own gstreamer plugin to do image processing stuff would make it more cleaner and better.

like v4l2src ! ffmpegcolorspace ! your_image_processing_plugin ! xvimagesink ?

Best
- Kapil

On Sat, Oct 29, 2011 at 2:23 AM, wally_bkg <[hidden email]> wrote:

Edward Hervey wrote:
>
>   Why aren't you using the v4l2src/v4l2sink elements ? If you have
> issues with them, please explain it in detail and/or file a bug report.
>

I have vrl2src elements at the start of my pipeline that ends with an
appsink, this worked fine with fixed video frame sizes, but now that I've
removed that restriction in the image processing algorithm the question is
how do I get the frame size from v4l2 without setting the pipeline to
playing?

I've had lots of issues with the gstreamer v4l2src tuner interface and never
could get it to reliably set the norm and select the input across the
various v4l2 capture devices on hand.  As to bug reports, the answer was
always to try a newer version, but when I tried the Ubuntu 10.04 gstreamer
PPA it broke all things multimedia and wasted a fair amount of time
recovering.  At that point it became easier to just use v4l2 calls to set
the norm and select the input, then build the pipeline.

Grossly simplified my code is

V4l2src->ffmpegcolorspace->fixed caps filter->appsink
duplicate the appsink buffer data
do some image processing on the duplicated buffer data
appsrc->xvimagesink (for realtime viewing, of the processed data)
appsrc->mp4 encoding->filesink ( original data to allow non-realtime
processing afterwards)

Works quite well with fixed caps.  I'm stuck on how to get the frame height
and width without fixed caps before the pipeline runs.   I have to build a
bunch of tables that depend on the frame size and takes a couple of seconds
to do (perhaps not the most efficient recursive code, but it only runs
once).

I could work around this by pre-building tables of the possibilities
(640x480, 704x480, & 720x480 so far, depending on the capture device, fixed
caps was just set to 640x480) but the real issue is I'd like the v4l2
timestamps from the driver to use to synchronize A/D data from another
process running on the same machine.  I asked about this earlier but got no
response:

http://gstreamer-devel.966125.n4.nabble.com/Getting-V4l2-timestamps-from-v4l2src-td3915764.html

Dealing with various frame sizes is at this point only needed to test my
changes (which I quickly did by changing the fixed caps  to match the
different capture card possibilities) in anticipation of someday using HD
video.  The timestamps to synchronize the A/D data with the video is an
action item.



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Get-v4l2src-width-and-height-before-pipeline-runs-tp3942212p3949188.html

Sent from the GStreamer-devel mailing list archive at Nabble.com.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
http://www.linkedin.com/in/kapilagrawal
www.mediamagictechnologies.com
(Consultancy - Multimedia, Gstreamer, ffmpeg, Streaming for platforms- linux, windows, TI, android)



_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[wally_bkg]

Kapil Agrawal wrote

Just a thought, may not be realted to this specific issue of yours
Writing your own gstreamer plugin to do image processing stuff would make
it more cleaner and better.

like v4l2src ! ffmpegcolorspace ! your_image_processing_plugin !
xvimagesink ?

Best
- Kapil

If you go way back in the archives, I debated this initially as appsrc and appsink were not part of the gstreamer shipped with Ubuntu 8.04.

The main reason I ended up going with appsrc/appsink is I'm quickly approaching retirement and found the process of downloading the template and getting the build environment working to build a plugin way too complicated and poorly documented to foist upon my replacement.

Appsrc/appsink lets me build a "normal" application, although there still are a lot of obscure bits to be installed, their installation can be largely automated with dpkg --set-selections and dpkg --get-selections

I did write a plugin for Avidemux to to be the "front end" for our offline processing, since gstreamer can't play the VOB files from a stand-alone DVD-R box.

There is no doubt that V4L2 is more of a problem than a solution as its a very poor abstraction exposing almost all the weirdness of the underlying hardware.  Using libv4l2 helps a lot.

[Stefan Sauer]

On 11/01/2011 02:48 PM, wally_bkg wrote:

> Kapil Agrawal wrote:
>> Just a thought, may not be realted to this specific issue of yours
>> Writing your own gstreamer plugin to do image processing stuff would make
>> it more cleaner and better.
>>
>> like v4l2src ! ffmpegcolorspace ! your_image_processing_plugin !
>> xvimagesink ?
>>
>> Best
>> - Kapil
>>
> If you go way back in the archives, I debated this initially as appsrc and
> appsink were not part of the gstreamer shipped with Ubuntu 8.04.
>
> The main reason I ended up going with appsrc/appsink is I'm quickly
> approaching retirement and found the process of downloading the template and
> getting the build environment working to build a plugin way too complicated
> and poorly documented to foist upon my replacement.

Have you noticed gst-plugins-bad/tools/gst-element-maker? Also another
way is to e.g. just git clone gst-plugin-bad, create a dir for your
plugin, copy the code for an existing similar enough plugin there and
rename things. In addition, you can also just have a local element in
your application, its just a gobject like any other + one call to
register it to gstreamer.

IMHO thats a lot less trouble than dealing with appsrc and appsink.

Stefan

> Appsrc/appsink lets me build a "normal" application, although there still
> are a lot of obscure bits to be installed, their installation can be largely
> automated with dpkg --set-selections and dpkg --get-selections
>
> I did write a plugin for Avidemux to to be the "front end" for our offline
> processing, since gstreamer can't play the VOB files from a stand-alone
> DVD-R box.
>
> There is no doubt that V4L2 is more of a problem than a solution as its a
> very poor abstraction exposing almost all the weirdness of the underlying
> hardware.  Using libv4l2 helps a lot.
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Get-v4l2src-width-and-height-before-pipeline-runs-tp3942212p3963547.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[wally_bkg]

Stefan Sauer wrote

Have you noticed gst-plugins-bad/tools/gst-element-maker? Also another
way is to e.g. just git clone gst-plugin-bad, create a dir for your
plugin, copy the code for an existing similar enough plugin there and
rename things. In addition, you can also just have a local element in
your application, its just a gobject like any other + one call to
register it to gstreamer.

IMHO thats a lot less trouble than dealing with appsrc and appsink.

Stefan

I haven't looked at (or downloaded) the source for gst-plugins-bad,  as I said I had trouble downloading the plugin template as git protocol is blocked here.  I did evaluate doing it as a plug in on Ubuntu 8.04 when I first started since there was no appsrc/appsink on that system.

I didn't find appsrc/appsink to be any trouble at all, as my entire binocular image processing application using it was only like 950 lines including comments (ignoring code in function call to the actual image processing).  Its mostly gstreamer boilerplate to setup the pipelines which I'd still have to do if I had the image processing in a plugin.  The biggest issue was working around the v4l2src/v4l2 bugs across the different capture hardware we have, and stumbling around in the dark to find settings for adequate quality in ffenc_mpeg4.

I'm not sure the gstreamer developers realize just how powerful the appsink/appsrc model is for using gstreamer as a front end to get a media frame and then again as a back end to view and encode a modified frame.