Generating frames from a paused video?

[Robert Schroll]

Hi all,

I'm trying to put together a simple video player with overlaid controls.
  At the moment, I'm using cairooverlay to draw the controls directly on
the video stream.  When I receive user input, I just update some state
variables so that when the next frame is drawn, it properly reflects the
user's actions by prelighting controls, etc.  This works pretty well
when the video is playing (low frame-rate videos have a bit of lag in
the controls, but I'm willing to live with it), but it means nothing is
updated when the video is paused.  I'm trying to work around this, but
I'm new to gstreamer and don't know which ideas are feasible.  Any
comments you have on which way to go (or if there's an entirely better
way to do this) would be appreciated.

The ideas I've had so far:
1) Find a way to force cairooverlay to emit a frame down the pipeline
when paused.  I could hook up a timer to call this occasionally when the
movie is paused.  But I suspect being paused is an all-or-nothing thing
for the pipeline, I can't have data flowing through only half of it.

2) Find some sort of equivalent of imagefreeze for videos, either as a
source or a filter.  When activated, this would replace the video frames
with a stream of that repeats the last frame over and over.  The
pipeline would always be in the "playing" state, so the controls would
always be updated.  But I don't know if such an element exists.

3) Use a videomixer to combine my desired video and a test source in
such a way that only the former is visible.  (This could also up the
refresh rate for low-frame rate videos.)  But I don't know if I can
pause the sources to a video mixer individually, or what this would do
to seeking.

4) What I'm doing right now is pausing the pipeline and then seeking to
the current position every 50ms.  This triggers a new frame to go down
the pipeline, on which I can draw the updated controls.  This works
seamlessly for some videos, but for others it introduces half-second
jumps on the pause, presumably because the seek is going to a keyframe
or something.  There also seems to be a problem with seeking to the
exact end of a video - I have to back up at least 1ns to get a new
frame, but some videos seem to require more.  In short, this method
seems sort of fragile, but it is the only one that I know how to make work.

So, is there anyway to make (1), (2), or (3) work, or does gstreamer
just not work that way?  Is there a way to improve (4), or are the
problems with it that I'm not seeing yet?  Is there a better way to
overlay controls than a cairooverlay?

As I said, I'm new to gstreamer, so please don't assume I've considered
something that seems obvious to you.  I'm working in Vala, but I'd be
happy with example code in any language.

Thanks in advance,
Robert
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Stefan Sauer]

On 05/20/2012 03:42 AM, Robert Schroll wrote:

Hi all,

I'm trying to put together a simple video player with overlaid controls.  At the moment, I'm using cairooverlay to draw the controls directly on the video stream.  When I receive user input, I just update some state variables so that when the next frame is drawn, it properly reflects the user's actions by prelighting controls, etc.  This works pretty well when the video is playing (low frame-rate videos have a bit of lag in the controls, but I'm willing to live with it), but it means nothing is updated when the video is paused.  I'm trying to work around this, but I'm new to gstreamer and don't know which ideas are feasible.  Any comments you have on which way to go (or if there's an entirely better way to do this) would be appreciated.

This is difficult. When the video is paused, the time is standing still and no updates are pushed to the screen. I wonder if we could do something clever using the recent VideoOverlayComposition api:
http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-libs/html/gst-plugins-base-libs-gstvideooverlaycomposition.html
Something like having a flag to mark them as 'live', so that video-sinks would re-render them even on paused frames.

One general problem with your approach though is, that your UI update rate is linked to the video framerate (but probably okay for you). You could mitigate that using a videorate element in front of the sink to fixate the fps to e.g. 30.

Stefan

The ideas I've had so far:
1) Find a way to force cairooverlay to emit a frame down the pipeline when paused.  I could hook up a timer to call this occasionally when the movie is paused.  But I suspect being paused is an all-or-nothing thing for the pipeline, I can't have data flowing through only half of it.

2) Find some sort of equivalent of imagefreeze for videos, either as a source or a filter.  When activated, this would replace the video frames with a stream of that repeats the last frame over and over.  The pipeline would always be in the "playing" state, so the controls would always be updated.  But I don't know if such an element exists.

3) Use a videomixer to combine my desired video and a test source in such a way that only the former is visible.  (This could also up the refresh rate for low-frame rate videos.)  But I don't know if I can pause the sources to a video mixer individually, or what this would do to seeking.

4) What I'm doing right now is pausing the pipeline and then seeking to the current position every 50ms.  This triggers a new frame to go down the pipeline, on which I can draw the updated controls.  This works seamlessly for some videos, but for others it introduces half-second jumps on the pause, presumably because the seek is going to a keyframe or something.  There also seems to be a problem with seeking to the exact end of a video - I have to back up at least 1ns to get a new frame, but some videos seem to require more.  In short, this method seems sort of fragile, but it is the only one that I know how to make work.

So, is there anyway to make (1), (2), or (3) work, or does gstreamer just not work that way?  Is there a way to improve (4), or are the problems with it that I'm not seeing yet?  Is there a better way to overlay controls than a cairooverlay?

As I said, I'm new to gstreamer, so please don't assume I've considered something that seems obvious to you.  I'm working in Vala, but I'd be happy with example code in any language.

Thanks in advance,
Robert
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim-Philipp Müller-2]

On Sat, 2012-05-19 at 21:42 -0400, Robert Schroll wrote:

Hi,

> I'm trying to put together a simple video player with overlaid controls.
>   At the moment, I'm using cairooverlay to draw the controls directly on
> the video stream.  When I receive user input, I just update some state
> variables so that when the next frame is drawn, it properly reflects the
> user's actions by prelighting controls, etc.  This works pretty well
> when the video is playing (low frame-rate videos have a bit of lag in
> the controls, but I'm willing to live with it), but it means nothing is
> updated when the video is paused.  I'm trying to work around this, but
> I'm new to gstreamer and don't know which ideas are feasible.  Any
> comments you have on which way to go (or if there's an entirely better
> way to do this) would be appreciated.

I think it would be best/easiest if you did the overlay outside of the
GStreamer pipeline. Have you considered using e.g. clutter and
clutter-gst?

Cheers
 -Tim

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Robert Schroll]

On 05/20/2012 04:18 PM, Stefan Sauer wrote:
> This is difficult.

In a way, I'm glad to hear this - I don't feel so dumb for not figuring
it out.

> [...] You could mitigate that using a videorate element in front of
> the sink to fixate the fps to e.g. 30.

If I stick with this solution, I'll check that out.

Thanks for the help,
Robert
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Robert Schroll]

On 05/21/2012 04:54 AM, Tim-Philipp Müller wrote:
> I think it would be best/easiest if you did the overlay outside of the
> GStreamer pipeline. Have you considered using e.g. clutter and
> clutter-gst?

I dismissed this at first, since I'm trying to show the video in only a
part of an existing Gtk.DrawingArea.  But surely there's a way to make
clutter render to that DrawingArea instead.  I will investigate more.

Thanks for the suggestion,
Robert
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel