XOverlay lost after setting playbin state to NULL or READY

[Ralph]

I'm developing a C# WPF application using OSSBuild and gstreamersharp. I'm using XOverlayAdapter to display video on a hosted WindowsForms control to which I have a Windows handle. Everything is working nicely except the situation when I set the playbin2 state to NULL or READY which is required for changing the source or changing the pipeline structure.
This is exactly what I'm doing (I stripped out all checks if the elements have been created correctly):

1. Create a playbin

      Gst.BasePlugins.PlayBin2 playBin = new Gst.BasePlugins.PlayBin2();
      playBin.PlayFlags &= ~((Gst.BasePlugins.PlayBin2.PlayFlagsType)(1 << 2));
      playBin.Bus.AddSignalWatch();
      playBin.Bus.EnableSyncMessageEmission();
      playBin.Bus.Message += new Gst.MessageHandler(OnPlayBinMessage);

2. Create a videosink

      Gst.Video.VideoSink videoSink = Gst.ElementFactory.Make("dshowvideosink") as Gst.Video.VideoSink;
      videoSink["force-aspect-ratio"] = true;

3. Associate the videosink with a WindowsForms control (screen) to display the video in a rectangular area in my WPF window.

      Gst.Interfaces.XOverlayAdapter overlayAdapter = new Gst.Interfaces.XOverlayAdapter(videoSink.Handle);
      overlayAdapter.XwindowId = (ulong)screen.Handle;

4. Attach the videosink to the playbin

      playBin.VideoSink = videoSink;

5. Set playbin's URI to some video file on my hard drive

      playBin.SetState(Gst.State.Null);
      playBin.Uri = @"file:///" + fileName.Replace('\\', '/');
      playBin.SetState(Gst.State.Paused);

6. Now I can play and pause the video by changing the state of the playbin

      playBin.SetState(Gst.State.Playing);
      playBin.SetState(Gst.State.Paused);

So far so good, everything is working smoothly, I can play, pause, seek the video displayed in a rectangular area in my window. The problem starts when I try to play another video file. According to all manuals, I should set the pipeline state to NULL or READY before changing the URI:

7. Open another video file

      playBin.SetState(Gst.State.Ready);
      playBin.Uri = @"file:///" + newFileName.Replace('\\', '/');
      playBin.SetState(Gst.State.Playing);

Unfortunately this causes the rectangular area in my window to become black and a separate video window is created by GStreamer. It has nothing to do with changing the URI, it is setting the playbin state to NULL or READY which disconnects the video output from the overlay and pops up a new window.
What am I doing wrong?

[Nathanael D. Noblet]

On 11/28/2011 06:48 AM, Ralph wrote:

> So far so good, everything is working smoothly, I can play, pause, seek the
> video displayed in a rectangular area in my window. The problem starts when
> I try to play another video file. According to all manuals, I should set the
> pipeline state to Null before changing the URI:

Try setting it to READY instead. NULL dereferences everything and should
be when you are done using the pipeline if I recall correctly.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Ralph]

> Try setting it to READY instead. NULL dereferences everything and should
> be when you are done using the pipeline if I recall correctly.

Unfortunately it doesn't work either, the result is the same - the rectangular display target goes blank and a new window is popped up.

[Stefan Sauer]

On 11/28/2011 03:48 PM, Ralph wrote:

> I'm developing a C# WPF application using OSSBuild and gstreamersharp. I'm
> using XOverlayAdapter to display video on a hosted WindowsForms control to
> which I have a Windows handle. Everything is working nicely except the
> situation when I set the playbin2 state to Null which is required for
> changing the source or changing the pipeline structure.
> This is exactly what I'm doing (I stripped out all checks if the elements
> have been created correctly):
>
> 1. Create a playbin
>
>       Gst.BasePlugins.PlayBin2 playBin = new Gst.BasePlugins.PlayBin2();
>       playBin.PlayFlags &= ~((Gst.BasePlugins.PlayBin2.PlayFlagsType)(1 <<
> 2));
>       playBin.Bus.AddSignalWatch();
>       playBin.Bus.EnableSyncMessageEmission();
>       playBin.Bus.Message += new Gst.MessageHandler(OnPlayBinMessage);
>
> 2. Create a videosink
>
>       Gst.Video.VideoSink videoSink =
> Gst.ElementFactory.Make("dshowvideosink") as Gst.Video.VideoSink;
>       videoSink["force-aspect-ratio"] = true;
>
> 3. Associate the videosink with a WindowsForms control (screen) to display
> the video in a rectangular area in my WPF window.
>
>       Gst.Interfaces.XOverlayAdapter overlayAdapter = new
> Gst.Interfaces.XOverlayAdapter(videoSink.Handle);
>       overlayAdapter.XwindowId = (ulong)screen.Handle;
>
> 4. Attach the videosink to the playbin
>
>       playBin.VideoSink = videoSink;
>
> 5. Set playbin's URI to some video file on my hard drive
>
>       playBin.SetState(Gst.State.Null);
>       playBin.Uri = @"file:///" + fileName.Replace('\\', '/');
>       playBin.SetState(Gst.State.Paused);
>
> 6. Now I can play and pause the video by changing the state of the playbin
>
>       playBin.SetState(Gst.State.Playing);
>       playBin.SetState(Gst.State.Paused);
>
> So far so good, everything is working smoothly, I can play, pause, seek the
> video displayed in a rectangular area in my window. The problem starts when
> I try to play another video file. According to all manuals, I should set the
> pipeline state to Null before changing the URI:
>
> 7. Open another video file
>
>       playBin.SetState(Gst.State.Null);
>       playBin.Uri = @"file:///" + newFileName.Replace('\\', '/');
>       playBin.SetState(Gst.State.Playing);

you could do a set_locked_state(TRUE) on the videosink (to avoid it
going down), then goto READY, change uri, go back to PAUSED,
set_locked_state(FALSE), go to PLAYING

Stefan

> Unfortunately this causes the rectangular area in my window to become black
> and a separate video window is created by GStreamer. It has nothing to do
> with changing the URI, it is setting the playbin state to Null which
> disconnects the video output from the overlay and pops up a new window.
> What am I doing wrong?
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/XOverlay-lost-after-setting-playbin-state-to-Null-tp4115442p4115442.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Ralph]

> you could do a set_locked_state(TRUE) on the videosink (to avoid it
> going down), then goto READY, change uri, go back to PAUSED,
> set_locked_state(FALSE), go to PLAYING

IT WORKS!!!
Fantastic, thank you very much.

[Stefan Sauer]

On 11/28/2011 06:25 PM, Ralph wrote:
>> you could do a set_locked_state(TRUE) on the videosink (to avoid it
>> going down), then goto READY, change uri, go back to PAUSED,
>> set_locked_state(FALSE), go to PLAYING
> IT WORKS!!!
> Fantastic, thank you very much.
Good to know. This could mean that you don't properly handle the
element-message to bind the overlay to the window handle or that the
sink is buggy.

Stefan

> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/XOverlay-lost-after-setting-playbin-state-to-NULL-or-READY-tp4115442p4116023.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[dhoyt]

> Good to know. This could mean that you don't properly handle the
> element-message to bind the overlay to the window handle or that the
> sink is buggy.

Any reason you're not using d3dvideosink?

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Ralph]

dhoyt wrote

> Good to know. This could mean that you don't properly handle the
> element-message to bind the overlay to the window handle or that the
> sink is buggy.

Any reason you're not using d3dvideosink?

Interesting, I had no idea such a sink existed.  Where can I find more information about it, examples, etc.?

[David Hoyt]

> Interesting, I had no idea such a sink existed.  Where can I find more
> information about it, examples, etc.?

Our discussion thread about it is here:
http://code.google.com/p/ossbuild/issues/detail?id=57

The source is available here:
http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/sys/d3dvideosink/
d3dvideosink.c

Just replace dshowvideosink with d3dvideosink. If you're using OSSBuild,
autovideosink should select d3dvideosink by default.

d3dvideosink was created because the direct show one locked up and crashed
all the time for me and seemed generally unusable for any serious use.
Direct show has a media graph/pipeline similar to gstreamer (the 2 are
somewhat analogous) and it also seemed redundant to create a 2nd media
pipeline and push buffers through it simply for display. Direct show, I'd
guess (but I don't *know*), uses direct 3d under the covers. So it made
sense to skip the middle man. dshowvideosink, I believe, does bypass most of
the direct show graph, so that certainly helps.

Another plus is that there's direct 3d headers now for gcc through mingw.org
or mingw-w64. As far as I know, you still need to have visual c++ to compile
the direct show elements because they need to use the direct show base
classes. So you can compile d3dvideosink using gcc (and others have), but
not dshowvideosink.

But I've been running d3dvideosink continuously now for months and months
without a problem. Seems to be pretty stable. But then again, I wrote most
of it. So I'm biased. :D



_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Ralph]

David Hoyt-2 wrote

> Interesting, I had no idea such a sink existed.  Where can I find more
> information about it, examples, etc.?

Our discussion thread about it is here:
http://code.google.com/p/ossbuild/issues/detail?id=57

The source is available here:
http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/sys/d3dvideosink/
d3dvideosink.c

Just replace dshowvideosink with d3dvideosink. If you're using OSSBuild,
autovideosink should select d3dvideosink by default.

d3dvideosink was created because the direct show one locked up and crashed
all the time for me and seemed generally unusable for any serious use.
Direct show has a media graph/pipeline similar to gstreamer (the 2 are
somewhat analogous) and it also seemed redundant to create a 2nd media
pipeline and push buffers through it simply for display. Direct show, I'd
guess (but I don't *know*), uses direct 3d under the covers. So it made
sense to skip the middle man. dshowvideosink, I believe, does bypass most of
the direct show graph, so that certainly helps.

Another plus is that there's direct 3d headers now for gcc through mingw.org
or mingw-w64. As far as I know, you still need to have visual c++ to compile
the direct show elements because they need to use the direct show base
classes. So you can compile d3dvideosink using gcc (and others have), but
not dshowvideosink.

But I've been running d3dvideosink continuously now for months and months
without a problem. Seems to be pretty stable. But then again, I wrote most
of it. So I'm biased. :D

Cheers, Dave, I will give it a try as soon as I solve my problem with signals.  My software is going to display 1-4 videos simultaneously and will be run on netbooks and tablets which are pretty slow and even tiniest performance gains are very important to me.

[dhoyt]

> Cheers, Dave, I will give it a try as soon as I solve my problem with
> signals.  My software is going to display 1-4 videos simultaneously and will
> be run on netbooks and tablets which are pretty slow and even tiniest
> performance gains are very important to me.

If it's any comfort, I've been running d3dvideosink in software that runs 16+ videos simultaneously. Most running it don't use that many at once, but the software supports it. Most do 5-10. But again, it's been running for months with no problems. But you should probably be using autovideosink so gstreamer can choose the best sink to use. If you use autovideosink and d3dvideosink isn't able to initialize for some reason, it should fall back to dshowvideosink.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Ralph]

dhoyt wrote

If it's any comfort, I've been running d3dvideosink in software that runs 16+ videos simultaneously. Most running it don't use that many at once, but the software supports it. Most do 5-10. But again, it's been running for months with no problems. But you should probably be using autovideosink so gstreamer can choose the best sink to use. If you use autovideosink and d3dvideosink isn't able to initialize for some reason, it should fall back to dshowvideosink.

WOW!!! I've just tested d3dvideosink and I am really impressed - it's FAST!!!  It is 24-32% faster than dshowvideosink depending on the machine.  Good job, Dave, really well done!

I have two problems though:

1. I have a video file with 5:4 aspect ratio (720x576), but d3dvideosink stretches it to 4:3.  It does not stretch the video to fill the available space, it deliberately changes the aspect ratio to 4:3 leaving black borders on top and bottom.  This is a sample 10-second video file: http://ralphos.com/video/test1.avi.

2. I cannot get it to work with a full screen window on my primary screen (1920x1200), it displays black surface.  It works without any problems if I place the full screen window on my secondary screen(1680x1050).  It is the same instance of a window, just hidden, repositioned and shown on another screen, I have no problems with it using other video sinks.  Is it a bug or am I doing something wrong?

[dhoyt]

> WOW!!! I've just tested d3dvideosink and I am really impressed - it's
> FAST!!!  It is 24-32% faster than dshowvideosink depending on the machine.
> Good job, Dave, really well done!

Glad to hear that!

> ...It does not stretch the video to fill the available
> space, it deliberately changes the aspect ratio to 4:3 leaving black borders
> on top and bottom.  This is a sample 10-second video file:
> http://ralphos.com/video/test1.avi.

Try the following pipelines:

gst-launch playbin2 uri=http://ralphos.com/video/test1.avi video-sink=dshowvideosink

gst-launch playbin2 uri=http://ralphos.com/video/test1.avi video-sink=d3dvideosink

gst-launch playbin2 uri=http://ralphos.com/video/test1.avi video-sink="d3dvideosink force-aspect-ratio=true"

Both those seem to be exactly alike. I see no black bars added to the video. Now the video *itself* has black bars on the side and bottom -- but that's got nothing to do with the video sink. Both behave exactly the same. d3dvideosink does have a force-aspect-ratio property you can use, but by default it should stretch to the window it's in (in each one of those example pipelines, try resizing the window during playback). You might want to examine your application's code and see if it's not doing something to cause the problem you're seeing.

> 2. I cannot get it to work with a full screen window on my primary screen
> (1920x1200), it displays black surface.  It works without any problems if I
> place the full screen window on my secondary screen(1680x1050).  It is the
> same instance of a window, just hidden, repositioned and shown on another
> screen, I have no problems with it using other video sinks.  Is it a bug or
> am I doing something wrong?

Sounds like an issue either with the direct 3d driver for your card or your application. How are you doing "full screen"? I don't recommend using a direct x style full screen -- instead create a normal window that fills the screen and use xoverlay to place the video in the appropriate window. My app takes over every monitor attached to a system -- so that's typically 2, 3, or 4 at a time w/ 4-5 videos running on each screen and never had a problem. It's likely your app that's the issue. I can't rule out a bug, but I can point to very similar scenarios where it's known to be working...so it's more unlikely it's on the d3dvideosink end. However, since it's working with other video sinks, it's definitely suspect. But, directdrawsink uses an older technology and direct show, if my memory serves me right, will fall back to direct draw if necessary.

What OS are you running this on? If it's Win XP, try re-running your app w/ the same setup but in Win 7 and see how well it does. It's my experience that manufacturer direct 3d drivers seem to be better w/ Win 7.

If it's still a problem, we can try and work on it to see what's causing the problem.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Ralph]

dhoyt wrote

Try the following pipelines:

gst-launch playbin2 uri=http://ralphos.com/video/test1.avi video-sink=dshowvideosink

gst-launch playbin2 uri=http://ralphos.com/video/test1.avi video-sink=d3dvideosink

gst-launch playbin2 uri=http://ralphos.com/video/test1.avi video-sink="d3dvideosink force-aspect-ratio=true"

Both those seem to be exactly alike. I see no black bars added to the video. Now the video *itself* has black bars on the side and bottom -- but that's got nothing to do with the video sink. Both behave exactly the same. d3dvideosink does have a force-aspect-ratio property you can use, but by default it should stretch to the window it's in (in each one of those example pipelines, try resizing the window during playback).

The dshowvideosink pipeline produces 5:4 video, and when I set force-aspect-ratio to true, the window can be resized and the aspect ratio of the video doesn't change.
Both d3dvideosink pipelines produce 4:3 video, the second one forces the wrong aspect ratio when I resize the window.  There is definitely something wrong with d3dvideosink on my machine.  If I could only zip or rar my machine and send it to you... ;-)

dhoyt wrote

 You might want to examine your application's code and see if it's not doing something to cause the problem you're seeing.

I turned off absolutely everything that might have any influence.  Still no change...

dhoyt wrote

Sounds like an issue either with the direct 3d driver for your card or your application. How are you doing "full screen"? I don't recommend using a direct x style full screen -- instead create a normal window that fills the screen and use xoverlay to place the video in the appropriate window. My app takes over every monitor attached to a system -- so that's typically 2, 3, or 4 at a time w/ 4-5 videos running on each screen and never had a problem. It's likely your app that's the issue. I can't rule out a bug, but I can point to very similar scenarios where it's known to be working...so it's more unlikely it's on the d3dvideosink end. However, since it's working with other video sinks, it's definitely suspect. But, directdrawsink uses an older technology and direct show, if my memory serves me right, will fall back to direct draw if necessary.

I'm doing it as you say - I create a borderless window and make it full screen.  I've just discovered that if I display the window in Normal state and resize it to fill the entire screen, then there is no problem.  It is setting the window state to FullScreen that causes problems.  Fortunately there is a workaround, but it is interesting why FullScreen state has any influence on displaying video, xoverlay gets the same handle to the same window.

dhoyt wrote

What OS are you running this on? If it's Win XP, try re-running your app w/ the same setup but in Win 7 and see how well it does. It's my experience that manufacturer direct 3d drivers seem to be better w/ Win 7.

I'm testing d3dvideosink with same results on two different machines:

1. Highly customised Dell PC
System: Windows7 Ultimate 64-bit SP1.
Graphics: ATI Radeon HD 5670.

2. Advent 4211-B netbook
System: Windows 7 Professional 32-bit
Graphics: Mobile Intel 945 Express

[Ralph]

Ralph wrote

dhoyt wrote

Sounds like an issue either with the direct 3d driver for your card or your application. How are you doing "full screen"? I don't recommend using a direct x style full screen -- instead create a normal window that fills the screen and use xoverlay to place the video in the appropriate window. My app takes over every monitor attached to a system -- so that's typically 2, 3, or 4 at a time w/ 4-5 videos running on each screen and never had a problem. It's likely your app that's the issue. I can't rule out a bug, but I can point to very similar scenarios where it's known to be working...so it's more unlikely it's on the d3dvideosink end. However, since it's working with other video sinks, it's definitely suspect. But, directdrawsink uses an older technology and direct show, if my memory serves me right, will fall back to direct draw if necessary.

I'm doing it as you say - I create a borderless window and make it full screen.  I've just discovered that if I display the window in Normal state and resize it to fill the entire screen, then there is no problem.  It is setting the window state to FullScreen that causes problems.  Fortunately there is a workaround, but it is interesting why FullScreen state has any influence on displaying video, xoverlay gets the same handle to the same window.

I wrote it too fast.  I don't know how I did it, but d3dvideosink doesn't work for Normal state windows stretched to cover the entire screen.  Maybe I used dshowvideosink by mistake?  The behavior is very interesting:

My screen resolution is 1920x1200, I create a WPF Window and associate it with the video sink using XOverlay:
        Gst.Interfaces.XOverlayAdapter overlayAdapter = new Gst.Interfaces.XOverlayAdapter(videoSink.Handle);
        overlayAdapter.XwindowId = (ulong)window.Handle;
- When I display the window in FullScreen state, no video is displayed.  We already know that.
- When I display the window in Normal state, but stretch it to cover the entire screen, no video is displayed either.
- When I display the window in Normal state and change its height to 1148 or less, the video is displayed correctly
Increasing window height to anything more than (screenHeight - 52) prevents the video from being displayed.  The limitation applies to height only, there are no problems using any width.

[Ralph]

Stefan Sauer wrote

On 11/28/2011 06:25 PM, Ralph wrote:
>> you could do a set_locked_state(TRUE) on the videosink (to avoid it
>> going down), then goto READY, change uri, go back to PAUSED,
>> set_locked_state(FALSE), go to PLAYING
> IT WORKS!!!
> Fantastic, thank you very much.
Good to know. This could mean that you don't properly handle the
element-message to bind the overlay to the window handle or that the
sink is buggy.
Stefan

Well... The solution with locking the videosink is not perfect, sometimes it works, but in some situations it freezes the whole application when changing the playbin state from PAUSED to READY.  This always happens when I try to preview a sequence of files very quickly, read duration of each file, etc.  When I use locking, and switch between the files too fast, it freezes the entire application.  Changing the file once every 10 seconds usually works, but not always.  I have tested it with both dshowvideosink and d3dvideosink and when I don't use videosink locking, a new video window is opened.
This is how I bind the window handle to the video sink:
      Gst.Interfaces.XOverlayAdapter overlayAdapter = new Gst.Interfaces.XOverlayAdapter(videoSink.Handle);
      overlayAdapter.XwindowId = (ulong)window.Handle;
I think that's the standard way of doing this, are ther any others?  Avoiding to lock the videosink would solve lots of problems.

[dhoyt]

> This is how I bind the window handle to the video sink:
>       Gst.Interfaces.XOverlayAdapter overlayAdapter = new
> Gst.Interfaces.XOverlayAdapter(videoSink.Handle);
>       overlayAdapter.XwindowId = (ulong)window.Handle;
> I think that's the standard way of doing this, are ther any others?
> Avoiding to lock the videosink would solve lots of problems.

WPF itself uses direct 3d, I believe (http://stackoverflow.com/questions/1373195/does-wpf-rely-on-direct3d-calls-even-for-standard-controls). So you may be experiencing issues as a result of that -- it may be trying to initialize and use a different version of direct 3d or the interaction between 2 differently managed direct 3d stacks, so to speak, could be giving rise to the problems you're seeing.

You might want to look at how to mix external direct 3d code with WPF. That might require changes to d3dvideosink or perhaps necessitate a WPF video sink.


_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Ralph]

Ralph wrote

1. I have a video file with 5:4 aspect ratio (720x576), but d3dvideosink stretches it to 4:3.  It does not stretch the video to fill the available space, it deliberately changes the aspect ratio to 4:3 leaving black borders on top and bottom.  This is a sample 10-second video file: http://ralphos.com/video/test1.avi.

One of the problems solved!
It was not a bug, it was an important feature!  When I looked at the properties of my video file using GSpot, I noticed that although the pixel aspect ratio was 1.25 (5:4), the reported display aspect ratio was 1.364 (15:11).  This means that the video was intended to be stretched and the information about it was embedded in the video file.  It seems that d3dvideosink is more clever than I thought and reads the display aspect ratio from the video file and resizes the image accordingly.

One more good reason for using d3dvideosink :)