Using GStreamer for a game emulation pipeline (and many other questions)

[athairus]

Hi everyone! I'm part of an open-source project called Phoenix (phoenix.vg). I've talked to people on IRC (#gstreamer) around two months ago about this idea of mine but I thought I'd post here to get a more formal conversation going.

Phoenix is a multi-system emulator for classic game consoles. It's powered by Qt 5 and Libretro (an API for emulators, in Libretro-speak they're called cores) and written in C++, QML and a bit of Objective-C and Python.

No need to get into too much detail here, but I'm planning on a massive variety of features in the future which behooves me to make it as easy as possible to play around with new ideas. One way to make this possible is to have a flexible pipeline system. During my research this project came on my radar early on along with Microsoft's Media Foundation, though it was obvious which was the more open-source friendly choice!

As I got further into developing my own custom pipeline system I started to get lost in its complexity. I was starting to do metaprogramming in C++, mostly to get elements to register themselves at compile time without having to explicitly register themselves in the pipeline manager. At this point I decided that I should stop and check out existing solutions to see if I could accomplish what I wanted much faster. Right now, the only actively maintained choice is GStreamer so here I am!

The way I see it, running a game emulator isn't too far removed from playing video. You have a stream of audio/video data that's been "decoded" and presented to the user. In the case of games you'll also have input to deal with. 

The plan is to develop a series of new elements and include them with each release. They're so domain-specific that I don't think it's worth getting them included into any of the gst-plugins-* repos. I imagine I'll either place the shared libraries near the executable or statically link them. Is the latter possible? What considerations should I have for the former?

I've noticed that there's a project called gstreamer-qt that's no longer maintained. What are my options for getting video output into the QML scene?

My project uses CMake and GStreamer uses automake. Is there any work being done on compiling GStreamer with CMake? Official FindGStreamer.cmake modules?

Lastly, I have the issue of input. I'm going to use SDL2 to manage connected controllers. I need to get controller input polled each frame before the Libretro element can emulate a frame. What approaches do I have? Another element before the Libretro element? I was told a while ago on IRC that I could maybe use something meant for DVD menu navigation? What is that?

Thanks!

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Sat, 2016-09-03 at 17:23 -0400, athairus wrote:

Hi,

Cool stuff!

> I've noticed that there's a project called gstreamer-qt that's no
> longer maintained. What are my options for getting video output into
> the QML scene?

There's a qmlglsink video sink in gst-plugins-bad that is based on the
GStreamer OpenGL library (libgstlgl).

Example how to use it here:
https://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/tests/examp
les/qt/qmlsink


> My project uses CMake and GStreamer uses automake. Is there any work
> being done on compiling GStreamer with CMake? Official
> FindGStreamer.cmake modules?

Nope. There's work being done on compiling GStreamer with Meson (mesonb
uild.com), but not CMake. We provide pkg-config files.


> Lastly, I have the issue of input. I'm going to use SDL2 to manage
> connected controllers. I need to get controller input polled each
> frame before the Libretro element can emulate a frame. What
> approaches do I have? Another element before the Libretro element? I
> was told a while ago on IRC that I could maybe use something meant
> for DVD menu navigation? What is that?

There's something called the GstNavigation interface in gst-plugins-
base as part of the video library. It allows applications (or video
sinks) to make elements in the pipeline aware of things such as mouse
pointer movements, clicks or key presses, but it's pretty much limited
to that. That way an element such as dvdnavsrc and friends can be
notified of these things and can select the right DVD menu items to
highlight/select and such. If that's suitable for your use case, I
don't know.

Minimal example:  gst-launch-1.0 videotestsrc ! navigationtest !
videoconvert ! ximagesink

Cheers
 -Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com

Join us at the GStreamer Conference: 10-11 October 2016 in Berlin,
Germany
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel