Evaluating use of GStreamer

[bdiesel]

Good Day All,

I am working on a project that required embedding data received from sensors
on serial and I2C connections on a video, using a Raspberry Pi.  I
originally wrote a C++ application using OpenCV to do this, but after
looking at GStreamer, it appears to be more suited to my requirements.

I have spent a day trying to understand how to use the text overlay, but
have come to the conclusion that this requires timing information in order
to embed changing text (using the subtitle mechanism).  I have also tried
numerous work-arounds e.g. piping serial data to a udp port, piping to a
text file and tailing it etc, but have not been successful.

So, I am trying to decide whether I need to spend the time learning how to
write a plugin that takes data from a serial port, and from an I2C stream,
and embed it.  Looking at the learning curve, (GObject, glib, GStreamer etc)
I'm not convinced that I should just continue with my original C++ OpenCV
implementation.

The application that I need to do is as follows:
- Run a rolling window of 30 sec of video data captured from the camera.
- Embed data received over a serial port onto each video frame.
- Monitor a sensor on an I2C interface.  This data also needs to be embedded
on the video.
- When the value of the data on the I2C interface goes below a certain
threshold, trigger the recording of the video buffer, and continue recording
until 30sec after the trigger event.

In short, I am wanting to record the 30sec prior to a trigger event, along
with the 30sec after, whilst embedding text on the video.

I have a number of questions:
1. Is this achievable using standard unix pipes and standard GStreamer
plugins such as fdsrc and text overlay etc.
2. If not, Is it a significant effort to develop my own plugins for this?
It appears that a plugin that can embed sensor data from a communication
port (udp, serial, I2C) whould be quite useful.  

I am happy to invest the time learning the toolkit, but I am concerned that
I would be biting off a lot.  I have plenty years C/C++ technical
development experience.

Apologies if such plugins already exist, I have spent numerous hours
searching for a solution.

Kind Regards,
  Bruce



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Martin Vachovski]

Hello Bruce,

If I understood correctly your use case, the textoverlay is the easy part:
Here is an example from the element's documentation, which overlays some text over the videoframes:

gst-launch-1.0 -v videotestsrc ! textoverlay text="Room A" valignment=top halignment=left font-desc="Sans, 72" ! autovideosink
The parameters like "text" are read/write so, you could change them during the pipeline's PLAYING state as a response to your app's business logic

I am not sure about the cyclic buffer:
If that's an option maybe you could consider post-processing of already stored video- you can compute the correct time if you know the
number of frames already passed and the frame rate or if you look into a GstBuffer's structure timestamp
But maybe somebody else would know a better solution

Cheers
Martin


________________________________________
From: gstreamer-devel <[hidden email]> on behalf of bdiesel <[hidden email]>
Sent: Tuesday, December 19, 2017 11:03 AM
To: [hidden email]
Subject: Evaluating use of GStreamer

Good Day All,

I am working on a project that required embedding data received from sensors
on serial and I2C connections on a video, using a Raspberry Pi.  I
originally wrote a C++ application using OpenCV to do this, but after
looking at GStreamer, it appears to be more suited to my requirements.

I have spent a day trying to understand how to use the text overlay, but
have come to the conclusion that this requires timing information in order
to embed changing text (using the subtitle mechanism).  I have also tried
numerous work-arounds e.g. piping serial data to a udp port, piping to a
text file and tailing it etc, but have not been successful.

So, I am trying to decide whether I need to spend the time learning how to
write a plugin that takes data from a serial port, and from an I2C stream,
and embed it.  Looking at the learning curve, (GObject, glib, GStreamer etc)
I'm not convinced that I should just continue with my original C++ OpenCV
implementation.

The application that I need to do is as follows:
- Run a rolling window of 30 sec of video data captured from the camera.
- Embed data received over a serial port onto each video frame.
- Monitor a sensor on an I2C interface.  This data also needs to be embedded
on the video.
- When the value of the data on the I2C interface goes below a certain
threshold, trigger the recording of the video buffer, and continue recording
until 30sec after the trigger event.

In short, I am wanting to record the 30sec prior to a trigger event, along
with the 30sec after, whilst embedding text on the video.

I have a number of questions:
1. Is this achievable using standard unix pipes and standard GStreamer
plugins such as fdsrc and text overlay etc.
2. If not, Is it a significant effort to develop my own plugins for this?
It appears that a plugin that can embed sensor data from a communication
port (udp, serial, I2C) whould be quite useful.

I am happy to invest the time learning the toolkit, but I am concerned that
I would be biting off a lot.  I have plenty years C/C++ technical
development experience.

Apologies if such plugins already exist, I have spent numerous hours
searching for a solution.

Kind Regards,
  Bruce



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Tue, 2017-12-19 at 04:03 -0700, bdiesel wrote:

Hi Bruce,

> I have spent a day trying to understand how to use the text overlay,
> but have come to the conclusion that this requires timing information
> in order to embed changing text (using the subtitle mechanism).  I
> have also tried numerous work-arounds e.g. piping serial data to a
> udp port, piping to a text file and tailing it etc, but have not been
> successful.

Not sure if I understand you correctly here. There are multiple ways to
make textoverlay show text on top of video. One is by feeding it timed
text as input on the text sink pad (the "subtitle way" if you like),
the other is to simply set its "text" property via g_object_set(). You
can do this at run-time from your application thread, in which case
it's up to your application when to change it / set it.

> The application that I need to do is as follows:
> - Run a rolling window of 30 sec of video data captured from the
> camera.
> - Embed data received over a serial port onto each video frame.
> - Monitor a sensor on an I2C interface.  This data also needs to be
> embedded on the video.
> - When the value of the data on the I2C interface goes below a
> certain threshold, trigger the recording of the video buffer, and
> continue recording until 30sec after the trigger event.
>
> In short, I am wanting to record the 30sec prior to a trigger event,
> along with the 30sec after, whilst embedding text on the video.
>
> I have a number of questions:
> 1. Is this achievable using standard unix pipes and standard
> GStreamer plugins such as fdsrc and text overlay etc.

Yes, that's achievable. I'm not sure whether you really need to grab
the data via fdsrc and pipes though, I would rather do this in your
application and then change the text property on textoverlay.

You can find some code here to start/stop recording with a backlog:

 https://people.freedesktop.org/~tpm/code/test-backlog-recording-h264.c

It's a bit more complicated than it needs to be, to make sure we start
on a keyframe and because it works around a bug/missing feature in
older versions of GStreamer.


> 2. If not, Is it a significant effort to develop my own plugins for
> this?  It appears that a plugin that can embed sensor data from a
> communication port (udp, serial, I2C) whould be quite useful.

Well. It's not really a lot of lines of code, but there's a bit of a
learning curve (GObject, GStreamer etc.). I'm not sure you need to do
this if you grab the sensor data in your app.

Cheers
 -Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[bdiesel]

Hi Martin,

Thanks for your response.  I had been trying to do this in a command line.
But working through the tutorials, I am getting a better understanding of
how to work with gstreamer programmatically which seems like the best was to
change the text on the fly.

With regard to buffering - I was thinking along the lines of creating a
queue that is 30 seconds long, then start recording from that queue when I
receive the trigger, and stop recording 60 seconds later.

Bruce



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[bdiesel]

Hi Tim,


> Not sure if I understand you correctly here. There are multiple ways to
> make textoverlay show text on top of video. One is by feeding it timed
> text as input on the text sink pad (the "subtitle way" if you like),
> the other is to simply set its "text" property via g_object_set(). You
> can do this at run-time from your application thread, in which case
> it's up to your application when to change it / set it.

Getting to grips with driving gstreamer programmatically now instead of the
command line, so I will hopefully be able to o this quite easily


> Yes, that's achievable. I'm not sure whether you really need to grab
> the data via fdsrc and pipes though, I would rather do this in your
> application and then change the text property on textoverlay.

Agreed, my unix piping skills are not quite up to scratch, will do it
programmatically.

> You can find some code here to start/stop recording with a backlog:

 > https://people.freedesktop.org/~tpm/code/test-backlog-recording-h264.c

Thanks for the reference, I will take a look

> Well. It's not really a lot of lines of code, but there's a bit of a
> learning curve (GObject, GStreamer etc.). I'm not sure you need to do
> this if you grab the sensor data in your app.

Agreed, getting a much better feel for driving gstreamer programmatically
without necessarily having to write plugins.

Thanks for the help!

Bruce



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[bdiesel]

Okay, had a look at the backlog recorder reference you gave me.  This looks
very much like what I need.  Thanks.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel