GStreamer-devel

Video processing(record\play) real time from camera with metadata

Classic

List

Threaded

13 messages Options

Mikl

Video processing(record\play) real time from camera with metadata

Hello

I am new in GStreamer and image processing. Sorry for stupid questions.

Currently, i am developing commercial application for recording video stream from IP camera in real time.
Also i need to add information to some of the frames. It will be like image metadata. For example, time of capturing, location, comment,... And of course i need to be able to read it later. As i understand, i need to use subtitles or it is another way exist? But in any way want to keep it inside of video file.

I already tried OpenCV. Not really happy with it because of codecs selection and no way to work with subtitles. You can find my questions here.
http://answers.opencv.org/question/100967/codecs-list/
http://answers.opencv.org/question/101044/add-subtitles-to-video/

Can you help me please to find a way to do it in GStreamer. Or it is a wrong tool for my task?

Thank you.

Tim Müller

Re: Video processing(record\play) real time from camera with metadata

On Wed, 2016-08-31 at 01:16 -0700, Mikl wrote:

Hi Mikl,

> Currently, i am developing commercial application for recording video
> stream from IP camera in real time.
> Also i need to add information to some of the frames. It will be like
> image metadata. For example, time of capturing, location, comment,...
> And of course i need to be able to read it later. As i understand, i
> need to use subtitles or it is another way exist? But in any way want
> to keep it inside of video file.
>
> I already tried OpenCV. Not really happy with it because of codecs
> selection
> and no way to work with subtitles. You can find my questions here.
> http://answers.opencv.org/question/100967/codecs-list/
> http://answers.opencv.org/question/101044/add-subtitles-to-video/
>
> Can you help me please to find a way to do it in GStreamer. Or it is
> a wrong tool for my task?

It all depends on your exact requirements (e.g. is it ok to decoder +
re-encode the video or do you want to pass through and save the encoded
video stream as-is?) and your work flow.

Capturing audio/video from an IP cam, and saving it to a container
format is not a big problem. Adding a text-based subtitle track of some
sort should also not be very doable (without decoding/re-encoding the
video).

Also depends on how you want to process the files later, and with which
applications, if it's all just for internal use or not.

There's KLV metadata for more "structured", standardised, less freeform
metadata. We only support that in MPEG-TS currently.

Cheers
-Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com

Join us at the GStreamer Conference: 10-11 October 2016 in Berlin,
Germany
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Mikl

Re: Video processing(record\play) real time from camera with metadata

Hello, Tim

Thank you for the answer.

I will try to give you more information, so may be you can help me.

Don`t hesitate to ask more, if something is unclear.

-Coding is a time related question. I need to capture 20 fps and more. It means i have 50 ms to capture, code and record. Not much even on fast machine. But, record raw data also not ideal. I have 1Gb every 50 sec from 20 frames of RGB 800x400.

-Decoding is not real time process. It will be handy for me to have all necessary information inside of one container.

-You want to say, what i can not add subtitles to the video using gstreamer? Is ANY way exist?

-In ideal situation i need to have video file with audio (this is for later) and subtitles. An i need to be able to pay it in a standard video player. As example of user i can give you TV broadcasting system. Best if i can stream it while recording.

It is an ideal goal. Do not focus too much and do not block yourself, please, if it is impossible. I need to be as close as possible.

Best regards, Michael.

From: Tim Müller [via GStreamer-devel] <ml-node+[hidden email]>
Sent: Sunday, September 4, 2016 10:29:56 PM
To: Michael Yarochkin
Subject: Re: Video processing(record\play) real time from camera with metadata

On Wed, 2016-08-31 at 01:16 -0700, Mikl wrote:

Hi Mikl,

If you reply to this email, your message will be added to the discussion below:

http://gstreamer-devel.966125.n4.nabble.com/Video-processing-record-play-real-time-from-camera-with-metadata-tp4679305p4679413.html

To unsubscribe from Video processing(record\play) real time from camera with metadata, click here.
NAML

Tim Müller

Re: Video processing(record\play) real time from camera with metadata

On Mon, 2016-09-05 at 00:20 -0700, Mikl wrote:

Hi,

> -You want to say, what i can not add subtitles to the video using
> gstreamer? Is ANY way exist?

That's not what I said. You *can* add subtitles with GStreamer, either
as a separate subtitle stream in the container or blended on top of the
actual video, as you prefer.

Cheers
-Tim
--
Tim Müller, Centricular Ltd - http://www.centricular.com

Join us at the GStreamer Conference: 10-11 October 2016 in Berlin,
Germany
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Mikl

Re: Video processing(record\play) real time from camera with metadata

Hi,

Thank you for fast answer.

My question now is HOW to do this. :)

Do you have also something to say about other information, i provided to you?

Cheers, Mikl

From: Tim Müller [via GStreamer-devel] <ml-node+[hidden email]>
Sent: Monday, September 5, 2016 9:50:29 AM
To: Michael Yarochkin
Subject: Re: Video processing(record\play) real time from camera with metadata

If you reply to this email, your message will be added to the discussion below:

http://gstreamer-devel.966125.n4.nabble.com/Video-processing-record-play-real-time-from-camera-with-metadata-tp4679305p4679416.html

To unsubscribe from Video processing(record\play) real time from camera with metadata, click here.
NAML

Tim Müller

Re: Video processing(record\play) real time from camera with metadata

On Mon, 2016-09-05 at 01:08 -0700, Mikl wrote:

Hi,

My question now is HOW to do this. :)

Well, you'll need to learn a bit of GStreamer, and again it depends on what you want to do exactly. You are a bit light on the details. Maybe start reading the GStreamer application development manual. How to capture from your IP camera, depends on the camera. If it offers rtsp you can use the 'rtspsrc' element, otherwise you might need other elements. Start capturing data, then depayload and remux to a matroska/mpeg-ts/whatever file. Then you can add a subtitle track with the 'appsrc' element, by pushing data into the pipeline from your app.

Do you have also something to say about other information, i provided to you?

Not really. I don't know what to say about that. Were there any questions? Nothing seemed problematic.

Cheers

-Tim

--

Tim Müller, Centricular Ltd - http://www.centricular.com

Join us at the GStreamer Conference: 10-11 October 2016 in Berlin, Germany

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Mikl

Re: Video processing(record\play) real time from camera with metadata

HI,

-Well, i am reading about GStreamer only 4 days. Learn a lot. Have some lines of code already. But it is still difficult for me.

-I don`t need to connect to camera. My data is already RAW RGB buffer in memory (char*).

-Maybe you can help me to construct pipeline initialization string. And i will start learning by making implementation.

My system is:

Source

-source video RGB24 RAW data by char* (800x400), 20 fps

-source audio not exist now, but want to prepare the structure

-source subtitles is text

Fast compression, if needed and possible

Probably i will need 'mixer'

Output

-local file in mkv(best i found for now) format

Need more details?

Thank you in advance.

From: Tim Müller [via GStreamer-devel] <ml-node+[hidden email]>
Sent: Monday, September 5, 2016 2:35:36 PM
To: Michael Yarochkin
Subject: Re: Video processing(record\play) real time from camera with metadata

On Mon, 2016-09-05 at 01:08 -0700, Mikl wrote:

Hi,

My question now is HOW to do this. :)

Do you have also something to say about other information, i provided to you?

Not really. I don't know what to say about that. Were there any questions? Nothing seemed problematic.

Cheers

-Tim

--

Tim Müller, Centricular Ltd - http://www.centricular.com

Join us at the GStreamer Conference: 10-11 October 2016 in Berlin, Germany

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

If you reply to this email, your message will be added to the discussion below:

http://gstreamer-devel.966125.n4.nabble.com/Video-processing-record-play-real-time-from-camera-with-metadata-tp4679305p4679423.html

To unsubscribe from Video processing(record\play) real time from camera with metadata, click here.
NAML

Mikl

Re: Video processing(record\play) real time from camera with metadata

In reply to this post by Tim Müller

Hello, Tim

I am still working on make it work.

Found already some answers

-i need to use MKV as a container

Also i found a good way to illustrate pipeline.

Using this i did some drawings of what i want to do.

Hope you can help me this time to complete my pipeline.

Maybe i also can help you in something.

Thank you in advance.

Best regards, Michael

On Mon, 2016-09-05 at 01:08 -0700, Mikl wrote:

Hi,

My question now is HOW to do this. :)

Do you have also something to say about other information, i provided to you?

Not really. I don't know what to say about that. Were there any questions? Nothing seemed problematic.

Cheers

-Tim

--

Tim Müller, Centricular Ltd - http://www.centricular.com

Join us at the GStreamer Conference: 10-11 October 2016 in Berlin, Germany

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

If you reply to this email, your message will be added to the discussion below:

http://gstreamer-devel.966125.n4.nabble.com/Video-processing-record-play-real-time-from-camera-with-metadata-tp4679305p4679423.html

To unsubscribe from Video processing(record\play) real time from camera with metadata, click here.
NAML

example.jpg (334K) Download Attachment

TT GStreamer pipeline.png (40K) Download Attachment

Mikl

Re: Video processing(record\play) real time from camera with metadata

In reply to this post by Tim Müller

Hello, Tim

Finally i understand how everything is working!

Now i know basic idea of how and why make pipeline like this.

Next step is make it work.

Can you help me please?

For the moment i am doing only video and from test source.

Here is my pipeline

videotestsrc ! video/x-raw,format=RGB,width=800,height=400,framerate=24/1 ! matroskamux ! filesink location=c:\gstreamer\test.mkv

I have file. It have size (big one). I can see structure using MKVToolNix.

videotestsrc will be replaced by appsrc. And i will not need capsfilter anymore.

But:

-i can not play it

-see no information

Even when i did simple pipeline

videotestsrc ! avimux ! filesink location=c:\gstreamer\test.avi

i was not able to see information about video, navigate in video.

And it was opened not in all players.

Also sending my source code. Maybe it will help you. And you will sea what i am busy and not only asking for help. :)

Can you helm me please what i am doing wrong?

Best regards, Michael

On Mon, 2016-09-05 at 01:08 -0700, Mikl wrote:

Hi,

My question now is HOW to do this. :)

Do you have also something to say about other information, i provided to you?

Not really. I don't know what to say about that. Were there any questions? Nothing seemed problematic.

Cheers

-Tim

--

Tim Müller, Centricular Ltd - http://www.centricular.com

Centricular · GStreamer, Multimedia and Graphics Expertise ...

www.centricular.com

Centricular home page ... GStreamer, Multimedia and Graphics Expertise. We can help you with anything in and around GStreamer, cross-platform multimedia, graphics and ...

Join us at the GStreamer Conference: 10-11 October 2016 in Berlin, Germany

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

If you reply to this email, your message will be added to the discussion below:

http://gstreamer-devel.966125.n4.nabble.com/Video-processing-record-play-real-time-from-camera-with-metadata-tp4679305p4679423.html

To unsubscribe from Video processing(record\play) real time from camera with metadata, click here.
NAML

codec error.jpg (191K) Download Attachment

TT pipeline.txt (2K) Download Attachment

Tim Müller

Re: Video processing(record\play) real time from camera with metadata

On Fri, 2016-09-16 at 06:49 -0700, Mikl wrote:

Hi,

> For the moment i am doing only video and from test source.
> Here is my pipeline
> videotestsrc ! video/x-
> raw,format=RGB,width=800,height=400,framerate=24/1 ! matroskamux
> ! filesink location=c:\gstreamer\test.mkv
> I have file. It have size (big one). I can see structure
> using MKVToolNix.
> videotestsrc will be replaced by appsrc. And i will not
> need capsfilter anymore.
>
> But:
> -i can not play it
> -see no information

Yes, it will be very big, because you are storing raw (unencoded) video
frames in the file. If you want to encode them, add a x264enc (for
example) before matroskamux.

> Even when i did simple pipeline
> videotestsrc ! avimux ! filesink location=c:\gstreamer\test.avi
> i was not able to see information about video, navigate in video.
> And it was opened not in all players.

I suspect not all players will be able to handle raw video in AVI.
Also, if you use gst-launch-1.0 (or your own app) you will need to make
sure that you shut down the pipeline 'cleanly', so that the muxer has
an opportunity to finalise the file properly (write indices, fix up the
header, etc.). In gst-launch-1.0 you do that by passing the -e command
line option, or using videotestsrc num-buffers=100 or such. In your
application you would have to send an EOS event on the pipeline with
gst_element_send_event(pipeline, gst_event_new_eos()) and then wait for
an EOS message on the bus before shutting down the pipeline with
gst_element_set_state().

Cheers
-Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com

Join us at the GStreamer Conference!
10-11 October 2016 in Berlin,
Germany
http://gstreamer.freedesktop.org/conference/

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Mikl

Re: Video processing(record\play) real time from camera with metadata

Hello, Tim

Thank you for the answers. It is really helping me.

Using gst_element_send_event(), now i see another error message from player.

It is a bit step for me. It means container is readable, but can not me played.

Thank you for help. I can look for this for a days.

But i do no really understand how my pipeline should looks like. something like this?

videotestsrc ! video/x-raw,format=RGB,width=800,height=400,framerate=24/1 ! ffmpegcolorspace ! x264enc ! matroskamux ! filesink location=c:\gstreamer\test.mkv

But:

1) i can not make

_videoConverter = gst_element_factory_make("ffmpegcolorspace","VideoConvertor");

It is returning null.

2) can i really use x264enc? is marked as "ugly"

https://www.freedesktop.org/software/gstreamer-sdk/data/docs/latest/gst-plugins-ugly-plugins-0.10/gst-plugins-ugly-plugins-x264enc.html

Am i right in my pipeline?

It is a way to do what i need?

Is any more simple way exist?

Thank you in advance.

Best regards, Michael.

From: Tim Müller [via GStreamer-devel] <ml-node+[hidden email]>
Sent: Friday, September 16, 2016 6:19:47 PM
To: Michael Yarochkin
Subject: Re: Video processing(record\play) real time from camera with metadata

On Fri, 2016-09-16 at 06:49 -0700, Mikl wrote:

Hi,

If you reply to this email, your message will be added to the discussion below:

http://gstreamer-devel.966125.n4.nabble.com/Video-processing-record-play-real-time-from-camera-with-metadata-tp4679305p4679636.html

To unsubscribe from Video processing(record\play) real time from camera with metadata, click here.
NAML

wrong codec.png (37K) Download Attachment

Mikl

Re: Video processing(record\play) real time from camera with metadata

In reply to this post by Tim Müller

Hello, Tim

I did this!

Pipeline:

videotestsrc ! video/x-raw,format=RGB,width=800,height=400,framerate=24/1 ! jpegenc ! matroskamux ! filesink location=c:\gstreamer\test.mkv

And then close after gst_element_send_event(_pipeline, gst_event_new_eos())

I can play it in

-VLC

-Media Player Classic

But not in Windows media player (12).

Probably JPEG id too simple for it.

And file size is small.

My next step is to select right codec:

-cross platform

-loss less

-fast

Can you point me to the guide, please, how to do it.

Best regards, Michael.

On Fri, 2016-09-16 at 06:49 -0700, Mikl wrote:

Hi,

If you reply to this email, your message will be added to the discussion below:

http://gstreamer-devel.966125.n4.nabble.com/Video-processing-record-play-real-time-from-camera-with-metadata-tp4679305p4679636.html

To unsubscribe from Video processing(record\play) real time from camera with metadata, click here.
NAML

wmp error.png (175K) Download Attachment

Mikl

Re: Video processing(record\play) real time from camera with metadata

In reply to this post by Tim Müller

Hello, Tim

I have a question about memory management.

Almost all functions returning pointer: for example gst_element_factory_make();

Do i need to "delete" it after i don`t need it?

Will you also have time to explain why?

My question in c++ code:

v1:

class Recorder

{

GstElement* _pipeline = nullptr;

void Start()

{

if(_pipeline !=nullptr) delete _pipeline ;

_pipeline = gst_pipeline_new("VideoRecorder");

//some code

}

v2:

class Recorder

{

GstElement* _pipeline;

void Start()

{

_pipeline = gst_pipeline_new("VideoRecorder");

//some code

}

v3:

class Recorder

{

void Start()

{

GstElement* _pipeline = gst_pipeline_new("VideoRecorder");

//some code

}

Best regards, Michael.

On Fri, 2016-09-16 at 06:49 -0700, Mikl wrote:

Hi,

If you reply to this email, your message will be added to the discussion below:

http://gstreamer-devel.966125.n4.nabble.com/Video-processing-record-play-real-time-from-camera-with-metadata-tp4679305p4679636.html

To unsubscribe from Video processing(record\play) real time from camera with metadata, click here.
NAML