GStreamer Video Capture and H264 Encoding with SEI

[steven414]

Hi，

I am trying to build an application which captures video from a Camera
(1920*1088).

The region 1920*1080 will be encoded into H264. While the region 1920*8 is
so-called embedded data, which should be extracted and added into SEI frame.
Then one original raw frame should become one I or P frame and one SEI frame
(embedded data).

Currently I have built the pipeline to do the H264 encoding part like below:
camerasrc -> videoconvert -> h264enc -> h264parse -> rtph264pay -> udpsink

Now I wonder:
1. How to create a corresponding SEI frame from the 1920*8 cropped region.
2. How to insert the SEI frame into H264 frame-by-frame.


<http://gstreamer-devel.966125.n4.nabble.com/file/t379728/Capture.png>



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mardi 03 novembre 2020 à 23:09 -0600, steven414 a écrit :

> Hi，
>
> I am trying to build an application which captures video from a Camera
> (1920*1088).
>
> The region 1920*1080 will be encoded into H264. While the region 1920*8 is
> so-called embedded data, which should be extracted and added into SEI frame.
> Then one original raw frame should become one I or P frame and one SEI frame
> (embedded data).
>
> Currently I have built the pipeline to do the H264 encoding part like below:
> camerasrc -> videoconvert -> h264enc -> h264parse -> rtph264pay -> udpsink
>
> Now I wonder:
> 1. How to create a corresponding SEI frame from the 1920*8 cropped region.

As this is custom way of embedding data, you'll have to add you own
filter, and you'll have to do that before videoconvert, as videoconvert
might endup stripping off these padding lines.

So basically, implement a video fitler (passthrough) that will analyze
the binary coming out of the camera. If the information is really per
frame, I'd use a custom BufferMeta to embedded the information,
otherwise I'd use serialized downstream element event.

> 2. How to insert the SEI frame into H264 frame-by-frame.

For that, you need a second filter, this one will read your BufferMeta,
and transform it into vendor specific SEI messages. You can push the
SEI NAL indepedently, though it will be up to you to place it properly.
Using alignment=au, you simply need to identify the first VLC and
prepend. If performance is a problem, consider using stream-format=avc,
this way you can avoid the costy scanning process. The payloader do
support avcc.

>
>
> <http://gstreamer-devel.966125.n4.nabble.com/file/t379728/Capture.png>
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[steven414]

Hi Nicolas Dufresne-5,

Thanks for your reply.

I am quite new to GStreamer and might not understand your comments
correctly.

1. As I understanded, I should use tee and crop to separate the video from
the camera first. Then use "gst_buffer_add_meta" to add binary into buffer.
How to get binary from the cam?

2. How to build a filter to read BufferMeta and transform it into vendor
specific SEI messages? Do you mean to use an app sink? Also let's say my SEI
should be payload TYPE 5, which is user_data_unregistered. The maximum
paylaod size for custom data is 256-16(UUID)=240 Byte. Therefore for 1920*8
region will result into multi SEI frames in order to embed all information
for one origianl frame. Will this cause a problem when do steaming and
playback?

Here is my current C code for your reference.  2K_Camera.c
<http://gstreamer-devel.966125.n4.nabble.com/file/t379728/2K_Camera.c>  
Thanks.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mercredi 04 novembre 2020 à 21:18 -0600, steven414 a écrit :

> Hi Nicolas Dufresne-5,
>
> Thanks for your reply.
>
> I am quite new to GStreamer and might not understand your comments
> correctly.
>
> 1. As I understanded, I should use tee and crop to separate the video from
> the camera first. Then use "gst_buffer_add_meta" to add binary into buffer.
> How to get binary from the cam?

I didn't suggest that. I suggested to develop a video filter element
that will just extract the information in-place and store is a meta.
That assumes you have acquire knowledge of GLib/GObject programming and
plugins writing (see plugins writer manual on our website).

>
> 2. How to build a filter to read BufferMeta and transform it into vendor
> specific SEI messages? Do you mean to use an app sink? Also let's say my SEI
> should be payload TYPE 5, which is user_data_unregistered. The maximum
> paylaod size for custom data is 256-16(UUID)=240 Byte. Therefore for 1920*8
> region will result into multi SEI frames in order to embed all information
> for one origianl frame. Will this cause a problem when do steaming and
> playback?

There is no short answer for this, I believe you will have to learn a
bit more about GStreamer programming in order to get ready for the
task.

>
> Here is my current C code for your reference.  2K_Camera.c
> <http://gstreamer-devel.966125.n4.nabble.com/file/t379728/2K_Camera.c>  
> Thanks.
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel