Pointer to I-frame extra data in H.264?

[Javiku]

Hi,

I am working on a project where I need to feed some application functions
with the output of an H.264 encoder (v4l2h264enc). Those functions expect a
pointer to the encoded data and a second pointer to the "extra data" from
the I-Frames. This extra data is defined in the application documentation as
follows:

"The extra ancillary data required to configure an H.264 codec must
correctly be provided. This must exist for all I-
frames. The structure of this data should contain concatenated NAL units in
Annex B format, along with their start
codes. For H.264, they are the SPS and PPS NAL units."

So far I managed to create an appsink, link it to a tee object and use a
callback to extract a buffer every time a new sample is generated. Then I
mapped that buffer to get the pointer to the data (map.data) and de data
size (map.size) as well. But I could not get any further because the
GstMapInfo structure does not provide any specific field for I-frames extra
data and I could not find it in the GstSample or GstBuffer structures
either. By the way, I am checking if the GST_BUFFER_FLAG_DELTA_UNIT flag is
not set to find the I-frames, which seems to work fine because the buffers
are way bigger when that happens (once every 30 frames at 30 fps).

I was wondering if there is a specific structure member where I could find
this extra data or if I should look for it somewhere else... maybe packed
within the data block? I had a look at the first bytes of my I-frames and
they start with 0x00 0x00 0x00 0x01 0x07 0x42 0x40 0x28... as other non
I-frames do (maybe P-frames). Could that data be the extra data (SPS and PPS
NAL units) anyways? The GST_BUFFER_FLAG_HEADER flag is never set though.

Thanks a lot for your help, best regards,
Javiku



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Thornton, Keith]

Hi,
The first four bytes in your example are the start code. The next byte indicates the NAL unit type.
Take a look at the description provided in e.g. https://yumichan.net/video-processing/video-compression/introduction-to-h264-nal-unit/ As you can see in his example, there are more than one NAl unit in the buffer. This is encoder specific.
Gruesse

-----Ursprüngliche Nachricht-----
Von: gstreamer-devel <[hidden email]> Im Auftrag von Javiku
Gesendet: Donnerstag, 25. Februar 2021 07:59
An: [hidden email]
Betreff: Pointer to I-frame extra data in H.264?

Hi,

I am working on a project where I need to feed some application functions with the output of an H.264 encoder (v4l2h264enc). Those functions expect a pointer to the encoded data and a second pointer to the "extra data" from the I-Frames. This extra data is defined in the application documentation as
follows:

"The extra ancillary data required to configure an H.264 codec must correctly be provided. This must exist for all I- frames. The structure of this data should contain concatenated NAL units in Annex B format, along with their start codes. For H.264, they are the SPS and PPS NAL units."

So far I managed to create an appsink, link it to a tee object and use a callback to extract a buffer every time a new sample is generated. Then I mapped that buffer to get the pointer to the data (map.data) and de data size (map.size) as well. But I could not get any further because the GstMapInfo structure does not provide any specific field for I-frames extra data and I could not find it in the GstSample or GstBuffer structures either. By the way, I am checking if the GST_BUFFER_FLAG_DELTA_UNIT flag is not set to find the I-frames, which seems to work fine because the buffers are way bigger when that happens (once every 30 frames at 30 fps).

I was wondering if there is a specific structure member where I could find this extra data or if I should look for it somewhere else... maybe packed within the data block? I had a look at the first bytes of my I-frames and they start with 0x00 0x00 0x00 0x01 0x07 0x42 0x40 0x28... as other non I-frames do (maybe P-frames). Could that data be the extra data (SPS and PPS NAL units) anyways? The GST_BUFFER_FLAG_HEADER flag is never set though.

Thanks a lot for your help, best regards, Javiku



--
Sent from: https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgstreamer-devel.966125.n4.nabble.com%2F&amp;data=04%7C01%7C%7C92f95a8be4ca4ea362ec08d8d97c9112%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C637498476320172732%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=wPU2%2FmvF8fupieZPYh4G75VUJ%2BWGGUZ79hf1eaWIMkY%3D&amp;reserved=0
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Fgstreamer-devel&amp;data=04%7C01%7C%7C92f95a8be4ca4ea362ec08d8d97c9112%7C28042244bb514cd680347776fa3703e8%7C1%7C0%7C637498476320172732%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=CRkIZ5pVCa1qW7caTWdGkOxS%2F2BLS2aEl%2B3FE%2FQFqvk%3D&amp;reserved=0
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeudi 25 février 2021 à 00:59 -0600, Javiku a écrit :
> Hi,
>
> I am working on a project where I need to feed some application functions
> with the output of an H.264 encoder (v4l2h264enc). Those functions expect a
> pointer to the encoded data and a second pointer to the "extra data" from
> the I-Frames. This extra data is defined in the application documentation as
> follows:

v4l2h264enc produces byte-stream (Annex B) stream, which means the headers are
inline. To get "extra-data" ala FFMPEG, you need to convert that to AVCc format.
The data will then be in the caps, under the field name codec_data with the type
GstBuffer. Something like:

  ... ! v4l2h264enc ! h264parse ! video/x-h264,profile=main,level=4.1,stream-format=avc ! ...

>
> "The extra ancillary data required to configure an H.264 codec must
> correctly be provided. This must exist for all I-
> frames. The structure of this data should contain concatenated NAL units in
> Annex B format, along with their start
> codes. For H.264, they are the SPS and PPS NAL units."
>
> So far I managed to create an appsink, link it to a tee object and use a
> callback to extract a buffer every time a new sample is generated. Then I
> mapped that buffer to get the pointer to the data (map.data) and de data
> size (map.size) as well. But I could not get any further because the
> GstMapInfo structure does not provide any specific field for I-frames extra
> data and I could not find it in the GstSample or GstBuffer structures
> either. By the way, I am checking if the GST_BUFFER_FLAG_DELTA_UNIT flag is
> not set to find the I-frames, which seems to work fine because the buffers
> are way bigger when that happens (once every 30 frames at 30 fps).
>
> I was wondering if there is a specific structure member where I could find
> this extra data or if I should look for it somewhere else... maybe packed
> within the data block? I had a look at the first bytes of my I-frames and
> they start with 0x00 0x00 0x00 0x01 0x07 0x42 0x40 0x28... as other non
> I-frames do (maybe P-frames). Could that data be the extra data (SPS and PPS
> NAL units) anyways? The GST_BUFFER_FLAG_HEADER flag is never set though.
>
> Thanks a lot for your help, best regards,
> Javiku
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel