Trying to understand gst_video_frame_map

[Ben Rush]

The following is more of an academic question, though I do have a practical reason for wanting to understand a bit more about this function. Bottom line is that I'm mostly exploring. 

First, I'll admit that I'm not an expert in media formats (all the various ways of encoding YUV, etc), most of the time I let the underlying library handle that part for me (converting between color spaces, creating packed vs. unpacked byte arrays for the frame data, etc). However, I'm trying to write my own simple "videotestsrc" in order to understand the basics of creating a GstPushSrc. I've got the thing written from the ground up (and it works in a gstreamer pipeline which dumps the video to an MP4) with the exception of the actual creation of the video frames. All I've got thus far is a green video being written out. 

So here's my question: say I've got a simple frame in YUV444 (https://gstreamer.freedesktop.org/documentation/design/mediatype-video-raw.html see AYUV) and I want to send this down to the next element as a source. How do I do it? The videotestsrc code is confusing with its creation of moving patterns and use of some helper class called "paintinfo"; I feel like it's making it hard for me to understand where I insert the data, and in precisely what format. According to the documentation on the link above, the format I'm playing with is YUV444

+-- + -- + -- + -- + +-- + -- + -- + -- +

| A0 | Y0 | U0 | V0| |A1 | Y1 | U1 | V1 | ...

+ -- + -- + -- + -- + +-- + -- + -- + -- +

In videotestsrc.c, line 1156 in  https://github.com/GStreamer/gst-plugins-base/blob/master/gst/videotestsrc/gstvideotestsrc.c the function "make_image" is surrounded by the gst_video_frame_map function.  And according to its documentation ( https://thiblahute.github.io/GStreamer-doc/gst-plugins-base-video-1.0/video-frame.html?gi-language=c#gst_video_frame_map ) I should be able to call it, use the macro GST_VIDEO_FRAME_PLANE_DATA to get a pointer to an underlying data buffer, and then write to it. However, if that macro returns a variable called "pixels" as a gchar*, the following code will cause exceptions (likely buffer overruns): 

for (int y = 0, i=0; y < frame.info.height; y++)

{

for (int x = 0; x < frame.info.width; x++, i++)

{

pixels[i * 4] = 0; 

pixels[i * 4 + 1] = Y;

pixels[i * 4 + 2] = U;

pixels[i * 4 + 3] = V;

}

} 

 
So I feel like I'm not really accessing the data right. The videotestsrc code uses some ORC library and appears to do something similar by creating an int32 and writing it atop the buffer (see line 148 of  https://github.com/GStreamer/gst-plugins-base/blob/master/gst/videotestsrc/gstvideotestsrcorc-dist.c ). 

Any input? I feel like I'm very, very close. I just need to figure out how to packetize the data  into the GstVideoFrame structure, I feel. 

Thanks in advance. 

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Ben Rush]

I believe I have figured out my issue. There is a "pack_func" that is attached to the GstVideoFrame's GstVideoInfo structure that all the bytes must pass through in order to be sent down the chain. Upon executing this function against my YUV data, I was able to get it to save a well-formatted and good-looking MP4 file. 

On Mon, Apr 22, 2019 at 4:54 PM Ben Rush <[hidden email]> wrote:

The following is more of an academic question, though I do have a practical reason for wanting to understand a bit more about this function. Bottom line is that I'm mostly exploring. 

First, I'll admit that I'm not an expert in media formats (all the various ways of encoding YUV, etc), most of the time I let the underlying library handle that part for me (converting between color spaces, creating packed vs. unpacked byte arrays for the frame data, etc). However, I'm trying to write my own simple "videotestsrc" in order to understand the basics of creating a GstPushSrc. I've got the thing written from the ground up (and it works in a gstreamer pipeline which dumps the video to an MP4) with the exception of the actual creation of the video frames. All I've got thus far is a green video being written out. 

So here's my question: say I've got a simple frame in YUV444 (https://gstreamer.freedesktop.org/documentation/design/mediatype-video-raw.html see AYUV) and I want to send this down to the next element as a source. How do I do it? The videotestsrc code is confusing with its creation of moving patterns and use of some helper class called "paintinfo"; I feel like it's making it hard for me to understand where I insert the data, and in precisely what format. According to the documentation on the link above, the format I'm playing with is YUV444

+-- + -- + -- + -- + +-- + -- + -- + -- +

| A0 | Y0 | U0 | V0| |A1 | Y1 | U1 | V1 | ...

+ -- + -- + -- + -- + +-- + -- + -- + -- +

In videotestsrc.c, line 1156 in  https://github.com/GStreamer/gst-plugins-base/blob/master/gst/videotestsrc/gstvideotestsrc.c the function "make_image" is surrounded by the gst_video_frame_map function.  And according to its documentation ( https://thiblahute.github.io/GStreamer-doc/gst-plugins-base-video-1.0/video-frame.html?gi-language=c#gst_video_frame_map ) I should be able to call it, use the macro GST_VIDEO_FRAME_PLANE_DATA to get a pointer to an underlying data buffer, and then write to it. However, if that macro returns a variable called "pixels" as a gchar*, the following code will cause exceptions (likely buffer overruns): 

for (int y = 0, i=0; y < frame.info.height; y++)

{

for (int x = 0; x < frame.info.width; x++, i++)

{

pixels[i * 4] = 0; 

pixels[i * 4 + 1] = Y;

pixels[i * 4 + 2] = U;

pixels[i * 4 + 3] = V;

}

} 

 
So I feel like I'm not really accessing the data right. The videotestsrc code uses some ORC library and appears to do something similar by creating an int32 and writing it atop the buffer (see line 148 of  https://github.com/GStreamer/gst-plugins-base/blob/master/gst/videotestsrc/gstvideotestsrcorc-dist.c ). 

Any input? I feel like I'm very, very close. I just need to figure out how to packetize the data  into the GstVideoFrame structure, I feel. 

Thanks in advance. 

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mardi 23 avril 2019 à 00:39 -0500, Ben Rush a écrit :

I believe I have figured out my issue. There is a "pack_func" that is attached to the GstVideoFrame's GstVideoInfo structure that all the bytes must pass through in order to be sent down the chain. Upon executing this function against my YUV data, I was able to get it to save a well-formatted and good-looking MP4 file. 

The pack/unpack function are used for generic colour handling, we also call these the slow paths. There very few unpack formats, so it reduces a lot the amount of code to support all possible format (used in videoconvert, videoscale, videotestsrc, etc). These formats are packed AYUV (8 bit per comp), AYUV64 (16bit per comp), ARGB (8 bit) and ARGB64 (16bit).

glad you find out about this,

Nicolas

On Mon, Apr 22, 2019 at 4:54 PM Ben Rush <[hidden email]> wrote:

The following is more of an academic question, though I do have a practical reason for wanting to understand a bit more about this function. Bottom line is that I'm mostly exploring. 

First, I'll admit that I'm not an expert in media formats (all the various ways of encoding YUV, etc), most of the time I let the underlying library handle that part for me (converting between color spaces, creating packed vs. unpacked byte arrays for the frame data, etc). However, I'm trying to write my own simple "videotestsrc" in order to understand the basics of creating a GstPushSrc. I've got the thing written from the ground up (and it works in a gstreamer pipeline which dumps the video to an MP4) with the exception of the actual creation of the video frames. All I've got thus far is a green video being written out. 

So here's my question: say I've got a simple frame in YUV444 (https://gstreamer.freedesktop.org/documentation/design/mediatype-video-raw.html see AYUV) and I want to send this down to the next element as a source. How do I do it? The videotestsrc code is confusing with its creation of moving patterns and use of some helper class called "paintinfo"; I feel like it's making it hard for me to understand where I insert the data, and in precisely what format. According to the documentation on the link above, the format I'm playing with is YUV444

+-- + -- + -- + -- + +-- + -- + -- + -- +

| A0 | Y0 | U0 | V0| |A1 | Y1 | U1 | V1 | ...

+ -- + -- + -- + -- + +-- + -- + -- + -- +

In videotestsrc.c, line 1156 in  https://github.com/GStreamer/gst-plugins-base/blob/master/gst/videotestsrc/gstvideotestsrc.c the function "make_image" is surrounded by the gst_video_frame_map function.  And according to its documentation ( https://thiblahute.github.io/GStreamer-doc/gst-plugins-base-video-1.0/video-frame.html?gi-language=c#gst_video_frame_map ) I should be able to call it, use the macro GST_VIDEO_FRAME_PLANE_DATA to get a pointer to an underlying data buffer, and then write to it. However, if that macro returns a variable called "pixels" as a gchar*, the following code will cause exceptions (likely buffer overruns): 

for (int y = 0, i=0; y < frame.info.height; y++)

{

for (int x = 0; x < frame.info.width; x++, i++)

{

pixels[i * 4] = 0; 

pixels[i * 4 + 1] = Y;

pixels[i * 4 + 2] = U;

pixels[i * 4 + 3] = V;

}

} 

 
So I feel like I'm not really accessing the data right. The videotestsrc code uses some ORC library and appears to do something similar by creating an int32 and writing it atop the buffer (see line 148 of  https://github.com/GStreamer/gst-plugins-base/blob/master/gst/videotestsrc/gstvideotestsrcorc-dist.c ). 

Any input? I feel like I'm very, very close. I just need to figure out how to packetize the data  into the GstVideoFrame structure, I feel. 

Thanks in advance. 

_______________________________________________

gstreamer-devel mailing list

[hidden email]

https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Ben Rush]

Thanks for your response, Nicolas. I appreciate you confirming my understanding. Much appreciated. 

On Tue, Apr 23, 2019 at 9:04 AM Nicolas Dufresne <[hidden email]> wrote:

Le mardi 23 avril 2019 à 00:39 -0500, Ben Rush a écrit :

I believe I have figured out my issue. There is a "pack_func" that is attached to the GstVideoFrame's GstVideoInfo structure that all the bytes must pass through in order to be sent down the chain. Upon executing this function against my YUV data, I was able to get it to save a well-formatted and good-looking MP4 file. 

The pack/unpack function are used for generic colour handling, we also call these the slow paths. There very few unpack formats, so it reduces a lot the amount of code to support all possible format (used in videoconvert, videoscale, videotestsrc, etc). These formats are packed AYUV (8 bit per comp), AYUV64 (16bit per comp), ARGB (8 bit) and ARGB64 (16bit).

glad you find out about this,

Nicolas

On Mon, Apr 22, 2019 at 4:54 PM Ben Rush <[hidden email]> wrote:

The following is more of an academic question, though I do have a practical reason for wanting to understand a bit more about this function. Bottom line is that I'm mostly exploring. 

First, I'll admit that I'm not an expert in media formats (all the various ways of encoding YUV, etc), most of the time I let the underlying library handle that part for me (converting between color spaces, creating packed vs. unpacked byte arrays for the frame data, etc). However, I'm trying to write my own simple "videotestsrc" in order to understand the basics of creating a GstPushSrc. I've got the thing written from the ground up (and it works in a gstreamer pipeline which dumps the video to an MP4) with the exception of the actual creation of the video frames. All I've got thus far is a green video being written out. 

So here's my question: say I've got a simple frame in YUV444 (https://gstreamer.freedesktop.org/documentation/design/mediatype-video-raw.html see AYUV) and I want to send this down to the next element as a source. How do I do it? The videotestsrc code is confusing with its creation of moving patterns and use of some helper class called "paintinfo"; I feel like it's making it hard for me to understand where I insert the data, and in precisely what format. According to the documentation on the link above, the format I'm playing with is YUV444

+-- + -- + -- + -- + +-- + -- + -- + -- +

| A0 | Y0 | U0 | V0| |A1 | Y1 | U1 | V1 | ...

+ -- + -- + -- + -- + +-- + -- + -- + -- +

In videotestsrc.c, line 1156 in  https://github.com/GStreamer/gst-plugins-base/blob/master/gst/videotestsrc/gstvideotestsrc.c the function "make_image" is surrounded by the gst_video_frame_map function.  And according to its documentation ( https://thiblahute.github.io/GStreamer-doc/gst-plugins-base-video-1.0/video-frame.html?gi-language=c#gst_video_frame_map ) I should be able to call it, use the macro GST_VIDEO_FRAME_PLANE_DATA to get a pointer to an underlying data buffer, and then write to it. However, if that macro returns a variable called "pixels" as a gchar*, the following code will cause exceptions (likely buffer overruns): 

for (int y = 0, i=0; y < frame.info.height; y++)

{

for (int x = 0; x < frame.info.width; x++, i++)

{

pixels[i * 4] = 0; 

pixels[i * 4 + 1] = Y;

pixels[i * 4 + 2] = U;

pixels[i * 4 + 3] = V;

}

} 

 
So I feel like I'm not really accessing the data right. The videotestsrc code uses some ORC library and appears to do something similar by creating an int32 and writing it atop the buffer (see line 148 of  https://github.com/GStreamer/gst-plugins-base/blob/master/gst/videotestsrc/gstvideotestsrcorc-dist.c ). 

Any input? I feel like I'm very, very close. I just need to figure out how to packetize the data  into the GstVideoFrame structure, I feel. 

Thanks in advance. 

_______________________________________________

gstreamer-devel mailing list

[hidden email]

https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel