Problem with h264parse element byte-stream output

[Amit Pandya]

Hi all,

Trying to decode H264 encoded file (mp4 container test_1280x720.mp4
<http://gstreamer-devel.966125.n4.nabble.com/file/t376003/test_1280x720.mp4>
) using following pipeline.

gst-launch-1.0 filesrc location=test_1280x720.mp4 ! qtdemux ! h264parse !
'video/x-h264, stream-format=(string)byte-stream' ! H264 decoder element

For above pipeline first input buffer received by H264 deoder from h264parse
seems not a complete frame (Refer first_buffer.264 first_buffer.264
<http://gstreamer-devel.966125.n4.nabble.com/file/t376003/first_buffer.264>
) - size of only few (~28)bytes only
Followed with rest of the first frame with second input buffer (Refer
second_buffer.264 second_buffer.264
<http://gstreamer-devel.966125.n4.nabble.com/file/t376003/second_buffer.264>
).
This is creating issues with H264 decode element as it expects complete
frame with first input buffer.

Here, "stream-format=(string)byte-stream" is forced using caps on output of
h264parse.

Shouldn't this force h264parse to send complete frame as first input buffer
for decoder ?

For further tryouts dumped the byte-stream output of h264parse for the same
mp4 file using below pipeline.

gst-launch-1.0 filesrc location=test_1280x720.mp4 ! qtdemux ! h264parse !
'video/x-h264, stream-format=(string)byte-stream' ! filesink
location=test_1280x720.264

Now, If use this dumped .264 using following pipline then first input buffer
received by H264 decoder from h264parse seems a complete frame (Refer
first_buffer_elemenary_input.264 first_buffer_elemenary_input.264
<http://gstreamer-devel.966125.n4.nabble.com/file/t376003/first_buffer_elemenary_input.264>
)

gst-launch-1.0 filesrc location=test_1280x720.264 ! h264parse !
'video/x-h264, stream-format=(string)byte-stream' ! H264 decoder element

Need help to understand why h264parse element is behaving differently in
above two cases of .mp4 container vs .264 elementary as input file.

-Thanks
 



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mer. 5 sept. 2018 13:27, Amit Pandya <[hidden email]> a écrit :

Hi all,

Trying to decode H264 encoded file (mp4 container test_1280x720.mp4
<http://gstreamer-devel.966125.n4.nabble.com/file/t376003/test_1280x720.mp4>
) using following pipeline.

gst-launch-1.0 filesrc location=test_1280x720.mp4 ! qtdemux ! h264parse !
'video/x-h264, stream-format=(string)byte-stream' ! H264 decoder element

For above pipeline first input buffer received by H264 deoder from h264parse
seems not a complete frame (Refer first_buffer.264 first_buffer.264
<http://gstreamer-devel.966125.n4.nabble.com/file/t376003/first_buffer.264>
) - size of only few (~28)bytes only
Followed with rest of the first frame with second input buffer (Refer
second_buffer.264 second_buffer.264
<http://gstreamer-devel.966125.n4.nabble.com/file/t376003/second_buffer.264>
).
This is creating issues with H264 decode element as it expects complete
frame with first input buffer.

Here, "stream-format=(string)byte-stream" is forced using caps on output of
h264parse.

Shouldn't this force h264parse to send complete frame as first input buffer
for decoder ?

No, you also need to specify alignment=au for complete frames. Hopefully this is all.

For further tryouts dumped the byte-stream output of h264parse for the same
mp4 file using below pipeline.

gst-launch-1.0 filesrc location=test_1280x720.mp4 ! qtdemux ! h264parse !
'video/x-h264, stream-format=(string)byte-stream' ! filesink
location=test_1280x720.264

Now, If use this dumped .264 using following pipline then first input buffer
received by H264 decoder from h264parse seems a complete frame (Refer
first_buffer_elemenary_input.264 first_buffer_elemenary_input.264
<http://gstreamer-devel.966125.n4.nabble.com/file/t376003/first_buffer_elemenary_input.264>
)

gst-launch-1.0 filesrc location=test_1280x720.264 ! h264parse !
'video/x-h264, stream-format=(string)byte-stream' ! H264 decoder element

Need help to understand why h264parse element is behaving differently in
above two cases of .mp4 container vs .264 elementary as input file.

-Thanks




--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Thornton, Keith]

Hi, a first buffer with length 28 might be an sps.

-----Ursprüngliche Nachricht-----
Von: gstreamer-devel [mailto:[hidden email]] Im Auftrag von Amit Pandya
Gesendet: Mittwoch, 5. September 2018 19:18
An: [hidden email]
Betreff: Problem with h264parse element byte-stream output

Hi all,

Trying to decode H264 encoded file (mp4 container test_1280x720.mp4 <http://gstreamer-devel.966125.n4.nabble.com/file/t376003/test_1280x720.mp4>
) using following pipeline.

gst-launch-1.0 filesrc location=test_1280x720.mp4 ! qtdemux ! h264parse !
'video/x-h264, stream-format=(string)byte-stream' ! H264 decoder element

For above pipeline first input buffer received by H264 deoder from h264parse seems not a complete frame (Refer first_buffer.264 first_buffer.264 <http://gstreamer-devel.966125.n4.nabble.com/file/t376003/first_buffer.264>
) - size of only few (~28)bytes only
Followed with rest of the first frame with second input buffer (Refer
second_buffer.264 second_buffer.264
<http://gstreamer-devel.966125.n4.nabble.com/file/t376003/second_buffer.264>
).
This is creating issues with H264 decode element as it expects complete frame with first input buffer.

Here, "stream-format=(string)byte-stream" is forced using caps on output of h264parse.

Shouldn't this force h264parse to send complete frame as first input buffer for decoder ?

For further tryouts dumped the byte-stream output of h264parse for the same
mp4 file using below pipeline.

gst-launch-1.0 filesrc location=test_1280x720.mp4 ! qtdemux ! h264parse !
'video/x-h264, stream-format=(string)byte-stream' ! filesink
location=test_1280x720.264

Now, If use this dumped .264 using following pipline then first input buffer received by H264 decoder from h264parse seems a complete frame (Refer
first_buffer_elemenary_input.264 first_buffer_elemenary_input.264 <http://gstreamer-devel.966125.n4.nabble.com/file/t376003/first_buffer_elemenary_input.264>
)

gst-launch-1.0 filesrc location=test_1280x720.264 ! h264parse !
'video/x-h264, stream-format=(string)byte-stream' ! H264 decoder element

Need help to understand why h264parse element is behaving differently in above two cases of .mp4 container vs .264 elementary as input file.

-Thanks
 



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Amit Pandya]

Hi Nicolas,

Tried specifying alignment=au as suggested using below pipeline but same
observation.

gst-launch-1.0 filesrc location=test_1280x720.mp4 ! qtdemux ! h264parse !
'video/x-h264, stream-format=(string)byte-stream, alignment=(string)au' !
H264 decoder element

For above pipeline also first input buffer received by H264 deoder from
h264parse is of (~28)bytes only & not a complete frame.

Let me know in case any further inputs.

PS: Please refer attached files
test_1280x720.mp4/first_buffer.264/second_buffer.264/first_buffer_elemenary_input.264
in case help with further analysis.  

-Thanks





--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeudi 06 septembre 2018 à 01:15 -0500, Amit Pandya a écrit :

> Hi Nicolas,
>
> Tried explicitly specifying "alignment=au" for complete frame as
> suggested
> though same observation,
>
> gst-launch-1.0 filesrc location=test_1280x720.mp4 ! qtdemux !
> h264parse !
> 'video/x-h264, stream-format=(string)byte-stream, ,
> alignment=(string)au' !
> H264 decoder element
>
> With above pipeline also first input buffer received by H264 deoder
> from
> h264parse is size of only few (~28)bytes & not a complete frame.
>
> Let me know in case nay further inputs.
>
> PS: I have already attached streams
> test_1280x720.mp4/first_buffer.264/second_buffer.264/first_buffer_ele
> menary_input.264
> for reference in case help with analysis.

Attachements don't work over the mailing list. You'll need to upload
somewhere and provide a link.

>
> -Thanks
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Amit Pandya]

Hi,

Resolved the issue & follwoing are the details for reference/inputs,

Problem seems to be with the encode pipeline used to generate input h264
container file (test_1280x720.mp4) using following pipeline,
H264 encoder element ! 'video/x-h264, stream-format=(string)byte-stream,
alignment=(string)au' ! h264parse ! qtmux ! filesink
location=test_1280x720.mp4 -e

In above pipeline our h264 encoder send 1st buffer consisting only
"sps+pps"(~28 bytes) & 2nd buffer consist of the "first frame data".
Now, h264parse element seems sending this 1st "sps+pps" buffer as is to
qtmux considering first frame to be muxed with mp4 container.

Now, when try to decode such generated h264 container mp4 file then the
first buffer sent out by h264parse is same "sps+pps"(~28 bytes) buffer & not
the complete frame creating issue with the decode pipeline.
gst-launch-1.0 filesrc location=test_1280x720.mp4 ! qtdemux ! h264parse !
'video/x-h264, stream-format=(string)byte-stream, alignment=(string)au' !
H264 decoder element      

For resolution of the issue handled this within our encoder merging first
two buffers making sure that first buffer sent out by encoder consist of
both "sps+pps+frame data" combined.

=> Also, Tried following experiment to further investigate comparative
behavior with H265.

Used follwing pipeline to generate input h265 container file
(test_1280x720_h265.mp4)
H265 encoder element ! 'video/x-h265, stream-format=(string)byte-stream,
alignment=(string)au' ! h265parse ! qtmux ! filesink
location=test_1280x720_h265.mp4 -e

In above pipeline our h265 encoder similraly send 1st buffer consisting only
"sps+pps" & 2nd buffer consist of the "first frame data".
h265parse element seems sending this 1st "sps+pps" buffer as is to qtmux
considering first frame to be muxed with mp4 container.

Now, when try to decode such generated h265 container mp4 file then the
first buffer sent out by h265parse is a complete frame (in contrast with
above observation from h264parse case)
gst-launch-1.0 filesrc location=test_1280x720_h265.mp4 ! qtdemux ! h265parse
! 'video/x-h265, stream-format=(string)byte-stream, alignment=(string)au' !
H265 decoder element      
 
Hence, It appears h265parse element is able to handle such scenario & take
care of forming first output buffer consisting of complete frame out of two
separate buffers received(1st:sps+pps & 2nd:frame data).

Though h264parse is not taking care of forming complete frame & sends out
first frame consist of sps+pps buffer only.

Not sure if this is a bug with h264parse element or
should not expect handling of such scenario by h264pasre element considering
still gst-plugins-bad.

Any inputs are appreciated.

 -Thanks



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Gregoire Gentil]

When using the snapshot application:

https://github.com/GStreamer/gst-plugins-base/blob/master/tests/examples/snapshot/snapshot.c

I have only modified the code to put 50% instead of 5% for the seek
duration.

gcc snapshot.c `pkg-config --cflags --libs gstreamer-1.0 gtk+-3.0`


it's working well with a mp4 file but it generates a gray image for a
mpeg-ts file. See two examples (mp4 and mpeg-ts) here:

http://gentil.com/tmp/snapshot.zip

I think that the reason is that the mpeg-ts has less information about
timestamp so it needs a few frames before it decodes properly a key frame.

How can the source code be patched so that multiple successive frames
are decoded before the sample is taken?

Note that if I play the mpeg-ts file and I seek, the first few frames
after the seek are also gray for the same reason explained above. Once
again, this is due to the lack of information of frame timestamp in the
mpeg-ts container,


Grégoire

PS: I have opened a bug
(https://bugzilla.gnome.org/show_bug.cgi?id=797155) but as it's more an
enhancement request of this application than a crashing bug, I post it
to the mailing list.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel