Tracking PTS of decoded H264 frames with leaky queue

[barapa]

I have an annex-b H264 stream that I am decoding via a gstreamer command-line
pipeline that looks something like:

gst-launch-1.0 fdsrc ! queue ! h264parse ! avdec_h264 ! videoconvert !
'video/x-raw, format=RGB'  ! fdsink sync=false

I run this as a subprocess from another program, communicating over
stdin/stdout. I cannot easily build gstreamer into this program, so for now,
I am stuck running it as a subprocess like this.

Before writing each frame to stdin, I know the PTS of the frame, and I need
to know this PTS on the RGB image I read out of stdout. At the moment, I
simply match the incoming to outgoing frames one-to-one. However, I want
gstreamer to be able to drop frames to keep up by introducing a leaky queue.
However, in doing so, I lose track of which PTS belongs to which output RGB
image.

Is there any way to write the PTS values into the stream somehow and recover
them after decoding (while continuing to use a command-line pipeline)? Or,
can I get gstreamer to somehow notify me of a dropped frame, preferably by
outputting something to stdout that I could parse?

Thanks



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Jan Alexander Steffens]

On Tue, Feb 5, 2019, 15:12 barapa <[hidden email] wrote:

I have an annex-b H264 stream that I am decoding via a gstreamer command-line
pipeline that looks something like:

gst-launch-1.0 fdsrc ! queue ! h264parse ! avdec_h264 ! videoconvert !
'video/x-raw, format=RGB'  ! fdsink sync=false

I run this as a subprocess from another program, communicating over
stdin/stdout. I cannot easily build gstreamer into this program, so for now,
I am stuck running it as a subprocess like this.

Before writing each frame to stdin, I know the PTS of the frame, and I need
to know this PTS on the RGB image I read out of stdout. At the moment, I
simply match the incoming to outgoing frames one-to-one. However, I want
gstreamer to be able to drop frames to keep up by introducing a leaky queue.
However, in doing so, I lose track of which PTS belongs to which output RGB
image.

Is there any way to write the PTS values into the stream somehow and recover
them after decoding (while continuing to use a command-line pipeline)? Or,
can I get gstreamer to somehow notify me of a dropped frame, preferably by
outputting something to stdout that I could parse?

Ending your pipeline with "! identity silent=0 ! fdsink sync=0" and starting gst-launch with -v (or was it -m?) should print buffer info to stderr right before writing each frame.

Needless to say this isn't considered a stable interface. Or a good one, for that matter.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[barapa]

Interesting - I will try that. With my input stream, it does not contain PTS for every frame (other than from the parameters preceding keyframes), so I also need to supply them. Is the only way to do this to containerize my input? If so, any suggestions for the format, at least for testing? This needs to be as close to real-time as possible.

On Tue, Feb 5, 2019 at 9:48 AM Jan Alexander Steffens <[hidden email]> wrote:

On Tue, Feb 5, 2019, 15:12 barapa <[hidden email] wrote:

I have an annex-b H264 stream that I am decoding via a gstreamer command-line
pipeline that looks something like:

gst-launch-1.0 fdsrc ! queue ! h264parse ! avdec_h264 ! videoconvert !
'video/x-raw, format=RGB'  ! fdsink sync=false

I run this as a subprocess from another program, communicating over
stdin/stdout. I cannot easily build gstreamer into this program, so for now,
I am stuck running it as a subprocess like this.

Before writing each frame to stdin, I know the PTS of the frame, and I need
to know this PTS on the RGB image I read out of stdout. At the moment, I
simply match the incoming to outgoing frames one-to-one. However, I want
gstreamer to be able to drop frames to keep up by introducing a leaky queue.
However, in doing so, I lose track of which PTS belongs to which output RGB
image.

Is there any way to write the PTS values into the stream somehow and recover
them after decoding (while continuing to use a command-line pipeline)? Or,
can I get gstreamer to somehow notify me of a dropped frame, preferably by
outputting something to stdout that I could parse?

Ending your pipeline with "! identity silent=0 ! fdsink sync=0" and starting gst-launch with -v (or was it -m?) should print buffer info to stderr right before writing each frame.

Needless to say this isn't considered a stable interface. Or a good one, for that matter.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Jan Alexander Steffens]

On Tue, Feb 5, 2019 at 4:29 PM Ben Rapaport <[hidden email]> wrote:

Interesting - I will try that. With my input stream, it does not contain PTS for every frame (other than from the parameters preceding keyframes), so I also need to supply them. Is the only way to do this to containerize my input? If so, any suggestions for the format, at least for testing? This needs to be as close to real-time as possible.

Yes; raw H.264 has no timing information. For a container format, I would wrap it in Matroska.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[MaZderMind]

Hi

Am 05.02.19 um 14:33 schrieb barapa:
> Is there any way to write the PTS values into the stream somehow and recover
> them after decoding (while continuing to use a command-line pipeline)?
Jou need to wrap the generated Frames in some kind of Container.
Matroska is a usual suspect but it might be to hard to parse in your
target application, if linking in 3rd party libraries is not possible,
as you suggested.
One of the simpler options (just a static, fixed length/format header)
would be rtpvrawpay, altough I think it will only carry a frame-counter
in its Timestamp-Field for raw video.

As far as I read your requirements this might be enough, because you
would be able to determine dropped or duplicated frames this way.

-Peter
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel