Can't decode frames for H264 stream encoded using NVENC over RTP

[Moiz]

I am testing out a sender and receiver pipeline using nvh264enc. I am able to
get this pipeline to work using x264enc however when I switch to nvh264,
although the snder pipeline works in that it starts a clock, I am not
getting any frames nor is the direct3d11 render window showing up when I run
the receiver pipeline.

I know nvh264enc is working because I am able to encode and save to an mp4
file, however when I switch to udpsink, it doesn't decode frames.

Here is the pipeline I used to send/receive h264 stream encoded using
x264enc over udp

Sender: .\gst-launch-1.0.exe -v dxgiscreencapsrc ! queue !
video/x-raw,framerate=20/1 ! videoconvert ! x264enc ! rtph264pay ! udpsink
host=127.0.0.1 port=5000
Receiver: .\gst-launch-1.0.exe -v udpsrc port=5000 caps =
"application/x-rtp, media=(string)video, clock-rate=(int)90000,
encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! decodebin !
videoconvert ! autovideosink

When I use this same pipeline switching x264enc to nvh264 enc, the sender
works but the receiver is not decoding frames nor is it opening the window,
so I tried playing around with my rtph264pay and nvh264enc caps however
still same result, this was the pipelines I tried

Sender: .\gst-launch-1.0.exe -v dxgiscreencapsrc ! queue !
video/x-raw,framerate=20/1 ! videoconvert ! nvh264enc !
"video/x-h264,profile=(string)high" ! rtph264pay !
"application/x-rtp,payload=(int)123" ! udpsink host=127.0.0.1 port=5000

Receiver: .\gst-launch-1.0.exe -v udpsrc port=5000 caps =
"application/x-rtp, media=(string)video, clock-rate=(int)90000,
encoding-name=(string)H264, payload=(int)123" ! rtph264depay ! decodebin !
videoconvert ! autovideosink

The receiver is stuck on "New clock: GstSystemClock" with no further output

To  ensure nvh264enc worked, I tried saving a screencap as mp4 and it worked
successfully and I was able to play the mp4 file using this pipeline

.\gst-launch-1.0.exe -e dxgiscreencapsrc width=1280 height=650 y=32 ! queue
! videorate drop-only=true ! video/x-raw,framerate=60/1 ! videoconvert !
video/x-raw,format=I420,framerate=60/1 ! nvh264enc ! h264parse ! mp4mux !
filesink location=nvh264.mp4


I am assuming it has something to do with the packetization happening
through rtph264pay or something on my receiver, but I am not able to figure
out what it is as I am not very familiar with conversion to packets and
sending packets over udp

Any guidance/direction would be appreciated greatly



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[e-mail]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Moiz]

Correction, the pipeline does work however only if the receiver is run first, how can I get it so that it runs even if I start the receiver command after sender? Why is this happening?



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[e-mail]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mer. 21 oct. 2020 01 h 15, Moiz <[hidden email]> a écrit :

I am testing out a sender and receiver pipeline using nvh264enc. I am able to
get this pipeline to work using x264enc however when I switch to nvh264,
although the snder pipeline works in that it starts a clock, I am not
getting any frames nor is the direct3d11 render window showing up when I run
the receiver pipeline.

I know nvh264enc is working because I am able to encode and save to an mp4
file, however when I switch to udpsink, it doesn't decode frames.

Here is the pipeline I used to send/receive h264 stream encoded using
x264enc over udp

Sender: .\gst-launch-1.0.exe -v dxgiscreencapsrc ! queue !
video/x-raw,framerate=20/1 ! videoconvert ! x264enc ! rtph264pay ! udpsink
host=127.0.0.1 port=5000
Receiver: .\gst-launch-1.0.exe -v udpsrc port=5000 caps =
"application/x-rtp, media=(string)video, clock-rate=(int)90000,
encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! decodebin !
videoconvert ! autovideosink

When I use this same pipeline switching x264enc to nvh264 enc, the sender
works but the receiver is not decoding frames nor is it opening the window,
so I tried playing around with my rtph264pay and nvh264enc caps however
still same result, this was the pipelines I tried

Sender: .\gst-launch-1.0.exe -v dxgiscreencapsrc ! queue !
video/x-raw,framerate=20/1 ! videoconvert ! nvh264enc !
"video/x-h264,profile=(string)high" ! rtph264pay !
"application/x-rtp,payload=(int)123" ! udpsink host=127.0.0.1 port=5000

A wild guess, but try adding h264parse. You should also check or force the video format, to avoid colorspace that might not be supported widely (like yuv 444). Another item to check is your key frame distance.

Receiver: .\gst-launch-1.0.exe -v udpsrc port=5000 caps =
"application/x-rtp, media=(string)video, clock-rate=(int)90000,
encoding-name=(string)H264, payload=(int)123" ! rtph264depay ! decodebin !
videoconvert ! autovideosink

To ensure nvh264enc worked, I tried saving a screencap as mp4 and it worked
successfully and I was able to play the mp4 file using this pipeline

.\gst-launch-1.0.exe -e dxgiscreencapsrc width=1280 height=650 y=32 ! queue
! videorate drop-only=true ! video/x-raw,framerate=60/1 ! videoconvert !
video/x-raw,format=I420,framerate=60/1 ! nvh264enc ! h264parse ! mp4mux !
filesink location=nvh264.mp4


I am assuming it has something to do with the packetization happening
through rtph264pay or something on my receiver, but I am not able to figure
out what it is as I am not very familiar with conversion to packets and
sending packets over udp

Any guidance/direction would be appreciated greatly



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Moiz]

I've tried adding h264parse with no avail, how can I go about setting the key
frame distance on nvh264enc? I know I can use key-int-max on x264enc but not
sure what is the equivalent on nvh264enc



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Gary Metalle]

Hi

I use the following code snippet to force a key frame on-demand (should work on any encoder):

        if (update) {
                GstPad *encoder_src_pad = NULL;
                encoder_src_pad = gst_element_get_static_pad(st->streamer.encoder, "src");
                if (encoder_src_pad) {
                        gst_pad_send_event(encoder_src_pad, gst_event_new_custom(GST_EVENT_CUSTOM_UPSTREAM,
                                gst_structure_new("GstForceKeyUnit", "all-headers",G_TYPE_BOOLEAN, TRUE, NULL)));
                        gst_object_unref(encoder_src_pad);
                }
        }

You basically find the src pad of your encoder and send a custom upstream event to force the key frame.

If you want this done at a regular interval then just create a background thread or timer at the required rate and make this call.

Gary.
________________________________________
From: gstreamer-devel <[hidden email]> on behalf of Moiz <[hidden email]>
Sent: 21 October 2020 20:42
To: [hidden email]
Subject: Re: Can't decode frames for H264 stream encoded using NVENC over RTP

I've tried adding h264parse with no avail, how can I go about setting the key
frame distance on nvh264enc? I know I can use key-int-max on x264enc but not
sure what is the equivalent on nvh264enc



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Moiz]

Solution was to set config-interval to -1 on rtph264pay!



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeudi 22 octobre 2020 à 22:27 -0500, Moiz a écrit :
> Solution was to set config-interval to -1 on rtph264pay!

Ok, so basically there was preriodic keyframe, but not headers
prepended. Glad you figure-out. It is also something that h264parse can
do.

Nicolas

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel