h264 gstshm and webrtc

[eadan]

Hi,
I am pretty new to gstreamer. I am trying to make video available to a number of services simultaneously on the raspberry pi platform (whilst using minimum possible CPU time). I have used libcamera to provide a hardware encoded h264 stream and also a raw RGB stream. both are terminated at a gst shmsink having piped the results from libcamera via an appsrc to it. I can use

 gst-launch-1.0 -v shmsrc socket_path=/tmp/shmsink.h264 ! queue ! filesink location=/tmp/shmsink.h264

and

 gst-launch-1.0 -v shmsrc socket_path=/tmp/shmsink.rgb ! queue ! filesink location=/tmp/shmsink.rgb

and both seem to work (almost  as required) with no significant CPU load. I can play the .h264 file in vlc/mplayer provided the stream started at the beginning, there seems to be some kind of header at the start which is then missing if I connect again later on in the stream and end up starting at some arbitrary point in the stream.

I can extract and re-encode the RGB variant with dd and imagemagick and confirm that I have valid frames  in raw RGB.

To start with I need to somehow resolve this synchronisation/media header issue which I assume requires some kind of container format. what would be the best way to go about this?

Next I want to be able push the h264 to a browser via webrtc, I have been having some trouble with this in the application so decided to try and narrow it down with a gst-launch pipeline. I think up to the rtph264pay element it would all be the same and the following webrtc parts are seemingly not possible from a gst pipeline hence I am using a udpsink here.

When I play the .h264 file produced above in vlc in the media codec window it tells me it is H264 MPEG4- AVC (part 10) (h264).

 I tried the following in the hope I could play the resulting stream with vlc or mplayer/mpv etc (have tried all).

gst-launch-1.0  shmsrc socket_path=/tmp/shmsink.h264 ! queue !  video/x-h264,stream-format=byte-stream ! h264parse ! rtph264pay ! udpsink host=<host_ip> port=5000
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:14.133368385
Setting pipeline to NULL ...
Freeing pipeline ...

Which looks like it is working, but I cannot play the resulting stream, mpv says

[ffmpeg/video] h264: non-existing PPS 0 referenced
[ffmpeg/video] h264: non-existing PPS 0 referenced
[ffmpeg/video] h264: decode_slice_header error
[ffmpeg/video] h264: no frame!

over and over.

I have tried with both stream-format=avc, and stream-format=byte-stream and various other combinations of caps but I have not been able to get a working result with any of these.

Having just written all this it suddenly occurred to me it might work if I start mpv first, and sure enough it does!

can anybody provide any pointers on how to resolve this this? I suspect it all comes down to the fact I seem to be generating h264 that is suitable for being contained in a file but not for live streaming, and I am not familiar enough yet with it all to know how to resolve this.

Many Thanks

Dan

[eadan]

answering my own question here, it seems V4L2_CID_MPEG_VIDEO_REPEAT_SEQ_HEADER whilst encoding the h264 is at least partially solving the problem.

[eadan]

I am now trying with the webrtc-unidirectional-h264 example from gst 1.19.1. With the video src set to videotestsrc and using firefox i get the test pattern in the browser. If I update it with the pipeline I am trying to use there seems to be some kind of failure to negotiate via SDP.

My pipeline becomes

      gst_parse_launch ("webrtcbin name=webrtcbin stun-server=stun://"
      STUN_SERVER " "
      " shmsrc socket-path=/tmp/shmsink.h264 ! queue max-size-time=100000000 ! h264parse ! "
      "rtph264pay config-interval=-1 name=payloader aggregate-mode=zero-latency ! "
      "application/x-rtp,media=video,encoding-name=H264,payload="
      RTP_PAYLOAD_TYPE " ! webrtcbin. "

And the SDP info dumped is:

Negotiation offer created:
v=0
o=- 5912664787617810588 0 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-options:trickle
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:HZfi9hhT1Uq/xvFr86atpTSwoPCFRvMs
a=ice-pwd:nkhQ+bXYXWLNxqP1fX0mW98zlo85MZpu
a=rtcp-mux
a=rtcp-rsize
a=sendonly
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=rtcp-fb:96 transport-cc
a=framerate:30
a=fmtp:96 packetization-mode=1;profile-level-id=428028;sprop-parameter-sets=J0KAKJWgHgCJ+WEAAAMAAQAAAwA85KAAJiWAAF9eb3uAeJE1,KM4CXIA=
a=ssrc:363880709 msid:user1503200852@host-bd97efb0 webrtctransceiver0
a=ssrc:363880709 cname:user1503200852@host-bd97efb0
a=mid:video0
a=fingerprint:sha-256 76:A7:03:41:13:AC:9B:49:B8:79:72:3B:98:F3:62:B4:80:CA:D4:38:63:2C:79:9F:DE:39:58:DA:9C:7C:34:E0
a=rtcp-mux-only
m=audio 9 UDP/TLS/RTP/SAVPF 97
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:REi0C+ANDEG1zbsUm48sq/nN1KMS1hAm
a=ice-pwd:/uXK1ncdM89UTSlujW2mjQHaQVG3HfAN
a=rtcp-mux
a=rtcp-rsize
a=sendonly
a=rtpmap:97 OPUS/48000/2
a=rtcp-fb:97 transport-cc
a=fmtp:97 sprop-stereo=0;sprop-maxcapturerate=48000
a=ssrc:1461109623 msid:user1503200852@host-bd97efb0 webrtctransceiver1
a=ssrc:1461109623 cname:user1503200852@host-bd97efb0
a=mid:audio1
a=fingerprint:sha-256 76:A7:03:41:13:AC:9B:49:B8:79:72:3B:98:F3:62:B4:80:CA:D4:38:63:2C:79:9F:DE:39:58:DA:9C:7C:34:E0
a=rtcp-mux-only

Received SDP:
v=0
o=mozilla...THIS_IS_SDPARTA-88.0.1 7255886693288437605 0 IN IP4 0.0.0.0
s=-
t=0 0
a=sendrecv
a=fingerprint:sha-256 C7:23:2E:0F:71:1C:CD:6C:2B:9F:10:4A:5F:76:E3:78:A0:03:18:0B:57:C7:2B:D2:C4:92:4D:D1:9B:11:0D:51
a=ice-options:trickle
a=msid-semantic:WMS *
m=video 0 UDP/TLS/RTP/SAVPF 120
c=IN IP4 0.0.0.0
a=inactive
a=mid:video0
a=rtpmap:120 VP8/90000
m=audio 9 UDP/TLS/RTP/SAVPF 97
c=IN IP4 0.0.0.0
a=recvonly
a=fmtp:97 maxplaybackrate=48000;stereo=1;useinbandfec=1
a=ice-pwd:bbd0195ca1a312bd4d28bd2caba33505
a=ice-ufrag:e7305812
a=mid:audio1
a=rtcp-mux
a=rtpmap:97 opus/48000/2
a=setup:active
a=ssrc:4046897550 cname:{0d455fa0-77e5-4449-b914-c515a98565ed}



What seems to be the relevant error is then

0:01:12.469235421  6849   0x55ac024b60 WARN               webrtcbin gstwebrtcbin.c:5372:_set_description_task:<webrtcbin> returning error: media 0 is missing or contains an empty 'ice-ufrag' attribute