No sound in HLS (.ts) generated by GStreamer (h264 + Opus --> MPEG2-TS)

[StreaminBaby]

I have a GStreamer pipeline running on a Raspberry Pi on my home's LAN that is multicasting a UDP video (h264) and audio (opus) stream.

Sending the stream:
 ------------------------------------------------------------------------
gst-launch-1.0 -v rpicamsrc vflip=true hflip=true \
               name=src preview=0 fullscreen=0 bitrate=10000000 \
               annotation-mode=time annotation-text-size=20 \
               ! video/x-h264,width=960,height=540,framerate=24/1 \
               ! h264parse \
               ! rtph264pay config-interval=1 pt=96 \
               ! queue max-size-bytes=0 max-size-buffers=0 \
               ! udpsink host=224.1.1.1 port=5001 auto-multicast=true\
               alsasrc device=plug:dsnooped provide-clock=false \
               ! audio/x-raw,rate=16000 \
                   ! audiorate \
               ! audioconvert \
               ! audioresample \
               ! opusenc \
               ! rtpopuspay \
               ! queue max-size-bytes=0 max-size-buffers=0 \
               ! udpsink host=224.1.1.1 port=5002 auto-multicast=true
 ------------------------------------------------------------------------



Receiving the streams, convert to HLS: I'm also using GStreamer to receive the audio and video streams

 ------------------------------------------------------------------------
VIDEO_CAPS="application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=(string)H264,payload=(int)96"
AUDIO_CAPS="application/x-rtp,media=(string)audio,clock-rate=(int)48000,encoding-name=(string)OPUS"

gst-launch-1.0 -v udpsrc address=224.1.1.1 port=5001 caps=$VIDEO_CAPS \
                ! queue \
                ! rtph264depay \
                ! h264parse \
                ! mpegtsmux name=mux \
                ! hlssink location="/var/www/picam-viewer/hls/%06d.ts" playlist-location="/var/www/picam-viewer/hls/list.m3u8" max-files=5 playlist-length=1 target-duration=5
                udpsrc address=224.1.1.1 port=5002 caps=$AUDIO_CAPS  \
                ! queue \
                ! rtpopusdepay \
                ! opusdec caps="audio/x-raw,rate=48000,channels=2" ! audioconvert ! voaacenc ! aacparse \
                ! mux.
 ------------------------------------------------------------------------

On the receiving side, I have tried many variations for the 2nd to last line (decoding Opus, converting to AAC), but in all cases I end up with HLS where the video works as expected, but there is no audio.



This is the verbose output I get from GStreamer when running the receiving pipeline:

 ------------------------------------------------------------------------
(.venv) pi@picroft:~ $ sudo ./BabySpiCroft-Setup-Files/GStreamer/receive-stream-to-hls.sh
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstUDPSrc:udpsrc0.GstPad:src: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal, parsed=(boolean)true
/GstPipeline:pipeline0/MpegTsMux:mux.GstPad:sink_65: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal, parsed=(boolean)true
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal, width=(int)960, height=(int)540, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)constrained-baseline, level=(string)4
/GstPipeline:pipeline0/MpegTsMux:mux.GstPad:sink_65: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal, width=(int)960, height=(int)540, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)constrained-baseline, level=(string)4
/GstPipeline:pipeline0/MpegTsMux:mux.GstPad:src: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188
/GstPipeline:pipeline0/GstHlsSink:hlssink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188
/GstPipeline:pipeline0/GstHlsSink:hlssink0/GstMultiFileSink:multifilesink0.GstPad:sink: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188
/GstPipeline:pipeline0/GstHlsSink:hlssink0.GstGhostPad:sink: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188
/GstPipeline:pipeline0/MpegTsMux:mux.GstPad:src: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188, streamheader=(buffer)< 47400030a600ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0000b00d0001c100000001e020a2c32941, 474020308b00ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0002b0280001c10000e041f00c050448444d5688040ffffcfc1be041f00a050848444d56ff1b443f5a3175c0 >
/GstPipeline:pipeline0/GstHlsSink:hlssink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188, streamheader=(buffer)< 47400030a600ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0000b00d0001c100000001e020a2c32941, 474020308b00ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0002b0280001c10000e041f00c050448444d5688040ffffcfc1be041f00a050848444d56ff1b443f5a3175c0 >
/GstPipeline:pipeline0/GstHlsSink:hlssink0/GstMultiFileSink:multifilesink0.GstPad:sink: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188, streamheader=(buffer)< 47400030a600ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0000b00d0001c100000001e020a2c32941, 474020308b00ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0002b0280001c10000e041f00c050448444d5688040ffffcfc1be041f00a050848444d56ff1b443f5a3175c0 >
/GstPipeline:pipeline0/GstHlsSink:hlssink0.GstGhostPad:sink: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188, streamheader=(buffer)< 47400030a600ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0000b00d0001c100000001e020a2c32941, 474020308b00ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0002b0280001c10000e041f00c050448444d5688040ffffcfc1be041f00a050848444d56ff1b443f5a3175c0 >
 ------------------------------------------------------------------------

I am unable to tell if there's anything useful in this output. I suspect there's some parameter that needs to be set properly that I am missing, I just don't know what.

Thanks!

[StreaminBaby]

This receiving pipeline did the trick (shoutout to u/thaytan on the GStreamer subreddit!)

------------------------------------------------------------------------
sudo rm /var/www/picam-viewer/hls/*.ts
sudo rm /var/www/picam-viewer/hls/*.m3u8

VIDEO_CAPS="application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=(string)H264,payload=(int)96"
AUDIO_CAPS="application/x-rtp,media=(string)audio,clock-rate=(int)48000,encoding-name=(string)OPUS"

sudo gst-launch-1.0 -v udpsrc address=224.1.1.1 port=5002 caps=$AUDIO_CAPS  \
                ! rtpopusdepay \
                ! opusdec \
                ! audioconvert \
                ! avenc_aac \
                ! queue \
                ! combine_to_hls.audio \
        udpsrc address=224.1.1.1 port=5001 caps=$VIDEO_CAPS \
                ! rtph264depay \
                ! h264parse \
                ! queue \
                ! hlssink2 location="/var/www/picam-viewer/hls/%06d.ts" playlist-location="/var/www/picam-viewer/hls/list.m3u8" max-files=5 playlist-length=3 target-duration=2 name=combine_to_hls
------------------------------------------------------------------------

The key seemed to be switching from `hlssink` to `hlssink2` which took care of the muxing for me.