WebRTC send audio/video with vp8 from RaspberryPi

[s0h0]

Hello,

Im sending my camera video stream to janus signaling server with this pipeline for h264:
raspivid -n -w 640 -h 480 -b 100000 -fps 2 -vf -hf -t 0 -pf baseline -o - | gst-launch-1.0 -v fdsrc fd=0 ! h264parse !  rtph264pay config-interval=2 pt=126 ! udpsink host=137.116.196.147 port=8004

In order to support audio i would like to change the streaming to VP8 with audio support.

I have tried following, but it shows "WARNING: erroneous pipeline: could not link fdsrc0 to videotestsrc0":
raspivid -n -w 640 -h 480 -b 100000 -fps 2 -vf -hf -t 0 -pf baseline -o - | gst-launch-1.0 -v fdsrc fd=0 ! videotestsrc ! video/x-raw-rgb,framerate=30/1 ! videoconvert ! vp8enc min_quantizer=13 cpu-used = 5 ! rtpvp8pay !  udpsink host=137.116.196.147 port=8004


Can anyone help me building the pipeline for that task?


Thank you,


Simon

[Wonchul Lee]

Hello,

I didn’t understand perfectly what you want to do, but the problem is,
you tried to link source element to source element. (fdsrc fd=0 ! videotestsrc)
The source element only has a src pad, so it just can link with downstream element.
If it is not intended you can easily fix it by removing videotestsrc.

> On Aug 7, 2015, at 10:15 AM, s0h0 <[hidden email]> wrote:
>
> Hello,
>
> Im sending my camera video stream to janus signaling server with this
> pipeline for h264:
> raspivid -n -w 640 -h 480 -b 100000 -fps 2 -vf -hf -t 0 -pf baseline -o - |
> gst-launch-1.0 -v fdsrc fd=0 ! h264parse !  rtph264pay config-interval=2
> pt=126 ! udpsink host=137.116.196.147 port=8004
>
> In order to support audio i would like to change the streaming to VP8 with
> audio support.
>
> I have tried following, but it shows "WARNING: erroneous pipeline: could not
> link fdsrc0 to videotestsrc0":
> raspivid -n -w 640 -h 480 -b 100000 -fps 2 -vf -hf -t 0 -pf baseline -o - |
> gst-launch-1.0 -v fdsrc fd=0 ! videotestsrc ! video/x-raw-rgb,framerate=30/1
> ! videoconvert ! vp8enc min_quantizer=13 cpu-used = 5 ! rtpvp8pay !  udpsink
> host=137.116.196.147 port=8004
>
>
> Can anyone help me building the pipeline for that task?
>
>
> Thank you,
>
>
> Simon
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/WebRTC-send-audio-video-with-vp8-from-RaspberryPi-tp4673026.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[s0h0]

Hello Wonchul,

thank you for your response. I tried that, if i remove the videotestsrc it says:
WARNING: erroneous pipeline: could not link fdsrc0 to videoconvert0

If i then also remove the videoconvert from pipeline, it says "WARNING: erroneous pipeline: could not link fdsrc0 to vp8enc0"

What i try to do, is sending a Video and Audio Stream from my RaspberryPI Camera Modul and the Microphone on the external Soundcard, to a WebRTC Signaling Server.

Any help is very much appreciated!


Best regards,

Simon

[Eric-BTG]

Hi,

raspivid outputs h264 payload isn't it?

You should either output raw video and then encode it in VP8 or either (to test): parse h264, decode h264 and then encode it in VP8..

Here you are trying to encode in VP8 h264 payloads.. which leads to erroneous pipeline..

Eric

[s0h0]

Ahh, thank you for that hint.
I will try doing that.

You have maybe some helpfull ideas how to do that? :)

[Nicolas Dufresne-3]

Le jeudi 06 août 2015 à 18:15 -0700, s0h0 a écrit :
> Hello,
>
> Im sending my camera video stream to janus signaling server with this
> pipeline for h264:
> raspivid -n -w 640 -h 480 -b 100000 -fps 2 -vf -hf -t 0 -pf baseline
> -o - |
> gst-launch-1.0 -v fdsrc fd=0 ! h264parse !  rtph264pay config
> -interval=2
> pt=126 ! udpsink host=137.116.196.147 port=8004

I would like to recommand using rpicamsrc from
https://github.com/thaytan/gst-rpicamsrc . It's more efficient.

>
> In order to support audio i would like to change the streaming to VP8
> with
> audio support.

VP8 is a video codec, it's not going to help you with audio. What you
should do is simply to capture audio, encode it the way you want (like
opusenc) RTP payload it and send it over it's own port. I'm not sure
if/how janus will figure-out, I support you are crafting an SDP ?

Nicolas

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Eric-BTG]

Try this pipeline :)

raspivid -n -w 640 -h 480 -b 100000 -fps 2 -vf -hf -t 0 -pf baseline -o - | gst-launch-1.0 -v fdsrc fd=0 ! h264parse ! decodebin ! video/x-raw-rgb,framerate=30/1 ! videoconvert ! vp8enc min_quantizer=13 cpu-used = 5 ! rtpvp8pay !  udpsink host=137.116.196.147 port=8004

Again, use it as a test only, as Nicolas sait this is not efficient.. (Moreover it fixes your pipeline issue, but does not resolve your problem..).

Eric

[s0h0]

Thank you for the help.

Main issue was the audio for me, now it is working for me with audio and video.
Looks like h264 is much simpler to use, and VP8 wont have a good performance on RaspberryPi.

I have a low badwidth of 1720kBit/s downstream and 400kBit/s upstream. I get a low of NACKS in Janus and gstream shows me an error after a while. Might the badwitdth the problem? Have you andy idea how to improve performance of the pipes?

New clock: GstSystemClock
WARNING: from element /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: Can't record audio fast enough
Additional debug info:
gstaudiobasesrc.c(857): gst_audio_base_src_create (): /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0:
Dropped 4240 samples. This is most likely because downstream can't keep up and is consuming samples too slowly.
 


This is how audio and video is working for me (video has a lot freezes and audio i can hear just fragments...)
#!/bin/bash -e

gst-launch-1.0 -v alsasrc device=plughw:Set ! mulawenc ! rtppcmupay ! udpsink host=137.116.196.147 port=8003 &

raspivid -n -w 640 -h 480 -b 100000 -fps 2 -vf -hf -t 0 -pf baseline -o - | gst-launch-1.0 -v fdsrc fd=0 ! h264parse !  rtph264pay config-interval=2 pt=126 ! udpsink host=137.116.196.147 port=8005



Thank you!

[Jan Schmidt-6]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512



On 08/08/15 11:36, s0h0 wrote:

> Thank you for the help.
>
> Main issue was the audio for me, now it is working for me with
> audio and video. Looks like h264 is much simpler to use, and VP8
> wont have a good performance on RaspberryPi.
>
> I have a low badwidth of 1720kBit/s downstream and 400kBit/s
> upstream. I get a low of NACKS in Janus and gstream shows me an
> error after a while. Might the badwitdth the problem? Have you andy
> idea how to improve performance of the pipes?

mulawenc at 8khz, 8-bit is 64 kbit of bandwidth, plus overhead. With
100kbit/s for the video, you should have enough upstream bandwidth
(although I'd expect the video quality to be terrible. Maybe at 2fps,
it'll be OK)

Try adding a queue into the audio chain, between alsasrc and mulawenc.
That will decouple capture from transmission into 2 threads.

> New clock: GstSystemClock WARNING: from element
> /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: Can't record audio fast
> enough Additional debug info: gstaudiobasesrc.c(857):
> gst_audio_base_src_create ():
> /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: Dropped 4240 samples.
> This is most likely because downstream can't keep up and is
> consuming samples too slowly.
>
>
>
> This is how audio and video is working for me (video has a lot
> freezes and audio i can hear just fragments...) #!/bin/bash -e /
> gst-launch-1.0 -v alsasrc device=plughw:Set ! mulawenc ! rtppcmupay
> ! udpsink host=137.116.196.147 port=8003 &
>
> raspivid -n -w 640 -h 480 -b 100000 -fps 2 -vf -hf -t 0 -pf
> baseline -o - | gst-launch-1.0 -v fdsrc fd=0 ! h264parse !
> rtph264pay config-interval=2 pt=126 ! udpsink host=137.116.196.147
> port=8005/

As suggested elsewhere, switch to using gst-rpicamsrc instead of
raspivid + fdsrc. You'll get lower CPU usage by not copying everything
through a file descriptor, and less inter-frame jitter that way - not
that that probably matters at 2 fps.

You can put both audio and video into one gst-launch-1.0 line, so they
share a clock and should retain sync over long periods of time.

gst-launch-1.0 -v alsasrc device=plughw:Set ! queue ! mulawenc !
rtppcmupay ! udpsink host=137.116.196.147 port=8003 \
rpicamsrc bitrate=100000 hflip=true vflip=true preview=false !
video/x-h264,profile=baseline,width=640,height=480,framerate=2/1 !
h264parse ! rtph264pay config-interval=2 pt=126 ! udpsink
host=137.116.196.147 port=8005

should do it.

> raspivid -n -w 640 -h 480 -b 100000 -fps 2 -vf -hf -t 0 -pf
> baseline
- -o - |
> gst-launch-1.0 -v fdsrc fd=0 ! h264parse !  rtph264pay
config-interval=2 pt=126 ! udpsink host=137.116.196.147 port=8005


>
>
> Thank you!
>
>
>
> -- View this message in context:
> http://gstreamer-devel.966125.n4.nabble.com/WebRTC-send-audio-video-wi
th-vp8-from-RaspberryPi-tp4673026p4673059.html
>
>
Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________ gstreamer-devel
> mailing list [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCgAGBQJVx3c/AAoJEEqdx1eGJfOeRaMP/11nAW4GKJcHUUtBo1S0UcG6
PiQzE1BdsjlWifGxh8V1S9VGS//szpKvMDDq/ei2cb38pjzZ2WAbjPL3bK0Ay5m3
W7Mm41sFMv9bfoHro/0gquiS13dSuk02qKx96LozQ4y4HGVPNxI/nq0/g71h6+GL
xvYymfx4MCRJroqetG79GvAKp0zTMIIBSYdTFMu2L57W9BDGj9WKDhsvTc3n8i+S
Ex2z0ugXqniZKtOse82QLM+lxgEa5mZLrftySZMuwojuqyh30rzdEyReQZxCtoaw
CmvMJJydwaNoV26HPhJRiuoCd5qy0Ky2LWYdX4QskUgiWJX9gcbOmv94jwy1fa1u
G7NZo0dA4OxmM2ttJyNUgDxWS+qhk8wRZxTzP1h9+3qB75UMzOnM/TGRnBXbzWjn
A5JBriCBiRqIEmvoQiE3TftrfAfUFwmRf8SLIHmY6n7b5BFosa5JFeHSW+eIB51G
ms5bWBhiClCrl2yoSmf6QLy1NHoqwCjgEIRgWGth8Rz4JR3qOCPWl6u5aOHg8+JM
/OLHodHPlMbUI0dWJC7c5UoQNb00Q39sXe05CfRXMQl6hY/ccmLRMDFwSpZwTD2m
e79HdBuwIGLgKZfwztwCSATUgpnnnucwHKQ8e6lKy00iiVrmrUmsg0FlBUCqmXAH
YOke7vVrhY5NnwSRuUH8
=a4cW
-----END PGP SIGNATURE-----
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[s0h0]

Wow, thank you for that great help!

When i try your command it says:
WARNING: from element /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: Can't record audio fast enough
Additional debug info:
gstaudiobasesrc.c(857): gst_audio_base_src_create (): /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0:
Dropped 10000 samples. This is most likely because downstream can't keep up and is consuming samples too slowly.


After some research i found that using pulsesrc (can not be executed via ssh...) instead of alsasrc should fix the problem. If i use pulsesrc, with correct device name as string, signaling server only receives the audio signal and dont start playing audio or video.

Do you have an idea why downstream cant keep up?


My SignalServer Konfig looks like that:
[gstreamer-monitor-h264]
type = rtp
id = 2
description = H.264 live stream coming from gstreamer
audio = yes
video = yes
audioport = 8003
audiopt = 111
audiortpmap = opus/48000/2
;audiortpmap = opus/16000/1
videoport = 8005
videopt = 126


Thank you!

[s0h0]

Hello,

i have tried many combinations so far but none of them is working.
Do you maybe have one more tip how to solve the issue?


Thanks a lot,

Simon

[Avnerus]

Not sure if you still need help on this, but since the audio encoding used by WebRTC is Opus, you need to use opusenc ! rtpopuspay and not mulawenc !rtppcmupay.
Try this one:

gst-launch-1.0 pulsesrc ! queue ! opusenc ! rtpopuspay ! udpsink host=127.0.0.1 port=8003 rpicamsrc
preview=false vflip=true ! x264enc tune=zerolatency speed-preset=ultrafast qp-min=18 pass=5 quantizer=21 !
video/x-h264,width=640,height=480,framerate=30/1 ! rtph264pay ! udpsink host=127.0.0.1 port=8004 sync=false