Putting together a complex gstreamer command line...

[nicko]

Hi,

I suspect this isn't the right place to ask questions about use, rather than development, of gstreamer - I have a complex question that no other forum has been able to help me on, so I would appreciate any pointers as to the best place to ask about using videomixer, multiple cameras (2 x v4l2src), H.264 and streaming over UDP to a remote host... I've got a long way, but I've hit a wall...

NB. I'm not a noob idiot - I'm an EE - it's just that gstreamer is a long way out of my comfort zone !

That and I need help !

Many thanks

Nick

[nicko]

For the sake of completeness, here is the question - I would really appreciate help on this as I'm completely stuck:

I have a Raspberry Pi Compute Module with 2 cameras. I'm trying to use gstreamer with v4l2src selecting /dev/video0 & /dev/video1 to continually run at about 20FPS and use videomixer to combine the images side-by-side then output H.264 over RTP to a UDP port (read by another host). Somewhere in the comand line, I'm going to need to include glshader to apply some distortion to each image before the merge...

The default (current) RPi v4l2src driver does not support two cameras, but as of today a beta is available that does, however it requires the beta 4.4.6 kernel.

The problem I'm having is in getting the mixer connected.

#!/bin/bash -x
#
# Script to start RPi Compute Module streaming over RTP (RFC3984)
# from both cameras
#
FPS=20                          # Frames per second
WIDTH=640                       # Image width
HEIGHT=480                      # Image height
UPLINK_HOST=192.168.1.73        # Receiving host
PORT=5200                       # UDP port
#
# TESTING WITH ONE CAMERA ONLY FOR THE MOMENT
#
function start_streaming
{
  gst-launch-1.0 -ve videomixer name=mixer  \
  ! x264enc \
  ! h264parse \
  ! rtph264pay config-interval=10 pt=96 \
  ! udpsink host=$UPLINK_HOST port=$PORT \
  v4l2src device=/dev/video0 \
  ! video/x-raw,format=AYUV,width=$WIDTH,height=$HEIGHT,framerate=$FPS/1 \
  ! mixer.
}

# Start streaming on both cameras simultaneously
echo Image size: $WIDTH x $HEIGHT
echo Frame rate: $FPS
echo Starting cameras 0 and 1 streaming to $UPLINK_HOST:$PORT
start_streaming

# Wait until everything has finished
wait

exit 0
# end

What I'm getting is the rather useless (to me) message:

WARNING: erroneous pipeline: could not link v4l2src0 to mixer

I've fiddled about rather a lot and got nowhere - it's probably something trivial, but be blowed if I can see it !

Many thanks

Nick

[Nicolas Dufresne-4]

As a first step, you should replace videomixer with compositor. The
compositor element have the same interface but works with live sources
correctly, and implement color conversion internally. Note that this is
all done in software and will quickly max out the CPU capacity.

As you want to apply GL shaders, best is to upload everything to GL in
the first place. This can be done using glupload and glcolorconvert
elements. The equivalent in GL to compositor is glvideomixer. You
should use fairly recent version of GStreamer (something Raspbian might
not offer). Not everything is supported in term of GL feature on this
platform.

Another point I'd like to bring, is that the v4l2 driver on RPI has
never been fantastic. I don't know if anyone have worked on it since,
but some people get better resulting using rpicamsrc, https://github.co
m/thaytan/gst-rpicamsrc

Nicolas

Le dimanche 03 avril 2016 à 05:30 -0700, nicko a écrit :

> For the sake of completeness, here is the question - I would really
> appreciate help on this as I'm completely stuck:
>
> I have a Raspberry Pi Compute Module with 2 cameras. I'm trying to
> use
> gstreamer with v4l2src selecting /dev/video0 & /dev/video1 to
> continually
> run at about 20FPS and use videomixer to combine the images side-by-
> side
> then output H.264 over RTP to a UDP port (read by another host).
> Somewhere
> in the comand line, I'm going to need to include glshader to apply
> some
> distortion to each image before the merge...
>
> The default (current) RPi v4l2src driver does not support two
> cameras, but
> as of today a beta is available that does, however it requires the
> beta
> 4.4.6 kernel.
>
> The problem I'm having is in getting the mixer connected.
>
> #!/bin/bash -x
> #
> # Script to start RPi Compute Module streaming over RTP (RFC3984)
> # from both cameras
> #
> FPS=20                          # Frames per second
> WIDTH=640                       # Image width
> HEIGHT=480                      # Image height
> UPLINK_HOST=192.168.1.73        # Receiving host
> PORT=5200                       # UDP port
> #
> # TESTING WITH ONE CAMERA ONLY FOR THE MOMENT
> #
> function start_streaming
> {
>   gst-launch-1.0 -ve videomixer name=mixer  \
>   ! x264enc \
>   ! h264parse \
>   ! rtph264pay config-interval=10 pt=96 \
>   ! udpsink host=$UPLINK_HOST port=$PORT \
>   v4l2src device=/dev/video0 \
>   ! video/x-
> raw,format=AYUV,width=$WIDTH,height=$HEIGHT,framerate=$FPS/1 \
>   ! mixer.
> }
>
> # Start streaming on both cameras simultaneously
> echo Image size: $WIDTH x $HEIGHT
> echo Frame rate: $FPS
> echo Starting cameras 0 and 1 streaming to $UPLINK_HOST:$PORT
> start_streaming
>
> # Wait until everything has finished
> wait
>
> exit 0
> # end
>
> What I'm getting is the rather useless (to me) message:
>
> WARNING: erroneous pipeline: could not link v4l2src0 to mixer
>
> I've fiddled about rather a lot and got nowhere - it's probably
> something
> trivial, but be blowed if I can see it !
>
> Many thanks
>
> Nick
>
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble
> .com/Putting-together-a-complex-gstreamer-command-line-
> tp4676713p4676714.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[RAJensen]

Hello Nicko.

Don't know so much about G-Streamer, but i did have a little fight with
videomixer. (I won)
Now i've got trouble with a tcpclientsrc & Python....
But I think that your Pipeline is incomplete regarding to your mixer.
First of all: I'm quite sure that it needs that your src use
do-timestamp=true and some where down the pipeline, some Caps must be
provided and some queue might help on roll.
Also importent: Mixer needs to know where to put mixed sources on
display. Like sink_0::xpos=0 sink_0::ypos=0 and display size must match
your 2 x stream width.
Consider to use a videotestsrc as a background with the width,
height,format,framerate and feed it to mixer and then mix your sources.
And use videoconvert to your sources.
I have a Rpi with tcpserversink streaming to a Win7 tcpclientsrc and can
mix 2 streams.
I set Caps according to my video before streams go out and a
videotestsrc as background for the mixer
I've not used v4l2src but raspivid and fdsrc.

Best regards Robin.

Den 03-04-2016 kl. 14:30 skrev nicko:

> For the sake of completeness, here is the question - I would really
> appreciate help on this as I'm completely stuck:
>
> I have a Raspberry Pi Compute Module with 2 cameras. I'm trying to use
> gstreamer with v4l2src selecting /dev/video0 & /dev/video1 to continually
> run at about 20FPS and use videomixer to combine the images side-by-side
> then output H.264 over RTP to a UDP port (read by another host). Somewhere
> in the comand line, I'm going to need to include glshader to apply some
> distortion to each image before the merge...
>
> The default (current) RPi v4l2src driver does not support two cameras, but
> as of today a beta is available that does, however it requires the beta
> 4.4.6 kernel.
>
> The problem I'm having is in getting the mixer connected.
>
> #!/bin/bash -x
> #
> # Script to start RPi Compute Module streaming over RTP (RFC3984)
> # from both cameras
> #
> FPS=20                          # Frames per second
> WIDTH=640                       # Image width
> HEIGHT=480                      # Image height
> UPLINK_HOST=192.168.1.73        # Receiving host
> PORT=5200                       # UDP port
> #
> # TESTING WITH ONE CAMERA ONLY FOR THE MOMENT
> #
> function start_streaming
> {
>    gst-launch-1.0 -ve videomixer name=mixer  \
>    ! x264enc \
>    ! h264parse \
>    ! rtph264pay config-interval=10 pt=96 \
>    ! udpsink host=$UPLINK_HOST port=$PORT \
>    v4l2src device=/dev/video0 \
>    ! video/x-raw,format=AYUV,width=$WIDTH,height=$HEIGHT,framerate=$FPS/1 \
>    ! mixer.
> }
>
> # Start streaming on both cameras simultaneously
> echo Image size: $WIDTH x $HEIGHT
> echo Frame rate: $FPS
> echo Starting cameras 0 and 1 streaming to $UPLINK_HOST:$PORT
> start_streaming
>
> # Wait until everything has finished
> wait
>
> exit 0
> # end
>
> What I'm getting is the rather useless (to me) message:
>
> WARNING: erroneous pipeline: could not link v4l2src0 to mixer
>
> I've fiddled about rather a lot and got nowhere - it's probably something
> trivial, but be blowed if I can see it !
>
> Many thanks
>
> Nick
>
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Putting-together-a-complex-gstreamer-command-line-tp4676713p4676714.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[nicko]

Thanks for the ideas guys.  As I'm very new to this, could anyone give me a complete command line to try?

Thanks

Nick

[RAJensen]

Go through this website / Wiki
You can see how to create your pipe. and use videomixer and more stuff.

Robin

Den 03-04-2016 kl. 16:03 skrev nicko:

Thanks for the ideas guys.  As I'm very new to this, could anyone give me a
complete command line to try?

Thanks

Nick



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Putting-together-a-complex-gstreamer-command-line-tp4676713p4676718.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[RAJensen]

Sorry to say but compositor1.4.4 is A BadPlugin in Gst-1.0.
And also is glvideomixer while videomixer is  Good-Plugin.
Did not have any luck with compositor and glvideomixer creates dark pictures.
I did not use glupload but glimagesink and it seemed to used a bit more CPU than videomixer.

Robin.

Den 03-04-2016 kl. 16:07 skrev Nicolas Dufresne:

As a first step, you should replace videomixer with compositor. The
compositor element have the same interface but works with live sources
correctly, and implement color conversion internally. Note that this is
all done in software and will quickly max out the CPU capacity.

As you want to apply GL shaders, best is to upload everything to GL in
the first place. This can be done using glupload and glcolorconvert
elements. The equivalent in GL to compositor is glvideomixer. You
should use fairly recent version of GStreamer (something Raspbian might
not offer). Not everything is supported in term of GL feature on this
platform.

Another point I'd like to bring, is that the v4l2 driver on RPI has
never been fantastic. I don't know if anyone have worked on it since,
but some people get better resulting using rpicamsrc, https://github.co
m/thaytan/gst-rpicamsrc

Nicolas

Le dimanche 03 avril 2016 à 05:30 -0700, nicko a écrit :

For the sake of completeness, here is the question - I would really
appreciate help on this as I'm completely stuck:

I have a Raspberry Pi Compute Module with 2 cameras. I'm trying to
use
gstreamer with v4l2src selecting /dev/video0 & /dev/video1 to
continually
run at about 20FPS and use videomixer to combine the images side-by-
side
then output H.264 over RTP to a UDP port (read by another host).
Somewhere
in the comand line, I'm going to need to include glshader to apply
some
distortion to each image before the merge...

The default (current) RPi v4l2src driver does not support two
cameras, but
as of today a beta is available that does, however it requires the
beta
4.4.6 kernel.

The problem I'm having is in getting the mixer connected.

#!/bin/bash -x
#
# Script to start RPi Compute Module streaming over RTP (RFC3984)
# from both cameras
#
FPS=20                          # Frames per second
WIDTH=640                       # Image width
HEIGHT=480                      # Image height
UPLINK_HOST=192.168.1.73        # Receiving host
PORT=5200                       # UDP port
#
# TESTING WITH ONE CAMERA ONLY FOR THE MOMENT
#
function start_streaming
{
  gst-launch-1.0 -ve videomixer name=mixer  \
  ! x264enc \
  ! h264parse \
  ! rtph264pay config-interval=10 pt=96 \
  ! udpsink host=$UPLINK_HOST port=$PORT \
  v4l2src device=/dev/video0 \
  ! video/x-
raw,format=AYUV,width=$WIDTH,height=$HEIGHT,framerate=$FPS/1 \
  ! mixer.
}

# Start streaming on both cameras simultaneously
echo Image size: $WIDTH x $HEIGHT
echo Frame rate: $FPS
echo Starting cameras 0 and 1 streaming to $UPLINK_HOST:$PORT
start_streaming

# Wait until everything has finished
wait

exit 0
# end

What I'm getting is the rather useless (to me) message:

WARNING: erroneous pipeline: could not link v4l2src0 to mixer

I've fiddled about rather a lot and got nowhere - it's probably
something
trivial, but be blowed if I can see it !

Many thanks

Nick




--
View this message in context: http://gstreamer-devel.966125.n4.nabble
.com/Putting-together-a-complex-gstreamer-command-line-
tp4676713p4676714.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[nicko]

I now have got a lot further - with running both cameras simultaneously using a new kernel (4.4.6+) and v4l2 driver for the Pi. However, the following gstreamer command just hangs consuming zero CPU - I can't see why. The pipeline goes into PLAYING but CPU usage is 0% and no data is streamed... the videomixer element is just placing the two streams side-by-side...

Thoughts appreciated!

Thanks

Nick
 
FPS=10                          # Frames per second
WIDTH=320                       # Image width
HEIGHT=240                      # Image height
UPLINK_HOST=192.168.1.73        # Receiving host
PORT=5200                       # UDP port
#
# Start streaming data
#
function start_streaming
{
  gst-launch-1.0 -v videomixer name=mixer sink_1::xpos=0 sink_2::xpos=$WIDTH \
  ! x264enc tune=zerolatency \
  ! h264parse \
  ! rtph264pay config-interval=10 pt=96 \
  ! queue \
  ! udpsink host=$UPLINK_HOST port=$PORT \
  v4l2src device=/dev/video0 \
  ! video/x-h264,width=$WIDTH,height=$HEIGHT,framerate=$FPS/1 \
  ! h264parse \
  ! decodebin \
  ! queue \
  ! mixer.sink_1 \
  v4l2src device=/dev/video1 \
  ! video/x-h264,width=$WIDTH,height=$HEIGHT,framerate=$FPS/1 \
  ! h264parse \
  ! decodebin \
  ! queue \
  ! mixer.sink_2
}
start_streamer

[Christophe Lafolet]

Hello Nicko

- videomixer need a video/x-raw format and your camera are configured in
video/x-h264
- In my release, x264enc need video/x-raw { I420, YV12, Y42B, Y444,
NV12, I420_10LE, I422_10LE, Y444_10LE }
so it seems you will need a videoconvert between videomixer and x264enc


Regards




Le 05/04/2016 12:30, nicko a écrit :

> I now have got a lot further - with running both cameras simultaneously using
> a new kernel (4.4.6+) and v4l2 driver for the Pi. However, the following
> gstreamer command just hangs consuming zero CPU - I can't see why. The
> pipeline goes into PLAYING but CPU usage is 0% and no data is streamed...
> the videomixer element is just placing the two streams side-by-side...
>
> Thoughts appreciated!
>
> Thanks
>
> Nick
>  
> FPS=10                          # Frames per second
> WIDTH=320                       # Image width
> HEIGHT=240                      # Image height
> UPLINK_HOST=192.168.1.73        # Receiving host
> PORT=5200                       # UDP port
> #
> # Start streaming data
> #
> function start_streaming
> {
>    gst-launch-1.0 -v videomixer name=mixer sink_1::xpos=0 sink_2::xpos=$WIDTH
> \
>    ! x264enc tune=zerolatency \
>    ! h264parse \
>    ! rtph264pay config-interval=10 pt=96 \
>    ! queue \
>    ! udpsink host=$UPLINK_HOST port=$PORT \
>    v4l2src device=/dev/video0 \
>    ! video/x-h264,width=$WIDTH,height=$HEIGHT,framerate=$FPS/1 \
>    ! h264parse \
>    ! decodebin \
>    ! queue \
>    ! mixer.sink_1 \
>    v4l2src device=/dev/video1 \
>    ! video/x-h264,width=$WIDTH,height=$HEIGHT,framerate=$FPS/1 \
>    ! h264parse \
>    ! decodebin \
>    ! queue \
>    ! mixer.sink_2
> }
> start_streamer
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Putting-together-a-complex-gstreamer-command-line-tp4676713p4676761.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel