ximagesrc and RTP streaming, evaluate solution and performance

[alexxtasi]

Hi all
My use case is grab the screen of a system and stream over the network
(gigabit) to another machine and I am evaluating gstreamer as a solution,
which I think is the only one. Cause I am new in gstreamer, I try some
pipelines based on examples over the web.

So far, I found out that my case includes
- the source machine (mostly with resolution 1080p or 720p and 25fps) :
ximagesrc <--> videoconvert <--> x264enc <--> rtph264pay <--> udpsink
- the receiver:
udpsrc <--> rtph264depay <--> avdec_h264 <--> videoconvert <--> videoscale
<--> ximagesink

and the exact pipelines I've tested, are:
- source
gst-launch-1.0 ximagesrc use-damage=0 \
        ! video/x-raw,framerate=25/1 \
        ! videoscale method=0 \
        ! video/x-raw \
        ! videoconvert \
        ! x264enc \
        ! rtph264pay \
        ! udpsink host=192.168.1.5 port=5601
(for windows machine I used gdiscreencapsrc and video/x-raw,format=I420)

- receiver
gst-launch-1.0 -e udpsrc port=5601 caps="application/x-rtp,
media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264,
payload=96" \
        ! rtpjitterbuffer \
        ! rtph264depay \
        ! avdec_h264 \
        ! videoconvert \
        ! videoscale \
        ! ximagesink sync=false

The results till now are disappointing, the delay is unacceptable. And since
I understand this happening while testing in a single machine (my pc and a
vm inside), this also happens and in a pc - raspberryPi test. Even when
raspberryPi sends low resolution (720 × 480) and framerate (10fps), which I
think is weird !!

Some could argue that raspberryPi isn't power enough for this, but there are
cases that it is a part of a gstreamer pipeline (underwater vehicles
streaming video from rpi to the ground pc using gstreamer).
In such a case I based my pipeline. The main difference is they use v4l2src
on the raspberryPi instead of ximagesrc, but the solution is working.

While searching this mail list, I found some topics about poor performance
in ximagesrc :
-  ximagesrc performance
<http://gstreamer-devel.966125.n4.nabble.com/ximagesrc-performance-td4657884.html>  
-  ximagesrc performace issue
<http://gstreamer-devel.966125.n4.nabble.com/ximagesrc-performace-issue-td4658073.html>  
-  ximagesrc at 144 fps
<http://gstreamer-devel.966125.n4.nabble.com/ximagesrc-at-144-fps-td4694010.html>  
as far as I understand from these, ximagesrc is old and also has many tricks
as "use-damage=false".

My questions are:
1- those delays, are the result of bad configuration of my pipelines ? (eg
use of rtpjitterbuffer, videoscale method=0, etc)
2- if yes, what pipeline should I use ?
3- is ximagesrc the appropriate element for my use case and  also is this
use case feasible at all ?

thanks in advance



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Daniel Johnson]

On Mon, May 4, 2020, 2:15 AM alexxtasi <[hidden email]> wrote:

My questions are:
1- those delays, are the result of bad configuration of my pipelines ? (eg
use of rtpjitterbuffer, videoscale method=0, etc)
2- if yes, what pipeline should I use ?
3- is ximagesrc the appropriate element for my use case and  also is this
use case feasible at all ?

thanks in advance

I've got a stream like this working from a raspberry pi 4 with 4 gigs of ram. I'm using the hardware h264 encoder on the raspberry pi using the v4l2 encoder element. I'm not sure if the new driver is the default yet or if you need to manually turn it on to make the encoder available from Gstreamer. It also has some weird bugs like having blue and red swapped in the recording. I fixed it with an extra capsfilter it shouldn't need. For 25 fps setting the monitor refresh rate to 25 or 50 fps might help slightly. 

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Arnaud Loonstra]

I've done a setup like that with h264 rtp streaming with very little
latency (hardly visible). It was with lower resolutions however! What
definitely matters is using x264enc zerolatency option!

On 03-05-2020 20:41, alexxtasi wrote:

> Hi all
> My use case is grab the screen of a system and stream over the network
> (gigabit) to another machine and I am evaluating gstreamer as a solution,
> which I think is the only one. Cause I am new in gstreamer, I try some
> pipelines based on examples over the web.
>
> So far, I found out that my case includes
> - the source machine (mostly with resolution 1080p or 720p and 25fps) :
> ximagesrc <--> videoconvert <--> x264enc <--> rtph264pay <--> udpsink
> - the receiver:
> udpsrc <--> rtph264depay <--> avdec_h264 <--> videoconvert <--> videoscale
> <--> ximagesink
>
> and the exact pipelines I've tested, are:
> - source
> gst-launch-1.0 ximagesrc use-damage=0 \
> ! video/x-raw,framerate=25/1 \
> ! videoscale method=0 \
> ! video/x-raw \
> ! videoconvert \
> ! x264enc \
> ! rtph264pay \
> ! udpsink host=192.168.1.5 port=5601
> (for windows machine I used gdiscreencapsrc and video/x-raw,format=I420)
>
> - receiver
> gst-launch-1.0 -e udpsrc port=5601 caps="application/x-rtp,
> media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264,
> payload=96" \
> ! rtpjitterbuffer \
> ! rtph264depay \
> ! avdec_h264 \
> ! videoconvert \
> ! videoscale \
> ! ximagesink sync=false
>
> The results till now are disappointing, the delay is unacceptable. And since
> I understand this happening while testing in a single machine (my pc and a
> vm inside), this also happens and in a pc - raspberryPi test. Even when
> raspberryPi sends low resolution (720 × 480) and framerate (10fps), which I
> think is weird !!
>
> Some could argue that raspberryPi isn't power enough for this, but there are
> cases that it is a part of a gstreamer pipeline (underwater vehicles
> streaming video from rpi to the ground pc using gstreamer).
> In such a case I based my pipeline. The main difference is they use v4l2src
> on the raspberryPi instead of ximagesrc, but the solution is working.
>
> While searching this mail list, I found some topics about poor performance
> in ximagesrc :
> -  ximagesrc performance
> <http://gstreamer-devel.966125.n4.nabble.com/ximagesrc-performance-td4657884.html>
> -  ximagesrc performace issue
> <http://gstreamer-devel.966125.n4.nabble.com/ximagesrc-performace-issue-td4658073.html>
> -  ximagesrc at 144 fps
> <http://gstreamer-devel.966125.n4.nabble.com/ximagesrc-at-144-fps-td4694010.html>
> as far as I understand from these, ximagesrc is old and also has many tricks
> as "use-damage=false".
>
> My questions are:
> 1- those delays, are the result of bad configuration of my pipelines ? (eg
> use of rtpjitterbuffer, videoscale method=0, etc)
> 2- if yes, what pipeline should I use ?
> 3- is ximagesrc the appropriate element for my use case and  also is this
> use case feasible at all ?
>
> thanks in advance
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[alexxtasi]

@Daniel Johnson and @Arnaud Loonstra thanks for the replies.

I have to make clear that RaspberryPi is not a requirement for me, just a
part of my tests.
The only requirement is "grab the screen".


Daniel Johnson wrote
> I've got a stream like this working from a raspberry pi 4 with 4 gigs of
> ram. I'm using the hardware h264 encoder on the raspberry pi using the
> v4l2 encoder element. I'm not sure if the new driver is the default yet or
> if you need to manually turn it on to make the encoder available from
> Gstreamer. It also has some weird bugs like having blue and red swapped in
> the recording. I fixed it with an extra capsfilter it shouldn't need. For
> 25 fps setting the monitor refresh rate to 25 or 50 fps might help
> slightly.

I understand you use v4l2src while I am using ximagesrc to grab the display.
Could you please share the pipeline so I can compare to mine and inspect
faults on my side ?


Arnaud Loonstra wrote
> I've done a setup like that with h264 rtp streaming with very little
> latency (hardly visible). It was with lower resolutions however! What
> definitely matters is using x264enc zerolatency option!

Hey, using "x264enc tune=zerolatency" made a huge improvement !!!! Thanks !!

So is this they way to go (stick with ximagesrc use-damage, x264enc
tune=zerolatency) ?

gst-launch-1.0 ximagesrc use-damage=0 \
        ! video/x-raw,framerate=30/1 \
        ! videoscale method=0 \
        ! video/x-raw \
        ! videoconvert \
        ! x264enc tune=zerolatency\
        ! rtph264pay \
        ! udpsink host=192.168.1.5 port=5601

should I configure anything else ?

Any advice ?
regards



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Arnaud Loonstra]

[snip]

> So is this they way to go (stick with ximagesrc use-damage, x264enc
> tune=zerolatency) ?
>
> gst-launch-1.0 ximagesrc use-damage=0 \
>          ! video/x-raw,framerate=30/1 \
>          ! videoscale method=0 \
>          ! video/x-raw \
>          ! videoconvert \
>          ! x264enc tune=zerolatency\
>          ! rtph264pay \
>          ! udpsink host=192.168.1.5 port=5601
>
> should I configure anything else ?
>
> Any advice ?
> regards
>

I wrote a guide ages ago:
https://z25.org/static/_rd_/videostreaming_intro_plab/index.html

I used this pipeline:

gst-launch-1.0 v4l2src ! \
  video/x-raw,width=640,height=480 ! \
  x264enc tune=zerolatency byte-stream=true \
  bitrate=3000 threads=2 ! \
  h264parse config-interval=1 ! \
  rtph264pay ! udpsink host=127.0.0.1 port=5000

I think nowadays you'll need to add a videoconvert for the v4l2src.
Should work with ximagesrc as well.

Rg,

Arnaud
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[alexxtasi]

Really thanks Arnaud,
that "tune" parameter did the trick for me (tune=zerolatency) !!

I'll keep searching for the params you used (byte-stream, bitrate and also
h264parse's config-interval sound interesting) and of course read your
guide.
And I'll come back if have any other question

regards,
Alex



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[alexxtasi]

Hi again !
following suggestions above, I end up with this pipeline on the "source"
pc's:

gst-launch-1.0 ximagesrc use-damage=true \
        ! video/x-raw,framerate=30/1 \
        ! videoscale method=0 \
        ! videoconvert \
        ! x264enc tune=zerolatency \
        ! rtph264pay \
        ! udpsink host=192.168.1.10 port=5601

in my tests I use two "source" machines, one running linux (with the above
script) on Intel i5 and the other running windows (on Intel i3) :

gst-launch-1.0.exe -e gdiscreencapsrc ! video/x-raw,framerate=30/1 !
videoscale method=0 ! videoconvert ! video/x-raw,format=I420 ! x264enc
tune=zerolatency ! rtph264pay ! udpsink host=192.168.1.10 port=5602

The "receiver" pc (linux) runs a pipeline showing those two streams side by
side in a videomixer.

This setup seams to work fine, apart from some delays in the frame coming
from the pc with the less powerful CPU (i3) and the strange ... viewing that
same stream in both two places on the the videomixer !!!!

- are those delays and strange things the result of low power CPU ?
- do I need to use Intel i5 or similar CPU in order to screen-grab and
stream without delays ??

regards


...
my testing pipeline on the "receiver" side is:

gst-launch-1.0 -e videomixer name=mixer ! videoconvert ! xvimagesink
sync=false \
        videotestsrc pattern="snow" !
"video/x-raw,framerate=10/1,width=1920,height=1080" ! textoverlay text="Live
from Pluto" valignment=top halignment=left font-desc="Sans, 6"
shaded-background=true auto-resize=true ! clockoverlay text="Edge City"
valignment=top halignment=right font-desc="Sans, 6" shaded-background=true
auto-resize=true ! alpha alpha=1.0 ! mixer. \
        udpsrc port=5601 caps="application/x-rtp, media=(string)video,
clock-rate=(int)90000, encoding-name=(string)H264, payload=96" \
        ! rtph264depay \
        ! avdec_h264 \
        ! videoconvert \
        ! videoscale \
        ! video/x-raw,width=960,height=540 \
        ! textoverlay text="CAM1" valignment=top halignment=left font-desc="Sans,
10" shaded-background=true auto-resize=true \
        ! alpha alpha=1.0 \
        ! videobox top=-270 left=-0 border-alpha=0 ! mixer. \
        udpsrc port=5602 caps="application/x-rtp, media=(string)video,
clock-rate=(int)90000, encoding-name=(string)H264, payload=96" \
        ! rtph264depay \
        ! avdec_h264 \
        ! videoconvert \
        ! videoscale \
        ! video/x-raw,width=960,height=540 \
        ! textoverlay text="CAM2" valignment=top halignment=left font-desc="Sans,
10" shaded-background=true auto-resize=true \
        ! alpha alpha=1.0 \
        ! videobox top=-270 left=-960 border-alpha=0 ! mixer.
...



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel