Raspberry Pi 4 is not working when the camera video size is increased.

[iotsystek]

I am new to Linux and GStreamer.  I am working on getting 4 - 1 mega pixel
USB 2.0 cameras working on a RPi4.  Currently I am using just 2 cameras for
testing.  The cameras are connected via USB 2.0 directly to the Pi (no
external hub).  The RPi4 has 256Mb GPU memory set and is being powered with
a precision bench power supply at 5.05 volts and does not complain about low
voltage.  There is no audio in this project.  My final application will be
embedded into my diesel truck and 5th wheel.

I have 2 different hardware test configurations:
1 A Raspberry Pi 4 with Buster OS dates Dec 2 2020.
        Each camera is connected directly to a Pi USB port.
        GStreamer version is 1.14.4

2 Linux Ubuntu 18.04 running in a virtual machine on my PC.
        1 camera is connected directly to a PC USB 3.0 port.
        1 camera is connected to a PC via a USB 3.0 hub.
        GStreamer version is 1.14.5

The following 2 gst-launch-1.0 commands are being used.  The only difference
between them are the size of the video 320x240 or 640x480.

gst-launch-1.0 videomixer name=mix ! videoconvert ! autovideosink sync=false
\
 v4l2src device=/dev/video0 ! videorate max-rate=6 ! queue ! videoconvert !
video/x-raw, format=AYUV, width=320, height=240 ! videobox border-alpha=0
top=0    left=0    ! mix. \
 v4l2src device=/dev/video2 ! videorate max-rate=6 ! queue ! videoconvert !
video/x-raw, format=AYUV, width=320, height=240 ! videobox border-alpha=0
top=0    left=-320 ! mix.

gst-launch-1.0 videomixer name=mix ! videoconvert ! ximagesink sync=false \
 v4l2src device=/dev/video0 ! videorate max-rate=6 ! queue ! videoconvert !
video/x-raw, format=AYUV, width=640, height=480 ! videobox border-alpha=0
top=0    left=0    ! mix. \
 v4l2src device=/dev/video2 ! videorate max-rate=6 ! queue ! videoconvert !
video/x-raw, format=AYUV, width=640, height=480 ! videobox border-alpha=0
top=0    left=-640 ! mix.

In the following write up my definition of working means that the video
starts up with gst-launch-1.0 and updates as expected when I move the camera
around. (Real time video)

Both of the above commands work on the PC VM.

On the RPi4 the smaller window always works but the larger window only kind
of works sometimes. With the larger window on the RPi4 on startup a window
does not always show up.  If a window does show up it will have only one
camera then after 10 to 20 seconds the second camera shows up.  The video
seems frozen at this point but if I move the cameras the image may update
after a 30 to 60 second or longer delay.

I have looked at the gst-launch-1.0 process and threads on both the RPi4 and
the PC VM.  As expected they look mostly the same but with one exception.
There seems to be a large delta time on the 2 queues. Also the pipeline
images created from the .dot files show no activity on one of the queues.
See Images.

Thanks in advance for any assistance with this problem.

<http://gstreamer-devel.966125.n4.nabble.com/file/t379857/GStreamer_PC_Threads.png>

<http://gstreamer-devel.966125.n4.nabble.com/file/t379857/GStreamer_RPi_Threads.png>

<http://gstreamer-devel.966125.n4.nabble.com/file/t379857/Queue.png>




--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le vendredi 29 janvier 2021 à 11:28 -0600, iotsystek a écrit :

> I am new to Linux and GStreamer.  I am working on getting 4 - 1 mega pixel
> USB 2.0 cameras working on a RPi4.  Currently I am using just 2 cameras for
> testing.  The cameras are connected via USB 2.0 directly to the Pi (no
> external hub).  The RPi4 has 256Mb GPU memory set and is being powered with
> a precision bench power supply at 5.05 volts and does not complain about low
> voltage.  There is no audio in this project.  My final application will be
> embedded into my diesel truck and 5th wheel.
>
> I have 2 different hardware test configurations:
> 1       A Raspberry Pi 4 with Buster OS dates Dec 2 2020.
>         Each camera is connected directly to a Pi USB port.
>         GStreamer version is 1.14.4
>
> 2       Linux Ubuntu 18.04 running in a virtual machine on my PC.
>         1 camera is connected directly to a PC USB 3.0 port.
>         1 camera is connected to a PC via a USB 3.0 hub.
>         GStreamer version is 1.14.5
>
> The following 2 gst-launch-1.0 commands are being used.  The only difference
> between them are the size of the video 320x240 or 640x480.
>
> gst-launch-1.0 videomixer name=mix ! videoconvert ! autovideosink sync=false
> \
>  v4l2src device=/dev/video0 ! videorate max-rate=6 ! queue ! videoconvert !
> video/x-raw, format=AYUV, width=320, height=240 ! videobox border-alpha=0
> top=0    left=0    ! mix. \
>  v4l2src device=/dev/video2 ! videorate max-rate=6 ! queue ! videoconvert !
> video/x-raw, format=AYUV, width=320, height=240 ! videobox border-alpha=0
> top=0    left=-320 ! mix.
>
> gst-launch-1.0 videomixer name=mix ! videoconvert ! ximagesink sync=false \
>  v4l2src device=/dev/video0 ! videorate max-rate=6 ! queue ! videoconvert !
> video/x-raw, format=AYUV, width=640, height=480 ! videobox border-alpha=0
> top=0    left=0    ! mix. \
>  v4l2src device=/dev/video2 ! videorate max-rate=6 ! queue ! videoconvert !
> video/x-raw, format=AYUV, width=640, height=480 ! videobox border-alpha=0
> top=0    left=-640 ! mix.
>
> In the following write up my definition of working means that the video
> starts up with gst-launch-1.0 and updates as expected when I move the camera
> around. (Real time video)
>
> Both of the above commands work on the PC VM.
>
> On the RPi4 the smaller window always works but the larger window only kind
> of works sometimes. With the larger window on the RPi4 on startup a window
> does not always show up.  If a window does show up it will have only one
> camera then after 10 to 20 seconds the second camera shows up.  The video
> seems frozen at this point but if I move the cameras the image may update
> after a 30 to 60 second or longer delay.

This looks like USB bandwidth issue. This is my guess, assuming 640x480 at 30fps
in YUY2 format, if my math is correct, you'll need 147.5 mb/s per stream. That
would requires 295 mb/s for the raw data, considering you need to add the UVC
header overhead. Also, the RPi USB driver is unlikely capable to reach the
theorical maximum of 480 mb/s.

What I'd try is to connect the cameras on the two USB 3.0 port, or one on each
type. If that still does not work, then perhaps you cameras support JPEG
compression, if so, request JPEG and decode in software.

>
> I have looked at the gst-launch-1.0 process and threads on both the RPi4 and
> the PC VM.  As expected they look mostly the same but with one exception.
> There seems to be a large delta time on the 2 queues. Also the pipeline
> images created from the .dot files show no activity on one of the queues.
> See Images.
>
> Thanks in advance for any assistance with this problem.
>
> <
> http://gstreamer-devel.966125.n4.nabble.com/file/t379857/GStreamer_PC_Threads.png
> >
>
> <
> http://gstreamer-devel.966125.n4.nabble.com/file/t379857/GStreamer_RPi_Threads.png
> >
>
> <http://gstreamer-devel.966125.n4.nabble.com/file/t379857/Queue.png>
>
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[iotsystek]

Nicolas Dufresne-5 wrote

> Le vendredi 29 janvier 2021 à 11:28 -0600, iotsystek a écrit :
>> I am new to Linux and GStreamer.  I am working on getting 4 - 1 mega
>> pixel
>> USB 2.0 cameras working on a RPi4.  Currently I am using just 2 cameras
>> for
>> testing.  The cameras are connected via USB 2.0 directly to the Pi (no
>> external hub).  The RPi4 has 256Mb GPU memory set and is being powered
>> with
>> a precision bench power supply at 5.05 volts and does not complain about
>> low
>> voltage.  There is no audio in this project.  My final application will
>> be
>> embedded into my diesel truck and 5th wheel.
>>
>> I have 2 different hardware test configurations:
>> 1       A Raspberry Pi 4 with Buster OS dates Dec 2 2020.
>>         Each camera is connected directly to a Pi USB port.
>>         GStreamer version is 1.14.4
>>
>> 2       Linux Ubuntu 18.04 running in a virtual machine on my PC.
>>         1 camera is connected directly to a PC USB 3.0 port.
>>         1 camera is connected to a PC via a USB 3.0 hub.
>>         GStreamer version is 1.14.5
>>
>> The following 2 gst-launch-1.0 commands are being used.  The only
>> difference
>> between them are the size of the video 320x240 or 640x480.
>>
>> gst-launch-1.0 videomixer name=mix ! videoconvert ! autovideosink
>> sync=false
>> \
>>  v4l2src device=/dev/video0 ! videorate max-rate=6 ! queue ! videoconvert
>> !
>> video/x-raw, format=AYUV, width=320, height=240 ! videobox border-alpha=0
>> top=0    left=0    ! mix. \
>>  v4l2src device=/dev/video2 ! videorate max-rate=6 ! queue ! videoconvert
>> !
>> video/x-raw, format=AYUV, width=320, height=240 ! videobox border-alpha=0
>> top=0    left=-320 ! mix.
>>
>> gst-launch-1.0 videomixer name=mix ! videoconvert ! ximagesink sync=false
>> \
>>  v4l2src device=/dev/video0 ! videorate max-rate=6 ! queue ! videoconvert
>> !
>> video/x-raw, format=AYUV, width=640, height=480 ! videobox border-alpha=0
>> top=0    left=0    ! mix. \
>>  v4l2src device=/dev/video2 ! videorate max-rate=6 ! queue ! videoconvert
>> !
>> video/x-raw, format=AYUV, width=640, height=480 ! videobox border-alpha=0
>> top=0    left=-640 ! mix.
>>
>> In the following write up my definition of working means that the video
>> starts up with gst-launch-1.0 and updates as expected when I move the
>> camera
>> around. (Real time video)
>>
>> Both of the above commands work on the PC VM.
>>
>> On the RPi4 the smaller window always works but the larger window only
>> kind
>> of works sometimes. With the larger window on the RPi4 on startup a
>> window
>> does not always show up.  If a window does show up it will have only one
>> camera then after 10 to 20 seconds the second camera shows up.  The video
>> seems frozen at this point but if I move the cameras the image may update
>> after a 30 to 60 second or longer delay.
>
> This looks like USB bandwidth issue. This is my guess, assuming 640x480 at
> 30fps
> in YUY2 format, if my math is correct, you'll need 147.5 mb/s per stream.
> That
> would requires 295 mb/s for the raw data, considering you need to add the
> UVC
> header overhead. Also, the RPi USB driver is unlikely capable to reach the
> theorical maximum of 480 mb/s.
>
> The Pi is booting from a SSD on a USB 3.0 port, the wireless keyboard and
> mouse are on a USB2.0 port and the 2 cameras are on 1 USB 3.0 and 1 USB2.0
> ports.  I have also tried putting both cameras on a USB 3.0 hub and
> connecting that to the Pi 3.0 port with the same results.
>
> What I'd try is to connect the cameras on the two USB 3.0 port, or one on
> each
> type. If that still does not work, then perhaps you cameras support JPEG
> compression, if so, request JPEG and decode in software.
>
> The cameras can output the following formats:
>
> v4l2-ctl --device /dev/video0 --list-formats-ext
>
> ioctl: VIDIOC_ENUM_FMT
> Type: Video Capture
>
> [0]: 'MJPG' (Motion-JPEG, compressed)
> Size: Discrete 1280x720
> Interval: Discrete 0.033s (30.000 fps)
> Size: Discrete 640x480
> Interval: Discrete 0.033s (30.000 fps)
> Size: Discrete 352x288
> Interval: Discrete 0.033s (30.000 fps)
> Size: Discrete 320x240
> Interval: Discrete 0.033s (30.000 fps)
> Size: Discrete 176x144
> Interval: Discrete 0.033s (30.000 fps)
> Size: Discrete 160x120
> Interval: Discrete 0.033s (30.000 fps)
> Size: Discrete 800x600
> Interval: Discrete 0.033s (30.000 fps)
> Size: Discrete 960x720
> Interval: Discrete 0.033s (30.000 fps)
> [1]: 'YUYV' (YUYV 4:2:2)
> Size: Discrete 1280x720
> Interval: Discrete 0.100s (10.000 fps)
> Size: Discrete 640x480
> Interval: Discrete 0.033s (30.000 fps)
> Size: Discrete 352x288
> Interval: Discrete 0.033s (30.000 fps)
> Size: Discrete 320x240
> Interval: Discrete 0.033s (30.000 fps)
> Size: Discrete 176x144
> Interval: Discrete 0.033s (30.000 fps)
> Size: Discrete 160x120
> Interval: Discrete 0.033s (30.000 fps)
> Size: Discrete 800x600
> Interval: Discrete 0.050s (20.000 fps)
> Size: Discrete 960x720
> Interval: Discrete 0.067s (15.000 fps)
>
> I have been unable to figure out the syntax to configure v4l2src to use
> MJPG.
>
> Also the GStreamer web site says the videomixer is limited to only 3
> different inputs, I quote: “Videomixer can accept AYUV, ARGB and BGRA
> video streams”.  I’m not sure which of these 3 I should use for best
> performance but I have found that if I don’t use one of them, then when
> the window is displayed only the last video source specified is displayed
> with the other 3 are blacked out.  Its like the boarders of the last
> window are blocking out the other videos.
>
>>
>> I have looked at the gst-launch-1.0 process and threads on both the RPi4
>> and
>> the PC VM.  As expected they look mostly the same but with one exception.
>> There seems to be a large delta time on the 2 queues. Also the pipeline
>> images created from the .dot files show no activity on one of the queues.
>> See Images.
>>
>> Thanks in advance for any assistance with this problem.
>>
>> <
>> http://gstreamer-devel.966125.n4.nabble.com/file/t379857/GStreamer_PC_Threads.png
>> >
>>
>> <
>> http://gstreamer-devel.966125.n4.nabble.com/file/t379857/GStreamer_RPi_Threads.png
>> >
>>
>> &lt;http://gstreamer-devel.966125.n4.nabble.com/file/t379857/Queue.png&gt; 
>>
>>
>>
>>
>> --
>> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
>> _______________________________________________
>> gstreamer-devel mailing list
>>

> gstreamer-devel@.freedesktop

>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
> _______________________________________________
> gstreamer-devel mailing list

> gstreamer-devel@.freedesktop

> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel





--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[iotsystek]

I have done further testing.  In both of the listed configurations below:

1) Both USB 2.0 cameras plugged into a USB 3.0 hub. The USB 3.0 hub plugged
into the RPi4 USB 2.0 port.
2) Both USB 2.0 cameras plugged into a USB 3.0 hub. The USB 3.0 hub plugged
into the RPi4 USB 3.0 port.

I ran the following GStreamer command line at max camera resolution (1280 x
720). I only changed the device=/dev/video(0 or 2) so that each camera could
be tested separately in each configuration. In all test cases the camera
provided real time video without any problems.

gst-launch-1.0 videomixer name=mix ! videoconvert ! autovideosink sync=false
\
 v4l2src device=/dev/video0 ! autovideoconvert ! video/x-raw, format=AYUV,
width=1280, height=720 ! videobox border-alpha=0 top=0    left=0    ! mix.

I do not believe the problem is with the USB port throughput.





--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel