IMX6 : Video streaming over serial port using Gstreamer and H264 encoder

[sk_gst]

Hello GStreamer Experts,

Recently, I have started to work on a project that aims at live video
streaming applications based on imx6 processors. A quick description of what
I have done so far and what I am trying to do:

1. Setup: imx6 board(Boundary devices Sabre Lite) acting as the video
server(using GStreamer imx plugins), and a PC running Ubuntu on it that
receives data from imx6 and streams the video using GStreamer features.
2. On the imx6 processor, I run the 'testvideosrc' and have successfully
streamed it using the RTP over UDP.
3. Since accessing the serial port is similar to writing and reading a file
in Linux, next I tried out dumping the video data from imx6 board to a
serial port and reading this serial port on the PC. For this, the baud rate
of both devices was configured to 115200 baud. The encoding 'bitrate' is
configured to 5Kbps. Here are the commands:

imx6: gst-launch-1.0 -v videotestsrc pattern=18 !
video/x-raw,width=100,height=50 ! imxvpuenc_h264 bitrate=5 ! h264parse !
filesink location=/dev/ttyUSB1

PC: CAPS=video/x-h264
PC:  gst-launch-1.0 -v filesrc location=/dev/ttyUSB1 ! $CAPS ! h264parse !
avdec_h264 ! autovideosink sync=true

There are no errors observed on the imx6 board.
However, I see the following errors at the PC side :
PC:# GST_DEBUG=3 gst-launch-1.0 -v filesrc location=/dev/ttyUSB1 ! $CAPS !
h264parse ! avdec_h264 ! autovideosink sync=true
 Setting pipeline to PAUSED …
0:00:00.066439392 15475 0x556d8a01d160 WARN basesrc
gstbasesrc.c:3583:gst_base_src_start_complete: pad not activated yet
 Pipeline is PREROLLING …
0:00:21.730466251 15475 0x556d8a000940 WARN capsfilter
gstcapsfilter.c:455:gst_capsfilter_prepare_buf: error: Filter caps do not
completely specify the output format
 0:00:21.730523691 15475 0x556d8a000940 WARN capsfilter
gstcapsfilter.c:455:gst_capsfilter_prepare_buf: error: Output caps are
unfixed: video/x-h264, width=(int)[ 1, 8192 ], height=(int)[ 1, 8192 ],
framerate=(fraction)[ 0/1, 2147483647/1 ]
 0:00:21.730676173 15475 0x556d8a000940 WARN basetransform
gstbasetransform.c:2159:default_generate_output: could not get buffer from
pool: error
 0:00:21.730742223 15475 0x556d8a000940 WARN basesrc
gstbasesrc.c:3055:gst_base_src_loop: error: Internal data stream error.
 0:00:21.730775478 15475 0x556d8a000940 WARN basesrc
gstbasesrc.c:3055:gst_base_src_loop: error: streaming stopped, reason error
(-5)
 ERROR: from element /GstPipeline:pipeline0/GstCapsFilter:capsfilter0:
Filter caps do not completely specify the output format
 Additional debug info:
 gstcapsfilter.c(455): gst_capsfilter_prepare_buf ():
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0:
 Output caps are unfixed: video/x-h264, width=(int)[ 1, 8192 ],
height=(int)[ 1, 8192 ], framerate=(fraction)[ 0/1, 2147483647/1 ]
 ERROR: pipeline doesn’t want to preroll.
 Setting pipeline to NULL …
Freeing pipeline …


4. Since, the encoding rate is 5Kbps (bitrate=5, as specified in the above
command), I suppose it is possible to send this amount of data via the
serial port. I realize that currently, the caps negotiation fails, however,
I am unsure of how to proceed with this.

5. On the PC side, reading the serial port with 'cat /dev/ttyUSB1' succeeds
with limited data. The data is unreadable(as expected), however it is not a
continuous stream.

Does anyone have an idea, on how to solve this.
My later test, would be to use an actual camera(MIPI) and try streaming it
over serial port. Does it seem feasible or is a completely crazy idea to do?



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[James Cameron]

On Mon, Jun 25, 2018 at 05:39:42AM -0700, vk_gst wrote:
> Hello GStreamer Experts,

I'm a serial port expert.  ;-)

> Recently, I have started to work on a project that aims at live
> video streaming applications based on imx6 processors. A quick
> description of what I have done so far and what I am trying to do:
>
> 1. Setup: imx6 board(Boundary devices Sabre Lite) acting as the
> video server(using GStreamer imx plugins), and a PC running Ubuntu
> on it that receives data from imx6 and streams the video using
> GStreamer features.
> 2. On the imx6 processor, I run the 'testvideosrc' and have
> successfully streamed it using the RTP over UDP.
> 3. Since accessing the serial port is similar to writing and reading
> a file in Linux,

Yes, to a certain extent.  There are some differences in behaviour;
they have a write rate in bits per second bounded by the baud rate,
and their read rate is bounded by the baud rate.  Transmit and receive
buffers change the rate in the very short term.  When an application
hits the rate limit, I/O is stalled.

> next I tried out dumping the video data from imx6 board to a serial
> port and reading this serial port on the PC. For this, the baud rate
> of both devices was configured to 115200 baud.

You might test the effective baud rates of each port are compatible,
that the drivers are implemented correctly, and that the electrical
edges of the signals are intact and slewing correctly.  Prepare a
large test set of random numbers; about 30 seconds worth.  Configure
the ports for raw data using stty, if not already done.  Use pv or cat
to send and receive the test set.  Use a binary comparison program to
look for changes or gaps.

> The encoding 'bitrate' is configured to 5Kbps. Here are the
> commands:
>
> imx6: gst-launch-1.0 -v videotestsrc pattern=18 !
> video/x-raw,width=100,height=50 ! imxvpuenc_h264 bitrate=5 ! h264parse !
> filesink location=/dev/ttyUSB1
>
> PC: CAPS=video/x-h264
> PC:  gst-launch-1.0 -v filesrc location=/dev/ttyUSB1 ! $CAPS ! h264parse !
> avdec_h264 ! autovideosink sync=true
>
> There are no errors observed on the imx6 board.
> However, I see the following errors at the PC side :
> PC:# GST_DEBUG=3 gst-launch-1.0 -v filesrc location=/dev/ttyUSB1 ! $CAPS !
> h264parse ! avdec_h264 ! autovideosink sync=true
>  Setting pipeline to PAUSED …
> 0:00:00.066439392 15475 0x556d8a01d160 WARN basesrc
> gstbasesrc.c:3583:gst_base_src_start_complete: pad not activated yet
>  Pipeline is PREROLLING …
> 0:00:21.730466251 15475 0x556d8a000940 WARN capsfilter
> gstcapsfilter.c:455:gst_capsfilter_prepare_buf: error: Filter caps do not
> completely specify the output format
>  0:00:21.730523691 15475 0x556d8a000940 WARN capsfilter
> gstcapsfilter.c:455:gst_capsfilter_prepare_buf: error: Output caps are
> unfixed: video/x-h264, width=(int)[ 1, 8192 ], height=(int)[ 1, 8192 ],
> framerate=(fraction)[ 0/1, 2147483647/1 ]
>  0:00:21.730676173 15475 0x556d8a000940 WARN basetransform
> gstbasetransform.c:2159:default_generate_output: could not get buffer from
> pool: error
>  0:00:21.730742223 15475 0x556d8a000940 WARN basesrc
> gstbasesrc.c:3055:gst_base_src_loop: error: Internal data stream error.
>  0:00:21.730775478 15475 0x556d8a000940 WARN basesrc
> gstbasesrc.c:3055:gst_base_src_loop: error: streaming stopped, reason error
> (-5)
>  ERROR: from element /GstPipeline:pipeline0/GstCapsFilter:capsfilter0:
> Filter caps do not completely specify the output format
>  Additional debug info:
>  gstcapsfilter.c(455): gst_capsfilter_prepare_buf ():
> /GstPipeline:pipeline0/GstCapsFilter:capsfilter0:
>  Output caps are unfixed: video/x-h264, width=(int)[ 1, 8192 ],
> height=(int)[ 1, 8192 ], framerate=(fraction)[ 0/1, 2147483647/1 ]
>  ERROR: pipeline doesn’t want to preroll.
>  Setting pipeline to NULL …
> Freeing pipeline …

Does this also occur if the data is saved to a file on the imx6 and
then read from file on the PC?

> 4. Since, the encoding rate is 5Kbps (bitrate=5, as specified in the
> above command), I suppose it is possible to send this amount of data
> via the serial port.

Yes, as it should be less than 115200 baud.  But you could also prove
it by counting bytes over a measured time.

> I realize that currently, the caps negotiation fails, however,
> I am unsure of how to proceed with this.

I've no suggestion on that.

> 5. On the PC side, reading the serial port with 'cat /dev/ttyUSB1'
> succeeds with limited data. The data is unreadable(as expected),
> however it is not a continuous stream.

It should not be continuous.  Your imx6 pipeline constrains the
bitrate of encoding output 5000 bits per second, and 115200 bits per
second is a faster clock, therefore the data will come in short
bursts.  The serial link will spend time idle.

> Does anyone have an idea, on how to solve this.  My later test,
> would be to use an actual camera(MIPI) and try streaming it over
> serial port. Does it seem feasible or is a completely crazy idea to
> do?

Properly working serial ports can transmit and receive digital video;
but I'm not confident that GStreamer elements can handle the unusual
time-based behaviours of serial ports, as compared to files.  I'd hope
they can, but I've not tried it.

--
James Cameron
http://quozl.netrek.org/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[sk_gst]

Hi James,

Thanks for your answer.

1. I do not think the IO is stalled, because at the PC side, the terminal
keeps running without any error messages. The below is the output of
terminal at the PC side:

root@linux-GP60-2PE:/home/linux# gst-launch-1.0 -v  filesrc
location=/dev/ttyUSB0 ! $CAPS  ! h264parse  ! avdec_h264 ! autovideosink
silent=false
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps =
video/x-h264, width=(int)100, height=(int)50, framerate=(fraction)10/1
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:sink: caps =
video/x-h264, width=(int)100, height=(int)50, framerate=(fraction)10/1


2. Before beginning this test, I have connected the serial devices via
HyperTerminal in window and tested by sending and receiving some random
bytes of data. It worked fine. However, I did not do any binary comparison
of the data received

3. I resolved the caps negotiation error with the following:
CAPS=video/x-h264,width=100,height=50,framerate=10/1

The output now is as shown in the point 1. There is no pop-up window that
displays the video received on the PC.

Do you have any more suggestions on to see whats going on behind the
commands? ;)
Also what other approach can I try to get GStreamer display the video?

Regards.






--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[sk_gst]

Hi James,

Here is more debugging info for this issue:

http://gstreamer-devel.966125.n4.nabble.com/Serial-port-interface-on-frame-grabber-td3998720.html


Do you have any more suggestions for this??



Regards



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[James Cameron]

I do think the IO does stall, easily demonstrated mathematically, but
I wasn't saying that was a problem.  Unless you have configured the
serial ports properly, this won't work.  You haven't answered those
parts of my mail, so I'm worried you may have missed it.  I'll go into
more detail.

To check serial port configuration;

stty < /dev/PORT

or

stty --all < /dev/PORT

Without configuration, the default is not binary safe.  Certain
characters cause process signals unless disabled with -icanon or raw.
An -echo is also useful as it prevents the receiving host from sending
back what was received.

An example test on Ubuntu 16.04 using two serial ports;

1.  make a test file of binary random numbers; pick a suitable size,
my choice was 1 MB for a 115200 baud test,

# dd if=/dev/urandom of=test bs=1024k count=1

2.  test the file

# sha256sum test

3.  set serial ports to raw, no echo,

# stty 115200 raw -echo < /dev/ttyUSB0
# stty 115200 raw -echo < /dev/ttyUSB1

4.  in one terminal, start receiving,

# pv /dev/ttyUSB1 > test.received

5.  in another terminal, start transmitting,

# pv test > /dev/ttyUSB0

6.  when the transmission is finished, stop the receiver with ctrl-c,

7.  compare the file sizes, they should be equal,

# ls -l tes*
-rw-r--r-- 1 root root 1048576 Jun 27 06:29 test
-rw-r--r-- 1 root root 1048576 Jun 27 06:32 test.received

8.  compare the file sha256sum,

9.  if there is a difference, use a binary comparison program, or
hexdump each file and use a textual differences program,

p.s. also remove or disable Modem Manager, as on Ubuntu it will often
probe any newly attached serial port.  The probing can interfere.

On Tue, Jun 26, 2018 at 03:55:58AM -0700, vk_gst wrote:

> Hi James,
>
> Thanks for your answer.
>
> 1. I do not think the IO is stalled, because at the PC side, the terminal
> keeps running without any error messages. The below is the output of
> terminal at the PC side:
>
> root@linux-GP60-2PE:/home/linux# gst-launch-1.0 -v  filesrc
> location=/dev/ttyUSB0 ! $CAPS  ! h264parse  ! avdec_h264 ! autovideosink
> silent=false
> Setting pipeline to PAUSED ...
> Pipeline is PREROLLING ...
> /GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps =
> video/x-h264, width=(int)100, height=(int)50, framerate=(fraction)10/1
> /GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:sink: caps =
> video/x-h264, width=(int)100, height=(int)50, framerate=(fraction)10/1
>
>
> 2. Before beginning this test, I have connected the serial devices via
> HyperTerminal in window and tested by sending and receiving some random
> bytes of data. It worked fine. However, I did not do any binary comparison
> of the data received
>
> 3. I resolved the caps negotiation error with the following:
> CAPS=video/x-h264,width=100,height=50,framerate=10/1
>
> The output now is as shown in the point 1. There is no pop-up window that
> displays the video received on the PC.
>
> Do you have any more suggestions on to see whats going on behind the
> commands? ;)
> Also what other approach can I try to get GStreamer display the video?
>
> Regards.
>
>
>
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
James Cameron
http://quozl.netrek.org/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel