How to send a series of zipped bitmaps files / raw video frames by tcp with gstreamer?

[F32]

Hi,

I'm trying to send a series of zipped bitmap files / raw frames over network (because it consumes much less cpu than h264).

The following command works and all files are copied correctly:

> gst-launch-1.0 multifilesrc location="/dev/shm/1/%05d.bmp" index=1 ! multifilesink location="%05d.bmp"

However, when I tried to transfer these files over tcp:

 > gst-launch-1.0 multifilesrc location="%05d.bmp" index=1 ! tcpclientsink port=3000

 > gst-launch-1.0 tcpserversrc port=3000 ! multifilesink location="%05d.bmp"

The result is incorrect. What I got is numerous 4KB files instead of large bitmap files.

I have learned that for raw video over tcp, we need to add a "rawvideoparse" element after tcpserversrc, but it does not seem to support generic compression (zip, gzip, etc.)

What is the simplest way to support transferring a series of zipped bitmap files / raw video frames by tcp with gstreamer? Do I need to extend the rawvideoaprse, write a special zipdec element, or something else?

Regards,

Windy

 

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le lundi 15 juin 2020 à 17:28 +0800, F32 a écrit :

> Hi,
>
> I'm trying to send a series of zipped bitmap files / raw frames over
> network (because it consumes much less cpu than h264).
> The following command works and all files are copied correctly:
>
> > gst-launch-1.0 multifilesrc location="/dev/shm/1/%05d.bmp" index=1
> ! multifilesink location="%05d.bmp"
>
> However, when I tried to transfer these files over tcp:
>
>  > gst-launch-1.0 multifilesrc location="%05d.bmp" index=1 !
> tcpclientsink port=3000
>  > gst-launch-1.0 tcpserversrc port=3000 ! multifilesink
> location="%05d.bmp"
>
> The result is incorrect. What I got is numerous 4KB files instead of
> large bitmap files.
>
> I have learned that for raw video over tcp, we need to add a
> "rawvideoparse" element after tcpserversrc, but it does not seem to
> support generic compression (zip, gzip, etc.)
>
> What is the simplest way to support transferring a series of zipped
> bitmap files / raw video frames by tcp with gstreamer? Do I need to
> extend the rawvideoaprse, write a special zipdec element, or
> something else?

The answer will depend on what you really intend to do. If generic file
transfer is your end goal, this is simply the wrong software ;-P But if
you do have some streaming use cases around that, we can likely help,
assuming we have more details. Of course, when you chose a streaming
method that is not following any existing standard, you might endup
having to implement some elements for it.

>
> Regards,
> Windy
>
>
>  
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[F32]

Here's more detailed background information:

I have a 3D (OpenGL) rendering software running on server A, and a media server based on gstreamer running on server B.

Server A is also the start point of the gstreamer pipeline. It fetches raw images from OpenGL, and puts images into gstreamer (e.g., the tcpclientsink element)

H264 / H265 is not used because it consumes a lot of cpu / gpu memory. However, sending raw images over network also consumes a lot of cpu / bandwidth, because raw images are just too large.

A trade-off is to use a lightweight compression algorithm (e.g., lz4 -1) which can compress 1GB bitmaps / raw images per second, per core. In my case, file size can be reduced by 90%.

Too achieve this, I'm thinking about:

 - creating a lz4enc element, which puts "zipped content size" + "zipped content" into a gstbuffer, and then sends it by tcp

 - creating a lz4dec element, which keeps reading bytes from tcpserversrc, and outputs a decompressed gstbuffer when both "zipped content size" and "zipped content" are received

Is it the correct way to achieve this?

(Also, I think that lightweight compression / decompression like lz4 is an interesting feature that might be officially added to gstreamer in the long term)

At 2020-06-15 21:46:40, "Nicolas Dufresne" <[hidden email]> wrote:
>Le lundi 15 juin 2020 à 17:28 +0800, F32 a écrit :
>> Hi,
>> 
>> I'm trying to send a series of zipped bitmap files / raw frames over
>> network (because it consumes much less cpu than h264).
>> The following command works and all files are copied correctly:
>> 
>> > gst-launch-1.0 multifilesrc location="/dev/shm/1/%05d.bmp" index=1
>> ! multifilesink location="%05d.bmp"
>> 
>> However, when I tried to transfer these files over tcp:
>> 
>>  > gst-launch-1.0 multifilesrc location="%05d.bmp" index=1 !
>> tcpclientsink port=3000
>>  > gst-launch-1.0 tcpserversrc port=3000 ! multifilesink
>> location="%05d.bmp"
>> 
>> The result is incorrect. What I got is numerous 4KB files instead of
>> large bitmap files.
>> 
>> I have learned that for raw video over tcp, we need to add a
>> "rawvideoparse" element after tcpserversrc, but it does not seem to
>> support generic compression (zip, gzip, etc.)
>> 
>> What is the simplest way to support transferring a series of zipped
>> bitmap files / raw video frames by tcp with gstreamer? Do I need to
>> extend the rawvideoaprse, write a special zipdec element, or
>> something else?
>
>The answer will depend on what you really intend to do. If generic file
>transfer is your end goal, this is simply the wrong software ;-P But if
>you do have some streaming use cases around that, we can likely help,
>assuming we have more details. Of course, when you chose a streaming
>method that is not following any existing standard, you might endup
>having to implement some elements for it.
>
>> 
>> Regards,
>> Windy
>> 
>> 
>>  
>> 
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>_______________________________________________
>gstreamer-devel mailing list
>[hidden email]
>https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

 

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mardi 16 juin 2020 à 11:07 +0800, F32 a écrit :

> Here's more detailed background information:
>
> I have a 3D (OpenGL) rendering software running on server A, and a
> media server based on gstreamer running on server B.
>
> Server A is also the start point of the gstreamer pipeline. It
> fetches raw images from OpenGL, and puts images into gstreamer (e.g.,
> the tcpclientsink element)
>
> H264 / H265 is not used because it consumes a lot of cpu / gpu
> memory. However, sending raw images over network also consumes a lot
> of cpu / bandwidth, because raw images are just too large.
>
> A trade-off is to use a lightweight compression algorithm (e.g., lz4
> -1) which can compress 1GB bitmaps / raw images per second, per core.
> In my case, file size can be reduced by 90%.
>
> Too achieve this, I'm thinking about:
>  - creating a lz4enc element, which puts "zipped content size" +
> "zipped content" into a gstbuffer, and then sends it by tcp
>  - creating a lz4dec element, which keeps reading bytes from
> tcpserversrc, and outputs a decompressed gstbuffer when both "zipped
> content size" and "zipped content" are received
> Is it the correct way to achieve this?

Perhaps you'd like to seek inspiration in PNG (or just use that). It's
a simple libz based RGB tile compressor, but to support the P
(portable) they introduce a magic cookie and pixel layout / image size
information. With this design, the receiver can:

  - Figure-out the image size and format (signalling free)
  - Decompress as data flows (tiles) or on multiple cores
  - Use low CPU (low complexity)
  - It's truly loss-less

Its bad for lossy network and do not have RTP integration (though
GStreamer offer a GST specific generaic payloader). It also does not
have lossy mechanism to be able to guaranty a compression budget. Of
course, there is other solution to like old JPEG 2K, Apple ProRES, RED
encoding, NDI (warning, bad reputation ;-P), etc. They are all about
visually loss-less/low-complexity and fixed budget networks/storage
speed.

>
> (Also, I think that lightweight compression / decompression like lz4
> is an interesting feature that might be officially added to gstreamer
> in the long term)

Adding niche and unique compression format, rather then using widely
adopted standard is not that interesting for maintainers. But if you
host your own project with a library and documentation, and provide a
plugin using that, this would be acceptable for upstream GStreamer
inclusion. Hosting your own plugins as part of your project is of
course totally fine ;-P

Instead of your own project, you could add to FFMPEG CODEC collection,
which unlike GStreamer is meant to offer compression algorithm directly
(clean room implementation, patent encumbered is apparently fine
there).

>
> At 2020-06-15 21:46:40, "Nicolas Dufresne" <[hidden email]>
> wrote:
> >Le lundi 15 juin 2020 à 17:28 +0800, F32 a écrit :
> >> Hi,
> >>
> >> I'm trying to send a series of zipped bitmap files / raw frames
> over
> >> network (because it consumes much less cpu than h264).
> >> The following command works and all files are copied correctly:
> >>
> >> > gst-launch-1.0 multifilesrc location="/dev/shm/1/%05d.bmp"
> index=1
> >> ! multifilesink location="%05d.bmp"
> >>
> >> However, when I tried to transfer these files over tcp:
> >>
> >>  > gst-launch-1.0 multifilesrc location="%05d.bmp" index=1 !
> >> tcpclientsink port=3000
> >>  > gst-launch-1.0 tcpserversrc port=3000 ! multifilesink
> >> location="%05d.bmp"
> >>
> >> The result is incorrect. What I got is numerous 4KB files instead
> of
> >> large bitmap files.
> >>
> >> I have learned that for raw video over tcp, we need to add a
> >> "rawvideoparse" element after tcpserversrc, but it does not seem
> to
> >> support generic compression (zip, gzip, etc.)
> >>
> >> What is the simplest way to support transferring a series of
> zipped
> >> bitmap files / raw video frames by tcp with gstreamer? Do I need
> to
> >> extend the rawvideoaprse, write a special zipdec element, or
> >> something else?
> >
> >The answer will depend on what you really intend to do. If generic
> file
> >transfer is your end goal, this is simply the wrong software ;-P But
> if
> >you do have some streaming use cases around that, we can likely
> help,
> >assuming we have more details. Of course, when you chose a streaming
> >method that is not following any existing standard, you might endup
> >having to implement some elements for it.
> >
> >>
> >> Regards,
> >> Windy
> >>
> >>
> >>  
> >>
> >> _______________________________________________
> >> gstreamer-devel mailing list
> >> [hidden email]
> >> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> >
> >_______________________________________________
> >gstreamer-devel mailing list
> >[hidden email]
> >https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
>  
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel