Injecting metadata from file MISP

[Fran Raga]

Hi devs

I am trying to inject a telemetry file (KLV) into a video to create a video MISB.

After many months looking I see that only Gstreamer is able to get it. Currently I am developing an open source project and it is already implemented and tested the demux in real time using ffmpeg but I can create the mux tool.
https://github.com/All4Gis/QGISFMV
What I am trying is:

D:\gstreamer\1.0\x86_64\bin\gst-launch-1.0.exe -v --gst-debug=3 -e mpegtsmux name=mux^
 ! filesink location=C:/Users/fran/Desktop/Multiplexor/MISB2.ts^
 filesrc location=C:/Users/fran/Desktop/Multiplexor/DJI_0047_MISB_F.klv^
 ! meta/x-klv ! mux. filesrc location=C:/Users/fran/Desktop/Multiplexor/DJI_0047.ts^
 ! video/x-h264, stream-format=byte-stream, alignment=au ! mux.

But I receive these errors:

Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
0:0/Gst0Pip:eli0ne:0pip.eli1ne01/Gs3tCa4psF2ilt8er:0cap0sfi0lte r0.Gs1tPad:1src: caps = met6a/x-klv, pars2ed=0(bool ean)t0rue
000/Gs0tPi1peliBne:2pipe6line00/M9peg6TsMu9x:m1ux.4Gst0Pad :sinkW_65A: cRaps N= me ta/ x-k lv,  par sed =(boo lean)true
      collectpads gstcollectpads.c:1804:gst_collect_pads_event_default:<collectpads0> GstCollectPads default collecting can only handle time segments. Non time segment ignored.

(gst-launch-1.0:11620): GStreamer-CRITICAL **: 11:27:59.766: gst_segment_to_running_time_full: assertion 'segment->format == format' failed
0/GstPipeline:pipelin:e0/G0stC0aps:Filter:ca0psfi0lte.r1.Gs1tPad:8src7: caps 0= video/5x-h52640, s0trea0m-f ormat1=(str1ing6)by2te-0stream,  ali0gnme0nt=(0str0ing0)au1
B26/Gs0tPi9peline:6pipel9ine00/MCpegT0sMux :muxW.GAstRPadN:s in k_6 6:  caps  =  vid eo/x -h264,  stream-for mat= (str in g)cbyte-sotreaml, alilgnmeentc=(strtingp)au
ads gstcollectpads.c:1804:gst_collect_pads_event_default:<collectpads0> GstCollectPads default collecting can only handle time segments. Non time segment ignored.

(gst-launch-1.0:11620): GStreamer-CRITICAL **: 11:27:59.839: gst_segment_to_running_time_full: assertion 'segment->format == format' failed
0:00:00.250626000 11620 000001B2609690C0 WARN             collectpads gstcollectpads.c:1409:gst_collect_pads_recalculate_waiting:<collectpads0> GstCollectPads has no time segment, assuming 0 based.
0:00:00.279205000 11620 000001B2609690C0 WARN             collectpads gstcollectpads.c:1409:gst_collect_pads_recalculate_waiting:<collectpads0> GstCollectPads has no time segment, assuming 0 based.
0:00:00.306901000 11620 000001B2609690C0 FIXME               basesink gstbasesink.c:3248:gst_base_sink_default_event:<filesink0> stream-start event without group-id. Consider implementing group-id handling in the upstream elements
/GstPipeline:pipeline0/MpegTsMux:mux.GstPad:src: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188
/GstPipeline:pipeline0/GstFileSink:filesink0.GstPad:sink: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188

The resulting video seems to have metadata but it does not look good (the image is corrupt i think)

Any help, suggestion about if I'm missing something, what am I doing wrong?

Gstreamer version:

gst-launch-1.0 version 1.16.0
GStreamer 1.16.0

thank you

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Tue, 2019-06-11 at 11:29 +0200, Fran Raga wrote:

Hi Fran,

> gst-launch-1.0.exe mpegtsmux name=mux ! filesink location=MISB2.ts
> filesrc location=DJI_0047_MISB_F.klv ! meta/x-klv ! mux.
> filesrc location=DJI_0047.ts
>  ! video/x-h264, stream-format=byte-stream, alignment=au ! mux.
>
> But I receive these errors:
> (snip)

So this part

 filesrc location=DJI_0047_MISB_F.klv ! meta/x-klv ! mux.

isn't going to work like that. meta/x-klv is supposed to be a parsed
format with one buffer per KLV chunk. People usually don't feed this in
from a file, but inject it live via an appsrc or such as they go along.
So you would need a klv parser here.

Then this part

 filesrc location=DJI_0047.ts ! video/x-h264, ... ! mux.

that's not really how things work. The file looks like it is an MPEG-TS
stream. So if you want to get to the video data inside that MPEG-TS
file you need to demux it, with the tsdemux element. So something like

 filesrc location=foo.ts ! tsdemux ! video/x-h264 ! mux.

might work, but usually one would put in a parser as well, in this case
h264parse:

 filesrc location=foo.ts ! tsdemux ! h264parse ! mux.

Cheers
 Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Fran Raga]

Thanks for the quick response.
I definitely don't know how to do it, I don't know Gstreamer.

What I have is an original video, the MISB video created with a commercial tool and the klv file extracted from that video.

What I want to generate is that video to not go through that commercial tool
I know that it has to be done in a flow, starting from a csv for example, but I wanted to test if I got it in a first attempt.

thanks

El mar., 11 jun. 2019 a las 13:15, Tim Müller (<[hidden email]>) escribió:

On Tue, 2019-06-11 at 11:29 +0200, Fran Raga wrote:

Hi Fran,

> gst-launch-1.0.exe mpegtsmux name=mux ! filesink location=MISB2.ts
> filesrc location=DJI_0047_MISB_F.klv ! meta/x-klv ! mux.
> filesrc location=DJI_0047.ts
>  ! video/x-h264, stream-format=byte-stream, alignment=au ! mux.
>
> But I receive these errors:
> (snip)

So this part

 filesrc location=DJI_0047_MISB_F.klv ! meta/x-klv ! mux.

isn't going to work like that. meta/x-klv is supposed to be a parsed
format with one buffer per KLV chunk. People usually don't feed this in
from a file, but inject it live via an appsrc or such as they go along.
So you would need a klv parser here.

Then this part

 filesrc location=DJI_0047.ts ! video/x-h264, ... ! mux.

that's not really how things work. The file looks like it is an MPEG-TS
stream. So if you want to get to the video data inside that MPEG-TS
file you need to demux it, with the tsdemux element. So something like

 filesrc location=foo.ts ! tsdemux ! video/x-h264 ! mux.

might work, but usually one would put in a parser as well, in this case
h264parse:

 filesrc location=foo.ts ! tsdemux ! h264parse ! mux.

Cheers
 Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Fran Raga]

Finally I made a parser to use the videos of a DJI without creating an MISB, and create klv packets for after demux (I do not know how create MISB)

It's pure python

https://github.com/All4Gis/QGISFMV/blob/dev/code/player/QgsMultiplexor.py  

I hope that it would be useful to someone

regards

El mar., 11 jun. 2019 a las 13:15, Tim Müller (<[hidden email]>) escribió:

On Tue, 2019-06-11 at 11:29 +0200, Fran Raga wrote:

Hi Fran,

> gst-launch-1.0.exe mpegtsmux name=mux ! filesink location=MISB2.ts
> filesrc location=DJI_0047_MISB_F.klv ! meta/x-klv ! mux.
> filesrc location=DJI_0047.ts
>  ! video/x-h264, stream-format=byte-stream, alignment=au ! mux.
>
> But I receive these errors:
> (snip)

So this part

 filesrc location=DJI_0047_MISB_F.klv ! meta/x-klv ! mux.

isn't going to work like that. meta/x-klv is supposed to be a parsed
format with one buffer per KLV chunk. People usually don't feed this in
from a file, but inject it live via an appsrc or such as they go along.
So you would need a klv parser here.

Then this part

 filesrc location=DJI_0047.ts ! video/x-h264, ... ! mux.

that's not really how things work. The file looks like it is an MPEG-TS
stream. So if you want to get to the video data inside that MPEG-TS
file you need to demux it, with the tsdemux element. So something like

 filesrc location=foo.ts ! tsdemux ! video/x-h264 ! mux.

might work, but usually one would put in a parser as well, in this case
h264parse:

 filesrc location=foo.ts ! tsdemux ! h264parse ! mux.

Cheers
 Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[danielopes92]

Hello Sirs,
I'am looking for a solution that streams H264 MPEG2-TS video over UDP-RTP
with KLV metadata. I'am using the gstreamer and until now i'am already able
to read frames from camera, compress them into H264 MPEG2-TS and send them
via RTP-UDP to a client. Now i would like to generate a KLV package with my
own platform attitude data and join it to my MPEG2-TS packages. These
packages will be received by the client that will see the video and decoded
metadata at the same time. I'am developing this in python using also the
opencv. Any help? This is my code:

...SOME CODE...

gst_str = ('v4l2src device=/dev/video{} ! '
               'video/x-raw, width=(int){}, height=(int){} ! '
               'videoconvert ! appsink').format(dev, width, height)
cap = cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)

gst_str_rtp = "appsrc ! videoconvert ! x264enc tune=zerolatency bitrate=1000
speed-preset=superfast ! mpegtsmux ! rtpmp2tpay ! udpsink
host=192.168.99.133 port=15004"

out = cv2.VideoWriter(gst_str_rtp, 0, 30, (1920, 1080), True)

while True:
        _, img = cap.read() # grab the next image frame from camera
        out.write(img)





--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Josh Doe]

On Wed, Apr 15, 2020 at 7:24 AM danielopes92 <[hidden email]> wrote:

Hello Sirs,
I'am looking for a solution that streams H264 MPEG2-TS video over UDP-RTP
with KLV metadata. I'am using the gstreamer and until now i'am already able
to read frames from camera, compress them into H264 MPEG2-TS and send them
via RTP-UDP to a client. Now i would like to generate a KLV package with my
own platform attitude data and join it to my MPEG2-TS packages. These
packages will be received by the client that will see the video and decoded
metadata at the same time. I'am developing this in python using also the
opencv. Any help?

Are you looking to embed synchronous or asynchronous KLV? I've not done it, but it seems mpegtsmux supports asynchronous KLV. RidgeRun has a metasrc element they sell which seems to be a convenience compared to using appsrc. Check the mailing lists, many people have not found this easy to do, and those who have succeeded don't seem to share their code.

For synchronous there's nothing ready-made it seems, though there are several patches floating around GitLab related to KLV. FWIW I use synchronous KLV with HD-SDI and GigE Vision streams.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Fran Raga]

After many years I found a clue. I have published what I think could be a
good starting point for multiplexing a video and ingesting telemetry (KLV).
It's only a draft but I think it's a good start. I'm not a gstreamer expert
https://gist.github.com/All4Gis/509fbe06ce53a0885744d16595811e6f

to extract the klv we are develop  this library to extract KLV data and
parse in real time,

https://github.com/All4Gis/QGISFMV/tree/master/code/klvdata


if you don't want to use it inside QGIS you can use the library separately.
https://github.com/paretech/klvdata



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel