gst-plugins-good: multipartmux extensions

[Robin Haberkorn]

Hello,

for a proprietary product I'm working on, I had to extend the
multipartmux element from gst-plugins-good.

This product is far from being released to the public, so licensing
issues are not (yet) urgent. Nevertheless they have a significant impact
on project structures etc., so I would like to get them sorted out as
early as possible.

My extensions add additional multipart HTTP headers. Firstly, an
X-Timestamp header is added with the buffer timestamp. This is useful
in our case if the pipeline clock is a real-time system clock, so we
can put individual frames recorded using a live source and muxed using
multipartmux into chronological relation with other frames (possibly
recorded by independant live sources).
This header is currently always sent but it would be easy to make that
configurable using a new property.

Furthermore I implemented support for adding custom HTTP header fields
generated from metadata associated with GstBuffers. Ie. I have
defined a GstMeta implementation that allows you to add some
GstStructure to a GstBuffer as meta data. If the name of the structure
is "http-headers" (could be easily changed of course), one HTTP header
is added per GstStructure field with a gst_value_serialize'd value.

Due to licensing, it is clear that those changes must be released
to the public once a product using them is shipped.

My question is whether you are willing and interested to merge all of
those changes back into the upstream project.
If yes, it makes sense to keep my modifications as minimal as possible;
keeping the type and short name of the modified element etc.
If not, I will have to create an independent fork, ie. a public project
containing the derived multipartmux element and GstMeta implementation.
It then also makes sense to modify the element's name to avoid
conflicts with the upstream multipartmux element.

Best regards,
Robin Haberkorn

--
Robin Haberkorn
Developer

metraTec GmbH
Werner-Heisenberg-Str. 1
39106 Magdeburg
Germany

[hidden email]
www.metratec.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Fr, 2016-03-11 at 20:34 +0100, Robin Haberkorn wrote:
>
> Furthermore I implemented support for adding custom HTTP header fields
> generated from metadata associated with GstBuffers. Ie. I have
> defined a GstMeta implementation that allows you to add some
> GstStructure to a GstBuffer as meta data. If the name of the structure
> is "http-headers" (could be easily changed of course), one HTTP header
> is added per GstStructure field with a gst_value_serialize'd value.

There's a custom event for similar purposes in souphttpsrc. Maybe those
two things can be merged somehow, the advantage of the event is that
it's independent of buffers (and IIRC it was a sticky event).

> My question is whether you are willing and interested to merge all of
> those changes back into the upstream project.
> If yes, it makes sense to keep my modifications as minimal as possible;
> keeping the type and short name of the modified element etc.

That certainly sounds interesting, as long as it is done in a backwards
compatible way :)

Note that multipart is not HTTP specific though, so let's try to keep
things somehow generic.


> My extensions add additional multipart HTTP headers. Firstly, an
> X-Timestamp header is added with the buffer timestamp. This is useful
> in our case if the pipeline clock is a real-time system clock, so we
> can put individual frames recorded using a live source and muxed
> using multipartmux into chronological relation with other frames
> (possibly recorded by independant live sources).

Independent of the other discussion, keep in mind that such header
requires you to somehow synchronize the clocks of the senders and
receivers and compensate for different clock rates of their own clocks.
At least if it is to be used for synchronization of the media.
You could add another header that gives information about which clock
is actually used here though (see e.g. RFC7273 which has that for RTP).

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Robin Haberkorn]

Hello Sebastian,

On Sat, 12 Mar 2016 11:41:53 +0200
Sebastian Dröge <[hidden email]> wrote:

> On Fr, 2016-03-11 at 20:34 +0100, Robin Haberkorn wrote:
> >
> > Furthermore I implemented support for adding custom HTTP header
> > fields generated from metadata associated with GstBuffers. Ie. I
> > have defined a GstMeta implementation that allows you to add some
> > GstStructure to a GstBuffer as meta data. If the name of the
> > structure is "http-headers" (could be easily changed of course),
> > one HTTP header is added per GstStructure field with a
> > gst_value_serialize'd value.
>
> There's a custom event for similar purposes in souphttpsrc. Maybe
> those two things can be merged somehow, the advantage of the event is
> that it's independent of buffers (and IIRC it was a sticky event).
>

Thanks for pointing that out. It's also entirely
undocumented :-)
The fact that events are independent of buffers is exactly why I did
not use them, though. I use meta data to convey information about the
contents of a buffer (always a video frame in my case). Take for
instance the result of a computer vision analysis on a video stream.

I think that sticky events would be OK for adding custom headers in
multipartmux. But what if you want to process the meta data in an
application using GstAppSink? AFAIK if I had events instead of GstMeta,
I could only get the event from the pipeline bus and I'd loose the
association with individual buffers.

Perhaps a good compromise would be to support both a custom GstMeta
type and souphttpsrc's http_headers event.
What do you think?

> > My question is whether you are willing and interested to merge all
> > of those changes back into the upstream project.
> > If yes, it makes sense to keep my modifications as minimal as
> > possible; keeping the type and short name of the modified element
> > etc.
>
> That certainly sounds interesting, as long as it is done in a
> backwards compatible way :)
>
> Note that multipart is not HTTP specific though, so let's try to keep
> things somehow generic.
>

Indeed it is not necessarily restricted to HTTP. Somebody could stream
out a video using E-Mail. Play By Email ;-). But seriously, multipart
messages are defined in RFC1341. For body parts it refers to RFC822
but it also specifically mentions that additional header fields can
occur. If they don't start with "Content-" they are to be ignored in
general. It also says: '"X-" fields may be created for experimental or
private purposes.' So adding timestamp and dynamic headers should be
allowed by the specs.
Furthermore, my changes would be backwards-compatible: X-Timestamp
would be disabled by default -- you will have to enable it using a new
property. The additional meta-data headers are only added if the
corresponding meta-data exists for the buffer. But it could also be
made entirely optional, e.g. by making the GstStructure name
configurable in multipartmux.

>
> > My extensions add additional multipart HTTP headers. Firstly, an
> > X-Timestamp header is added with the buffer timestamp. This is
> > useful in our case if the pipeline clock is a real-time system
> > clock, so we can put individual frames recorded using a live source
> > and muxed using multipartmux into chronological relation with other
> > frames (possibly recorded by independant live sources).
>
> Independent of the other discussion, keep in mind that such header
> requires you to somehow synchronize the clocks of the senders and
> receivers and compensate for different clock rates of their own
> clocks. At least if it is to be used for synchronization of the media.

I'm using PTP to synchronize the real-time system clocks. That should do
the job.
Currently I'm using linuxptp to synchronize the system clock and I'm
enforcing the realtime system clock on my pipeline.

I'm aware that you have implemented a dedicated GstPtpClock at
Centricular. However I don't see the benefit in using it in favour
linuxptp+system clock. Perhaps you can enlighten me.

> You could add another header that gives information about which clock
> is actually used here though (see e.g. RFC7273 which has that for
> RTP).
>

You mean something like the ts-refclk? Could be useful in the long run,
yes. Esp. to debug synchronization issues in more complex setups I
guess. The information that PTP is used as the clock source is not
really useful - in my case at least - since that's a defined part of the
product.
But how to report the PTP server in the context of Gstreamer?
GstPtpClock could probably report that. That wouldn't be
clock-agnostic though, as GStreamer does not seem to support
generic GstClock identification.
When using some extern way to synchronize the system clock, I don't
think any automatic clock identification will work though. So why not
just make that user/application settable? There could be yet another
multipartmux property for adding arbitrary header fields to each
multipart segment not derived from the GstBuffer. Something like
souphttpsrc's "extra-headers". The application may then manage
header fields identifying the clock source.
What do you think?

Best regards,
Robin
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Sa, 2016-03-12 at 21:09 +0100, Robin Haberkorn wrote:

> Hello Sebastian,
>
> On Sat, 12 Mar 2016 11:41:53 +0200
> Sebastian Dröge <[hidden email]> wrote:
>
> >
> > On Fr, 2016-03-11 at 20:34 +0100, Robin Haberkorn wrote:
> > >
> > >
> > > Furthermore I implemented support for adding custom HTTP header
> > > fields generated from metadata associated with GstBuffers. Ie. I
> > > have defined a GstMeta implementation that allows you to add some
> > > GstStructure to a GstBuffer as meta data. If the name of the
> > > structure is "http-headers" (could be easily changed of course),
> > > one HTTP header is added per GstStructure field with a
> > > gst_value_serialize'd value.
> > There's a custom event for similar purposes in souphttpsrc. Maybe
> > those two things can be merged somehow, the advantage of the event is
> > that it's independent of buffers (and IIRC it was a sticky event).
> >
> Thanks for pointing that out. It's also entirely undocumented :-)
>
> The fact that events are independent of buffers is exactly why I did
> not use them, though. I use meta data to convey information about the
> contents of a buffer (always a video frame in my case). Take for
> instance the result of a computer vision analysis on a video stream.
>
> I think that sticky events would be OK for adding custom headers in
> multipartmux. But what if you want to process the meta data in an
> application using GstAppSink? AFAIK if I had events instead of GstMeta,
> I could only get the event from the pipeline bus and I'd loose the
> association with individual buffers.
>
> Perhaps a good compromise would be to support both a custom GstMeta
> type and souphttpsrc's http_headers event.
> What do you think?

My idea was that both the event and the meta could share the
representation at least, and that for non-per-buffer headers you could
use the event and for per-buffer headers the meta. I.e. having the
event as the default values and let the meta override them.

> > > My question is whether you are willing and interested to merge all
> > > of those changes back into the upstream project.
> > > If yes, it makes sense to keep my modifications as minimal as
> > > possible; keeping the type and short name of the modified element
> > > etc.
> > That certainly sounds interesting, as long as it is done in a
> > backwards compatible way :)
> >
> > Note that multipart is not HTTP specific though, so let's try to keep
> > things somehow generic.
> >
> Indeed it is not necessarily restricted to HTTP. Somebody could stream
> out a video using E-Mail. Play By Email ;-).

There's a curlsmtpsink ;)

>  But seriously, multipart 
> messages are defined in RFC1341. For body parts it refers to RFC822
> but it also specifically mentions that additional header fields can
> occur. If they don't start with "Content-" they are to be ignored in
> general. It also says: '"X-" fields may be created for experimental or
> private purposes.' So adding timestamp and dynamic headers should be
> allowed by the specs.
> Furthermore, my changes would be backwards-compatible: X-Timestamp
> would be disabled by default -- you will have to enable it using a new
> property. The additional meta-data headers are only added if the
> corresponding meta-data exists for the buffer. But it could also be
> made entirely optional, e.g. by making the GstStructure name
> configurable in multipartmux.

Makes sense

> > > My extensions add additional multipart HTTP headers. Firstly, an
> > > X-Timestamp header is added with the buffer timestamp. This is
> > > useful in our case if the pipeline clock is a real-time system
> > > clock, so we can put individual frames recorded using a live source
> > > and muxed using multipartmux into chronological relation with other
> > > frames (possibly recorded by independant live sources).
> > Independent of the other discussion, keep in mind that such header
> > requires you to somehow synchronize the clocks of the senders and
> > receivers and compensate for different clock rates of their own
> > clocks. At least if it is to be used for synchronization of the media.
> I'm using PTP to synchronize the real-time system clocks. That should do
> the job.
> Currently I'm using linuxptp to synchronize the system clock and I'm
> enforcing the realtime system clock on my pipeline.
>
> I'm aware that you have implemented a dedicated GstPtpClock at
> Centricular. However I don't see the benefit in using it in favour
> linuxptp+system clock. Perhaps you can enlighten me.

linuxptp+system clock requires you to synchronize your system clock to
one specific PTP clock, and it requires root permissions to actually do
so. With the GStreamer PTP clock you can synchronize your media to
different PTP/NTP/etc servers without affecting your system clock time
and without requiring root permissions.

> > You could add another header that gives information about which clock
> > is actually used here though (see e.g. RFC7273 which has that for
> > RTP).
> >
> You mean something like the ts-refclk? Could be useful in the long run,
> yes. Esp. to debug synchronization issues in more complex setups I
> guess. The information that PTP is used as the clock source is not
> really useful - in my case at least - since that's a defined part of the
> product.

The main advantage here is IMHO that you have this information next to
media instead of having to distribute it somehow else (like making it a
requirement of your product that the system clock is synchronized to
that specific PTP clock).

> But how to report the PTP server in the context of Gstreamer?
> GstPtpClock could probably report that. That wouldn't be
> clock-agnostic though, as GStreamer does not seem to support
> generic GstClock identification.

See https://bugzilla.gnome.org/show_bug.cgi?id=760005 and
https://bugzilla.gnome.org/show_bug.cgi?id=762259

Yes, it actually checks for those two clocks and handles them
specifically right now. The number of possible clocks here seems rather
limited, so I didn't want to overengineer it :)

> When using some extern way to synchronize the system clock, I don't
> think any automatic clock identification will work though. So why not
> just make that user/application settable? There could be yet another
> multipartmux property for adding arbitrary header fields to each
> multipart segment not derived from the GstBuffer. Something like
> souphttpsrc's "extra-headers". The application may then manage
> header fields identifying the clock source.
> What do you think?

Makes sense. But the "ts-refclk" (vocals seem to be expensive) is not
really required for getting your patches merged anyway IMHO :)

If that ever becomes necessary for the RFC7273 implementation in the
RTSP server or jitterbuffer (see patches), API could be added for
specifying that the system clock is indeed synced to one specific
PTP/NTP/etc clock. Or it could even be API on the system clock.


FWIW, note that the system clock used by default is the monotonic one
and not the real-time one. So you will get rather arbitrary values
there instead of the wallclock time according to the PTP clock. You can
change that though (just be careful when the clock changes then, like
when your system suspends and then suddenly there is a huge clock jump
or the system clock jumps backwards or forwards because of linuxptp
adjustments, or ...).

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Robin Haberkorn]

Hello Sebastian,

On Sun, 13 Mar 2016 11:28:31 +0200
Sebastian Dröge <[hidden email]> wrote:

> ...
>
> My idea was that both the event and the meta could share the
> representation at least, and that for non-per-buffer headers you could
> use the event and for per-buffer headers the meta. I.e. having the
> event as the default values and let the meta override them.
>
Yes, makes sense.

> ...
>
> linuxptp+system clock requires you to synchronize your system clock to
> one specific PTP clock, and it requires root permissions to actually
> do so. With the GStreamer PTP clock you can synchronize your media to
> different PTP/NTP/etc servers without affecting your system clock time
> and without requiring root permissions.
>
Both of these advantages are irrelevant to me as I'm developing an
embedded product. My biggest concern on using GstPtpClock is however
that it is a recent feature only implemented in 2015. I'd have to update
my on-board GStreamer - or backport GstPtpClock - which may be a lot
of work. It's not always pleasant to upgrade a large project like
GStreamer in an embedded system as all components must fit together.
(Of course I will have similar problems with my multipartmux changes,
but it's relatively isolated.) So for the sake of simplicity I'm staying
with linuxptp at least for the time being.

btw. I had compatibility problems with OpenCV and thus patched
gst-plugins-bad to compile with OpenCV 3. I should probably post that
here. Unfortunately, the code base of gst-plugins-bad has changed so
much recently that this patch won't work anymore. At the same time
gst-plugins-bad is still not OpenCV 3 compatible. So when I upgrade
gstreamer, I will have to completely rewrite that, too. At least the
patch may then be merged upstream, finally adding support for OpenCV 3.

> ...
> > You mean something like the ts-refclk? Could be useful in the long
> > run, yes. Esp. to debug synchronization issues in more complex
> > setups I guess. The information that PTP is used as the clock
> > source is not really useful - in my case at least - since that's a
> > defined part of the product.
>
> The main advantage here is IMHO that you have this information next to
> media instead of having to distribute it somehow else (like making it
> a requirement of your product that the system clock is synchronized to
> that specific PTP clock).
>

My use case is probably much different than what you have in mind.
Clients receiving the video data will not have to sync against the same
clock. But I acknowledge that it's useful elsewhere.

> ...
>
> See https://bugzilla.gnome.org/show_bug.cgi?id=760005 and
> https://bugzilla.gnome.org/show_bug.cgi?id=762259
>
> Yes, it actually checks for those two clocks and handles them
> specifically right now. The number of possible clocks here seems
> rather limited, so I didn't want to overengineer it :)
>
Alright, I will add similar code for NTP and PTP clocks.

> ...
> > When using some extern way to synchronize the system clock, I don't
> > think any automatic clock identification will work though. So why
> > not just make that user/application settable? There could be yet
> > another multipartmux property for adding arbitrary header fields to
> > each multipart segment not derived from the GstBuffer. Something
> > like souphttpsrc's "extra-headers". The application may then manage
> > header fields identifying the clock source.
> > What do you think?
>
> Makes sense. But the "ts-refclk" (vocals seem to be expensive) is not
> really required for getting your patches merged anyway IMHO :)
>

What would you like to call it instead? X-Timestamp-RefClock?

If you use "extra-headers" for externally synced system clocks, you can
of course choose whatever name you like.

"extra-headers" would consequently be merged into the event and meta
data headers (after both of them).

> If that ever becomes necessary for the RFC7273 implementation in the
> RTSP server or jitterbuffer (see patches), API could be added for
> specifying that the system clock is indeed synced to one specific
> PTP/NTP/etc clock. Or it could even be API on the system clock.
>
Mhh... I don't think that's a good idea. What if you sync your system
clock to something else, like NTP? This can hardly be mapped generically
to SystemClock properties or methods. And even if it could be by adding
some new properties, you could not ask the OS about the relevant
properties; so you would have to update the SystemClock properties from
your App anyway using whatever means necessary.
So why not just use the "extra-headers" way of delegating that task to
the application?

>
> FWIW, note that the system clock used by default is the monotonic one
> and not the real-time one. So you will get rather arbitrary values
> there instead of the wallclock time according to the PTP clock. You
> can change that though (just be careful when the clock changes then,
> like when your system suspends and then suddenly there is a huge
> clock jump or the system clock jumps backwards or forwards because of
> linuxptp adjustments, or ...).
>
I'm aware of that and have done that. That's also why I'd like to keep
a rather idiosyncratic X-Timestamp format: it is simply the sum of
the buffer's running time and the element's base time in nanoseconds,
resulting in an absolute timestamp with regard the pipeline clock's
origin or epoch.
If you set up a realtime system clock, that just happens to be the UNIX
epoch and the X-Timestamp happens to be in nanoseconds since the UNIX
epoch. If you use some other monotonic clock, the timestamp value will
probably be useless but at least easy to describe. That's why it makes
little sense to use something like the NTP timestamp format since it
has an explicit epoch and I can only translate the buffer's timestamp
to an NTP timestamp when I know a little bit about the pipeline's clock.
Otherwise I would have to __assume__ something like the UNIX epoch,
complicating the timestamp semantics (if the pipeline clock is NOT
based on the UNIX epoch).

Best regards,
Robin
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-4]

Le mardi 15 mars 2016 à 21:28 +0100, Robin Haberkorn a écrit :
> btw. I had compatibility problems with OpenCV and thus patched
> gst-plugins-bad to compile with OpenCV 3. I should probably post that
> here. Unfortunately, the code base of gst-plugins-bad has changed so
> much recently that this patch won't work anymore. At the same time
> gst-plugins-bad is still not OpenCV 3 compatible. So when I upgrade
> gstreamer, I will have to completely rewrite that, too. At least the
> patch may then be merged upstream, finally adding support for OpenCV
> 3.

This works has already been done over 1.6 and should be complete in
1.8. All element are now ported to C++.

cheers,
Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Di, 2016-03-15 at 18:31 -0400, Nicolas Dufresne wrote:

> Le mardi 15 mars 2016 à 21:28 +0100, Robin Haberkorn a écrit :
> >
> > btw. I had compatibility problems with OpenCV and thus patched
> > gst-plugins-bad to compile with OpenCV 3. I should probably post
> > that
> > here. Unfortunately, the code base of gst-plugins-bad has changed
> > so
> > much recently that this patch won't work anymore. At the same time
> > gst-plugins-bad is still not OpenCV 3 compatible. So when I upgrade
> > gstreamer, I will have to completely rewrite that, too. At least
> > the
> > patch may then be merged upstream, finally adding support for
> > OpenCV
> > 3.
> This works has already been done over 1.6 and should be complete in
> 1.8. All element are now ported to C++.

This here is still pending:
https://bugzilla.gnome.org/show_bug.cgi?id=760473

Making things work with OpenCV 3.1 and without the legacy/deprecated
APIs.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Di, 2016-03-15 at 21:28 +0100, Robin Haberkorn wrote:

>
> > linuxptp+system clock requires you to synchronize your system clock to
> > one specific PTP clock, and it requires root permissions to actually
> > do so. With the GStreamer PTP clock you can synchronize your media to
> > different PTP/NTP/etc servers without affecting your system clock time
> > and without requiring root permissions.
> >
> Both of these advantages are irrelevant to me as I'm developing an
> embedded product. My biggest concern on using GstPtpClock is however
> that it is a recent feature only implemented in 2015. I'd have to update
> my on-board GStreamer - or backport GstPtpClock - which may be a lot
> of work. It's not always pleasant to upgrade a large project like
> GStreamer in an embedded system as all components must fit together.
> (Of course I will have similar problems with my multipartmux changes,
> but it's relatively isolated.) So for the sake of simplicity I'm staying
> with linuxptp at least for the time being.

If you're using something like yocto for building your system images,
or even a normal Linux distro, it should be relatively easy to upgrade
GStreamer to a newer version. Definitely easier than backporting all
those changes :) Unless you also have more changes that would need to
be forward-ported.
Doing that would be a good idea independent of staying with linuxptp or
not, which is an acceptable solution in many situations indeed.


> btw. I had compatibility problems with OpenCV and thus patched
> gst-plugins-bad to compile with OpenCV 3. I should probably post that
> here. Unfortunately, the code base of gst-plugins-bad has changed so
> much recently that this patch won't work anymore. At the same time
> gst-plugins-bad is still not OpenCV 3 compatible. So when I upgrade
> gstreamer, I will have to completely rewrite that, too. At least the
> patch may then be merged upstream, finally adding support for OpenCV 3.

See https://bugzilla.gnome.org/show_bug.cgi?id=760473


> > The main advantage here is IMHO that you have this information next to
> > media instead of having to distribute it somehow else (like making it
> > a requirement of your product that the system clock is synchronized to
> > that specific PTP clock).
> >
> My use case is probably much different than what you have in mind.
> Clients receiving the video data will not have to sync against the same
> clock. But I acknowledge that it's useful elsewhere.

What is your use case? Your clients get those buffers with the
timestamps you put on them, and then just write them out as metadata
instead of using them for synchronization?

> > If that ever becomes necessary for the RFC7273 implementation in the
> > RTSP server or jitterbuffer (see patches), API could be added for
> > specifying that the system clock is indeed synced to one specific
> > PTP/NTP/etc clock. Or it could even be API on the system clock.
> >
> Mhh... I don't think that's a good idea. What if you sync your system
> clock to something else, like NTP? This can hardly be mapped generically
> to SystemClock properties or methods. And even if it could be by adding
> some new properties, you could not ask the OS about the relevant
> properties; so you would have to update the SystemClock properties from
> your App anyway using whatever means necessary.
> So why not just use the "extra-headers" way of delegating that task to
> the application?

That's exactly what I proposed, yes. Some API on the system clock so
that the application can specify a clock the system clock is synced to,
so the inside the pipeline elements can use this information for
whatever they want to.

I also don't think it makes sense to somehow autodetect this, it must
be the application in the end that provides this information.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mercredi 16 mars 2016 à 09:57 +0200, Sebastian Dröge a écrit :
> This here is still pending:
> https://bugzilla.gnome.org/show_bug.cgi?id=760473
>
> Making things work with OpenCV 3.1 and without the legacy/deprecated
> APIs.

Oops, we forgot this one, will be in next version then.

Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Robin Haberkorn]

On Wed, 16 Mar 2016 10:20:34 +0200
Sebastian Dröge <[hidden email]> wrote:

> ...
>
> If you're using something like yocto for building your system images,
> or even a normal Linux distro, it should be relatively easy to upgrade
> GStreamer to a newer version. Definitely easier than backporting all
> those changes :) Unless you also have more changes that would need to
> be forward-ported.
> Doing that would be a good idea independent of staying with linuxptp
> or not, which is an acceptable solution in many situations indeed.
>

Indeed I'm using Yocto. But something being easy in theory is not
always easy in practice. Upgrading Gstreamer will very likely require
me to upgrade a lot of other packages as well. Not all of them will be
readily available. Also I will have to forward-port BSP and other
patches on top of Gstreamer. Speaking out of experience working many
years in the embedded Linux business, this sort of stuff __can__ be
annoying and is certainly not done in 5 minutes.

Since OpenCV 3 support will be finished in v1.8 -- as Nicolas Dufresne
has pointed out -- I think it would be wise to wait for that release.

> ...
> > My use case is probably much different than what you have in mind.
> > Clients receiving the video data will not have to sync against the
> > same clock. But I acknowledge that it's useful elsewhere.
>
> What is your use case? Your clients get those buffers with the
> timestamps you put on them, and then just write them out as metadata
> instead of using them for synchronization?
>
Exactly.

> > > If that ever becomes necessary for the RFC7273 implementation in
> > > the RTSP server or jitterbuffer (see patches), API could be added
> > > for specifying that the system clock is indeed synced to one
> > > specific PTP/NTP/etc clock. Or it could even be API on the system
> > > clock.
> > >
> > Mhh... I don't think that's a good idea. What if you sync your
> > system clock to something else, like NTP? This can hardly be mapped
> > generically to SystemClock properties or methods. And even if it
> > could be by adding some new properties, you could not ask the OS
> > about the relevant properties; so you would have to update the
> > SystemClock properties from your App anyway using whatever means
> > necessary. So why not just use the "extra-headers" way of
> > delegating that task to the application?
>
> That's exactly what I proposed, yes. Some API on the system clock so
> that the application can specify a clock the system clock is synced
> to, so the inside the pipeline elements can use this information for
> whatever they want to.
>
> I also don't think it makes sense to somehow autodetect this, it must
> be the application in the end that provides this information.
>

I still don't get why this needs a special API on the system clock side
then. Let the application simply use the proposed "extra-headers"
property on multipartmux.

Best regards,
Robin
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Robin Haberkorn]

On Sun, 13 Mar 2016 11:28:31 +0200
Sebastian Dröge <[hidden email]> wrote:

> On Sa, 2016-03-12 at 21:09 +0100, Robin Haberkorn wrote:
> > ...
> > Perhaps a good compromise would be to support both a custom GstMeta
> > type and souphttpsrc's http_headers event.
> > What do you think?
>
> My idea was that both the event and the meta could share the
> representation at least, and that for non-per-buffer headers you could
> use the event and for per-buffer headers the meta. I.e. having the
> event as the default values and let the meta override them.
>

This raises a new problem though. It makes sense to extend
multipartdemux to parse additional headers as well, so the changes are
symmetric and consistent. But which headers are parsed into a GstMeta
and which ones will be transmitted via events?
This information was lost when all structures were merged.
I see three possibilities to resolve that: Either all headers will
produce one GstMeta in multipartdemux; or all headers will produce one
event; or multipartmux will have to encode the header's source into the
header's name or field, so multipartdemux can infer what to do with
them.
For the time being I will not extend multipartdemux, though.

Best regards,
Robin
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Mi, 2016-03-16 at 14:39 -0400, Nicolas Dufresne wrote:
> Le mercredi 16 mars 2016 à 09:57 +0200, Sebastian Dröge a écrit :
> >
> > This here is still pending:
> > https://bugzilla.gnome.org/show_bug.cgi?id=760473
> >
> > Making things work with OpenCV 3.1 and without the
> > legacy/deprecated
> > APIs.
> Oops, we forgot this one, will be in next version then.

I'd like to get it into 1.8.1 if possible.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Mi, 2016-03-16 at 19:39 +0100, Robin Haberkorn wrote:

> On Wed, 16 Mar 2016 10:20:34 +0200
> Sebastian Dröge <[hidden email]> wrote:
>
> >
> > ...
> >
> > If you're using something like yocto for building your system images,
> > or even a normal Linux distro, it should be relatively easy to upgrade
> > GStreamer to a newer version. Definitely easier than backporting all
> > those changes :) Unless you also have more changes that would need to
> > be forward-ported.
> > Doing that would be a good idea independent of staying with linuxptp
> > or not, which is an acceptable solution in many situations indeed.
> >
> Indeed I'm using Yocto. But something being easy in theory is not
> always easy in practice. Upgrading Gstreamer will very likely require
> me to upgrade a lot of other packages as well. Not all of them will
> be readily available. Also I will have to forward-port BSP and other
> patches on top of Gstreamer. Speaking out of experience working many
> years in the embedded Linux business, this sort of stuff __can__ be
> annoying and is certainly not done in 5 minutes.

True, my point was that it's going to be less work than backporting
lots of changes to an older version.

> > That's exactly what I proposed, yes. Some API on the system clock so
> > that the application can specify a clock the system clock is synced
> > to, so the inside the pipeline elements can use this information for
> > whatever they want to.
> >
> > I also don't think it makes sense to somehow autodetect this, it must
> > be the application in the end that provides this information.
> >
> I still don't get why this needs a special API on the system clock side
> then. Let the application simply use the proposed "extra-headers"
> property on multipartmux.

It allows to have this information be used in a generic way by
arbitrary elements, and for the elements to request this information
from the clock. Instead of having the application worry about telling
all elements that might care about this in a custom way.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Robin Haberkorn]

Hello again,

I just wanted to give an update here. Already some months ago, I
prepared to contribute the changes discussed here back to the
community. I rewrote them to work on top of gst-plugins-good v1.2.4.
However I did not yet implement the planned multipartmux changes
for clock source reporting, http_headers event support or
extra-headers property, etc.

What I did was just the minimum in order to comply with the LGPL should
we release our product. (Since it's easy to publish our gst-plugins-good
fork even if the changes don't get upstreamed.)

Still, I'm sending you these changes now as patches for the time being,
so they won't get lost and you can already have a look at them. When I
implement the remaining features (they're on my very long TODO
list...), I will probably publish my gst-plugins-good fork and create a
proper Bugzilla ticket.

Best regards,
Robin Haberkorn

On Wed, 16 Mar 2016 21:53:11 +0100
Robin Haberkorn <[hidden email]> wrote:

> On Sun, 13 Mar 2016 11:28:31 +0200
> Sebastian Dröge <[hidden email]> wrote:
>
> > On Sa, 2016-03-12 at 21:09 +0100, Robin Haberkorn wrote:
> > > ...
> > > Perhaps a good compromise would be to support both a custom
> > > GstMeta type and souphttpsrc's http_headers event.
> > > What do you think?
> >
> > My idea was that both the event and the meta could share the
> > representation at least, and that for non-per-buffer headers you
> > could use the event and for per-buffer headers the meta. I.e.
> > having the event as the default values and let the meta override
> > them.
> >
> This raises a new problem though. It makes sense to extend
> multipartdemux to parse additional headers as well, so the changes are
> symmetric and consistent. But which headers are parsed into a GstMeta
> and which ones will be transmitted via events?
> This information was lost when all structures were merged.
> I see three possibilities to resolve that: Either all headers will
> produce one GstMeta in multipartdemux; or all headers will produce one
> event; or multipartmux will have to encode the header's source into
> the header's name or field, so multipartdemux can infer what to do
> with them.
> For the time being I will not extend multipartdemux, though.
>
> Best regards,
> Robin
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
Robin Haberkorn
Developer

metraTec GmbH
Niels-Bohr-Str. 5
39106 Magdeburg
Germany

[hidden email]
www.metratec.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel