Adding typefind function for a new Gst Type, "tensors"

[MyungJoo Ham]

Dear Gstreamer Developers,


We have been working on developing Gstreamer plugins to handle
general neural network models implemented with on-the-shelf frameworks
(such as Tensorflow or Caffe) as media filters.


I have a few questions on how to register a new custom "typefind" for
a new non-media type, "tensor/tensors".



Because the "sources" are general media streams, such as "camera", "mic",
or "media files", the to-be-proposed type, "other/tensor" and
"other/tensors", do not have their file format, yet.
Their meta data are handled as pad-capabilities and
the buffer between elements contains the tensor data only
(similar to what video/x-raw does)

However, assuming that we need to register "typefind",
later to be considered to become "good plugin" some day,
it appears that we need to define headers as if they are
stored as files. (for debugging and testing purpose, we may need this anyway)



Q1. Is it ok to define "other/tensor(s)" type to have some headers
(including dimensions of tensors) when it is stored in file and
not to have any meta data in inter-gst-element transfer (like video/x-raw)?

Q2. If Q1 == NO, is it ok to define "other/tensorsave" to represent
the saved file format for tensors and use "other/tensor(s)" for
video/x-raw-like handling?

Q3. Or is it ok not to register any and possible to upstream to
gst-plugins-good with these new media types (other/tensor and other/tensors)?



As I've mentioned in Gstreamer-IRC back in March (sorry, I was away from
this project for a while and returned), we've finally got internal approval
to fully open sources (In LGPL-2.1.)
We will be migrating the code to github.com/nnsuite soon, probably in this week.


The full migration requires CI/CD systems to be migrated as well,
so the migration of developing process might take some time.



We hope, someday, we can upstream the plugins to gst-plugins-good. :)





Cheers,
MyungJoo

--

MyungJoo Ham (함명주), Ph.D.
Autonomous Machine Lab., AI Center, Samsung Research.
Cell: [phone]


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[David Ing]

I do not have an answer to your questions, but I noticed a documentation page:

https://gstreamer.freedesktop.org/documentation/plugin-development/advanced/media-types.html#table-of-other-types

At the very bottom of that page, I noticed an empty table "Table of Other Types" which seems to suggest that you can define any type you want (with any prefix, not just "audio/" or "video/").

I am guessing that you would simply write a plugin with a typefind function like this ...

// Executes typefind logic for "cust/om" media type
// @param ptr  A pointer to the first byte of the stream.
static void gst_custom_typefind(GstTypeFind *typeFind, gpointer ptr)
{
    int64_t offset = 0;
    uint32_t size = 100;
    const guint8 *data = gst_type_find_peek(typeFind, offset, size);

    if (data && data[0] == 'C' && data[1] == 'U' && data[2] == 'S' && data[3] == 'T')
    {
        GstCaps* caps = ...
        gst_type_find_suggest(typeFind, GST_TYPE_FIND_MAXIMUM, gst_caps_new_simple("cust/om", caps));
        gst_caps_unref(caps);
    }
}

And in your `plugin_init` method, you would need to register the typefind function using `gst_type_find_register`.

Notice that in the typefind function, you can define the caps of the stream (such as the dimensionality of your tensor).  I believe that the caps structure can contain any fields you want.

Next your plugin should define a pad template having SINK caps which are compatible with the caps that would be detected by your typefind function .  Your plugin can have whatever SRC caps you want (i.e. "video/x-raw") ... but it needs to contain the code which transforms the stream from SINK to SRC.

I can't see any reason why that wouldn't work, but I am just a novice:  Someone should correct me if I am wrong.

On Mon, Jul 23, 2018 at 1:07 AM, MyungJoo Ham <[hidden email]> wrote:

Dear Gstreamer Developers,


We have been working on developing Gstreamer plugins to handle
general neural network models implemented with on-the-shelf frameworks
(such as Tensorflow or Caffe) as media filters.


I have a few questions on how to register a new custom "typefind" for
a new non-media type, "tensor/tensors".



Because the "sources" are general media streams, such as "camera", "mic",
or "media files", the to-be-proposed type, "other/tensor" and
"other/tensors", do not have their file format, yet.
Their meta data are handled as pad-capabilities and
the buffer between elements contains the tensor data only
(similar to what video/x-raw does)

However, assuming that we need to register "typefind",
later to be considered to become "good plugin" some day,
it appears that we need to define headers as if they are
stored as files. (for debugging and testing purpose, we may need this anyway)



Q1. Is it ok to define "other/tensor(s)" type to have some headers
(including dimensions of tensors) when it is stored in file and
not to have any meta data in inter-gst-element transfer (like video/x-raw)?

Q2. If Q1 == NO, is it ok to define "other/tensorsave" to represent
the saved file format for tensors and use "other/tensor(s)" for
video/x-raw-like handling?

Q3. Or is it ok not to register any and possible to upstream to
gst-plugins-good with these new media types (other/tensor and other/tensors)?



As I've mentioned in Gstreamer-IRC back in March (sorry, I was away from
this project for a while and returned), we've finally got internal approval
to fully open sources (In LGPL-2.1.)
We will be migrating the code to github.com/nnsuite soon, probably in this week.


The full migration requires CI/CD systems to be migrated as well,
so the migration of developing process might take some time.



We hope, someday, we can upstream the plugins to gst-plugins-good. :)





Cheers,
MyungJoo

--

MyungJoo Ham (함명주), Ph.D.
Autonomous Machine Lab., AI Center, Samsung Research.
Cell: [phone]


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[David Ing]

Sorry, I borked that code example a bit.

The typefind function should be more like this

// Executes typefind logic for "cust/om" media type
// @param ptr  A pointer to the first byte of the stream.
static void gst_custom_typefind(GstTypeFind *typeFind, gpointer ptr)
{
    int64_t offset = 0;
    uint32_t size = 100;
    const guint8 *data = gst_type_find_peek(typeFind, offset, size);

    if (data && data[0] == 'C' && data[1] == 'U' && data[2] == 'S' && data[3] == 'T')
    {
        // extract other data from the stream so you can feed it into the caps
        // replace the `...` in the next statement with the caps data.  See documentation
        // for `gst_caps_new_simple` for more information.

        GstCaps* caps = gst_caps_new_simple("cust/om", ...);
        gst_type_find_suggest(typeFind, GST_TYPE_FIND_MAXIMUM, caps);
        gst_caps_unref(caps);
    }
}

On Mon, Jul 23, 2018 at 9:36 AM, David Ing <[hidden email]> wrote:

I do not have an answer to your questions, but I noticed a documentation page:

https://gstreamer.freedesktop.org/documentation/plugin-development/advanced/media-types.html#table-of-other-types

At the very bottom of that page, I noticed an empty table "Table of Other Types" which seems to suggest that you can define any type you want (with any prefix, not just "audio/" or "video/").

I am guessing that you would simply write a plugin with a typefind function like this ...

// Executes typefind logic for "cust/om" media type
// @param ptr  A pointer to the first byte of the stream.
static void gst_custom_typefind(GstTypeFind *typeFind, gpointer ptr)
{
    int64_t offset = 0;
    uint32_t size = 100;
    const guint8 *data = gst_type_find_peek(typeFind, offset, size);

    if (data && data[0] == 'C' && data[1] == 'U' && data[2] == 'S' && data[3] == 'T')
    {
        GstCaps* caps = ...
        gst_type_find_suggest(typeFind, GST_TYPE_FIND_MAXIMUM, gst_caps_new_simple("cust/om", caps));
        gst_caps_unref(caps);
    }
}

And in your `plugin_init` method, you would need to register the typefind function using `gst_type_find_register`.

Notice that in the typefind function, you can define the caps of the stream (such as the dimensionality of your tensor).  I believe that the caps structure can contain any fields you want.

Next your plugin should define a pad template having SINK caps which are compatible with the caps that would be detected by your typefind function .  Your plugin can have whatever SRC caps you want (i.e. "video/x-raw") ... but it needs to contain the code which transforms the stream from SINK to SRC.

I can't see any reason why that wouldn't work, but I am just a novice:  Someone should correct me if I am wrong.

On Mon, Jul 23, 2018 at 1:07 AM, MyungJoo Ham <[hidden email]> wrote:

Dear Gstreamer Developers,


We have been working on developing Gstreamer plugins to handle
general neural network models implemented with on-the-shelf frameworks
(such as Tensorflow or Caffe) as media filters.


I have a few questions on how to register a new custom "typefind" for
a new non-media type, "tensor/tensors".



Because the "sources" are general media streams, such as "camera", "mic",
or "media files", the to-be-proposed type, "other/tensor" and
"other/tensors", do not have their file format, yet.
Their meta data are handled as pad-capabilities and
the buffer between elements contains the tensor data only
(similar to what video/x-raw does)

However, assuming that we need to register "typefind",
later to be considered to become "good plugin" some day,
it appears that we need to define headers as if they are
stored as files. (for debugging and testing purpose, we may need this anyway)



Q1. Is it ok to define "other/tensor(s)" type to have some headers
(including dimensions of tensors) when it is stored in file and
not to have any meta data in inter-gst-element transfer (like video/x-raw)?

Q2. If Q1 == NO, is it ok to define "other/tensorsave" to represent
the saved file format for tensors and use "other/tensor(s)" for
video/x-raw-like handling?

Q3. Or is it ok not to register any and possible to upstream to
gst-plugins-good with these new media types (other/tensor and other/tensors)?



As I've mentioned in Gstreamer-IRC back in March (sorry, I was away from
this project for a while and returned), we've finally got internal approval
to fully open sources (In LGPL-2.1.)
We will be migrating the code to github.com/nnsuite soon, probably in this week.


The full migration requires CI/CD systems to be migrated as well,
so the migration of developing process might take some time.



We hope, someday, we can upstream the plugins to gst-plugins-good. :)





Cheers,
MyungJoo

--

MyungJoo Ham (함명주), Ph.D.
Autonomous Machine Lab., AI Center, Samsung Research.
Cell: [phone]


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[David Ing]

I apologize for all the emails.  I botched something else:  I had incorrect comments around the `ptr` argument.

Should be:

// Executes typefind logic for "cust/om" media type
// @param ptr  An optional pointer to userdata
static void gst_custom_typefind(GstTypeFind *typeFind, gpointer ptr)

On Mon, Jul 23, 2018 at 10:00 AM, David Ing <[hidden email]> wrote:

Sorry, I borked that code example a bit.

The typefind function should be more like this

// Executes typefind logic for "cust/om" media type
// @param ptr  A pointer to the first byte of the stream.
static void gst_custom_typefind(GstTypeFind *typeFind, gpointer ptr)
{
    int64_t offset = 0;
    uint32_t size = 100;
    const guint8 *data = gst_type_find_peek(typeFind, offset, size);

    if (data && data[0] == 'C' && data[1] == 'U' && data[2] == 'S' && data[3] == 'T')
    {
        // extract other data from the stream so you can feed it into the caps
        // replace the `...` in the next statement with the caps data.  See documentation
        // for `gst_caps_new_simple` for more information.

        GstCaps* caps = gst_caps_new_simple("cust/om", ...);
        gst_type_find_suggest(typeFind, GST_TYPE_FIND_MAXIMUM, caps);
        gst_caps_unref(caps);
    }
}

On Mon, Jul 23, 2018 at 9:36 AM, David Ing <[hidden email]> wrote:

I do not have an answer to your questions, but I noticed a documentation page:

https://gstreamer.freedesktop.org/documentation/plugin-development/advanced/media-types.html#table-of-other-types

At the very bottom of that page, I noticed an empty table "Table of Other Types" which seems to suggest that you can define any type you want (with any prefix, not just "audio/" or "video/").

I am guessing that you would simply write a plugin with a typefind function like this ...

// Executes typefind logic for "cust/om" media type
// @param ptr  A pointer to the first byte of the stream.
static void gst_custom_typefind(GstTypeFind *typeFind, gpointer ptr)
{
    int64_t offset = 0;
    uint32_t size = 100;
    const guint8 *data = gst_type_find_peek(typeFind, offset, size);

    if (data && data[0] == 'C' && data[1] == 'U' && data[2] == 'S' && data[3] == 'T')
    {
        GstCaps* caps = ...
        gst_type_find_suggest(typeFind, GST_TYPE_FIND_MAXIMUM, gst_caps_new_simple("cust/om", caps));
        gst_caps_unref(caps);
    }
}

And in your `plugin_init` method, you would need to register the typefind function using `gst_type_find_register`.

Notice that in the typefind function, you can define the caps of the stream (such as the dimensionality of your tensor).  I believe that the caps structure can contain any fields you want.

Next your plugin should define a pad template having SINK caps which are compatible with the caps that would be detected by your typefind function .  Your plugin can have whatever SRC caps you want (i.e. "video/x-raw") ... but it needs to contain the code which transforms the stream from SINK to SRC.

I can't see any reason why that wouldn't work, but I am just a novice:  Someone should correct me if I am wrong.

On Mon, Jul 23, 2018 at 1:07 AM, MyungJoo Ham <[hidden email]> wrote:

Dear Gstreamer Developers,


We have been working on developing Gstreamer plugins to handle
general neural network models implemented with on-the-shelf frameworks
(such as Tensorflow or Caffe) as media filters.


I have a few questions on how to register a new custom "typefind" for
a new non-media type, "tensor/tensors".



Because the "sources" are general media streams, such as "camera", "mic",
or "media files", the to-be-proposed type, "other/tensor" and
"other/tensors", do not have their file format, yet.
Their meta data are handled as pad-capabilities and
the buffer between elements contains the tensor data only
(similar to what video/x-raw does)

However, assuming that we need to register "typefind",
later to be considered to become "good plugin" some day,
it appears that we need to define headers as if they are
stored as files. (for debugging and testing purpose, we may need this anyway)



Q1. Is it ok to define "other/tensor(s)" type to have some headers
(including dimensions of tensors) when it is stored in file and
not to have any meta data in inter-gst-element transfer (like video/x-raw)?

Q2. If Q1 == NO, is it ok to define "other/tensorsave" to represent
the saved file format for tensors and use "other/tensor(s)" for
video/x-raw-like handling?

Q3. Or is it ok not to register any and possible to upstream to
gst-plugins-good with these new media types (other/tensor and other/tensors)?



As I've mentioned in Gstreamer-IRC back in March (sorry, I was away from
this project for a while and returned), we've finally got internal approval
to fully open sources (In LGPL-2.1.)
We will be migrating the code to github.com/nnsuite soon, probably in this week.


The full migration requires CI/CD systems to be migrated as well,
so the migration of developing process might take some time.



We hope, someday, we can upstream the plugins to gst-plugins-good. :)





Cheers,
MyungJoo

--

MyungJoo Ham (함명주), Ph.D.
Autonomous Machine Lab., AI Center, Samsung Research.
Cell: [phone]


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[MyungJoo Ham]

>
>I do not have an answer to your questions, but I noticed a documentation
>page:
>
>https://gstreamer.freedesktop.org/documentation/plugin-development/advanced/media-types.html#table-of-other-types
>
>
>At the very bottom of that page, I noticed an empty table "Table of Other
>Types" which seems to suggest that you can define any type you want (with
>any prefix, not just "audio/" or "video/").

Thanks.

We have implemented typefind function in the plugin init that finds
"other/tensorsave" that recognize "saved" format of tensors (with headers).

Then, the type(cap) goes between related gstreamer elements is
header-less "other/tensor", which resembles "video/x-raw".

I wanted to ask if this is a desired ("good enough") approach.


Anyway, we have started opening "NNStreamer" plugins at
github.com/nnsuite/nnstreaer , which is to allow gstreamer
pipelines to use recent neural network frameworks and their models
as yet another media filters.


Cheers,
MyungJoo
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel