How to operate on GstMapInfo

[dingoegret]

I'm sending an rtp stream of audio to gstreamer and I have a pipeline with appsink. I've got a g-signal_connect callback function registered on 'new-sample' and this works fine. In that callback function I'm retrieving the audio data via the function:

static void new_sample(GstElement *sink) {
        GstSample *sample;
        g_signal_emit_by_name(sink, "pull-sample", &sample, NULL);

        if(sample) {
                GstBuffer* buffer = gst_sample_get_buffer(sample);
                GstMapInfo info;
                gst_buffer_map(buffer, &info, GST_MAP_READ);

                //fwrite(info.data, info.size, 1, myfile);
                std::cout << info.data[info.size + 10] << std::endl;

                gst_buffer_unmap (buffer, &info);
                gst_sample_unref(sample);
        }
}


Now the fwrite line seems to be working properly. The audio data gets written to a file and that file plays the audio fine. Here is what I don't get though, and keep in mind I am c/c++/memory management programming noob. What format is the info.data (*GstMapInfo.data) in? Is it an array? A pointer to something? An array of characters or zeroes and ones or what exactly. The specific problem I face is that I am trying to use a google speech api

#include "google/cloud/speech/v1beta1/cloud_speech.grpc.pb.h"

and one particular function, which is

StreamingRecognizeRequest::set_audio_content(const void* value, size_t size)

accepts a void*? What does that mean? iS this compatible with GstMapInfo.data? Furthermore, after some tests it appears that this function only works properly (google sends back transcribed results) if you send it at minimum 16 kilobytes of data increments and no less. So even if I figure out how to work with the GstMapInfo.data to use it with set_audio_content, I will have to also figure out how to chunk the data into 16kb each.

Here is an example function which uses google speech api for reference

static void MicrophoneThreadMain(
        grpc::ClientReaderWriterInterface<StreamingRecognizeRequest,
                StreamingRecognizeResponse>* streamer,
        char* file_path) {
    StreamingRecognizeRequest request;
    std::ifstream file_stream(file_path);
    const size_t chunk_size = 64 * 1024;
    std::vector<char> chunk(chunk_size);
    while (true) {
        // Read another chunk from the file.
        std::streamsize bytes_read =
                file_stream.rdbuf()->sgetn(&chunk[0], chunk.size());
        // And write the chunk to the stream.
        request.set_audio_content(&chunk[0], bytes_read);
        std::cout << "Sending " << bytes_read / 1024 << "k bytes." << std::endl;
        streamer->Write(request);
        if (bytes_read < chunk.size()) {
            // Done reading everything from the file, so done writing to the stream.
            streamer->WritesDone();
            break;
        } else {
            // Wait a second before writing the next chunk.
            std::this_thread::sleep_for(std::chrono::seconds(1));
        }
    }
}

Any insight into this would be greatly appreciated

[Andrew Grace]

> On Mar 13, 2017, at 11:17 PM, dingoegret <[hidden email]> wrote:
>
> I'm sending an rtp stream of audio to gstreamer and I have a pipeline with
> appsink. I've got a g-signal_connect callback function registered on
> 'new-sample' and this works fine. In that callback function I'm retrieving
> the audio data via the function:
>
> static void new_sample(GstElement *sink) {
>        GstSample *sample;
>        g_signal_emit_by_name(sink, "pull-sample", &sample, NULL);
>
>        if(sample) {
>                GstBuffer* buffer = gst_sample_get_buffer(sample);
>                GstMapInfo info;
>                gst_buffer_map(buffer, &info, GST_MAP_READ);
>
>                //fwrite(info.data, info.size, 1, myfile);
>                std::cout << info.data[info.size + 10] << std::endl;
>
>                gst_buffer_unmap (buffer, &info);
>                gst_sample_unref(sample);
>        }
> }
>
>
> Now the fwrite line seems to be working properly. The audio data gets
> written to a file and that file plays the audio fine. Here is what I don't
> get though, and keep in mind I am c/c++/memory management programming noob.

Gstreamer is written in C, not C++.  C is a simpler language, so I'd avoid C++ if possible.  The GLib library fills in much of what the standard C library lacks.  For trying stuff out, I'd also recommend using the Python bindings pygi (python gobject inspection).

> What format is the info.data (*GstMapInfo.data) in? Is it an array? A
> pointer to something? An array of characters or zeroes and ones or what
> exactly. The specific problem I face is that I am trying to use a google
> speech api
>
> #include "google/cloud/speech/v1beta1/cloud_speech.grpc.pb.h"
>
> and one particular function, which is
>
> StreamingRecognizeRequest::set_audio_content(const void* value, size_t size)
>
> accepts a void*? What does that mean?

A void* (void pointer) is awkward syntax (I think C overuses the star - makes things confusing in the beginning).  Void ptr is a data type that lets you accept a pointer of any type without the C compiler complaining of a type error.  You can cast any pointer to void* and vice versa.  It's up to you to know that what you are doing makes sense.  I've seen it used a lot, especially for libraries that create opaque objects (a struct allocated at the other end of that void pointer handle).

If C is totally new for you, maybe check out the classic K&R C book.  But there are also many tutorials online.  I even liked the coverage of C in the Big Nerd Ranch Objective-C book (the first couple chapters are just plain C).  They write in a very clear and concise way in that book.

> iS this compatible with
> GstMapInfo.data? Furthermore, after some tests it appears that this function
> only works properly (google sends back transcribed results) if you send it
> at minimum 16 kilobytes of data increments and no less. So even if I figure
> out how to work with the GstMapInfo.data to use it with set_audio_content, I
> will have to also figure out how to chunk the data into 16kb each.
>
> Here is an example function which uses google speech api for reference
>
> static void MicrophoneThreadMain(
>        grpc::ClientReaderWriterInterface<StreamingRecognizeRequest,
>                StreamingRecognizeResponse>* streamer,
>        char* file_path) {
>    StreamingRecognizeRequest request;
>    std::ifstream file_stream(file_path);
>    const size_t chunk_size = 64 * 1024;
>    std::vector<char> chunk(chunk_size);
>    while (true) {
>        // Read another chunk from the file.
>        std::streamsize bytes_read =
>                file_stream.rdbuf()->sgetn(&chunk[0], chunk.size());
>        // And write the chunk to the stream.
>        request.set_audio_content(&chunk[0], bytes_read);
>        std::cout << "Sending " << bytes_read / 1024 << "k bytes." <<
> std::endl;
>        streamer->Write(request);
>        if (bytes_read < chunk.size()) {
>            // Done reading everything from the file, so done writing to the
> stream.
>            streamer->WritesDone();
>            break;
>        } else {
>            // Wait a second before writing the next chunk.
>            std::this_thread::sleep_for(std::chrono::seconds(1));
>        }
>    }
> }
>
> Any insight into this would be greatly appreciated
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/How-to-operate-on-GstMapInfo-tp4682202.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le lundi 13 mars 2017 à 20:17 -0700, dingoegret a écrit :

> I'm sending an rtp stream of audio to gstreamer and I have a pipeline
> with
> appsink. I've got a g-signal_connect callback function registered on
> 'new-sample' and this works fine. In that callback function I'm
> retrieving
> the audio data via the function:
>
> static void new_sample(GstElement *sink) {
>         GstSample *sample;
>         g_signal_emit_by_name(sink, "pull-sample", &sample, NULL);
>
>         if(sample) {
>                 GstBuffer* buffer = gst_sample_get_buffer(sample);
>                 GstMapInfo info;
>                 gst_buffer_map(buffer, &info, GST_MAP_READ);
>
>                 //fwrite(info.data, info.size, 1, myfile);
>                 std::cout << info.data[info.size + 10] << std::endl;
>
>                 gst_buffer_unmap (buffer, &info);
>                 gst_sample_unref(sample);
>         }
> }
>
>
> Now the fwrite line seems to be working properly. The audio data gets
> written to a file and that file plays the audio fine. Here is what I
> don't
> get though, and keep in mind I am c/c++/memory management programming
> noob.
> What format is the info.data (*GstMapInfo.data) in? Is it an array? A
> pointer to something? An array of characters or zeroes and ones or
> what
> exactly. The specific problem I face is that I am trying to use a
> google
> speech api

It's an array of bytes, guint8 *, see:

https://developer.gnome.org/gstreamer/stable/gstreamer-GstMemory.html#G
stMapInfo

>
> #include "google/cloud/speech/v1beta1/cloud_speech.grpc.pb.h"
>
> and one particular function, which is 
>
> StreamingRecognizeRequest::set_audio_content(const void* value,
> size_t size)
>
> accepts a void*? What does that mean? iS this compatible with
> GstMapInfo.data? Furthermore, after some tests it appears that this
> function

Yes, all pointers are compatible with void *. In general you don't even
have to cast, but in C++ it's more restrictive.

> only works properly (google sends back transcribed results) if you
> send it
> at minimum 16 kilobytes of data increments and no less. So even if I
> figure
> out how to work with the GstMapInfo.data to use it with
> set_audio_content, I
> will have to also figure out how to chunk the data into 16kb each.

GStreamer provides GstAdapter, which is a utility to accumulate buffers
and chunk them. See:

https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-lib
s/html/GstAdapter.html

>
> Here is an example function which uses google speech api for
> reference
>
> static void MicrophoneThreadMain(
>         grpc::ClientReaderWriterInterface<StreamingRecognizeRequest,
>                 StreamingRecognizeResponse>* streamer,
>         char* file_path) {
>     StreamingRecognizeRequest request;
>     std::ifstream file_stream(file_path);
>     const size_t chunk_size = 64 * 1024;
>     std::vector<char> chunk(chunk_size);
>     while (true) {
>         // Read another chunk from the file.
>         std::streamsize bytes_read =
>                 file_stream.rdbuf()->sgetn(&chunk[0], chunk.size());
>         // And write the chunk to the stream.
>         request.set_audio_content(&chunk[0], bytes_read);
>         std::cout << "Sending " << bytes_read / 1024 << "k bytes." <<
> std::endl;
>         streamer->Write(request);
>         if (bytes_read < chunk.size()) {
>             // Done reading everything from the file, so done writing
> to the
> stream.
>             streamer->WritesDone();
>             break;
>         } else {
>             // Wait a second before writing the next chunk.
>             std::this_thread::sleep_for(std::chrono::seconds(1));
>         }
>     }
> }
>
> Any insight into this would be greatly appreciated
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble
> .com/How-to-operate-on-GstMapInfo-tp4682202.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[dingoegret]

I'm attempting to use GstAdapter but I can't seem to get it to be included for the compiler.

I get a undefined reference to `gst_adapter_push'
undefined reference to `gst_adapter_available'
undefined reference to `gst_adapter_map'
collect2: error: ld returned 1 exit status

Is this normal? I've tried `#include <gst/base/gstadapter.h>` and the way I'm compiling it is by

g++ -std=c++11 -I/usr/local/include -pthread -I/home/dingo/googleapis/gens  -c -o main4.o main4.cpp `pkg-config --cflags --libs gstreamer-1.0`
&&
g++ main4.o googleapis.ar -L/usr/local/lib `pkg-config --libs grpc++ grpc gstreamer-1.0` -Wl,--no-as-needed -lgrpc++_reflection -Wl,--as-needed -lprotobuf -lpthread -ldl -o main

I never had a problem compiling with gstreamer until using GstAdapter. What am I doing wrong?


> View this message in context: http://gstreamer-devel.966125.n4.nabble
> .com/How-to-operate-on-GstMapInfo-tp4682202.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [e-mail]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
_______________________________________________
gstreamer-devel mailing list
[e-mail]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel


signature.asc (188 bytes)
<http://gstreamer-devel.966125.n4.nabble.com/attachment/4682220/0/signature.asc>


Quoted from:
http://gstreamer-devel.966125.n4.nabble.com/How-to-operate-on-GstMapInfo-tp4682202p4682220.html

[dingoegret]

Okay I have it working so far and I'm able to chunk at 16kb. Is the data coming out of gst_adapter_map still a byte array? Why is it returned as a gconstpointer (void*)? What's the difference between void* and a pointer to a unsigned char? How are they connected to the concept of byte arrays? I just want to uderstand the basic data happening here. In higher level languages data is less ambiguous. An int is an int and a string is a string, even if they are implemented differently underneath so I'm coming from that background I'm really confused what a byte array is and why isnt the data type set as something like byte[] instead of a pointer to an unsigned char*.

[Andrew Grace]

Sent from my iPhone
> On Mar 15, 2017, at 5:52 AM, dingoegret <[hidden email]> wrote:
>
> Okay I have it working so far and I'm able to chunk at 16kb. Is the data
> coming out of gst_adapter_map still a byte array? Why is it returned as a
> gconstpointer (void*)? What's the difference between void* and a pointer to
> a unsigned char?

Those are different data types.  void pointer is a pointer (holds a memory address), and any kind of pointer (ptr to int, ptr to float, ptr to char, whatever) can be casted to void ptr amd vice versa.

Unsigned char is a datatype that holds one byte of data.  Unless you are doing bit twiddling, pointer to unsigned char or pointer to char are interchangable dealing with a buffer.

If this is new then consider reading the K&R C book (isbn 0131103628).  Its very old, but it is short and sweet.
 
> How are they connected to the concept of byte arrays?

Byte arrays sounds like a python term.  Python has a nice abstraction for representing a block of memory.  C's abstraction is more old-school... but with faster performance.  A char* buffer is the same as a python byte array conceptually, though Python wraps the memory buffer in a python object.  C does not... C is less abstracted from the underlying machine instructions... and faster.

> I
> just want to uderstand the basic data happening here. In higher level
> languages data is less ambiguous.

C feels less ambiguous when you get familiar with it.  It's just less abstracted from the machine lang than a higher level lang.  I've heard C described as a high level assembly language.

> An int is an int and a string is a string,
> even if they are implemented differently underneath so I'm coming from that
> background I'm really confused what a byte array is and why isnt the data
> type set as something like byte[] instead of a pointer to an unsigned char*.
>
With C, you really just need take time and do some reading or tutorials.  It has features you dont see used elsewhere and lacks other features.  For example pointers and arrays in C are closely related.  You can treat pointers as arrays and arrays as pointers.


>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/How-to-operate-on-GstMapInfo-tp4682202p4682228.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mercredi 15 mars 2017 à 02:52 -0700, dingoegret a écrit :
> Okay I have it working so far and I'm able to chunk at 16kb. Is the data
> coming out of gst_adapter_map still a byte array? Why is it returned as a
> gconstpointer (void*)?

Ask the author. Just cast it to const guint8 * and your good. It's
const because when you use gst_adapter_map(), writing to the memory may
have no effect on the gst_buffers inside the adapter. This method may
make a copy if the chunk was overlapping two memory segments.

Note that you should probably use gst_adapter_take() here.

> What's the difference between void* and a pointer to
> a unsigned char? How are they connected to the concept of byte arrays? I
> just want to uderstand the basic data happening here. In higher level

The alignment of a void* is unknown. It would point to a structure, a
function, data array etc. Though, in this case, it's just data, which
is application specific. The adapter is media agnostic remember. In
your case, you know what this data is.


> languages data is less ambiguous. An int is an int and a string is a string,
> even if they are implemented differently underneath so I'm coming from that
> background I'm really confused what a byte array is and why isnt the data
> type set as something like byte[] instead of a pointer to an unsigned char*.

All languages are similar (or more limited) in regard to byte array.
This paragraph is just a rant in my opinion. It's not helpful to get
help from community members.

Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[dingoegret]

with gst_adapter_available i can get the size (gsize) of bytesof char data in the adapter.

with gst_adapter_map i can get the actual data as const void* (gconstpointer).

now if i cast the gconstpointer to an unsigned *char, does that mean i get back an array of pointers to char bytes where the size of the array of pointers is the returned gsize? as in i could access data[0], data[1], data[2] ... data[returned_gsize]? and get back pointers to the actual singular bytes of supposed unsigned chars? this is what im trying to understand. What is the data?

[Andrew Grace]

Sent from my iPhone

> On Mar 16, 2017, at 5:30 AM, dingoegret <[hidden email]> wrote:
>
> with gst_adapter_available i can get the size (gsize) of bytesof char data in
> the adapter.
>
> with gst_adapter_map i can get the actual data as const void*
> (gconstpointer).
>
> now if i cast the gconstpointer to an unsigned *char, does that mean i get
> back an array of pointers to char bytes where the size of the array of
> pointers is the returned gsize? as in i could access data[0], data[1],
> data[2] ... data[returned_gsize]? and get back pointers to the actual
> singular bytes of supposed unsigned chars? this is what im trying to
> understand. What is the data?

That sounds right, except I'd expect the last valid array index to be size-1.  The unsigned chars are your bytes.  Each unsigned char is 8 bits in width (1 byte). The cast does not change the underlying bit string, just how you interpret that string as a number.  If it's unsigned char, the bit values of each byte are interpreted in the range [0,255], and if it is signed char is [-128,127].  Char is used rather than int just because it is guaranteed to be exactly 8 bits in width on every playform (because stuff like data type sizes vary per platform with C).



>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/How-to-operate-on-GstMapInfo-tp4682202p4682252.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le 16 mars 2017 5:45 AM, "dingoegret" <[hidden email]> a écrit :

with gst_adapter_available i can get the size (gsize) of bytesof char data in
the adapter.

with gst_adapter_map i can get the actual data as const void*
(gconstpointer).

now if i cast the gconstpointer to an unsigned *char, does that mean i get
back an array of pointers to char bytes where the size of the array of
pointers is the returned gsize? as in i could access data[0], data[1],
data[2] ... data[returned_gsize]? and get back pointers to the actual
singular bytes of supposed unsigned chars? this is what im trying to
understand. What is the data?

Yes, the size is in bytes. And by indexing it you get the value, not the pointer. Also, index starts at 0, so the max is data[size - 1]

--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/How-to-operate-on-GstMapInfo-tp4682202p4682252.html

Sent from the GStreamer-devel mailing list archive at Nabble.com.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[dingoegret]

Okay so there is definitely a difference with the returned data from gstreamer and returned data from say the

char* file_path = "resources/sample3.wav";
std::ifstream file_stream(file_path);
const size_t chunk_size = 16384;
std::vector<char> chunk(chunk_size);
std::streamsize bytes_read = file_stream.rdbuf()->sgetn(&chunk[0], chunk.size());

because I am sending the "resources/sample3.wav" to gstreamer, retrieving 16kb of data via GstAdapter and changing the gconstpointer to a vector

size = 16384;
gconstpointer data = gst_adapter_map (adapter, size);
char* temp = const_cast<char*>(reinterpret_cast<const char*>(static_cast<const unsigned char*>(data)));
std::vector<char> new_chunk(temp, temp + size);

new_chunk == chunk //false..

my pipeline isnt doing anything to the sample3.wav file. All I am doing is

gst_element_link_many(udpsrc, rtpL16depay, appsink, NULL);
g_signal_connect(appsink, "new-sample", G_CALLBACK (new_sample), NULL);

and I'm sending the sample3.wav via the gst-launch command

sudo gst-launch-1.0 filesrc location=~/gstreamer/resources/sample3.wav ! wavparse ! audioconvert ! rtpL16pay ! udpsink port=5000

I'm confused why the bytes of audio data come out differently in gstreamer from the bytes of audio data coming out from the file stream reader

[Nicolas Dufresne-5]

Le vendredi 17 mars 2017 à 02:13 -0700, dingoegret a écrit :
> sudo gst-launch-1.0 filesrc
> location=~/gstreamer/resources/sample3.wav !
> wavparse ! audioconvert ! rtpL16pay ! udpsink port=5000
>
> I'm confused why the bytes of audio data come out differently in
> gstreamer
> from the bytes of audio data coming out from the file stream reader

Isn't that kind of evident ? wavparse removed the wav header,
audioconvert will convert to L16 as needed and rtpL16pay will add RTP
payload.

Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel