Audio-resampling isn't actually performed?

[Lusine]

Hi Folks,

I have my custom audio plugin based on audioaggregator.

I have 2 input audio streams with different sample rates and I'm using audioresample element to resample the audio streams to the same rate before passing them to my custom plugin. In my custom plugin I've overwritten aggregate function where I print GstMapInfo.size of audio buffers and I expect that buffers from 2 sinkpads which resampled before have the same size but they are different, is it mean that audio resampling wasn't actually performed before?

Regards,

Lusine

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Matthew Waters]

The size of the input audio buffer can be different or the same for any number of reasons.  e.g. the input chains upstream may produce buffers of a completely different number of audio samples. There is too little information to point you in any meaningful direction.

Cheers
-Matt

On 12/11/20 2:51 am, Lusine Hayrapetyan wrote:

Hi Folks,

I have my custom audio plugin based on audioaggregator.

I have 2 input audio streams with different sample rates and I'm using audioresample element to resample the audio streams to the same rate before passing them to my custom plugin. In my custom plugin I've overwritten aggregate function where I print GstMapInfo.size of audio buffers and I expect that buffers from 2 sinkpads which resampled before have the same size but they are different, is it mean that audio resampling wasn't actually performed before?

Regards,

Lusine

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Lusine]

Hi Matt,
Thanks for your answer. I will try to provide more details:
I have 2 audio streams with 22050 and 48000 audio sample rates and I have my
custom audio plugin based on GstAudioAggregator class where I only overwrite
aggregate() virtual method - it's called when buffers are queued on all
sinkpads. Please note that before connecting to my custom plugin both
streams are resampled to 48000 rate with audioresample element. Here is the
aggregate function:

static GstFlowReturn aggregate(GstAggregator * agg, gboolean timeout)
{
        GstFlowReturn ret;
        gboolean pad_eos = false;
        GstBuffer *outbuf = NULL;
        GST_OBJECT_LOCK(agg);
        for (size_t i = 0; i < g_list_length(GST_ELEMENT(agg)->sinkpads); ++i)
        {
                GstAudioAggregatorPad *aaggpad = (GstAudioAggregatorPad *)
(g_list_nth_data(GST_ELEMENT(agg)->sinkpads, i));
                // This prints 48000 for both of the pads. Which is expected because
streams are resampled before.
                g_print("GST_AUDIO_INFO_RATE %d\n", GST_AUDIO_INFO_RATE (&aaggpad->info));
               
                GstAggregatorPad *aggpad = (GstAggregatorPad *)
(g_list_nth_data(GST_ELEMENT(agg)->sinkpads, i));
                pad_eos = gst_aggregator_pad_is_eos(aggpad);
                if (!pad_eos)
                {
                        outbuf = gst_aggregator_pad_pop_buffer(aggpad);
                        GstMapInfo map;
                        if (gst_buffer_map(outbuf, &map, GST_MAP_READ))
                        {
                                // Here audio buffer size is 4096 for the first input and 8916 for the
second input.
                                // Is it expected? Why audio buffers have different sizes? Isn't it
expected to have the same size because they have the same audio rate.
                                g_print("outbuf map.size %d\n", map.size);
                        }
                }
        }
}

Please see my comments in the above code. Thank you very much for your time.
Regads,
Lusine



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Matthew Waters]

On 12/11/20 8:00 pm, Lusine wrote:

> Hi Matt,
> Thanks for your answer. I will try to provide more details:
> I have 2 audio streams with 22050 and 48000 audio sample rates and I have my
> custom audio plugin based on GstAudioAggregator class where I only overwrite
> aggregate() virtual method - it's called when buffers are queued on all
> sinkpads. Please note that before connecting to my custom plugin both
> streams are resampled to 48000 rate with audioresample element. Here is the
> aggregate function:
>
> static GstFlowReturn aggregate(GstAggregator * agg, gboolean timeout)
> {
> GstFlowReturn ret;
> gboolean pad_eos = false;
> GstBuffer *outbuf = NULL;
> GST_OBJECT_LOCK(agg);
> for (size_t i = 0; i < g_list_length(GST_ELEMENT(agg)->sinkpads); ++i)
> {
> GstAudioAggregatorPad *aaggpad = (GstAudioAggregatorPad *)
> (g_list_nth_data(GST_ELEMENT(agg)->sinkpads, i));
> // This prints 48000 for both of the pads. Which is expected because
> streams are resampled before.
> g_print("GST_AUDIO_INFO_RATE %d\n", GST_AUDIO_INFO_RATE (&aaggpad->info));
>
> GstAggregatorPad *aggpad = (GstAggregatorPad *)
> (g_list_nth_data(GST_ELEMENT(agg)->sinkpads, i));
> pad_eos = gst_aggregator_pad_is_eos(aggpad);
> if (!pad_eos)
> {
> outbuf = gst_aggregator_pad_pop_buffer(aggpad);
> GstMapInfo map;
> if (gst_buffer_map(outbuf, &map, GST_MAP_READ))
> {
> // Here audio buffer size is 4096 for the first input and 8916 for the
> second input.
> // Is it expected? Why audio buffers have different sizes? Isn't it
> expected to have the same size because they have the same audio rate.

No, the buffer size is not a direct relation with the sample rate of the
audio.  There can be any number of samples within an audio buffer that
still have the same sample rate.

It sounds like you probably want to base your element on
GstAudioAggregator though for the audio-specifics.

> g_print("outbuf map.size %d\n", map.size);
> }
> }
> }
> }
>
> Please see my comments in the above code. Thank you very much for your time.
> Regads,
> Lusine
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Thu, 2020-11-12 at 20:44 +1100, Matthew Waters wrote:

No, the buffer size is not a direct relation with the sample rate of the 

audio.  There can be any number of samples within an audio buffer that 

still have the same sample rate.

It sounds like you probably want to base your element on 

GstAudioAggregator though for the audio-specifics.

Or alternatively you could make use of GstAdapter for converting the input buffers with arbitrary number of samples into equally-sized buffers.

However GstAudioAggregator also takes care of inter-stream synchronization and various other things that are not too easy to reimplement, so if you need that it would be a better idea to use it instead.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Thu, 2020-11-12 at 12:03 +0200, Sebastian Dröge wrote:

On Thu, 2020-11-12 at 20:44 +1100, Matthew Waters wrote:

No, the buffer size is not a direct relation with the sample rate of the 

audio.  There can be any number of samples within an audio buffer that 

still have the same sample rate.

It sounds like you probably want to base your element on 

GstAudioAggregator though for the audio-specifics.

Or alternatively you could make use of GstAdapter for converting the input buffers with arbitrary number of samples into equally-sized buffers.

However GstAudioAggregator also takes care of inter-stream synchronization and various other things that are not too easy to reimplement, so if you need that it would be a better idea to use it instead.

And I forgot to add that when using GstAudioAggregator you should not override GstAggregator::aggregate() but instead GstAudioAggregator::create_output_buffer() and GstAudioAggregator::aggregate_one_buffer().

If you override the former then all the processing logic of GstAudioAggregator is disabled.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Thu, 2020-11-12 at 05:16 -0600, Lusine wrote:
> Hi Folks,
> Thanks for your feedback. My element is based on GstAudioAggregator and it's
> pretty simple - I only overwrite base aggregate function in my element.

If you do that you don't get any of the features of GstAudioAggregator.

You must only override the create_output_buffer() and
aggregate_one_buffer() virtual methods.

>
> [...]
>
> It  works correctly when both my inputs(sink pads) have the same audio rate
> but in case of inputs have different sample rates I get the following error
> message even though I do resampling(bringing to the same audio rate with
> audioresample) before passing to the audio aggregator element:
>
> ERROR                  qtmux gstqtmux.c:4535:gst_qt_mux_add_buffer:
> decreasing DTS value 0:00:08.845333332 < 0:00:18.612919076
>
>
> Please let me know what I'm missing. Why this code works when my input files
> have the same audio rate and why I get this error when both my inputs have
> different rates even though I resample the audio streams to the same rate
> with audioresample element before passing them to my custom plugin.

That suggests that you don't synchronize streams correctly inside your
custom element. Hard to say without a runnable testcase for this
problem.

But this is unlikely to happen if you use GstAudioAggregator correctly,
i.e. don't override the GstAggregator::aggregate() virtual method.

--
Sebastian Dröge, Centricular Ltd · https://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel