Asynchrounous Audio Sample Rate Conversion with gstreamer

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Asynchrounous Audio Sample Rate Conversion with gstreamer

Maik Scholz
Hi,

in some <span id="spans0e0" tooltip="{&lt;b&gt;use cases&lt;/b&gt;} &lt;i&gt;replaces&lt;/i&gt; &lt;br/&gt;{&lt;b&gt;&lt;s style='color:Red;'&gt;usecases&lt;/s&gt;&lt;/b&gt;}: wrong word" onmouseover="showtooltip(this,300);" onmouseout="hideddrivetip();" class="sac" onclick="showMenu(this);">use cases (broadcast digital radio or IP radio), the transmitter is sending an audio stream with a <span id="spans0e1" tooltip="{&lt;b&gt;nominal&lt;/b&gt;}: relating to or constituting or bearing or giving a name&lt;br/&gt;&lt;i&gt;replaces&lt;/i&gt;&lt;br/&gt; {&lt;b&gt;&lt;s style='color:Red;'&gt;norminal&lt;/s&gt;&lt;/b&gt;}: wrong word" onmouseover="showtooltip(this,300);" onmouseout="hideddrivetip();" class="sac" onclick="showMenu(this);">nominal sampling rate (e.g. 48kHz).
When a receiver, expecting 48KHz, wants to play <span id="spans0e2" tooltip="{&lt;b&gt;these samples&lt;/b&gt;} &lt;i&gt;replaces&lt;/i&gt; &lt;br/&gt;{&lt;b&gt;&lt;s style='color:Red;'&gt;this samples&lt;/s&gt;&lt;/b&gt;}: wrong word" onmouseover="showtooltip(this,300);" onmouseout="hideddrivetip();" class="sac" onclick="showMenu(this);">these samples on a local sound sink, then there is always the problem,
that the transmitter clock <span id="spans0e3" tooltip="{&lt;b&gt;slightly differs&lt;/b&gt;} &lt;i&gt;replaces&lt;/i&gt; &lt;br/&gt;{&lt;b&gt;&lt;s style='color:Red;'&gt;sligly differ&lt;/s&gt;&lt;/b&gt;}: wrong word" onmouseover="showtooltip(this,300);" onmouseout="hideddrivetip();" class="sac" onclick="showMenu(this);">slightly differs from the receiver clock.
For that issue, I need <span id="spans0e4" tooltip="{&lt;b&gt;an&lt;/b&gt;} &lt;i&gt;replaces&lt;/i&gt; &lt;br/&gt;{&lt;b&gt;&lt;s style='color:Red;'&gt;a&lt;/s&gt;&lt;/b&gt;}: wrong word" onmouseover="showtooltip(this,300);" onmouseout="hideddrivetip();" class="sac" onclick="showMenu(this);">an <span id="spans0e5" tooltip="{&lt;b&gt;asynchronous&lt;/b&gt;}: (digital communication) pertaining to a transmission technique that does not require a common clock between the communicating devices; timing signals are derived from special characters in the data stream itself&lt;br/&gt;&lt;i&gt;replaces&lt;/i&gt;&lt;br/&gt; {&lt;b&gt;&lt;s style='color:Red;'&gt;asynchrounous&lt;/s&gt;&lt;/b&gt;}: wrong word" onmouseover="showtooltip(this,300);" onmouseout="hideddrivetip();" class="sac" onclick="showMenu(this);">asynchronous sampling rate <span id="spans0e6" tooltip="{&lt;b&gt;converter&lt;/b&gt;}: a device for changing one substance or form or state into another&lt;br/&gt;&lt;i&gt;replaces&lt;/i&gt;&lt;br/&gt; {&lt;b&gt;&lt;s style='color:Red;'&gt;conmverter&lt;/s&gt;&lt;/b&gt;}: wrong word" onmouseover="showtooltip(this,300);" onmouseout="hideddrivetip();" class="sac" onclick="showMenu(this);">converter!

Is this possible the available <span style="text-decoration:none;background-color:yellow;" tooltip="{gstreamer} : unknown word" onmouseover="showtooltip(this,300);" onmouseout="hideddrivetip();">gstreamer plugins?
<span id="spans0e8" tooltip="{&lt;b&gt;Do&lt;/b&gt;} &lt;i&gt;replaces&lt;/i&gt; &lt;br/&gt;{&lt;b&gt;&lt;s style='color:Red;'&gt;Does&lt;/s&gt;&lt;/b&gt;}: wrong word" onmouseover="showtooltip(this,300);" onmouseout="hideddrivetip();" class="sac" onclick="showMenu(this);">Do <span style="text-decoration:none;background-color:yellow;" tooltip="{audioresample} : unknown word" onmouseover="showtooltip(this,300);" onmouseout="hideddrivetip();">audioresample support <span id="spans0e10" tooltip="{&lt;b&gt;asynchronous&lt;/b&gt;}: (digital communication) pertaining to a transmission technique that does not require a common clock between the communicating devices; timing signals are derived from special characters in the data stream itself&lt;br/&gt;&lt;i&gt;replaces&lt;/i&gt;&lt;br/&gt; {&lt;b&gt;&lt;s style='color:Red;'&gt;asynchrounous&lt;/s&gt;&lt;/b&gt;}: wrong word" onmouseover="showtooltip(this,300);" onmouseout="hideddrivetip();" class="sac" onclick="showMenu(this);">asynchronous clocks?

My example:
My Transmitter (sending samples with (48-1)kHz):
gst-launch-0.10 audiotestsrc freq=300 ! audio/x-raw-int, endianness="(int)1234", signed="(boolean)true", width="(int)16", depth="(int)16", rate="(int)47000", channels="(int)1" ! tcpclientsink host=localhost port=3000

My Receiver: (Expeting 48kHz)
gst-launch-0.10 tcpserversrc host=localhost port=3000 ! audio/x-raw-int, endianness="(int)1234", signed="(boolean)true", width="(int)16", depth="(int)16", rate="(int)48000", channels="(int)1" ! audioconvert ! audioresample ! audio/x-raw-int, endianness="(int)1234", signed="(boolean)true", width="(int)16", depth="(int)16", rate="(int)48000", channels="(int)1" ! autoaudiosink

Because the transmitter clock (47kHz) <span id="spans0e0" tooltip="{&lt;b&gt;differs&lt;/b&gt;} &lt;i&gt;replaces&lt;/i&gt; &lt;br/&gt;{&lt;b&gt;&lt;s style='color:Red;'&gt;differ&lt;/s&gt;&lt;/b&gt;}: wrong word" onmouseover="showtooltip(this,300);" onmouseout="hideddrivetip();" class="sac" onclick="showMenu(this);">differs from the receiver clock (48kHz), the sound drops <span id="spans0e2" tooltip="{&lt;b&gt;over&lt;/b&gt;} &lt;i&gt;replaces&lt;/i&gt; &lt;br/&gt;{&lt;b&gt;&lt;s style='color:Red;'&gt;after&lt;/s&gt;&lt;/b&gt;}: wrong word" onmouseover="showtooltip(this,300);" onmouseout="hideddrivetip();" class="sac" onclick="showMenu(this);">over short time.

Thank you for any hint in advance.

Maik

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Asynchrounous Audio Sample Rate Conversion with gstreamer

Sebastian Dröge-3
On Fr, 2016-04-29 at 21:11 +0200, Maik Scholz wrote:

> Hi, 
>
> in some use cases (broadcast digital radio or IP radio), the
> transmitter is sending an audio stream with a nominal sampling rate
> (e.g. 48kHz).
> When a receiver, expecting 48KHz, wants to play these samples on a
> local sound sink, then there is always the problem, that the
> transmitter clock slightly differs from the receiver clock.
> For that issue, I need an asynchronous sampling rate converter!
>
> Is this possible the available gstreamer plugins?
> Do audioresample support asynchronous clocks?
audioresample not, it's not really audioresample's job to do this kind
of work (but in your case you can use audioresample to resample the
audio as you know the ratio between sender and receiver rate).

This is something that can be implemented on various layers, e.g. with
RTP this would happen based on the RTP timestamps and arrival times
inside the rtpjitterbuffer. You don't have any timestamps inside the
stream so that won't do it.

In your case you would have to implement something similar somewhere
downstream of the tcpserversrc. Something that correlates the amount of
data received per time unit with the duration of that time unit and
then extrapolates the relative rate difference between the server and
receiver clock, and then accordingly timestamps the media, or directly
resamples the media (for which you can again use audioresample here).
One main problem in your case is that you use TCP, and due to all the
buffering that happens in TCP you will have it harder to calculate this
than with UDP. And as you don't have timestamps inside the stream, you
will have to assume that your media is continuous which might also not
be true.


And then (if you don't resample anywhere but just adjust timestamps),
it's the job of the audio sink in the end of make up for the difference
between timestamps and actual samples received. It will be default just
"skip" (skew) but you can configure it also to resample, this can be
done with the slave-method property on all audio sinks.
However the resampling that is currently implemented there is very
minimal and audioresample will give you better quality, but that's
something that should be fixed.


Does that answer your question? There's no standard solution for this
(as your streaming protocol is not and has many use case specific
assumptions) and you have to implement something yourself with the help
of existing GStreamer code.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

signature.asc (968 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Asynchrounous Audio Sample Rate Conversion with gstreamer

Sebastian Dröge-3
In reply to this post by Maik Scholz
On Fr, 2016-04-29 at 21:11 +0200, Maik Scholz wrote:


> My example:
> My Transmitter (sending samples with (48-1)kHz):
> gst-launch-0.10 audiotestsrc freq=300 ! audio/x-raw-int,
> endianness="(int)1234", signed="(boolean)true", width="(int)16",
> depth="(int)16", rate="(int)47000", channels="(int)1" ! tcpclientsink
> host=localhost port=3000
>
> My Receiver: (Expeting 48kHz)
> gst-launch-0.10 tcpserversrc host=localhost port=3000 ! audio/x-raw-
> int, endianness="(int)1234", signed="(boolean)true", width="(int)16",
> depth="(int)16", rate="(int)48000", channels="(int)1" ! audioconvert
> ! audioresample ! audio/x-raw-int, endianness="(int)1234",
> signed="(boolean)true", width="(int)16", depth="(int)16",
> rate="(int)48000", channels="(int)1" ! autoaudiosink
Also some notes regarding your specific pipelines. First of all, don't
use GStreamer 0.10 but use a recent 1.x version, e.g. 1.8.1. 0.10 is no
longer maintained since more than 3 years and you're unlikely to get
much support for it anywhere.

Then, you should make sure that downstream of tcpserversrc always a
multiple of full samples arrives. That is, in your case an even buffer
size. This is not guaranteed because TCP is a stream protocol.
The audioparse element can ensure that this is the case for you.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

signature.asc (968 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Asynchrounous Audio Sample Rate Conversion with gstreamer

Maik Scholz
Hi,
thank you very much for the hints.
With your help, my pipeline is running quite well with ONE audio source.

My target is, having multiply audio sources with different sample rates and different external clock sources.
That includes small sample rate mismatches (+- 50Hz) and drifts.

My mixer pipeline prototype:
GSTLAUNCH=gst-launch-1.0
STREAMDECHOST=127.0.0.1
PORT_SRC1=41001
PORT_SRC2=42001
PORT_SINK_DEBUG_1=45002

${GSTLAUNCH} \
                audiomixer name=sink_audiomixer latency=256000000 ! \
                tee name=sink_tee \
                \
                sink_tee. ! \
                queue name=outtcp_queue ! \
                tcpserversink name=tcpserversink host="0.0.0.0" port=${PORT_SINK_DEBUG_1} \
                \
                sink_tee. ! \
                queue name=outdsp_queue ! \
                autoaudiosink name=autoaudiosink sync=true slave-method=resample \
                \
                sink_tee. ! \
                queue name=file_queue ! \
                audioconvert name=fileconvert ! \
                filesink name=file_filesink location=/dev/null \
                \
                udpsrc name=src1_src port=${PORT_SRC1} do-timestamp=true ! \
                audio/x-raw,format=S16LE,channels=2,rate=32000 ! \
                queue name=src1_queue min-threshold-time=0 ! \
                audiorate name=src1_audiorate tolerance=256000000 skip-to-first=false ! \
                audioconvert name=src1_audioconvert ! \
                audioresample name=src1_audioresample quality=1 ! \
                audio/x-raw,format=S16LE,channels=2,rate=48000 ! \
                tee name=src1_tee \
                \
                src1_tee. ! \
                volume name=src1_volume volume=0.5 ! \
                sink_audiomixer. \
                \
                udpsrc name=src2_src port=${PORT_SRC2} do-timestamp=true ! \
                audio/x-raw,format=S16LE,channels=2,rate=48000 ! \
                queue name=src2_queue min-threshold-time=0 ! \
                audiorate name=src2_audiorate tolerance=256000000 skip-to-first=false ! \
                audioconvert name=src2_audioconvert ! \
                audioresample name=src2_audioresample quality=4 ! \
                audio/x-raw,format=S16LE,channels=2,rate=48000 ! \
                tee name=src2_tee \
                \
                src2_tee. ! \
                volume name=src2_volume volume=0.5 ! \
                sink_audiomixer.
               

My SRC1 audio transmitter (32000 - 50Hz):
gst-launch-1.0 audiotestsrc freq=100 ! audio/x-raw,format=S16LE,channels=2,rate=31950 ! udpsink host="127.0.0.1" port=41001

My SRC2 audio transmitter (48000 + 50Hz):
gst-launch-1.0 audiotestsrc freq=300 ! audio/x-raw,format=S16LE,channels=2,rate=48050 ! udpsink host="127.0.0.1" port=42001

When I run these pipelines with even the latest gst version, then I hear "clicks" from time to time.
I expect, that the audiomixer has some problems with the differ sampling frequencies.

My assumption is, I need a asynchronous SRC for each input (src1_audioresample+src2_audioresample).
How can I implement this with available elements. Or if not available, what is the best way to implement these?

Why is the audiomixer located in the bad plugins? What means bad?
Reply | Threaded
Open this post in threaded view
|

Re: Asynchrounous Audio Sample Rate Conversion with gstreamer

Carlos Rafael Giani
In most cases, there is only one clock (-> audio clock = pipeline
clock), but in your case, the pipeline clock is the monotonic system
clock, so audiobasesink needs to compensate for the drift.
The clicks are most likely coming from the "skew" drift compensation
method in the audiobasesink. This method looks at the drift, and if a
threshold is reached, moves the playout pointer (= an offset withing a
ring buffer), effectively inserting nullsamples or discarding samples,
depending on the skew direction.

You can hook in your own drift compensation method if you set the
audiobasesink's "slave-method" property to "custom". We use that in a
project to finetune the audio clock's speed via a PLL by an integer
number of ppm. It would be the same with an ASRC.

On 2016-05-25 15:04, Maik Scholz wrote:

> Hi,
> thank you very much for the hints.
> With your help, my pipeline is running quite well with ONE audio source.
>
> My target is, having multiply audio sources with different sample rates and
> different external clock sources.
> That includes small sample rate mismatches (+- 50Hz) and drifts.
>
> *My mixer pipeline prototype:*
> GSTLAUNCH=gst-launch-1.0
> STREAMDECHOST=127.0.0.1
> PORT_SRC1=41001
> PORT_SRC2=42001
> PORT_SINK_DEBUG_1=45002
>
> ${GSTLAUNCH} \
>                  audiomixer name=sink_audiomixer latency=256000000 ! \
>                  tee name=sink_tee \
>                  \
>                  sink_tee. ! \
>                  queue name=outtcp_queue ! \
>                  tcpserversink name=tcpserversink host="0.0.0.0"
> port=${PORT_SINK_DEBUG_1} \
>                  \
>                  sink_tee. ! \
>                  queue name=outdsp_queue ! \
>                  autoaudiosink name=autoaudiosink sync=true
> slave-method=resample \
>                  \
>                  sink_tee. ! \
>                  queue name=file_queue ! \
>                  audioconvert name=fileconvert ! \
>                  filesink name=file_filesink location=/dev/null \
>                  \
>                  udpsrc name=src1_src port=${PORT_SRC1} do-timestamp=true ! \
>                  audio/x-raw,format=S16LE,channels=2,rate=32000 ! \
>                  queue name=src1_queue min-threshold-time=0 ! \
>                  audiorate name=src1_audiorate tolerance=256000000
> skip-to-first=false ! \
>                  audioconvert name=src1_audioconvert ! \
>                  audioresample name=src1_audioresample quality=1 ! \
>                  audio/x-raw,format=S16LE,channels=2,rate=48000 ! \
>                  tee name=src1_tee \
>                  \
>                  src1_tee. ! \
>                  volume name=src1_volume volume=0.5 ! \
>                  sink_audiomixer. \
>                  \
>                  udpsrc name=src2_src port=${PORT_SRC2} do-timestamp=true ! \
>                  audio/x-raw,format=S16LE,channels=2,rate=48000 ! \
>                  queue name=src2_queue min-threshold-time=0 ! \
>                  audiorate name=src2_audiorate tolerance=256000000
> skip-to-first=false ! \
>                  audioconvert name=src2_audioconvert ! \
>                  audioresample name=src2_audioresample quality=4 ! \
>                  audio/x-raw,format=S16LE,channels=2,rate=48000 ! \
>                  tee name=src2_tee \
>                  \
>                  src2_tee. ! \
>                  volume name=src2_volume volume=0.5 ! \
>                  sink_audiomixer.
>                
>
> *My SRC1 audio transmitter (32000 - 50Hz):*
> gst-launch-1.0 audiotestsrc freq=100 !
> audio/x-raw,format=S16LE,channels=2,rate=31950 ! udpsink host="127.0.0.1"
> port=41001
>
> *My SRC2 audio transmitter (48000 + 50Hz):*
> gst-launch-1.0 audiotestsrc freq=300 !
> audio/x-raw,format=S16LE,channels=2,rate=48050 ! udpsink host="127.0.0.1"
> port=42001
>
> When I run these pipelines with even the latest gst version, then I hear
> "clicks" from time to time.
> I expect, that the audiomixer has some problems with the differ sampling
> frequencies.
>
> My assumption is, I need a asynchronous SRC for each input
> (src1_audioresample+src2_audioresample).
> How can I implement this with available elements. Or if not available, what
> is the best way to implement these?
>
> Why is the audiomixer located in the bad plugins? What means bad?
>
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Asynchrounous-Audio-Sample-Rate-Conversion-with-gstreamer-tp4677252p4677765.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Asynchrounous Audio Sample Rate Conversion with gstreamer

Maik Scholz
Hi,

My pipeline simplified:
SRC1: sine with 32kHz+50Hz=>
 NETWORK=>
  udpsrc(expected sample rate is 32kHz)=>
   queue=>
    audiorate=>
     audioconvert=>
      audioresample=>|
                            |audiomixer=>queue=>autoaudiosink(sync=true,slave-method=resample)=>Audio-Out
      audioresample=>|
     audioconvert=>
    audiorate=>
   queue=>
  udpsrc(expected sample rate is 48kHz)=>
 NETWORK=>
SRC2: sine with 48kHz-50Hz=>

>You can hook in your own drift compensation method if you set the
>audiobasesink's "slave-method" property to "custom". We use that in a
>project to finetune the audio clock's speed via a PLL by an integer
>number of ppm. It would be the same with an ASRC.

Sorry, I have some concerns about this proposal.
How should this compensate the clock drifts between (Audio-Out) and
two different inputs (SRC1 and SRC2)?

My expectation is, that I need something in the SRC pipeline parts.
This compensates each SRC clock to the main pipeline clock.
The ASRC in the basesink/autoaudiosink is responsible to synchronize the
main pipeline clock to the sink clock.

In the gstreamer audiomixer manual:
>Unlike the adder element audiomixer properly synchronises all input streams.
What does this mean in this context? Is there a ASRC inside the audiomixer?

An idea for would be adding a custom element in front of each "audioresample" element.
This "ASRC-SampleRate-Checker", could observe the SRCx buffer timestamps and
calculate the real input sample rate.
By knowing this, the element could adapt/change the rate in the output caps in a way,
that the following audioresample is compensating the source clock difference.
Could this work (inaudible)?

If there is any other solution for this, I am happy for all proposals.

Maik