GStreamer-devel

Audio mixing with ADDER results in broken sound

Classic

List

Threaded

3 messages Options

Karl Phillip

Audio mixing with ADDER results in broken sound

Hi all,

I'm using QtGStreamer-0.10 to play a mp3 file and I'm having difficulties finding out why the resulting sound is so horrible. I wrote the smallest application possible to reproduce the problem I'm facing.

I'm working on a "audio mixer" component that allows you to have multiple audio players instantiated and running. It's job is to mix all audio received by the players and chose to do this task myself because the soundcard I have is terrible and can't deal with 8, 9 audio players simultaneously. So I'm hoping to make it's job easier by sending it only one audio stream.

Let's jump into the code! The application below sets up 2 pipelines: the first named "_ms_pipeline" (as in master pipeline), and its responsible for mixing the audio using only 2 elements: an ADDER linked to AUTOAUDIOSINK.

The second pipeline, named only "pipeline", loads the file from the disk through FILESRC, decodes it, does some stuff using the following elements: APPSRC > VOLUME > AUDIOCONVERT > AUDIORESAMPLE > CAPSFILTER.

At the end I connect the src pad of CAPSFILTER to a sink pad of the ADDER, then set the state of the pipelines to QGst::StatePlaying to start playing the audio.

The result is awful.

#include <QtGui/QApplication>

#include <QList>
#include <QPointer>
#include <QObject>

#include <QGlib/Error>
#include <QGlib/Connect>
#include <QGst/Init>
#include <QGst/Bus>
#include <QGst/Pipeline>
#include <QGst/Parse>
#include <QGst/Message>
#include <QGst/Utils/ApplicationSink>
#include <QGst/Utils/ApplicationSource>
#include <QGst/ElementFactory>
#include <QGst/Element>
#include <QGst/Pad>
#include <QGst/Bin>

#include <iostream>

QGst::PipelinePtr _ms_pipeline;
QGst::ElementPtr _ms_adder;
QGst::ElementPtr _ms_output;

QGst::ElementPtr appsink;

class MySink : public QGst::Utils::ApplicationSink
{
public:
    MySink(QGst::Utils::ApplicationSource *src)
    : QGst::Utils::ApplicationSink(), m_src(src) {}

protected:
    virtual void eos()
    {
        m_src->endOfStream();
    }

    virtual QGst::FlowReturn newBuffer()
    {
        m_src->pushBuffer(pullBuffer());
        return QGst::FlowOk;
    }

private:
    QGst::Utils::ApplicationSource *m_src;
};

class Player : QObject
{
public:
    Player(const char* filename);
    ~Player() {};

    MySink m_sink;
    QGst::Utils::ApplicationSource m_src;

    void onNewDecodedPad(QGst::PadPtr pad)
    {
        std::cout << "Player::onNewDecodePad\n";
        QGst::CapsPtr caps = pad->caps();
        QGst::StructurePtr structure = caps->internalStructure(0);
        if (structure->name().contains("audio/x-raw"))
            std::cout << "Player::onNewDecodePad audio/x-raw\n";

        QGst::PadPtr saidaDecoder = appsink->getStaticPad("sink");
        if (pad->link(saidaDecoder) != QGst::PadLinkOk)
            std::cout << "Player::onNewDecodePad: Failed link !!!\n";

    }
};

int main(int argc, char *argv[])
{
    if (argc < 2)
    {
        std::cout << "Usage: qtplayer <file.mp3>\n";
        return -1;
    }

    QApplication app(argc, argv);

    Player player(argv[1]);

    return app.exec();
}

Player::Player(const char* filename)
: m_sink(&m_src)
{
    QGst::init(0, NULL);

   // Master pipeline setup

    _ms_pipeline = QGst::Pipeline::create("mypipe");
    if (!_ms_pipeline)
    {
        qDebug() << "MasterSink: Failed creating pipeline !!!";
        return;
    }
    _ms_pipeline->setState(QGst::StateNull);

    _ms_adder = QGst::ElementFactory::make("adder","audiomixer");
    _ms_pipeline->add(_ms_adder);

    _ms_output = QGst::ElementFactory::make("autoaudiosink","audio_out");
    _ms_pipeline->add(_ms_output);

    if (!_ms_adder->link(_ms_output))
    {
        qDebug() << "MasterSink: Failed linking to _output !!!";
        return;
    }

    _ms_pipeline->setState(QGst::StateReady);
    std::cout << "_ms_pipeline: StateReady\n";

    // This 2nd pipeline loads the file, decodes it and sends the audio stream to the master pipeline

    QGst::PipelinePtr pipeline = QGst::Pipeline::create();

    QGst::ElementPtr filesrc = QGst::ElementFactory::make("filesrc");
    filesrc->setProperty("location", filename);
    pipeline->add(filesrc);

    QGst::BinPtr decodebin = QGst::ElementFactory::make("decodebin2").dynamicCast<QGst::Bin>();
    pipeline->add(decodebin);
    if (!filesrc->link(decodebin))
    {
        qDebug() << "Failed: filesrc > decodebin !!!";
        return;
    }

    appsink = QGst::ElementFactory::make("appsink");
    pipeline->add(appsink);

    m_sink.setElement(appsink);

    QGst::ElementPtr appsrc = QGst::ElementFactory::make("appsrc");
    m_src.setElement(appsrc);
    m_src.setFormat(QGst::FormatDefault);

    _ms_pipeline->add(appsrc);

    QGst::ElementPtr vol = QGst::ElementFactory::make("volume");
    vol->setProperty("volume", 0.7);
    QGst::ElementPtr convert = QGst::ElementFactory::make("audioconvert");
    QGst::ElementPtr resample = QGst::ElementFactory::make("audioresample");

    QGst::ElementPtr filtercaps = QGst::ElementFactory::make("capsfilter");
    filtercaps->setProperty("caps", QGst::Caps::fromString("audio/x-raw-int, endianness=(int)1234, channels=(int)1, width=(int)16, depth=(int)16, signed=(boolean)true, rate=(int)11025"));

    _ms_pipeline->add(vol);
    _ms_pipeline->add(convert);
    _ms_pipeline->add(resample);
    _ms_pipeline->add(filtercaps);

    if (!appsrc->link(vol))
    {
        qDebug() << "Failed: appsrc > vol !!!";
        return;
    }

    if (!vol->link(convert))
    {
        qDebug() << "Failed: filesrc > decodebin !!!";
        return;
    }

    if (!convert->link(resample))
    {
        qDebug() << "Failed: filesrc > decodebin !!!";
        return;
    }

    if (!resample->link(filtercaps))
    {
        qDebug() << "Failed: filesrc > decodebin !!!";
        return;
    }

    QGst::PadPtr sinkpad = _ms_adder->getRequestPad("sink%d");
    if (!sinkpad)
    {
        qDebug() << "Failed: sinkpad !!!";
        return;
    }

    QGst::PadPtr saidaConverters = filtercaps->getStaticPad("src");
    if (saidaConverters->link(sinkpad) != QGst::PadLinkOk)
    {
        qDebug() << "Failed: saida > sinkpad !!!";
        return;
    }

    QGlib::connect(decodebin, "pad-added", this, &Player::onNewDecodedPad);

    pipeline->setState(QGst::StatePlaying);
    std::cout << "pipeline: StatePlaying\n";

    QGst::State state;
    pipeline->getState(&state, NULL, 0);
    std::cout << "pipeline: state is " << state << " while playing would be: " << QGst::StatePlaying << "\n";

    QGst::State ms_state;
    _ms_pipeline->getState(&ms_state, NULL, 0);
    if (ms_state != QGst::StatePlaying)
    {
        std::cout << "_ms_pipeline: StatePlaying *AGAIN* was: " << ms_state << "\n";
        _ms_pipeline->setState(QGst::StateNull);
        _ms_pipeline->setState(QGst::StatePlaying);
    }

    _ms_pipeline->getState(&ms_state, NULL, 0);
    std::cout << "_ms_pipeline: state is " << ms_state << "\n";
}

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Matthias Dodt

AW: Audio mixing with ADDER results in broken sound

Hi Karl!

You should always place an audioconvert + audioresample right before the audiosink, see:

http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good-plugins/html/gst-plugins-good-plugins-directsoundsink.html

I would remove the audioresample from the first pipeline. In the 2nd:

adder audioconvert audioresample autosoundsink

You can try your pipelines via gst-launch before implementing them right away-

cheers,

mat

Von: gstreamer-devel-bounces+mdodt=[hidden email] [mailto:gstreamer-devel-bounces+mdodt=[hidden email]] Im Auftrag von Karl Phillip
Gesendet: 23 November 2011 14:00
An: [hidden email]
Betreff: Audio mixing with ADDER results in broken sound

George Kiagiadakis

Re: Audio mixing with ADDER results in broken sound

In reply to this post by Karl Phillip

On Wed, Nov 23, 2011 at 2:59 PM, Karl Phillip <[hidden email]> wrote:

> Hi all,
>
> I'm using QtGStreamer-0.10 to play a mp3 file and I'm having difficulties
> finding out why the resulting sound is so horrible. I wrote the smallest
> application possible to reproduce the problem I'm facing.
>
> I'm working on a "audio mixer" component that allows you to have multiple
> audio players instantiated and running. It's job is to mix all audio
> received by the players and chose to do this task myself because the
> soundcard I have is terrible and can't deal with 8, 9 audio players
> simultaneously. So I'm hoping to make it's job easier by sending it only one
> audio stream.
>
> Let's jump into the code! The application below sets up 2 pipelines: the
> first named "_ms_pipeline" (as in master pipeline), and its responsible for
> mixing the audio using only 2 elements: an ADDER linked to AUTOAUDIOSINK.
>
> The second pipeline, named only "pipeline", loads the file from the disk
> through FILESRC, decodes it, does some stuff using the following elements:
> APPSRC > VOLUME > AUDIOCONVERT > AUDIORESAMPLE > CAPSFILTER.
>
> At the end I connect the src pad of CAPSFILTER to a sink pad of the ADDER,
> then set the state of the pipelines to QGst::StatePlaying to start playing
> the audio.
>

Why are you doing this with two pipelines instead of one? The problem
is likely a synchronization problem between the two pipelines. Try
linking directly the pads in the first pipeline without using
appsink/appsrc.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel