GStreamer-devel

Gstreamer with multithread app (Qt)

Classic

List

Threaded

16 messages Options

Matias Hernandez Arellano

Gstreamer with multithread app (Qt)

It's possible to use gstreamer in a multithread app?

I have an Producer/Consumer application where i have two groups 1 with 1 Producer and 1 Cosumer (Capture Video with Opencv/Process Image) and other group equal ... Take de Process Frame / Stream the frames...

The communication between threads works good, but know i need to add the stream part ..
I have another app where i test the functionality of AppSrc and works!!! ..
So i need to add this (Read from the Buffer where i have images, push into AppSrc and Stream over network) in a thread.

is this possible??

I was working with Qt and i have qt-gstreamer installed and working..

Any example of how can i do that? (with Qt-gstreamer or simple gst ?? )

Thanks in advance.

--
Matías Hernandez Arellano
Ingeniero de Software/Proyectos en VisionLabs S.A
CDA Archlinux-CL
www.msdark.archlinux.cl

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Tim-Philipp Müller-2

Re: Gstreamer with multithread app (Qt)

On Thu, 2011-04-07 at 18:20 -0400, Matias Hernandez Arellano wrote:

> It's possible to use gstreamer in a multithread app?

Yes.

> I have an Producer/Consumer application where i have two groups 1 with
> 1 Producer and 1 Cosumer (Capture Video with Opencv/Process Image) and
> other group equal ... Take de Process Frame / Stream the frames...

Have you considered writing a simple source element that captures and
processes the frame directly in a GStreamer pipeline?

> The communication between threads works good, but know i need to add
> the stream part .. I have another app where i test the functionality of
> AppSrc and works!!! .. So i need to add this (Read from the Buffer
> where i have images, push into AppSrc and Stream over network) in a
> thread.
>
> is this possible??

Why would this not be possible? Have you tried it and run into problems?

GStreamer does (almost) all of its processing in dedicated threads of
its own ("streaming threads"). You should be able to push buffers into
appsrc from any application thread (they will get queued internally in
appsrc and then processed in the GStreamer streaming thread).

Cheers
-Tim

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Matias Hernandez Arellano

Re: Gstreamer with multithread app (Qt)

> Have you considered writing a simple source element that captures and
> processes the frame directly in a GStreamer pipeline?

I can't do that... the processing process it's very complex and it's different for every ambient where the application works, but i can try...

> Why would this not be possible? Have you tried it and run into problems?
>
> GStreamer does (almost) all of its processing in dedicated threads of
> its own ("streaming threads"). You should be able to push buffers into
> appsrc from any application thread (they will get queued internally in
> appsrc and then processed in the GStreamer streaming thread).

Well i try... in the StreamTread .. i read data from the Buffer that i create (this buffer contains processing's frames)...
I use QGst for this.

I use the QGst::Init in the StreamThread constructor ... and configure the pipeline ...
But when i run the app .. i get this:
QObject::startTimer: QTimer can only be used with threads started with QThread

and nothing happen (the app don't crash but neither capture/process/stream) .. so i think maybe i can't use the init method of the gsreamer in the thread maybe i need to do in the main thread... but this doesn't work ...

So if there is any example of how to do that or guideline i would very appreciate (using QGst, the C or C++ interface of Gstreamer)...

Thanks a lot

El 08-04-2011, a las 4:34, Tim-Philipp Müller escribió:

> On Thu, 2011-04-07 at 18:20 -0400, Matias Hernandez Arellano wrote:
>
>> It's possible to use gstreamer in a multithread app?
>
> Yes.
>
>> I have an Producer/Consumer application where i have two groups 1 with
>> 1 Producer and 1 Cosumer (Capture Video with Opencv/Process Image) and
>> other group equal ... Take de Process Frame / Stream the frames...
>
> Have you considered writing a simple source element that captures and
> processes the frame directly in a GStreamer pipeline?
>
>> The communication between threads works good, but know i need to add
>> the stream part .. I have another app where i test the functionality of
>> AppSrc and works!!! .. So i need to add this (Read from the Buffer
>> where i have images, push into AppSrc and Stream over network) in a
>> thread.
>>
>> is this possible??
>
> Why would this not be possible? Have you tried it and run into problems?
>
> GStreamer does (almost) all of its processing in dedicated threads of
> its own ("streaming threads"). You should be able to push buffers into
> appsrc from any application thread (they will get queued internally in
> appsrc and then processed in the GStreamer streaming thread).
>
> Cheers
> -Tim
>
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Matías Hernandez Arellano
Ingeniero de Software/Proyectos en VisionLabs S.A
CDA Archlinux-CL
www.msdark.archlinux.cl

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Matias Hernandez Arellano

Re: Gstreamer with multithread app (Qt)

In reply to this post by Tim-Philipp Müller-2

(me again)...
Well i try it a lot of things and finally i get a proof of concept of the final application semi functional using Qt+OpenCV+QGstreamer ...
but.. yet.. i have a problem..

Here some snippets of code

main.cpp
int main(int argc, char *argv[]){
QCoreApplication a(argc, argv);
QGst::init(&argc,&argv);
FrameBuffer *buffer = new FrameBuffer(3);
FrameBuffer *buffer2 = new FrameBuffer(3);
CaptureThread *capture = new CaptureThread(buffer,0);
ProcessingThread *processing = new ProcessingThread(buffer,buffer2,capture->getSourceWidth(),capture->getSourceHeight());
StreamThread *stream = new StreamThread(buffer2);
QCoreApplication::connect(processing,SIGNAL(newframe()),stream,SLOT(update()),Qt::QueuedConnection);
capture->start();
processing->start();
stream->start();
return a.exec();;
}

buffer and buffer2 are Circular Buffers that hold cv::Mat (OpenCV Images).
buffer hold cv::Mat obtained with CaptureThread. ProcessingThread pull images from here.
ProcessingThread push images into buffer2 and StreamThread pull images from here.

The cycle works good...
now the StreamThread

StreamThread::StreamThread(FrameBuffer *b, QObject *parent): ImageBuffer(b),QThread(parent){
QString pipe_str = QString("appsrc name=\"appsrc\" caps=\"video/x-raw-rgb,width=320,height=240\" ! queue ! videoparse format=14 width=%1 height=%1 ! videorate "
" ! videoscale ! video/x-raw-rgb,width=320,height=240 "
" ! queue ! ffmpegcolorspace ! jpegenc ! queue "
" ! tcpserversink port=5000 ").arg(WIDTH,HEIGHT);
this->pipeline = QGst::Parse::launch(pipe_str).dynamicCast<QGst::Pipeline>();
QGlib::connect(this->pipeline->bus(),"message",this,&StreamThread::onBusMessage);
this->pipeline->bus()->addSignalWatch();
this->pipeline->setState(QGst::StatePlaying);
this->m_src.setElement(this->pipeline->getElementByName("appsrc"));
}
void StreamThread::onBusMessage(const QGst::MessagePtr &message){
switch (message->type()) {
case QGst::MessageEos:
quit();
break;
case QGst::MessageError:
qCritical() << message.staticCast<QGst::ErrorMessage>()->error();
break;
default:
break;
}
}
void StreamThread::run(){
this->exec();
}

//This SLOT fires up when ProcessingThread calls the newImage() SIGNAL ..
// this signal ocurrs when a new processing image was push into the buffer
void StreamThread::update(){
if(!this->m_src.element().isNull()){
this->pipeline->setState(QGst::StatePlaying);
QGst::BufferPtr buffer = QGst::Buffer::create(WIDTH*HEIGHT*3*sizeof(quint8*));
quint8 *data = buffer->data();
Mat frame = this->ImageBuffer->getFrame();
data = (quint8*)frame.data;
QGst::FlowReturn ret = this->m_src.pushBuffer(buffer);
if(ret!=QGst::FlowOk){
qDebug()<<"Error pushing data";
}
}
}
StreamThread::~StreamThread(){
this->pipeline->setState(QGst::StateNull);
}

When i run this app i don't get errors, just this warning (this doesn't show if i don't execute StreamThread)
QObject::startTimer: QTimer can only be used with threads started with QThread
So i suppose that QGstreamer use QTimer in some hidden part.

And... if i try to see the stream using
gst-launch tcpclientsrc host=127.0.0.1 port=5000 ! jpegdec ! queue ! video/x-raw-rgb,width=320,height=240 ! ffmpegcolorspace ! queue ! glimagesink sync=false

I get:
Estableciendo el conducto a PAUSA ?
El conducto est? PREPAR?NDOSE ?

But the pipeline never goes to PLAYING and nothing happen (i can't see the result)...

Any idea??

Thanks in advance

El 08-04-2011, a las 4:34, Tim-Philipp Müller escribió:

Michael Joachimiak

Re: Gstreamer with multithread app (Qt)

Does your StreamThread inherits from QThread? If not you have the answer.

I think generally you have a problem here because you are using singal/slot mechanism which was designed for GUI not for normal (fast) operation. I could not find it in the documentation (somebody please correct me if wrong) but my guess is that signal/slot mechanism is using qtimer.
Excerpt from the documentation: "In general, emitting a signal that is connected to some slots, is approximately ten times slower than calling the receivers directly, with non-virtual function calls." From here http://doc.qt.nokia.com/latest/signalsandslots.html.

So if you want some real-time processing to happen and you have a producer-consumer scenario I would suggest to use semaphores (QSemaphore ) for that.

Otherways (if qtimer is involved in multithreading) you will get performance issues sooner or later.
Checked on my own skin.

If not I guess it should

2011/4/13 Matias Hernandez Arellano <[hidden email]>

(me again)...
Well i try it a lot of things and finally i get a proof of concept of the final application semi functional using Qt+OpenCV+QGstreamer ...
but.. yet.. i have a problem..

Here some snippets of code

main.cpp
int main(int argc, char *argv[]){
QCoreApplication a(argc, argv);
QGst::init(&argc,&argv);
FrameBuffer *buffer = new FrameBuffer(3);
FrameBuffer *buffer2 = new FrameBuffer(3);
CaptureThread *capture = new CaptureThread(buffer,0);
ProcessingThread *processing = new ProcessingThread(buffer,buffer2,capture->getSourceWidth(),capture->getSourceHeight());
StreamThread *stream = new StreamThread(buffer2);
QCoreApplication::connect(processing,SIGNAL(newframe()),stream,SLOT(update()),Qt::QueuedConnection);
capture->start();
processing->start();
stream->start();
return a.exec();;
}

buffer and buffer2 are Circular Buffers that hold cv::Mat (OpenCV Images).
buffer hold cv::Mat obtained with CaptureThread. ProcessingThread pull images from here.
ProcessingThread push images into buffer2 and StreamThread pull images from here.

The cycle works good...
now the StreamThread

StreamThread::StreamThread(FrameBuffer *b, QObject *parent): ImageBuffer(b),QThread(parent){
QString pipe_str = QString("appsrc name=\"appsrc\" caps=\"video/x-raw-rgb,width=320,height=240\" ! queue ! videoparse format=14 width=%1 height=%1 ! videorate "
" ! videoscale ! video/x-raw-rgb,width=320,height=240 "
" ! queue ! ffmpegcolorspace ! jpegenc ! queue "
" ! tcpserversink port=5000 ").arg(WIDTH,HEIGHT);
this->pipeline = QGst::Parse::launch(pipe_str).dynamicCast<QGst::Pipeline>();
QGlib::connect(this->pipeline->bus(),"message",this,&StreamThread::onBusMessage);
this->pipeline->bus()->addSignalWatch();
this->pipeline->setState(QGst::StatePlaying);
this->m_src.setElement(this->pipeline->getElementByName("appsrc"));
}
void StreamThread::onBusMessage(const QGst::MessagePtr &message){
switch (message->type()) {
case QGst::MessageEos:
quit();
break;
case QGst::MessageError:
qCritical() << message.staticCast<QGst::ErrorMessage>()->error();
break;
default:
break;
}
}
void StreamThread::run(){
this->exec();
}

//This SLOT fires up when ProcessingThread calls the newImage() SIGNAL ..
// this signal ocurrs when a new processing image was push into the buffer
void StreamThread::update(){
if(!this->m_src.element().isNull()){
this->pipeline->setState(QGst::StatePlaying);
QGst::BufferPtr buffer = QGst::Buffer::create(WIDTH*HEIGHT*3*sizeof(quint8*));
quint8 *data = buffer->data();
Mat frame = this->ImageBuffer->getFrame();
data = (quint8*)frame.data;
QGst::FlowReturn ret = this->m_src.pushBuffer(buffer);
if(ret!=QGst::FlowOk){
qDebug()<<"Error pushing data";
}
}
}
StreamThread::~StreamThread(){
this->pipeline->setState(QGst::StateNull);
}

When i run this app i don't get errors, just this warning (this doesn't show if i don't execute StreamThread)

QObject::startTimer: QTimer can only be used with threads started with QThread

So i suppose that QGstreamer use QTimer in some hidden part.

And... if i try to see the stream using
gst-launch tcpclientsrc host=127.0.0.1 port=5000 ! jpegdec ! queue ! video/x-raw-rgb,width=320,height=240 ! ffmpegcolorspace ! queue ! glimagesink sync=false

I get:
Estableciendo el conducto a PAUSA ?
El conducto est? PREPAR?NDOSE ?

But the pipeline never goes to PLAYING and nothing happen (i can't see the result)...

Any idea??

Thanks in advance

El 08-04-2011, a las 4:34, Tim-Philipp Müller escribió:

> On Thu, 2011-04-07 at 18:20 -0400, Matias Hernandez Arellano wrote:
>
>> It's possible to use gstreamer in a multithread app?
>
> Yes.
>
>> I have an Producer/Consumer application where i have two groups 1 with
>> 1 Producer and 1 Cosumer (Capture Video with Opencv/Process Image) and
>> other group equal ... Take de Process Frame / Stream the frames...
>
> Have you considered writing a simple source element that captures and
> processes the frame directly in a GStreamer pipeline?
>
>> The communication between threads works good, but know i need to add
>> the stream part .. I have another app where i test the functionality of
>> AppSrc and works!!! .. So i need to add this (Read from the Buffer
>> where i have images, push into AppSrc and Stream over network) in a
>> thread.
>>
>> is this possible??
>
> Why would this not be possible? Have you tried it and run into problems?
>
> GStreamer does (almost) all of its processing in dedicated threads of
> its own ("streaming threads"). You should be able to push buffers into
> appsrc from any application thread (they will get queued internally in
> appsrc and then processed in the GStreamer streaming thread).
>
> Cheers
> -Tim
>
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Matías Hernandez Arellano
Ingeniero de Software/Proyectos en VisionLabs S.A
CDA Archlinux-CL
www.msdark.archlinux.cl

_______________________________________________

gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
Your Sincerely
Michal Joachimiak

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Matias Hernandez Arellano

Re: Gstreamer with multithread app (Qt)

Well.. i will try!
El 12-04-2011, a las 17:03, Michael Joachimiak escribió:

> Does your StreamThread inherits from QThread? If not you have the answer.
>
> I think generally you have a problem here because you are using singal/slot
> mechanism which was designed for GUI not for normal (fast) operation. I
> could not find it in the documentation (somebody please correct me if wrong)
> but my guess is that signal/slot mechanism is using qtimer.
> Excerpt from the documentation: "In general, emitting a signal that is
> connected to some slots, is approximately ten times slower than calling the
> receivers directly, with non-virtual function calls." From here
> http://doc.qt.nokia.com/latest/signalsandslots.html.
>
> So if you want some real-time processing to happen and you have a
> producer-consumer scenario I would suggest to use semaphores (QSemaphore )
> for that.
>
> Otherways (if qtimer is involved in multithreading) you will get performance
> issues sooner or later.
> Checked on my own skin.
>
>
>
> If not I guess it should
>
> 2011/4/13 Matias Hernandez Arellano <[hidden email]>
>
>> (me again)...
>> Well i try it a lot of things and finally i get a proof of concept of the
>> final application semi functional using Qt+OpenCV+QGstreamer ...
>> but.. yet.. i have a problem..
>>
>> Here some snippets of code
>>
>> main.cpp
>> int main(int argc, char *argv[]){
>> QCoreApplication a(argc, argv);
>> QGst::init(&argc,&argv);
>> FrameBuffer *buffer = new FrameBuffer(3);
>> FrameBuffer *buffer2 = new FrameBuffer(3);
>> CaptureThread *capture = new CaptureThread(buffer,0);
>> ProcessingThread *processing = new
>> ProcessingThread(buffer,buffer2,capture->getSourceWidth(),capture->getSourceHeight());
>> StreamThread *stream = new StreamThread(buffer2);
>>
>> QCoreApplication::connect(processing,SIGNAL(newframe()),stream,SLOT(update()),Qt::QueuedConnection);
>> capture->start();
>> processing->start();
>> stream->start();
>> return a.exec();;
>> }
>>
>> buffer and buffer2 are Circular Buffers that hold cv::Mat (OpenCV Images).
>> buffer hold cv::Mat obtained with CaptureThread. ProcessingThread pull
>> images from here.
>> ProcessingThread push images into buffer2 and StreamThread pull images from
>> here.
>>
>> The cycle works good...
>> now the StreamThread
>>
>> StreamThread::StreamThread(FrameBuffer *b, QObject *parent):
>> ImageBuffer(b),QThread(parent){
>> QString pipe_str = QString("appsrc name=\"appsrc\"
>> caps=\"video/x-raw-rgb,width=320,height=240\" ! queue ! videoparse format=14
>> width=%1 height=%1 ! videorate "
>> " ! videoscale !
>> video/x-raw-rgb,width=320,height=240 "
>> " ! queue ! ffmpegcolorspace ! jpegenc !
>> queue "
>> " ! tcpserversink port=5000
>> ").arg(WIDTH,HEIGHT);
>> this->pipeline =
>> QGst::Parse::launch(pipe_str).dynamicCast<QGst::Pipeline>();
>>
>> QGlib::connect(this->pipeline->bus(),"message",this,&StreamThread::onBusMessage);
>> this->pipeline->bus()->addSignalWatch();
>> this->pipeline->setState(QGst::StatePlaying);
>> this->m_src.setElement(this->pipeline->getElementByName("appsrc"));
>> }
>> void StreamThread::onBusMessage(const QGst::MessagePtr &message){
>> switch (message->type()) {
>> case QGst::MessageEos:
>> quit();
>> break;
>> case QGst::MessageError:
>> qCritical() << message.staticCast<QGst::ErrorMessage>()->error();
>> break;
>> default:
>> break;
>> }
>> }
>> void StreamThread::run(){
>> this->exec();
>> }
>>
>> //This SLOT fires up when ProcessingThread calls the newImage() SIGNAL ..
>> // this signal ocurrs when a new processing image was push into the buffer
>> void StreamThread::update(){
>> if(!this->m_src.element().isNull()){
>> this->pipeline->setState(QGst::StatePlaying);
>> QGst::BufferPtr buffer =
>> QGst::Buffer::create(WIDTH*HEIGHT*3*sizeof(quint8*));
>> quint8 *data = buffer->data();
>> Mat frame = this->ImageBuffer->getFrame();
>> data = (quint8*)frame.data;
>> QGst::FlowReturn ret = this->m_src.pushBuffer(buffer);
>> if(ret!=QGst::FlowOk){
>> qDebug()<<"Error pushing data";
>> }
>> }
>> }
>> StreamThread::~StreamThread(){
>> this->pipeline->setState(QGst::StateNull);
>> }
>>
>> When i run this app i don't get errors, just this warning (this doesn't
>> show if i don't execute StreamThread)
>> QObject::startTimer: QTimer can only be used with threads started with
>> QThread
>> So i suppose that QGstreamer use QTimer in some hidden part.
>>
>> And... if i try to see the stream using
>> gst-launch tcpclientsrc host=127.0.0.1 port=5000 ! jpegdec ! queue !
>> video/x-raw-rgb,width=320,height=240 ! ffmpegcolorspace ! queue !
>> glimagesink sync=false
>>
>> I get:
>> Estableciendo el conducto a PAUSA ?
>> El conducto est? PREPAR?NDOSE ?
>>
>> But the pipeline never goes to PLAYING and nothing happen (i can't see the
>> result)...
>>
>> Any idea??
>>
>> Thanks in advance
>>
>>
>> El 08-04-2011, a las 4:34, Tim-Philipp Müller escribió:
>>
>>> On Thu, 2011-04-07 at 18:20 -0400, Matias Hernandez Arellano wrote:
>>>
>>>> It's possible to use gstreamer in a multithread app?
>>>
>>> Yes.
>>>
>>>> I have an Producer/Consumer application where i have two groups 1 with
>>>> 1 Producer and 1 Cosumer (Capture Video with Opencv/Process Image) and
>>>> other group equal ... Take de Process Frame / Stream the frames...
>>>
>>> Have you considered writing a simple source element that captures and
>>> processes the frame directly in a GStreamer pipeline?
>>>
>>>> The communication between threads works good, but know i need to add
>>>> the stream part .. I have another app where i test the functionality of
>>>> AppSrc and works!!! .. So i need to add this (Read from the Buffer
>>>> where i have images, push into AppSrc and Stream over network) in a
>>>> thread.
>>>>
>>>> is this possible??
>>>
>>> Why would this not be possible? Have you tried it and run into problems?
>>>
>>> GStreamer does (almost) all of its processing in dedicated threads of
>>> its own ("streaming threads"). You should be able to push buffers into
>>> appsrc from any application thread (they will get queued internally in
>>> appsrc and then processed in the GStreamer streaming thread).
>>>
>>> Cheers
>>> -Tim
>>>
>>>
>>> _______________________________________________
>>> gstreamer-devel mailing list
>>> [hidden email]
>>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
>> Matías Hernandez Arellano
>> Ingeniero de Software/Proyectos en VisionLabs S.A
>> CDA Archlinux-CL
>> www.msdark.archlinux.cl
>>
>>
>>
>>
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
>
>
>
> --
> Your Sincerely
> Michal Joachimiak
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Matias Hernandez Arellano

Re: Gstreamer with multithread app (Qt)

In reply to this post by Michael Joachimiak

But... i'm using QWaitConditions to the Circula Buffer to solve the Consumer/Producer problem...

And i have only one SIGNAL/SLOT .. the idea to use this is i need a method to make the push into appsrc ..

When i was trying with C interface of gstreamer i use callbacks (needData, enoughData, etc) with appsrc so, when i put the pipeline in Playing State, appsrc call the needData function and push buffers into it ..

But with QGst i don't find a method to mimic this, so i think in using this SIGNAL/SLOT game to tell to the StreamThread there is a image to push into appsrc ...

i will try with other thougs...
El 12-04-2011, a las 17:03, Michael Joachimiak escribió:

Matias Hernandez Arellano

Re: Gstreamer with multithread app (Qt)

In reply to this post by Michael Joachimiak

(Sorry i write a lot of emails but i have a lot of ideas/doubts...)

I remove the SIGNAL/SLOT so in the StreamThread i use this:

void StreamThread::run(){
for(;;){
Mat frame = this->ImageBuffer->getFrame();
if(!this->m_src.element().isNull() && !frame.empty()){
this->pipeline->setState(QGst::StatePlaying);
QGst::BufferPtr buffer = QGst::Buffer::create(WIDTH*HEIGHT*3*sizeof(quint8*));
quint8 *data = buffer->data();
data = (quint8*)frame.data;
QGst::FlowReturn ret = this->m_src.pushBuffer(buffer);
switch(ret){
case QGst::FlowNotNegotiated:
qDebug()<<"Not Negotiated";
break;
case QGst::FlowNotSupported:
qDebug()<<"Not Supported";
break;
case QGst::FlowUnexpected:
qDebug()<<"Unexpected";
break;
case QGst::FlowWrongState:
qDebug()<<"Wrong State";
break;
}
}
}
}

the ImageBuffer->getFrame() pull data from the ImageBuffer .. this is a implementation of a Circular Buffer with WaitConditions to ensure that the buffer isn't empty when the pull operation occurs.

So, i check if the appsrc.element() is valid and if the frame obtained is valid and then push into Appsrc using pushBuffer method and check the return value of this operation.. and is QGst::FlowOk .. but in the other side when i try to receive the stream

gst-launch tcpclientsrc host=127.0.0.1 port=5000 ! jpegdec ! queue ! video/x-raw-rgb,width=320,height=240 ! ffmpegcolorspace ! queue ! glimagesink sync=false

i gen nothing .. the pipeline don't put into Playing State.

So the problem (maybe) is not the QTimer problem .. i don't use this and neither SIGNALS and SLOTS

Any idea or guideline will be very appreciate

Thanks in advance..

El 12-04-2011, a las 17:03, Michael Joachimiak escribió:

George Kiagiadakis

Re: Gstreamer with multithread app (Qt)

In reply to this post by Matias Hernandez Arellano

Hello,

On Wed, Apr 13, 2011 at 12:34 AM, Matias Hernandez Arellano
<[hidden email]> wrote:

> (me again)...
> Well i try it a lot of things and finally i get a proof of concept of the final application semi functional using Qt+OpenCV+QGstreamer ...
> but.. yet.. i have a problem..
>
> Here some snippets of code
>
> main.cpp
> int main(int argc, char *argv[]){
> QCoreApplication a(argc, argv);
> QGst::init(&argc,&argv);
> FrameBuffer *buffer = new FrameBuffer(3);
> FrameBuffer *buffer2 = new FrameBuffer(3);
> CaptureThread *capture = new CaptureThread(buffer,0);
> ProcessingThread *processing = new ProcessingThread(buffer,buffer2,capture->getSourceWidth(),capture->getSourceHeight());
> StreamThread *stream = new StreamThread(buffer2);
> QCoreApplication::connect(processing,SIGNAL(newframe()),stream,SLOT(update()),Qt::QueuedConnection);
> capture->start();
> processing->start();
> stream->start();
> return a.exec();;
> }
>
> buffer and buffer2 are Circular Buffers that hold cv::Mat (OpenCV Images).
> buffer hold cv::Mat obtained with CaptureThread. ProcessingThread pull images from here.
> ProcessingThread push images into buffer2 and StreamThread pull images from here.
>
> The cycle works good...
> now the StreamThread
>
> StreamThread::StreamThread(FrameBuffer *b, QObject *parent): ImageBuffer(b),QThread(parent){
> QString pipe_str = QString("appsrc name=\"appsrc\" caps=\"video/x-raw-rgb,width=320,height=240\" ! queue ! videoparse format=14 width=%1 height=%1 ! videorate "
> " ! videoscale ! video/x-raw-rgb,width=320,height=240 "
> " ! queue ! ffmpegcolorspace ! jpegenc ! queue "
> " ! tcpserversink port=5000 ").arg(WIDTH,HEIGHT);

Why on earth are you using so many threads here. Each queue starts a
new thread; additionally, appsrc starts a new thread too and maintains
an internal queue (making the second queue right after it completely
useless).

> this->pipeline = QGst::Parse::launch(pipe_str).dynamicCast<QGst::Pipeline>();
> QGlib::connect(this->pipeline->bus(),"message",this,&StreamThread::onBusMessage);
> this->pipeline->bus()->addSignalWatch();

This is where the QTimer starts, inside the signal watch. You can see
the code here [1] if you want. This timer expires every once in a
while and polls the GstBus for new messages. It's purpose is to
deliver messages to the main thread. I'm not sure why it shows that
error message though on the output... I'll check it, it doesn't seem
normal.

> this->pipeline->setState(QGst::StatePlaying);
> this->m_src.setElement(this->pipeline->getElementByName("appsrc"));
> }
> void StreamThread::onBusMessage(const QGst::MessagePtr &message){
> switch (message->type()) {
> case QGst::MessageEos:
> quit();
> break;
> case QGst::MessageError:
> qCritical() << message.staticCast<QGst::ErrorMessage>()->error();
> break;
> default:
> break;
> }
> }
> void StreamThread::run(){
> this->exec();
> }

I wonder if you understand that this event loop here is completely
useless. The update() slot will NEVER be called in this thread,
because this QThread object, being a QObject itself, lives in the main
thread and all of its slots are called in the context of the main
thread. So, the whole "StreamThread" could actually be a QObject
instead of a QThread and still do the same job. Let me add that since
appsrc itself handles queuing, you don't really need this extra thread
here.

> //This SLOT fires up when ProcessingThread calls the newImage() SIGNAL ..
> // this signal ocurrs when a new processing image was push into the buffer
> void StreamThread::update(){
> if(!this->m_src.element().isNull()){
> this->pipeline->setState(QGst::StatePlaying);

Why are you setting the pipeline *again* to playing state?

> QGst::BufferPtr buffer = QGst::Buffer::create(WIDTH*HEIGHT*3*sizeof(quint8*));
> quint8 *data = buffer->data();
> Mat frame = this->ImageBuffer->getFrame();
> data = (quint8*)frame.data;

I hope you realize that this statement is void, it copies no actual
data. You need to copy the data with memcpy() instead.

> QGst::FlowReturn ret = this->m_src.pushBuffer(buffer);

...and effectively here you are pushing an empty buffer. No wonder why
the other end doesn't receive anything.

> if(ret!=QGst::FlowOk){
> qDebug()<<"Error pushing data";
> }
> }
> }
> StreamThread::~StreamThread(){
> this->pipeline->setState(QGst::StateNull);
> }
>
> When i run this app i don't get errors, just this warning (this doesn't show if i don't execute StreamThread)
> QObject::startTimer: QTimer can only be used with threads started with QThread
> So i suppose that QGstreamer use QTimer in some hidden part.
>
> And... if i try to see the stream using
> gst-launch tcpclientsrc host=127.0.0.1 port=5000 ! jpegdec ! queue ! video/x-raw-rgb,width=320,height=240 ! ffmpegcolorspace ! queue ! glimagesink sync=false
>
> I get:
> Estableciendo el conducto a PAUSA ?
> El conducto est? PREPAR?NDOSE ?
>
> But the pipeline never goes to PLAYING and nothing happen (i can't see the result)...
>
> Any idea??
>
> Thanks in advance

Regards,
George

[1]. http://gstreamer.freedesktop.org/data/doc/gstreamer/head/qt-gstreamer/html/bus_8cpp_source.html
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Matias Hernandez Arellano

Re: Gstreamer with multithread app (Qt)

Why on earth are you using so many threads here. Each queue starts a
new thread; additionally, appsrc starts a new thread too and maintains
an internal queue (making the second queue right after it completely
useless).

The true.. i find/read some example and this work.... but i ever think about this, but "if work don't touch it" ... xD

   this->pipeline = QGst::Parse::launch(pipe_str).dynamicCast<QGst::Pipeline>();
   QGlib::connect(this->pipeline->bus(),"message",this,&StreamThread::onBusMessage);
   this->pipeline->bus()->addSignalWatch();

This is where the QTimer starts, inside the signal watch. You can see
the code here [1] if you want. This timer expires every once in a
while and polls the GstBus for new messages. It's purpose is to
deliver messages to the main thread. I'm not sure why it shows that
error message though on the output... I'll check it, it doesn't seem
normal.

So if i remove this (the connect and addSignalWatch) i don't use QTimer, but i can't get errors messages from the pipeline.

void StreamThread::run(){
this->exec();
}

I modify this to run a infinite loop where i push data into appsrc.

Why are you setting the pipeline *again* to playing state?

A mistake ... this was for testing purpose ...

   Mat frame = this->ImageBuffer->getFrame();
   data = (quint8*)frame.data;

I hope you realize that this statement is void, it copies no actual
data. You need to copy the data with memcpy() instead.

   QGst::FlowReturn ret = this->m_src.pushBuffer(buffer);

I do this cause i asking in other thread about the use of QGst::Buffer and i get this:

The QGst::Buffer class has a data() method which gives you a pointer
to the allocated buffer. You can use this pointer to write on the
buffer.

Btw, the above line of code looks terribly wrong to me... Here is how
you should use it:

QGst::BufferPtr b = QGst::Buffer::create(size);
quint8 *data = b->data();
//use the data pointer to write.

But i don't know if this is right.. so i try with memcpy. When i testing with C interface of Gstreamer i do this:

memcpy(GST_BUFFER_DATA(buffer),img_data,data_size);

even i use this GST_BUFFER_DATA(buffer) = img_data; GST_BUFFER_SIZE(buffer) = 640*480*3*sizeof(uchar*);

but now i don't know how copy the data into QGst::Buffer... this don't work ...

memcpy(data,frame.data,640*480*3*sizeof(uchar*))

Sorry if my questions are too silly but i'm a newbie with this..

Thanks a lot!!

El 12-04-2011, a las 17:26, George Kiagiadakis escribió:

Hello,

On Wed, Apr 13, 2011 at 12:34 AM, Matias Hernandez Arellano
<[hidden email]> wrote:
(me again)...
Well i try it a lot of things and finally i get a proof of concept of the final application semi functional using Qt+OpenCV+QGstreamer ...
but.. yet.. i have a problem..

Here some snippets of code

main.cpp
int main(int argc, char *argv[]){
QCoreApplication a(argc, argv);
QGst::init(&argc,&argv);
FrameBuffer *buffer = new FrameBuffer(3);
FrameBuffer *buffer2 = new FrameBuffer(3);
CaptureThread *capture = new CaptureThread(buffer,0);
ProcessingThread *processing = new ProcessingThread(buffer,buffer2,capture->getSourceWidth(),capture->getSourceHeight());
StreamThread *stream = new StreamThread(buffer2);
QCoreApplication::connect(processing,SIGNAL(newframe()),stream,SLOT(update()),Qt::QueuedConnection);
capture->start();
processing->start();
stream->start();
return a.exec();;
}

buffer and buffer2 are Circular Buffers that hold cv::Mat (OpenCV Images).
buffer hold cv::Mat obtained with CaptureThread. ProcessingThread pull images from here.
ProcessingThread push images into buffer2 and StreamThread pull images from here.

The cycle works good...
now the StreamThread

StreamThread::StreamThread(FrameBuffer *b, QObject *parent): ImageBuffer(b),QThread(parent){
QString pipe_str = QString("appsrc name=\"appsrc\" caps=\"video/x-raw-rgb,width=320,height=240\" ! queue ! videoparse format=14 width=%1 height=%1 ! videorate "
" ! videoscale ! video/x-raw-rgb,width=320,height=240 "
" ! queue ! ffmpegcolorspace ! jpegenc ! queue "
" ! tcpserversink port=5000 ").arg(WIDTH,HEIGHT);

Why on earth are you using so many threads here. Each queue starts a
new thread; additionally, appsrc starts a new thread too and maintains
an internal queue (making the second queue right after it completely
useless).

this->pipeline = QGst::Parse::launch(pipe_str).dynamicCast<QGst::Pipeline>();
QGlib::connect(this->pipeline->bus(),"message",this,&StreamThread::onBusMessage);
this->pipeline->bus()->addSignalWatch();

This is where the QTimer starts, inside the signal watch. You can see
the code here [1] if you want. This timer expires every once in a
while and polls the GstBus for new messages. It's purpose is to
deliver messages to the main thread. I'm not sure why it shows that
error message though on the output... I'll check it, it doesn't seem
normal.

this->pipeline->setState(QGst::StatePlaying);
this->m_src.setElement(this->pipeline->getElementByName("appsrc"));
}
void StreamThread::onBusMessage(const QGst::MessagePtr &message){
switch (message->type()) {
case QGst::MessageEos:
quit();
break;
case QGst::MessageError:
qCritical() << message.staticCast<QGst::ErrorMessage>()->error();
break;
default:
break;
}
}
void StreamThread::run(){
this->exec();
}

I wonder if you understand that this event loop here is completely
useless. The update() slot will NEVER be called in this thread,
because this QThread object, being a QObject itself, lives in the main
thread and all of its slots are called in the context of the main
thread. So, the whole "StreamThread" could actually be a QObject
instead of a QThread and still do the same job. Let me add that since
appsrc itself handles queuing, you don't really need this extra thread
here.

//This SLOT fires up when ProcessingThread calls the newImage() SIGNAL ..
// this signal ocurrs when a new processing image was push into the buffer
void StreamThread::update(){
if(!this->m_src.element().isNull()){
this->pipeline->setState(QGst::StatePlaying);

Why are you setting the pipeline *again* to playing state?

QGst::BufferPtr buffer = QGst::Buffer::create(WIDTH*HEIGHT*3*sizeof(quint8*));
quint8 *data = buffer->data();
Mat frame = this->ImageBuffer->getFrame();
data = (quint8*)frame.data;

I hope you realize that this statement is void, it copies no actual
data. You need to copy the data with memcpy() instead.

QGst::FlowReturn ret = this->m_src.pushBuffer(buffer);

...and effectively here you are pushing an empty buffer. No wonder why
the other end doesn't receive anything.

if(ret!=QGst::FlowOk){
qDebug()<<"Error pushing data";
}
}
}
StreamThread::~StreamThread(){
this->pipeline->setState(QGst::StateNull);
}

When i run this app i don't get errors, just this warning (this doesn't show if i don't execute StreamThread)
QObject::startTimer: QTimer can only be used with threads started with QThread
So i suppose that QGstreamer use QTimer in some hidden part.

And... if i try to see the stream using
gst-launch tcpclientsrc host=127.0.0.1 port=5000 ! jpegdec ! queue ! video/x-raw-rgb,width=320,height=240 ! ffmpegcolorspace ! queue ! glimagesink sync=false

I get:
Estableciendo el conducto a PAUSA ?
El conducto est? PREPAR?NDOSE ?

But the pipeline never goes to PLAYING and nothing happen (i can't see the result)...

Any idea??

Thanks in advance

Regards,
George

[1]. http://gstreamer.freedesktop.org/data/doc/gstreamer/head/qt-gstreamer/html/bus_8cpp_source.html
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Matías Hernandez Arellano
Ingeniero de Software/Proyectos en VisionLabs S.A
CDA Archlinux-CL
www.msdark.archlinux.cl

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

George Kiagiadakis

Re: Gstreamer with multithread app (Qt)

On Wed, Apr 13, 2011 at 1:59 AM, Matias Hernandez Arellano
<[hidden email]> wrote:

>    Mat frame = this->ImageBuffer->getFrame();
>
>    data = (quint8*)frame.data;
>
> I hope you realize that this statement is void, it copies no actual
> data. You need to copy the data with memcpy() instead.
>
>    QGst::FlowReturn ret = this->m_src.pushBuffer(buffer);
>
>
> I do this cause i asking in other thread about the use of QGst::Buffer and i
> get this:
> The QGst::Buffer class has a data() method which gives you a pointer
> to the allocated buffer. You can use this pointer to write on the
> buffer.
>
> Btw, the above line of code looks terribly wrong to me... Here is how
> you should use it:
>
> QGst::BufferPtr b = QGst::Buffer::create(size);
> quint8 *data = b->data();
> //use the data pointer to write.

Yes, I meant to write on the location where data is pointing at, using
some function like memcpy() or something similar. This is basic C/C++.

> But i don't know if this is right.. so i try with memcpy. When i testing
> with C interface of Gstreamer i do this:
> memcpy(GST_BUFFER_DATA(buffer),img_data,data_size);
> even i use this GST_BUFFER_DATA(buffer) = img_data; GST_BUFFER_SIZE(buffer)
> = 640*480*3*sizeof(uchar*);
> but now i don't know how copy the data into QGst::Buffer... this don't work
> ...
>
> memcpy(data,frame.data,640*480*3*sizeof(uchar*))

What do you mean it doesn't work? Does it crash? Are you sure about
that sizeof(uchar*) there?

Try:
memcpy(data, frame.data, 640*480*3);

Regards,
George
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Matias Hernandez Arellano

Re: Gstreamer with multithread app (Qt)

> Try:
> memcpy(data, frame.data, 640*480*3);

This work, but now .. i have a crash (EXEC_BAD_ACCESS) with this QGst::FlowReturn ret = this->m_src.pushBuffer(buffer);

Works for a while and then crash....

any idea??

Thanks again again again
El 13-04-2011, a las 15:30, George Kiagiadakis escribió:

> On Wed, Apr 13, 2011 at 1:59 AM, Matias Hernandez Arellano
> <[hidden email]> wrote:
>> Mat frame = this->ImageBuffer->getFrame();
>>
>> data = (quint8*)frame.data;
>>
>> I hope you realize that this statement is void, it copies no actual
>> data. You need to copy the data with memcpy() instead.
>>
>> QGst::FlowReturn ret = this->m_src.pushBuffer(buffer);
>>
>>
>> I do this cause i asking in other thread about the use of QGst::Buffer and i
>> get this:
>> The QGst::Buffer class has a data() method which gives you a pointer
>> to the allocated buffer. You can use this pointer to write on the
>> buffer.
>>
>> Btw, the above line of code looks terribly wrong to me... Here is how
>> you should use it:
>>
>> QGst::BufferPtr b = QGst::Buffer::create(size);
>> quint8 *data = b->data();
>> //use the data pointer to write.
>
> Yes, I meant to write on the location where data is pointing at, using
> some function like memcpy() or something similar. This is basic C/C++.
>
>> But i don't know if this is right.. so i try with memcpy. When i testing
>> with C interface of Gstreamer i do this:
>> memcpy(GST_BUFFER_DATA(buffer),img_data,data_size);
>> even i use this GST_BUFFER_DATA(buffer) = img_data; GST_BUFFER_SIZE(buffer)
>> = 640*480*3*sizeof(uchar*);
>> but now i don't know how copy the data into QGst::Buffer... this don't work
>> ...
>>
>> memcpy(data,frame.data,640*480*3*sizeof(uchar*))
>
> What do you mean it doesn't work? Does it crash? Are you sure about
> that sizeof(uchar*) there?
>
> Try:
> memcpy(data, frame.data, 640*480*3);
>
>
> Regards,
> George
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Matias Hernandez Arellano

Re: Gstreamer with multithread app (Qt)

In reply to this post by George Kiagiadakis

> Try:
> memcpy(data, frame.data, 640*480*3);

This work, but now .. i have a crash (EXEC_BAD_ACCESS) with this QGst::FlowReturn ret = this->m_src.pushBuffer(buffer);

Works for a while and then crash....

any idea??

Thanks again aga
El 13-04-2011, a las 15:30, George Kiagiadakis escribió:

Matias Hernandez Arellano

Re: Gstreamer with multithread app (Qt)

In reply to this post by George Kiagiadakis

I continue with this... finally i don't get crash problems now...

this is the StreamThread :
Constructor:

QString pipe_str = QString("appsrc name=\"appsrc\" caps=\"video/x-raw-rgb,width=320,height=240\" ! videoparse format=14 width=%1 height=%1 ! videorate "
" ! videoscale ! video/x-raw-rgb,width=320,height=240 "
" ! ffmpegcolorspace ! jpegenc "
" ! tcpserversink port=5000 ").arg(WIDTH,HEIGHT);
this->pipeline = QGst::Parse::launch(pipe_str).dynamicCast<QGst::Pipeline>();
this->m_src.setElement(this->pipeline->getElementByName("appsrc"));
this->pipeline->setState(QGst::StatePlaying);
this->m_src.setLive(true); //with this the application don't crash

The Loop of the Thread
for(;;){
Mat frame = this->ImageBuffer->getFrame();
if(!this->m_src.element().isNull() && !frame.empty()){
this->pipeline->setState(QGst::StatePlaying);
QGst::BufferPtr buffer = QGst::Buffer::create(WIDTH*HEIGHT*3*sizeof(quint8*));
quint8 *data = buffer->data();
memcpy(data,frame.data,frame.elemSize());
if(!buffer.isNull()){
QGst::FlowReturn ret = this->m_src.pushBuffer(buffer);
if(ret!=QGst::FlowOk)
qDebug()<<"Error";
}
}
}

The applicaiton don't crash but if i try in other machine (or even the same) with this pipeline
gst-launch tcpclientsrc host=127.0.0.1 port=5000 ! jpegdec ! queue ! video/x-raw-rgb,width=320,height=240 ! ffmpegcolorspace ! queue ! glimagesink sync=false

I can't get something.. the pipeline never goes to PLAYING State .... and i don't find how can i fix this....

if any one can help with this .. please!!!

Thanks a lot

El 13-04-2011, a las 15:30, George Kiagiadakis escribió:

Matias Hernandez Arellano

Re: Gstreamer with multithread app (Qt)

In reply to this post by George Kiagiadakis

I continue with this... finally i don't get crash problems now...

this is the StreamThread :
Constructor:

QString pipe_str = QString("appsrc name=\"appsrc\" caps=\"video/x-raw-rgb,width=320,height=240\" ! videoparse format=14 width=%1 height=%1 ! videorate "
" ! videoscale ! video/x-raw-rgb,width=320,height=240 "
" ! ffmpegcolorspace ! jpegenc "
" ! tcpserversink port=5000 ").arg(WIDTH,HEIGHT);
this->pipeline = QGst::Parse::launch(pipe_str).dynamicCast<QGst::Pipeline>();
this->m_src.setElement(this->pipeline->getElementByName("appsrc"));
this->pipeline->setState(QGst::StatePlaying);
this->m_src.setLive(true); //with this the application don't crash

The Loop of the Thread
for(;;){
Mat frame = this->ImageBuffer->getFrame();
if(!this->m_src.element().isNull() && !frame.empty()){
this->pipeline->setState(QGst::StatePlaying);
QGst::BufferPtr buffer = QGst::Buffer::create(WIDTH*HEIGHT*3*sizeof(quint8*));
quint8 *data = buffer->data();
memcpy(data,frame.data,frame.elemSize());
if(!buffer.isNull()){
QGst::FlowReturn ret = this->m_src.pushBuffer(buffer);
if(ret!=QGst::FlowOk)
qDebug()<<"Error";
}
}
}

The applicaiton don't crash but if i try in other machine (or even the same) with this pipeline
gst-launch tcpclientsrc host=127.0.0.1 port=5000 ! jpegdec ! queue ! video/x-raw-rgb,width=320,height=240 ! ffmpegcolorspace ! queue ! glimagesink sync=false

I can't get something.. the pipeline never goes

El 13-04-2011, a las 15:30, George Kiagiadakis escribió:

Matias Hernandez Arellano

Re: Gstreamer with multithread app (Qt)

In reply to this post by George Kiagiadakis

Me again, but i can't figure out why this can't work...

I have this into the run method of a thread:
for(;;){
Mat frame = this->ImageBuffer->getFrame(); //Get a OpenCV Frame from Circular Buffer
if(!this->m_src.element().isNull() && !frame.empty()){
QGst::BufferPtr buffer = QGst::Buffer::create(640*480*3);
quint8 *data = buffer->data();
memcpy(data,(quint8*)frame.data,640*480*3);
if(!buffer.isNull()){
QGst::FlowReturn ret = this->m_src.pushBuffer(buffer);
if(ret!=QGst::FlowOk)
qDebug()<<"Error";
}
buffer.clear();
}
}

But if i try to see the result (using a receive pipeline into a terminal) the pipeline don't go to PLAYING state so i though i don' copy the data into the buffer the right way.. so i try to see what is in the buffer (what it push into appsrc).. using this qDebug()<<buffer->data();
qDebug()<<buffer->size();

So the buffer->size() show me the exact size i put (640*480*3) .. and the buffer->data() always show me the same value 0x137847000, so the memory direction never change????? .. so i think i push wrong data into the appsrc and obviously i can't see anything in the "client" ...

How can i accomplish this..????

Any help???

Thanks in advance...

El 13-04-2011, a las 15:30, George Kiagiadakis escribió: