GStreamer-devel

Debugging a blocked GStreamer pipeline

Classic

List

Threaded

5 messages Options

Todd Fischer

Debugging a blocked GStreamer pipeline

Hi,

We are seeing a behavior where we run a GStreamer application (doing audio / video decoding), that runs continuously for several days, then suddenly locks up in the middle of an A/V stream. Our best guess is there is a defect in the ALSA output driver. We believe this because if we exit the application and try aplay, it doesn't work.

I am wondering if there is a debug GStreamer logger element in existence or if one is even possible or helpful. Such a logger element could be put anywhere in the pipeline. The logger would have circular buffers to keep track of all potentially interesting recent history, such as pad activity, bus activity, and any other relevant information. The circular buffer entries would all be timestamped. When some event occurs (a file exists, a message/signal is received, etc), the element would dump the history, and continue capturing new data.

This idea is after the pipeline locks up, you could cause the history logger to dump it data, and then get an idea of what is suppose to be happening that isn't not occurring.

Does such a logging element exist? If not, does it make any sense to develop?

Todd

------------------------------------------------------------------------------
This SF.net Dev2Dev email is sponsored by:

Show off your parallel programming skills.
Enter the Intel(R) Threading Challenge 2010.
http://p.sf.net/sfu/intel-thread-sfd
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Nicolas Bertrand-4

Re: Debugging a blocked GStreamer pipeline

Hi,
Maybe you can use the GST_DEBUG env variable.
cf :
http://www.gstreamer.net/data/doc/gstreamer/head/gstreamer/html/gst-running.html
http://www.gstreamer.net/data/doc/gstreamer/head/manual/html/section-checklist-debug.html

Nico

Todd Fischer a écrit :

> Hi,
>
> We are seeing a behavior where we run a GStreamer application (doing
> audio / video decoding), that runs continuously for several days, then
> suddenly locks up in the middle of an A/V stream. Our best guess is
> there is a defect in the ALSA output driver. We believe this because
> if we exit the application and try aplay, it doesn't work.
>
> I am wondering if there is a debug GStreamer logger element in
> existence or if one is even possible or helpful. Such a logger
> element could be put anywhere in the pipeline. The logger would have
> circular buffers to keep track of all potentially interesting recent
> history, such as pad activity, bus activity, and any other relevant
> information. The circular buffer entries would all be timestamped.
> When some event occurs (a file exists, a message/signal is received,
> etc), the element would dump the history, and continue capturing new data.
>
> This idea is after the pipeline locks up, you could cause the history
> logger to dump it data, and then get an idea of what is suppose to be
> happening that isn't not occurring.
>
> Does such a logging element exist? If not, does it make any sense to
> develop?
>
> Todd
>
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> This SF.net Dev2Dev email is sponsored by:
>
> Show off your parallel programming skills.
> Enter the Intel(R) Threading Challenge 2010.
> http://p.sf.net/sfu/intel-thread-sfd
> ------------------------------------------------------------------------
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>

------------------------------------------------------------------------------
This SF.net Dev2Dev email is sponsored by:

Show off your parallel programming skills.
Enter the Intel(R) Threading Challenge 2010.
http://p.sf.net/sfu/intel-thread-sfd
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Stefan Sauer

Re: Debugging a blocked GStreamer pipeline

In reply to this post by Todd Fischer

Am 02.09.2010 04:15, schrieb Todd Fischer:

> Hi,
>
> We are seeing a behavior where we run a GStreamer application (doing audio /
> video decoding), that runs continuously for several days, then suddenly locks up
> in the middle of an A/V stream. Our best guess is there is a defect in the ALSA
> output driver. We believe this because if we exit the application and try
> aplay, it doesn't work.
>
> I am wondering if there is a debug GStreamer logger element in existence or if
> one is even possible or helpful. Such a logger element could be put anywhere in
> the pipeline. The logger would have circular buffers to keep track of all
> potentially interesting recent history, such as pad activity, bus activity, and
> any other relevant information. The circular buffer entries would all be
> timestamped. When some event occurs (a file exists, a message/signal is
> received, etc), the element would dump the history, and continue capturing new data.
>
> This idea is after the pipeline locks up, you could cause the history logger to
> dump it data, and then get an idea of what is suppose to be happening that isn't
> not occurring.
>
> Does such a logging element exist? If not, does it make any sense to develop?

You can use gst-tracelib. It is a ld-preload thing for linux boxes. It can log
all dataflow (buffers, events, messages and queries) as well as structural
changes. Logging can be file, memory or socket.

Stefan

>
> Todd
>
>
>
>
> ------------------------------------------------------------------------------
> This SF.net Dev2Dev email is sponsored by:
>
> Show off your parallel programming skills.
> Enter the Intel(R) Threading Challenge 2010.
> http://p.sf.net/sfu/intel-thread-sfd
>
>
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Todd Fischer

Re: Debugging a blocked GStreamer pipeline

Hi Stefan,

Thanks for the information about gst-tracelib. We have used this library in the past to produce some great performance graphs.

I see the library supports a GSTTL_LOG_SIZE variable. Does that keep the first log data until the buffer is full, or does it overwrite old data with the most current information?

Todd

On Sun, 2010-09-05 at 22:33 +0300, Stefan Kost wrote:

Am 02.09.2010 04:15, schrieb Todd Fischer:
> Hi,
> 
> We are seeing a behavior where we run a GStreamer application (doing audio /
> video decoding), that runs continuously for several days, then suddenly locks up
> in the middle of an A/V stream.  Our best guess is there is a defect in the ALSA
> output driver.  We believe this because if we exit the application and try
> aplay, it doesn't work.
> 
> I am wondering if there is a debug GStreamer logger element in existence or if
> one is even possible or helpful.  Such a logger element could be put anywhere in
> the pipeline. The logger would have circular buffers to keep track of all
> potentially interesting recent history, such as pad activity, bus activity, and
> any other relevant information. The circular buffer entries would all be
> timestamped. When some event occurs (a file exists, a message/signal is
> received, etc), the element would dump the history, and continue capturing new data.
> 
> This idea is after the pipeline locks up, you could cause the history logger to
> dump it data, and then get an idea of what is suppose to be happening that isn't
> not occurring.
> 
> Does such a logging element exist?  If not, does it make any sense to develop?

You can use gst-tracelib. It is a ld-preload thing for linux boxes. It can log
all dataflow (buffers, events, messages and queries) as well as structural
changes. Logging can be file, memory or socket.

Stefan

> 
> Todd
> 
> 
> 
> 
> ------------------------------------------------------------------------------
> This SF.net Dev2Dev email is sponsored by:
> 
> Show off your parallel programming skills.
> Enter the Intel(R) Threading Challenge 2010.
> http://p.sf.net/sfu/intel-thread-sfd
> 
> 
> 
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Stefan Sauer

Re: Debugging a blocked GStreamer pipeline

Am 06.09.2010 01:07, schrieb Todd Fischer:
> Hi Stefan,
>
> Thanks for the information about gst-tracelib. We have used this library in the
> past to produce some great performance graphs.
>
> I see the library supports a GSTTL_LOG_SIZE variable. Does that keep the first
> log data until the buffer is full, or does it overwrite old data with the most
> current information?

If the buffer is full it counts and tells at the end how much it would have
needed. Then you can run again with a bigger size.

If you would need a different scheme let me know, it can probably be added easily.

Stefan

>
> Todd
>
> On Sun, 2010-09-05 at 22:33 +0300, Stefan Kost wrote:
>> Am 02.09.2010 04:15, schrieb Todd Fischer:
>> > Hi,
>> >
>> > We are seeing a behavior where we run a GStreamer application (doing audio /
>> > video decoding), that runs continuously for several days, then suddenly locks up
>> > in the middle of an A/V stream. Our best guess is there is a defect in the ALSA
>> > output driver. We believe this because if we exit the application and try
>> > aplay, it doesn't work.
>> >
>> > I am wondering if there is a debug GStreamer logger element in existence or if
>> > one is even possible or helpful. Such a logger element could be put anywhere in
>> > the pipeline. The logger would have circular buffers to keep track of all
>> > potentially interesting recent history, such as pad activity, bus activity, and
>> > any other relevant information. The circular buffer entries would all be
>> > timestamped. When some event occurs (a file exists, a message/signal is
>> > received, etc), the element would dump the history, and continue capturing new data.
>> >
>> > This idea is after the pipeline locks up, you could cause the history logger to
>> > dump it data, and then get an idea of what is suppose to be happening that isn't
>> > not occurring.
>> >
>> > Does such a logging element exist? If not, does it make any sense to develop?
>>
>> You can use gst-tracelib. It is a ld-preload thing for linux boxes. It can log
>> all dataflow (buffers, events, messages and queries) as well as structural
>> changes. Logging can be file, memory or socket.
>>
>> Stefan
>>
>> >
>> > Todd
>> >
>> >
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > This SF.net Dev2Dev email is sponsored by:
>> >
>> > Show off your parallel programming skills.
>> > Enter the Intel(R) Threading Challenge 2010.
>> > http://p.sf.net/sfu/intel-thread-sfd
>> >
>> >
>> >
>> > _______________________________________________
>> > gstreamer-devel mailing list
>> > [hidden email] <mailto:[hidden email]>
>> > https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>>