GStreamer-devel

Gstreamer / Multithreading problem

Classic

List

Threaded

5 messages Options

klex0

Gstreamer / Multithreading problem

Hello,

I use Gstreamer together with the Python Twisted Framework. There seems to
be a race condition when sending events in the pipeline.
With many parallel pipelines which I dynamically create and terminate, a
thread hangs reproducibly in the marked line. This seems to happen during a
context switch of a twisted thread within this line.

log.msg("TERM: gen EOS")
eos = Gst.Event.new_eos()
log.msg("TERM: send EOS")
self.pipeline.send_event(eos) <-- Here it hangs
log.msg("TERM: EOS sent to pipline")

Has anyone an idea about this ?

Regards,
alex

--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Nicolas Dufresne-5

Re: Gstreamer / Multithreading problem

Le ven. 5 oct. 2018 08 h 41, klex0 <[hidden email]> a écrit :

Hello,

I use Gstreamer together with the Python Twisted Framework. There seems to
be a race condition when sending events in the pipeline.
With many parallel pipelines which I dynamically create and terminate, a
thread hangs reproducibly in the marked line. This seems to happen during a
context switch of a twisted thread within this line.

log.msg("TERM: gen EOS")
eos = Gst.Event.new_eos()
log.msg("TERM: send EOS")
self.pipeline.send_event(eos) <-- Here it hangs
log.msg("TERM: EOS sent to pipline")

Has anyone an idea about this ?

Interactions between GStreame and the Python run-time is single threaded. As a side effect, some cause that is supposed to run in seperate thread may cause deadlocks.

I have heard that recent Python API could help with this, but that would need some rework of the GObject auto-generated bindings (and research, and load of testing).

Best is to add a small native layer that deals with the threaded callbacks, or avoid threaded callbacks.

Regards,
alex

--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

klex0

Re: Gstreamer / Multithreading problem

Hi Nicolas,

thank you very much. I removed all threading and it doesn't seem to be a
problem with multithreading.
I think there is a bug in the rtp_jitterbuffer in the function
gst_rtp_buffer_ext_timestamp() when there is a big jump in the timestamps.
When I remove the jitterbuffer from my pipeline, everything works.

--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Nicolas Dufresne-5

Re: Gstreamer / Multithreading problem

Le mar. 9 oct. 2018 08 h 42, klex0 <[hidden email]> a écrit :

Hi Nicolas,

thank you very much. I removed all threading and it doesn't seem to be a
problem with multithreading.
I think there is a bug in the rtp_jitterbuffer in the function
gst_rtp_buffer_ext_timestamp() when there is a big jump in the timestamps.
When I remove the jitterbuffer from my pipeline, everything works.

What are the size of these jumps ? Did you configure the clock-rate in a way that you don't overflow on big jumps ? In RTP it's application responsibility to pick the right click rate. This rate is how the timestamp are scaled down to fit into 32bit timestamp. A jump must not be bigger then MAXINT / 2. Bigger jumps will lead to a clock resync, and may not be appropriate.

--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

klex0

Re: Gstreamer / Multithreading problem

Hi Nicolas,

here is an example:

1 0.000000 192.168.1.225 → 10.128.1.182 SIP/SDP 1394 Request: INVITE <a href="sip:8905105@10.128.1.182:5060;line=wbvyarue" class="">sip:8905105@10.128.1.182:5060;line=wbvyarue |

2 0.034750 10.128.1.182 → 192.168.1.225 SIP 568 Status: 180 Ringing |

3 4.291404 10.128.1.182 → 192.168.1.225 RTCP 110 Sender Report Source description

4 4.312332 10.128.1.182 → 192.168.1.225 SIP/SDP 948 Status: 200 Ok |

5 4.314067 192.168.1.225 → 10.128.1.182 SIP 452 Request: ACK <a href="sip:8905105@10.128.1.182:5060;line=wbvyarue" class="">sip:8905105@10.128.1.182:5060;line=wbvyarue |

6 4.362510 192.168.1.225 → 10.128.1.182 SIP/sipfrag 665 Request: INFO <a href="sip:8905105@10.128.1.182:5060;line=wbvyarue" class="">sip:8905105@10.128.1.182:5060;line=wbvyarue | , with Sipfrag(From: "callcenter" callcenter...)

7 4.392291 192.168.1.225 → 10.128.1.182 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0xDC2D2E57, Seq=16122, Time=160, Mark

8 4.417775 10.128.1.182 → 192.168.1.225 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x7DA7E15E, Seq=19404, Time=1369368212, Mark

9 4.418509 10.128.1.182 → 192.168.1.225 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x7DA7E15E, Seq=19405, Time=1369368372

10 4.419393 10.128.1.182 → 192.168.1.225 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x7DA7E15E, Seq=19406, Time=1369368532

11 4.432259 192.168.1.225 → 10.128.1.182 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0xDC2D2E57, Seq=16123, Time=160, Mark

12 4.435693 10.128.1.182 → 192.168.1.225 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x7DA7E15E, Seq=19407, Time=1369368692

13 4.440873 10.128.1.182 → 192.168.1.225 SIP 419 Status: 200 Ok |

14 4.482457 10.128.1.182 → 192.168.1.225 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x7DA7E15E, Seq=19408, Time=1369368852

15 4.483059 10.128.1.182 → 192.168.1.225 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x7DA7E15E, Seq=19409, Time=1369369012

16 4.492230 192.168.1.225 → 10.128.1.182 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0xDC2D2E57, Seq=16124, Time=2667591785

17 4.512334 192.168.1.225 → 10.128.1.182 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0xDC2D2E57, Seq=16125, Time=2667591945

18 4.518921 10.128.1.182 → 192.168.1.225 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x7DA7E15E, Seq=19410, Time=1369369172

19 4.519658 10.128.1.182 → 192.168.1.225 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x7DA7E15E, Seq=19411, Time=1369369332

20 4.532355 192.168.1.225 → 10.128.1.182 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0xDC2D2E57, Seq=16126, Time=2667592105

...

10729 111.517164 10.128.1.182 → 192.168.1.225 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x7DA7E15E, Seq=24761, Time=1370225332

10730 111.532725 192.168.1.225 → 10.128.1.182 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0xDC2D2E57, Seq=21476, Time=2668448105

10731 111.538454 10.128.1.182 → 192.168.1.225 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x7DA7E15E, Seq=24762, Time=1370225492

10732 111.552794 192.168.1.225 → 10.128.1.182 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0xDC2D2E57, Seq=21477, Time=160, Mark

10733 111.557090 10.128.1.182 → 192.168.1.225 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x7DA7E15E, Seq=24763, Time=1370225652

10734 111.577114 10.128.1.182 → 192.168.1.225 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x7DA7E15E, Seq=24764, Time=1370225812

10735 111.584062 192.168.1.225 → 10.128.1.182 SIP 661 Request: BYE <a href="sip:8905105@10.128.1.182:5060;line=wbvyarue" class="">sip:8905105@10.128.1.182:5060;line=wbvyarue |

10736 111.607864 10.128.1.182 → 192.168.1.225 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x7DA7E15E, Seq=24765, Time=1370225972

10737 111.617072 10.128.1.182 → 192.168.1.225 RTP 214 PT=ITU-T G.711 PCMA, SSRC=0x7DA7E15E, Seq=24766, Time=1370226132

10738 111.626308 10.128.1.182 → 192.168.1.225 SIP 587 Status: 200 OK |

10739 111.756027 10.128.1.182 → 192.168.1.225 RTCP 62 Goodbye

I already set the clock-rate for the rtp stream to 8000 but I don’t think the timestamps get scaled. I’m getting many of these warnings:

0:00:00.836097511 8936 0x55ee997c71e0 WARN default gstrtpbuffer.c:1296:gst_rtp_buffer_ext_timestamp: Cannot unwrap, any wrapping took place yet. Returning 0 without updating extended timestamp.

Interesting is, when I run this pipeline alone, my program completes, when I run this pipeline together with other pipelines in parallel, which are also working, sometimes it hangs during the sending of EOS in _another_ Pipeline.

Viele Grüße,
Alexander Kordecki

—
Dipl. Ing. Beratung und Softwareentwicklung
Alexander Kordecki für BigData, Datenbanken,
Veitstr. 34a Netzwerke und Telekommunikation
13507 Berlin [hidden email]
Germany 030 / 303 660 660

Am 09.10.2018 um 15:29 schrieb Nicolas Dufresne <[hidden email]>:

Le mar. 9 oct. 2018 08 h 42, klex0 <[hidden email]> a écrit :
Hi Nicolas,

thank you very much. I removed all threading and it doesn't seem to be a
problem with multithreading.
I think there is a bug in the rtp_jitterbuffer in the function
gst_rtp_buffer_ext_timestamp() when there is a big jump in the timestamps.
When I remove the jitterbuffer from my pipeline, everything works.

What are the size of these jumps ? Did you configure the clock-rate in a way that you don't overflow on big jumps ? In RTP it's application responsibility to pick the right click rate. This rate is how the timestamp are scaled down to fit into 32bit timestamp. A jump must not be bigger then MAXINT / 2. Bigger jumps will lead to a clock resync, and may not be appropriate.

--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel