Hi folks,
I'm experiencing an issue wherein a temporary load spike on a system causes shmsink to stop delivering frames permanently.
(a cron job is doing it to me in real life, but I can replicate the issue with "stress -m 5 -t 5")
My pipeline is decklinkvideosrc ! videoconvert ! shmsink -> snowmix
(the issue can be duplicated using filesrc location=test.mp4 ! decodebin in place of the decklinkvideosrc)
more precicely:
gst-launch-1.0 -v decklinkvideosrc mode=19 connection=sdi device-number=0 name=decoder ! videorate ! videoscale ! videoconvert ! video/x-raw, format=\(string\)BGRA, pixel-aspect-ratio=\(fraction\)1/1, interlace-mode=\(string\)progressive, width=\(int\)1280, height=\(int\)720, framerate=\(fraction\)30/1 ! queue leaky=2 ! shmsink socket-path=/tmp/feed1-control-pipe shm-size=100000000 wait-for-connection=0 sync=false
So, my shm size is roughly 27 frames.
It appears that during the load spike frames pile up in the shm space.
When load returns to normal, the shm client reads and ACKs every one of the backlogged buffers, but they do not seem to be properly freed in the shmsink.
If everything recovers before the shm space is filled (say, 25 frames have accumulated in the ShmAllocSpace) then everything seems to recover fine, frames flow normally.
But if shmsink ever fills the ShmAllocSpace then I'm boned. the client will, when it recovers, read all the buffers that are in there, but no more will ever be written.
When it ends up in this state, this function: shm_alloc_space_alloc_block in shmalloc.c is being called for each new frame and is returning NULL.
When this is happening, the ShmAllocSpace linked list of blocks has 27 entries. It stays at 27 forever.
I watched sp_writer_recv in shmpipe.c and I can see an ACK come in for every one of those 27 buffers. The ShmPipe's linked list of buffers get decremented from 27 down to a normal level, but the ShmAllocSpace's list of blocks never goes down.
I tried to understand how blocks in the ShmAllocSpace get released or re-used but I'm failing to wrap my head around it.
Any thoughts would be welcome.
For the moment I can just throw a huge shm-size at the problem (300 frames?) and hope I never have a load issue that lasts that long.
That works but isn't a very satisfying solution.
Locked
