What is the bitrate of your stream? For high bitrates, it could be a network stack buffer overflow. You can increase the UDP buffers using:
sysctl -w net.core.rmem_max=<value>
sysctl -w net.core.rmem_default=<value>
sysctl -w net.core.wmem_max=<value>
sysctl -w net.core.wmem_default=<value>
value would be in bytes. Normally a couple of megabytes should be sufficient.