GStreamer-devel

Internal data stream error due to stream position with interleave and audiomixer

Classic

List

Threaded

3 messages Options

Mark W

Internal data stream error due to stream position with interleave and audiomixer

I am developing a gstreamer app in C that receives multiple RTP audio streams over UDP, sends a copy of each to a WAV file, and sends another copy to an alsasrc. To select which stream goes to alsasrc, I am using an audiomixer element and I set the volume on the sink pads appropriately. This all works when I have a single channel going to alsasrc.

I need to have two streams going to alsasrc, each of which can be selected independently. For this, I have set up an interleave element and a second audiomixer. Each of the 8 incoming streams has a tee, sending the output to each audiomixer.

This also works, but only when I use a single stream for input. When I add a second stream, I get an Internal data stream error, reason: not-negotiated. With debug on, I chased this back to the audiomixer element, where I got the following message:
0:00:15.634352592 ^[[332m 1297^[[00m   0x243920 ^[[37mDEBUG ^[[00m ^[[00m          audiomixer gstaudiomixer.c:402:gst_audiomixer_setcaps:<audiomixer0:sink_1>^[[00m got input caps
audio/x-raw, layout=(string)interleaved, rate=(int)16000, format=(string)S16LE, channels=(int)1, channel-mask=(bitmask)0x0000000000000001, but current caps are
audio/x-raw, layout=(string)interleaved, rate=(int)16000, format=(string)S16LE, channels=(int)1, channel-mask=(bitmask)0x0000000000000001

The caps are the same. I dug deeper, adding my own GST_DEBUG statements and found that gst_audio_info_is_equal was returning false to gst_audiomixer_setcaps, because of this section:
if (memcmp (info->position, other->position,
          GST_AUDIO_INFO_CHANNELS (info) * sizeof (GstAudioChannelPosition)) !=
      0)
    return FALSE;

I have tried many combinations of properties for the audiomixer and interleave elements. I admit I do not completely understand how to use the start-time or alignment threshold properties, but they seem like that could lead to a solution.

I am stuck using gstreamer 1.12.2 for this embedded product, using yocto, branch rocko. I have a hunch that 1.14.4, from yocto branch thud, might resolve this problem, but I would rather not mix up versions in our SDK if I can help it, and it can be difficult to port versions across yocto branches.

The following is a simplified gst-launch pipeline which results in a very similar failure. In this example, the error propogates back to the udpsrc rather than coming up in audiomixer.

gst-launch-1.0 -v interleave name=i ! \
audioconvert ! \
alsasink sync=false buffer-time=1000 \
\
udpsrc port=5000 buffer-size=1000 ! \
"application/x-rtp, media=(string)audio, clock-rate=(int)16000, encoding-name=(string)L16" ! \
rtpL16depay ! \
queue ! \
audio/x-raw,format=S16BE,layout=interleaved,rate=16000,channels=1 ! \
audioconvert ! \
"audio/x-raw,channels=1,channel-mask=(bitmask)0x1" ! \
tee name=t0 ! queue ! audiomixer name=mix0 !\
i.sink_0 \
\
udpsrc port=5001 buffer-size=1000 ! \
"application/x-rtp, media=(string)audio, clock-rate=(int)16000, encoding-name=(string)L16" ! \
rtpL16depay ! \
queue ! \
audio/x-raw,format=S16BE,layout=interleaved,rate=16000,channels=1 ! \
audioconvert ! \
"audio/x-raw,channels=1,channel-mask=(bitmask)0x2" ! \
tee name=t1 ! queue ! audiomixer name=mix1 !\
i.sink_1 \
t0. ! queue ! mix1. \
t1. ! queue ! mix0.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Nicolas Dufresne-5

Re: Internal data stream error due to stream position with interleave and audiomixer

Le ven. 19 juill. 2019 15 h 40, <[hidden email]> a écrit :

I am developing a gstreamer app in C that receives multiple RTP audio streams over UDP, sends a copy of each to a WAV file, and sends another copy to an alsasrc. To select which stream goes to alsasrc, I am using an audiomixer element and I set the volume on the sink pads appropriately. This all works when I have a single channel going to alsasrc.

I need to have two streams going to alsasrc, each of which can be selected independently. For this, I have set up an interleave element and a second audiomixer. Each of the 8 incoming streams has a tee, sending the output to each audiomixer.

This also works, but only when I use a single stream for input. When I add a second stream, I get an Internal data stream error, reason: not-negotiated. With debug on, I chased this back to the audiomixer element, where I got the following message:
0:00:15.634352592 ^[[332m 1297^[[00m   0x243920 ^[[37mDEBUG ^[[00m ^[[00m          audiomixer gstaudiomixer.c:402:gst_audiomixer_setcaps:<audiomixer0:sink_1>^[[00m got input caps
audio/x-raw, layout=(string)interleaved, rate=(int)16000, format=(string)S16LE, channels=(int)1, channel-mask=(bitmask)0x0000000000000001, but current caps are
audio/x-raw, layout=(string)interleaved, rate=(int)16000, format=(string)S16LE, channels=(int)1, channel-mask=(bitmask)0x0000000000000001

The caps are the same. I dug deeper, adding my own GST_DEBUG statements and found that gst_audio_info_is_equal was returning false to gst_audiomixer_setcaps, because of this section:
if (memcmp (info->position, other->position,
          GST_AUDIO_INFO_CHANNELS (info) * sizeof (GstAudioChannelPosition)) !=
      0)
    return FALSE;

I have tried many combinations of properties for the audiomixer and interleave elements. I admit I do not completely understand how to use the start-time or alignment threshold properties, but they seem like that could lead to a solution.

I am stuck using gstreamer 1.12.2 for this embedded product, using yocto, branch rocko. I have a hunch that 1.14.4, from yocto branch thud, might resolve this problem, but I would rather not mix up versions in our SDK if I can help it, and it can be difficult to port versions across yocto branches.

I do the same in a more complicated way, I'm pretty sure I can help. I'm on 1.10.3 still for that application, but this didn't work so well without patches, which I believe most are covered in 1.14+.

The difference between audiomixer and audiointerleaver in such case, is that adding interleaver pads changes the caps. The other issue is that you need a channel mapping, I have never figured out how to do that from gst-launch-1.0.

The following is a simplified gst-launch pipeline which results in a very similar failure. In this example, the error propogates back to the udpsrc rather than coming up in audiomixer.

gst-launch-1.0 -v interleave name=i ! \
audioconvert ! \
alsasink sync=false buffer-time=1000 \
\
udpsrc port=5000 buffer-size=1000 ! \
"application/x-rtp, media=(string)audio, clock-rate=(int)16000, encoding-name=(string)L16" ! \
rtpL16depay ! \
queue ! \
audio/x-raw,format=S16BE,layout=interleaved,rate=16000,channels=1 ! \
audioconvert ! \
"audio/x-raw,channels=1,channel-mask=(bitmask)0x1" ! \
tee name=t0 ! queue ! audiomixer name=mix0 !\
i.sink_0 \
\
udpsrc port=5001 buffer-size=1000 ! \
"application/x-rtp, media=(string)audio, clock-rate=(int)16000, encoding-name=(string)L16" ! \
rtpL16depay ! \
queue ! \
audio/x-raw,format=S16BE,layout=interleaved,rate=16000,channels=1 ! \
audioconvert ! \
"audio/x-raw,channels=1,channel-mask=(bitmask)0x2" ! \
tee name=t1 ! queue ! audiomixer name=mix1 !\
i.sink_1 \
t0. ! queue ! mix1. \
t1. ! queue ! mix0.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Mark W

Re: Internal data stream error due to stream position with interleave and audiomixer

Thank you for your response. The gst-launch script here is only for illustration. My C app is over 1000 lines of code and I do not wish to post it to the mailing list.

I have duplicated the crash of the gst-launch script on an Ubuntu desktop running gstreamer 1.14.1, but I still don't know if 1.14.4 would help. I have run into build issues in other parts of my yocto build when I try to bring in the 1.14.4 recipe from another yocto branch.

So, I would be very interested in seeing the patches you used.

I have the following caps set on my interleave element in the C app, which I think sets the channel mask correctly:
interleave_0_caps = gst_caps_from_string("audio/x-raw,channels=1,channel-mask=(bitmask)0x1");
...
interleave_1_caps = gst_caps_from_string("audio/x-raw,channels=1,channel-mask=(bitmask)0x2");
...
gst_element_link_filtered (audiomixer0, audioconvert_interleave_0, interleave_0_caps)
...
gst_element_link_filtered (audiomixer1, audioconvert_interleave_1, interleave_1_caps)

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel