Decklink multi-channel audio source

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Decklink multi-channel audio source

jpoet
I noticed that support for 8 and 16 channel audio was added to decklinkaudiosink.  Are there any plans to support to decklinkaudiosrc as well?

I came across https://bugzilla.gnome.org/attachment.cgi?id=274910&action=diff , but that is quite old.  I have not even tried to apply it.

Thanks,

John



_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Decklink multi-channel audio source

Sebastian Dröge-3
On Mon, 2016-08-29 at 21:03 +0000, John P Poet wrote:
> I noticed that support for 8 and 16 channel audio was added to
> decklinkaudiosink.  Are there any plans to support to
> decklinkaudiosrc as well?
>
> I came across https://bugzilla.gnome.org/attachment.cgi?id=274910&act
> ion=diff , but that is quite old.  I have not even tried to apply it.

It does not apply anymore and needs a major update.

Adding support for 8 and 16 channels is relatively simple and can be
done similar to what that patch does, however decklink does not provide
any information about the location/layout of channels and as such we
would have to handle it as unpositioned.

Do you want to update the patch?

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

signature.asc (949 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Decklink multi-channel audio source

jpoet
On Tue, Aug 30, 2016 at 12:42 AM Sebastian Dröge <[hidden email]> wrote:
On Mon, 2016-08-29 at 21:03 +0000, John P Poet wrote:
> I noticed that support for 8 and 16 channel audio was added to
> decklinkaudiosink.  Are there any plans to support to
> decklinkaudiosrc as well?
>
> I came across https://bugzilla.gnome.org/attachment.cgi?id=274910&act
> ion=diff , but that is quite old.  I have not even tried to apply it.

It does not apply anymore and needs a major update.

Adding support for 8 and 16 channels is relatively simple and can be
done similar to what that patch does, however decklink does not provide
any information about the location/layout of channels and as such we
would have to handle it as unpositioned.

Do you want to update the patch?

I am willing to try.  I have hacked on other peoples code a little bit, when necessary, but do not have a high degree of comfort when working on GStreamer internals.

Is there some other src module I can use as an example?  Searching for similar code, I see:

gst-plugins-bad/ext/flite/gstflitetestsrc.c
gst-plugins-base/gst/audioconvert/gstaudioconvert.c

Based on those two, I am guessing I need to create a "fixate" function.  I need to understand how the caps are managed...

John


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Decklink multi-channel audio source

Sebastian Dröge-3
On Tue, 2016-08-30 at 19:41 +0000, John P Poet wrote:

> On Tue, Aug 30, 2016 at 12:42 AM Sebastian Dröge <[hidden email]> wrote:
> > On Mon, 2016-08-29 at 21:03 +0000, John P Poet wrote:
> > > I noticed that support for 8 and 16 channel audio was added to
> > > decklinkaudiosink.  Are there any plans to support to
> > > decklinkaudiosrc as well?
> > >
> > > I came across https://bugzilla.gnome.org/attachment.cgi?id=274910&act
> > > ion=diff , but that is quite old.  I have not even tried to apply it.
> >
> > It does not apply anymore and needs a major update.
> >
> > Adding support for 8 and 16 channels is relatively simple and can be
> > done similar to what that patch does, however decklink does not provide
> > any information about the location/layout of channels and as such we
> > would have to handle it as unpositioned.
> >
> > Do you want to update the patch?
>
> I am willing to try.  I have hacked on other peoples code a little bit, when necessary, but do not have a high degree of comfort when working on GStreamer internals.
>
> Is there some other src module I can use as an example?  Searching for similar code, I see: 
>
> gst-plugins-bad/ext/flite/gstflitetestsrc.c
> gst-plugins-base/gst/audioconvert/gstaudioconvert.c
>
> Based on those two, I am guessing I need to create a "fixate" function.  I need to understand how the caps are managed...
You would need to:
1) Add 8 / 16 channels to the template caps like in the audio 
2) Handle those in gst_decklink_audio_src_set_caps() (EnableAudioInput)
3) Check what the hardware actually supports and do something like
   in gst_decklink_audio_sink_get_caps()
4) Implement fixate to select a meaningful number of channels if
   you still get multiple in there. 2 seems like the best choice
   as we know the channel layout for that

In general for 8 / 16 channels, the channel-mask in the caps would be
0, which means unpositioned.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

signature.asc (949 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Decklink multi-channel audio source

jpoet
On Wed, Aug 31, 2016 at 1:00 AM Sebastian Dröge <[hidden email]> wrote:
On Tue, 2016-08-30 at 19:41 +0000, John P Poet wrote:
> On Tue, Aug 30, 2016 at 12:42 AM Sebastian Dröge <[hidden email]> wrote:
> > On Mon, 2016-08-29 at 21:03 +0000, John P Poet wrote:
> > > I noticed that support for 8 and 16 channel audio was added to
> > > decklinkaudiosink.  Are there any plans to support to
> > > decklinkaudiosrc as well?
> > >
> > > I came across https://bugzilla.gnome.org/attachment.cgi?id=274910&act
> > > ion=diff , but that is quite old.  I have not even tried to apply it.
> >
> > It does not apply anymore and needs a major update.
> >
> > Adding support for 8 and 16 channels is relatively simple and can be
> > done similar to what that patch does, however decklink does not provide
> > any information about the location/layout of channels and as such we
> > would have to handle it as unpositioned.
> >
> > Do you want to update the patch?
>
> I am willing to try.  I have hacked on other peoples code a little bit, when necessary, but do not have a high degree of comfort when working on GStreamer internals.
>
> Is there some other src module I can use as an example?  Searching for similar code, I see: 
>
> gst-plugins-bad/ext/flite/gstflitetestsrc.c
> gst-plugins-base/gst/audioconvert/gstaudioconvert.c
>
> Based on those two, I am guessing I need to create a "fixate" function.  I need to understand how the caps are managed...

You would need to:
1) Add 8 / 16 channels to the template caps like in the audio 
2) Handle those in gst_decklink_audio_src_set_caps() (EnableAudioInput)
3) Check what the hardware actually supports and do something like
   in gst_decklink_audio_sink_get_caps()
4) Implement fixate to select a meaningful number of channels if
   you still get multiple in there. 2 seems like the best choice
   as we know the channel layout for that

In general for 8 / 16 channels, the channel-mask in the caps would be
0, which means unpositioned.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com

Hi Sebastian,

I am attaching my attempt at this.  Unfortunately it does not quite work.  I cannot figure out how to get 8 channels of audio encoded into ac3.  With this pipeline:

```
gst-launch-1.0 decklinkvideosrc device-number=${DEVICE} mode=0 ! queue ! autovideoconvert ! deinterlace mode=2 ! queue ! avenc_mpeg2video bitrate=800000 ! mpegvideoparse ! mpegtsmux name=mux ! filesink location=video.ts decklinkaudiosrc device-number=${DEVICE} audio-channels=8 ! audioconvert ! avenc_ac3 bitrate=640000 ! ac3parse ! queue ! mux.
```

The resulting file has:
```
Audio
ID                                       : 66 (0x42)
Menu ID                                  : 1 (0x1)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Codec ID                                 : 129
Duration                                 : 3s 485ms
Bit rate mode                            : Constant
Bit rate                                 : 640 Kbps
Channel(s)                               : 1 channel
Channel positions                        : Front: C
Sampling rate                            : 48.0 KHz
Bit depth                                : 16 bits
Compression mode                         : Lossy
Stream size                              : 272 KiB (15%)
Language                                 : English

```

I have read that the audio from the Decklink is single-track, mult-channel PCM.  I thought maybe I needed to deinterlace it for the ac3 encoder to process it correctly, so I tried:
```
 gst-launch-1.0 decklinkvideosrc device-number=${DEVICE} mode=0 ! queue ! autovideoconvert ! deinterlace mode=2 ! queue ! avenc_mpeg2video bitrate=800000 ! mpegvideoparse ! mpegtsmux name=mux ! filesink location=video.ts decklinkaudiosrc device-number=${DEVICE} audio-channels=8 ! audioconvert ! "audio/x-raw,channels=8" ! deinterleave ! audioconvert ! avenc_ac3 bitrate=640000 ! ac3parse ! queue ! mux.
```
But either that pipeline is incorrect for that purpose, or it just doesn't help.

I built and ran https://github.com/lu-zero/bmdtools/blob/master/bmdcapture.cpp ,  and it produces 8 channel PCM audio.  As far as I can tell, that code is not doing any conversion, it just is writing the PCM audio straight from the Decklink card/driver.

I tried coming up with a pipeline which does not convert the audio, but was unsuccessful.

I also found the patch that added 8/16 channel support to ffmpeg: https://ffmpeg.org/pipermail/ffmpeg-devel/2015-January/167444.html

but it does not really shed any light on the situation.

Note: If I set the channel mask to zero, then those pipelines don't run at all.

Any help would be appreciated.

Thanks,

John




_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

decklink-multichannel.patch (13K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Decklink multi-channel audio source

Sebastian Dröge-3
On Wed, 2016-09-07 at 23:14 +0000, John P Poet wrote:

> Hi Sebastian,
>
> I am attaching my attempt at this.

Thanks, can you put it into Bugzilla? We're using that for patch review
/ discussions:

  https://bugzilla.gnome.org/enter_bug.cgi?product=GStreamer

> Unfortunately it does not quite work.  I cannot figure out how to get
> 8 channels of audio encoded into ac3.  With this pipeline:
>
> ```
> gst-launch-1.0 decklinkvideosrc device-number=${DEVICE} mode=0 !
> queue ! autovideoconvert ! deinterlace mode=2 ! queue !
> avenc_mpeg2video bitrate=800000 ! mpegvideoparse ! mpegtsmux name=mux
> ! filesink location=video.ts decklinkaudiosrc device-number=${DEVICE}
> audio-channels=8 ! audioconvert ! avenc_ac3 bitrate=640000 ! ac3parse
> ! queue ! mux.
> ```
> [...]
Are the caps set to 8 channels on decklinkaudiosrc?

Also avenc_ac3 only supports up to 6 channels, not 8. So audioconvert
would convert to something else.


The number of channels should also be negotiated via caps, similar to
what other elements are doing, instead of a property.

> Note: If I set the channel mask to zero, then those pipelines don't
> run at all.

The problem with setting it to anything else is that you give meaning
to each channel (the position), which you don't have with decklink. You
have 8 unpositioned channels and if anything you can know their
positions from the application.

As such, using 0 as channel mask is the right thing to do here. And it
would be the applications job to set the actual channel mask via a
capssetter or similar.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

signature.asc (949 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Decklink multi-channel audio source

jpoet
On Thu, Sep 8, 2016 at 12:33 AM Sebastian Dröge <[hidden email]> wrote:
On Wed, 2016-09-07 at 23:14 +0000, John P Poet wrote:

> Hi Sebastian,
>
> I am attaching my attempt at this.

Thanks, can you put it into Bugzilla? We're using that for patch review
/ discussions:

  https://bugzilla.gnome.org/enter_bug.cgi?product=GStreamer

Will do.
 
> Unfortunately it does not quite work.  I cannot figure out how to get
> 8 channels of audio encoded into ac3.  With this pipeline:
>
> ```
> gst-launch-1.0 decklinkvideosrc device-number=${DEVICE} mode=0 !
> queue ! autovideoconvert ! deinterlace mode=2 ! queue !
> avenc_mpeg2video bitrate=800000 ! mpegvideoparse ! mpegtsmux name=mux
> ! filesink location=video.ts decklinkaudiosrc device-number=${DEVICE}
> audio-channels=8 ! audioconvert ! avenc_ac3 bitrate=640000 ! ac3parse
> ! queue ! mux.
> ```
> [...]

Are the caps set to 8 channels on decklinkaudiosrc?

Yes.
 
Also avenc_ac3 only supports up to 6 channels, not 8. So audioconvert
would convert to something else.

Okay.  I assumed it could handle 7.1.  If AC3 is a bad choice, what would you recommend?
 
The number of channels should also be negotiated via caps, similar to
what other elements are doing, instead of a property.

> Note: If I set the channel mask to zero, then those pipelines don't
> run at all.

The problem with setting it to anything else is that you give meaning
to each channel (the position), which you don't have with decklink. You
have 8 unpositioned channels and if anything you can know their
positions from the application.

As such, using 0 as channel mask is the right thing to do here. And it
would be the applications job to set the actual channel mask via a
capssetter or similar.

I am far from an expert with gstreamer.  It would help me greately if someone could provide me with a pipeline what should work in this situation.  Maybe an example of writing 8/16 channels of PCM (LPCM?) audio to a container, or any other audio encoding which can handle that many channels.

Thank you,

John
  
 

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel