vaapih264enc and x264enc usage for encoding MVC / stereo?

[Jérôme Carretero]

Hi,



I'm running out of google-fu to find documentation on how it's possible
to try and use h264 encoding with MVC profiles.


I am under the impression that the following should work to encode
side-by-side stereoscopic video:

... ! vaapih264enc ! video/x-h264,profile=stereo-high ! h264parse ! ...

But the resulting videos don't seem to be in the right profile.


Then, there is MVC, and I really had trouble finding *anything* on that.



Best regards,

--
Jérôme
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Victor Jaquez]

Hi,

First thing to check is does your hardware support MVC?

You can know that by running `vainfo`

And you should see

      VAProfileH264MultiviewHigh      : VAEntrypointEncSlice
      VAProfileH264StereoHigh         : VAEntrypointEncSlice


Second thing is that setting the profile via capsfilters is not supported yet
by the vaapi encoders: https://bugzilla.gnome.org/show_bug.cgi?id=757941

Finally, I haven't tested MVC encoding personally, but grepping the source
code the way how MVC encoding is selected is if the number of views in
upstream is equal or bigger than 2.


vmjl

On 12/28/16 at 12:32pm, Jérôme Carretero wrote:

> Hi,
>
>
>
> I'm running out of google-fu to find documentation on how it's possible
> to try and use h264 encoding with MVC profiles.
>
>
> I am under the impression that the following should work to encode
> side-by-side stereoscopic video:
>
> ... ! vaapih264enc ! video/x-h264,profile=stereo-high ! h264parse ! ...
>
> But the resulting videos don't seem to be in the right profile.
>
>
> Then, there is MVC, and I really had trouble finding *anything* on that.
>
>
>
> Best regards,

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Jérôme Carretero]

Hi Víctor,


(adding Jan because I saw
https://gstreamer.freedesktop.org/data/events/gstreamer-conference/2015/Jan%20Schmidt%20-%20Stereoscopic%203D%20Redux.pdf)


Yes, the hardware supports MVC encoding (intel SkyLake).
Definitely the fact that the downstream caps weren't used wasn't
helping there.

After some further Googling based on your hint “MVC encoding is selected
is if the number of views in upstream is equal or bigger than 2”, I
found in the glstereomix docs the "multiview-mode=side-by-side" which
is added at the output of glstereomix.


Because I already have mixed video in one of my cases (top-bottom or
left-right), so I tried:

  ! ... \
  ! video/x-raw,format=I420,multiview-mode=side-by-side \
  ! vaapih264enc \
  ! h264parse ! ...

But to no avail.


Is the StereoHigh encoder expecting these side-by-side buffers, and I'm
missing "metadata", or does it prefer to use separate buffers?


To "reproduce" a similar situation:

# Get a random SBS video on youtube, I believe it's clearly fair use

youtube-dl EgD5_7u2NvQ
mv *EgD5_7u2NvQ*.mp4 sbs.mp4

# Attempt to recompress using StereoHigh

gst-launch-1.0 --verbose --no-fault \
 filesrc location=sbs.mp4 \
 ! qtdemux name=demux \
 demux.audio_0 ! queue ! fakesink \
 demux.video_0 \
 ! queue name=q_dec\
  ! h264parse \
  ! avdec_h264 \
  ! vaapipostproc \
 ! queue name=q_enc \
  ! video/x-raw,format=I420,multiview-mode=side-by-side \
  ! vaapih264enc \
  ! h264parse \
  ! mp4mux fragment-duration=1000 \
  ! filesink location=test-stereohigh.mp4

GST_DEBUG=GST_CAPS:5 gst-launch-1.0 \
 filesrc location=test-stereohigh.mp4 ! qtdemux ! h264parse ! fakesink \
 2>&1 | grep stream-format | head -n 1

# video/x-h264, parsed=(boolean)true, stream-format=(string){ avc, avc3,
# byte-stream }, alignment=(string){ au, nal }


Regards,

--
Jérôme

On Wed, 28 Dec 2016 19:38:22 +0100 Víctor M. Jáquez L.
<[hidden email]> wrote:

> Hi,
>
> First thing to check is does your hardware support MVC?
>
> You can know that by running `vainfo`
>
> And you should see
>
>       VAProfileH264MultiviewHigh      : VAEntrypointEncSlice
>       VAProfileH264StereoHigh         : VAEntrypointEncSlice
>
>
> Second thing is that setting the profile via capsfilters is not
> supported yet by the vaapi encoders:
> https://bugzilla.gnome.org/show_bug.cgi?id=757941
>
> Finally, I haven't tested MVC encoding personally, but grepping the
> source code the way how MVC encoding is selected is if the number of
> views in upstream is equal or bigger than 2.
>
>
> vmjl
>
> On 12/28/16 at 12:32pm, Jérôme Carretero wrote:
> > Hi,
> >
> >
> >
> > I'm running out of google-fu to find documentation on how it's
> > possible to try and use h264 encoding with MVC profiles.
> >
> >
> > I am under the impression that the following should work to encode
> > side-by-side stereoscopic video:
> >
> > ... ! vaapih264enc ! video/x-h264,profile=stereo-high !
> > h264parse ! ...
> >
> > But the resulting videos don't seem to be in the right profile.
> >
> >
> > Then, there is MVC, and I really had trouble finding *anything* on
> > that.
> >
> >
> >
> > Best regards,  

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Jérôme Carretero]

Hi,


Did not get a reply... is there any documentation available or I should
dig in the code?


Thanks,

--
cJ


On Wed, 28 Dec 2016 17:49:37 -0500
Jérôme Carretero <[hidden email]> wrote:

> Hi Víctor,
>
>
> (adding Jan because I saw
> https://gstreamer.freedesktop.org/data/events/gstreamer-conference/2015/Jan%20Schmidt%20-%20Stereoscopic%203D%20Redux.pdf)
>
>
> Yes, the hardware supports MVC encoding (intel SkyLake).
> Definitely the fact that the downstream caps weren't used wasn't
> helping there.
>
> After some further Googling based on your hint “MVC encoding is
> selected is if the number of views in upstream is equal or bigger
> than 2”, I found in the glstereomix docs the
> "multiview-mode=side-by-side" which is added at the output of
> glstereomix.
>
>
> Because I already have mixed video in one of my cases (top-bottom or
> left-right), so I tried:
>
>   ! ... \
>   ! video/x-raw,format=I420,multiview-mode=side-by-side \
>   ! vaapih264enc \
>   ! h264parse ! ...
>
> But to no avail.
>
>
> Is the StereoHigh encoder expecting these side-by-side buffers, and
> I'm missing "metadata", or does it prefer to use separate buffers?
>
>
> To "reproduce" a similar situation:
>
> # Get a random SBS video on youtube, I believe it's clearly fair use
>
> youtube-dl EgD5_7u2NvQ
> mv *EgD5_7u2NvQ*.mp4 sbs.mp4
>
> # Attempt to recompress using StereoHigh
>
> gst-launch-1.0 --verbose --no-fault \
>  filesrc location=sbs.mp4 \
>  ! qtdemux name=demux \
>  demux.audio_0 ! queue ! fakesink \
>  demux.video_0 \
>  ! queue name=q_dec\
>   ! h264parse \
>   ! avdec_h264 \
>   ! vaapipostproc \
>  ! queue name=q_enc \
>   ! video/x-raw,format=I420,multiview-mode=side-by-side \
>   ! vaapih264enc \
>   ! h264parse \
>   ! mp4mux fragment-duration=1000 \
>   ! filesink location=test-stereohigh.mp4
>
> GST_DEBUG=GST_CAPS:5 gst-launch-1.0 \
>  filesrc location=test-stereohigh.mp4 ! qtdemux ! h264parse !
> fakesink \
>  2>&1 | grep stream-format | head -n 1  
>
> # video/x-h264, parsed=(boolean)true, stream-format=(string){ avc,
> avc3, # byte-stream }, alignment=(string){ au, nal }
>
>
> Regards,
>

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[sreerenj]

set "num-views=2" to vaapih264enc

On Thu, Jan 26, 2017 at 6:22 PM, Jérôme Carretero <[hidden email]> wrote:

Hi,


Did not get a reply... is there any documentation available or I should
dig in the code?


Thanks,

--
cJ

On Wed, 28 Dec 2016 17:49:37 -0500
Jérôme Carretero <[hidden email]> wrote:

> Hi Víctor,
>
>
> (adding Jan because I saw
> https://gstreamer.freedesktop.org/data/events/gstreamer-conference/2015/Jan%20Schmidt%20-%20Stereoscopic%203D%20Redux.pdf)
>
>
> Yes, the hardware supports MVC encoding (intel SkyLake).
> Definitely the fact that the downstream caps weren't used wasn't
> helping there.
>
> After some further Googling based on your hint “MVC encoding is
> selected is if the number of views in upstream is equal or bigger
> than 2”, I found in the glstereomix docs the
> "multiview-mode=side-by-side" which is added at the output of
> glstereomix.
>
>
> Because I already have mixed video in one of my cases (top-bottom or
> left-right), so I tried:
>
>   ! ... \
>   ! video/x-raw,format=I420,multiview-mode=side-by-side \
>   ! vaapih264enc \
>   ! h264parse ! ...
>
> But to no avail.
>
>
> Is the StereoHigh encoder expecting these side-by-side buffers, and
> I'm missing "metadata", or does it prefer to use separate buffers?
>
>
> To "reproduce" a similar situation:
>
> # Get a random SBS video on youtube, I believe it's clearly fair use
>
> youtube-dl EgD5_7u2NvQ
> mv *EgD5_7u2NvQ*.mp4 sbs.mp4
>
> # Attempt to recompress using StereoHigh
>
> gst-launch-1.0 --verbose --no-fault \
>  filesrc location=sbs.mp4 \
>  ! qtdemux name=demux \
>  demux.audio_0 ! queue ! fakesink \
>  demux.video_0 \
>  ! queue name=q_dec\
>   ! h264parse \
>   ! avdec_h264 \
>   ! vaapipostproc \
>  ! queue name=q_enc \
>   ! video/x-raw,format=I420,multiview-mode=side-by-side \
>   ! vaapih264enc \
>   ! h264parse \
>   ! mp4mux fragment-duration=1000 \
>   ! filesink location=test-stereohigh.mp4
>
> GST_DEBUG=GST_CAPS:5 gst-launch-1.0 \
>  filesrc location=test-stereohigh.mp4 ! qtdemux ! h264parse !
> fakesink \
>  2>&1 | grep stream-format | head -n 1
>
> # video/x-h264, parsed=(boolean)true, stream-format=(string){ avc,
> avc3, # byte-stream }, alignment=(string){ au, nal }
>
>
> Regards,
>

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Victor Jaquez]

On 01/26/17 at 06:25pm, Sreerenj Balachandran wrote:

> set "num-views=2" to vaapih264enc
>
> On Thu, Jan 26, 2017 at 6:22 PM, Jérôme Carretero <[hidden email]>
> wrote:
>
> > Hi,
> >
> >
> > Did not get a reply... is there any documentation available or I should
> > dig in the code?

Oops! Sorry, this thread slip out of my head.

As Sree already replied, with the property num-views you should set MVC
encoding.

Could you comment this issue in bug

https://bugzilla.gnome.org/show_bug.cgi?id=757941 ???

To keep it in mind.

Thanks

vmjl

> >
> >
> > Thanks,
> >
> > --
> > cJ
> >
> >
> > On Wed, 28 Dec 2016 17:49:37 -0500
> > Jérôme Carretero <[hidden email]> wrote:
> >
> > > Hi Víctor,
> > >
> > >
> > > (adding Jan because I saw
> > > https://gstreamer.freedesktop.org/data/events/gstreamer-
> > conference/2015/Jan%20Schmidt%20-%20Stereoscopic%203D%20Redux.pdf)
> > >
> > >
> > > Yes, the hardware supports MVC encoding (intel SkyLake).
> > > Definitely the fact that the downstream caps weren't used wasn't
> > > helping there.
> > >
> > > After some further Googling based on your hint “MVC encoding is
> > > selected is if the number of views in upstream is equal or bigger
> > > than 2”, I found in the glstereomix docs the
> > > "multiview-mode=side-by-side" which is added at the output of
> > > glstereomix.
> > >
> > >
> > > Because I already have mixed video in one of my cases (top-bottom or
> > > left-right), so I tried:
> > >
> > >   ! ... \
> > >   ! video/x-raw,format=I420,multiview-mode=side-by-side \
> > >   ! vaapih264enc \
> > >   ! h264parse ! ...
> > >
> > > But to no avail.
> > >
> > >
> > > Is the StereoHigh encoder expecting these side-by-side buffers, and
> > > I'm missing "metadata", or does it prefer to use separate buffers?
> > >
> > >
> > > To "reproduce" a similar situation:
> > >
> > > # Get a random SBS video on youtube, I believe it's clearly fair use
> > >
> > > youtube-dl EgD5_7u2NvQ
> > > mv *EgD5_7u2NvQ*.mp4 sbs.mp4
> > >
> > > # Attempt to recompress using StereoHigh
> > >
> > > gst-launch-1.0 --verbose --no-fault \
> > >  filesrc location=sbs.mp4 \
> > >  ! qtdemux name=demux \
> > >  demux.audio_0 ! queue ! fakesink \
> > >  demux.video_0 \
> > >  ! queue name=q_dec\
> > >   ! h264parse \
> > >   ! avdec_h264 \
> > >   ! vaapipostproc \
> > >  ! queue name=q_enc \
> > >   ! video/x-raw,format=I420,multiview-mode=side-by-side \
> > >   ! vaapih264enc \
> > >   ! h264parse \
> > >   ! mp4mux fragment-duration=1000 \
> > >   ! filesink location=test-stereohigh.mp4
> > >
> > > GST_DEBUG=GST_CAPS:5 gst-launch-1.0 \
> > >  filesrc location=test-stereohigh.mp4 ! qtdemux ! h264parse !
> > > fakesink \
> > >  2>&1 | grep stream-format | head -n 1
> > >
> > > # video/x-h264, parsed=(boolean)true, stream-format=(string){ avc,
> > > avc3, # byte-stream }, alignment=(string){ au, nal }
> > >
> > >
> > > Regards,
> > >
> >
> > _______________________________________________
> > gstreamer-devel mailing list
> > [hidden email]
> > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> >

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Jérôme Carretero]

Hi,


I will comment on the bug, trying to get further into encoding, first:
from your answers, progress was made but there still something that's
not right.


Reference encodings with vaapi and x264 (I am generating a test video
that I assume has stacked side-by-side left-right views in the same
frame buffer):

gst-launch-1.0 --verbose --no-fault \
 videotestsrc num-buffers=60 \
 ! video/x-raw,format=I420,width=2048,height=1024,framerate=30/1 \
 ! vaapih264enc rate-control=cbr bitrate=2000 \
 ! h264parse \
 ! matroskamux streamable=true \
 ! filesink location=tmp-va.mkv

gst-launch-1.0 --verbose --no-fault \
 videotestsrc num-buffers=60 \
 ! video/x-raw,format=I420,width=2048,height=1024,framerate=30/1 \
 ! queue max-size-time=10000000000 \
 ! x264enc bitrate=2000 \
 ! h264parse \
 ! matroskamux streamable=true \
 ! filesink location=tmp-x264.mkv


Trying to use num-views=2 with vaapi:


gst-launch-1.0 --verbose --no-fault \
 videotestsrc num-buffers=60 \
 ! video/x-raw,format=I420,width=2048,height=1024,framerate=30/1,multiview-mode=side-by-side \
 ! vaapih264enc rate-control=cbr bitrate=2000 num-views=2 \
 ! h264parse \
 ! matroskamux streamable=true \
 ! filesink location=tmp-va-sbs.mkv


mediainfo tmp-va-sbs.mkv says:

MultiView_Count                          : 2
MultiView_Layout                         : Right Eye

(not sure why "Right Eye" is said):

During playback, the image has an aspect ratio of 4:1 instead of 2:1:

Input #0, matroska,webm, from 'tmp-va-mvc.mkv':
  Metadata:
    encoder         : GStreamer matroskamux version 1.10.2
    creation_time   : 2017-01-28T19:01:48.000000Z
  Duration: N/A, start: 0.033000, bitrate: N/A
    Stream #0:0(eng): Video: h264 (High), yuv420p(progressive), 2048x1024 [SAR 1:1 DAR 2:1], SAR 2:1 DAR 4:1, 30 fps, 30 tbr, 1k tbn, 60 tbc (default)
    Metadata:
      title           : Video
      stereo_mode     : left_right
    Side data:
      stereo3d: side by side


And there are lots of:

Error while decoding frame!
[NULL @ 0x7fa732aa7b60]missing picture in access unit with size 16898
[h264 @ 0x7fa732aa7b60]no frame!



Then I tried using x264 to see what happens:

gst-launch-1.0 --verbose --no-fault \
 videotestsrc num-buffers=60 \
 ! video/x-raw,format=I420,width=2048,height=1024,framerate=30/1,multiview-mode=side-by-side \
 ! queue max-size-time=10000000000 \
 ! x264enc bitrate=2000 \
 ! h264parse \
 ! matroskamux streamable=true \
 ! filesink location=tmp-x264-sbs.mkv

Same mediainfo, same unexpected aspect ratio, but no playback warnings.



Best regards,

--
Jérôme


On Fri, 27 Jan 2017 09:39:35 +0100
Víctor M. Jáquez L. <[hidden email]> wrote:

> On 01/26/17 at 06:25pm, Sreerenj Balachandran wrote:
> > set "num-views=2" to vaapih264enc
> >
> > On Thu, Jan 26, 2017 at 6:22 PM, Jérôme Carretero
> > <[hidden email]> wrote:
> >  
> > > Hi,
> > >
> > >
> > > Did not get a reply... is there any documentation available or I
> > > should dig in the code?  
>
> Oops! Sorry, this thread slip out of my head.
>
> As Sree already replied, with the property num-views you should set
> MVC encoding.
>
> Could you comment this issue in bug
>
> https://bugzilla.gnome.org/show_bug.cgi?id=757941 ???
>
> To keep it in mind.
>
> Thanks
>
> vmjl
>
> > >
> > >
> > > Thanks,
> > >
> > > --
> > > cJ
> > >
> > >
> > > On Wed, 28 Dec 2016 17:49:37 -0500
> > > Jérôme Carretero <[hidden email]> wrote:
> > >  
> > > > Hi Víctor,
> > > >
> > > >
> > > > (adding Jan because I saw
> > > > https://gstreamer.freedesktop.org/data/events/gstreamer- 
> > > conference/2015/Jan%20Schmidt%20-%20Stereoscopic%203D%20Redux.pdf)  
> > > >
> > > >
> > > > Yes, the hardware supports MVC encoding (intel SkyLake).
> > > > Definitely the fact that the downstream caps weren't used wasn't
> > > > helping there.
> > > >
> > > > After some further Googling based on your hint “MVC encoding is
> > > > selected is if the number of views in upstream is equal or
> > > > bigger than 2”, I found in the glstereomix docs the
> > > > "multiview-mode=side-by-side" which is added at the output of
> > > > glstereomix.
> > > >
> > > >
> > > > Because I already have mixed video in one of my cases
> > > > (top-bottom or left-right), so I tried:
> > > >
> > > >   ! ... \
> > > >   ! video/x-raw,format=I420,multiview-mode=side-by-side \
> > > >   ! vaapih264enc \
> > > >   ! h264parse ! ...
> > > >
> > > > But to no avail.
> > > >
> > > >
> > > > Is the StereoHigh encoder expecting these side-by-side buffers,
> > > > and I'm missing "metadata", or does it prefer to use separate
> > > > buffers?
> > > >
> > > >
> > > > To "reproduce" a similar situation:
> > > >
> > > > # Get a random SBS video on youtube, I believe it's clearly
> > > > fair use
> > > >
> > > > youtube-dl EgD5_7u2NvQ
> > > > mv *EgD5_7u2NvQ*.mp4 sbs.mp4
> > > >
> > > > # Attempt to recompress using StereoHigh
> > > >
> > > > gst-launch-1.0 --verbose --no-fault \
> > > >  filesrc location=sbs.mp4 \
> > > >  ! qtdemux name=demux \
> > > >  demux.audio_0 ! queue ! fakesink \
> > > >  demux.video_0 \
> > > >  ! queue name=q_dec\
> > > >   ! h264parse \
> > > >   ! avdec_h264 \
> > > >   ! vaapipostproc \
> > > >  ! queue name=q_enc \
> > > >   ! video/x-raw,format=I420,multiview-mode=side-by-side \
> > > >   ! vaapih264enc \
> > > >   ! h264parse \
> > > >   ! mp4mux fragment-duration=1000 \
> > > >   ! filesink location=test-stereohigh.mp4
> > > >
> > > > GST_DEBUG=GST_CAPS:5 gst-launch-1.0 \
> > > >  filesrc location=test-stereohigh.mp4 ! qtdemux ! h264parse !
> > > > fakesink \  
> > > >  2>&1 | grep stream-format | head -n 1  
> > > >
> > > > # video/x-h264, parsed=(boolean)true,
> > > > stream-format=(string){ avc, avc3, # byte-stream },
> > > > alignment=(string){ au, nal }
> > > >
> > > >
> > > > Regards,
> > > >  
> > >
> > > _______________________________________________
> > > gstreamer-devel mailing list
> > > [hidden email]
> > > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> > >  
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel