gnonlin composition - filling queues

[Olivier Aubert-2]

Hello again

Now that I know how to extract a single A+V segment from a file, I want
to extract multiple ones. The context is a video annotation application
(http://www.advene.org/), where I can define annotations (start
timecode, end timecode, some content) on video files. From the
annotations, I can get multiple visualisations, one of them being a
virtual montage of interesting annotations. I already have an
event-based interactive version, which does not use gnonlin.
Now, I want to be able to save it to a file, and that is where I am
stuck. The following code takes a list of annotations (segments
basically, position and duration information) and generates a
gnlcomposition from them: two gnlfilesources (audio, video) for each
annotation.

The problem is that the pipeline blocks because the vmuxqueue is full
(the current-level-time is higher than max-size-time (1s) ). Setting the
queues to leaky=1 is not satisfactory, it just moves the issue
elsewhere.

What triggers the queue filling is the change of gnlfilesource
(incidentally, the code works with 1 annotation only), so my current
analysis is that the timestamp discontinuity bothers the queue, even
though I have put identity + videorate elements to try to fix the
timestamp issues.

I join the code + gst-debug messages below. Thanks in advance to anyone
who has some advice to share here.

Regards,
Olivier

--- montage renderer code ---

CAPS_VIDEO_STRING = 'video/x-raw-yuv'
CAPS_VIDEO = gst.caps_from_string(CAPS_VIDEO_STRING)
CAPS_AUDIO_STRING = 'audio/x-raw-int;audio/x-raw-float'
CAPS_AUDIO = gst.caps_from_string(CAPS_AUDIO_STRING)

class MontageRenderer(object):
    def __init__(self, controller, elements=None):
        self.controller = controller
        # self.elements is a list of annotations
        if elements is None:
            elements = []
        self.elements = elements
        self.encoding_pipe = None

    def render(self, outputfile, progress_callback = None):
        sourcefile = self.controller.get_default_media()
        if not sourcefile:
            return
        pipedef = "gnlcomposition name=videocomp caps=%s ! queue ! progressreport name=progress silent=true update-freq=1 ! identity single-segment=true ! ffmpegcolorspace ! videorate ! theoraenc ! queue name=vmuxqueue ! oggmux name=mux ! filesink name=sink gnlcomposition name=audiocomp caps=%s ! queue ! identity single-segment=true ! audioconvert ! audiorate ! vorbisenc ! queue name=amuxqueue ! mux." % (CAPS_VIDEO_STRING, CAPS_AUDIO_STRING)
        pipe = gst.parse_launch(pipedef)
        videocomp = pipe.get_by_name('videocomp')
        audiocomp = pipe.get_by_name('audiocomp')
        sink = pipe.get_by_name('sink')
        bus = pipe.get_bus()
        bus.enable_sync_message_emission()

        def filesource(a, pos, caps):
            """Create a filesource.
            """
            e = gst.element_factory_make('gnlfilesource')
            e.set_property("location", sourcefile)
            e.set_property("caps", caps)
            e.set_property("start", pos * gst.MSECOND)
            e.set_property("duration", a.fragment.duration * gst.MSECOND)
            e.set_property("media-start", a.fragment.begin * gst.MSECOND)
            e.set_property("media-duration", a.fragment.duration * gst.MSECOND)
            return e

        pos = 0
        for a in self.elements:
            e = filesource(a, pos, CAPS_VIDEO)
            videocomp.add(e)
            e = filesource(a, pos, CAPS_AUDIO)
            audiocomp.add(e)
            pos += a.fragment.duration

        sink.set_property("location", outputfile)

        def on_bus_message(bus, message):
            if message.type == gst.MESSAGE_STATE_CHANGED:
                old, new, pending = message.parse_state_changed()
                print "STATE", old.value_nick, new.value_nick, pending.value_nick
            elif message.type == gst.MESSAGE_EOS:
                print " EOS"
                pipe.set_state(gst.STATE_NULL)
                progress_callback(None)
            elif message.structure:
                s=message.structure
                print "MSG " + bus.get_name() + ": " + s.to_string()
                if s.get_name() == 'progress' and progress_callback is not None:
                    progress_callback(s['percent-double'] / 100)
            return True

        bus.add_signal_watch()
        bus.connect('message', on_bus_message)
        pipe.set_state(gst.STATE_PLAYING)
        self.encoding_pipe = pipe
        return True

    def finalize(self):
        if self.encoding_pipe is not None:
            self.encoding_pipe.set_state(gst.STATE_NULL)
            self.encoding_pipe = None


--- GST_DEBUG output ---
0:00:30.433191931  1064 0x9d883fc0 DEBUG                 ffmpeg gstffmpegdec.c:1764:gst_ffmpegdec_video_frame:<ffdec_mpeg45> picture: display 0
0:00:30.433206807  1064  0xb238738 DEBUG             multiqueue gstmultiqueue.c:1589:single_queue_check_full:<multiqueue5> queue 1: visible 4/5, bytes 3749/2097152, time 182583334/0
0:00:30.433208833  1064 0x9d883fc0 DEBUG                 ffmpeg gstffmpegdec.c:1766:gst_ffmpegdec_video_frame:<ffdec_mpeg45> picture: opaque 0xb18acb8
0:00:30.433225944  1064  0xb238738 DEBUG             multiqueue gstmultiqueue.c:874:apply_buffer:<multiqueue5> queue 1, last_stop updated to 0:00:23.072541666
0:00:30.433228528  1064 0x9d883fc0 DEBUG                 ffmpeg gstffmpegdec.c:1768:gst_ffmpegdec_video_frame:<ffdec_mpeg45> repeat_pict:0
0:00:30.433245710  1064  0xb238738 DEBUG             multiqueue gstmultiqueue.c:797:update_time_level:<multiqueue5> queue 1, sink 0:00:01.872541666, src 0:00:01.637166666
0:00:30.433096246  1064  0xacaede0 DEBUG          basetransform gstbasetransform.c:1399:gst_base_transform_prepare_output_buffer:<ffmpegcsp0> discard buffer, reuse input buffer
0:00:30.433248853  1064 0x9d883fc0 DEBUG                 ffmpeg gstffmpegdec.c:1770:gst_ffmpegdec_video_frame:<ffdec_mpeg45> interlaced_frame:0
0:00:30.434205135  1064  0xb238738 DEBUG               avidemux gstavidemux.c:4522:gst_avi_demux_advance:<avidemux5> moved forwards from 576 to 577
0:00:30.434210443  1064 0x9d883fc0 DEBUG                 ffmpeg gstffmpegdec.c:1792:gst_ffmpegdec_video_frame:<ffdec_mpeg45> timestamp discont, we have DTS as timestamps
0:00:30.434232373  1064 0x9d883fc0 DEBUG                 ffmpeg gstffmpegdec.c:1815:gst_ffmpegdec_video_frame:<ffdec_mpeg45> DTS as timestamps, interpolate
0:00:30.434242011  1064  0xb238738 DEBUG                basesrc gstbasesrc.c:2031:gst_base_src_update_length:<source> reading offset 6688422, length 6, size 79857152, segment.stop -1, maxsize 79857152
0:00:30.434259262  1064  0xb238738 DEBUG                basesrc gstbasesrc.c:2121:gst_base_src_get_range:<source> calling create offset 6688422 length 6, time 0
0:00:30.434259961  1064 0x9d883fc0 DEBUG                  query gstquery.c:359:gst_query_new: creating new query 0xadc34e0 8
0:00:30.434282799  1064  0xb238738 DEBUG                basesrc gstbasesrc.c:1995:gst_base_src_do_sync:<source> no sync needed
0:00:30.434281612  1064 0x9d883fc0 DEBUG               GST_PADS gstpad.c:3674:gst_pad_query:<multiqueue5:src0> sending query 0xadc34e0
0:00:30.434298095  1064  0xb238738 DEBUG                basesrc gstbasesrc.c:2162:gst_base_src_get_range:<source> buffer ok
0:00:30.434302844  1064 0x9d883fc0 DEBUG               GST_PADS gstpad.c:3674:gst_pad_query:<avidemux5:video_00> sending query 0xadc34e0
0:00:30.434314578  1064  0xb238738 DEBUG               avidemux gstavidemux.c:4702:gst_avi_demux_loop_data:<avidemux5> Pushing buffer of size 6, ts 0:00:23.040000000, dur 0:00:00.040000000, off 576, off_end 577
0:00:30.434324076  1064 0x9d883fc0 DEBUG                 ffmpeg gstffmpegdec.c:1952:gst_ffmpegdec_video_frame:<ffdec_mpeg45> return flow 0, out 0xb18acb8, len 24927
0:00:30.434332038  1064  0xb238738 DEBUG             multiqueue gstmultiqueue.c:1589:single_queue_check_full:<multiqueue5> queue 0: visible 4/209, bytes 35683/2097152, time 160000000/0
0:00:30.434344610  1064 0x9d883fc0 DEBUG         queue_dataflow gstqueue.c:963:gst_queue_chain:<queue0> queue is full, waiting for free space
0:00:30.434347473  1064  0xb238738 DEBUG             multiqueue gstmultiqueue.c:874:apply_buffer:<multiqueue5> queue 0, last_stop updated to 0:00:23.080000000
0:00:30.434275396  1064  0xacaede0 DEBUG          basetransform gstbasetransform.c:2036:gst_base_transform_handle_buffer:<ffmpegcsp0> element is in passthrough
0:00:30.434361582  1064  0xb238738 DEBUG             multiqueue gstmultiqueue.c:797:update_time_level:<multiqueue5> queue 0, sink 0:00:01.880000000, src 0:00:01.680000000
0:00:30.434376318  1064  0xb238738 DEBUG               avidemux gstavidemux.c:4522:gst_avi_demux_advance:<avidemux5> moved forwards from 576 to 577
0:00:30.434372826  1064  0xacaede0 DEBUG              videorate gstvideorate.c:696:gst_video_rate_chain:<videorate0> got buffer with timestamp 0:00:07.880000000
0:00:30.434394477  1064  0xb238738 DEBUG                basesrc gstbasesrc.c:2031:gst_base_src_update_length:<source> reading offset 6711828, length 622, size 79857152, segment.stop -1, maxsize 79857152
0:00:30.435127334  1064  0xb238738 DEBUG                basesrc gstbasesrc.c:2121:gst_base_src_get_range:<source> calling create offset 6711828 length 622, time 0
0:00:30.435148216  1064  0xb238738 DEBUG                basesrc gstbasesrc.c:1995:gst_base_src_do_sync:<source> no sync needed
0:00:30.435161626  1064  0xb238738 DEBUG                basesrc gstbasesrc.c:2162:gst_base_src_get_range:<source> buffer ok
0:00:30.435176293  1064  0xb238738 DEBUG               avidemux gstavidemux.c:4702:gst_avi_demux_loop_data:<avidemux5> Pushing buffer of size 622, ts 0:00:23.072541666, dur 0:00:00.025916667, off 18446744073709551615, off_end 18446744073709551615
0:00:30.435195290  1064  0xb238738 DEBUG             multiqueue gstmultiqueue.c:1589:single_queue_check_full:<multiqueue5> queue 1: visible 5/5, bytes 5016/2097152, time 235375000/0
0:00:30.435210935  1064  0xb238738 DEBUG             multiqueue gstmultiqueue.c:1512:single_queue_overrun_cb:<multiqueue5> queue 0: visible 5/209, bytes 35689/2097152, time 200000000/0
0:00:30.435225253  1064  0xb238738 DEBUG             multiqueue gstmultiqueue.c:1512:single_queue_overrun_cb:<multiqueue5> queue 1: visible 5/5, bytes 5016/2097152, time 235375000/0
0:00:30.435239081  1064  0xb238738 DEBUG             multiqueue gstmultiqueue.c:1589:single_queue_check_full:<multiqueue5> queue 1: visible 5/5, bytes 5016/2097152, time 235375000/0
0:00:30.478657663  1064  0xacaede0 DEBUG         queue_dataflow gstqueue.c:963:gst_queue_chain:<vmuxqueue> queue is full, waiting for free space





------------------------------------------------------------------------------
The Palm PDK Hot Apps Program offers developers who use the
Plug-In Development Kit to bring their C/C++ apps to Palm for a share
of $1 Million in cash or HP Products. Visit us here for more details:
http://p.sf.net/sfu/dev2dev-palm
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel