Dynamically updating filesink location at run-time, on the fly

[lsmithso]

I've seen this asked lots of times but have never found a direct
answer. I hope you don't mind my  asking the question  again.

I want to programatically change the file location that filesink
writes to as the program runs. I an saving an audio stream as a series
of short flac files. A new file is written after 30 seconds, or a
period of silence, whichever comes first. The source is any one of
mmssrc, souphttpsrc or pulsesrc. Ideally no src data will be lost
while switching files. The silence detector is either level or cutter
- I'm not sure yet.

I have tried naively setting the filesink location attribute but get
the 'cannot change the open file' error. multifilesink won't help
because it writes to a sequence of fixed length files, and my files
will be of varying lengths. I haven't tried fdsink but I suspect I'll
get the same error as for filesink.


I suspect the answer involved sending an eos and changing the state of
the file sink, and would have to be controlled by bus messages, but
I'm not sure what the sequence of operations should be.


This seems such a simple and obvious things to want to do. Is there a
reason I shouldn't do it, and should the answer to this be in the FAQ?


Thanks for any insights.




--
Les Smithson
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Paddy]

I had much pain in getting this to work for audio & video - for one tho it was quite easy.

Assuming you have a pipeline that looks like this:
audiosrc -->  encoder --> mux --> filesink

then you'll need to change it to:
audiosrc --> encoder --> queue --> muxsink_bin
where muxsink_bin is a bin
ghostpad --> mux --> filesink

then the procedure is:
1 - Block the queue srcpad using gst_pad_set_blocked_async()
2 - In the blocked callback:
2a - unlink muxsink_bin with gst_pad_unlink()
2b - send an EOS event to the muxsink_bin sink pad with gst_pad_send_event()
2b - create a new muxsink_bin
2c - set filesink location
2d - add the new bin to the pipeline with gst_bin_add()
2e - sync with parent using gst_element_sync_state_with_parent()
2f - link it to the queue srcpad with gst_pad_link()
2g - unblock the queue srcpad with gst_pad_set_blocked_async(). When the unblocked callback occurs you're recording again & no data has been lost. No action is required in the unblocked callback

3 - handle the EOS & delete the old muxsink_bin. I had a msg handler that I installed in my bin_init() function using "gstbin_class->handle_message = GST_DEBUG_FUNCPTR(msg_handler)" & in the handler:
3a - lock the bin state with gst_element_set_locked_state()
3b - set the state to NULL with gst_element_set_state()
3c - remove it from the pipeline with gst_bin_remove()

That's it. The only thing to be mindful of is that data must be flowing thru the pipeline for this to work.

Cheers

Paddy

[lsmithso]

Wow! that sounds complicated, but not as complicated as the work
around I came up with. This involved spawning a child process and
writing a continuous pipe of raw data down a fdsink. The child process
read the pipe and wrote the data to file until signalled. At each
signal it created a new pipeline to encode to a new output file.  This
uses the pipe to buffer data while the child is switching files and
encoding. I am not completely sure its not losing data though.

I've never programmed pads or bins before.  I guess blocking a pad
stops it sending data to the next element and it buffers it
instead. While its blocked you can remove and add the next element in
the pipeline then unblock the pad without loss of data. Contrast this
with setting the state of the sink to say PAUSED, which pauses the
whole pipeline.

Is this surmise correct? What is the purpose of the bin here?

I'll certainly give this a go, and thanks for your help.




Paddy writes: > I had much pain
in getting this to work for audio & video - for one tho it > was quite
easy.
 >
 > Assuming you have a pipeline that looks like this:
 > audiosrc -->  encoder --> mux --> filesink
 >
 > then you'll need to change it to:
 > audiosrc --> encoder --> queue --> muxsink_bin
 > where muxsink_bin is a bin
 > ghostpad --> mux --> filesink
 >
 > then the procedure is:
 > 1 - Block the queue srcpad using gst_pad_set_blocked_async()
 > 2 - In the blocked callback:
 > 2a - unlink muxsink_bin with gst_pad_unlink()
 > 2b - send an EOS event to the muxsink_bin sink pad with gst_pad_send_event()
 > 2b - create a new muxsink_bin
 > 2c - set filesink location
 > 2d - add the new bin to the pipeline with gst_bin_add()
 > 2e - sync with parent using gst_element_sync_state_with_parent()
 > 2f - link it to the queue srcpad with gst_pad_link()
 > 2g - unblock the queue srcpad with gst_pad_set_blocked_async(). When the
 > unblocked callback occurs you're recording again & no data has been lost. No
 > action is required in the unblocked callback
 >
 > 3 - handle the EOS & delete the old muxsink_bin. I had a msg handler that I
 > installed in my bin_init() function using "gstbin_class->handle_message =
 > GST_DEBUG_FUNCPTR(msg_handler)" & in the handler:
 > 3a - lock the bin state with gst_element_set_locked_state()
 > 3b - set the state to NULL with gst_element_set_state()
 > 3c - remove it from the pipeline with gst_bin_remove()
 >
 > That's it. The only thing to be mindful of is that data must be flowing thru
 > the pipeline for this to work.
 >
 > Cheers
 >
 > Paddy
 >
 >
 >
 > --
 > View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Dynamically-updating-filesink-location-at-run-time-on-the-fly-tp4660569p4660577.html
 > Sent from the GStreamer-devel mailing list archive at Nabble.com.
 > _______________________________________________
 > gstreamer-devel mailing list
 > [hidden email]
 > http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
Les Smithson

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-3]

Le dimanche 16 juin 2013 à 22:52 +0100, [hidden email] a écrit :

Is this surmise correct? What is the purpose of the bin here?

The procedure described by Paddy is correct. The bin is used to make make it easier to drop the spurious EOS message (GstBin has a handy virtual method). This EOS message is caused by the EOS event send at step 2b. This EOS event is require to properly close the file descriptor and thus ensure file integrity, though the EOS message would cause the application to terminate.

Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Edward Hervey]

Hi,

  Just to point out that multifilesink has multiple ways of deciding
when to split/create a new file. See the next-file property:

===================
  next-file           : When to start a new file
                        flags: readable, writable
                        Enum "GstMultiFileSinkNext" Default: 0, "buffer"
                           (0): buffer           - New file for each
buffer
                           (1): discont          - New file after each
discontinuity
                           (2): key-frame        - New file at each key
frame (Useful for MPEG-TS segmenting)
                           (3): key-unit-event   - New file after a
force key unit event
                           (4): max-size         - New file when the
configured maximum file size would be exceeded with the next buffer or
buffer list
====================

  You could therefore just send a key-unit-event from somewhere in your
pipeline to force the generation of a new file.

   Edward

On Fri, 2013-06-14 at 22:18 +0100, [hidden email] wrote:

> I've seen this asked lots of times but have never found a direct
> answer. I hope you don't mind my  asking the question  again.
>
> I want to programatically change the file location that filesink
> writes to as the program runs. I an saving an audio stream as a series
> of short flac files. A new file is written after 30 seconds, or a
> period of silence, whichever comes first. The source is any one of
> mmssrc, souphttpsrc or pulsesrc. Ideally no src data will be lost
> while switching files. The silence detector is either level or cutter
> - I'm not sure yet.
>
> I have tried naively setting the filesink location attribute but get
> the 'cannot change the open file' error. multifilesink won't help
> because it writes to a sequence of fixed length files, and my files
> will be of varying lengths. I haven't tried fdsink but I suspect I'll
> get the same error as for filesink.
>
>
> I suspect the answer involved sending an eos and changing the state of
> the file sink, and would have to be controlled by bus messages, but
> I'm not sure what the sequence of operations should be.
>
>
> This seems such a simple and obvious things to want to do. Is there a
> reason I shouldn't do it, and should the answer to this be in the FAQ?
>
>
> Thanks for any insights.
>
>
>
>

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Paddy]

I suppose it looks complicated - but only because I've detailed every step in English, in C it would look a lot more concise. In the code there's only 2 more functions to create; the blocked handler & EOS handler - which between them call a dozen or so functions.

Loving your solution btw.

Correct: pad blocking allows elements downstream of the block to be changed without data loss. Have a look at part-block.txt in the docs/design directory. The bin make it easier to create mux->filesink elements & trap the EOS, as Nicolas said.

Edward - I looked at key-unit-event but dropped the idea as I was not sure that this event would not occur naturally in a pipeline. Maybe you know: is a key-unit-event guaranteed not to occur in a pipeline other than programmatically by my app ??

Cheers,

Paddy

[chmario]

Paddy:
I am following your procedure, but  I have some problems with it;
First I am unable to delete the old muxSink , currently i am not deleting the elements, i just create a new musk sink over the old one;

An the oter problem is that the all the files except for the first are corrupted.

I hope you can help me

[shakir]

Hello Edward,

I am trying the key-unit-event approach on my multifilesink branch of tee element, however the files are not playable. I was wondering if you can split files on user input. Broadly I am trying to have the recording functionality parallel to playback

here is the snippet

------------------------------------------------------------------------------------------------------------------------
/* Create the elements */
   source = gst_element_factory_make ("udpsrc", "source");
   tee = gst_element_factory_make("tee", "t");
   mux  = gst_element_factory_make("matroskamux", "mux");
   dpl = gst_element_factory_make("rtph264depay", "dpl");
   parser = gst_element_factory_make("h264parse", "parser");
   valve = gst_element_factory_make("valve", "valve");

   //g_object_get (G_OBJECT (source), "port", &port, NULL);


     sink = gst_element_factory_make ("udpsink", "sink");
     sink2 = gst_element_factory_make ("multifilesink", "sink2");

      //g_object_get (G_OBJECT (sink), "port", &port2, NULL);
      //g_object_get (G_OBJECT (sink), "host", &host, NULL);
      if (!pipeline || !source || !sink) {
         if(!pipeline){
            GST_ERROR ("pipeline could be created.\n");
         }
         if(!source){
             GST_ERROR ("source could be created.\n");
          }
         if(!sink){
              GST_ERROR ("sink could be created.\n");
           }
           return;
        }

      /* Build the pipeline */
      gst_bin_add_many (GST_BIN (pipeline), source, dpl, parser, mux, tee, valve, sink2, sink, NULL);
        if (gst_element_link (source, tee) != TRUE) {
            GST_ERROR ("source, tee could not be linked.\n");
            gst_object_unref (pipeline);
            return;
        }
        if (gst_element_link (tee, valve) != TRUE) {
            GST_ERROR ("source, tee could not be linked.\n");
            gst_object_unref (pipeline);
            return;
        }
        if (gst_element_link (tee, sink) != TRUE) {
            GST_ERROR ("tee, sink could not be linked.\n");
            gst_object_unref (pipeline);
            return;
        }
        if (gst_element_link (valve, dpl) != TRUE) {
            GST_ERROR ("tee, dpl could not be linked.\n");
            gst_object_unref (pipeline);
            return;
        }
        if (gst_element_link (dpl, parser) != TRUE) {
            GST_ERROR ("dpl, parser could not be linked.\n");
            gst_object_unref (pipeline);
            return;
        }
        if (gst_element_link (parser, mux) != TRUE) {
            GST_ERROR ("parser, mux sink could not be linked.\n");
            gst_object_unref (pipeline);
            return;
        }
        if (gst_element_link (mux, sink2) != TRUE) {
            GST_ERROR ("mux, sink2 could not be linked.\n");
            gst_object_unref (pipeline);
            return;
        }
        GST_ERROR ("Elements GOT LINKED.\n");
        g_object_set (valve, "drop", 0, NULL);
        g_object_set (source,"port", 4999,"caps", filtercaps, NULL);
        g_object_set (sink2, "location", pathto, "next-file", 3,  "max-files" , 100, NULL);
.
.
.
static void gst_native_stop_recording (JNIEnv* env, jobject thiz) {

GstEvent* event = gst_video_event_new_downstream_force_key_unit (GST_CLOCK_TIME_NONE,GST_CLOCK_TIME_NONE,GST_CLOCK_TIME_NONE, TRUE, event_num);
  gst_event_ref(event);
  event_num++;
 gboolean tmp = gst_element_send_event(mux, event);

 if(tmp == TRUE){
    GST_DEBUG("WORKED");
    }
 else{
    GST_DEBUG("FAILED");
 }
  gst_event_unref(event);
}
.
.
.
------------------------------------------------------------------------------------------------------------------