Saving a live stream

[Hayden Andrews]

Hi guys,

We have a live A/V Ogg stream that runs for a couple of hours, but we
want to be able to save an (approximately) 40 min section of this stream
to a file.

First of all, I wrote a quick Perl script which reads directly from the
appropriate HTTP port (an Icecast Server) and writes to a file.
Everything looks fine with this file except ..
  1) Suppose we record from 40 mins to 1:20 of the live stream. When we
play the file, the current time starts at the time we started recording
from rather than from 00:00. Although the run time is correct (40 mins),
the current time progresses from 40mins to 1:20.
  2) When someone tries to edit the file to trim each end and make any
other minor changes that are required, seeking takes forever and the
editor takes for ever to do things that would normally be quite quick
with a normal file. I believe that if the file has a play time of about
40 mins, then the player says that it will take about 40 mins to do many
tasks.

I figure that since a live stream has an unknown length, and since it is
not possible to seek when viewing a live stream, that there is missing
info in the live stream that needs to be inserted while saving. So, I
also figure that it would be easy to insert the relevant requirements
while saving a live stream to file.....

So, I edited my script to use Gstreamer instead. I have used souphttpsrc
! filesink and I've used souphttpsrc ! decodebin ! ...... ! oggmux !
filesink. I hoped that the filesink would expect to insert [what ever is
needed] and would do [something] automatically. But we still get files
which are difficult to edit. So, I tried setting do-timestamp=true in
southttpsrc which produced a file, which when played by vlc, showed
either a very short (2-3 seconds) or unknown run time, and crashed the
player when I tried seeking within the file being played.

So, what should I be doing? Any ideas.... I expect that this is a simple
and normal scenario as I'm sure that it is more than normal to be saving
portions of live streams to file for later editing/playing.

Cheers,


Hayden

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Arnout Vandecappelle]

On Saturday 24 January 2009 09:58:42 Hayden Andrews wrote:
> First of all, I wrote a quick Perl script which reads directly from the
> appropriate HTTP port (an Icecast Server) and writes to a file.
> Everything looks fine with this file except ..
>   1) Suppose we record from 40 mins to 1:20 of the live stream. When we
> play the file, the current time starts at the time we started recording
> from rather than from 00:00. Although the run time is correct (40 mins),
> the current time progresses from 40mins to 1:20.

 That's because the buffer timestamps are retained.  You have to re-stamp the
buffers.  I ended up writing an element to do exactly that:
http://bugzilla.gnome.org/show_bug.cgi?id=561224 - that bugzilla entry also
contains a remark on how it should be done by setting base_time, but I don't
think that will work.  As discussed earlier on the list (see
http://thread.gmane.org/gmane.comp.video.gstreamer.devel/22806 ), you can
also try an identity with single-segment=true, although I also doubt that
will work :-).

 In any case, you'll need to do demuxing and remuxing to get the timestamps in
your output file.  So your pipeline will look like this:

filesrc location=... ! oggdemux ! identity single-segment=true \
  ! oggmux ! filesink location=...


 Of course, you'll encounter additional problems with audio: you'll need to
split the pipeline and add queue's, and I'm pretty sure something will go
wrong with the synchronisation between the timestamps of the audio and video
channel.

>   2) When someone tries to edit the file to trim each end and make any
> other minor changes that are required, seeking takes forever and the
> editor takes for ever to do things that would normally be quite quick
> with a normal file. I believe that if the file has a play time of about
> 40 mins, then the player says that it will take about 40 mins to do many
> tasks.
>
> I figure that since a live stream has an unknown length, and since it is
> not possible to seek when viewing a live stream, that there is missing
> info in the live stream that needs to be inserted while saving. So, I
> also figure that it would be easy to insert the relevant requirements
> while saving a live stream to file.....

 Indeed, the output file doesn't contain an index, which means the editor
needs to scan the entire file in order to seek correctly.  The index is only
written when the muxer receives an EOS event.  Unfortunately, as far as I
know, there is no way to force an EOS on a stream started with gst-launch.  
How do you select the part you want using GStreamer anyway?

 It may pay off to take a look at gentrans: http://gentrans.sourceforge.net/

 Regards,
 Arnout

--
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 [phone]
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  D206 D44B 5155 DF98 550D  3F2A 2213 88AA A1C7 C933

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Hayden Andrews]

Arnout Vandecappelle wrote:

On Saturday 24 January 2009 09:58:42 Hayden Andrews wrote:
  

First of all, I wrote a quick Perl script which reads directly from the
appropriate HTTP port (an Icecast Server) and writes to a file.
Everything looks fine with this file except ..
  1) Suppose we record from 40 mins to 1:20 of the live stream. When we
play the file, the current time starts at the time we started recording
from rather than from 00:00. Although the run time is correct (40 mins),
the current time progresses from 40mins to 1:20.
    

 That's because the buffer timestamps are retained.  You have to re-stamp the 
buffers.  I ended up writing an element to do exactly that: 
http://bugzilla.gnome.org/show_bug.cgi?id=561224 - that bugzilla entry also 
contains a remark on how it should be done by setting base_time, but I don't 
think that will work.  As discussed earlier on the list (see 
http://thread.gmane.org/gmane.comp.video.gstreamer.devel/22806 ), you can 
also try an identity with single-segment=true, although I also doubt that 
will work :-).

 In any case, you'll need to do demuxing and remuxing to get the timestamps in 
your output file.  So your pipeline will look like this:

filesrc location=... ! oggdemux ! identity single-segment=true \
  ! oggmux ! filesink location=...


 Of course, you'll encounter additional problems with audio: you'll need to 
split the pipeline and add queue's, and I'm pretty sure something will go 
wrong with the synchronisation between the timestamps of the audio and video 
channel.
  

I'm going to try the 'identity' idea as your element is not yet in the build of GStreamer that is on the production machine.
I'm not filled with confidence when you suggest that something else is bound to go wrong! Is this not a fairly standard scenario? We are just trying to save portion of a live stream to a file that can then be edited easily, and played with reasonable timings. Anyway, I will try the new script with an identity element for each of the audio/video streams.

I figure that since a live stream has an unknown length, and since it is
not possible to seek when viewing a live stream, that there is missing
info in the live stream that needs to be inserted while saving. So, I
also figure that it would be easy to insert the relevant requirements
while saving a live stream to file.....
    

 Indeed, the output file doesn't contain an index, which means the editor 
needs to scan the entire file in order to seek correctly.  The index is only 
written when the muxer receives an EOS event.  Unfortunately, as far as I 
know, there is no way to force an EOS on a stream started with gst-launch.  
How do you select the part you want using GStreamer anyway?
  

I'm using the CPAN GStreamer module, not gst-launch. We are currently kicking off the script via a web page. So, the person, who is videoing/streaming the event, at the require time, hits 'record stream' on a web page which kicks off the Perl script. When they want to stop recording, they press 'Stop Recording' which sends a signal to the Perl Script. When the script receives the signal, it calls quit() on the Glib::MainLoop object. I guess that I should be sending this EOS to the pipe before calling quit() ??? How do I go about doing that?

Thanks heaps for your help, so far, Arnout.

Hayden

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Arnout Vandecappelle]

On Friday 30 January 2009 11:02:07 Hayden Andrews wrote:
> I'm going to try the 'identity' idea as your element is not yet in the
> build of GStreamer that is on the production machine.

 I doubt it ever will be since there are no sponsors for it.

> I'm not filled with confidence when you suggest that something else is
> bound to go wrong! Is this not a fairly standard scenario?
[snip]

 Although I never actually tried it, I think the identity single-segment thing
will do want you want automatically if you select your chunk using a seek
event.  The seek event will be interpreted by the demuxer, and I believe it
will send a synchronous newsegment on all its src pads.


> I guess that I should be sending this EOS to the
> pipe before calling quit() ??? How do I go about doing that?

 I don't know how to do it in perl, but it's probably similar to the way you
do it in C.

 You have to push an EOS event to an element or a pad upstream from the mixer.  
You can choose any element, but probably the source element (souphttpsrc?) is
the easiest.  In C it would be something like:

    gst_pad_send_event(srcelement, gst_event_new_eos());

 Actually, it's possible that just setting the source element state to NULL is
sufficient to trigger the EOS.  You anyway have to set the source element
state to NULL, otherwise it will keep on pushing buffers after the EOS, which
is not allowed.

 After sending the EOS, you can wait for an EOS message to appear on the bus.  
This message indicates that all sinks have recieved an EOS.  After that you
can tear down the pipeline.

 Regards,
 Arnout

--
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 [phone]
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  D206 D44B 5155 DF98 550D  3F2A 2213 88AA A1C7 C933

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Hayden Andrews]

Hi Arnout,

Thanks heaps again for your reply. I tried the 'identity' idea at the end of last week, and the test was apparently very successful .... except that the person who tested it stopped the stream source before stopping the saveStream script, so the script exited on it's own accord because of the received EOS. I have only just this morning updated the script, and it will hopefully be tested in the next few days.

Now, when the Perl script receives a signal to exit, it blocks the src pad of the souphttpsrc element, and then sends an EOS signal to the next element in the pipe. The script then ends on it's own accord upon seeing the EOS signal on the bus.

Thanks for your help! Much appreciated! Hopefully this test will produce what we want! :)

H

Arnout Vandecappelle wrote:

On Friday 30 January 2009 11:02:07 Hayden Andrews wrote:
  

I'm going to try the 'identity' idea as your element is not yet in the
build of GStreamer that is on the production machine.
    

 I doubt it ever will be since there are no sponsors for it.

  

I'm not filled with confidence when you suggest that something else is
bound to go wrong! Is this not a fairly standard scenario?
    

[snip]

 Although I never actually tried it, I think the identity single-segment thing 
will do want you want automatically if you select your chunk using a seek 
event.  The seek event will be interpreted by the demuxer, and I believe it 
will send a synchronous newsegment on all its src pads.


  

I guess that I should be sending this EOS to the
pipe before calling quit() ??? How do I go about doing that?
    

 I don't know how to do it in perl, but it's probably similar to the way you 
do it in C.

 You have to push an EOS event to an element or a pad upstream from the mixer.  
You can choose any element, but probably the source element (souphttpsrc?) is 
the easiest.  In C it would be something like:

    gst_pad_send_event(srcelement, gst_event_new_eos());

 Actually, it's possible that just setting the source element state to NULL is 
sufficient to trigger the EOS.  You anyway have to set the source element 
state to NULL, otherwise it will keep on pushing buffers after the EOS, which 
is not allowed.

 After sending the EOS, you can wait for an EOS message to appear on the bus.  
This message indicates that all sinks have recieved an EOS.  After that you 
can tear down the pipeline.

 Regards,
 Arnout

  

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel