Summer Coding: Screen Recording on the Free Desktop

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Summer Coding: Screen Recording on the Free Desktop

cellenro
Posted to kde-devel + gstreamer-devel due to interest
of having both perspectives.

There is a clear need for an integrated screen
recording program that covers recording, editing, and
publishing with a focus on usability.  Current tools
make it difficult compared to other platforms to
record and share knowledge effectively on GNU/Linux.
For example the process the Ubuntu screen cast team
uses
(https://wiki.ubuntu.com/ScreencastTeam/RecordingScreencasts)
is more complex than the situation on proprietary
operating systems with already established screen
recording software.

Ogg Theora is not the best codec for screen recording
because even at higher resolutions it is hard to see
text clearly. Part of the project will be to evaluate
the feasibility of using Driac as the codec for
recording and editing. Below is an outline of the
proposal. I look forward to constructive feedback.


Technology:
        Qt Jambi 4.4 -
(http://trolltech.com/company/newsroom/announcements/press.2008-03-11.2168449860)

        JRuby - jruby.codehaus.org
        GStreamer gstreamer.freedesktop.org
        Schrödinger implementation of Driac (diracvideo.org)

License:
        GPL v3

The language used will be Ruby, with the graphics
framework Qt, and making use of  Schrödinger and
GStreamer components.  The program will be released
using the GPL v3 license, although the current Qt
Jambi preview license may prevent this until Qt 4.4 is
officially released under GPL v3 also.


Format
        support Driac/Schrodinger via GStreamer plugin
        GStreamer for rendering / non-linear video editing --
using Ruby bindings
                - preview -- playbin (like in totem?)
                - video editing -- GNonLin
        http://www.xfree86.org/current/xwininfo.1.html --
looks useful

Based on my research the above technology will be
useful to fully implement the specification of the
screen recording program (found below).  I hope to
gain a better understanding of the optimal way to
create the application with fully open source software
and no patented encumbered codecs.  So if you know who
I should be talking to (or links to additional
resources), please let me know! This will be my first
significant independent open source project and I look
forward to learning from other free software
developers. Based on the technology choices it seems
that the KDE and Gstreamer communities are a good
starting point.


Platforms:
        The application will be developed against Ubuntu 7.10
and created from the start with the intent to be
cross-platform. The screen recording software will
first fully support Linux and then Windows and OS X
(Intel). Windows and the Mac OS already have
professional grade screen capture utilities, although
all platforms have a lack of open source options. This
project aims to change that.


Work flow:
        1.) Select -> 2) Record -> 3) Preview -> 4) Edit ->
5) Render

        Screencasting occurs in five phases.
        First the area to be recorded is selected, then that
area is recorded, after recording the user decides to
keep or retake the video, and then the video is
edited, finally the video is published for consumption
in the desired format.

Development direction:
        The software development will initially focus on the
core of the program, which is selecting and recording
the screen, before implementing other functionality.

        Priority 1
                Selection & screen recording (Driac codec)
        Priority 2
                Audio recording
        Priority 3
                Video / Audio editing


1) Select Area to Record -- illustrative gui img of
record
(http://img338.imageshack.us/img338/2264/34503177vy4.png)


Audio and additional video sources (ex webcam), if
any, chosen by the user.

Two modes of selecting
        0) draw rectangle on screen (8 resize handles, able
to move by dragging in center) . This is similar to
xvidcap except the selection is movable.
       
        1) select window (interactive selection by moving
over window, frame highlights, then click to select).
Once selection of area is made it is refined in either
of two ways.  
A: interacting with the drawn selection grid -
dragging on resize handles, dragging in center to move
(inspired by GIMP rectangle select tool)
B: interacting with the gui widget - typing in new
height/width values, selecting saved height/widths
from past sessions
Note: if window was selected, resizing selection (via
selection grid or gui widget) will cause the window to
resize. The selection grid becomes the resize handler
of the window.
        2) select components of window, for example just want
the canvas of the gimp program -- this enables
automatic detection of area size without having to
precisely draw out a rectangle

2) Record
After selection is made and record button is pressed
then the application goes into recording mode.
Selection grid changes color to denote the new mode.
In recording mode the user is able to pause/resume,
finish,  or cancel the recording
       
3) Preview -- illustrative GUI img of preview
(http://img259.imageshack.us/img259/4373/65638300ql4.png)
After recording is finished the preview mode is
entered. In this mode the user is able to quickly
review the recording just created and decides to save
or delete it.  After choosing, presented with the
option to go into edit mode or back into record mode.

4) Edit
        Edit mode, functionality related to what pitivi
(pitivi.org) attempts to do.
        Must support deleting selected parts of recording,
splicing recording into two recordings at specific
point
        Time line should support reordering of the recordings
        With audio component, time line should have audio
editing functionality integrated.  audio functions
largely similar to jokosher (jokosher.org)

5) Render
        render to desired output format for publishing, ex:
FLV for web video.

Potential future features:
        1. custom xml based format for saving extra data
generated at record time to be useful in editing
automation. (in addition to saving the video)
        2. inserting geometry/text into the video, drawing
attention
        3. webcam / external video integration -- 'picture in
picture'
        4. capturing of opengl ex physics simulations (Phun)
        5. playback faster/slower -- variable speed
        6. capturing vector screenshots of applications (http://lists.freedesktop.org/archives/cairo/2007-April/010256.html)


      ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page.
http://www.yahoo.com/r/hs


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Summer Coding: Screen Recording on the Free Desktop

Thomas Dybdahl Ahle-2
On Tue, 2008-03-25 at 02:08 -0700, cellenro cellenro wrote:
> Ogg Theora is not the best codec for screen recording
> because even at higher resolutions it is hard to see
> text clearly. Part of the project will be to evaluate
> the feasibility of using Driac as the codec for
> recording and editing.

Then what about extending istanbul with driac support?

--
Best Regards,
Med Venlig Hilsen,
Thomas


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Summer Coding: Screen Recording on the Free Desktop

David Schleef
In reply to this post by cellenro
On Tue, Mar 25, 2008 at 02:08:52AM -0700, cellenro cellenro wrote:
> Ogg Theora is not the best codec for screen recording
> because even at higher resolutions it is hard to see
> text clearly. Part of the project will be to evaluate
> the feasibility of using Driac as the codec for
> recording and editing.

Schroedinger isn't really optimal for screencasting, either.  Fixing
this would make a great GSoC project, so I just added it to the
Dirac/Schroedinger ideas page
(http://www.diracvideo.org/wiki/GSoC2008Ideas).  Mainly, someone needs
to add an "encode screencast" mode to schroedinger, and have it make
encoding decisions based on this information.  Some of the intelligence
it needs is:

 - text is not noise that should be filtered out

 - 99% of pictures are nearly identical to the previous frame, or have
   large changes (e.g., a new window)

 - the noise level in pictures is 0.

 - detect picture sequences that look like normal video (say, someone
   screencasting totem playing a video) and fall back to normal
   encoding.

In most cases, a good encoder should be able to encode a screencast
(nearly) losslessly without creating a giant file.  This is not currently
true.



dave...


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Summer Coding: Screen Recording on the Free Desktop

cellenro
In reply to this post by Thomas Dybdahl Ahle-2

--- Thomas Dybdahl Ahle <[hidden email]> wrote:

> On Tue, 2008-03-25 at 02:08 -0700, cellenro cellenro
> wrote:
> > Ogg Theora is not the best codec for screen
> recording
> > because even at higher resolutions it is hard to
> see
> > text clearly. Part of the project will be to
> evaluate
> > the feasibility of using Driac as the codec for
> > recording and editing.
>
> Then what about extending istanbul with driac
> support?
>
> --
> Best Regards,
> Med Venlig Hilsen,
> Thomas
>
>
Extending istanbul is not the optimal way to implement
the specification for the screen recording application
because of the chosen technology.

The goal of the project is to bring a professional
screen casting application to GNU/Linux. Creating a
new project, as a result of this different direction,
seems to be the best way forward.

Thanks for your interest.


      ____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Summer Coding: Screen Recording on the Free Desktop

Sjoerd Simons
In reply to this post by cellenro
On Tue, Mar 25, 2008 at 02:08:52AM -0700, cellenro cellenro wrote:
> Technology:
> Qt Jambi 4.4 -
> (http://trolltech.com/company/newsroom/announcements/press.2008-03-11.2168449860)
>
> JRuby - jruby.codehaus.org
> GStreamer gstreamer.freedesktop.org
> Schrödinger implementation of Driac (diracvideo.org)

Interesting choice of technologies. May i ask why you've choosen for JRuby and
thus a java VM (java is still a bit of an odd kid in the open source world) ?
And which set of gstreamer java bindings you intend to use?

  Sjoerd
--
All science is either physics or stamp collecting.
                -- Ernest Rutherford

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Summer Coding: Screen Recording on the Free Desktop

Sjoerd Simons
In reply to this post by cellenro
On Tue, Mar 25, 2008 at 05:19:20PM -0700, cellenro cellenro wrote:

>
> --- Thomas Dybdahl Ahle <[hidden email]> wrote:
>
> > On Tue, 2008-03-25 at 02:08 -0700, cellenro cellenro wrote:
> > > Ogg Theora is not the best codec for screen > recording
> > > because even at higher resolutions it is hard to see
> > > text clearly. Part of the project will be to
> > > evaluate the feasibility of using Driac as the codec for recording
> > > and editing.
> >
> > Then what about extending istanbul with driac
> > support?

> Extending istanbul is not the optimal way to implement
> the specification for the screen recording application
> because of the chosen technology.

If you want to use this argument then you will have to explain what is wrong
with the technologies Istanbul uses for your purposes and why your chosen set
is better.

> The goal of the project is to bring a professional
> screen casting application to GNU/Linux. Creating a
> new project, as a result of this different direction,
> seems to be the best way forward.

Ditto. Please explain why it will be hard to modify istanbul to reach your
goals. I'm not saying your wrong, but at some rationale to support your
arguments is really needed.

  Sjoerd
--
If it smells it's chemistry, if it crawls it's biology, if it doesn't work
it's physics.

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Summer Coding: Screen Recording on the Free Desktop

cellenro
In reply to this post by cellenro
> Interesting choice of technologies. May i ask why
> you've choosen for JRuby and
> thus a java VM (java is still a bit of an odd kid in
> the open source world) ?
> And which set of gstreamer java bindings you intend
> to use?
>
>   Sjoerd
There was a great presentation on JRuby at FOSDEM 08
(fosdem.org/2008/schedule/events/jruby) that talks
about JRuby's excellent experience with the open
source process, calling it a success story. The
OpenJDK project (openjdk.java.net) is making
significant progress toward a fully open source JVM.
I think that projects such as this one play a useful
role in accelerating Java's transition into the open
source world.

"the JNA project[1] has been used to successfully
build bindings
for the gstreamer multimedia framework.  The resulting
bindings are pure
java (except for a dependency on JNA itself), and run
without modification on
Linux/x86, Linux/amd64, Windows/x86, MacOSX/x86,
Solaris(x86, sparc, sparcv9)
and FreeBSD/x86." --
http://mail.openjdk.java.net/pipermail/challenge-discuss/2008-February.txt

I plan on using the gstreamer-java bindings
(http://code.google.com/p/gstreamer-java/)

There have been some recent discussions on using
gstreamer-java and Gnonlin from JRuby and it looks
promising.

jruby and gstreamer-java
http://groups.google.com/group/gstreamer-java/browse_thread/thread/6be0fde2713f1eb5

Gstreamer-java and Gnonlin / DV
http://groups.google.com/group/gstreamer-java/browse_thread/thread/58a96d3dcb0ce3c1



> If you want to use this argument then you will have
> to explain what is wrong
> with the technologies Istanbul uses for your
> purposes and why your chosen set
> is better.
>
> > The goal of the project is to bring a professional
> > screen casting application to GNU/Linux. Creating
> a
> > new project, as a result of this different
> direction,
> > seems to be the best way forward.
>
> Ditto. Please explain why it will be hard to modify
> istanbul to reach your
> goals. I'm not saying your wrong, but at some
> rationale to support your
> arguments is really needed.
>
>   Sjoerd
Thank you for your important question.

The technologies Istanbul uses are:
GPL v2
GTK
Python

and was last modified (according to the ChangeLog)
2007-02-23
The actual programming logic of how the application
facilitates the screen recording process is
significantly different from the design I outline in
the initial post.

Here are some of the motivations behind the
technological selections made for this project:
 * make it easier for developers to use JRuby to
create desktop multimedia applications on GNU/Linux by
documenting the process and overcoming potential
limitations of the technology (ex: using GStreamer
from JRuby)
 * use the Qt 4.4 GUI framework with JRuby (trolltech
has a posting about using Qt with Jython on one of
their blogs which also uses the JVM)

Progress made in creating this application will have
potential positive effects on all screen recording
software for GNU/Linux. One example is that if
Schroedinger is extended with an encode screencast
mode, other projects such as Istanbul can take
advantage of it. Platforms like JRuby and Jython
empower developers to create more open source software
by enabling languages to be used together instead of
in isolation.

> Hi,
>
> in fact, some weeks ago I played with the thought of
> extending KSnapshot
> with regard to supporting screen recording. I wanted
> to use ffmpeg (which
> IMHO delivers very good quality when using the
> "qtrle" codec) as recorder
> and use the capabilities of KSnapshot only for
> selecting the window or
> region and starting/stopping the recording.
>
> This would support X11 only, though.
>
> Regards,
> Bernd
The qtrle codec
(wiki.multimedia.cx/index.php?title=QTRLE)  does seem
to be capable of good quality screen recording. I
looked into other lossless codecs utilized by existing
proprietary screen recording software but the legal
situation of using them in GPL software was not very
clear to me. This lead  to the evaluation of Driac's
potential because they have a crystal clear legal
situation and the technology is excellent. According
to a recent presentation on GStreamer it was stated
that they also have a clear legal situation (in
contrast to ffmpeg) and that Schroedinger is being
provided as a GStreamer plugin is a nice bonus.

For the first phase of the application I think the
best way will be to develop the screen recording
functionality with JRuby and Qt 4.4 using a place
holder codec ogg Theora via gstreamer with the intent
of replacing the codec with Schroedinger once
development on its "screencast mode" starts.

Editing the recorded video will be explored with the
Gnonlin plugin and looking at what the PiTiVi project
has accomplished so far.


      ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page.
http://www.yahoo.com/r/hs


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel