appsrc->avimux->appsink - very slow

[Nicholas Butts]

I'm trying to mux an MPEG4-ES stream with an uncompressed audio stream into an avimux filter and then to an appsink. I have this working on Ubuntu 10.04. Now I am moving to an ARM based embedded system and this is REALLY slow. It takes on-average 12.7 ms to performing the muxing. This seems extremely slow to me. Has anyone else attempted this? Is there a way to disable all of the threading and make a single threaded application?

--
Nicholas Butts
SENIOR FIRMWARE ENGINEER

Appareo Systems, LLC
1810 NDSU Research Circle N
Fargo, ND 58102

P: [phone]
F: [phone]

http://www.appareo.com

[hidden email]



NOTICE: This message {including attachments} is covered by the Electronic Communication Privacy Act, 18 U.S.C. sections 2510-2521, is CONFIDENTIAL and may also be protected by ATTORNEY-CLIENT OR OTHER PRIVILEGE. If you believe that it has been sent to you in error, do not read it. If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited. Please reply to the sender that you have received the message in error and then delete it.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[michael smith-6-3]

You haven't really provided any information about what is being slow.
12.7 ms for an... indeteminate amount of work may or may not be
reasonable. Using AVI is a terrible idea (it's a poorly designed
format and has been entirely obsolete for quite a few years now), but
is unlikely to be the cause of whatever problems you're having.

You cannot disable use of threads in gstreamer; it's very much core to
the entire design.

Mike



On Thu, Aug 25, 2011 at 2:28 PM, Nicholas Butts <[hidden email]> wrote:

> I'm trying to mux an MPEG4-ES stream with an uncompressed audio stream into
> an avimux filter and then to an appsink. I have this working on Ubuntu
> 10.04. Now I am moving to an ARM based embedded system and this is REALLY
> slow. It takes on-average 12.7 ms to performing the muxing. This seems
> extremely slow to me. Has anyone else attempted this? Is there a way to
> disable all of the threading and make a single threaded application?
>
> --
> Nicholas Butts
> SENIOR FIRMWARE ENGINEER
>
> Appareo Systems, LLC
> 1810 NDSU Research Circle N
> Fargo, ND 58102
>
> P: [phone]
> F: [phone]
>
> http://www.appareo.com
>
> [hidden email]
>
>
>
> NOTICE: This message {including attachments} is covered by the Electronic
> Communication Privacy Act, 18 U.S.C. sections 2510-2521, is CONFIDENTIAL and
> may also be protected by ATTORNEY-CLIENT OR OTHER PRIVILEGE. If you believe
> that it has been sent to you in error, do not read it. If you are not the
> intended recipient, you are hereby notified that any retention,
> dissemination, distribution, or copying of this communication is strictly
> prohibited. Please reply to the sender that you have received the message in
> error and then delete it.
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Stefan Sauer]

On 08/25/11 23:28, Nicholas Butts wrote:
> I'm trying to mux an MPEG4-ES stream with an uncompressed audio stream
> into an avimux filter and then to an appsink. I have this working on
> Ubuntu 10.04. Now I am moving to an ARM based embedded system and this
> is REALLY slow. It takes on-average 12.7 ms to performing the muxing.
> This seems extremely slow to me. Has anyone else attempted this? Is
> there a way to disable all of the threading and make a single threaded
> application?
>
Even if you could disable the threading, I doubt that this would be the
source of the slowness. Check that you don't copy data where it can be
avoided. Also using appsrc and appsink would make me think that your
design isn't that good.

Stefan
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicholas Butts]

I'm capturing 720P video 20 FPS in a MPEG-4-ES stream, so I have 50ms to capture and process a frame. I think my problem has to do with the time stamps. I can write the raw ES streams to disk and then I can play it back without any problems. But when I read the raw frames and try to mux it, then get the data back our and write it I get corrupted frames.

Here is the code I am using to push buffers into the pipeline:
    GstBuffer * vidbuf = gst_buffer_try_new_and_alloc(vidLength);
    if (vidbuf == NULL)
    {
        Log::logError("[VideoMuxer::mux] Failed to allocate a video buffer");
    }
    GST_BUFFER_DURATION(vidbuf) = GST_CLOCK_TIME_NONE;
    GST_BUFFER_TIMESTAMP(vidbuf) = GST_CLOCK_TIME_NONE;
    GST_BUFFER_OFFSET(vidbuf) = GST_BUFFER_OFFSET_NONE;
    GST_BUFFER_OFFSET_END(vidbuf) = GST_BUFFER_OFFSET_NONE;
//    GST_BUFFER_DURATION(vidbuf) = nsDuration;
//    GST_BUFFER_TIMESTAMP(vidbuf) = nsTime;
    gst_buffer_set_data(vidbuf, vid, vidLength);
    error = gst_app_src_push_buffer(GST_APP_SRC(_videoSrc), vidbuf);
    if (error != GST_FLOW_OK)
    {
        Log::logError("[VideoMuxer::mux] Failed to push data into _videoSrc");
    }

I've commented out the code that had the duration and current time. It would generate garbage data if I kept the time stamps. The way the code is written above it will not produce any data into a file. I can get this to work on the PC by reading the MPEG4-ES data and parsing frames from this file and then pushing it into the same pipeline. But when I do this on the embedded system it doesn't work.

Another odd thing is the MPEG-4 hardware compressor produces weird frame times. Here is what ffprobe says about my MPEG4-ES streams. Note that this example uses 24 FPS instead of the 20 FPS of the production system:
Input #0, m4v, from '2010_12_17_Good.m4v':
  Duration: N/A, bitrate: N/A
    Stream #0.0: Video: mpeg4, yuv420p, 1280x720 [PAR 1:1 DAR 16:9], 24k tbr, 1200k tbn, 24k tbc
[STREAM]
codec_name=mpeg4
codec_long_name=MPEG-4 part 2
decoder_time_base=1/24000
codec_type=video
r_frame_rate=24000.000000
r_frame_rate_num=24000
r_frame_rate_den=1
width=1280
height=720
gop_size=12
has_b_frames=1
sample_aspect_ratio=1/1
display_aspect_ratio=16/9
pix_fmt=yuv420p
index=0
time_base=1/1200000
start_time=N/A
duration=N/A
nb_frames=0
[/STREAM]

Now when I mux this into a MOV file (based on an earlier post about the inadequacies of AVI files). I get the following from ffprobe:
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'output.mov':
  Duration: 00:00:03.69, start: 0.000000, bitrate: 3517 kb/s
    Stream #0.0(eng): Video: mpeg4, yuv420p, 1280x720 [PAR 1:1 DAR 16:9], 23.83 tbr, 2400 tbn, 24k tbc
    Stream #0.1(eng): Audio: pcm_s16le, 32000 Hz, stereo, s16, 1024 kb/s
[STREAM]
codec_name=mpeg4
codec_long_name=MPEG-4 part 2
decoder_time_base=1/24000
codec_type=video
language=p�F    ��F   
r_frame_rate=23.833333
r_frame_rate_num=143
r_frame_rate_den=6
width=1280
height=720
gop_size=12
has_b_frames=1
sample_aspect_ratio=1/1
display_aspect_ratio=16/9
pix_fmt=yuv420p
index=0
time_base=1/2400
language=eng
start_time=0.000000
duration=3.654167
nb_frames=0
[/STREAM]
[STREAM]
codec_name=pcm_s16le
codec_long_name=PCM signed 16-bit little-endian
decoder_time_base=0/1
codec_type=audio
language=��F    p�F   
sample_rate=32000.000000
channels=2
bits_per_sample=16
index=1
time_base=1/32000
language=eng
start_time=0.000000
duration=3.696000
nb_frames=0
[/STREAM]


I suspect I am doing something wrong with the time stamps of the data. Can anyone help with an example of how to correctly mux an already compressed video and audio streams together?

Thanks in advance!!!

On Thu, Aug 25, 2011 at 6:39 PM, Stefan Kost <[hidden email]> wrote:

On 08/25/11 23:28, Nicholas Butts wrote:
> I'm trying to mux an MPEG4-ES stream with an uncompressed audio stream
> into an avimux filter and then to an appsink. I have this working on
> Ubuntu 10.04. Now I am moving to an ARM based embedded system and this
> is REALLY slow. It takes on-average 12.7 ms to performing the muxing.
> This seems extremely slow to me. Has anyone else attempted this? Is
> there a way to disable all of the threading and make a single threaded
> application?
>

Even if you could disable the threading, I doubt that this would be the
source of the slowness. Check that you don't copy data where it can be
avoided. Also using appsrc and appsink would make me think that your
design isn't that good.

Stefan

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel