Sync Video and Audio

[killerrats]

gst-launch-1.0 --gst-debug=3 -e rtspsrc location=[IP] name=source !
rtpjitterbuffer ! rtph264depay ! h264parse config-interval=1 ! tee name=vtee
! mux.video splitmuxsink async-handling=true max-size-bytes=1048576
max-files=360 muxer=avimux location=video%06d.avi name=mux source. !
rtpjitterbuffer ! rtpmp4gdepay ! aacparse ! tee name=atee ! mux.audio_0

I put together an application but somehow the video and audio are off by
0.5-1sec. It will hit a hickup sometime and then after that the video and
audio will be off quite a bit. It has got up to 1min off where the audio
will show up before movement.

h264parse error:
the only thing that happened was h264parse error:
"couldn't find associated picture parameter set with id: 2

How can you adjust the video and audio sync? I have put sync=true on the
filesink in the splitmuxsink. any ideas?



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[killerrats]

gst-launch-1.0 --gst-debug=3 -e rtspsrc location=[IP] name=source !
rtpjitterbuffer ! rtph264depay ! h264parse config-interval=1 ! tee name=vtee
! mux.video splitmuxsink async-handling=true max-size-bytes=1048576
max-files=360 location=video%06d.avi name=mux source. !
rtpjitterbuffer ! rtpmp4gdepay ! aacparse ! tee name=atee ! mux.audio_0

Okay so i used the mp4mux element and the video and audio are spot on when
watching them.

Error that comes up and doesn't let it go anywhere after that. Any ideas?

13:15:20.737978710  5888   02ED4EB0 WARN qtmux
gstqtmux.c:4553:gst_qt_mux_add_buffer:muxer> error: Buffer has no PTS.
ERROR: from element
/GstPipeline:pipeline/GstSplitMuxSinkNpc:splitmuxsink/GstMP4
Mux:muxer: Could not multiplex stream.Additional debug info:
gstqtmux.c(4553): gst_qt_mux_add_buffer ():
/GstPipeline:pipeline/GstSplitMuxSin
kNpc:splitmuxsink/GstMP4Mux:muxer:Buffer has no PTS.



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mercredi 22 août 2018 à 17:20 -0500, killerrats a écrit :

> gst-launch-1.0 --gst-debug=3 -e rtspsrc location=[IP] name=source !
> rtpjitterbuffer ! rtph264depay ! h264parse config-interval=1 ! tee name=vtee
> ! mux.video splitmuxsink async-handling=true max-size-bytes=1048576
> max-files=360 location=video%06d.avi name=mux source. !
> rtpjitterbuffer ! rtpmp4gdepay ! aacparse ! tee name=atee ! mux.audio_0
>
> Okay so i used the mp4mux element and the video and audio are spot on when
> watching them.
>
> Error that comes up and doesn't let it go anywhere after that. Any ideas?
>
> 13:15:20.737978710  5888   02ED4EB0 WARN qtmux
> gstqtmux.c:4553:gst_qt_mux_add_buffer:muxer> error: Buffer has no PTS.
> ERROR: from element
> /GstPipeline:pipeline/GstSplitMuxSinkNpc:splitmuxsink/GstMP4
> Mux:muxer: Could not multiplex stream.Additional debug info:
> gstqtmux.c(4553): gst_qt_mux_add_buffer ():
> /GstPipeline:pipeline/GstSplitMuxSin
> kNpc:splitmuxsink/GstMP4Mux:muxer:Buffer has no PTS.

We removed the naive PTS generator (copying DTS into PTS) and added
this error, so that GStreamer stop producing invalid ISOMP4. So this
error is telling you that something before the muxer is not setting
PTS. We know some work is needed, someone need to find the time to fix
this properly.

>
>
>
> -----
> ------------------------------
> Gstreamer 1.14.1
> ------------------------------
> Windows
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[killerrats]

will the qtmux do the same thing as well if i use it?



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Mandar Joshi]

Nicolas,
Is this workaround not valid any more?
https://lists.freedesktop.org/archives/gstreamer-devel/2017-October/065833.html

Regards
Mandar Joshi
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[killerrats]

I at least found the equivalent to the other method you showed. the only thing is that haven't been able to figure out how to apply it. gst_base_parse_set_pts_interpolation(GstBaseParse,gboolean); GstBaseParse

---

Sent from the GStreamer-devel mailing list archive at Nabble.com.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[killerrats]

I searched for gstreamer pts dts or something like that and came across
multiqueue that I had used awhile back. In the gstreamer 1.7.1 says
"multiqueue: Use buffer DTS if present, else PTS". could this still be
active and useful?



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeu. 23 août 2018 22:12, Mandar Joshi <[hidden email]> a écrit :

Nicolas,
Is this workaround not valid any more?
https://lists.freedesktop.org/archives/gstreamer-devel/2017-October/065833.html

Sure you can hack the parser like in this example, or you could just copy the DTS into the PTS with a pad probe, assuming you don't have B-Frames. We should implement this heuristic in h264parse, just need someone to have the time.

Regards
Mandar Joshi
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[killerrats]

Is this the correct way of doing this?

GstPadProbeType ProbeType = GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM;

gulong id = gst_pad_add_probe(gst_element_get_static_pad(h264parse, "sink"),
ProbeType, Check_buffer_pts_probe, NULL, NULL);

GstPadProbeReturn Check_buffer_pts_probe(GstPad* pad, GstPadProbeInfo* info,
gpointer user_data)
{
        if (GST_EVENT_TYPE(GST_PAD_PROBE_INFO_DATA(info)) == GST_EVENT_EOS)
        {
                return GST_PAD_PROBE_OK;
        }

        GstBuffer * buffer = gst_pad_probe_info_get_buffer(info);
       
        GST_BUFFER_PTS(buffer) = GST_BUFFER_DTS(buffer);

        return GST_PAD_PROBE_OK;
}



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[killerrats]

Is this the correct way of doing this?

GstPadProbeType ProbeType = GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM;

gulong id = gst_pad_add_probe(gst_element_get_static_pad(h264parse, "sink"),
ProbeType, Check_buffer_pts_probe, NULL, NULL);

GstPadProbeReturn Check_buffer_pts_probe(GstPad* pad, GstPadProbeInfo* info,
gpointer user_data)
{
        if (GST_EVENT_TYPE(GST_PAD_PROBE_INFO_DATA(info)) == GST_EVENT_EOS)
        {
                return GST_PAD_PROBE_OK;
        }

        GstBuffer * buffer = gst_pad_probe_info_get_buffer(info);

        GST_BUFFER_PTS(buffer) = GST_BUFFER_DTS(buffer);

        return GST_PAD_PROBE_OK;


}



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[killerrats]

sorry for the repeat. i thought it didn't submit the other reply yesterday.

well it looks like this works.
GstElement* parse = gst_element_factory_make("h264parse",NULL);
gst_base_parse_set_pts_interpolation(GST_BASE_PARSE(parse), TRUE);

In the long run it seems to have not underrun for the splitmuxsink like it
did without apply that gst_base_parse_set_pts_interpolation().



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[killerrats]

That h264parse definitely works. the only thing is that the video and audio
are off from each other after several hours. I put sync = true on the mp4mux
in my application before adding to the splitmuxsink. any ideas?



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[killerrats]

does anybody know if filesink ignores sync=true?

I read this  sync ignored
<https://lists.freedesktop.org/archives/gstreamer-bugs/2010-June/065302.html>
. is this still true or has this been fixed?




-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mar. 18 sept. 2018 19 h 12, killerrats <[hidden email]> a écrit :

does anybody know if filesink ignores sync=true?

I read this  sync ignored
<https://lists.freedesktop.org/archives/gstreamer-bugs/2010-June/065302.html>
. is this still true or has this been fixed?

This ancient bug is marked as resolved. Unless there is a regression, it should still be this way.

-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[killerrats]

It takes about a day before I notice the video and audio start to go out of
sync. I set the value qos to true as well on the mp4mux. Do I have to do
anything to check if video and audio are out of sync along the pipeline?



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[killerrats]

okay I'm trying a new approach by creating two sources to separate between
audio and video in the same bin and try to see if that syncs it better than
having one source give out the both medias.

unless anybody thinks that it doesn't matter which way result is the same?



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[killerrats]

gst-launch-1.0 -e rtspsrc location=[IP] latency=66 name=source !
rtpjitterbuffer do-lost=true ! rtph264depay ! identity ! h264parse
config-interval=1 ! avdec_h264 ! openh264enc ! h264parse ! tee name=vtee !
mq.sink_1 multiqueue name=mq ! mux.video splitmuxsink async-handling=true
max-size-bytes=4194304 max-files=360 muxer=mp4mux location=video%06d.avi
name=mux mq.src_2 ! mux.audio_1 source. ! rtpjitterbuffer do-lost=true !
rtpmp4gdepay ! identity ! aacparse ! avdec_aac ! avenc_aac ! aacparse ! tee
name=atee ! mq.sink_2

found out if I do this for each side of the branches for audio and video.

depay -> parse -> decode -> encode -> parse -> tee

if i leave the decode -> encode -> parse out of it the pipeline won't go out
of sync. i apparently don't need to have two sources.



-----
------------------------------
Gstreamer 1.14.2
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel