WebRTC to MKV or hlssink2 - problems with audio+video streams together

[d3im]

Hi,
I'm using gst 1.16.2 and trying to modify webrtc-sendrecv.c to save received
streams to file (HLS would be perfect) instead of play with autosinks.

When I save only one stream it seem OK - audio alone is fine video alone
also OK.

When I try to put them together it quits:
ERROR:/var/tmp/portage/media-libs/gst-plugins-good-1.16.2/work/gst-plugins-good-1.16.2/gst/multifile/gstsplitmuxsink.c:2008:handle_gathered_gop:
assertion failed: (queued_gop_time >= 0)
Bail out!
ERROR:/var/tmp/portage/media-libs/gst-plugins-good-1.16.2/work/gst-plugins-good-1.16.2/gst/multifile/gstsplitmuxsink.c:2008:handle_gathered_gop:
assertion failed: (queued_gop_time >= 0)
(SIGABRT)

Is there any technique how to do it correctly?

I enclose graph of it:
0.dot <http://gstreamer-devel.966125.n4.nabble.com/file/t379376/0.dot>  
0.pdf <http://gstreamer-devel.966125.n4.nabble.com/file/t379376/0.pdf>  



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Mandar Joshi]

Hello deim31,

This is just something that popped up in the head when I first saw your problem. No guarantees it will work but I think it is certainly worth an attempt.

> When I save only one stream it seem OK - audio alone is fine video alone
> also OK.

In the callback to the pad-added signal, audio and video seem to be arriving one by one and not together.

Can you try creating a common bin starting the bin only after both audio and video have arrived?

If possible, posting your pad-added callback would certainly help.

>
> When I try to put them together it quits:
> ERROR:/var/tmp/portage/media-libs/gst-plugins-good-1.16.2/work/gst-plugins-good-1.16.2/gst/multifile/gstsplitmuxsink.c:2008:handle_gathered_gop:
> assertion failed: (queued_gop_time >= 0)
> Bail out!
> ERROR:/var/tmp/portage/media-libs/gst-plugins-good-1.16.2/work/gst-plugins-good-1.16.2/gst/multifile/gstsplitmuxsink.c:2008:handle_gathered_gop:
> assertion failed: (queued_gop_time >= 0)
> (SIGABRT)

Does something apart from hlssink2 work? Like "... ! matroskamux ! filesink" or "... ! matroskamux ! splitmuxsink"

Regards

Mandar

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[d3im]

hlssink is created in function start_pipeline:
hlssink = gst_element_factory_make ("hlssink2", NULL);

/*
 * to hlssink - with pad requests*/
static void
handle_media_stream_d (GstPad * pad, GstElement * pipe, const char *type,
GstElement * hlssink)
{
  GstPad *qpad;
  GstElement *q, *q1, *conv, *resample, *sink, *enc, *parse, *fr_rate;
  GstPadLinkReturn ret;
  GstCaps *fr_caps;

  GstPad *hls_pad, *enc_pad;
  gchar *name;

  gst_print ("Trying to handle stream %s\n", type);

  q = gst_element_factory_make ("queue", NULL);
  g_assert_nonnull (q);
  q1 = gst_element_factory_make ("queue", NULL);
  g_assert_nonnull (q1);

  if (g_strcmp0 (type, "audio") == 0) {
        conv = gst_element_factory_make ("audioconvert", NULL);
        g_assert_nonnull (conv);
    /* Might also need to resample, so add it just in case.
     * Will be a no-op if it's not required. */
    resample = gst_element_factory_make ("audioresample", NULL);
    g_assert_nonnull (resample);
    enc = gst_element_factory_make ("faac", NULL); //avenc_aac
    g_assert_nonnull (enc);
    parse = gst_element_factory_make ("aacparse", NULL);
    g_assert_nonnull (parse);

    gst_bin_add_many (GST_BIN (pipe), q, conv, resample, enc, q1, parse,
NULL);
    gst_element_sync_state_with_parent (q);
    gst_element_sync_state_with_parent (conv);
    gst_element_sync_state_with_parent (resample);
    gst_element_sync_state_with_parent (enc);
    gst_element_sync_state_with_parent (q1);
    gst_element_sync_state_with_parent (parse);
    //gst_element_sync_state_with_parent (hlssink);
    gst_element_link_many (q, conv, resample, enc, q1, parse, NULL);

  } else { // video
        conv = gst_element_factory_make ("videoconvert", NULL);
        g_assert_nonnull (conv);
        enc = gst_element_factory_make ("x264enc", NULL);
        g_object_set(enc,
"tune",0x00000004,"pass",5,"speed-preset",1,"key-int-max",10,NULL);

        fr_rate = gst_element_factory_make("videorate", NULL);
        fr_caps = gst_caps_new_simple("video/x-raw",
                  "framerate", GST_TYPE_FRACTION, 25, 2,
                  NULL);

        g_assert_nonnull (enc);
        parse = gst_element_factory_make ("h264parse", NULL);
        g_assert_nonnull (parse);
    gst_bin_add_many (GST_BIN (pipe), q, conv, fr_rate, enc, q1, parse,
NULL);
    gst_element_sync_state_with_parent (q);
    gst_element_sync_state_with_parent (conv);
    gst_element_sync_state_with_parent (fr_rate);
    gst_element_sync_state_with_parent (enc);
    gst_element_sync_state_with_parent (q1);
    gst_element_sync_state_with_parent (parse);

    gst_element_link_many (q, conv, fr_rate, NULL);
    gst_element_link_filtered(fr_rate,enc,fr_caps);
    gst_element_link_many (enc, q1, parse, NULL);

  }

  // add to hls audio/video sink pads
  enc_pad = gst_element_get_static_pad(parse, "src");
  hls_pad = gst_element_get_request_pad(hlssink, type);
  name = gst_pad_get_name(hls_pad);
  gst_print ("pad name %s\n", name);

// tried to sync hlssink only once but it's not better
  gst_print ("state %d\n", state);
  if (state > 0) {
          gst_print ("SYNCING\n", NULL);
          gst_element_sync_state_with_parent (hlssink);
  } else {
          state++;
  }

  gst_pad_link(enc_pad, hls_pad);

  qpad = gst_element_get_static_pad (q, "sink");

  ret = gst_pad_link (pad, qpad);
  g_assert_cmphex (ret, ==, GST_PAD_LINK_OK);
}

"... ! matroskamux ! filesink" is the same but doesn't exits - it produces
mkv file with only audio playable (only audio stream is visible in mkv file,
don't know if it fills video also, but size seem be bigger than just audio)

I think common bin could help. Never done this before. Should I make bin
only from hlssink or longer pipe?

Thanks,
deim31



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Jan Schmidt-3]

Hi,

On 3/11/20 12:26 am, d3im wrote:

> Hi,
> I'm using gst 1.16.2 and trying to modify webrtc-sendrecv.c to save received
> streams to file (HLS would be perfect) instead of play with autosinks.
>
> When I save only one stream it seem OK - audio alone is fine video alone
> also OK.
>
> When I try to put them together it quits:
> ERROR:/var/tmp/portage/media-libs/gst-plugins-good-1.16.2/work/gst-plugins-good-1.16.2/gst/multifile/gstsplitmuxsink.c:2008:handle_gathered_gop:
> assertion failed: (queued_gop_time >= 0)
> Bail out!
> ERROR:/var/tmp/portage/media-libs/gst-plugins-good-1.16.2/work/gst-plugins-good-1.16.2/gst/multifile/gstsplitmuxsink.c:2008:handle_gathered_gop:
> assertion failed: (queued_gop_time >= 0)

I fixed that problem recently in
https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/-/commit/1316dd9c653dc8e6e6eff23ba922549745e4daf3 
- but it's only in git master at this point.

I think one solution for you would be to add a buffer probe on the audio
stream and drop any audio before the first video frame arrives.

Cheers,

Jan.

> (SIGABRT)
>
> Is there any technique how to do it correctly?
>
> I enclose graph of it:
> 0.dot <http://gstreamer-devel.966125.n4.nabble.com/file/t379376/0.dot>
> 0.pdf <http://gstreamer-devel.966125.n4.nabble.com/file/t379376/0.pdf>
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Mandar Joshi]

> I think one solution for you would be to add a buffer probe on the audio
> stream and drop any audio before the first video frame arrives.

This solution works for me. Here's the prototype code I used to test

gulong audioprobe_id;
GstPad *audioparse_pad;

GstPadProbeReturn audioparse_drop_buffer (GstPad *pad, GstPadProbeInfo *info,
                                          gpointer user_data) {
  g_message ("Need to drop this audio buffer");
  return GST_PAD_PROBE_DROP;
}

//Need To Drop Audio buffers until video arrives
static void
handle_media_stream (GstPad *pad, GstElement *pipe, const char *convert_name,
  const char *sink_name) {
    GstElement *q, *audioconvert, *videoconvert, *faac, *aacparse;
    GstElement *audiosink, *videosink;
    GstPad *qpad, *p1, *p2;
    static GstElement *hlssink2 = NULL;

    GstElement *x264enc, *h264parse;
    GstPad *videoparse_pad;

    gboolean ret;

    if (g_str_equal (convert_name, "audioconvert")) {
      g_message ("Received Audio");
      q = gst_element_factory_make ("queue", "queue");
      audioconvert = gst_element_factory_make ("audioconvert", "audioconvert");
      hlssink2 = gst_element_factory_make ("hlssink2", "hlssink2");
      g_object_set (hlssink2, "max-files", 5, "target-duration", 4, NULL);
      faac = gst_element_factory_make ("avenc_aac", "avenc_aac");
      aacparse = gst_element_factory_make ("aacparse", "aacparse");

      /*audiosink = gst_element_factory_make ("autoaudiosink", "autoaudiosink");
      gst_bin_add_many (GST_BIN (pipe), q, audioconvert, audiosink, NULL);*/

      gst_bin_add_many (GST_BIN (pipe), q, audioconvert, faac, aacparse,
                        hlssink2, NULL);
      gst_element_sync_state_with_parent (q);
      gst_element_sync_state_with_parent (audioconvert);
      gst_element_sync_state_with_parent (faac);
      gst_element_sync_state_with_parent (aacparse);
      //gst_element_sync_state_with_parent (audiosink);
      //gst_element_link_many (q, audioconvert, audiosink, NULL);
      gst_element_link_many (q, audioconvert, faac, aacparse, NULL);

      //Drop Audio Buffers
      audioparse_pad = gst_element_get_static_pad (aacparse, "src");
      audioprobe_id = gst_pad_add_probe (audioparse_pad, GST_PAD_PROBE_TYPE_BUFFER,
              audioparse_drop_buffer, NULL, NULL);

      //Regular pad linking
      p1 = gst_element_get_request_pad (hlssink2, "audio");
      audioparse_pad = gst_element_get_static_pad (aacparse, "src");
      ret = gst_pad_link (audioparse_pad, p1);

      g_message ("Audio Pad Link: %d", ret);
      g_assert_cmphex (ret, ==, GST_PAD_LINK_OK);

      qpad = gst_element_get_static_pad (q, "sink");
      ret = gst_pad_link (pad, qpad);
      g_assert_cmphex (ret, ==, GST_PAD_LINK_OK);
    }
    else if (g_str_equal (convert_name, "videoconvert")) {
      g_message ("Received Video");
      q = gst_element_factory_make ("queue", "videoqueue");
      videoconvert = gst_element_factory_make ("videoconvert", "videoconvert");
      x264enc = gst_element_factory_make ("x264enc", "x264enc");
      h264parse = gst_element_factory_make ("h264parse", "h264parse");

      gst_bin_add_many (GST_BIN (pipe), q, videoconvert, x264enc, h264parse,
                        NULL);
      gst_element_sync_state_with_parent (q);
      gst_element_sync_state_with_parent (videoconvert);
      gst_element_sync_state_with_parent (x264enc);
      gst_element_sync_state_with_parent (h264parse);
      //gst_element_sync_state_with_parent (audiosink);
      //gst_element_link_many (q, audioconvert, audiosink, NULL);
      gst_element_link_many (q, videoconvert, x264enc, h264parse, NULL);

      //Regular pad linking
      p1 = gst_element_get_request_pad (hlssink2, "video");
      videoparse_pad = gst_element_get_static_pad (h264parse, "src");
      ret = gst_pad_link (videoparse_pad, p1);
      g_message ("Video Pad Link: %d", ret);
      g_assert_cmphex (ret, ==, GST_PAD_LINK_OK);

      qpad = gst_element_get_static_pad (q, "sink");
      ret = gst_pad_link (pad, qpad);
      g_assert_cmphex (ret, ==, GST_PAD_LINK_OK);
      gst_pad_remove_probe (audioparse_pad, audioprobe_id);
    }
}

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[d3im]

I've managed to create custom bin and connect those two decodebins (audio,
video) to them and MKV and HLS outputs are with both streams now.

I face another "cardinal" problem - broken video frames. From beginning
video (and audio) works fine. After a while video image broke (as on
enclose) audio still ok then.
<http://gstreamer-devel.966125.n4.nabble.com/file/t379376/mpv-shot0051.jpg>

I tried Vivaldi and Opera browsers on Linux machine. Same results.
I use Vivaldi for online conferencing without this broken frames (Jitsi)



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[d3im]

To elaborate more on this problem - my situation:

original sendrecv example. (doesnt matter if it goes to autovideosink or to MKV or HLS it's always the same)

I modified html page to share screen (getDisplayMedia instead of getUserMedia)

- use it in Vivaldi or Opera.

I first share whole screen and it works well several seconds then image breaks.

Today I tried share window. Now I know the larger shared window is the sooner image breaks.

800x800px is almost OK. When I play for example YT video in Firefox window I share, it breaks sometimes and I can see moving things but with wrong colors but it in several seconds refreshes (maybe new I frame?).

The big problem is in text share or for example slides. Broken image seem never refreshes.

1920x1200 is problem.

Is it known problem? Would for example Windows or Mac work better?

On other side - I use the same screen share with Jitsi Meet and it never break image - codec is the same - VP8.

Thanks

Dne 11. 11. 20 v 14:04 d3im napsal(a):

I've managed to create custom bin and connect those two decodebins (audio,
video) to them and MKV and HLS outputs are with both streams now.

I face another "cardinal" problem - broken video frames. From beginning
video (and audio) works fine. After a while video image broke (as on
enclose) audio still ok then.
<http://gstreamer-devel.966125.n4.nabble.com/file/t379376/mpv-shot0051.jpg> 

I tried Vivaldi and Opera browsers on Linux machine. Same results.
I use Vivaldi for online conferencing without this broken frames (Jitsi)



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[GStreamer-devel mailing list]

d3im wrote
> "... ! matroskamux ! filesink" is the same but doesn't exits - it produces
> mkv file with only audio playable (only audio stream is visible in mkv
> file,
> don't know if it fills video also, but size seem be bigger than just
> audio)
>
> I think common bin could help. Never done this before. Should I make bin
> only from hlssink or longer pipe?


We have exactly same problems .mkv files only has audio and video is empty.
If we only record video it also works.

You write that you solved and successfully record .mkv files with audio and
video, how can do that?

Thanks.




--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel