JVM blocked when using gst1-java-core

[Xinheng]

Hi everyone,

My application is in Java so I use the non-official gstreamer java library gst1-java-core. In my application, I use appsrc to load a hls stream. The java code to build the pipeline I think is similar to C code so I attach here:

src = (AppSrc) ElementFactory.make("appsrc", "source");
src.setLive(true);
src.setMaxBytes(APPSRC_MAX_QUEUE_SIZE);
src.set("max-bytes", APPSRC_MAX_QUEUE_SIZE);
src.setBlocksize(READ_BUFFER_SIZE);
src.set("blocksize", READ_BUFFER_SIZE);
src.setStreamType(Type.STREAM);
src.set("emit-signals", true);
src.set("block", true);

dataListener = new AppSrc.NEED_DATA() {
      public void needData(AppSrc appsrc, int buffer_size) {
             if (state == ETAT.URL_LOADED || state == ETAT.LOADING_STREAM || state == ETAT.PLAYING){
                    Buffer buffer = new Buffer(buffer_size);
                    byte[] bytesbuffer = new byte[buffer_size];
                    int read_bytes = 0;
                    try {
                          if (input_stream != null) {
                                 attente_sec = 3;
                                 read_bytes = ((HlsInputStreamPerformance)input_stream).read(bytesbuffer, 0, buffer_size,attente_sec);
                          }
                          if (read_bytes > 0){
                                 ByteBuffer bb = buffer.map(false);
                                 bb.put(bytesbuffer,0,read_bytes);
                                 buffer.unmap();
                                 appsrc.pushBuffer(buffer);
                                 bb.clear();
                          }
                          buffer.dispose();
                          if ( read_bytes < buffer_size){
                                 setEtat(ETAT.STOPPING);
                                 src.endOfStream();
                                 stopFromNeedData();
                                 return;
                          }
                    } catch (IOException e) {
                          e.printStackTrace();
                    }
             } else {
                    src.endOfStream();
                    return;
             }
      }
};
src.connect(dataListener);

videoBin = new Bin("Video Bin");
Element tsDemux = ElementFactory.make("tsdemux", "tsdemux");
Element h264parse = ElementFactory.make("h264parse", "h264parse");
avdec_h264 = ElementFactory.make("avdec_h264", "H264 Decoder");
Element videoconvert = ElementFactory.make("videoconvert", "videoconvert");
Element videorate = ElementFactory.make("videorate", "videorate");
Element capsfilter = ElementFactory.make("capsfilter", "capsfilter");
StringBuilder capsString = new StringBuilder("video/x-raw, framerate=6/1, ");

if (ByteOrder.nativeOrder() == ByteOrder.LITTLE_ENDIAN) {
      capsString.append("format=BGRx");
} else {
      capsString.append("format=xRGB");
}

Caps caps = Caps.fromString(capsString.toString());
capsfilter.set("caps", caps);
videoBin.addMany(h264parse,avdec_h264,videoconvert,videorate,capsfilter,sink_element);
h264parse.link(avdec_h264);
avdec_h264.link(videoconvert);
videoconvert.link(videorate);
videorate.link(capsfilter);
capsfilter.link(sink_element);
videoBin.addPad(new GhostPad("sink", h264parse.getStaticPad("sink")));

pipe = new Pipeline("main pipeline");
pipe.add(src);
input_buffer = ElementFactory.make("queue", "input_buffer");
input_buffer.set("max-size-time", 1 * 1000000000);
pipe.add(tsDemux);
pipe.add(input_buffer);
pipe.add(videoBin);
src.link(tsDemux);
tsDemux.link(input_buffer);

audioBin = new Bin("Audio Bin");
Element queue = ElementFactory.make("queue", "queue");
Element aacparse = ElementFactory.make("aacparse", "aacparse");
Element faad = ElementFactory.make("faad", "faad");
Element autoaudiosink = ElementFactory.make("autoaudiosink", "autoaudiosink");
autoaudiosink.set("sync", false);
volumeControl = ElementFactory.make("volume", "volume");
volumeControl.set("volume", VOLUME_0);
audioBin.addMany(queue,aacparse, faad,volumeControl,autoaudiosink);
Element.linkMany(queue,aacparse, faad,volumeControl,autoaudiosink);
audioBin.addPad(new GhostPad("sink", queue.getStaticPad("sink")));
pipe.add(audioBin);

tsDemux.connect(new Element.PAD_ADDED() {
      @Override
      public void padAdded(Element element, Pad pad) {
              Caps caps = pad.getCaps();
              Structure struct = caps.getStructure(0);
              if (struct.getName().startsWith("audio/")) {
                    Pad audioPad = audioBin.getStaticPad("sink");
                    if (pad.isLinked()) {
                           return;
                    }
                    pad.link(audioPad);
              } else if (struct.getName().startsWith("video/")) {
                    Pad videoPad = videoBin.getStaticPad("sink");
                    if (pad.isLinked()) {
                           return;
                    }
                    pad.link(videoPad);
              }
      }
});

This is my creation of pipeline. The variable "state" isn't something in gstreamer and the function "stopFromNeedData()" will launch a thread to call pipe.stop(). I hope you guys could understand my code without too much difficulty.

I use a test video lasting about 2 seconds which will be shown in a window.
The pipeline will be built only one time.
Every time I open the video window the pipeline will load the video and play it. Every time I close the window or the video runs out of the time, the pipeline will be stopped.

I tried to test the stability of my application so I programmatically open and close the window again and again. But always after thousands iterations, no response from JVM but no special errors (neither gstreamer nor java) shown in console. I assume that it encountered a deadlock on gstreamer level but I cannot find any solution to debug or even solve it. Hope you guys can help me.

If someone dosen't understand some part of my test, ask me without hesitation. Thank you!

[Neil C Smith]

Hi,

On Wed, 22 Mar 2017 at 08:57 silver <[hidden email]> wrote:

I tried to test the stability of my application so I programmatically open
and close the window again and again. But always after thousands iterations,
no response from JVM but no special errors (neither gstreamer nor java)
shown in console. I assume that it encountered a deadlock on gstreamer level
but I cannot find any solution to debug or even solve it. Hope you guys can
help me.

Have you tried profiling and a thread dump on the Java side?  When updating the 0.10 bindings for gst1-java-core I fixed a load of memory leaks, but it's possible your code is finding another one.  Is memory usage increasing?  Are any particular objects not being garbage collected?  Does the thread dump show a potential deadlock inside the JVM?

Note that you should probably be calling buffer.map(true) - the boolean is a shortcut for mapping the buffer with the write flag.

I'm not sure what your stopFromNeedData() call is doing, but it should probably use Gst,invokeLater().

Best wishes.

Neil

--

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Xinheng]

Neil C Smith wrote

Hi,

On Wed, 22 Mar 2017 at 08:57 silver <[hidden email]> wrote:

> I tried to test the stability of my application so I programmatically open
> and close the window again and again. But always after thousands
> iterations,
> no response from JVM but no special errors (neither gstreamer nor java)
> shown in console. I assume that it encountered a deadlock on gstreamer
> level
> but I cannot find any solution to debug or even solve it. Hope you guys can
> help me.
>

Have you tried profiling and a thread dump on the Java side?  When updating
the 0.10 bindings for gst1-java-core I fixed a load of memory leaks, but
it's possible your code is finding another one.  Is memory usage
increasing?  Are any particular objects not being garbage collected?  Does
the thread dump show a potential deadlock inside the JVM?

Note that you should probably be calling buffer.map(true) - the boolean is
a shortcut for mapping the buffer with the write flag.

I'm not sure what your stopFromNeedData() call is doing, but it should
probably use Gst,invokeLater().

Best wishes.

Neil
--
Neil C Smith
Artist & Technologist
www.neilcsmith.net

Praxis LIVE - hybrid visual IDE for creative coding - www.praxislive.org

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Thanks for your response. I'm always trying the profiler in visualVM but it seems not able to connect to the jvm and I don't know why. But I'm sure that the memory usage is relatively stable.

I'm trying your propose of invokeLater() but the test really takes time :/

In addition, I think I should mention my process of pipeline:
Before: 1. when opening the window, I did pipe.ready()(GST_STATE_NULL->GST_STATE_READY) and then pipe.play()(GST_STATE_READY->GST_STATE_PAUSED->GST_STATE_PLAYING)
           2. when closing the video or it runs out of time, I did pipe.stop()(GST_STATE_PLAYING->GST_STATE_PAUSED->GST_STATE_READY->GST_STATE_NULL)
Now(the last test): I comment pipe.stop()

I don't know which is more correct or both are OK or none. The last time I tested, it ended at the 9000th iteration and then no response. In the console I could see that when opening the windows,it was changing the state element by element but just casually it died.

Last logs for state:
INFOS: Source = autoaudiosink-actual-sink-directsound message = GstMessageStateChanged, old-state=(GstState)GST_STATE_PLAYING, new-state=(GstState)GST_STATE_PAUSED, pending-state=(GstState)GST_STATE_VOID_PENDING;
INFOS: Source = autoaudiosink message = GstMessageStateChanged, old-state=(GstState)GST_STATE_PLAYING, new-state=(GstState)GST_STATE_PAUSED, pending-state=(GstState)GST_STATE_VOID_PENDING;
INFOS: Source = volume message = GstMessageStateChanged, old-state=(GstState)GST_STATE_PLAYING, new-state=(GstState)GST_STATE_PAUSED, pending-state=(GstState)GST_STATE_VOID_PENDING;
INFOS: Source = faad message = GstMessageStateChanged, old-state=(GstState)GST_STATE_PLAYING, new-state=(GstState)GST_STATE_PAUSED, pending-state=(GstState)GST_STATE_VOID_PENDING;
INFOS: Source = aacparse message = GstMessageStateChanged, old-state=(GstState)GST_STATE_PLAYING, new-state=(GstState)GST_STATE_PAUSED, pending-state=(GstState)GST_STATE_VOID_PENDING;
INFOS: Source = queue message = GstMessageStateChanged, old-state=(GstState)GST_STATE_PLAYING, new-state=(GstState)GST_STATE_PAUSED, pending-state=(GstState)GST_STATE_VOID_PENDING;
INFOS: Source = Audio Bin message = GstMessageStateChanged, old-state=(GstState)GST_STATE_PLAYING, new-state=(GstState)GST_STATE_PAUSED, pending-state=(GstState)GST_STATE_VOID_PENDING;
INFOS: Source = GstVideoComponent message = GstMessageStateChanged, old-state=(GstState)GST_STATE_PLAYING, new-state=(GstState)GST_STATE_PAUSED, pending-state=(GstState)GST_STATE_VOID_PENDING;

[Xinheng]

Neil C Smith wrote

Hi,

On Wed, 22 Mar 2017 at 08:57 silver <[hidden email]> wrote:

> I tried to test the stability of my application so I programmatically open
> and close the window again and again. But always after thousands
> iterations,
> no response from JVM but no special errors (neither gstreamer nor java)
> shown in console. I assume that it encountered a deadlock on gstreamer
> level
> but I cannot find any solution to debug or even solve it. Hope you guys can
> help me.
>

Have you tried profiling and a thread dump on the Java side?  When updating
the 0.10 bindings for gst1-java-core I fixed a load of memory leaks, but
it's possible your code is finding another one.  Is memory usage
increasing?  Are any particular objects not being garbage collected?  Does
the thread dump show a potential deadlock inside the JVM?

Note that you should probably be calling buffer.map(true) - the boolean is
a shortcut for mapping the buffer with the write flag.

I'm not sure what your stopFromNeedData() call is doing, but it should
probably use Gst,invokeLater().

Best wishes.

Neil
--
Neil C Smith
Artist & Technologist
www.neilcsmith.net

Praxis LIVE - hybrid visual IDE for creative coding - www.praxislive.org

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

One more thing, what endOfStream() really dose? I realized that it dosn't change the state of elements.