Wrong PTS in AppSink and duplicate frames
Locked
 
|
Hello everyone,
I've been struggling with this for months now. I've looked in all the GStreamer docs, read the whole Internet, but couldn't find a cure. I would really be grateful for some help. Thanks in advance. Here is what I'm trying to accomplish (simplified): - seek to an exact decoded raw audio frame position - read a number of frames - (do some processing) - seek exactly past the previously read frames - read some more frames - (and so on) What I'm expecting to see: - A contiguous buffer seamlessly memcpy'd together from AppSink's buffers, that is basically a perfect copy of the decoded raw audio data. This works OK for WAV files. This is what happens instead: - For FLAC: after a seek, the first GstSample that I pull from AppSink is always offset by exactly 1 frame, so I get a duplicate between the end of the previous call to my read function and the current one - For various other formats: most GstSamples' GstBuffer->pts are either -1, 0, or +1 frames of their expected position - so I either get 1 duplicate, get it right, or I'm 1 frame short I've got the same results with hundreds of various test files, GStreamer 1.14.5 and 1.16.2, on both 32 and 64bit versions of Linux. ***************************************************************************************** Here is the pipeline: --------------------- "filesrc location=FILE ! decodebin ! audioconvert ! audio/x-raw ! appsink name=sink sync=FALSE" And here is my read function (simplified): ------------------------------------------ void read(gchar *lBuffer, guint64 nStartFrame, guint64 nFramesToRead, GstAudioInfo *pAudioInfo, GstPipeline *pPipeline) { guint64 nBytesToRead = nFramesToRead * pAudioInfo->bpf; guint64 nBytesRead = 0; // We need this temp buffer because of the offset thing gchar *lBytes = g_malloc(nBytesToRead + 8192); guint nOffsetBytes = 0; //We are paused here gint64 nTime = GST_FRAMES_TO_CLOCK_TIME(nStartFrame, pAudioInfo->rate); gst_element_seek_simple(GST_ELEMENT_CAST(pPipeline), GST_FORMAT_TIME, GST_SEEK_FLAG_ACCURATE | GST_SEEK_FLAG_FLUSH, nTime); gst_element_get_state(GST_ELEMENT_CAST(pPipeline), NULL, NULL, GST_CLOCK_TIME_NONE); gst_element_query_position(GST_ELEMENT_CAST(pPipeline), GST_FORMAT_TIME, &nTime); // Very important to continue where we left off g_assert(GST_CLOCK_TIME_TO_FRAMES(nTime, pAudioInfo->rate) == nStartFrame); gst_element_set_state(GST_ELEMENT_CAST(pPipeline), GST_STATE_PLAYING); GstAppSink *pAppSink = GST_APP_SINK_CAST(gst_bin_get_by_name(GST_BIN_CAST(pPipeline), "sink")); while (!gst_app_sink_is_eos(pAppSink)) { GstSample *pSample = gst_app_sink_pull_sample(pAppSink); if (pSample) { GstBuffer *pBuffer = gst_sample_get_buffer(pSample); gint64 nOfset = GST_CLOCK_TIME_TO_FRAMES(pBuffer->pts, pAudioInfo->rate); gint64 nWantOffset = (nBytesRead / pAudioInfo->bpf) + nStartFrame; if (nOfset > nWantOffset) { //WAV and FLAC never get here g_error("Bad pBuffer->pts"); } else if (nOfset < nWantOffset && nOffsetBytes == 0) { // FLAC reports multiple bad PTSs, but only the first one must be corrected nOffsetBytes = (nWantOffset - nOfset) * pAudioInfo->bpf; } GstMapInfo pMapInfo; gboolean bSuccess = gst_buffer_map(pBuffer, &pMapInfo, GST_MAP_READ); if (bSuccess) { memcpy(lBytes + nBytesRead, pMapInfo.data, pMapInfo.size); nBytesRead += pMapInfo.size; gst_buffer_unmap(pBuffer, &pMapInfo); } else { g_error("gst_buffer_map failed"); } gst_sample_unref(pSample); } else { g_error("gst_app_sink_pull_sample failed"); } // Normally, it would be: if (nBytesRead == nBytesToRead) if (nBytesRead >= nBytesToRead + nOffsetBytes) { break; } } gst_object_unref(pAppSink); gst_element_set_state(GST_ELEMENT_CAST(pPipeline), GST_STATE_PAUSED); memcpy(lBuffer, lBytes + nOffsetBytes, nBytesToRead); g_free(lBytes); } _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: Wrong PTS in AppSink and duplicate frames
|
|
Le samedi 11 avril 2020 à 00:33 +0200, Robert Tari a écrit :
> Hello everyone, > > I've been struggling with this for months now. I've looked in all the > GStreamer docs, read the whole Internet, but couldn't find a cure. I > would really be grateful for some help. Thanks in advance. > > Here is what I'm trying to accomplish (simplified): > > - seek to an exact decoded raw audio frame position > - read a number of frames > - (do some processing) > - seek exactly past the previously read frames > - read some more frames > - (and so on) > > What I'm expecting to see: > - A contiguous buffer seamlessly memcpy'd together from AppSink's > buffers, that is basically a perfect copy of the decoded raw audio data. > This works OK for WAV files. > > This is what happens instead: > - For FLAC: after a seek, the first GstSample that I pull from AppSink > is always offset by exactly 1 frame, so I get a duplicate between the > end of the previous call to my read function and the current one > - For various other formats: most GstSamples' GstBuffer->pts are either > -1, 0, or +1 frames of their expected position - so I either get 1 > duplicate, get it right, or I'm 1 frame short > > I've got the same results with hundreds of various test files, GStreamer > 1.14.5 and 1.16.2, on both 32 and 64bit versions of Linux. It could be that the seeks are not accurate enough for you needs. Best is to look in the appropriate parsers to find out if the seek is effectively accurate for this codec or container. You may be able to improve it, or not, since there exist formats that makes it impossible, unless.... Unless you decode from the start and skip to the accurate position you want. If that level of accuracy is required for you case, this might be what you want to do. It's of course much slower. > > > ***************************************************************************************** > > > Here is the pipeline: > --------------------- > > "filesrc location=FILE ! decodebin ! audioconvert ! audio/x-raw ! > appsink name=sink sync=FALSE" > > > And here is my read function (simplified): > ------------------------------------------ > > void read(gchar *lBuffer, guint64 nStartFrame, guint64 nFramesToRead, > GstAudioInfo *pAudioInfo, GstPipeline *pPipeline) > { > guint64 nBytesToRead = nFramesToRead * pAudioInfo->bpf; > guint64 nBytesRead = 0; > // We need this temp buffer because of the offset thing > gchar *lBytes = g_malloc(nBytesToRead + 8192); > guint nOffsetBytes = 0; > > //We are paused here > > gint64 nTime = GST_FRAMES_TO_CLOCK_TIME(nStartFrame, > pAudioInfo->rate); > gst_element_seek_simple(GST_ELEMENT_CAST(pPipeline), > GST_FORMAT_TIME, GST_SEEK_FLAG_ACCURATE | GST_SEEK_FLAG_FLUSH, nTime); > gst_element_get_state(GST_ELEMENT_CAST(pPipeline), NULL, NULL, > GST_CLOCK_TIME_NONE); > > gst_element_query_position(GST_ELEMENT_CAST(pPipeline), > GST_FORMAT_TIME, &nTime); > > > // Very important to continue where we left off > g_assert(GST_CLOCK_TIME_TO_FRAMES(nTime, pAudioInfo->rate) == > nStartFrame); > > gst_element_set_state(GST_ELEMENT_CAST(pPipeline), > GST_STATE_PLAYING); > GstAppSink *pAppSink = > GST_APP_SINK_CAST(gst_bin_get_by_name(GST_BIN_CAST(pPipeline), "sink")); > > while (!gst_app_sink_is_eos(pAppSink)) > { > GstSample *pSample = gst_app_sink_pull_sample(pAppSink); > > if (pSample) > { > GstBuffer *pBuffer = gst_sample_get_buffer(pSample); > gint64 nOfset = GST_CLOCK_TIME_TO_FRAMES(pBuffer->pts, > pAudioInfo->rate); > gint64 nWantOffset = (nBytesRead / pAudioInfo->bpf) + > nStartFrame; > > if (nOfset > nWantOffset) > { > //WAV and FLAC never get here > g_error("Bad pBuffer->pts"); > } > else if (nOfset < nWantOffset && nOffsetBytes == 0) > { > // FLAC reports multiple bad PTSs, but only the first > one must be corrected > nOffsetBytes = (nWantOffset - nOfset) * pAudioInfo->bpf; > } > > GstMapInfo pMapInfo; > gboolean bSuccess = gst_buffer_map(pBuffer, &pMapInfo, > GST_MAP_READ); > > if (bSuccess) > { > memcpy(lBytes + nBytesRead, pMapInfo.data, > pMapInfo.size); > nBytesRead += pMapInfo.size; > gst_buffer_unmap(pBuffer, &pMapInfo); > } > else > { > g_error("gst_buffer_map failed"); > } > > gst_sample_unref(pSample); > } > else > { > g_error("gst_app_sink_pull_sample failed"); > } > > // Normally, it would be: if (nBytesRead == nBytesToRead) > if (nBytesRead >= nBytesToRead + nOffsetBytes) > { > break; > } > } > > gst_object_unref(pAppSink); > gst_element_set_state(GST_ELEMENT_CAST(pPipeline), > GST_STATE_PAUSED); > > memcpy(lBuffer, lBytes + nOffsetBytes, nBytesToRead); > g_free(lBytes); > } > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
Thanks for the answer.
If you look at my code sample you can see I have this check: g_assert(GST_CLOCK_TIME_TO_FRAMES(nTime, pAudioInfo->rate) == nStartFrame); It never fails. If it was the first buffer after the seek that has the wrong PTS, I could simply offset my temp buffer (as I do with FLAC right now), and read the rest. The problem is after the seek: when repeatedly pulling AppSink buffers, they do not always follow the previous pull. If I read the complete file from position 0 into the AppSink, I have the same issue: some AppSink buffers do not start where previously pulled buffer ended. By the way, I tried replacing AppSink with GStreamSink and reading whole files into a buffer: the output is 100% identical to the AppSink method: a few frames are duplicated, and a few are overlapped. Maybe there is an element to add to the pipeline, or a property that could control this behaviour? On 2020-04-12 03:36, Nicolas Dufresne wrote: > Le samedi 11 avril 2020 à 00:33 +0200, Robert Tari a écrit : >> Hello everyone, >> >> I've been struggling with this for months now. I've looked in all the >> GStreamer docs, read the whole Internet, but couldn't find a cure. I >> would really be grateful for some help. Thanks in advance. >> >> Here is what I'm trying to accomplish (simplified): >> >> - seek to an exact decoded raw audio frame position >> - read a number of frames >> - (do some processing) >> - seek exactly past the previously read frames >> - read some more frames >> - (and so on) >> >> What I'm expecting to see: >> - A contiguous buffer seamlessly memcpy'd together from AppSink's >> buffers, that is basically a perfect copy of the decoded raw audio >> data. >> This works OK for WAV files. >> >> This is what happens instead: >> - For FLAC: after a seek, the first GstSample that I pull from AppSink >> is always offset by exactly 1 frame, so I get a duplicate between the >> end of the previous call to my read function and the current one >> - For various other formats: most GstSamples' GstBuffer->pts are >> either >> -1, 0, or +1 frames of their expected position - so I either get 1 >> duplicate, get it right, or I'm 1 frame short >> >> I've got the same results with hundreds of various test files, >> GStreamer >> 1.14.5 and 1.16.2, on both 32 and 64bit versions of Linux. > > It could be that the seeks are not accurate enough for you needs. Best > is to look in the appropriate parsers to find out if the seek is > effectively accurate for this codec or container. You may be able to > improve it, or not, since there exist formats that makes it impossible, > unless.... > > Unless you decode from the start and skip to the accurate position you > want. If that level of accuracy is required for you case, this might be > what you want to do. It's of course much slower. > >> >> >> ***************************************************************************************** >> >> >> Here is the pipeline: >> --------------------- >> >> "filesrc location=FILE ! decodebin ! audioconvert ! audio/x-raw ! >> appsink name=sink sync=FALSE" >> >> >> And here is my read function (simplified): >> ------------------------------------------ >> >> void read(gchar *lBuffer, guint64 nStartFrame, guint64 nFramesToRead, >> GstAudioInfo *pAudioInfo, GstPipeline *pPipeline) >> { >> guint64 nBytesToRead = nFramesToRead * pAudioInfo->bpf; >> guint64 nBytesRead = 0; >> // We need this temp buffer because of the offset thing >> gchar *lBytes = g_malloc(nBytesToRead + 8192); >> guint nOffsetBytes = 0; >> >> //We are paused here >> >> gint64 nTime = GST_FRAMES_TO_CLOCK_TIME(nStartFrame, >> pAudioInfo->rate); >> gst_element_seek_simple(GST_ELEMENT_CAST(pPipeline), >> GST_FORMAT_TIME, GST_SEEK_FLAG_ACCURATE | GST_SEEK_FLAG_FLUSH, nTime); >> gst_element_get_state(GST_ELEMENT_CAST(pPipeline), NULL, NULL, >> GST_CLOCK_TIME_NONE); >> >> gst_element_query_position(GST_ELEMENT_CAST(pPipeline), >> GST_FORMAT_TIME, &nTime); >> >> >> // Very important to continue where we left off >> g_assert(GST_CLOCK_TIME_TO_FRAMES(nTime, pAudioInfo->rate) == >> nStartFrame); >> >> gst_element_set_state(GST_ELEMENT_CAST(pPipeline), >> GST_STATE_PLAYING); >> GstAppSink *pAppSink = >> GST_APP_SINK_CAST(gst_bin_get_by_name(GST_BIN_CAST(pPipeline), >> "sink")); >> >> while (!gst_app_sink_is_eos(pAppSink)) >> { >> GstSample *pSample = gst_app_sink_pull_sample(pAppSink); >> >> if (pSample) >> { >> GstBuffer *pBuffer = gst_sample_get_buffer(pSample); >> gint64 nOfset = GST_CLOCK_TIME_TO_FRAMES(pBuffer->pts, >> pAudioInfo->rate); >> gint64 nWantOffset = (nBytesRead / pAudioInfo->bpf) + >> nStartFrame; >> >> if (nOfset > nWantOffset) >> { >> //WAV and FLAC never get here >> g_error("Bad pBuffer->pts"); >> } >> else if (nOfset < nWantOffset && nOffsetBytes == 0) >> { >> // FLAC reports multiple bad PTSs, but only the first >> one must be corrected >> nOffsetBytes = (nWantOffset - nOfset) * >> pAudioInfo->bpf; >> } >> >> GstMapInfo pMapInfo; >> gboolean bSuccess = gst_buffer_map(pBuffer, &pMapInfo, >> GST_MAP_READ); >> >> if (bSuccess) >> { >> memcpy(lBytes + nBytesRead, pMapInfo.data, >> pMapInfo.size); >> nBytesRead += pMapInfo.size; >> gst_buffer_unmap(pBuffer, &pMapInfo); >> } >> else >> { >> g_error("gst_buffer_map failed"); >> } >> >> gst_sample_unref(pSample); >> } >> else >> { >> g_error("gst_app_sink_pull_sample failed"); >> } >> >> // Normally, it would be: if (nBytesRead == nBytesToRead) >> if (nBytesRead >= nBytesToRead + nOffsetBytes) >> { >> break; >> } >> } >> >> gst_object_unref(pAppSink); >> gst_element_set_state(GST_ELEMENT_CAST(pPipeline), >> GST_STATE_PAUSED); >> >> memcpy(lBuffer, lBytes + nOffsetBytes, nBytesToRead); >> g_free(lBytes); >> } >> _______________________________________________ >> gstreamer-devel mailing list >> [hidden email] >> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel > > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
«
Return to GStreamer-devel
|
1 view|%1 views
| Free forum by Nabble | Edit this page |