Bug 918 - FFMPEGMediaPlayer -vid -2 fail to decode and queue audio packets
Summary: FFMPEGMediaPlayer -vid -2 fail to decode and queue audio packets
Alias: None
Product: Jogl
Classification: JogAmp
Component: util (show other bugs)
Version: 2
Hardware: All all
: --- enhancement
Assignee: Sven Gothel
Depends on:
Reported: 2013-12-03 19:43 CET by Xerxes Rånby
Modified: 2013-12-11 21:14 CET (History)
0 users

See Also:
Type: ---
SCM Refs:
c6ab1bc932d9a0b3897b24d289e4d561bf8bb65c a1be0f69bacb315e40a017b8997ef1c610da576e 8a8ed735f6631b2da7bf605c5c3dda4e0fc13905 d42b236ef139dcc8d8713535893c7870b55d420c a0b23035bf9de987f5ad1b00fe3630c102f4e513
Workaround: ---


Note You need to log in before you can comment on or make changes to this bug.
Description Xerxes Rånby 2013-12-03 19:43:19 CET
The current FFMPEGMediaPlayer implementation call the native packet decoder natives.readNextPacket0 for both audio and video during getNextTexture(GL)
this implementation impede use of the FFMPEGMediaPlayer GLMediaPlayer backend for audio only playback when getNextTexture(GL) is not called periodically.

FFMPEGMediaPlayer should decode audio and video packets in a separate thread during play state in order to support audio only use. By making the decoder multithreaded may as well help reduce latency by making sure that both video and audio frames are queued up in time before playback.
Comment 1 Xerxes Rånby 2013-12-03 21:40:06 CET
It is possible to reproduce this bug using the MovieSimple test by simply passing -vid -2 and produces no audio output for audio only playback.

passing -vid -1 work and produce audio output for audio only playback.

-2 == GLMediaPlayer.STREAM_ID_NONE == OFF
-1 == AUTO

java -cp jogamp-all-platforms/jar/jogl-test.jar:jogamp-all-platforms/jar/gluegen-rt.jar:jogamp-all-platforms/jar/jogl-all.jar:jogamp-all-platforms/jar/joal.jar com.jogamp.opengl.test.junit.jogl.demos.es2.av.MovieSimple -vid -2
Comment 2 Xerxes Rånby 2013-12-04 22:02:35 CET
Main issue:
StreamWorker did not get created and started for -vid -2 
inside src/jogl/classes/jogamp/opengl/util/av/GLMediaPlayerImpl.java
textureCount = 0; for -vid -2

StreamWorker is responsible for decoding both audio and video in a separate thread.

TODO: review StreamWorker for -vid -2
TODO: review StreamWorker, GLMediaPlayer and FFMPEGMediaPlayer when to stop using an audio only stream. The current implementation decide to stop when it encounter an end of stream _video_ frame.

Suggested fix:

diff --git a/src/jogl/classes/jogamp/opengl/util/av/GLMediaPlayerImpl.java b/src
index 86e19c9..7bc61b7 100644
--- a/src/jogl/classes/jogamp/opengl/util/av/GLMediaPlayerImpl.java
+++ b/src/jogl/classes/jogamp/opengl/util/av/GLMediaPlayerImpl.java
@@ -528,7 +528,7 @@ public abstract class GLMediaPlayerImpl implements GLMediaPl
             this.vid = vid;
             this.aid = aid;
             if ( this.streamLoc != null ) {
-                if( TEXTURE_COUNT_MIN < textureCount ) {
+                if( streamWorker == null ) {
                     streamWorker = new StreamWorker();
                 } else {
                     new Thread() {
Comment 3 Sven Gothel 2013-12-11 06:22:02 CET
    Bug 918 (1/2): Use StreamWorker in 'Audio Only' mode, since no 'getNextTexture(..)' is issued here!
    Thanks to Xerxes to analyze this issue thoroughly.
    TODO: Implement EOS for 'Audio Only' and test seek, pause, etc .. - Apply manual tests in MovieSimple
Comment 4 Sven Gothel 2013-12-11 21:14:45 CET
    GLMediaPlayer: Fix Deadlock if EOS happens after pause/resume (seek) - 
   Tested w/ seeking 'Audio Only' and Matroska
    Test stream was default of MovieSimple:
    while disabling video (-vid -2)

    Determine StreamWorker usage after init
      - To support audio only files, we need to determine to use StreamWorker
        after completion of stream-init.
    Fix seek(..)
      - FFMPeg: pos0 needs to use aPTS for audio-only
      - Clip target time [0..duration[
    Fallback for EOS Detection
      In case the backend does not report proper EOS:
      - Utilize 'nullFramesCount >= MAX' -> EOS,
        where MAX is number of frames for 3s play duraction
        and where 'nullFramesCount' is increased if no valid packet is available
        and no decoded-video or -audio in the queue.
      - Utilize pts > duration -> EOS
    MovieSimple uses full GLEventListener for 'Audio Only' as well to test seek
      - Matroska seek for audio-only leads to EOS ..
      - MP4 audio-only seek works
      - Use audio-pts in audio-only to calc target time
      - A, V and A+V
      - Pause, Stop and Seek
      - GNU/Linux

    ALAudioSink: Fix stop and flush of OpenAL source
    - stopImpl() shall always issue alSourceStop(..) if state is not STOPPED
    - Remove 'flush' hint for dequeueBuffer(..), we perform proper flush in respective method, see below
    - flush() needs to issue:
       - stopImpl() - which should already dequeue all buffers
       - Explicitly dequeue all buffers: via 'alSourcei(alSource[0], AL.AL_BUFFER, 0)'
       - Then dequeue manually processed buffers: dequeueBuffer( false /* wait */ );
       - And dequeue _all_ buffers: dequeueForceAll();

    Fix EOS Regression: Only use pts>duration for EOS if duration > 0 
   (camera or other sources may not have duration)
    Regression of commit 8a8ed735f6631b2da7bf605c5c3dda4e0fc13905

Tested on Android as well ..