Changes

Fix GLES3/GLES2 relation / Use bouml 4.21.1 (Note: SVG fonts .. are too tiny, using png)

Use uml's png files (svg font is broken)

GLContext: Refine API doc (CONTEXT_NOT_CURRENT.. values) / Exception message

GLDrawableFactory: Expose createDummyDrawable(..) / GLDrawableFactoryImpl: Make impl. methods final.

createDummyDrawable(..) is useful for efficient shared context w/o actually rendering to this dummy drawable's framebuffer

GLFBODrawableImpl.initialize(): Unrealize when already unrealized shall not throw an exception - only dump DEBUG info!

MouseEvent API doc: Add W3C PointerEvent Reference

ES2 NEWT Tests: GearsES2/RedSquareES2 - Don'w use AWTRobot if AWT is n/a (Ji Gong manual testing)

GLMediaPlayer: Add multithreaded decoding w/ textureCount > 2 where available EGL/FFMPeg. WIP!

Off-thread decoding:
  If validated (impl) textureCount > 2, decoding happens on extra thread.
  If decoding requires GL context, a shared context is created for decoding thread.

API Changes:
- initGLStream(..): Adds 'textureCount' as argument.
- TextureSequence.TexSeqEventListener.newFrameAvailable(..) exposes the new frame available
- TextureSequence.TextureFrame exposes the PTS (video)

Implementation:

- 'int validateTextureCount(int)': implementation decides whether textureCount can be > 2, i.e. off-thread decoding allowed,
  default is NO w/ textureCount==2!

- 'boolean requiresOffthreadGLCtx()': implementation decides whether shared context is required for off-thread decoding

- 'syncFrame2Audio(TextureFrame frame)': implementation shall handle a/v sync, due to audio stream details (pts, buffered frames)

- FFMPEGMediaPlayer extends GLMediaPlayerImpl, no more EGLMediaPlayerImpl (redundant)

+++

- SyncedRingbuffer: Expose T[] array

+++

TODO:
 - syncAV!
 - test Android

Fix regression of commit 6332e13b2f0aa9818d37802302f04c90a4fa4239 causing optional OMX to fail to compile

SyncedRingbuffer: peek() don't throw an exception

GLMediaPlayer Multithreaded Decoding: AudioSink (Part-2) - WIP

- AudioSink.AudioDataFormat
  - add fixedP (fixed-point or floating-point)

- AudioSink
  - rename 'buffer count' to 'frame count'
  - add setPlaySpeed(..)
  - add isPlaying()
  - add play()
  - add pause()
  - add flush()
  - add: getFrameCount(), getQueuedFrameCount(), getFreeFrameCount(), getEnqueuedFrameCount(),
  - rename: writeData() -> enqueueData(..)

- ALAudioSink
  - multithreaded usage

  - make ALCcontext current per thread, now required for multithreaded use
    Use RecursiveLock encapsulating the ALCcontext's makeCurrent/release/destroy,
    since the native operations seem to be buggy.
    NOTE: Think about adding these general methods to ALCcontext

  - implement new methods

-

GLMediaPlayer Multithreaded Decoding: GLMediaPlayer* (Part-3) - WIP

- GLMediaPlayer
  - Remove State.Stopped and method stop() - redundant, use pause() / destroy()
  - Add notion of stream IDs
  - Add API doc: State / Stream-ID incl. html-anchor
  - Expose video/audio PTS, ..
  - Expose optional AudioSink
  - Min multithreaded textureCount is 4 (EGL* and FFMPEG*)

- GLMediaPlayerImpl
  - Move AudioSink rel. impl. to this class,
    allowing a tight video implementation reusing logic.

  - Remove 'synchronized' methods, synchronize on State
    where applicable

  - implement new methods (see above)

  - playSpeed is handled partially in AudioSink.
    If it exeeds AudioSink's capabilities, drop audio and rely solely on video sync.

  - video sync (WIP)
    - video pts delay based on geometric weight
    - reset video SCR if 'out of range', resync w/ PTS
    -

  - FramePusher
    - allow interruption when pausing/stopping,
      while waiting for next avail free frame to decode.

- FFMPEGMediaPlayer
  - Add proper AudioDataFormat negotiation AudioSink <-> libav
  - Parse libav's SampleFormat
  - Remove AudioSink interaction (moved to GLMediaPlayerImpl)

- Tests (MovieSimple, MovieCube):
  - Add aid/vid selection
  - Add KeyListener for actions: seek(..), play()/pause(), setPlaySpeed(..)
  - Dump perf-string each 2s

- TODO:
  - Add audio sync in AudioSink, similar to GLMediaPlayer's weighted video delay,
    here: drop audio frames.

Fix Bug 815: GL*: Change glIs<Buffer>Enabled() -> glIs<Buffer>Bound() to reflect semanics - Also fix the exception message (enabled/disabled -> bound/unbound)

Reason of change: Avoid confusion and point to the cause!

API change:
  glIsVBOArrayEnabled() -> glIsVBOArrayBound()
  glIsVBOElementArrayEnabled() -> glIsVBOElementArrayBound()
  glIsPBOPackEnabled() -> glIsPBOPackBound()
  glIsPBOUnpackEnabled() -> glIsPBOUnpackBound()

Exception message change:
  "must be enabled to call this method" -> "must be bound to call this method"
  "must be disabled to call this method" -> "must be unbound to call this method"

SyncedRingbuffer: Add 'reset(boolean full)', simplify 'clear(..)'.

'reset(boolean full)' enables user to reset ringbuffer pointer and assume it's empty or full,
while 'clear()' shall only remove all references .. etc.

Provide unit test for Bug 817

Refine unit test Bug 817 c19f9916bc59765e5aaf307b8ecd5d25dfcad6b6 / Add pbuffer and fbo testing

Fix Bug 817 (1/2): GLPixelAttributes checks arguments and queried bytesPerPixel

GLPixelAttributes checks arguments (componentCount, format / type)
and the queried bytesPerPixel.

Fix Bug 817 (2/2): GLContextImpl's getDefaultPixelDataType()/getDefaultPixelDataFormat() use defaults (fix)

GLContextImpl's getDefaultPixelDataType()/getDefaultPixelDataFormat()
uses default values if GL query fails.

SyncedRingbuffer Cleanup: private fields, clarify reset(boolean)

GLMediaPlayer Multithreaded Decoding: GLMediaPlayer* (Part-4) - WIP

- Use Platform.currentTimeMillis() for accurate timing!

- GLMediaPlayer / GLMediaPlayerImpl
  - Add DEBUG_NATIVE property jogl.debug.GLMediaPlayer.Native
    for verbose impl. messages, i.e. ffmpeg/libav

  - Add 'synchronization' section in GLMediaPlayer API doc (WIP)

  - Use passive non-blocking video synchronization,
    i.e. repeat frames instead of 'sleep'.
    Thx to Xerxes's suggestion.

  - Add flushing of cached decoded frames,
    allowing to remove complicated 'videoSCR_reset_latch'

  - FramePusher (threaded decoding):
    - Always create a shared context!
    - Release context while pausing
    - Pre/post 'getNextTextureImpl()' actions only
      at makeCurrent/release.
    - newFrameAvailable(..) signal after decoded frame is enqueued

- FFMPEGDynamicLibraryBundleInfo
  - Bind add. functions of libavcodec:
    +         "av_init_packet",
    +         "av_new_packet",
    +         "av_destruct_packet",
  - Bind add. functions of libavformat:
    +         "avformat_seek_file",
    +         "av_read_play",
    +         "av_read_pause",
  - DEBUG property := FFMPEGMediaPlayer.DEBUG || DynamicLibraryBundleInfo.DEBUG;

- FFMPEGMediaPlayer
  - Use libavformat's 'av_read_play()' and 'av_read_pause()',
    which may get utilized for network streams, e.g. RTSP

  - getNextTextureImpl(..):
    - Fix retry loop
    - Use postNextTextureImpl/preNextTextureImpl if desired (PSM)

  - Native:
    - Use fixed my_av_q2i32(..) macro (again)
    - Use INVALID_PTS marker (synced w/ Java code)
    - DEBUG: Dump more detailed frame information
    - TODO: Consider passing frame_delay, especially for repeated frames!

- Tests (MovieSimple, MovieCube):
  - Refine KeyEvents control for seek and speed.

- TODO:
  - Proper audio clock calculation - difficult w/ OpenAL !
  - Video / Audio sync:
    - seek !
    - streams w/ very async A/V frames
    - Test Streams:
      - Five-minute-sync-test.mp4
      - Audio-Video-Sync-Test-Calibration-23.98fps-24fps.mp4
      - sound_in_sync_test.mp4
      - big_buck_bunny_1080p_surround.avi

GLMediaPlayerImpl: Refine getNextTexture(..) DEBUG output, put 'last SCR delay' in regular println.

GLMediaPlayer: Use URI instead of URL / Misc refinements

- GLMediaPlayer: Use URI instead of URL, allowing passing a non resolved location
  - Java's URL doesn't allow 'other' protocols, i.e. RTSP

- GLMediaPlayer: Add Table of test streams and their location ..

- FFMPEGMediaPlayer
  - Handle av_read_play/pause response on java side, ignore error - simply dump in DEBUG_NATIVE mode

SyncedRingbuffer moved to GlueGen, commit 30475c6bbeb9a5d48899b281ead8bb305679028d

AudioSink: Add END_OF_STREAM_PTS, initSink(..) args: frameGrowAmount and frameLimit allowing an optional used Ringbuffer to grow in implementation.

Texture: Add 'toString()' and 'getTextureObject()' w/o GL instance to read existing object name.

TextureSequence: Add END_OF_STREAM_PTS, remove 'blocking' from getNextTexture(..), may blocks .. or not, depending on implementation and state.

GLMediaPlayer Multithreaded Decoding: GLMediaPlayer* (Part-5) - WIP

- Update/fix GLMediaPlayer API doc

- GLMediaEventListener: Add event bits for all state changes to be delivered via attributesChanged(..)

- StreamWorker / Decoder Thread:
  - Use StreamWorker only !
  - Handle exceptions on StreamWorker via StreamException
  - Handles stream initialization and decoding (-> initStream(..))

- Split initGLStream(..) -> initStream(..) + initGL(GL)
  - allow initStream(..)'s implementation being executed on StreamWorker
  - allow GL initialization to be 'postponed' when stream is read,
    i.e. non blocking stream initialization (UI .. etc)

- Handle EOS via END_OF_STREAM_PTS -> pause/event

- Video: Use lock-free LFRingbuffer, similar to
  ALAudioSink (commit f18a94b3defef16e98badd6d99f2422609aa56c5)

+++

- FFMPEGDynamicLibraryBundleInfo
 - Add avcodec's:
     - avcodec_get_frame_defaults, avcodec_free_frame (54.28.0), avcodec_flush_buffers,
 - Add avutil's:
     - av_frame_unref (55.0.0)
 - Add avformat's:
     - avformat_seek_file (??)

+++

- FFMPEGMediaPlayer Native:
  - add 'snoop' video frames for a/v frame count relation.
    disabled per default, since no more needed due to ALAudioSink's
    grow-buffer usage of LFRingbuffer.

  - use sp_avcodec_free_frame if available

  - 'useRefCountedFrames=1' for libav 55.0 to cache more than one audio frame,
     not used since ALAudioSink's OpenAL usage does not require it (copies data once).
     Note: the above snooped-video frame count is used here.

  - use only one cached audio-frame (-> see above, OpenAL copies data once),
    while reusing the NIO buffer!

  - Perform OpenGL sync (glFinish) in native code!

  - find proper PTS value, i.e. either frame's PTS or DTS,
    see 'PTSStats'.

- FFMPEGMediaPlayer Java:
  - use private fields

  - simplified code due to above changes.

+++

Working Tests: MovieSimple and MovieCube

TODO-1: Fix
  - Android
  - OMXGLMediaPlayer

TODO-2:
  - Fix issue where async audio frames arrive much later than 1st video frame, i.e. around 300ms.
  - Default TextureCount .. maybe 3 ?
  - Adding Audio synchronization ?
  - Find 'truth' about correlation of audio and video PTS values,
    currently, we assume both to be unrelated ?

FFMPEGMediaPlayer: Transform URI spaces '%20' to ' ' manually, libav doesn't work well w/ URI encoded names.

JOGL/Util: Add generic 'TimeFrameI' for key frame animation of 'things' basen on integer milliseconds.

*AudioSink: Add byte/time calc to AudioDataFormat, *AudioFrame OO changes, Reuses ALAudioFrames to ease GC, Ringbuffer changes

- Adapt to Ringbuffer changes of GlueGen commit f9f881e59c78e3036cb3f956bc97cfc3197f620d

- Favor AudioSink 'AudioFrame enqueueData(int pts, ByteBuffer bytes, int byteCount)',
  - Impl. shall reuse AudioFrame's instead of creating them on the fly
  - User shall simply pass the net data required, while receiving an internal AudioFrame

- Add byte/time calc to AudioDataFormat:
  - Add getDuration(byteCount) and getByteCount(ms).

- *AudioFrame OO changes:
  - abstract AudioFrame extends TimeFrameI
  - allow setting of all components to reuse instanced (GC clean)

- ALAudioSink reuses ALAudioFrames to ease GC:
  - Remove creating temporary objects to ease GC

  - ALAudioFrame holds ALBuffer name, remove ActiveBuffer type.

  - Use ALAudioFrame similar to TextureFrame in GLMediaPlayerImpl,
    i.e. fill them in 'full' Ringbuffer and move them in-between 'full'/'playing' Ringbuffer.

  -

GLMediaPlayer Multithreaded Decoding: GLMediaPlayer* (Part-6) - DONE

Multithreaded decoding and API should be considered stable by now,
minor changes may apply if Android/OMX impl. requires it.

We still need to solve TODO's as listed below, copied from 474ce65081ecd452215bc07ab866666cb11ca8b1.

+++

- *TextureFrame OO changes:
  - TextureFrame extends TimeFrameI

- GLMediaPlayerImpl*
  - Adapt to Ringbuffer changes of GlueGen commit f9f881e59c78e3036cb3f956bc97cfc3197f620d

  - Fix impl. method's API doc
    - getNextTextureImpl(..) returns video PTS

  - Fix audio-only playback

  - frame dropping shall only happen if:
     - previous frame has not been dropped
     - frame is too later
     - one decoded frame is already available

  - Don't block for decoder anymore:
     - nextFrame = "videoFramesDecoded.getBlocking() -> videoFramesDecoded.get()";
     No 'next decoded frame avail' only could mean:
       - slow decoding/hardware
       - slow transport
     hence we shall not block rendering.

  - Add DEBUG output if using last frame

  - Add integer property 'jogl.debug.GLMediaPlayer.StreamWorker.delay' in milliseconds
    to simulate slow decoding, i.e. delay is added in StreamWorker after decoding
    before pushing new frame to Ringbuffer.

  - FFMPEGMediaPlayer:
    - audioFrameLimitWithVideo 128 -> 64
    - audioFrameLimitAudioOnly 128 -> 32
    - uses AudioSink's 'enqueueData(int pts, ByteBuffer bytes, int byteCount)'
    - fixes for audio-only playback

+++

Working Tests: MovieSimple and MovieCube

TODO-1: Fix
  - Android
  - OMXGLMediaPlayer

TODO-2:
  - Fix issue where async audio frames arrive much later than 1st video frame, i.e. around 300ms.
  - Default TextureCount .. maybe 3 ?
  - Adding Audio synchronization ?
  - Find 'truth' about correlation of audio and video PTS values,
    currently, we assume both to be unrelated ?

AudioSink.init(..) abstract 'frame count' -> duration [ms] allowing non-frame based AudioSink's to deal w/ desired queue sizes.

- Rename AudioSink.initSink(..) -> AudioSink.init(..)

- Move: "int initialFrameCount, int frameGrowAmount, int frameLimit" to
        "int initialQueueSize, int queueGrowAmount, int queueLimit"
  based on milliseconds instead of frame count.

- Passing hint 'float frameDuration' to calculate frame count for fame based audio sink, i.e. ALAudioSink.

- Adding sensible static final default values

- AudioDataFormat: Add convenient conversion routines (samples/bytes/frame-count)

- FFMPEGMediaPlayer: Retrieve audio frame size in samples per channel, pass it to AudioSink.init(..)
  to properly calculate frame count/limits based on duration.

GLMediaPlayer/AudioSink: Add set[Audio]Volume(float v)/get[Audio]Volume() allowing to change the audio volume.

AndroidGLMediaPlayerAPI14: Fix implementation to coop w/ threaded decoder / Add EOS detection, setAudioVolume(..)

GLMediaPlayerImpl.initStreamGL(..):
  Only require a minimum texture count of 2,
  which is the bare minimum to allow our algorithm to work,
  i.e. having a 'lastFrame' and avail/playing ringbuffer have each one frame.

Android's MediaPlayer API can only deal w/ one SurfaceTexture,
hence we have to fake a second SurfaceTextureFrame w/ same content
to allow our implementation to work w/ the threaded decoder (min 2 frames).

GLAutoDrawableBase: DEBUG code - Avoid NPE

GLES3Impl: Add NOP 'finalizeInit()' method, to avoid GLContentImpl not finding it and dumping a DEBUG message

libav/ffmpeg: Move current lavc53.lavf53.lavu51 (version 8) header files to their sub-folders; Add lavc54.lavf54.lavu52 (version 9) header files

lavc54.lavf54.lavu52:
  - Begin testing w/ version 9
  - Header files are binary incompatible (data types)

Default is still lavc53.lavf53.lavu51

NullGLMediaPlayer: Fix reported VID (fake 0), no AID, textureCount == 2

libav/ffmpeg: Prepare for lavc54.lavf54.lavu52

- Add compile-time/runtime version check, fail if major versions do not match
  assuming binary incompatibility

- Add: 'av_find_input_format' for future video input support

- Manually map '/dev/video<NUM>' to video input - not working yet.
  - WINDOWS: Set file to '<NUM>'
  - Set input format string depending on OS

ffmpeg/libav: lavc54.lavf54.lavu52 -> lavc54.lavf54.lavu52.lavr01: Adding libavresample of version 9

ffmpeg/libav: remove unsued header files

AudioSink: Fixe type names ; Enhance AudioFormat negotiation ; ALAudioSink adds AL_SOFT_buffer_samples support w/ full AL caps

- Fixe type names:
  - Remove AudioDataType, we only support PCM here anyways
  - AudioDataFormat -> AudioFormat / Add 'planar' attribute to distingush packed/planar data type
  - Validate float types

- Enhance AudioFormat negotiation
  - Add 'isSupported(AudioFormat format)' which _shall_ be used before 'init(..)'
    to test/negotiate format
  - Add getMaxSupportedChannels(), which may be used w/ getPreferredFormat() if orig requested format fails
    via 'isSupported(..)'
  - 'init(..)' returns boolean only.

- ALAudioSink adds AL_SOFT_buffer_samples support w/ full AL caps
  - Determine whether AL_SOFT_buffer_samples is supported
  - Use new JOAL ALHelper to convert AudioFormat -> AL-types,
    which also answers the 'isSupported(..)' query.
  - Now allows multiple: channles, sample-types, etc.

libav/ffmpeg: version9: Add libavresample support ; Proper AudioFormat negotiation w/ AudioSink; Misc

- Add libavresample support
  - Resample if avail && (!AV_SAMPLE_FMT_S16 || !prefSampleRate || !sinkSupported)
  - Resample to: prefSampleRate (if set), AV_SAMPLE_FMT_S16 and min(channelCount, maxChannelCount)

- Proper AudioFormat negotiation w/ AudioSink;
  - Utilize AudioSink's 'isSupported(AudioFormat)'

- Misc
  - use 'av_get_bytes_per_sample(fmt)' always, don't assume 2

ffmpeg/libav: Remove 'dead' audio/video frame count relation snoop-code

libav/ffmpeg: Compile/Link 2 versions of native FFMPEGMediaPlayer methods FFMPEGNatives -> FFMPEGv08Natives + FFMPEGv09Natives

Enables FFMPEGMediaPlayer to work w/ either ffmpeg/libav version 8 or 9 w/ same JOGL binary

Same C source code is compiled against
  1: version 0.8  FFMPEGv08Natives lavc53.lavf53.lavu51
  2: version 0.9  FFMPEGv09Natives lavc54.lavf54.lavu52.lavr01

FFMPEGv08Natives and FFMPEGv09Natives implements FFMPEGNatives,
native C code uses CPP '##' macro concatenation to produce unique function names.

To enable 'cpp' to find the libav* header files matching the desired version,
we have placed them in the c-file's folder, issued '#include "path/file.h"
and added symbolic links to allow finding same module and 'sister modules':

ls -l libavformat/
..
lrwxrwxrwx 1 sven sven    13 Aug 26 12:56 libavcodec -> ../libavcodec
lrwxrwxrwx 1 sven sven    14 Aug 26 12:56 libavformat -> ../libavformat
lrwxrwxrwx 1 sven sven    12 Aug 26 12:57 libavutil -> ../libavutil
..

At static init FFMPEGDynamicLibraryBundleInfo, determines the runtime version
and instantiates the matching FFMPEGNatives, or null if non matches.

FFMPEGMediaPlayer still compares the compile-time and runtime versions.

FFMPEGMediaPlayer passes it's own instance to FFMPEGNatives for callbacks.

FFMPEGNatives*: Add missing license header

ALAudioSink: Remove force DEBUG info

GLMediaPlayer: Add camera input / FFMPEG: Fix 'av_packet' leak and add missing symbol 'av_realloc'.

- Add camera input
  - Use URI w/ scheme 'camera' to determine camera input is desired,
    use URI host as camera id.
    E.g. 'camera://0' for 1st camera.

  - AndroidGLMediaPlayerAPI14: Via 'Camera'

  - FFMPEG*: Via libavdevice, device name and input format

  - TODO: Add controls to manipulate camera if available

- FFMPEG*
  - Add symbols
    - avcodec_register_all
    - av_realloc (was missing)
    - avdevice_register_all

  - Load libavdevice (opt)

  - Camera:
    - Use <ID> (windows) and /dev/video<ID> other OS
    - simply find the input format in native code

  - Support YUYV422 (used in video4linux2, etc.)
    - Stuff 2x 16bpp (YUYV) into one RGBA pixel!
    - Add texture format for 16bpp
    - Add texture lookup shader

  - Fix av_packet leak in readNextImpl(..)
    - Restore orig pointer and size values,
      we may have moved along within packet.
      Then call av_free_packet().

  - Use null AudioSink if audio-id is NONE