Bug 1472 - GLMediaPlayer Enhance AV Synchronization
Summary: GLMediaPlayer Enhance AV Synchronization
Status: RESOLVED FIXED
Alias: None
Product: Jogl
Classification: JogAmp
Component: video (show other bugs)
Version: 2.6.0
Hardware: All all
: P4 major
Assignee: Sven Gothel
URL:
Depends on: 1473
Blocks:
  Show dependency treegraph
 
Reported: 2023-10-03 03:06 CEST by Sven Gothel
Modified: 2023-10-16 15:29 CEST (History)
0 users

See Also:
Type: DEFECT
SCM Refs:
jogl e066250b908f0f2187d468d2c5ab586341031753 joal 05529d4d6cd857e5bcd87be3b342c3fe7ba7655e gluegen 52725b4c6525487f93407f529dc0a758b387a4fc joal d3de587eae8ed8b5b5bc62647da0f95297144294 joal 12029f1ec1d8afa576e1ac61655f318cc37c1d16 jogl b1956113f5601b0cc6ac525d3918a0dfa8d240af jogl 2c80bb2e6eb12e155d747daf8a08362396d5e0fc jogl bd2aa98d57c0c03bbface35000ed1c4bac6470e2 gluegen 2a91bec43a7a58efc22dd36e919308ecc50d9b4a jogl ee29eaf770d0d3627e5989dcae58a87cf27184d3 jogl dac49f32e09d61f5ba9be53a21d351843a48761b jogl a35ca1763c773b4533b9b9850a9842522303ad01
Workaround: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sven Gothel 2023-10-03 03:06:55 CEST

    
Comment 1 Sven Gothel 2023-10-03 03:14:43 CEST
Culprit might be the ALAudioSink changes right before our 2.5.0 release.
Comment 2 Sven Gothel 2023-10-04 10:03:06 CEST
Pushed the regression part down to Bug 1473.

Regardless, GLMediaPlayerImpl's AV sync resolution may need refinement since some streams are not playing well.

+++

Notable, Bug 1473 also added better tracking of audio_pts in ALAudioSink,
but whether we can use it to align streams is another questions - see below.

+++

The current Sync Model 
=========================

It leaves audio untouched as it should be IMHO,
as a change in audio stream always leads to uncomfy disruptions.
OK.

It uses the system clock sync'ed on first video_pts as reference.
It may update this reference if it gets out of band too much.

Then code acts as usual:
A) video-diff > threshold (video in future): Repaint frame
B) video-diff > threshold (video too late): Drop one

Now case (A) may lead to disruption if happening too much 
and whole sync is "out of hand" in special streams.
This needs more investigation.
Comment 3 Sven Gothel 2023-10-04 12:06:32 CEST
commit e066250b908f0f2187d468d2c5ab586341031753

Refine AVSync debug output (WIP)
    
Using `-Djogl.debug.GLMediaPlayer.AVSync -Djogamp.debug.AudioSink` to trace AVSync issues for now.
Comment 4 Sven Gothel 2023-10-04 16:32:03 CEST
Joal 05529d4d6cd857e5bcd87be3b342c3fe7ba7655e
Add more comprehensive DEBUG info to detect sync issues
Comment 5 Sven Gothel 2023-10-15 07:14:06 CEST
GlueGen commit 52725b4c6525487f93407f529dc0a758b387a4fc 

    Bug 1472: Enhance AV Sync: Pass through PTS object in AudioSink, tracking last PTS value against System Clock Reference (SCR)
    
    This allows passing through actual SCR with the PTS when enqueueing audio frames
    and utilizing it when querying actual time lapsed PTS value (interpolation SCR to currentMillis).

+++

JOAL commit d3de587eae8ed8b5b5bc62647da0f95297144294

   ALAudioSink: getPTS() returns time-adjusted last dequeued PTS, new updateQueue() dequeues w/o wait 1st, then returns adjusted PTS; Simplify/split waitFroReleaded*(); Use TSPrinter for DEBUG
    
    Returning the time-adjusted PTS from the last dequeued frame seems to be the most accurate
    value we can deliver.
    
    Hence we store the Clock.currentMillis() in playing_pts_t0 when updating playing_pts
    and add the difference to current Clock.currentMillis() when retrieving.

+++

JOAL commit 12029f1ec1d8afa576e1ac61655f318cc37c1d16 

    Bug 1472: Enhance AV Sync: Pass through PTS object in ALAudioSink, tracking last PTS value against System Clock Reference (SCR)
    
    See GlueGen commit 52725b4c6525487f93407f529dc0a758b387a4fc

+++
Comment 6 Sven Gothel 2023-10-15 07:14:49 CEST
JOGL commit b1956113f5601b0cc6ac525d3918a0dfa8d240af

    Bug 1472: Enhance GLMediaPlayer AV Sync: Utilize SCR aware audio PTS used as master-clock, enabling proper AV sync w/ untouched audio
    
    We can finally utilize the added pass through audio PTS, see commits
    - GlueGen 52725b4c6525487f93407f529dc0a758b387a4fc
    - JOAL 12029f1ec1d8afa576e1ac61655f318cc37c1d16
    
    This enables us to use the audio PTS as the master-clock and adjust video to the untouched audio.
    
    In case no audio is selected/playing or audio is muted,
    we sync merely on the system-clock (SCR) w/o audio.
    
    AV granularity is 22ms, however, since the ALAudioSink PTS may be a little late,
    it renders even a slightly better sync in case of too early audio (d_apts < 0).
    
    Since video frames are sync'ed to audio, the resync procedure may result
    in a hysteresis swinging into sync. This might be notable at start
    and when resumed audio or after seek.
    
    We leave the audio frames untouched to reduce processing burden
    and allow non-disrupted listening.
    
    Passed AV sync tests
    - Five-minute-sync-test.mp4
    - Audio-Video-Sync-Test-Calibration-23.98fps-24fps.mp4
    - Audio-Video-Sync-Test-2.mkv
Comment 7 Sven Gothel 2023-10-15 07:15:33 CEST
JOGL commit 2c80bb2e6eb12e155d747daf8a08362396d5e0fc

    Bug 1472: Enhance GLMediaPlayer AV Sync: Fix FFMPEGMediaPlayer's audio resample: Use swr_get_out_samples(..) to calculate the required output sample count
    
    Notable when playing audio with e.g. 24k sample rate on a 48k OpenAL ALAudioSink,
    the sample duration was cut in half due to erroneous resampling missing half the required samples.
    
    Using swr_get_out_samples(..) resolves this issue pre swr_convert(..), the resampling.
Comment 8 Sven Gothel 2023-10-15 07:18:07 CEST
Resolved for FFMPEGMediaPlayer w/ ALAudioSink (default).

A bit more testing on NullGLMediaPlayer and AndroidGLMediaPlayerAPI14.
Comment 9 Sven Gothel 2023-10-15 19:04:16 CEST
JOGL commit bd2aa98d57c0c03bbface35000ed1c4bac6470e2

    Bug 1472: Enhance GLMediaPlayer AV Sync: Fine tune AV heuristics to (multiple of) audio_dequeued_ms
    
    The case of lagging audio to the audio master-clock (by experience) is a rare and probably synthethic case
    of the AV async videos, was
      dt_a > MAX_VIDEO_ASYNC && d_apts > 0
    now with increased threshold max_adelay = Math.max( 4*audio_dequeued_ms, 4*MAX_VIDEO_ASYNC )
      dt_a > max_adelay && d_apts > 0
    
    In conjunction the video-repeat case (video pts > SCR) shall use a higher threshold _when_
    detecting, i.e. min1_audio_queued_ms = Math.max( 2*audio_dequeued_ms, 2*MAX_VIDEO_ASYNC )
    to ensure enough buffered audio exists (2 audio-frames) before the next getNextTexture()
    hits within vsync.
    
    This early detection and min1_audio_queued_ms threshold
    is double of the late threshold for video-repeat min0_audio_queued_ms = Math.max( audio_dequeued_ms, MAX_VIDEO_ASYNC ),
    when the to-be repeated frame shall be displayed within getNextThreshold().
    Failing this requirement (1 audio-frame) will discard it and gather the next video-frame,
    allowing to fill the audio-buffer. A subsequent AV sync shall correct the difference.
    Strategy is less intervention on less buffered-audio.
    
    This shorter tolerance also reduces some video lag or stuttering on 24fps -> 60fps films.
Comment 10 Sven Gothel 2023-10-15 19:43:53 CEST
OK .. now let's adjust the playback-speed behavior, 
which is not considered with dropped audio.
Comment 11 Sven Gothel 2023-10-16 04:15:14 CEST
gluegen commit 2a91bec43a7a58efc22dd36e919308ecc50d9b4a

    PTS: get(currentMillis): Employ rounding in integer conversion

+++

JOGL

commit ee29eaf770d0d3627e5989dcae58a87cf27184d3

    Bug 1472: GLMediaPlayer: Expose SCR PTS and encourage its usage, removes user from selecting video or audio PTS.

commit dac49f32e09d61f5ba9be53a21d351843a48761b

    Bug 1472: GLMediaPlayer: Handle setPlaySpeed() like setAudioVolume() if stream is not yet initialized or playing, cached clipped value and set at initGL(..)


commit a35ca1763c773b4533b9b9850a9842522303ad01

    Bug 1472: GLMediaPlayer.getNextTexture(): Fix playSpeed > 1; Handle !hasVideoFrame upfront, simplifying sync-block
    
    Fix playSpeed > 1: dt_v (keep playSpeed factor), allow drop frame on no-frames w/ playSpeed > 2
Comment 12 Sven Gothel 2023-10-16 15:29:38 CEST
Resolved for FFMPEGMediaPlayer w/ ALAudioSink (default).