Bug 735

Summary: Screen is not updated when using invoke() method on OSX
Product: [JogAmp] Jogl Reporter: ac <andres.colubri>
Component: macosxAssignee: Sven Gothel <sgothel>
Status: RESOLVED FIXED    
Severity: major CC: jeffgemail, xerxes
Priority: ---    
Version: 2   
Hardware: pc_all   
OS: macosx   
Type: --- SCM Refs:
jogl 4fe6cf06f74085cc90d1f5b003fd717234d43ced jogl 05d8a9490497413df11fd2faf07e34d648130966 jogl 24b63b18e6ca3f657350d6c80e4914eadce60164 jogl c24f44036971bf58b5c47a6e1f7d9f186c67e789 jogl d846b04928ecfcfb319e75d7ed114d357edbeb89
Workaround: ---
Bug Depends on: 753    
Bug Blocks:    
Attachments: Eclipse project with test case.

Description ac 2013-05-14 17:54:01 CEST
I wrote an application that uses (GLCanvas/GLWindow).invoke() to run my frame rendering code through a GLRunnable. It works without any issues on Windows 7 (with Java 7), but it doesn't properly update the screen on OSX. The frame starts by clearing the color buffer to red, and it then runs a raymarching shader. Only the red color filling the entire screen is the visible output of this application. Even if the shader is modified to simply output a constant color for all pixels, the result continues to be the red color from the clear command. 

The application can switch between AWT and NEWT as the toolkit, the uploaded test case is set to use NEWT at the moment. In that case, and if the following two lines in the initGL() method:

applet.setLayout(new BorderLayout());
applet.add(newtCanvas, BorderLayout.CENTER);

are replaced by:

newtWindow.setSize(applet.width, applet.height); 
newtWindow.setVisible(true); 

then the output of the rendering is properly displayed in a separate window.

This issue is consistent across 10.6, 10.7 and 10.8, with the same output described above across all cases. The following test configurations were used:

* Macbook Pro 2010, NVidia Geforce 320M, OSX 10.6.8
* iMac 2009, NVidia Geforce GT 130, OSX 10.7.5
* Macbook Pro 2011, AMD Radeon HD 6490M, Intel HD Graphics 3000, OSX 10.8.3

On the OSX 10.7 and 10.8 systems, Java 6 and 7 were used, with identical results.
Comment 1 ac 2013-05-14 17:58:33 CEST
Created attachment 465 [details]
Eclipse project with test case.

Jogl jars removed due to size limitation in bugzilla. Tested with jogl 2.0-b982 and gluegen 2.0-b667.
Comment 2 Xerxes RĂ„nby 2013-05-21 12:59:33 CEST
Try add
applet.validate();
after adding the newtCanvas to the applet.

AWT require you to call to the validate() method on the last line is necessary to repaint the applet and to ensure that the applet is correctly displayed by the layout manager.

Since JOGL do lazy initialization the GL canvas is only initialized if it have been displayed properly, thus a missing validate may prevent JOGL to initialize.
Comment 3 ac 2013-05-21 16:05:56 CEST
Hello Xerxes, thanks for your suggestion. I added the call applet.validate() right after applet.add(newtCanvas, BorderLayout.CENTER) but doesn't seem to make any difference, either on Windows or OSX (tested on 10.6 and 10.8). On the former, the program renders correctly, on the later, it only shows a red screen as I reported originally.

I wonder if this is really a bug (either in JOGL or at the level of the OS), or whether I'm doing something fundamentally wrong in the setup of JOGL. The fact that rendering occurs normally on Windows makes me think that this is indeed due to a bug somewhere in the OSX implementation of JOGL or in the OS itself (maybe all the recent NSView/CALayer changes have something to to with this problem?).

Since we are trying to reach the final release of Processing 2.0 in the near future, I'd need to be able to determine if this is due to some error on my side, to a fixable bug in JOGL, or to a more fundamental issue in OSX. Depending on the answer to this question, I will have to handle this issue differently in the Processing code. Ideally, it is something we can fix entirely, otherwise I'd need to do add some kind of workaround.

Hopefully Sven will be able to take a look at this and give us his opinion.

Andres
Comment 4 Sven Gothel 2013-06-15 02:05:38 CEST
.. working on it now, finally.
Comment 5 Sven Gothel 2013-06-15 07:53:22 CEST
.. I love the landscape shader .. will adopt it in demos for sure - THX for picking this one ! :)

Reproduced lack of drawing ..

Looking at your code .. your looping and draw code
looks quite complicated at 1st glance .. well.

Dunno why you don't use a simple GLEventListener style w/ Animator,
however - it's your valid choice I do respect.

Will analyze reason for failure.
Comment 6 ac 2013-06-15 21:37:23 CEST
The reason for the complexity of the looping/drawing code is to reproduce Processing's architecture, where an animation thread started from the main process drives the rendering loop, and also is designed to support different renderers (JAVA2D-based, OpenGL-based, etc). 

At some point in the development of the OpenGL renderer in Processing, I used an animator object in combination with the GLEventListener, but the animator seemed superflous since the framerate was directed by the animation thread I mentioned above.

From some discussions on the jogamp forum (http://forum.jogamp.org/synchronization-issues-tt4029115.html), it seemed to me that the most correct way to trigger the rendering of individual frames from the animation loop was through the use of the invoke() method, which is what lead to the issue I reported here.
Comment 7 Sven Gothel 2013-06-16 00:30:01 CEST
(In reply to comment #6)
> The reason for the complexity of the looping/drawing code is to reproduce
> Processing's architecture, where an animation thread started from the main
> process drives the rendering loop, and also is designed to support different
> renderers (JAVA2D-based, OpenGL-based, etc). 

ok, sorry .. I should have know - was more about the below issues (invoke, animator).

> 
> At some point in the development of the OpenGL renderer in Processing, I
> used an animator object in combination with the GLEventListener, 

Yes .. and this [should] work well .. and does here ..

> but the
> animator seemed superflous since the framerate was directed by the animation
> thread I mentioned above.

that can be handled ..

Pls note that the GLAnimatorControl interface API
duty is not only to manage refresh, but to orchestrate refresh
w/ toolkit refresh - and also allows the exclusive context thread feature (ECT).

I see that both latter features are attempted manually in your test code,
however - I doubt this is sufficient.

> 
> From some discussions on the jogamp forum
> (http://forum.jogamp.org/synchronization-issues-tt4029115.html), it seemed
> to me that the most correct way to trigger the rendering of individual
> frames from the animation loop was through the use of the invoke() method,
> which is what lead to the issue I reported here.

This is more like an abuse of the invoke(..) feature.
I.e. invoke was intended to add a GLRunnable to the renderer queue 
to be executed when rendering (display) _from_ a different thread, e.g. UI.
invoke() does trigger display(), if no GLAnimatorControl is set in the GLAutoDrawable.
Hence it simply complicates things, where you just could call 'display()'
w/ an GLEventListener attached.

My first attempt to clarify code was to add an 
[LandscapeES2 implements GLEventlistener', and replace your invoke()
w/ a simple GLAutoDrawable.display() call, where GLAutoDrawable is either GLCanvas or GLWindow.

I will go into the details of this design w/ a few variations,
besides I still have to find the actual culprit of the 'now show' in your orig. test case.

Long story short:
  - Sure it should work w/ invoke() .. but it's like a bit overkill.
  - The GLAnimatorControl issues you might had are solvable.

.. stay tuned :)
Comment 8 Sven Gothel 2013-06-16 04:31:31 CEST
See 
  <http://jogamp.org/git/?p=jogl.git;a=commit;h=4fe6cf06f74085cc90d1f5b003fd717234d43ced>

All below test cases work fine w/ GNU/Linux.

If not mentioned, same behavior w/ java6 and java7 on OSX.

Bug735Inv0AppletAWT:
/**
 * Original test case.
 * <br/>
 * OSX Results:
 * <pre>
 *   - Only RED Clear Color
 * </pre>
 */

Bug735Inv1AppletAWT:
/**
 * Difference to orig. Bug735Inv0AppletAWT:
 * <pre>
 *   - MANUAL_FRAME_HANDLING false
 *   - MANUAL_FRAME_HANDLING: impl using pass through GLContext instead of static
 * </pre>
 * OSX Results:
 * <pre>
 *   - Only RED Clear Color
 * </pre>
 */

Bug735Inv2AppletAWT:
/**
 * Difference to orig. Bug735Inv0AppletAWT:
 * <pre>
 *   - Use GLEventListener
 *   - Add GLEventListener to GLAutoDrawable
 *   - Use GLAutoDrawable.display() instead of GLAutoDrawable.invoke(true, GLRunnable { init / render })
 *   - Removed MANUAL_FRAME_HANDLING, obsolete due to GLAutoDrawable/GLEventListener 
 * </pre>
 * OSX Results:
 * <pre>
 *   - Visible content
 *   - Stuttering, non-fluent and slow animation
 * </pre>
 */

Bug735Inv3AppletAWT:
/**
 * Difference to orig. Bug735Inv0AppletAWT:
 * <pre>
 *   - Use GLEventListener
 *   - Add GLEventListener to GLAutoDrawable
 *   - Use GLAutoDrawable.display() instead of GLAutoDrawable.invoke(true, GLRunnable { init / render })
 *   - Removed MANUAL_FRAME_HANDLING, obsolete due to GLAutoDrawable/GLEventListener 
 *   - Use Animator
 *   - Remove applet, component sizes, use frame based size via validate
 *   - Run frame validation/visibility on AWT-EDT
 *   - Add Wait-For-Key after init (perf-test)
 * </pre>
 * OSX Results:
 * <pre>
 *   - Visible content
 *   - Stuttering, non-fluent and slow animation
 * </pre>
 */

Bug735Inv4AppletAWT:
/**
 * Difference to orig. Bug735Inv0AppletAWT:
 * <pre>
 *   - Use GLEventListener
 *   - Add GLEventListener to GLAutoDrawable
 *   - Use GLAutoDrawable.display() instead of GLAutoDrawable.invoke(true, GLRunnable { init / render })
 *   - Removed MANUAL_FRAME_HANDLING, obsolete due to GLAutoDrawable/GLEventListener 
 *   - Use Animator
 *   - Remove component sizes, use frame based size via validate
 *   - Run frame validation/visibility on AWT-EDT
 *   - Add Wait-For-Key after init (perf-test)
 *   - Remove intermediate applet!
 * </pre>
 * OSX Results:
 * <pre>
 *   - Visible content
 *   - Java6: Fluent animation
 *   - Java7: Stuttering, non-fluent and slow animation
 * </pre>
 */
Comment 9 Sven Gothel 2013-06-16 04:41:18 CEST
Note that 'TestLandscapeES2NewtCanvasAWT' and 'Bug735Inv4AWT' expose same behavior,
i.e. 
  java6: fluent
  java7: stuttering

Note: if running same tests w/ GearsES2 instead of LandscapeES2, 
no stuttering appeared. See TestGearsES2NewtCanvasAWT vs TestLandscapeES2NewtCanvasAWT.

Stuttering: Maybe .. there is a correlation of the rendering period,
since we may block the CALayer animation ?

This needs to be investigated for sure.

++

Still need to resolve why Bug735Inv0AppletAWT and Bug735Inv1AppletAWT
don't show the content at all.

.. ?
Comment 10 Sven Gothel 2013-06-16 16:23:35 CEST
(In reply to comment #9)
> 
> Stuttering: Maybe .. there is a correlation of the rendering period,
> since we may block the CALayer animation ?
> 
Yes, on the OSX mini w/ NV .. the GPU utilization is maxed out w/ the shader itself.
If I increase the swap-interval to 5, fluent animation appears.
It seems that we need to adapt to this situation, i.e. if rendering is 'slower' increase the 
wait period to not hog the GPU. Otherwise .. no GPU cycles left for the CALayer thread.
Too bad that multithreading doesn't work well here.
Note: It's not the context locking which blocks CALayer .. simply the GPU utilization.
Comment 11 ac 2013-06-16 19:41:36 CEST
Hi Sven, thanks for looking into this. 

I'm also glad to see that you were able to reproduce the stuttering issue, which was in fact the problem I faced originally. My use of invoke was basically an attempt to solve the stuttering. So far, the only way I was able to reduce the stuttering was by adding a glFinish() call at the end of the frame rendering, which of course shouldn't be done, at least from my understanding of glFinish.

I will take a look at GLAnimatorControl, thanks for pointing that out.

(In reply to comment #10)
> (In reply to comment #9)
> > 
> > Stuttering: Maybe .. there is a correlation of the rendering period,
> > since we may block the CALayer animation ?
> > 
> Yes, on the OSX mini w/ NV .. the GPU utilization is maxed out w/ the shader
> itself.
> If I increase the swap-interval to 5, fluent animation appears.
> It seems that we need to adapt to this situation, i.e. if rendering is
> 'slower' increase the 
> wait period to not hog the GPU. Otherwise .. no GPU cycles left for the
> CALayer thread.
> Too bad that multithreading doesn't work well here.
> Note: It's not the context locking which blocks CALayer .. simply the GPU
> utilization.
Comment 12 Sven Gothel 2013-06-16 19:59:46 CEST
(In reply to comment #11)
> Hi Sven, thanks for looking into this. 
> 
> I'm also glad to see that you were able to reproduce the stuttering issue,
> which was in fact the problem I faced originally. My use of invoke was
> basically an attempt to solve the stuttering. So far, the only way I was
> able to reduce the stuttering was by adding a glFinish() call at the end of
> the frame rendering, which of course shouldn't be done, at least from my
> understanding of glFinish.

I have moved the stuttering issue to it's own bug, Bug 753.

Thank you for the hint glFinish(), maybe it gives a 'timeout',
allowing the CALayer thread to utilize the GPU.

I will copy this dialog to Bug 753 ..

> 
> I will take a look at GLAnimatorControl, thanks for pointing that out.

If you read the variations and their changes, 
it should be 'easy' to use our Animator.

After Bug 753, I will have a 2nd look into it .. but till now, I couldn't find the
'gotcha' lines .. :(
Comment 13 Sven Gothel 2013-06-17 06:24:42 CEST
(In reply to comment #12)
> After Bug 753, I will have a 2nd look into it .. but till now, I couldn't
> find the
> 'gotcha' lines .. :(

Solved this mystery: glViewport(..) was missing - duh :)

How did I find it ?

Enabled property jogl.debug.TraceGL ..

Since this was not a JOGL bug, marked INVALID.
Comment 14 Sven Gothel 2013-06-17 07:34:26 CEST
Re-opened: glViewport should have been set by GLAutoDrawable, i.e. GLDrawableHelper.
It was not, due to no GLEventListener attached .. but we should expose same behavior!
Comment 15 Sven Gothel 2013-06-17 13:19:48 CEST
GLAutoDrawable must issue glViewport(..) even w/o GLEventListener
  - Same behavior w/ or w/o GLEventListener requires to issue glViewport, always.
  - No glViewport(..) required in user code.