Bug 629

Summary: AWT-to-NEWT event modifier translation does not preserve mouse button state
Product: [JogAmp] Newt Reporter: Rob Hatcherson <rob.hatcherson>
Component: x11Assignee: Rob Hatcherson <rob.hatcherson>
Status: RESOLVED FIXED    
Severity: major CC: gouessej, rami.santina, sgothel
Priority: ---    
Version: 1   
Hardware: pc_x86_32   
OS: linux   
Type: --- SCM Refs:
jogl 811e3791b98fea0dfa3b7d301cb532c54df8dc82 jogl 13168c99ff9e8bf71c83f1be7afee270a3db4074 jogl 7ec58d773535e86e413e3c581a5e5bd734cb1269 jogl 58eb9d922c2a16a2724aaa2188c56784d440fb99 jogl f3e2467decf2a5ee47cfb458f7071772f39ea9c6 jogl d45f777e7969769eaa9b7d7f3eb5b34c10268368 jogl 17fde6c517b4ccf91b07c7e86974019bcbfee642
Workaround: ---
Bug Depends on:    
Bug Blocks: 656    
Attachments: test.sh output

Description Rob Hatcherson 2012-10-12 19:14:37 CEST 
Fedora 12 (2.6.32.26-175.fc12.i686.PAE)
Intel/32
JRE/JDK 7u7
GeForce 9800 GT + NVIDIA driver 304.37
JOGAMP/JOGL custom build from 10/09/2012 repos (RC11 in progress)

test.sh output is attached (FYI I had to tweek that script to get it to work)

Additional discussion leading up to this bug can be found in the following forum topic:
http://forum.jogamp.org/Mouse-Button-Modifiers-Lost-In-Newt-mouseDragged-Events-tp4026456.html

One of NEWT's goals is to provide a common event model so graphics code doesn't need to be aware of AWT, SWT, etc.  Event translation code exists to map event data between these specific window toolkit events and NEWT events.

In the Java and JOGL versions noted above the AWT-to-NEWT event modifier translation does not preserve mouse button state for at least some types of mouse events.

The problem occurs due to a quirk of how the AWT sets its event modifiers.  The AWT maintains both "modifiers" and "extended modifiers", where NEWT only has the notion of "modifiers".  In the AWT the modifiers include the usual ALT, META, CTRL, etc., as well as some button flags that are, apparently, no longer the button flags of choice.  Based on the AWT javadocs the preferred mouse button state flags are part of the "extended modifiers", and this has been the case since at least JDK 1.5.

Further confusing the issue is that in the AWT the masks for BUTTON2 and BUTTON3 are also associated with ALT and META respectively, and this has some implications for how NEWT maps the modifiers since it maintains distinct mask bits for the mouse buttons (1,2,3...) and the customary modifiers (ALT, META, CTRL, ...).

The current AWT-to-NEWT modifier translation does not pay attention to any of the button flags in either the AWT modifiers or the AWT extended modifiers explicitly, but it needs to because it relies on the button state to be there in its (NEWT's) modifiers to return proper values from at least the isButtonDown(someButton) and getButtonsDown() methods of the NEWT InputEvent class.

One possible fix that I have implemented and tested (and for which I will submit a patch candidate through github assuming I can figure out how to do so without messing something up) is to add some code to AWTNewtEventFactory to also check the extended modifiers for the mouse button state.
Comment 1 Rob Hatcherson 2012-10-12 19:15:38 CEST 
Created attachment 376 [details]
test.sh output
Comment 2 Sven Gothel 2012-10-12 22:09:27 CEST 
Thank you Rob!
Comment 3 Rob Hatcherson 2012-10-18 20:20:15 CEST 
Status:

I have implemented a candidate patch for this particular variation of the problem.

At your request I have also implemented a candidate comprehensive JUnit test for general testing of NEWT event modifier preservation for at least .../awt/GLCanvas, NewtCanvasAWT, and NewtCanvasSWT.  I think to test .../swt/GLCanvas we would need a SWT equivalent to the AWT...Adapter classes (?); I might be wrong about this, as I am still absorbing what all these pieces do.

However...

While the unit tests work fine for .../awt/GLCanvas at least on Linux, they work less fine for NewtCanvasAWT and NewtCanvasSWT.

Examples:

- When the robot clicks in the test window when using NewtCanvasAWT and NewtCanvasSWT, there appear to be synthesized EVENT_MOUSE_EXITED and EVENT_MOUSE_ENTERED events prior to MOUSE_PRESSED, and the final MOUSE_CLICKED is omitted.

- Event modifiers don't seem to be there in the expected form when using NewtCanvasAWT and NewtCanvasSWT.

I'm still working on making sure these are real problems and not some artifact of my test code.  If these persist should I go straight to Bugzilla with them and start discussion there?  In particular, skip cluttering up the forum with this kind of low-level discussion?  I probably would:

- File a low-priority enhancement bug on AWT...Adapter equivalents for SWT.
- File a bug about the superfluous synthesized EVENT_MOUSE_EXITED/EVENT_MOUSE_ENTERED events.
- File a bug about the missing MOUSE_CLICKED.
- File a new bug for the other missing modifier cases.

Since the AWT is a common case and the .../awt/GLCanvas tests work, I am inclined at the moment to submit the work I have so far for general comments.  If you're satisfied with it then those changes will at least fix the NEWT event modifiers for the common .../awt/GLCanvas case.  Then we can discuss these other issues in the new bugs noted above.

Preferences?
Comment 4 Sven Gothel 2012-10-26 17:01:29 CEST 
(In reply to comment #3)
> Status:
> 
> I have implemented a candidate patch for this particular variation of the
> problem.
> 
> At your request I have also implemented a candidate comprehensive JUnit test
> for general testing of NEWT event modifier preservation for at least
> .../awt/GLCanvas, NewtCanvasAWT, and NewtCanvasSWT.  I think to test
> .../swt/GLCanvas we would need a SWT equivalent to the AWT...Adapter classes
> (?); I might be wrong about this, as I am still absorbing what all these pieces
> do.
> 
> However...
> 
> While the unit tests work fine for .../awt/GLCanvas at least on Linux, they
> work less fine for NewtCanvasAWT and NewtCanvasSWT.
> 
> Examples:
> 
> - When the robot clicks in the test window when using NewtCanvasAWT and
> NewtCanvasSWT, there appear to be synthesized EVENT_MOUSE_EXITED and
> EVENT_MOUSE_ENTERED events prior to MOUSE_PRESSED, and the final MOUSE_CLICKED
> is omitted.

Ofc it might not be possible to have the very same events produced 
when using the native window parenting (NewtCanvasAWT, etc).
Keep in mind that the visible NEWT window is on top of the native AWT/SWT peer
and hence the focus cycle _is_ a bit more complicated .. and produces more events.

The AWT / NEWT focus handling in this case is indeed a huge mess
due to unpredicted / different behavior on some platforms.

> 
> - Event modifiers don't seem to be there in the expected form when using
> NewtCanvasAWT and NewtCanvasSWT.

Key and mouse events are coming from NEWT itself, i.e. the native subsystem.

> 
> I'm still working on making sure these are real problems and not some artifact
> of my test code.  If these persist should I go straight to Bugzilla with them
> and start discussion there?  In particular, skip cluttering up the forum with
> this kind of low-level discussion?  I probably would:
> 
> - File a low-priority enhancement bug on AWT...Adapter equivalents for SWT.
> - File a bug about the superfluous synthesized
> EVENT_MOUSE_EXITED/EVENT_MOUSE_ENTERED events.
> - File a bug about the missing MOUSE_CLICKED.
> - File a new bug for the other missing modifier cases.
> 
> Since the AWT is a common case and the .../awt/GLCanvas tests work, I am
> inclined at the moment to submit the work I have so far for general comments. 
> If you're satisfied with it then those changes will at least fix the NEWT event
> modifiers for the common .../awt/GLCanvas case.  Then we can discuss these
> other issues in the new bugs noted above.
> 
> Preferences?

Well, if you don't break current unit tests w/ your changes
- or can correct / describe such changes in behavior - I agree.

This task (unifying event handling) is a heavy one but surely required 
to provide a reliable system for production.
See Bug 601 and Bug 631 .. even for key events the story ain't over yet.
Comment 5 Sven Gothel 2012-10-31 17:17:54 CET 
FYI .. NEWT's (native) KeyEvent handling has been refined (see Bug 601 and 631, as well as 
KeyEvent class and rel. commits.)

Maybe it helps - available next week to continue working on this issue.
Comment 6 Rob Hatcherson 2012-10-31 17:59:12 CET 
OK.  I've been keeping up with your master on github every few days.  I haven't pushed the changes to my clone of your master yet though, simply due to time constraints.  I'll try to put priority on this next week.

You're right - an overhaul to the event system will be a substantial task.  Since the existing infrastructure is very far along it's probably best at this point to smooth out its few remaining rough spots and get it working as consistently as possible across AWT/SWT/NEWT/etc.  Then we'll have time to step back and think about whether or not an improved approach is warranted.
Comment 7 Sven Gothel 2012-12-24 08:08:17 CET 
    TODO: Unify synthesization of NEWT event artifacts as described by the above,
    allowing NEWT event translation to benefit from same code to gain same semantics.
    Notable: AWTNewtEventFactory and SWTNewtEventFactory
Comment 8 Sven Gothel 2012-12-24 08:18:14 CET 
(In reply to comment #7)
>     TODO: Unify synthesization of NEWT event artifacts as described by the
> above,
>     allowing NEWT event translation to benefit from same code to gain same
> semantics.
>     Notable: AWTNewtEventFactory and SWTNewtEventFactory

See Bug 656