Created attachment 337 [details]
This is a potential fix for the GLCanvas Linux memory leak. Needs to be reviewed.

*Setup:

JOGL2.0 RC5, Ubuntu 11.10 (64-bit), GeForce 285GTX

*Problem:

XLib11 requires that each XWindow opened with XOpenWindow also be closed with XCloseWindow (See X11Util). This is not the case for GLCanvas, which seems to call XOpenWindow when it is attached to a frame, but it doesn't properly close the XWindow when you remove it from a frame.

In a 3D editing application, where windows are opened and closed a lot, this could be problematic. Especially since Ubuntu seems to have a limit of maximum 180 XWindows open before it refuses to open a new one.

*Steps to reproduce:

1) Create a GLCanvas
2) Add GLCanvas to frame
3) Remove GLCanvas from frame
4) Dispose the GLCanvas

(If you repeat this 200 times, you will get an exception telling you you can't open a new XOpenWindow)

See my attachment for a test case to reproduce. You can call X11Util.dumpOpenDisplayConnections() to see how many
XWindows are actually open. This number will grow over time, and it shouldn't.

*Expected result:

XWindow is closed

*Actual result: 

XWindow remains open -> This indicates a memory leak.

*Analysis:

The problem is that when you remove the GLCanvas from the frame, the removeNotify is called, which disposes the GLCanvas. This in itself is fine, but it doesn't dispose the native surface, it only disposes the awt surface.

If you look at GLCanvas.DisposeAbstractGraphicsDeviceAction, you will see the following line:

(GLCanvas:884) boolean closed = awtConfig.getScreen().getDevice().close();

This line calls awtConfig.getScreen(), which refers to an AWTGraphicsScreen that is created at construction time. Calling close on the device of this screen will do nothing, because all implementations of AbstractGraphicsDevice close methods do nothing, apart from the X11GraphicsDevice, which closes the XWindow.

So the problem is either that the wrong screen is initialized at construction

*Possible Fix (1):

Calling 

awtConfig.getNativeGraphicsConfiguration().getScreen().getDevice().close();

right after line 884 in DisposeAbstractGraphicsDeviceAction fixes the issue, but I'm not sure if this is correct. This will call the native X11 screen to close the device when destroyed, which plugs the memory leak.

*Possible Fix (2):

Initialize the screen in the GLCanvas.awtConfig field to be the X11GraphicsScreen instead of AWTGraphicScreen. This is done in the getGraphicsConfiguration() method of GLCanvas.

Again, I'm not sure what the impact of this fix would be.

*Workaround:

The workaround I'm using currently in my application is to override the destroy of my own GLCanvas, and call close the native device myself before the canvas is destroyed. The problem is that after the GLCanvas is destroyed, the reference to getNativeGraphicsConfiguration() will be null, so I have to do it before this happens.