Bug 981 - com.jogamp.opencl.[Low|High]LevelBindingTest fail on OS X 10.6 with java.lang.OutOfMemoryError: Direct buffer memory
Summary: com.jogamp.opencl.[Low|High]LevelBindingTest fail on OS X 10.6 with java.lang...
Status: RESOLVED FIXED
Alias: None
Product: Jocl
Classification: JogAmp
Component: opencl (show other bugs)
Version: 1
Hardware: All macosx
: --- normal
Assignee: Wade Walker
URL:
Depends on:
Blocks:
 
Reported: 2014-02-22 19:56 CET by Wade Walker
Modified: 2014-02-25 02:48 CET (History)
1 user (show)

See Also:
Type: ---
SCM Refs:
af62da5b7ee3d99da7dc9364f1240fa7a8f5968e
Workaround: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Wade Walker 2014-02-22 19:56:27 CET
It looks like these tests allocate large direct buffers, and the system just doesn't do GC quickly enough to prevent these errors. The stack trace is:

java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:632)
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:97)
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
	at com.jogamp.common.nio.Buffers.newDirectByteBuffer(Buffers.java:73)
	at com.jogamp.opencl.HighLevelBindingTest.vectorAddGMTest(HighLevelBindingTest.java:323)

I propose using something like this to deallocate the large (~10MB) direct buffers created in these tests:

    /**
     * Uses reflection to call the "clean" method of a DirectByteBuffer.
     * Needed because the JVM may not clean these often or quickly enough.
     * Uses reflection because this interface isn't public, so may
     * break if the class internals change.
     *
     * @param bb The ByteBuffer to cleaned. Does nothing if it's not direct. 
     */
	public static void cleanDirectByteBuffer(ByteBuffer bb) {

        if(!bb.isDirect())
		    return;

        try {
		    // get cleaner
		    Method cleanerMethod = bb.getClass().getMethod("cleaner");
		    cleanerMethod.setAccessible(true);
		    Object cleaner = cleanerMethod.invoke(bb);
		  
		    // invoke clean method
		    Method cleanMethod = cleaner.getClass().getMethod("clean");
		    cleanMethod.setAccessible(true);
		    cleanMethod.invoke(cleaner);
	    }
	    catch( Exception e ) {
	    	System.err.println( "Failed to clean DirectByteBuffer. Java implementation details may have changed." );
	    }
    }
Comment 1 Wade Walker 2014-02-22 20:09:10 CET
Or, alternately, I could test how much direct memory is available, then make sure to use less than that. Originally I thought that perhaps the tests were failing because they were all running in one JVM, and old allocations were piling up, but it could just be that there's only 32MB of direct memory available in the JVM, and the test tries to allocate 3 buffers of 11MB each, and fails on the third one.

I'll try this first, since it seems more plausible.
Comment 2 Wade Walker 2014-02-22 21:46:29 CET
Submitted pull request to fix this: https://github.com/JogAmp/jocl/pull/4
Here's my branch: https://github.com/WadeWalker/jocl/tree/fix_jocl_bug_981_out_of_direct_buffer_memory

Both these tests pass on Windows 7 x86 with -XX:MaxDirectMemorySize=16m set to simulate the problem we're seeing on the older Mac JVM.
Comment 3 Sven Gothel 2014-02-23 00:00:14 CET
(In reply to comment #2)
> Submitted pull request to fix this: https://github.com/JogAmp/jocl/pull/4
> Here's my branch:
> https://github.com/WadeWalker/jocl/tree/
> fix_jocl_bug_981_out_of_direct_buffer_memory
> 
> Both these tests pass on Windows 7 x86 with -XX:MaxDirectMemorySize=16m set
> to simulate the problem we're seeing on the older Mac JVM.

Nice!

Merged.

If you have pushed a commit and since you are the maintainer,
pls add the git-sha1 to the SCM-Refs field so we can query the commit in bugzilla
and find it's discussion.

.. did it here:

+++

52a618fa844fa19dce19e18c527991ef422b1c43
    Fix memory problems in High/LowLevelBindingTests.
    
    These tests now adaptively reduce the global work size until they
    successfully allocate memory for their DirectByteBuffers. This
    makes the tests work on JVMs where XX:MaxDirectMemorySize is
    smaller than the modern defaults. These tests were failing on
    OS X 10.6 for this reason.
Comment 4 Wade Walker 2014-02-23 00:24:42 CET
Will do -- I didn't realize I had the privileges to push to the repo myself. That's why I was submitting everything to you as pull requests :)
Comment 5 Sven Gothel 2014-02-23 00:51:47 CET
(In reply to comment #4)
> Will do -- I didn't realize I had the privileges to push to the repo myself.
> That's why I was submitting everything to you as pull requests :)

Since you are now a JOCL maintainer .. it's setup properly now,
i.e. we both can push to jocl* to allow us to ease pushing things fwd.

Ofc course .. if any of us is uncertain about a commit, 
we may refrain from pushing into the master branch and use a 
dedicated one for review ..
Comment 6 Wade Walker 2014-02-25 02:45:38 CET
Merged in the fix after testing successfully on Mac OS X, Windows 7, Ubuntu 12.04, and Oracle Solaris 11.
Comment 7 Wade Walker 2014-02-25 02:48:15 CET
Oops, last comment and SCM ref were for bug 978... changing it back.