Bug 1024

Summary: On jar file location agnosticism
Product: [JogAmp] General Reporter: Mark Raynsford <org.jogamp>
Component: buildsAssignee: Mark Raynsford <org.jogamp>
Status: RESOLVED FIXED    
Severity: major CC: sgothel
Priority: ---    
Version: unspecified   
Hardware: All   
OS: all   
See Also: https://jogamp.org/bugzilla/show_bug.cgi?id=1023
Type: --- SCM Refs:
Workaround: ---
Bug Depends on: 1023, 1032    
Bug Blocks:    

Description Mark Raynsford 2014-06-23 19:53:04 CEST
The following is a summary of the discussion that occurred on IRC.

There is currently an issue with using JOGL from package systems such as Maven, Gradle, etc.

Currently, JOGL (and the other projects such as JOAL) perform the following to locate their own native libraries:

1. One of $JAR := {jogl-all.jar, jogl-all-mobile.jar, jogl-all-noawt.jar, ...} is placed on the classpath, .

2. The JOGL code determines the current location in the filesystem of $JAR, and then uses the basename of $JAR to locate the current platform's native jar by appending extra strings to the name (an example native jar name is "jogl-all-natives-linux-amd64.jar"). It assumes that this native jar exists in the same directory as the original jar.

3. The native jar is then unpacked to a temporary directory, resulting in a native library, which is then loaded.

The basic problem is the assumption in (2): Many build systems essentially give no control over the location to which jar files will be downloaded.

In Maven, the jar files (including any attached native jar files) for package "org.jogamp.jogl:jogl-all:2.1.5" will be downloaded to Maven's filesystem cache at:

  $HOME/.m2/repository/org/jogamp/jogl/jogl-all/2.1.5/

The names of the jar files will be changed such that they all start with the artifact name ("jogl-all") and version number ("2.1.5"):

  $HOME/.m2/repository/org/jogamp/jogl/jogl-all/2.1.5/jogl-all-2.1.5.jar
  $HOME/.m2/repository/org/jogamp/jogl/jogl-all/2.1.5/jogl-all-2.1.5-natives-linux-amd64.jar

This is more or less an internal implementation detail of Maven that we've exploited in order to produce Maven packages that appear to work (in other words, we've constructed packages such that the native libraries will be downloaded into the correct directory in the cache [as a side effect of how they're attached to a particular package as extra artfiacts]).

Unfortunately, other systems that use Maven packages (such as Gradle), don't lay out their own filesystem caches in the same manner and so JOGL is unable to find its own natives. According to this bug:

  http://issues.gradle.org/browse/GRADLE-2553

... Not only has the behaviour changed at least once, but in recent versions of Gradle *every single artifact will be placed in its own directory*. It doesn't matter how we tried to structure packages in this case, there's just no way to make Gradle download them into one directory in order to please JOGL.

The situation is compounded by the fact that JOGL actually uses
aliasing for some packages. Specifically, when using the jogl-all-noawt
package, the JOGL loader will look for jogl-all-*-natives.jar instead
of jogl-all-noawt-*-natives.jar). The purpose of this is to reduce
duplication; the basic assumption that JOGL makes is that all of its
own native jar files will exist in the same directory. See
https://jogamp.org/bugzilla/show_bug.cgi?id=1023. We can't fix this
one in the Maven packages without disabling aliasing and actually
shipping identical copies of the native jar files but attached to
the jogl-all-noawt package (so that Maven will assign them the correct
names)

One possible solution is to have each of these separate build systems run one final step at the end of the build in order to aggregate all of the jars into one directory. The problem here is that it essentially forces extra complexity onto the end-users for every project they make that uses JOGL. It's also not clear how this would interact with various IDEs (Eclipse, for example, uses the dependency information in Maven projects to configure the classpath, but does not otherwise execute Maven - meaning that anything the user did to manipulate JOGL libraries in a Maven build would not be executed when run from Eclipse).

The basic problem seems to be that JOGL cannot determine the right course of action in all these cases (obviously impossible), but it could easily do the right thing if the user provided a bit more information.

I'd like to propose making the loading process (more) controllable with properties.

Basically the JOGL (and JOAL, etc) code will check for a property called com.jogamp.native_jar_loading_type, which can be set to one of the following values (the name of this property, and the name of all the values below are obviously up for debate):

1. "same_directory"

This gives the traditional JOGL behaviour as things stand now. It's also the default, and will be used by JOGL if the user doesn't define a com.jogamp.native_jar_loading_type property.

2. "classpath"

For various technical reasons, JOGL doesn't inspect the classpath to find native directories. However, as a direct consequence of how Maven and Gradle handle dependencies, the full paths to each of the native jars will be (completely unconditionally, at least in Maven's case) placed on the classpath! This means that even if all of the jar files exist in different directories (Gradle), JOGL can still find them.

Tentatively, I might suggest a third value:

3. "same_directory_without_aliasing"

This gives the traditional JOGL behaviour but with aliasing disabled. In effect, when the "jogl-all-noawt" package looks for its native jars, it'll look for "jogl-all-noawt-*-natives.jar" instead of "jogl-all-*-natives.jar". This would
allow for fixing bug 1023 with some minor changes to the current Maven packages.

If we had the above, suddenly our Maven packages become a lot easier to rearrange (because we don't have to desperately try to get Maven to download everything to the right place), and resistant to any future repository layout changes. We'd also gain the ability for people to use Gradle as their build system (it outright won't work at all, at the moment).
Comment 1 Mark Raynsford 2014-06-23 20:06:11 CEST
I forget: Wasn't there a way to turn off all of this stuff entirely and force the user to set -Djava.library.path with the locations of the natives?
Comment 2 Mark Raynsford 2014-06-23 20:06:33 CEST
(In reply to comment #1)
> I forget: Wasn't there a way to turn off all of this stuff entirely and
> force the user to set -Djava.library.path with the locations of the natives?

A fourth value of "none" is what I was getting at...
Comment 3 Sven Gothel 2014-06-23 20:31:54 CEST
> 
> 1. "same_directory"
> 
> This gives the traditional JOGL behaviour as things stand now. It's also the
> default, and will be used by JOGL if the user doesn't define a
> com.jogamp.native_jar_loading_type property.

Negative impact: adds some complexity

> 
> 2. "classpath"
> 
> For various technical reasons, JOGL doesn't inspect the classpath to find
> native directories. However, as a direct consequence of how Maven and Gradle
> handle dependencies, the full paths to each of the native jars will be
> (completely unconditionally, at least in Maven's case) placed on the
> classpath! This means that even if all of the jar files exist in different
> directories (Gradle), JOGL can still find them.

Negative impact: Using ClassLoader and hence adding alot complexity
as well as having a performance impact.

Inspecting the classpath to find the right native jar file would
required:
  - adding an identifying class into the jar file
    to find it - and query the jar files purpose (platform)

  - using the ClassLoader, i.e. making resource lookup more expensive
    since all ClassLoader resources, 
    as well as all ClassLoader would need to be traversed.
    
  - requires networking applications to transfer all jar files to
    the client, e.g. Applets.
    Our current solution only transfer the requested native jar file
    to the client.    

> 
> Tentatively, I might suggest a third value:
> 
> 3. "same_directory_without_aliasing"
> 
> This gives the traditional JOGL behaviour but with aliasing disabled. In
> effect, when the "jogl-all-noawt" package looks for its native jars, it'll
> look for "jogl-all-noawt-*-natives.jar" instead of "jogl-all-*-natives.jar".
> This would
> allow for fixing bug 1023 with some minor changes to the current Maven
> packages.

Negative impact: Duplication of artifacts, but no code, performance
                 or 'process' sideffects.
Strong UP-VOTE!

I.e. simply produce matching native jar files for all 'all-<variant>'
java jar files, while removing the 'aliasing' altogether.

This also _REMOVES_ code complexity, which is good.

> 
> If we had the above, suddenly our Maven packages become a lot easier to
> rearrange (because we don't have to desperately try to get Maven to download
> everything to the right place), and resistant to any future repository
> layout changes. We'd also gain the ability for people to use Gradle as their
> build system (it outright won't work at all, at the moment).
Comment 4 Mark Raynsford 2014-06-23 21:58:25 CEST
Having talked about this on IRC, it seems that the best approach would be to essentially let the classloader do the work of classpath inspection by adding classes to the native jars that can be queried at runtime.

Basically:

1. Attempt to load native jars using what would have been the "same_directory_without_aliasing" behaviour described. This eliminates some complexity that the current aliasing introduces, and makes it possible to fix bug 1023.

2. If the above loading failed, assume that the native jar files are on the classpath and therefore can be looked up by class. The classloader will attempt to look up a class named, for example, "jogl_linux_amd64" which will be inserted into the jogl-all-natives-linux-amd64.jar at JOGL build time. The location of the jar file containing that class can obviously then be derived from the resulting class and the unpacking of jar files can continue as normal.

3. If the class loading failed, give up.

Advantage of this is that it's not necessary to inspect a platform-specific classpath that may or may not actually contain valid URIs.
Comment 5 Sven Gothel 2014-07-11 15:32:53 CEST
Solved via Bug 1023 and Bug 1032 commits.

Gradle works as confirmed by 'zubzub' on IRC:
  (02:26:21 PM) zubzub: huray!
  (02:26:24 PM) zubzub: \o/

Maven should work ..

Please reopen for refinements.