Bug 1172

Summary: Always file native libs in 'natives/os.and.arch'; Allow using maven-assembly-plugin single jar deployment
Product: [JogAmp] Gluegen Reporter: dylan.millikin
Component: coreAssignee: Xerxes Rånby <xerxes>
Status: RESOLVED FIXED    
Severity: major CC: gouessej, kjolhede, org.jogamp, sgothel, xerxes, xerxes
Priority: ---    
Version: 2.3.2   
Hardware: All   
OS: all   
Type: FEATURE SCM Refs:
3f73bbbd44721cc666e4d3505fcf163490636ba8 41d89263109d20dbcfcc7a642c88a290b4877b5f b4ad01b53421a58ccfe7028a520cf3e06d6b6742
Workaround: ---
Bug Depends on:    
Bug Blocks: 1118    

Description dylan.millikin 2015-07-08 00:42:01 CEST
I'm running ubuntu 14.04LTS and when I run the following piece of code I get an error:

GLProfile glp = GLProfile.getDefault();

The error is:

Exception in thread "main" java.lang.UnsatisfiedLinkError: /tmp/jogamp_0000/file_cache/jln7005008932607663889/jln8160798677507740369/libgluegen-rt.so: /tmp/jogamp_0000/file_cache/jln7005008932607663889/jln8160798677507740369/libgluegen-rt.so: cannot open shared object file: No such file or directory (Possible cause: can't load this .so (machine code=0xb7) on a AMD 64-bit platform))
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1937)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1822)
        at java.lang.Runtime.load0(Runtime.java:809)
        at java.lang.System.load(System.java:1086)
        at com.jogamp.common.jvm.JNILibLoaderBase.loadLibraryInternal(JNILibLoaderBase.java:575)
        at com.jogamp.common.jvm.JNILibLoaderBase.access$000(JNILibLoaderBase.java:63)
        at com.jogamp.common.jvm.JNILibLoaderBase$DefaultAction.loadLibrary(JNILibLoaderBase.java:95)
        at com.jogamp.common.jvm.JNILibLoaderBase.loadLibrary(JNILibLoaderBase.java:459)
        at com.jogamp.common.os.DynamicLibraryBundle$GlueJNILibLoader.loadLibrary(DynamicLibraryBundle.java:421)
        at com.jogamp.common.os.Platform$1.run(Platform.java:317)
        at java.security.AccessController.doPrivileged(Native Method)
        at com.jogamp.common.os.Platform.<clinit>(Platform.java:287)
        at com.jogamp.opengl.GLProfile.<clinit>(GLProfile.java:146)
        at com.dmill.enki.App.main(App.java:30)


From checking, the files are on the disk but they don't have execution rights (don't know if this is relevant)

bellow is more information about the system:

 > java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

 > jogl installed via maven :
<dependency>
   <groupId>org.jogamp.gluegen</groupId>
   <artifactId>gluegen-rt-main</artifactId>
   <version>2.3.1</version>
</dependency>
<dependency>
   <groupId>org.jogamp.jogl</groupId>
   <artifactId>jogl-all-main</artifactId>
   <version>2.3.1</version>
</dependency>

> uname -a
Linux ubuntu 3.13.0-53-generic #89-Ubuntu SMP Wed May 20 10:34:39 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Comment 1 Xerxes Rånby 2015-07-08 09:37:26 CEST
machine code=0xb7 is EM_AARCH64

it appears that the AMD 64 libgluegen-rt.so has been overwritten on your system with the AARCH 64 libgluegen-rt.so.
This can happen if you unpack all jogamp jar's into the same folder or into the same jar.

Please describe how you have deployed your application.

Do this error occur after you have exported your application using Eclipse runnable jar (unpack all files into the jar) or some other kind of "onejar" deployment option?
Comment 2 dylan.millikin 2015-07-25 16:23:23 CEST
Sorry for the delay in response.
I've simply imported the library with maven as stated in my original post.
I then:
mvn package // which produces a jar with dependencies
java -cp target/my-app-1.0-SNAPSHOT-with-dependencies.jar com.mycompany.app.App

The application then creates a temp folder with the libgluegen-rt.so and then I get the error.

I haven't one any extra installation. It may be possible that netbeans is doing something in the background but I would find that surprising.
Comment 3 dylan.millikin 2015-07-25 16:40:04 CEST
Thinking about it I guess this counts as a one-jar deployement. Do you know how this could be fixed? The builde section of the pom.xml is as follows:

  <build>
      <plugins>
          <plugin>
            <!-- NOTE: We don't need a groupId specification because the group is
                 org.apache.maven.plugins ...which is assumed by default.
             -->
            <artifactId>maven-assembly-plugin</artifactId>
            <version>2.5.5</version>
            <configuration>
              <descriptorRefs>
                <descriptorRef>jar-with-dependencies</descriptorRef>
              </descriptorRefs>
            </configuration>
            <executions>
            <execution>
              <id>make-assembly</id> <!-- this is used for inheritance merges -->
              <phase>package</phase> <!-- bind to the packaging phase -->
              <goals>
                <goal>single</goal>
              </goals>
            </execution>
          </executions>
          </plugin>
      </plugins>
  </build>
Comment 4 Sven Gothel 2015-08-05 17:00:57 CEST
Well, as Xerxes pointed out, this is a maven deployment issue.
In jogl-demos, Mark has provided a test for using our maven repos
and they are working.

In case you like to use a one-jar, a.k.a. fat-jar approach,
you can right now, look at Bug 1145.
However, we have no fat-jar in our maven repo (yet).
IMHO .. this is not intended .. but dunno.

Adding Mark to the CC list, in case he likes to join the conversation.

Pls reopen bug in case it turns out this issue is our fault.
Comment 5 Xerxes Rånby 2015-08-06 01:12:54 CEST
(In reply to comment #4)
> Well, as Xerxes pointed out, this is a maven deployment issue.
> In jogl-demos, Mark has provided a test for using our maven repos
> and they are working.
> 
> In case you like to use a one-jar, a.k.a. fat-jar approach,
> you can right now, look at Bug 1145.
> However, we have no fat-jar in our maven repo (yet).
> IMHO .. this is not intended .. but dunno.
> 
> Adding Mark to the CC list, in case he likes to join the conversation.
> 
> Pls reopen bug in case it turns out this issue is our fault.

I can propose a solution to fix overwrites of natives that occur when using:
* the maven maven-assembly-plugin single goal jar deployment as described in this bug
* the eclipse "one-jar" -> export -> java runnable jar file -> * "Extract required libraries into generated JAR"
deployment options.

We can avoid the overwrites of natives if JogAmp do:
* change the location of the native library inside the "single" native jars 
  -> to use the same in-jar folder structure for native jars as the fat-jar.

Example: the location of the native library inside the "single" native jar for gluegen-rt linux-armv6hf 
gluegen-rt-natives-linux-armv6hf.jar
is currently 
/libgluegen-rt.so

If this was changed to use same in-jar folder structure for native jars as the fat-jar then the new in-jar location would be:
/natives/linux-armv6hf/libgluegen-rt.so

Such a change would allow users to use the two new deployment options that extract all jars into one as mentioned above.
Comment 6 Julien Gouesse 2015-08-11 19:37:34 CEST
(In reply to comment #5)
> (In reply to comment #4)
> > Well, as Xerxes pointed out, this is a maven deployment issue.
> > In jogl-demos, Mark has provided a test for using our maven repos
> > and they are working.
> > 
> > In case you like to use a one-jar, a.k.a. fat-jar approach,
> > you can right now, look at Bug 1145.
> > However, we have no fat-jar in our maven repo (yet).
> > IMHO .. this is not intended .. but dunno.
> > 
> > Adding Mark to the CC list, in case he likes to join the conversation.
> > 
> > Pls reopen bug in case it turns out this issue is our fault.
> 
> I can propose a solution to fix overwrites of natives that occur when using:
> * the maven maven-assembly-plugin single goal jar deployment as described in
> this bug
> * the eclipse "one-jar" -> export -> java runnable jar file -> * "Extract
> required libraries into generated JAR"
> deployment options.
> 
> We can avoid the overwrites of natives if JogAmp do:
> * change the location of the native library inside the "single" native jars 
>   -> to use the same in-jar folder structure for native jars as the fat-jar.
> 
> Example: the location of the native library inside the "single" native jar
> for gluegen-rt linux-armv6hf 
> gluegen-rt-natives-linux-armv6hf.jar
> is currently 
> /libgluegen-rt.so
> 
> If this was changed to use same in-jar folder structure for native jars as
> the fat-jar then the new in-jar location would be:
> /natives/linux-armv6hf/libgluegen-rt.so
> 
> Such a change would allow users to use the two new deployment options that
> extract all jars into one as mentioned above.

I agree with this change, I'm all for it, it would ease the use of build tools that can't rely on jogamp-fat.jar, they would just have to preserve the location of the native libraries, for example with Maven and Gradle.
Comment 7 Xerxes Rånby 2015-08-12 12:50:15 CEST
Branch to merge:
https://github.com/xranby/gluegen/tree/bug1172

Implements: Use the same in-jar folder structure for native jars as the fat-jar.

gluegen and jogl
    cd make
    ant
    ant junit.run
passes.
Comment 8 Sven Gothel 2015-08-18 02:56:55 CEST
3f73bbbd44721cc666e4d3505fcf163490636ba8
  Use the same in-jar folder structure for native jars as the fat-jar

  Patches macro 'native.tag.jar' to use the native/os.and.arch
  structure as used in fat-jar, as used by all modules
  for native jar production.
Comment 9 Sven Gothel 2015-08-18 03:31:50 CEST
commit 41d89263109d20dbcfcc7a642c88a290b4877b5f

    TempJarCache: Only copy native library files from 
    'natives/os.and.arch', reducing JAR search.
    
    Since all native libraries are now contained within 'natives/os.and.arch',
    we don't need to search the whole JAR file anymore
    but simply can copy the content of the defined folder - if existing.
Comment 10 Sven Gothel 2015-08-18 03:35:35 CEST
Changed title to match patch, was:
"Fix Failure to load libgluegen-rt.so after maven maven-assembly-plugin single goal jar deployment"
Comment 11 Sven Gothel 2015-08-18 12:21:45 CEST
b4ad01b53421a58ccfe7028a520cf3e06d6b6742
  Add performance counter for native-jar lookup: 
  Property 'jogamp.debug.JNILibLoader.Perf'

Xerxes could confirm, regarding 
commit 41d89263109d20dbcfcc7a642c88a290b4877b5f, comment 9:
"we reduced startup time with ~half a second on the raspberry pi by reducing places to look for natives etc"
Comment 12 Sven Gothel 2015-08-18 12:30:33 CEST
On a fast desktop machine (sgothel):
  libgluegen-rt.jar + jogl-all.jar -> 14 ms

On a raspi2 (xerxes):
  libgluegen-rt.jar + jogl-all.jar -> 80 ms
Comment 13 Xerxes Rånby 2015-08-28 01:40:41 CEST
*** Bug 1118 has been marked as a duplicate of this bug. ***