Bug 1246

Summary: gluegen: please add support for Linux ppc64le
Product: [JogAmp] Gluegen Reporter: gilles.filippini
Component: coreAssignee: Sven Gothel <sgothel>
Status: RESOLVED FIXED    
Severity: enhancement CC: gilles.filippini, gouessej
Priority: P4    
Version: 2.4.0   
Hardware: Other   
OS: linux   
Type: FEATURE SCM Refs:
gluegen 6d87df8b109f045433575cd94b22ba8d8150903a
Workaround: ---
Attachments: ppc64le support proposal

Description gilles.filippini 2015-10-05 22:58:15 CEST
Hi,

The only difference with LP64_UNIX seems to be the page size of 65536.
I tried to do this by myself but the hardware detection / description is so scattered over several java classes that I had to give up.

Many thanks in advance.
Comment 1 Sven Gothel 2015-10-06 03:31:43 CEST
- dunno whether it is scattered -> static MachineDescription

- Since it is an ELF binary, we should be able to detect it.

- As you mentioned, the PAGESIZE is different and the
  static (assumed) may not match the queried one at runtime,
  but AFAIK we prefer the runtime one .. no?
  So does it crash -> exception?

- Then you can simply try to remove that criteria from the 
  comparison .. IMHO OK.

- Does the ELF reader properly detect the platform?

- Also -> version 2.4.0
Comment 2 gilles.filippini 2015-10-06 21:40:34 CEST
Created attachment 754 [details]
ppc64le support proposal
Comment 3 gilles.filippini 2015-10-06 21:41:50 CEST
Hi,

I gave it another try and got it to build eventually.
Patch attached. Comments welcome.

Thanks.
Comment 4 Sven Gothel 2015-10-06 23:28:01 CEST
(In reply to gilles.filippini from comment #3)
> Hi,
> 
> I gave it another try and got it to build eventually.
> Patch attached. Comments welcome.
> 
> Thanks.

Hi Gilles(?),

first of all KUDOS for your approach diving into our
ant build system and data!

After the 2.3.2 release I like to take your patches!

As mentioned earlier, we may be able to enhance the generic
validation, so the pagesize criteria does not hit.
This will allow us not needing to add the static MachineData config
and be more versatile for other similar LP64 arch, 
see comment 1.

But this can be done in a followup commit.

For now, it would be great, if you can offer your patch
either as:
  - git patch (email formatted)
  - git pull request

This way your authorship is being preserved
and this is required to allow an code audit to defend our IP.
(i.e. that we don't violate rights of others .. i.e. your employer)

Thank you, Sven
Comment 5 gilles.filippini 2015-10-07 11:04:19 CEST
(In reply to Sven Gothel from comment #4)
> first of all KUDOS for your approach diving into our
> ant build system and data!

Thanks :)

> 
> After the 2.3.2 release I like to take your patches!
> 
> As mentioned earlier, we may be able to enhance the generic
> validation, so the pagesize criteria does not hit.
> This will allow us not needing to add the static MachineData config
> and be more versatile for other similar LP64 arch, 
> see comment 1.
> 
> But this can be done in a followup commit.
> 
> For now, it would be great, if you can offer your patch
> either as:
>   - git patch (email formatted)
>   - git pull request
> 
> This way your authorship is being preserved
> and this is required to allow an code audit to defend our IP.
> (i.e. that we don't violate rights of others .. i.e. your employer)

Will do, but before I need to test this patch usability.
I've somewhat adapted it to the release 2.2.4 currently in Debian but failed to run the classical OneTriangleAWT example:

Exception in thread "main" java.lang.ExceptionInInitializerError
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:278)
	at javax.media.nativewindow.NativeWindowFactory$3.run(NativeWindowFactory.java:338)
	at javax.media.nativewindow.NativeWindowFactory$3.run(NativeWindowFactory.java:334)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.media.nativewindow.NativeWindowFactory.initSingleton(NativeWindowFactory.java:334)
	at javax.media.opengl.GLProfile.initProfilesForDefaultDevices(GLProfile.java:1690)
	at javax.media.opengl.GLProfile.access$000(GLProfile.java:77)
	at javax.media.opengl.GLProfile$1.run(GLProfile.java:201)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.media.opengl.GLProfile.initSingleton(GLProfile.java:187)
	at javax.media.opengl.GLProfile.getProfileMap(GLProfile.java:2246)
	at javax.media.opengl.GLProfile.get(GLProfile.java:959)
	at javax.media.opengl.GLProfile.getDefault(GLProfile.java:693)
	at javax.media.opengl.GLProfile.getDefault(GLProfile.java:704)
	at OneTriangleAWT.main(OneTriangleAWT.java:19)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 9
	at jogamp.nativewindow.jawt.JAWT.size(JAWT.java:29)
	at jogamp.nativewindow.jawt.JAWT.create(JAWT.java:33)
	at jogamp.nativewindow.jawt.JAWTUtil.getJAWT(JAWTUtil.java:246)
	at jogamp.nativewindow.jawt.JAWTUtil.<clinit>(JAWTUtil.java:330)
	... 16 more

Does this backtrace tell you something?

Thanks,

_gilles.
Comment 6 gilles.filippini 2015-10-07 15:40:57 CEST
(In reply to gilles.filippini from comment #5)
> I've somewhat adapted it to the release 2.2.4 currently in Debian but failed
> to run the classical OneTriangleAWT example:

As I understand it jogl needs to be patched as well for this example to work :/

Is there a simple way to check for gluegen usability which does not require jogl?

Thanks,

_g.
Comment 7 Sven Gothel 2015-10-07 23:20:01 CEST
(In reply to gilles.filippini from comment #6)
> (In reply to gilles.filippini from comment #5)
> > I've somewhat adapted it to the release 2.2.4 currently in Debian but failed
> > to run the classical OneTriangleAWT example:
> 
> As I understand it jogl needs to be patched as well for this example to work
> :/

Yes, sorry, this is a bit annoying I know.
That would be the next thing on you platter of support:
  - joal, jogl and jocl

> 
> Is there a simple way to check for gluegen usability which does not require
> jogl?

- just run the gluegen unit tests.

all via ant, or a single one as I do:

cd gluegen/make
vi scripts/runtest.sh
bash scripts/runtest.sh ../build

IMHO the following test should be sufficient,
triggering the 'loading path':
  com.jogamp.common.util.TestPlatform01 

~Sven
Comment 8 gilles.filippini 2015-10-08 22:58:48 CEST
How are these source files in src/java/jogamp/common/os/elf/ supposed to be generated?

Ehdr.java
Shdr.java

Is there an option to force the target generate.os.sources during the build?

Thanks,

_g.
Comment 9 gilles.filippini 2015-10-09 00:09:14 CEST
One more step. The ArrayIndexOutOfBoundsException is solved and the OneTriangleAWT now fails with this error:

libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00003fffa95ee284, pid=9911, tid=70366626443680
...

Since I test through a vnc session on a remote Debian porterbox I can't tell where the problem is. Could this kind of error come from a faulty gluegen / jogl build?

Thanks,

_g.
Comment 10 gilles.filippini 2015-10-09 01:07:32 CEST
(In reply to gilles.filippini from comment #9)
> One more step. The ArrayIndexOutOfBoundsException is solved and the
> OneTriangleAWT now fails with this error:
> 
> libGL error: No matching fbConfigs or visuals found
> libGL error: failed to load driver: swrast
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x00003fffa95ee284, pid=9911, tid=70366626443680
> ...
> 
> Since I test through a vnc session on a remote Debian porterbox I can't tell
> where the problem is. Could this kind of error come from a faulty gluegen /
> jogl build?
> 
> Thanks,
> 
> _g.

Found a stack trace:
Stack: [0x00003fff26cc0000,0x00003fff26ec0000],  sp=0x00003fff26ebd710,  free space=2037k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x54e284]  jni_GetLongField+0x74
V  [libjvm.so+0x566b68]  jni_GetDirectBufferAddress+0x118
C  [libnativewindow_x11.so+0x42ec]  Java_jogamp_nativewindow_x11_X11Lib_XRenderFindVisualFormat1+0x54
j  jogamp.nativewindow.x11.X11Lib.XRenderFindVisualFormat1(JJ)Ljava/nio/ByteBuffer;+0
j  jogamp.nativewindow.x11.X11Lib.XRenderFindVisualFormat(JJ)Ljogamp/nativewindow/x11/XRenderPictFormat;+2
j  jogamp.opengl.x11.glx.X11GLXGraphicsConfiguration.XVisual2XRenderMask(JJ)Ljogamp/nativewindow/x11/XRenderDirectFormat;+2
j  jogamp.opengl.x11.glx.X11GLXGraphicsConfiguration.XVisualInfo2GLCapabilities(Lcom/jogamp/nativewindow/x11/X11GraphicsDevice;Ljavax/media/opengl/GL
Profile;Ljogamp/nativewindow/x11/XVisualInfo;IZ)Ljogamp/opengl/x11/glx/X11GLCapabilities;+256
j  jogamp.opengl.x11.glx.X11GLXGraphicsConfigurationFactory.chooseGraphicsConfigurationXVisual(Ljavax/media/opengl/GLCapabilitiesImmutable;Ljavax/med
ia/opengl/GLCapabilitiesImmutable;Ljavax/media/opengl/GLCapabilitiesChooser;Lcom/jogamp/nativewindow/x11/X11GraphicsScreen;I)Ljogamp/opengl/x11/glx/X
11GLXGraphicsConfiguration;+269
j  jogamp.opengl.x11.glx.X11GLXGraphicsConfigurationFactory.chooseGraphicsConfigurationStatic(Ljavax/media/opengl/GLCapabilitiesImmutable;Ljavax/medi
a/opengl/GLCapabilitiesImmutable;Ljavax/media/opengl/GLCapabilitiesChooser;Lcom/jogamp/nativewindow/x11/X11GraphicsScreen;I)Ljogamp/opengl/x11/glx/X1
1GLXGraphicsConfiguration;+164
j  jogamp.opengl.x11.glx.X11GLXDrawableFactory.createMutableSurfaceImpl(Ljavax/media/nativewindow/AbstractGraphicsDevice;ZLjavax/media/opengl/GLCapab
ilitiesImmutable;Ljavax/media/opengl/GLCapabilitiesImmutable;Ljavax/media/opengl/GLCapabilitiesChooser;Ljavax/media/nativewindow/UpstreamSurfaceHook;
)Ljavax/media/nativewindow/ProxySurface;+69
j  jogamp.opengl.x11.glx.X11GLXDrawableFactory.createDummySurfaceImpl(Ljavax/media/nativewindow/AbstractGraphicsDevice;ZLjavax/media/opengl/GLCapabil
itiesImmutable;Ljavax/media/opengl/GLCapabilitiesImmutable;Ljavax/media/opengl/GLCapabilitiesChooser;II)Ljavax/media/nativewindow/ProxySurface;+24
j  jogamp.opengl.x11.glx.X11GLXDrawableFactory$SharedResourceImplementation.createSharedResource(Ljava/lang/String;)Ljogamp/opengl/SharedResourceRunn
er$Resource;+133
j  jogamp.opengl.SharedResourceRunner.run()V+267
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
V  [libjvm.so+0x53a59c]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x46c
V  [libjvm.so+0x7c5c14]  os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x34
V  [libjvm.so+0x539014]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x224
V  [libjvm.so+0x5856a8]  thread_entry(JavaThread*, Thread*)+0xb8
V  [libjvm.so+0x945a58]  JavaThread::thread_main_inner()+0x198
V  [libjvm.so+0x945d68]  JavaThread::run()+0x2a8
V  [libjvm.so+0x7bc050]  java_start(Thread*)+0x160
C  [libpthread.so.0+0x89a4]  start_thread+0xf4
Comment 11 Sven Gothel 2015-10-09 03:18:42 CEST
(In reply to gilles.filippini from comment #8)
> How are these source files in src/java/jogamp/common/os/elf/ supposed to be
> generated?
> 
> Ehdr.java
> Shdr.java
> 
> Is there an option to force the target generate.os.sources during the build?
> 

you could issue their generation manually via ant.

however, we choose to pre-generate certain files.
I would need to check whether those were post-edited or not.
Comment 12 Sven Gothel 2015-10-09 03:20:09 CEST
(In reply to gilles.filippini from comment #9)
> One more step. The ArrayIndexOutOfBoundsException is solved and the
> OneTriangleAWT now fails with this error:
> 
> libGL error: No matching fbConfigs or visuals found
> libGL error: failed to load driver: swrast
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x00003fffa95ee284, pid=9911, tid=70366626443680
> ...
> 
> Since I test through a vnc session on a remote Debian porterbox I can't tell
> where the problem is. Could this kind of error come from a faulty gluegen /
> jogl build?
> 

I assume you produce jogl similar to gluegen, i.e. w/ your compiler flags?

But 1st things 1st: did you succeed w/ running some or all 
GlueGen unit tests?
Not only report the failures - reporting success is also very welcome! :)
Comment 13 gilles.filippini 2015-10-09 22:04:36 CEST
(In reply to Sven Gothel from comment #12)
> But 1st things 1st: did you succeed w/ running some or all 
> GlueGen unit tests?
> Not only report the failures - reporting success is also very welcome! :)

All tests but 6 pass:

TestVersionSemantics doesn't build (I don't have semver).

test01URLCompositioning:
There was 1 failure:
1) test01URLCompositioning(com.jogamp.common.util.TestIOUtilURICompose)
java.net.MalformedURLException: unknown protocol: asset

testTempJarCache02AddNativeLibs:
There was 1 failure:
1) testTempJarCache02AddNativeLibs(com.jogamp.common.util.TestTempJarCache)
java.io.FileNotFoundException: /home/pini/gluegen2-2.2.4/build/gluegen2-rt-natives-linux-ppc64le.jar (No such file or directory)

testJarUtilFlat01:
There were 4 failures:TempFileCache: removeAll(/tmp/jogamp_0000/file_cache/jln8133976196119522112/jln7418965195467938792/jogamp/common/jvm)

1) testJarUtilFlat01(com.jogamp.common.util.TestJarUtil)TempFileCache: removeAll(/tmp/jogamp_0000/file_cache/jln8133976196119522112/jln7418965195467938792/jogamp/common/jvm/JVMUtil.class)

test00TempJarCacheSimplePath:
There were 2 failures:
1) test00TempJarCacheSimplePath(com.jogamp.common.net.TestNetIOURIReservedCharsBug908)
java.lang.UnsatisfiedLinkError: Can't load library: /home/pini/gluegen2-2.2.4/libgluegen2-rt.so

pcppMacroDefinitionTest:
There was 1 failure:
1) pcppMacroDefinitionTest(com.jogamp.gluegen.test.junit.generation.PCPPTest)
java.lang.ExceptionInInitializerError

_g.
Comment 14 gilles.filippini 2015-10-10 01:25:56 CEST
I'm back on the 2.3.1 release because somebody has just uploaded this version of gluegen to Debian experimental.

I'm now puzzled by the tests results where I read in many places:
Duplicate/Compatible MachineDataInfo in StaticConfigs: Elements [7: PPC_64_UNIX(7)] and [6: LP64_UNIX(6)]
MachineDataInfoStatic: PPC_64_UNIX(7): MachineDataInfo: runtimeValidated false, 32Bit false, primitive size / alignment:
  int8    1 / 1, int16   2 / 2
  int     4 / 4, long    8 / 8
  int32   4 / 4, int64   8 / 8
  float   4 / 4, double  8 / 8, ldouble 16 / 16
  pointer 8 / 8, page    65536
MachineDataInfoStatic: LP64_UNIX(6): MachineDataInfo: runtimeValidated false, 32Bit false, primitive size / alignment:
  int8    1 / 1, int16   2 / 2
  int     4 / 4, long    8 / 8
  int32   4 / 4, int64   8 / 8
  float   4 / 4, double  8 / 8, ldouble 16 / 16
  pointer 8 / 8, page    4096

Why is this an error? Obviously the page sizes are different.

Thanks,

_g.
Comment 15 gilles.filippini 2015-10-10 02:37:12 CEST
I've inhibited the "Duplicate/Compatible MachineDataInfo" error and there is only 4 failures left:

1) TestJarUtil fails loading inexistant gluegen2-rt-natives-linux-ppc64le.jar which is not generated by the Debian package. I guess I can ignore this error.

2) TestUri99LaunchOnReservedCharPathBug908 fails loading libgluegen2-rt.so with this error: 
n/a - Native Library /home/pini/debian/gluegen2/debian/libgluegen2-jni/usr/lib/jni/libgluegen2-rt.so already loaded in another classloader

3) and 4) have the same pattern: TestStructGen01 and TestStructGen02 fail with:
1) test01(com.jogamp.gluegen.test.junit.structgen.TestStructGen02)
java.lang.InternalError: Not set
        at jogamp.common.os.MachineDataInfoRuntime.getStatic(MachineDataInfoRuntime.java:78)
        at com.jogamp.gluegen.test.junit.structgen.Pixel.<clinit>(Pixel.java:18)
        at com.jogamp.gluegen.test.junit.structgen.TestStructGen02.test01(TestStructGen02.java:23)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
...

I guess the latter is the most annoying. What do you think?

Thanks,

_g
Comment 16 gilles.filippini 2015-10-10 13:58:24 CEST
Hi,

The good new is that I have exactly the same failures on arch amd64 :)

Thanks,

_g.
Comment 17 gilles.filippini 2015-10-12 11:16:27 CEST
Pull request created.
==> https://github.com/sgothel/gluegen/pull/29

Thanks,

_g.
Comment 18 gilles.filippini 2015-10-17 14:06:37 CEST
Good news: the patch was finally tested on a baremetal ppc64le machine, and it works.
===> https://lists.debian.org/debian-powerpc/2015/10/msg00035.html
Comment 19 Julien Gouesse 2015-10-19 15:57:35 CEST
(In reply to gilles.filippini from comment #17)
> Pull request created.
> ==> https://github.com/sgothel/gluegen/pull/29
> 
> Thanks,
> 
> _g.

Great job :) I have just looked at your pull request, it's minimal, it does the job, big kudos to you.
Comment 20 gilles.filippini 2015-10-28 14:53:52 CET
Hi,

Now I better understand comment 1. I've canceled the previous pull request and created another one with relaxed constraint on runtime pagesize.
==> https://github.com/sgothel/gluegen/pull/30

Thanks,

_g.
Comment 21 Sven Gothel 2019-03-30 01:43:33 CET
Merged Gilles pull request, GlueGen commit 6d87df8b109f045433575cd94b22ba8d8150903a

In case this is not sufficient to build all modules, please re-open and advise.