JogAmp IRC Chat Logs

#jogamp @ irc.freenode.net - 20140623 05:05:27 (UTC)

20140623 05:05:27 -jogamp- Previous @ http://jogamp.org/log/irc/jogamp_20140622050527.html
20140623 05:05:27 -jogamp- This channel is logged @ http://jogamp.org/log/irc/jogamp_20140623050527.html
20140623 06:06:12 * monsieur_max (~maxime@anon) Quit (Quit: Leaving.)
20140623 06:58:58 * eclesia (~husky@anon) has joined #jogamp
20140623 06:59:02 <eclesia> hi
20140623 07:39:05 * monsieur_max (~maxime@anon) has joined #jogamp
20140623 08:32:23 * bbbruce (~bx@anon) has joined #jogamp
20140623 09:51:36 * [Mike] (~Mike]@anon) Quit ()
20140623 11:10:02 * jvanek (jvanek@anon) Quit (Quit: Leaving)
20140623 11:34:18 * zzuegg|afk is now known as zzuegg
20140623 12:20:48 * jvanek (jvanek@anon) has joined #jogamp
20140623 13:27:48 <zubzub> Catched FileNotFoundException: /home/zubzub/.m2/repository/org/jogamp/jogl/jogl-all-noawt/2.1.5-01/jogl-all-natives-linux-amd64.jar
20140623 13:28:05 <zubzub> however the file is in:
20140623 13:28:07 <zubzub> /home/zubzub/.m2/repository/org/jogamp/jogl/jogl-all/2.1.5-01/
20140623 13:28:56 <zubzub> (probably from when I still had the dep with awt)
20140623 13:38:36 <zubzub> the no-awt native deps
20140623 13:38:52 <zubzub> all have a jogl-all artifact id
20140623 13:39:01 <zubzub> with a native classifier
20140623 13:39:14 <zubzub> yet for some reason
20140623 13:40:48 <zubzub> it looks for them in the jogl-all-noawt
20140623 13:43:11 <zubzub> ofcourse this will not work
20140623 13:54:40 <sgothel> jogl-all-noawt gets aliased w/ jogl-all for native jar lookup purposes ..
20140623 13:54:44 <sgothel> (the file name)
20140623 13:55:01 <sgothel> all files (java + native jars) must be in same URI
20140623 14:14:31 * rmk0 eyes everything
20140623 14:15:02 <rmk0> hrm
20140623 14:15:52 <rmk0> that sounds like a maven package problem
20140623 14:16:23 * monsieur_max1 (~maxime@anon) has joined #jogamp
20140623 14:17:02 <rmk0> ... i don't see a way to solve that
20140623 14:17:24 <rmk0> maven isn't going to allow those natives to appear in the correct directory
20140623 14:17:49 <rmk0> zubzub: can you file a bug so that this one doesn't get lost?
20140623 14:18:41 <rmk0> sgothel: what would be the consequences of removing that aliasing?
20140623 14:19:06 * monsieur_max (~maxime@anon) Quit (Ping timeout: 240 seconds)
20140623 14:19:12 <rmk0> assuming that we also produced jogl-all-noawt-natives-linux-amd64.jar and so on
20140623 14:19:34 <rmk0> (would just be copies of jogl-all-natives-*.jar)
20140623 14:21:16 <zubzub> rmk0: I will try to reproduce on another machine this evening first
20140623 14:21:34 <zubzub> as I did not seem to have the problem there
20140623 14:21:39 <sgothel> @Mark: jogl stops working w/ no-awt :)
20140623 14:22:03 <sgothel> i.e. we have certain aliasing .. let me find the commit .. / code
20140623 14:22:27 <zubzub> what is the reason you try to load the native jar directly from the FS and not from the classpath?
20140623 14:22:49 <zubzub> name clashes?
20140623 14:22:58 <sgothel> we try to load the native jar directly from the FS - yes
20140623 14:23:16 <sgothel> we don't want to 'spam' the ClassLoader .. hell no
20140623 14:23:25 <sgothel> then .. we have no classes in native jar files
20140623 14:23:54 <sgothel> we really need the native jar file, unzip it .. copy to temp folder and offer its content for our native lib loading
20140623 14:24:15 <rmk0> sgothel: i meant if we also produced native jar files named jogl-all-noawt-natives*.jar that are just exact copies of the ordinary jogl-all-natives* jars
20140623 14:24:56 <zubzub> sgothel: but why look them up and not include them on the classpath?
20140623 14:25:39 <zubzub> (& extract them to a tmp folder and native load them)
20140623 14:25:44 <rmk0> with maven, the natives will be on the classpath... there's no way to stop that either
20140623 14:25:56 <rmk0> (just so everyone's clear)
20140623 14:26:05 <sgothel> @zuzub: we attempt to load a native file .. like liblala.so
20140623 14:26:20 <sgothel> so the native jar file name .. marks the platform
20140623 14:26:41 <sgothel> and again: don't waste ClassLoader .. it makes things more expensive for application class-lookup
20140623 14:27:09 <zubzub> I don't understand
20140623 14:27:29 <zubzub> I see what jogl is doing now
20140623 14:27:36 <zubzub> it looks up a jar at runtime
20140623 14:27:40 <zubzub> extracts the native file
20140623 14:27:45 <zubzub> native loads it
20140623 14:27:48 <sgothel> you daisy chain ClassLoader .. if you try to find something in there .. a visitor walks through all of 'em ..
20140623 14:28:00 <sgothel> until found
20140623 14:28:17 <sgothel> then - the native library file has a platform agnostic name
20140623 14:28:49 <sgothel> then - you need to copy the native lib file anyways for every classloader ! otherwise you can't use them w/ many (internal JVM restriction)
20140623 14:29:11 <sgothel> It's all explained in TempJarFile/Cache .. ..
20140623 14:30:20 <zubzub> why not give it a gnostic file name
20140623 14:30:26 <zubzub> load/link it once to the jvm
20140623 14:30:36 <zubzub> (find it first on the classpath)
20140623 14:30:39 <zubzub> done
20140623 14:30:40 <zubzub> ?
20140623 14:31:41 <sgothel> further more: with this logic, we will load the required file online on a 'per-need' basis
20140623 14:31:48 <sgothel> think applets etc
20140623 14:32:15 <sgothel> ^^ b/c it will not help w/ all the other issues
20140623 14:32:58 <sgothel> @Mark: I don't get the issue .. really
20140623 14:33:30 <sgothel> IMHO all jar and native jar files needed shall reside in one folder/URI for execution
20140623 14:33:40 <sgothel> don't know the maven implications .. hmm
20140623 14:34:03 <zubzub> sgothel: I still don't understand what the issue is. I will try to read the TempJarFile/Cache.
20140623 14:34:20 <zubzub> and see how osgi solves it
20140623 14:34:49 <sgothel> 1) one native lib copy per class-loader
20140623 14:34:55 <sgothel> 2) load native jars on demand
20140623 14:35:02 <rmk0> sgothel: we've organized the maven packages such that the jogl-all natives appear in the same directory as the jogl-all jar in the filesystem (in maven's filesystem cache)
20140623 14:35:06 <sgothel> 3) agnostic native lib file
20140623 14:35:27 <sgothel> 4) don't waster performance - don't append _all_ native jar files to ClassLoader
20140623 14:35:54 <sgothel> great
20140623 14:36:07 <rmk0> jogl-all-noawt is a different package and therefore doesn't have the jogl-all natives in the same directory as the jogl-all-noawt jar
20140623 14:36:09 <sgothel> and also jogl-all-noawt ?
20140623 14:36:11 <sgothel> ah
20140623 14:36:17 <rmk0> because maven just outright won't allow that
20140623 14:36:31 <sgothel> so it needs a copy I am afraid ..
20140623 14:36:34 <rmk0> i could make it so that there are copies of the jogl-all natives instead named jogl-all-noawt*
20140623 14:36:42 <rmk0> but that won't work either due to the aliasing
20140623 14:36:54 <sgothel> no renaming
20140623 14:36:57 <rmk0> i don't control the initial parts of the filenames, those are decided by maven
20140623 14:37:30 <sgothel> can noawt folder have all-noawt java jar and all-native jar (copy) ?
20140623 14:37:59 <sgothel> in the end .. the user may choose one configuration, usually
20140623 14:38:32 * zzuegg (zzuegg@anon) Quit (Ping timeout: 272 seconds)
20140623 14:38:47 <rmk0> like i said... i don't control the names. if i added the natives to the jogl-all-noawt package, they'd end up being named like jogl-all-noawt-2.1.5-natives-android-armv6.jar and so on
20140623 14:38:54 <rmk0> there's no way around that, is just how maven works
20140623 14:39:03 <sgothel> ah
20140623 14:39:14 <sgothel> .. let me double check ..
20140623 14:39:20 <rmk0> the way we're handling things now in maven is sort of borderline abusive
20140623 14:39:31 <sgothel> jogl-all-noawt.jar -> jogl-all package ?
20140623 14:39:48 <rmk0> .. not sure what you're asking
20140623 14:40:07 <zubzub> 1) why one lib copy per class-loader? the native lib is linked to the jvm and becomes visible to all classes (?), 2) yes, on demand. ClassFromNativeJar.getClassLoader().getResource(), 3) why agnostic lib file name? when searching for the jar it's not agnostic either? 4) how does this impact performance? arent resource loaded by the classloader on demand (ie streamed)?
20140623 14:41:29 <sgothel> JNILibLoaderBase: private static final String[] joglDeployCfg = new String[] { "-noawt", "-mobile", "-core", "-android" };
20140623 14:41:50 <sgothel> ^^ those suffixes maybe stripped if appended to all-*
20140623 14:42:02 <sgothel> so it's more than awt/noawt, also android ..
20140623 14:43:50 <sgothel> @zuzub: 1) read JVM specs / constraints - you will find it ; 2) on demand over the network dude 3) since thats how it is now 4) All CL lookups walk through all daisy chained CLs (parent) until found or .. not
20140623 14:44:39 <zubzub> sgothel, dude, I'm not question your solution. Just trying to understand.
20140623 14:44:48 <sgothel> also 3) .. gnostic lib names moves the complication from jar packaging to each module .. which is quite expensive and inconvenient
20140623 14:45:10 <sgothel> ay .. but also try to find the buzz words in the spec/inet :)
20140623 14:46:30 <sgothel> @Mark: It seems we need to think about a 'new' solution :)
20140623 14:47:14 <sgothel> i.e. a maven packaging matching our use case, hoping maven is not like SAP .. where users need to follow their suite :)
20140623 14:47:51 <rmk0> urhur
20140623 14:48:02 <sgothel> Note: Allowing more 'failed' lookups .. to find a resource is expensive and will slowdown startup experience
20140623 14:48:09 <sgothel> .. sure, we know that one
20140623 14:48:33 <sgothel> so, allowing a 'all-noawt' native jar is no option really
20140623 14:48:44 <sgothel> and sure we hate to have duplicates anyways
20140623 14:49:27 <sgothel> goal would be to have the artifacts aligned similar as in jogamp .. i.e. all jars shall reside in one folder location for jogamp stuff
20140623 14:49:48 <sgothel> at least this is how it is when using our builds, webserver .. etc
20140623 14:49:56 <rmk0> the only way to achieve that in maven would be to have one package
20140623 14:50:09 <sgothel> 'till now, we were just lucky I guess .. hmm
20140623 14:50:32 <sgothel> how did you test jogl-demos w/ android ?
20140623 14:50:42 <sgothel> ah .. via the apk .. right
20140623 14:50:48 <sgothel> no need for native jars .. hmm
20140623 14:50:54 <rmk0> right
20140623 14:51:06 <sgothel> so .. there can't be a final post-processing stage ?
20140623 14:51:17 <sgothel> a final module .. which collects cached artifacts ?
20140623 14:51:37 <sgothel> (for runtime purposes only ..)
20140623 14:51:38 <rmk0> i think a lot of people do that anyway
20140623 14:51:42 <sgothel> (maven runtime :)
20140623 14:51:56 <rmk0> i generally dump everything into one lib/ directory and have a shell script that adds everything in there to the classpath and runs the program
20140623 14:52:15 <rmk0> but the dependencies have to be in the right place in the maven cache in order to allow things to work correctly when running from inside eclipse and so on
20140623 14:52:21 <rmk0> where a full maven build hasn't necessarily taken place
20140623 14:52:55 <rmk0> we're basically entirely at the mercy of where these systems like to save their cached jars... there's an open gradle bug about our maven packages
20140623 14:53:01 <sgothel> we use it for build dependencies ..
20140623 14:53:11 <rmk0> all of the systems use maven packages, but they all have different rules about how/where they'll be saved in the filesystem
20140623 14:53:20 <rmk0> we have no control over any of that
20140623 14:53:22 <sgothel> so a last jogamp module could fetch jars to it's own artifact folder .. no ?
20140623 14:53:41 <sgothel> so a module doesn't know about it's artifact location ?
20140623 14:53:48 <rmk0> don't think so, no
20140623 14:53:53 <sgothel> ah .. oops
20140623 14:54:05 <sgothel> the classloader does I guess
20140623 14:54:13 <sgothel> otherwise ..
20140623 14:54:16 <rmk0> maven will blindly add the jar files of every module upon which the current module depends
20140623 14:54:21 <rmk0> to the classpath, i mean
20140623 14:54:25 <rmk0> is no way to stop that happening either
20140623 14:54:31 <sgothel> yeah .. so we can snoop the classpath ?
20140623 14:54:37 <sgothel> and copy ..
20140623 14:55:00 <rmk0> for maven, at least
20140623 14:55:11 <sgothel> so collect-module runs a simple class utilizing our many GluegenVersion, JoglVersion .. and copies those jar files exposed by it
20140623 14:55:26 <sgothel> *Version does expose it's location ..
20140623 14:55:28 <rmk0> ... you think it'd be worth making this part of the API?
20140623 14:55:39 <rmk0> like, allowing the user to decide how jogamp stuff finds itself?
20140623 14:55:52 <sgothel> we have it in already .. runtime version stuff ..
20140623 14:56:01 <rmk0> not familiar
20140623 14:56:40 <sgothel> http://jogamp.org/wiki/index.php/Jogamp_Versioning_and_Releases#Runtime_Version_Check
20140623 14:56:46 * rmk0 eyes it
20140623 14:57:22 <sgothel> this allows users to give proper bug reports .. so you know git-sha1 etc ..
20140623 14:57:28 * zubzub thinks it would be very nice if you could finally use gradle with jogl in his ide
20140623 14:57:37 * zubzub and dump maven
20140623 14:57:42 <sgothel> all queried via the JAR file's manifest
20140623 14:57:59 <rmk0> sgothel: not sure how this allows people to control how jogl finds its natives and so on
20140623 14:58:02 <rmk0> maybe that's not what you meant
20140623 14:58:02 <sgothel> we pass the data from our jenkins slaves .. etc
20140623 14:58:22 <sgothel> it may allow a maven module to find the jogl jar files .. and copy it
20140623 14:58:39 <sgothel> of all those maven variants all, all-noawt, .. etc
20140623 14:59:01 <sgothel> but - it sure would be another ugly step .. hmm
20140623 14:59:26 <rmk0> zubzub: the osgi stuff doesn't use a classpath at all, does it?
20140623 15:03:07 <zubzub> it does afaik
20140623 15:03:28 <zubzub> you tell in your manifest where the native lib is
20140623 15:03:31 <zubzub> and for what arch
20140623 15:04:00 <zubzub> the framework then uses that info to load the native lib with that bundle's classloader
20140623 15:04:05 <rmk0> oh... may just be ceylon then
20140623 15:05:21 <zubzub> http://blog.vogella.com/2010/07/27/osgi/
20140623 15:06:46 <rmk0> hm
20140623 15:08:24 <rmk0> the way LWJGL handles this in maven is as uninteresting as possible
20140623 15:08:49 <rmk0> they use a plugin called maven-natives... literally just unpacks a given list of archives to a directory and adds them to -Djava.library.path
20140623 15:09:27 <rmk0> obviously lwjgl doesn't have any of the same native loading stuff that jogl has
20140623 15:09:37 <rmk0> requires the person running the program to set the library path
20140623 15:09:51 <bbbruce> :(
20140623 15:10:47 <zubzub> the way qtjambi handles it, is by extracting the native library from a resource on the classpath
20140623 15:11:27 <zubzub> ie the way I described earlier
20140623 15:11:48 <zubzub> but because jogl <3 applets, that's not an option :(
20140623 15:12:46 <rmk0> i don't know enough about applets to comment on what is and isn't possible with them
20140623 15:13:10 <rmk0> i'm not even sure if they get filesystem access, but presumably they do if TempJarCache exists
20140623 15:15:45 <rmk0> so... right
20140623 15:16:33 <rmk0> the basic requirement is that the native jar files need to be in the same directory as the current module's jar
20140623 15:16:53 <rmk0> realistically, that requirement has to be even stronger: all jogamp jars need to be in the same directory
20140623 15:17:52 <rmk0> we need to support applets (apparently), compiling from maven (meaning the jar files have to be where maven thinks they should be), running from eclipse, running from eclipse as a maven project, compiling from gradle...
20140623 15:18:59 <rmk0> "something" has to get all of those files into one directory that jogl then knows how to locate
20140623 15:19:17 <sgothel> (back .. reading)
20140623 15:20:14 <rmk0> we've not decided whether that "something" has to be in the build processes of each (no idea how to achieve that in eclipse), or done by jogl at runtime
20140623 15:20:45 <rmk0> in maven and maven/eclipse, we can always assume that all of the native jars will be on the classpath (because there's no way to stop that happening)
20140623 15:20:52 <rmk0> otherwise... we can't assume that
20140623 15:20:57 <rmk0> don't know about gradle
20140623 15:21:13 <zubzub> with gradle they will be on the classpath as well
20140623 15:21:16 <rmk0> ok
20140623 15:21:43 <zubzub> because they are marked as a (runtime+compile time) dep
20140623 15:21:52 <rmk0> yeah, same reason as in maven
20140623 15:22:03 <sgothel> "all jogamp jars need to be in the same directory" <- not necessarily, i.e. gluegen, jogl, joal .. can stay apart
20140623 15:22:31 <rmk0> ok
20140623 15:22:51 <rmk0> although i think it might be simpler with that requirement left in, actually... considering jogl-noawt
20140623 15:23:06 <sgothel> logic is: find me the native jar file of this class's containing class-jar file
20140623 15:23:21 <rmk0> due to the aliasing, jogl-noawt is going to try to look in the jogl-noawt module directory and the natives aren't going to be there
20140623 15:23:27 <rmk0> (in maven)
20140623 15:23:38 <zubzub> correct
20140623 15:23:52 <sgothel> I may be open up for the change of removing the aliasing and deploying it as a copy
20140623 15:24:23 <zubzub> rmk0: in essense it's the same problem as the gradle bug
20140623 15:24:27 <sgothel> this removes one complication in code .. adds duplicate native jar files (well ..) .. if it helps .. just an option, not wanting to block the world here :)
20140623 15:24:28 <rmk0> yeah
20140623 15:24:46 <sgothel> so .. you guys tell me - whether this is a MUST_DO_IT
20140623 15:24:53 <rmk0> not sure yet!
20140623 15:25:00 <sgothel> then .. let's open a bug report .. usual drill :)
20140623 15:25:10 <rmk0> i am concerned about not breaking the ability to run programs in eclipse without having run a full maven build
20140623 15:25:49 <sgothel> hard sentence .. w/ 2 negations .. hmm .. :)
20140623 15:25:53 <rmk0> as far as i can tell, eclipse' maven plugin basically consults the maven dependency information to set up the classpath, but doesn't actually run any other parts of the build
20140623 15:26:32 <rmk0> so someone running a program in eclipse hasn't necessarily executed a maven build (and therefore hasn't necessarily done whatever setup is needed to smash jar files into a single directory)
20140623 15:26:47 <sgothel> if the 'use copy instead of aliased' fixes other issues as well .. dunno
20140623 15:27:11 <sgothel> but you tell me .. cont. w/ trying to break GlueGen .. autch
20140623 15:27:24 <rmk0> hehe, well, seems that none of the other maven packages will work at the moment - had no idea this aliasing stuff was happening
20140623 15:27:27 <sgothel> (more struct access ..)
20140623 15:28:02 <rmk0> zubzub: what was the number of that gradle bug?
20140623 15:29:23 <zubzub> dont' know
20140623 15:29:34 <zubzub> the only bugreport I find for it is on gradle itself
20140623 15:29:36 <zubzub> http://issues.gradle.org/browse/GRADLE-2553
20140623 15:29:52 <rmk0> thought somebody here filed one... assumed it was you
20140623 15:30:06 <zubzub> I don't think I filed one
20140623 15:30:14 <zubzub> I mentioned it but was told 'works as designed'
20140623 15:30:55 * jvanek (jvanek@anon) Quit (Quit: Leaving)
20140623 15:31:00 <zubzub> so I switched back to maven
20140623 15:31:18 <rmk0> so we can't assume that gradle will download the natives attached to each package into the same directory
20140623 15:31:22 <zubzub> and put a 'write own jogl native lib loader' somewhere at the back of my long todo list :)
20140623 15:31:26 <rmk0> is basically one step "worse" than maven
20140623 15:31:42 <zubzub> rmk0: no gradle with dld each artifact in it's own dir
20140623 15:31:49 <rmk0> so i mean... even fixing the current packages and removing the aliasing wouldn't solve this
20140623 15:31:53 <zubzub> and the ide will add them all to the classpath
20140623 15:32:01 <zubzub> rmk0: correct
20140623 15:32:19 <sgothel> how about 'no classpath' artifacts .. ? can be marked as such ?
20140623 15:32:46 <sgothel> looks like we need to 'copy' then .. i.e. all-noawt: java-jar + native-jar -> remove aliasing
20140623 15:32:46 <zubzub> you can mark stuff as pure compile time
20140623 15:32:56 <zubzub> but then you have no idea where to find them
20140623 15:33:00 <sgothel> and pure runtime / no classpath ?
20140623 15:33:03 <sgothel> well ..
20140623 15:33:46 <zubzub> unless you do some pre/post compile time deployment
20140623 15:33:51 <sgothel> @Mark: you tell me about your choice, now really back to hacking :)
20140623 15:34:22 <zubzub> which you could do with a custom gradle script/plugin
20140623 15:34:29 <rmk0> sgothel: haven't made one yet! still trying to collect data and understand all of the problems...
20140623 15:34:30 <zubzub> (gradle is awesome like that)
20140623 15:35:57 <zubzub> anyway, I'm glad there is some talk about it :)
20140623 15:36:12 <zubzub> I'm sure it's not a blocker for most people, but it would make a lot of them happy :)
20140623 15:36:28 <rmk0> would like to avoid having to have custom gradle or maven plugins, mainly because as i said earlier: when running from inside eclipse and so on, there's no guarantee that those plugins have been executed
20140623 15:36:34 * eclesia use maven, really happy with it in netbeans
20140623 15:36:44 <rmk0> would mean that you'd have to run a maven/gradle build, and then have the IDE effectively use the cached results
20140623 15:37:00 <rmk0> if the IDE could even be made to understand where the jar files are... is a mess
20140623 15:37:21 <rmk0> at least at the moment, everything goes on the classpath (even if that's not ideal for other reasons)
20140623 15:37:30 <rmk0> so the IDE isn't dependent on any other state
20140623 15:37:31 <zubzub> rmk0: true, but I don't see a better solution, aside from extracting the native libs from the classpath
20140623 15:38:42 <rmk0> i'm still leaning towards having this be explicitly part of the API, so the user would call JOGL.loadFromClasspath() or JOGL.loadFromJarCache() or whatever... but i suspect i'm going to get hit with furniture for wanting something like that
20140623 15:38:59 <zubzub> rmk0: sounds great to me
20140623 15:39:17 <rmk0> it seems like it's not possible for JOGL to always get the right answer automatically, but if it had some help from humans at runtime...
20140623 15:39:19 <zubzub> or a property passed on
20140623 15:39:35 <zubzub> -DloadFromClasspath
20140623 15:39:41 <rmk0> imagine a property would be less intrusive at the moment... no new API
20140623 15:39:55 <zubzub> and you keep the current as default when no property is specified
20140623 15:41:36 <zubzub> anyway
20140623 15:41:38 * zubzub goes home
20140623 15:42:40 <rmk0> would still like to see the aliasing removed or made optional somehow, in order to "fix" the current maven packages
20140623 15:58:29 * monsieur_max1 (~maxime@anon) Quit (Quit: Leaving.)
20140623 15:58:51 <eclesia> changing subject, any news on making newt frames not visible in the desktop bar ?
20140623 16:03:36 * eclesia (~husky@anon) has left #jogamp
20140623 16:19:30 * zzuegg (zzuegg@anon) has joined #jogamp
20140623 16:32:26 * monsieur_max (~maxime@anon) has joined #jogamp
20140623 17:23:20 <rmk0> sgothel: writing up all of the above into a bug
20140623 17:23:54 <rmk0> i wouldn't mind trying to implement this one
20140623 17:24:05 <rmk0> haven't contributed significantly to the code outside of the maven packages... is about time!
20140623 17:24:24 <rmk0> would like to understand the code better too
20140623 17:54:22 <rmk0> sgothel: https://jogamp.org/bugzilla/show_bug.cgi?id=1024 <- summary of it all, and a proposed solution
20140623 17:57:05 <rmk0> zubzub: ^^
20140623 18:06:43 <zubzub> \o/
20140623 18:19:14 <zubzub> rmk0: bug report looks very good!
20140623 18:20:07 <rmk0> let's hope there doesn't turn out to be some horrible reason it's not doable
20140623 18:22:53 <sgothel> back ..
20140623 18:23:04 <sgothel> thank you Mark - ofc .. you can go ahead!
20140623 18:23:45 <sgothel> .. reading
20140623 18:28:32 <rmk0> hrm, "artfiacts"
20140623 18:28:37 * rmk0 adjusts f and i key
20140623 18:32:06 <sgothel> https://jogamp.org/bugzilla/show_bug.cgi?id=1024#c3
20140623 18:32:19 <sgothel> 3. "same_directory_without_aliasing" <- UP-VOTE (strong) :)
20140623 18:32:41 <sgothel> hope I added the right arguments ..
20140623 18:33:15 <sgothel> when I think about it now .. duh! Why haven't we done this earlier :)
20140623 18:33:46 <rmk0> i should mention that i've obviously been adding all of the native jars to the classpath without any obvious performance issues since the start
20140623 18:33:53 <rmk0> didn't have a choice due to maven, etc
20140623 18:34:01 <sgothel> well ..
20140623 18:34:09 <sgothel> but trust me - there are alot!
20140623 18:34:09 <rmk0> from what you've said, this wouldn't be any good for applets
20140623 18:34:19 <rmk0> but... people seem to be coping with those anyway
20140623 18:34:31 <rmk0> as in, they could stick to the old behaviour
20140623 18:34:46 <sgothel> big apps .. w/ many elements in jar files .. all need to be unpacked etc .. yes, maybe networking .. and so on and so forth
20140623 18:35:10 <sgothel> hence we are discussing big-fat-jar for a big app .. w/ like 100 modules (each one jar) :)
20140623 18:35:16 <sgothel> it takes some time :)
20140623 18:35:50 <sgothel> sure maybe not so visible on the super duper machine w/ ssd and 16 cores :)
20140623 18:36:11 <sgothel> but if there are blocking cross-dependencies .. bingo
20140623 18:36:42 <sgothel> for (3): just remove the aliasing .. and change jogl's ant build .. done
20140623 18:37:00 <rmk0> think this may be a non-issue, not because i'm claiming the performance issues don't exist, but rather that they only happen during development
20140623 18:37:36 <sgothel> na .. I mean other apps .. having 100+ jars in classpath
20140623 18:38:23 <rmk0> what i mean is, people will probably be actually *deploying* their programs without all this stuff on the classpath, because they can arrange for the jogl jars to appear in the same directory on deployment
20140623 18:38:45 <rmk0> but during development, from within an IDE, they might use behaviour 2 instead
20140623 18:39:13 <rmk0> in that case, they can probably tolerate the extra cost
20140623 18:39:20 <sgothel> then still need to add complexity to identify the jar file at runtime, plus the issue that deployment-runtime follows a diff code-path than developer-runtime
20140623 18:39:48 <sgothel> the latter is an issue IMHO
20140623 18:40:26 <rmk0> i don't disagree, but gradle users have no choice in the matter
20140623 18:40:46 <rmk0> gradle says "stuff is going here whether you like it or not"
20140623 18:40:51 <sgothel> yeah .. but wouldn't it work w/ solution (3) ?
20140623 18:41:00 <rmk0> nope
20140623 18:41:04 <sgothel> (yes .. incl. the classpath if they must)
20140623 18:41:12 <sgothel> hmm
20140623 18:41:17 <rmk0> recent versions of gradle put every jar in a different directory
20140623 18:41:25 <rmk0> like... each of the native jars goes in its own directory
20140623 18:41:25 <sgothel> ah .. sorry
20140623 18:41:39 <rmk0> realize that could be interpreted multiple ways in the bug report...
20140623 18:41:43 <sgothel> in this case .. maybe it's (3) + gradle-magic
20140623 18:42:21 <sgothel> (3) to fix it in general .. plus magic/complexity for gradle
20140623 18:44:41 <rmk0> not sure i understand why 2 is vastly more expensive than the original behaviour. seems like it'd be a matter of "genenerate native jar name ('jogl-all-natives-linux-amd64.jar'), split classpath string, iterate over each element; if the basename of an element is equal to the native jar name, use that full path to unpack the native jar as normal"
20140623 18:46:26 <rmk0> it seems equivalent in terms of i/o to the traditional behaviour, it's just that the classpath string is inspected in order to grab the path
20140623 18:47:42 * zzuegg (zzuegg@anon) Quit (Ping timeout: 245 seconds)
20140623 18:48:49 <rmk0> i mean it's not like we're constructing random filenames and trying to open them (resulting in network activity for applets)... we still have the exact names, we just also have the exact paths in the classpath, rather than assuming a directory name based on the current jar
20140623 18:49:23 <rmk0> you understand this madness better than i do... am i missing something?
20140623 18:52:21 * zzuegg (zzuegg@anon) has joined #jogamp
20140623 18:54:03 <sgothel> we derive the URI for the native jar files from the 'calling' class's jar file
20140623 18:54:17 <sgothel> i.e. the path and the 'prefix'
20140623 18:54:31 <sgothel> then we append the suffix '-natives-<os.and.arch>'
20140623 18:54:46 <sgothel> so for graddle to work .. we would need a hint to manipulate the path
20140623 18:54:57 <sgothel> something like: ../../native/
20140623 18:55:22 <sgothel> _or_ a hint to find the path, like another class to look-up from the class path, which jar file is in said path :)
20140623 18:55:29 * [Mike] (~Mike]@anon) has joined #jogamp
20140623 18:55:38 <sgothel> you can follow ?
20140623 18:55:47 <rmk0> urhur
20140623 18:55:59 <sgothel> Wade once added a custom URL locator .. hmm
20140623 18:56:38 <sgothel> JarUtil.Resolver <- he added that one
20140623 18:57:03 <sgothel> maybe we can enhance the 'plugin' allowing a graddle impl.
20140623 18:57:50 <sgothel> the hint-class could reside in the native jar file in this case .. hmm
20140623 18:58:06 <sgothel> something like 'os.and.arch' class maybe ? :)
20140623 18:58:16 <sgothel> then we could lookup the native jar file by the classpath
20140623 18:58:36 <sgothel> so we need to produce this class at compile time on the target
20140623 18:58:52 <sgothel> I guess that would work
20140623 18:59:16 <sgothel> we could use this as a fallback after the current mechanism (w/o aliasing) fails .. to support graddle
20140623 18:59:30 <sgothel> where the current mechanism might be disabled by a property, if you like
20140623 18:59:32 <rmk0> this ... seems a lot more complicated than looking at the classpath
20140623 18:59:55 <sgothel> but it would work in a generic manner ..
20140623 19:00:02 <sgothel> and gives you a proper URI
20140623 19:00:16 <sgothel> we then need to open the native jar file ..
20140623 19:00:32 <rmk0> doesn't the classpath give you a proper URI?
20140623 19:00:49 <sgothel> I have my strong doubt, since it is platform dependent :)
20140623 19:01:32 <sgothel> injecting a os.and.arch class in the native jars is not so bad
20140623 19:02:15 <sgothel> from that point on, we have the URI and can cont. w/ our existing code ..
20140623 19:02:15 <rmk0> i don't think it's going to be possible to predict gradle paths sanely... the directories are named based on the hashes of "something"
20140623 19:02:29 <sgothel> yeah .. but at runtime, we know
20140623 19:02:42 <sgothel> i.e. if we have the class, we know it's jar file
20140623 19:02:49 <sgothel> and hence have it's URI, can open .. etc
20140623 19:03:18 <rmk0> er... how do we know the path of the natives at runtime?
20140623 19:03:36 <rmk0> that's what i was referring to... they'll be downloaded and inserted into oddly named directories
20140623 19:03:41 <sgothel> injecting class 'module-os.and.arch' into the native jar file
20140623 19:04:15 <sgothel> so class-lookup by ClassLoader of module-os.and.arch, gives us the class at runtime
20140623 19:04:26 <sgothel> and we can have it's jar and jar-location
20140623 19:04:31 <rmk0> ...
20140623 19:05:06 <sgothel> like, class 'gluegen_linux-amd64'
20140623 19:05:10 <sgothel> etc etc
20140623 19:05:30 <sgothel> or to be correct gluegen-rt_linux-amd64
20140623 19:05:50 <sgothel> one class for each module, like jogl-all jogl-all-noawt .. etc
20140623 19:06:33 <sgothel> we have all the data at build time and already use it
20140623 19:06:43 <sgothel> Platform produces os.and.arch
20140623 19:06:58 <sgothel> complying w/ our build system
20140623 19:07:01 <rmk0> i feel like i've missed something important; you're talking about placing classes in the native jars so that they can be looked up with ClassLoader
20140623 19:07:09 <sgothel> yes
20140623 19:07:17 <rmk0> but they aren't on the classpath, so how does the ClassLoader find them?
20140623 19:07:31 <sgothel> oh .. sorry, I thought they are w/ graddle
20140623 19:07:38 <rmk0> oh, right...
20140623 19:07:39 <sgothel> ^^ this is all about graddle
20140623 19:07:53 <sgothel> so the class name must identify the module and os.and.arch
20140623 19:08:10 <sgothel> it will be slower than current code-path .. but will work in all cases
20140623 19:08:16 <rmk0> think i understand now
20140623 19:08:16 <sgothel> so IMHO a fallback
20140623 19:08:20 <sgothel> great
20140623 19:08:28 <sgothel> sorry for being too cryptic
20140623 19:08:37 <rmk0> no, is fine, this stuff isn't exactly straightforward
20140623 19:08:40 <rmk0> if the native jars are on the classpath, the ClassLoader will return the URI
20140623 19:08:50 <rmk0> if they aren't, fall back to the traditional method
20140623 19:09:07 <sgothel> I prefer the other way around, since CL lookup is expensive
20140623 19:09:15 <sgothel> and graddle is not our #1 platform :)
20140623 19:09:44 <rmk0> what... try to open the jar files, and if that fails, do a class lookup?
20140623 19:09:44 <sgothel> hence: (3) + (inject ID-class in native jar)
20140623 19:09:56 <sgothel> I know ..
20140623 19:10:07 <sgothel> but (3) suits our major customers
20140623 19:10:14 <sgothel> and (4) .. well, graddle
20140623 19:10:43 <rmk0> oh, wasn't disputing, just wanted to be sure i knew what you meant
20140623 19:10:54 <rmk0> open the jar files, try a class lookup if opening fails
20140623 19:11:19 <sgothel> if tradition+3 fails .. go-class-lookup
20140623 19:11:24 <rmk0> i'd... have expected i/o to be more expensive than a class lookup, but i suppose that'll have to be measured
20140623 19:11:29 <sgothel> may disable traditional via property
20140623 19:11:40 <sgothel> class-lookup is I/O as well
20140623 19:11:46 <rmk0> right
20140623 19:11:51 <sgothel> i.e. visit all classloader .. all resources .. etc
20140623 19:11:53 <rmk0> potentially a lot more I/O, i suppose
20140623 19:11:57 <sgothel> yes
20140623 19:12:10 <sgothel> deal - great :)
20140623 19:12:13 <rmk0> sounds ok!
20140623 19:12:18 <sgothel> dinner .. back in 30min
20140623 19:12:32 <rmk0> you know, one of these days, the java people will come up with a packaging standard that includes native libraries and actually works
20140623 19:12:37 <rmk0> we can throw all of this stuff away
20140623 19:12:47 <rmk0> i'm ABSOLUTELY UNQUESTIONABLY CERTAIN they will manage it /o\
20140623 19:13:13 * rmk0 waves
20140623 19:30:34 <sgothel> hehe
20140623 19:31:00 <sgothel> I was avoiding the native-jar-loading stuff for a while .. until I realized .. w/o it .. it's all crap
20140623 19:31:13 <sgothel> i.e. even simple applets won't work .. and similar use cases
20140623 19:31:28 <sgothel> quick dinner .. but hmmmm delicious
20140623 19:32:24 <sgothel> oh .. another problem w/ using 'classpath' is, it's not available at runtime AFAIK
20140623 19:32:25 <sgothel> :)
20140623 19:32:35 <sgothel> i.e. finding something in it at runtime
20140623 19:32:46 <sgothel> (not using a classloader look-up)
20140623 19:35:51 <rmk0> not available at runtime?
20140623 19:36:08 <sgothel> it is ?
20140623 19:38:19 * rmk0 twists brain
20140623 19:38:36 <rmk0> it is in maven land... don't know about elsewhere
20140623 19:38:45 <sgothel> sorry .. maybe, but never used it at runtime
20140623 19:38:55 <rmk0> but ... if it wasn't available at runtime, how would class loading work?
20140623 19:39:05 <rmk0> the default classloader, anyway
20140623 19:39:08 <sgothel> 'they' know ofc ..
20140623 19:39:31 <rmk0> irrelevant anyway, this class loading approach seems better
20140623 20:26:11 <zubzub> you'd need a package for the native jars as well if you ever want them to work in osgi
20140623 20:26:37 <zubzub> ie the if the current native jars don't have a package declared, osgi won't know what to do with them
20140623 20:27:01 <zubzub> so adding a locator class is an extra + if you want to use it in osgi
20140623 20:27:06 <sgothel> the classloader lookup w/ injected 'signature/ID class' would cover that
20140623 20:27:22 <zubzub> the thing is though
20140623 20:27:37 <zubzub> osgi will not let you find the class simply through it's name
20140623 20:28:10 <zubzub> as each jar (bundle) will live in it's own classloader
20140623 20:28:17 <rmk0> ceylon won't either, as far as i know
20140623 20:28:24 <rmk0> not outside of the current package
20140623 20:28:32 <zubzub> rmk0: correct
20140623 20:28:45 <zubzub> unless you register the class somewhere central
20140623 20:28:54 <zubzub> and query this something
20140623 20:28:56 <sgothel> so here .. it needs to package java-jar + native-jar to work
20140623 20:29:42 <zubzub> alternatively, you could declare a native lib section in the native's jar manifest
20140623 20:29:47 <zubzub> and let osgi do the native lib loading
20140623 20:29:49 <rmk0> the classloader approach would mean that we'd no longer have to worry about in which package the maven natives were provided
20140623 20:30:09 <rmk0> because we'd not have to be trying to arrange for maven to pick the right names for the jar files
20140623 20:30:12 <rmk0> ♥
20140623 20:30:42 <zubzub> rmk0: idd that solution would solve the maven noawt and gradle problems
20140623 20:31:10 <zubzub> but locating the 'native locator class' will not work if you search for it by name in osgi (and ceylon)
20140623 20:31:34 <rmk0> well even if we didn't do this, we'd be including natives along with the noawt packages from now on, and removing the aliasing
20140623 20:31:34 <zubzub> unless you let that class register itself and a central 'locator' service
20140623 20:31:52 <zubzub> *with a central
20140623 20:32:37 <zubzub> (just making you aware of this)
20140623 20:32:44 <rmk0> urhur
20140623 20:33:20 <rmk0> suspect the ceylon solution will bypass all of this stuff if i ever get around to it
20140623 20:33:52 <rmk0> what they have is so far removed that it really needs all of the jars to be repacked into one and use their built-in native loading stuff
20140623 20:34:39 <zubzub> sounds a lot like osgi
20140623 20:34:50 <rmk0> urhur
20140623 20:35:06 <rmk0> think we've had this conversation... it's jboss modules, which i think is basically a rip-off of osgi
20140623 20:35:17 <rmk0> they're extremely similar
20140623 20:35:27 <zubzub> yeah
20140623 20:35:39 <rmk0> they don't even use jars, they have their own thing
20140623 20:35:59 <rmk0> some sort of ... "ceylon module" archive
20140623 20:35:59 <zubzub> a zipfile with a manifest file embedded in it? :p
20140623 20:36:06 <rmk0> hehe, pretty much
20140623 20:44:55 <zubzub> rmk0: how do you plan on locating the native jar helper class?
20140623 20:45:53 <rmk0> is actually sgothel's idea... says the classloader will be able to find it by name
20140623 20:47:18 <zubzub> but doesn't that trigger the same performance problem of finding the actualy native library through Class.geResourceAsStream()?
20140623 20:47:35 <zubzub> (or ClassLoader.getResource...)
20140623 20:47:57 <sgothel> sure .. and it is mentioned
20140623 20:48:24 <sgothel> sure they all use same impl. underneath .. CL.getResource .. CL.find(..)
20140623 20:49:21 <sgothel> hence we don't use this path by default ^^^ (3, skip aliasing .. otherwise traditional native jar loading)
20140623 20:49:33 <sgothel> then .. try the CL way
20140623 20:50:24 <zubzub> and locate & unpack the jar so you could reuse the current native lib loading?
20140623 20:50:52 <zubzub> hmmm why not let the locator class register itself somewhere instead of doing a class lookup?
20140623 20:51:14 <zubzub> that way it should work in osgi, and you don't have to scan the classpath
20140623 20:51:14 <sgothel> _and_ to allow multiple CL's (the copy gives us a unique native-lib-location per CL, a requirement)
20140623 20:52:21 <sgothel> a loaded class is always cached .. hence 2nd lookup should be fast AFAIK
20140623 20:52:44 <rmk0> i'm curious about this locator class registering itself
20140623 20:53:38 <rmk0> .. i don't have anythin to add to that statement. i'm just curious!
20140623 20:53:40 <rmk0> it ... sounds good
20140623 20:53:45 <rmk0> g
20140623 20:53:48 <sgothel> :)
20140623 20:54:02 <sgothel> using existing mechanisms .. and minimizing code change in GlueGen
20140623 20:55:26 <zubzub> the only thing is that I don't know how you would trigger teh locator class to be loaded
20140623 20:55:33 <zubzub> as initially nothing will need it
20140623 20:55:38 <zubzub> hance it will not be loaded afaik
20140623 20:55:47 <zubzub> anway
20140623 20:55:52 <rmk0> that's what i was thinking... i was hoping you knew something i didn't
20140623 20:55:55 <zubzub> I'm gong to sleep
20140623 20:55:57 <sgothel> in 'java' .. findClass ..
20140623 20:56:05 <sgothel> good night
20140623 20:56:10 <rmk0> byee!
20140623 20:56:13 <zubzub> maybe there is some kind of jar loader mechanism that you can hook into
20140623 20:56:16 * void256 (~void@anon) has joined #jogamp
20140623 20:56:31 <zubzub> osgi has bundle (jar) activators that get called when a jar is loaded
20140623 20:56:34 <zubzub> anway
20140623 20:56:35 <zubzub> nn!
20140623 20:56:46 <sgothel> @Mark: if you read TempJarCache, JarUtil .. it's all there already (class lookup, jar URI lookup ..)
20140623 20:57:05 <sgothel> ReflectionUtil <- for class lookup
20140623 20:57:13 <rmk0> sgothel: was referring to a class registering itself if nothing has prompted it to load
20140623 20:57:24 <sgothel> findClass .. does it
20140623 20:57:45 <sgothel> i.e. traversing through all CLs .. and all resources in each CL's classpath
20140623 20:57:48 <rmk0> i'd class that as something else prompting it to load
20140623 20:57:53 <sgothel> ^^ the expensive stuff :)
20140623 21:20:27 * monsieur_max (~maxime@anon) Quit (Quit: Leaving.)
20140623 22:03:33 * void256 (~void@anon) Quit (Remote host closed the connection)
20140624 03:51:48 * [Mike] (~Mike]@anon) Quit ()
20140624 05:05:27 -jogamp- Continue @ http://jogamp.org/log/irc/jogamp_20140624050527.html