Bug 607 - JavaFX 2+ and JOGL should work together
Summary: JavaFX 2+ and JOGL should work together
Status: IN_PROGRESS
Alias: None
Product: Jogl
Classification: JogAmp
Component: javafx (show other bugs)
Version: tbd
Hardware: All all
: P5 enhancement
Assignee: Sven Gothel
URL:
Depends on: 1364
Blocks:
  Show dependency treegraph
 
Reported: 2012-07-20 00:06 CEST by Arnaud Couturier
Modified: 2019-11-19 21:58 CET (History)
7 users (show)

See Also:
Type: FEATURE
SCM Refs:
Workaround: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Arnaud Couturier 2012-07-20 00:06:39 CEST
There is currently no way to integrate JOGL in a pure JavaFX 2+ GUI.

Swing is deprecated, and JavaFX is the new way to make GUIs in Java.
So this possibility would be useful:
- to take advantage of JavaFX
- and necessary on the long term
Comment 1 Julien Gouesse 2012-07-29 12:38:00 CEST
Hi

There are several technical aspects driving such an interoperability with JavaFX 2 almost impossible or very difficult to implement. I advise you to look at this document:
http://docs.oracle.com/javafx/2/architecture/jfxpub-architecture.htm

At first, I have found no way of forcing JavaFX to use its Java2D backend or in the best case its OpenGL backend. It means that there is a risk of conflict at driver level with Direct3D. I posted a question on the mailing list of JavaFX developers and they have never replied.

Then, I have to think about another way of supporting this technology. As you can see on the document, JavaFX uses its own equivalent of NEWT (Glass windowing toolkit) and its own scenegraph (Prism).

In my humble opinion, the support of JavaFX with hardware acceleration goes beyond the scope of any Java binding for the OpenGL API, it is rather a potential feature for an existing scenegraph supporting JOGL 2.0. Such a scenegraph would have to replace Prism and to allow you to use plain JOGL code in render delegates. Moreover, it would have to rely on NEWT and to modify Quantum so that it uses NEWT instead of Glass WT. Glass WT and NEWT may have some similarities.

To conclude, we can look at the source code of JavaFX (OpenJFX) to investigate deeper but this feature can only be implemented in Ardor3D with "my" renderer based on JOGL 2.0, it is none of JogAmp's concern.
Comment 2 Arnaud Couturier 2012-08-03 03:23:24 CEST
(In reply to comment #1)
Understood.
Nice article.

I see it's in the JavaFX roadmap (http://www.oracle.com/technetwork/java/javafx/overview/roadmap-1446331.html) to add "3D Graphics Support", but I doubt they mean low level access to the hardware. On the bright side, their development might unlock  access to their rendering pipeline, so that we can hook ourselves somehow.

Anyway, this isn't going to happen soon, so I'll just use Swing with applications that need OpenGL, and JavaFX with the others. I'll see whether JavaFX in Swing is practical or not.

Thanks for taking the time to investigate the issue :)
Comment 3 Julien Gouesse 2012-08-26 13:11:13 CEST
It seems possible to override com.sun.javafx.sg.prism.NGNode.render() in order to create an external drawable with GLDrawableFactory.createExternalGLDrawable() so that we can do whatever we want with the OpenGL context and render with JOGL 2.0. However, there is still no way to force JavaFX to use OpenGL under Windows. The suggested solution only works under Mac OS X and probably under GNU Linux (but it requires at least GLX 1.3).

I have no access to the source code of Quantum (com.sun.javafx.tk.quantum) and Glass. We have to force com.sun.javafx.sg.BaseNode or another class to create an overridden node and use it as a delegate.

Keep in mind that such an hacky solution could be broken by changes in JavaFX/OpenJFX. It would be safer to be able to override a node of the public API.

Someone succeeded in mixing JavaFX 2 with OpenGL under Mac here:
http://www.java-gaming.org/topics/jogl-in-javafx/26881/msg/242906/view.html#msg242906
Comment 4 Julien Gouesse 2012-08-26 13:47:42 CEST
NGNode is abstract like BaseNode and I don't find its child classes.
Comment 5 Julien Gouesse 2013-02-22 13:31:47 CET
Hi

I would like to know Sven's opinion about my reply here:
http://www.java-gaming.org/topics/javafx-3d-api-and-opengl/28797/msg/262932/view.html#msg262932
Comment 6 Julien Gouesse 2013-03-14 10:50:06 CET
It is possible to force the use of OpenGL with "-Dprism.order=es2,es1" or force the use of Java2D with "-Dprism.order=j2d" but then Direct3D must be disabled in Java2D too.
Comment 7 Martin 2013-03-21 12:43:41 CET
Hi,
Is there any downloadable source example of JOGL2 integration to JavaFX?
Thanks in advance,
Martin
Comment 8 Julien Gouesse 2013-03-21 13:38:41 CET
(In reply to comment #7)
> Hi,
> Is there any downloadable source example of JOGL2 integration to JavaFX?
> Thanks in advance,
> Martin

There is no such example working with JavaFX 2.0. "Deejay" published some source code on BitBucket. In theory, rendering a JavaFX widget in a FBO is possible. You can use JavaFX in a Swing application thanks to JFXPanel. However, there is no more fully JOGL-based backend unlike what was available for JavaFX 1 several years ago.

I don't want to spend any time on a hacky solution that could be broken by a change in an obscure private API and Oracle wants to keep JavaFX self-contained.

Keep in mind that mixing JavaFX with JOGL might drive your application unstable. Some developers had to use dirty kludges to make it possible. For example, JavaFX seems to cause troubles in AWT on some platforms.

I will invest some time in implementing this interoperability if and only if the JogAmp Foundation considers JavaFX to be really viable on the long term and if it is modularized enough to allow me not to rewrite everything.

There is one possibility that I have not yet tested: it is possible to force the use of the Java 2D backend in Prism and GLG2D allows to use JOGL 2.0 in Java 2D. It would probably be less slow than rendering JavaFX widgets in a FBO but less fast than a fully hardware accelerated pipeline. Feel free to contribute ;)
Comment 9 Martin 2013-03-27 15:04:25 CET
Thanks for the suggestions.
I could find Dejay's works here: https://bitbucket.org/dejayberlin/joglfxpipeline/

If I come with something helpfull, no worry I'll contribute :)
Comment 10 Julien Gouesse 2013-03-27 15:40:08 CET
A member of Java-Gaming.org used our competitor to render JavaFX widgets in a FBO, maybe you could port his code to JOGL 2.0. It would probably work reliably and we could use GLOffscreenAutoDrawable to get something more viable on low end graphics cards.
Comment 11 Julien Gouesse 2013-07-30 14:39:57 CEST
I don't know whether the selected toolkit may influence the threading, it can be set with "-Djavafx.toolkit=", the default value is "glass", the others are:
com.sun.javafx.tk.DummyToolkit
com.sun.javafx.tk.prism.TVToolkit
com.sun.javafx.tk.prism.PrismToolkit
com.sun.javafx.tk.swing.SwingToolkit
com.sun.javafx.pgstub.StubToolkit
Comment 12 Julien Gouesse 2013-07-30 14:55:44 CEST
The ES2 pipeline is not shipped by default with the JRE under Windows:
https://forums.oracle.com/message/11016899#11016899

JavaFX users will have an hard time if they are to use the software renderer with JOGL 2.0 especially on low end machines.
Comment 13 Julien Gouesse 2013-12-19 14:04:46 CET
Oracle will ease the interoperability with natives APIs, the implementation of a GLNode is planned:
https://oracleus.activeevents.com/2013/connect/fileDownload/session/C3B800733D28DB105AB5FC964843D1AB/CON1905_Heidrich.pptx

Oracle seems to be aware of some blocking aspects.
Comment 14 Julien Gouesse 2013-12-19 16:31:29 CET
An experimental API of JavaFX 8 allows to use a single thread for JavaFX and Swing, you can use -Djavafx.embed.singleThread=true to do that:
http://stackoverflow.com/a/18675377

You can already use JavaFX in a Swing application thanks to JFXPanel but now you can use Swing in a JavaFX application thanks to SwingNode:
http://download.java.net/jdk8/jfxdocs/javafx/embed/swing/SwingNode.html

It allows to use existing JOGL Swing based UIs with JavaFX until we provide a better solution.
Comment 15 Julien Gouesse 2014-01-14 17:27:01 CET
Christoph worked on JavaFX with JOGL too:
https://bitbucket.org/redrezo/prism-jogl-playground/src

According to him, I have to put the JOGL pipeline for OpenJFX/JavaFX into com.sun.prism.jogl.JOGLPipeline and load it from the same classloader than jfxrt.jar to make it work, I have to set prism.order to "jogl" to respect the naming convention.

http://j661.sourceforge.net already uses JavaFX, Swing and JOGL. I have to look at EmbeddedWindowsManager.

As I said in the comment 13, Oracle employees are aware of the difficulties to use JavaFX with native APIs:
http://www.parleys.com/play/524ee4dbe4b0ab14e307d7b1

I'll have to decide how to implement this enhancement:
- just render the 3D scene into an offscreen FBO and put the result into a JavaFX node. Render a JavaFX node into an image and use it as an OpenGL texture as it is done in this project: http://tinyurl.com/n9naya8
- use Prism/Glass/Quantum to implement another thin pipeline
- get rid of Prism/Glass/Quantum, use NEWT, provide a completely alternative implementation of OpenJFX/JavaFX
Comment 16 Julien Gouesse 2014-08-17 02:37:15 CEST
aqd ported a known example using OpenJFX/JavaFX to JOGL:
https://github.com/AqD/JOGL-FX
Comment 17 Julien Gouesse 2014-08-20 12:23:11 CEST
The ES2 pipeline is shipped with Oracle Java for Windows since at least Java 1.8 update 11 which improves the performance when using JOGL.
Comment 18 Julien Gouesse 2014-08-20 12:23:59 CEST
(In reply to comment #17)
> The ES2 pipeline is shipped with Oracle Java for Windows since at least Java
> 1.8 update 11 which improves the performance when using JOGL.

Of course, use -Dprism.order=es2 to make it work.
Comment 19 AqD 2014-08-23 17:53:42 CEST
On windows ES2 JNI is shipped (prism-es2.dll) but the required Java classes are NOT.

You have to use jfxrt.jar from Linux or other version, and then compile the following classes (source inside javafx-src.zip):

com.sun.prism.es2.WinGLContext
com.sun.prism.es2.WinGLDrawable
com.sun.prism.es2.WinGLFactory
com.sun.prism.es2.WinGLPixelFormat


Put the four classes into jfxrt.jar and then the ES2 pipeline can be used. During the operations it's better to enable verbose messages by -Dprism.verbose=true.



It works fine but on AMD chips there might be graphic artifacts.
Comment 20 Sven Gothel 2015-03-05 02:32:08 CET
Terminology:
============

[JavaFX]
If we use the product name 'JavaFX' here,
we mean the combination of its:
  - Windowing Toolkit (Non public Glass + ...)   
  - UI Toolkit (Non public Prism + ...)

Their separation is not guaranteed,
interoperability w/ other toolkits is an issue
and parts of the implementation is not public.


[NEWT, GraphUI]
On the other hand (JogAmp), we have well separated
building blocks:
  - NEWT: Public Windows Toolkit
  - GraphUI: Upcoming Public UI Toolkit


[Fragile]
The latter might be an issue in case the strategy
uses non public APIs and is therefor not supported by the JavaFX vendor. 
We mark those as instable [Fragile].


[SlowPath]
Performance might be an issue, in case we are not able
to render the content directly into a surface layer
which will be composited by JavaFX, but need to copy
the rendered content from OpenGL and paste it into an JavaFX node.
We mark those as slow path [SlowPath].


[Portability]
JavaFX integration strategy might not be available on
all desired target platforms at all
or only w/ reduced performance [SlowPath].


[Static-UI-Separation]
Further, we may not be able to integrate the OpenGL rendered artifacts seamlessly 
into the overall 'user screen/window', as it is possible w/ [GraphUI].
Here we are restricted to old school spatial separation of
3D content and UI elements, e.g. having an opaque 3D scene rectangular area 
and the UI elements 'on the side'.
This almost represents C3D's current UI experience using Swing.

Such opaque spatial separation design has the following draw-backs,
and hence are also pro-arguments for [GraphUI].

  [1] Pre-occupying space w/ UI elements -> losing display area.

      But a seamless 3D-UI [GraphUI] will simply be placed       
      into the 3D scene _on_demand_ !

  [2] Toggling UI visibility to benefit from bigger display area,
      e.g. fullscreen, will resize the 3D scene and cause irritation.

      But [GraphUI] can simply be enabled/disabled while       
      the main 3D scene remains unchanged.

  [3] UI elements cannot be associated w/ 3D scene by location,
      since they are separated spatially!

      But [GraphUI]'s seamless integration into the scene allows
      association by location, rotation and size.

  [4] UI elements cannot contain 3D elements from the scene

      But [GraphUI] is able to share content from the main 3D scene.

  [5] UI elements will not benefit from high performance GPU rendering.

      But [GraphUI] is rendered using hw accelerated OpenGL the same way as
      the main 3D scene.


Strategies:
=============

All below strategies will suffer from above mentioned [Static-UI-Separation]!

Only [GraphUI] would allow a seamless 3D scene integration.

[A] Including [JavaFX] UI elements as a Swing node
    within border layout's border components
    while maintaining a heavyweight center 3D main scene.

    JFXPanel: 
      <http://docs.oracle.com/javase/8/javafx/api/javafx/embed/swing/JFXPanel.html>

    Current C3D already utilizes a similar approach w/o JavaFX,
    i.e. mixing Swing lightweight and one AWT heavyweight component
    containing the NEWT child window.

    - [Portability]

    CLASSIFICATION: It has to be evaluated whether JavaFX's     
    implementation allows this strategy, i.e. mixing a heavyweight     
    AWT Canvas w/ NewtCanvasAWT and other Swing/JavaFX lightweight components.
    At least this strategy seems worthwhile to investigate,     
    however difficulties based on platform and implementation differences
    shall be expected.

[B] Rendering OpenGL offscreen while copy its content
    and paste it into a [JavaFX] node, similar to GLJPanel for Swing.  

    Canvas:
      <http://docs.oracle.com/javase/8/javafx/api/javafx/scene/canvas/Canvas.html>

    - or - 

    WritableImage:
      <http://docs.oracle.com/javase/8/javafx/api/javafx/scene/image/WritableImage.html>

    - [SlowPath]          

    This strategy has a huge performance impact,
    i.e. the OpenGL pixel-read and [JavaFX] node write operation
    for each frame will be very slow!

    CLASSIFICATION: Since C3D is a demanding high-performance application
    rendering the main 3D scene fluently at 10-60 fps,
    this methodology is not recommended to achieve great results.

[C] Using [JavaFX] non public API to implement     
    opaque heavyweight component like NewtCanvasAWT.

    - [Fragile]
    - [Portability]

    The [JavaFX] non public API is used to retrieve     
    the native window handle, allowing to attach a NEWT child window.

    Hack to retrieve the native window handle:
      <https://stackoverflow.com/questions/15034407/how-can-i-get-the-window-handle-hwnd-for-a-stage-in-javafx>

    - [Performance]

    Here the child window would need to be offset to its JavaFX parent     
    to only cover the desired part at the desired location of its JavaFX parent window.
    Whether this is possible and portable remains to be seen.

    CLASSIFICATION: Since this methodology is not public and may change at any given time
    it is not recommended for stable product development.
    Further constraints to achieve proper child window positioning     
    also reduce this strategies merits by a huge margin.
Comment 21 Sven Gothel 2015-03-05 02:32:58 CET
(In reply to comment #20)

Added analysis to public, due to forum request(s).
Comment 22 Julien Gouesse 2015-06-22 14:36:26 CEST
AqD's instructions might be useful for Windows users:
https://github.com/AqD/JOGL-FX#jogl-on-javafx-es2
Comment 23 Julien Gouesse 2015-10-15 17:27:56 CEST
A contributor has implemented a pipeline for JavaFX/OpenJFX based on JOGL 2:
https://github.com/pepe1914/jfx-zoglpipeline
Comment 24 Sven Gothel 2019-03-28 18:14:32 CET
Bug 1364 fully implements strategy [C] as outlined in comment 20

[C] Using [JavaFX] non public API to implement     
    opaque heavyweight component like NewtCanvasAWT.

Julien's suggestion in comment 23 of replacing OpenJFX's 
Prism and Glass implemention with a JOGL and NEWT based pipeline
has been also taken into consideration for the future, depending on demand.

Notable: Current OpenJFX implementation has codepath for
- GNU/Linux X11
- Windows
- Mac OSX
- Mac iOS
Comment 25 Sven Gothel 2019-03-30 02:02:51 CET
Bug 1364 gives us the first native NEWT access to JavaFX, hence we can push other possible options like glass replacement with NEWT and prism path using JOGL a bit further away. However, we keep this earmarked.
Comment 26 Xerxes Rånby 2019-05-16 11:17:16 CEST
OpenJFX upstream is working on a new public api that enables a new future strategy

[D] Using [JavaFX] new public API to implement     
    opaque heavyweight component like NewtCanvasAWT.

https://bugs.openjdk.java.net/browse/JDK-8091324 - Allow FX to interoperate with 3rd party (native) OpenGL visualizations
https://bugs.openjdk.java.net/browse/JDK-8167148 - Add native rendering support by supporting WritableImages backed by NIO ByteBuffers
Export

This is the first pull-request for openjfx that implement this new API
https://github.com/javafxports/openjdk-jfx/pull/472 - JDK-8167148 : Add native rendering support by supporting WritableImages backed by NIO ByteBuffers #472

By using this API we can address some of the shortcommings of strategy [C] as it improves [Fragile] and [Portability]

The downside with this strategy is
- [No public OpenJFX release with this API]
Comment 27 Sven Gothel 2019-08-20 06:48:23 CEST
(In reply to Xerxes Rånby from comment #26)
> OpenJFX upstream is working on a new public api that enables a new future
> strategy
> 
> [D] Using [JavaFX] new public API to implement     
>     opaque heavyweight component like NewtCanvasAWT.
> 
> https://bugs.openjdk.java.net/browse/JDK-8091324 - Allow FX to interoperate
> with 3rd party (native) OpenGL visualizations
> https://bugs.openjdk.java.net/browse/JDK-8167148 - Add native rendering
> support by supporting WritableImages backed by NIO ByteBuffers
> Export
> 
> This is the first pull-request for openjfx that implement this new API
> https://github.com/javafxports/openjdk-jfx/pull/472 - JDK-8167148 : Add
> native rendering support by supporting WritableImages backed by NIO
> ByteBuffers #472
> 
> By using this API we can address some of the shortcommings of strategy [C]
> as it improves [Fragile] and [Portability]
> 
> The downside with this strategy is
> - [No public OpenJFX release with this API]

It is impressive that nobody contacted me.
Thank you for the heads up though.

However, this path is slow as it would require at least
1) One FBO copy and the FBO usage in the first place
2) Another composition round (merging the image data)

In other news, I have played with the idea to replace 
OpenJFX's glass with newt (back where it was) 
and also use JOGL as the underlying OpenGL renderer.

However, I have already added 'overlay' support, 
see below.
Comment 28 Sven Gothel 2019-08-20 06:48:57 CEST
commit a4ec6556f9ef3a409cceb9bfdb0d19dfc7d98d4a
Author: Sven Gothel <sgothel@jausoft.com>
Date:   Tue Mar 19 11:40:34 2019 +0100

    JavaFX: Adding JavaFX Support for NEWT utilizing native Window parenting via NewtCanvasJFX
    
    NewtCanvasJFX, a JavaFX Canvas Node, allows attaching a native NEWT Window to the JavaFX Node's native Window (if attached).
    
    The mechanism is similar to NewtCanvasAWT.
    
    Current implementation supports placing the NEWT Window
    into the JavaFX scene of the native window correctly,
    as well as the following different lifecycles
    - attach NewtCanvasJFX to already visible group->scene->window
    - attach NewtCanvasJFX to not yet visible or attached group->scene->window
    - attach NEWT Window before or after NewtCanvasJFX's visibility
    
    The above is covered by unit test: TestNewtCanvasJFXGLn
    
    This is the initial commit for JavaFX support and has been tested on
    - OpenJDK 8 + OpenJFX 8
    - GNU/Linux X11
Comment 29 Sven Gothel 2019-08-20 06:55:45 CEST
See Bug 1364 for the NewtCanvasJFX work.

(Sorry folks, too early or too late. Missed that I already posted this there)