Bug 1021 - Add generic support for stereoscopic renderer, demo w/ OculusVR
Summary: Add generic support for stereoscopic renderer, demo w/ OculusVR
Alias: None
Product: Jogl
Classification: JogAmp
Component: util (show other bugs)
Version: 2
Hardware: All all
: --- enhancement
Assignee: Sven Gothel
Depends on: 1022 1025
Blocks: 1116
  Show dependency treegraph
Reported: 2014-06-14 08:41 CEST by Sven Gothel
Modified: 2015-01-08 13:24 CET (History)
1 user (show)

See Also:
Type: ---
SCM Refs:
40d7d0d18ed11b8dbb0f16137f82eef828b822a4 8a4dacf38c9d78bc0aafae91cbd256148835ee9b 589070da2be7e097b7415503011387ad1f6e14d5 b54e87d2cebf6aa0def18f172cbfa111c4efc837 d7d8d11abad1e4e17532068333dfe9dddf2942a2 ae17eca383cce89a000062c3a86e415a6753066b 36327e24cf586b50bf18e87d7d13d53eb41cf1d9 d44e8ada30d62149c5d4d4b8fdba7cc33f8c765b 62bc3219736ea003e69d8a63dc4ce29841d4129e f8f0f051604721bceaee214b8e5218fd47d2eb9e jogl 38e51e4a5f6f35c658df10f6d48a33e3ffaea2f3 jogl 8be4b87a9740229f09b1dc2b7265e3e73c4397dd jogl 609b3f6d28bb13b589cde815acdb0e72e58ebe44 oculusvr-sdk a18c0cec906c4ac8c171d9000fd1c54e9302c9a4
Workaround: ---


Note You need to log in before you can comment on or make changes to this bug.
Description Sven Gothel 2014-06-14 08:41:00 CEST
General 3D/VR:

  - Provide GLAutoDrawable VRRenderer:

    - controlling FBO's per eye

    - performing optional stereoscopic operations (shader):
      - distortion
      - color-correction
      - ..

    - perform specific display-sync operations

    - may utilize other impl. specific operations

    - surface-size and pixel-size maybe different,
      this is similar to already supported HiDPI

    - Hook in VRRenderer driver, e.g. OculusVR

OculusVR VRRenderer driver:

  - GlueGen / JNI binding to native library

  - Implement VRRenderer specifics to OculusVR
Comment 1 Sven Gothel 2014-06-17 08:42:22 CEST
To seamlessly use libovr via GlueGen/Java,
we require compound [array] call-by-value support.
See Bug 1022.
Comment 2 Sven Gothel 2014-06-19 06:20:34 CEST
with the last required GlueGen, i.e. Bug 1011 comment 5,
i.e. supporting compound-arrays in structs and allowing enum-c 
identifiers to define array-length, bring-up on GNU/Linux x86_64 
is successful.

Bring-up test only initializes the ovrHmdHandle
data structure by OVR native code.

Current evaluation build compiles all OVR-SDK source
files itself w/o using provided libovr.a.
We also skip the GL dependent renderer of the SDK,
i.e. we prefer to utilize our JOGL 'barrel distortion' renderer.
This eases the 'chicken-egg' problem of OVR SDK dependencies,
i.e. libGL, libGLESv2 .. etc 

Since the OVR source code is C++, we may still have to figure out
how to / and whether we shall link libstdc++ statically 
to remove platform dependencies.

Next bring-up test shall include a working visual demo.
Comment 3 Sven Gothel 2014-06-19 06:21:17 CEST
(In reply to comment #2)
> with the last required GlueGen, i.e. Bug 1011 comment 5,
 Ooops - Bug 1022 comment 5,
Comment 4 Sven Gothel 2014-06-19 06:54:25 CEST

Branch: oculusvr

  Add OculusSDK 0.3.2 Source Code w/o Samples, docs or binaries (libs or tools)

  As described in comment 2
Comment 5 Sven Gothel 2014-06-20 00:16:15 CEST
40d7d0d18ed11b8dbb0f16137f82eef828b822a4 ...

Re-created oculusvr branch: Please delete previous one!



Moved OculusVR SDK to it's own sub-module:


with merged sources for 
  - Linux
  - Windows
  - OSX


Continue w/ bring-up for Windows ..
Comment 6 Sven Gothel 2014-06-20 08:23:09 CEST
  - Bring-up on Windows w/ Mingw64
  - Solve static linkage of stdc++

  - Bring-up on OSX w/ clang/clang++ 
  - Split native build: cc, c++, linkage

  - Add note about 'Oculus VR Rift SDK Software License' in LICENSE.txt

- Merged branch oculusvr -> master, ..
- Removed branch oculusvr
Comment 7 Sven Gothel 2014-07-01 21:31:19 CEST

Add OculusVR distortion renderer (single FBO and dual FBO)


Adoption (aka usage) is quite simple, see:

It uses the new support classes to render stereo manually w/ all the distortions.

The sensors shall work implicit via the 'eye pose'
and the derived modelview matrix.
However, we have to see how that works w/ 'rotating the head',
no DK2 available here.
Comment 8 Sven Gothel 2014-07-01 21:32:26 CEST
In case more stereo renderer are required,
we shall add subsequent bug reports.

Closing for now.
Comment 9 Sven Gothel 2014-07-03 09:38:56 CEST
Commit d44e8ada30d62149c5d4d4b8fdba7cc33f8c765b:

Refine API in regards to proper package names, interface
and high-level access to eye specific constant parameter
and variable eye movement.


Commit 36327e24cf586b50bf18e87d7d13d53eb41cf1d9 introduced 'GLEventListener2'

Move javax.media.opengl.GLEventListener2
  -> com.jogamp.opengl.util.CustomRendererListener
    -> com.jogamp.opengl.util.stereo.StereoRendererListener

StereoRendererListener adds stereoscopic specific:

  public void reshapeEye(final GLAutoDrawable drawable, final int x, final int y, final int width, final int height,
                         final EyeParameter eyeParam, final EyePose eyePose);

.. see below.


Add com.jogamp.opengl.util.stereo:
  - EyeParameter (Constant eye parameters, like IPD and FOV)
  - EyePose (Current eye position and orientation)


Add com.jogamp.opengl.math.FovHVHalves to support
non-centered bi-directional FOV for lenses.

Add respective FloatUtil.makePerspective(.. FovHVHalves fovhv ) variant.
Comment 10 Sven Gothel 2014-07-03 13:57:29 CEST
Commit 62bc3219736ea003e69d8a63dc4ce29841d4129e

Turns out that using bilinear texture filtering w/o multisampling is 'good enough' ..

- OVRSBSRenderer*:
  - Add texture filtering parameter (bilinear/none)

  - Adjust FBO texture params (alpha, filtering)
    No alpha needed in FBO sink/target!

- Demo: Pass all parameters w/ arguments, use sane good defaults.
Comment 11 Sven Gothel 2014-07-05 04:06:47 CEST
Make OVR access vendor agnostic: 
Package 'com.jogamp.opengl.util.stereo' contains all public interfaces/classes

Renamed interfaces:
  CustomRendererListener -> CustomGLEventListener
  StereoRendererListener -> StereoGLEventListener

New vendor agnostic 'stuff' in com.jogamp.opengl.util.stereo:
  1 - StereoDeviceFactory
    To create a vendor specific StereoDeviceFactory instance,
    which creates the StereoDevice.

  2 - StereoDevice
    For vendor specific implementation.
    Can create StereoDeviceRenderer.

  3 - StereoDeviceRenderer
    For vendor specific implementation.

  4 - StereoClientRenderer
    Vendor agnostic client StereoGLEventListener renderer,
    using a StereoDeviceRenderer.
    Now supports multiple StereoGLEventListener, via add/remove.

- MovieSBSStereo demo-able via StereoDemo01
  can show SBS 3D movies.
Comment 12 Sven Gothel 2014-07-07 15:22:59 CEST
I did another round of abstraction
adding generic impl. of StereoDevice and StereoDeviceRenderer
and isolating the lense distortion math from the OVR SDK[1].

The generic impl. may utilize:
  - monoscopic display (*)
  - stereoscopic display (*)
    - Side-By-Side (SBS) (*)
      - without lenses (*)
      - with lenses (*)
    - Other techniques 
      - OpenGL Quad Stereo Buffer (+)
      - Top-Bottom (+) 
      - Interlaced (+)
      - Color Separation (+)

Goal is to harmonize stereo rendering w/ this package.

Note, the isolating lense distortion math from the OVR SDK[1]
is bound to the SDK's license since I have copied parts of it.
Hence we may only be able to distribute this part 
within the oculusvr-sdk sub-module due to it's license
which sort of forbids deploying only parts IMHO.
Comment 13 Sven Gothel 2014-07-07 23:49:37 CEST
(In reply to comment #12)

> The generic impl. may utilize:
>   - monoscopic display (*)
>   - stereoscopic display (*)
>     - Side-By-Side (SBS) (*)
>       - without lenses (*)
>       - with lenses (*)


commit jogl 38e51e4a5f6f35c658df10f6d48a33e3ffaea2f3

Bug 1021: Add GenericStereoDevice* Supporting custom configurations; Hook-in oculusvr-sdk java distortion-mesh calculation if available

StereoDeviceFactory support new GenericStereoDeviceFactory, with it's GenericStereoDevice and GenericStereoDeviceRenderer.

GenericStereoDevice maintains different configurations, triggered either by passing a GenericStereoDevice.Config
instance directly or by the device-index parameter:

  - 0: monoscopi device: No post-processing

  - 1: stereoscopic device SBS: No post-processing

  - 2: stereoscopic device SBS + Lenses: Distortion post-processing
       (only available w/ oculusvr-sdk sub-module)

Producing a 'GenericStereoDevice.Config' instance is self containing
and may extend if supporting more device types like top-bottom, interlaced etc.

StereoDemo01 handles all use-cases and may be used as a test-bed
to add and experiment with stereoscopy, devices and settings.


commit oculusvr-sdk a18c0cec906c4ac8c171d9000fd1c54e9302c9a4

Add java code of DistortionMeshProducer for lens usage, as included in Oculus VR RIFT SDK 0.3.2 (w/ dependency on JOGL)

This code has been added to the oculusvr-sdk sub-module
due to the unclear license situation.

While including these parts w/ the whole SDK,
we do not violate the RIFT SDK licence - to our knowledge.


Code implements JOGL's
'jogamp.opengl.util.stereo.DistortionMesh.Producer' interface.

The implementation heavily uses RIFT SDK mathematical equations,
comments, algorithms and constants (DK1).

RIFT SDK C++ code has been partially ported to Java.

Purpose it to add a development testbed for JOGL
without requiring to bind to the native SDL library
and to experiment w/ some parameters.

This module may be used for educational purposes with JOGL's

If 'compiled in' with the SDK, reflection it utilized
in one of GenericStereoDevice configuration to load this functionality.

This module is not useful for real VR integration,
since it missies proper parameter detection
as well as sensor support.

For real VR integration, the fully fledged
'jogamp.opengl.oculusvr.OVRStereoDevice' shall be used instead.

The latter is part of JOGL and binds dynamically into the RIFT SDK
native library, which is provided within this sub-module (oculusvr-sdk).
Comment 14 Sven Gothel 2014-08-07 00:15:07 CEST
  Fix OculusVR detection, need to probe HMD count, 
  otherwise native SDK code crashes w/ NPE


  - StereoDevice.DeviceType: Add API doc
  - StereoDevice: Add getFactory()
  - GenericStereoDevice
    - Use common static vars for configurations for simplicity
    - Fix createRenderer(..)'s eyeViewport in case no post-processing
      is performed, i.e. needs viewport X offset.
  - StereoDemo01
    - Use 'movie' eyePosition instead of default if:
      - using a movie player _and_ using lenses!
    - Fix NEWT window pixel-unit size after window creation!