Bug 1417 - Android: Support building using current SDK and NDK v20 as of 2019-12-10
Summary: Android: Support building using current SDK and NDK v20 as of 2019-12-10
Status: RESOLVED FIXED
Alias: None
Product: General
Classification: JogAmp
Component: builds (show other bugs)
Version: 2.4.0
Hardware: All all
: P4 normal
Assignee: Sven Gothel
URL:
Depends on: 1419
Blocks: 1363
  Show dependency treegraph
 
Reported: 2019-12-10 06:23 CET by Sven Gothel
Modified: 2020-02-04 01:00 CET (History)
3 users (show)

See Also:
Type: DEFECT
SCM Refs:
gluegen 74e5bbe097a07d1275a2d6452d1e2336a8d5c44a ant-cpptasks 4b859d7b63e5cf4aa74f740d4ae7f46aa13788c9 ant-cpptasks 6bdee37988009d3006ccf99bac1ccea497141afd ant-cpptasks e1933c74b9c256332cd12c9a1094e4687b535112 gluegen b6a91a035c14f1e4d142517f311007e48ea9c0c1 gluegen 4f7273e017e2deb2928a76066604ced03609f921 gluegen 19761070e12f65fe7a07e6d9b7a5221a1623b12d gluegen eb6dfabd7427a1836f6b1d2080422f98808ea0a4 gluegen 51ef5eadd9db020412d3a3716b4ab5a25b0522fb ant-cpptasks ec14765d6baab247fa37d62845c139adc367af2a ant-cpptasks 79a8fed9fddab5429a3457c3077ee5012b405a51 ant-cpptasks 07aeed47e484bfec85e749ea721756b8a2571e00 ant-cpptasks 8353e0cc85705ca8f81f8563ad975026978b6745 gluegen 53ae28fd2bc501b5de54fdbdf915d88f823eb11a gluegen 26d06324312b8fe6980ff29ab9d443cc5ec2657f gluegen aeae740ba2569d839820bfc20c43a8550ab8ed33 gluegen ae4cdcf3490a1bf493f39ab5c8c6b596d3af71de joal b8e20cd8238c2f3694068f501ba06b9a6f654645 jogl 07de253dae77b39da944a946381bd6efec0066ba jogl c6ba090a0030c177d7e60f797a3ec25fecfe3546 openal-soft f344ef3aa3a42c4e81376c3d8ad6507ad2b8a20c openal-soft 44f5d564bacfd8adcbb1d59763ed8970667824af joal 8907cb8550def10640f9d812853fbadf3f2989f5 openal-soft 0d14d30808cfe7b9e3413353e3eef8a0f201399a joal 647eaa5fc88a9f4dfc0ec421f5ae79474d1117af joal 768677b5b56a9cfea3accf4b7638d217eceade44 jogl 784dcfa2094f3fb235ca4b60395708a1f87c0b1b gluegen d190301a82a961ac3246890206200086842059bd gluegen 6e98be1a1bc819b47e2cc52eb35d23a7489fdf1b gluegen faa37334e2912e02cb58e738570a127cc6402b37 gluegen 1390bc236de531646aaa4525a17de1a533b1fa06 joal c4f73b67ebbb554cde69b6f59ab377a38864df72 jogl d91f8581eedf73663abab25524147d2bcab9109b jogl 95ca88ba975c7296098bbd34bec2d922aeb2b563 jocl 22b41b9a232527d8c2b2a2b03e03005273529df7 jocl 023fff1af43a172d4c2a0e18fdbb78e6ba817cca jocl 4287fdd657b946c9b605361692dff56b2b99ab1a
Workaround: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sven Gothel 2019-12-10 06:23:17 CET
The layout of today's SDK and NDK of the Android tool has changes.

Aligned with Android SDK build-tools 29 and NDK 20 as of 2019-12-10

It sort of is no more easily possible to get a complete SDK tarball/zip archive,
the official supported way is to use Android-Studio and fetch all parts from there.

Doing so, I the copied the ~/Android/Sdk -> /opt-linux-x86_64/android-sdk-linux_x86_64, which I also use for the official crosscompilation.

Variable names shall be borrowed from the cmake template
~/Android/Sdk/ndk/20.1.5948944/build/cmake/android.toolchain.cmake
We shall only use ANDROID_API_LEVEL instead of ANDROID_PLATFORM_LEVEL, as it describes the API level.

The updated gluegen/make/scripts/setenv-android-tools.sh
shall be use preset values for:

# - ANDROID_HOME - defaults to one of
#        ~/Android/Sdk
#        /opt-linux-x86_64/android-sdk-linux_x86_64
#        /opt/android-sdk-linux_x86_64
#        /usr/local/android-sdk-linux_x86_64
#
# - ANDROID_API_LEVEL - defaults to 24
#
# - ANDROID_HOST_TAG - defaults to linux-x86_64
#
# - ANDROID_ABI - defaults to x86_64, one of
#        armeabi-v7a 
#        arm64-v8a 
#        x86_64
#        x86

Reusing the standard environment variable names should ease migration
and the cmake template also gives clues.

Desired target ABIs currently are: armeabi-v7a, arm64-v8a, x86_64, x86
Comment 1 Sven Gothel 2019-12-10 06:33:27 CET
gluegen 74e5bbe097a07d1275a2d6452d1e2336a8d5c44a

Android: Support building using current SDK and NDK v20 as of 2019-12-10 (GlueGen w/ android-aarch64 patched)
    
    Default Platform: Android 7.0 Nougat, API level 24 supports Java 1.8
    
    Reworked gluegen/make/scripts/setenv-android-tools.sh,
    following canonical environment variables and sets all
    required variables for cross-compilation.
    See file's comments!
    
    ant files:
      - android.version -> android.api.level (default 24)
      - android.jar defaults to ${gluegen.root}/make/lib/android-sdk/24/android.jar (provided)
Comment 2 Sven Gothel 2019-12-10 06:39:34 CET
TODO: Complete patching all previously supported target compiler configurations
  gluegen/make/lib/gluegen-cpptasks-android-<abi>.xml 
and their build scripts 
  gluegen/make/scripts/make.gluegen.all.android-<abi>-cross.sh

Do the same for joal, jogl and jocl build scripts and test.

Ensure the build node is updated accordingly for jenkins.

Finally kick off 2.4.0 pre-release build ;-)
Comment 3 Sven Gothel 2019-12-10 18:51:36 CET
ant-cpptasks 4b859d7b63e5cf4aa74f740d4ae7f46aa13788c9
Android clang support: Eclipse Code Cleanup

ant-cpptasks 6bdee37988009d3006ccf99bac1ccea497141afd
Android clang support: ant build update: java 1.8

ant-cpptasks e1933c74b9c256332cd12c9a1094e4687b535112
Android clang support: Introduce clangLinkerOption '-target <abi>'
Comment 4 Sven Gothel 2019-12-10 19:12:06 CET
gluegen patches:

b6a91a035c14f1e4d142517f311007e48ea9c0c1
Complete 1st round: make/lib/gluegen-cpptask-android-<abi>.xml and scripts

4f7273e017e2deb2928a76066604ced03609f921
Update cpptasks.jar

19761070e12f65fe7a07e6d9b7a5221a1623b12d
Add new x86 MachineDataInfo, differs due to smaller 'long double' than regular x86_32_unix

eb6dfabd7427a1836f6b1d2080422f98808ea0a4
Android: Manifest*-android* can no more have 'Name: <folder>' directives for apk

51ef5eadd9db020412d3a3716b4ab5a25b0522fb
Resulting dynamic libraries require the 'soname' to be set since API level 23
Comment 5 Sven Gothel 2019-12-10 19:18:34 CET
OK, validated all gluegen-rt-android-<abi>.apk for x86 (emulator), armv7 and aarch64:
- installed
- run/tested
Comment 6 Sven Gothel 2019-12-12 13:27:26 CET
ant-cpptasks ec14765d6baab247fa37d62845c139adc367af2a
ant-cpptasks 79a8fed9fddab5429a3457c3077ee5012b405a51
ant-cpptasks 07aeed47e484bfec85e749ea721756b8a2571e00
New CCTask parameter 'writesoname': Adds GnuLinker and clang linker '-h name' options, writing SONAME for shares libraries

ant-cpptasks 8353e0cc85705ca8f81f8563ad975026978b6745

gluegen 53ae28fd2bc501b5de54fdbdf915d88f823eb11a
Update cpptask, using implicit 'soname' setup, which scales better for all platforms.
Comment 7 Sven Gothel 2019-12-24 16:43:55 CET
gluegen 26d06324312b8fe6980ff29ab9d443cc5ec2657f
    Bug 1417 - Android: Classify SYSROOTs in NDK for native libraries (link) and headers (compile)
    
    Utilizing '-sysroot' with the linker, we require standard FS layout.
    
    '~/Android/Sdk/ndk/20.1.5948944/platforms/android-24/arch-arm64'
    contains the proper std FS layout for libraries under usr/lib,
    but lacks the header files.
    We call this 'ANDROID_TOOLCHAIN_SYSROOT0' and shall be used for linking.
    
    '~/Android/Sdk/ndk/20.1.5948944/toolchains/llvm/prebuilt/linux-x86_64/sysroot'
    native libraries are not under std FS layout,
    but headers are under std FS layout for compilation under usr/include.
    We call this 'ANDROID_TOOLCHAIN_SYSROOT1' and shall be used for compilation.
    
    Identified while working on openal-soft Android compilation,
    where we need to pass ANDROID_TOOLCHAIN_SYSROOT0

gluegen aeae740ba2569d839820bfc20c43a8550ab8ed33
    Bug 1417 - Android: Add ANDROID_TOOLCHAIN_SYSROOT1_INC_ARCH for arch specific includes like asm/types.h
    
    Also adapt make/lib/gluegen-cpptask-android* files:
    - Set C++ stdlib = libstdc++
    - Set isysroot (same as sysroot)
    - Add isystem: *_SYSROOT1_INC, *_SYSROOT1_INC_ARCH and *_SYSROOT1_INC/c++/v1

gluegen ae4cdcf3490a1bf493f39ab5c8c6b596d3af71de
    Bug 1417 - Android: Add ANDROID_TOOLCHAIN_SYSROOT1_INC_STL for LLVM's C++ STL headers (default since NDK r18)
    
    Using LLVM's c++_shared as of NDK r18: https://developer.android.com/ndk/guides/cpp-support.html
    LLVM's c++ headers must come before other system header!
    Also see https://github.com/android/ndk/issues/452 and https://gitlab.kitware.com/cmake/cmake/issues/17059
    
    Also adapt make/lib/gluegen-cpptask-android* files:
    - Compiler
    -- align for potential C++ usage, i.e.:
    -- don't use '-sysroot' not '-isysroot'
    -- IF using C++ STL, prepend ANDROID_TOOLCHAIN_SYSROOT1_INC_STL
    -- just add normal sysroot headers via '-isystem'
    
    - Linker
    -- Use ANDROID_TOOLCHAIN_SYSROOT1_LIB1 and ANDROID_TOOLCHAIN_SYSROOT1_LIB2 for '-L' search path
    -- where ANDROID_TOOLCHAIN_SYSROOT1_LIB2 is for 'libc++' -> 'c++_shared'
Comment 8 Sven Gothel 2019-12-24 16:50:09 CET
joal b8e20cd8238c2f3694068f501ba06b9a6f654645
Bug 1417 - Android: OpenAL-Soft toolchain.android.cmake cleanup and NDK r20 adoption llvm: clang libc++
Comment 9 Sven Gothel 2019-12-24 19:30:12 CET
jogl 07de253dae77b39da944a946381bd6efec0066ba
Bug 1417 - Android: Adopt to gluegen/make/scripts/setenv-android-tools.sh changes
Comment 10 Sven Gothel 2019-12-25 02:10:29 CET
jogl c6ba090a0030c177d7e60f797a3ec25fecfe3546

Bug 1417 - Android: Disable native FFmpeg binding for Android
Comment 11 Sven Gothel 2019-12-25 04:17:53 CET
openal-soft f344ef3aa3a42c4e81376c3d8ad6507ad2b8a20c
    Bug 1417 - Android: OpenAL-Soft toolchain.android.cmake cleanup and NDK r20 adoption llvm: clang libc++
    
    All properties are now aligned with gluegen/make/scripts/setenv-android-tools.sh.
    User must set ANDROID_NDK, ANDROID_HOST_TAG, ANDROID_API_LEVEL and one of
    NDK_CPU_X86_64, NDK_CPU_X86, NDK_CPU_ARM64, NDK_CPU_ARM_V7A, NDK_CPU_ARM.

openal-soft 44f5d564bacfd8adcbb1d59763ed8970667824af
    Bug 1417 - Android: OpenAL-Soft toolchain.android.cmake: ANDROID_API_LEVEL [21..22] only, link against c++_static!
    
    NDK >= r19 using API > 22: 'U __register_atfork'
       __register_atfork; # introduced=23
    See <https://github.com/android/ndk/issues/964>
    
    libc++_shared.so is not provided on device,
    either link statically or deploy the shared library.
    We link statically now.

joal 8907cb8550def10640f9d812853fbadf3f2989f5
    Bug 1417 - Android: OpenAL-Soft: ANDROID_API_LEVEL [21..22] only, link against c++_static!
Comment 12 Sven Gothel 2019-12-25 05:51:54 CET
Gluegen + Joal bring-up test success on Android: 
- x86 (sim), 
- armv7a (Nexus 10) and 
- aarch64 (Nexus 9)

> 'GlueGen's Version' was able to load native lib and retrieve info

> OALTest1 (Joal/OpenAL) was able to load openal and play the sample

+++

openal-soft 0d14d30808cfe7b9e3413353e3eef8a0f201399a
  Bug 1417 - Android: OpenAL-Soft toolchain.android.cmake: Only build libOpenAL.so on Android

joal 647eaa5fc88a9f4dfc0ec421f5ae79474d1117af
  OpenAL-Soft toolchain.android.cmake: Only build libOpenAL.so on Android

joal 768677b5b56a9cfea3accf4b7638d217eceade44
  Jar Manifest joalversion-android can no more have 'Name: <folder>' directives for apk
    
  The 'Name <folder>' leads new apk to claim the missing resources (the folder actually)
  and fails to install the apk.
    
  Hence removed tagging the folders sealed.
    
  See gluegen commit eb6dfabd7427a1836f6b1d2080422f98808ea0a4

+++
Comment 13 Sven Gothel 2019-12-25 08:43:08 CET
jogl 784dcfa2094f3fb235ca4b60395708a1f87c0b1b

    Bug 1156 Regression (Bug 1417): Probe whether 'eglGetPlatformDisplay(..)' is available before using

See Bug 1156 comment 32 for details.

+++

JOGL bring-up test success on Android: 
- x86 (sim), 
- armv7a (Nexus 10) and 
- aarch64 (Nexus 9)

> Tested a few activities ..

+++

TODO: JOCL
Comment 14 Sven Gothel 2019-12-26 05:28:12 CET
JOGL bring-up (incl GlueGen + JOAL) test success on Android: 
- x86 (sim), 
- armv7a (Nexus 10) and 
- aarch64 (Nexus 9)

Tested a few activities incl 'GraphUI1p' (w/ GraphUI + Video + JOAL)

+++

gluegen d190301a82a961ac3246890206200086842059bd
  Add adb scripts to force-stop and clear all jogamp tasks and packages

gluegen 6e98be1a1bc819b47e2cc52eb35d23a7489fdf1b
  jogamp-env.xml: Elaborate on Android SDK API versions required [min=21, target=28], javac w/ 24, some NDK's clang with 22.

gluegen faa37334e2912e02cb58e738570a127cc6402b37
  Context.MODE_WORLD_READABLE causes SecurityException on API >= 24, use Context.MODE_PRIVATE for temp cache

gluegen 1390bc236de531646aaa4525a17de1a533b1fa06
  Use d8 for dex'ing w/ implicit Java8 desugaring (macro: aapt.signed)
    
  The new SDK d8 dex'ing tool performs Java8 desugaring, hence allows using Java8 code style.
    
  We use d8 instead of dx in the 'aapt.signed' macro of 'jogamp-androidtasks.xml'.
    
  For the proper desugaring, we pass 'android-min.jar and 'target.rt.jar' Android and Java8
  system runtime jar-files to d8's classpath.
    
  Macro 'aapt.signed' also receives the new optional element 'd8-classpaths'
  to allows caller to pass further required jar-files to the d8 classpath.
  See build-test.xml.

joal c4f73b67ebbb554cde69b6f59ab377a38864df72  
  APK Manifest [min=21, target=28]; Adopt to GlueGen's 'aapt.signed' d8 usage

jogl d91f8581eedf73663abab25524147d2bcab9109b
  Add adb scripts for x86 [re-]install

jogl 95ca88ba975c7296098bbd34bec2d922aeb2b563
  APK Manifest [min=21, target=28]; Adopt to GlueGen's 'aapt.signed' d8 usage
Comment 15 Sven Gothel 2019-12-26 18:40:22 CET
Jocl bring-up test success on Android: 
- x86 (sim), 
- armv7a (Nexus 10) and 
- aarch64 (Nexus 9)

On armv7a (Nexus 10) JOCL is even available and could be retrieved.

This bug is now closed as fixed.

+++

jocl 22b41b9a232527d8c2b2a2b03e03005273529df7
  Adapt to GlueGen's 'aapt.signed' d8 usage and use manifest SDK API [min=21, target=28]
    
  Also fix all make/scripts using Java11 as 'host compiler' and Java8 as target-rt-jar,
  as well as adapt to GlueGen's new Android 'setenv-android-tools.sh' for new SDK/NDK as of 2019-12-10.

jocl 023fff1af43a172d4c2a0e18fdbb78e6ba817cca
  Android: Manifest*-android* can no more have 'Name: <folder>' directives for apk
    
  The 'Name <folder>' leads new apk to claim the missing resources (the folder actually)
  and fails to install the apk.
    
  Hence removed tagging the folders sealed.
Comment 16 Sven Gothel 2019-12-26 19:32:52 CET
jocl 4287fdd657b946c9b605361692dff56b2b99ab1a
  Native CC: Set usehighleveltool=true (e.g. ld via clang) writesoname=true (required by android)
Comment 17 Sven Gothel 2020-02-04 01:00:26 CET
Retroactive: This bug blocked Bug 1363, as Java-8 minimum requirement also requires us to utilize the current Android SDK + NDK.

Especially comment 14 described usage of d8 is related here, besides others:

gluegen 1390bc236de531646aaa4525a17de1a533b1fa06
  Use d8 for dex'ing w/ implicit Java8 desugaring (macro: aapt.signed)