Prerequisites and Configurations for DL4J in Android

Contents

  • Prerequisites
  • Required Dependencies
  • Managing Dependencies with ProGuard
  • Memory Management
  • Saving and Loading Networks on AndroidWhile neural networks are typically run on powerful computers using multiple GPUs, the compatibility of Deeplearning4J with the Android platform makes using DL4J neural networks in android applications a possibility. This tutorial will cover the basics of setting up android studio for building DL4J applications. Several configurations for dependencies, memory management, and compilation exclusions needed to mitigate the limitations of low powered mobile device are outlined below. If you just want to get a DL4J app running on your device, you can jump ahead to a simple demo application which trains a neural network for Iris flower classification available here.

Prerequisites

  • Android Studio 2.2 or newer, which can be downloaded here.
  • Android Studio version 2.2 and higher comes with the latest OpenJDK embedded; however, it is recommended to have the JDK installed on your own as you are then able to update it independent of Android Studio. Android Studio 3.0 and later supports all of Java 7 and a subset of Java 8 language features. Java JDKs can be downloaded from Oracle’s website.
  • Within Android studio, the Android SDK Manager can be used to install Android Build tools 24.0.1 or later, SDK platform 24 or later, and the Android Support Repository.
  • An Android device or an emulator running API level 21 or higher. A minimum of 200 MB of internal storage space free is recommended.It is also recommended that you download and install IntelliJ IDEA, Maven, and the complete dl4j-examples directory for building and building and training neural nets on your desktop instead of android studio.

Required Dependencies

In order to use Deeplearning4J in your Android projects, you will need to add the following dependencies to your app module’s build.gradle file. Depending on the type of neural network used in your application, you may need to add additional dependencies.

  1. compile (group: 'org.deeplearning4j', name: 'deeplearning4j-core', version: '1.0.0-beta4') {
  2.     exclude group: 'org.bytedeco.javacpp-presets', module: 'opencv-platform'
  3.     exclude group: 'org.bytedeco.javacpp-presets', module: 'leptonica-platform'
  4.     exclude group: 'org.bytedeco.javacpp-presets', module: 'hdf5-platform'
  5.     exclude group: 'org.nd4j', module: 'nd4j-base64'
  6. }
  7. compile group: 'org.nd4j', name: 'nd4j-native', version: '1.0.0-beta4'
  8. compile group: 'org.nd4j', name: 'nd4j-native', version: '1.0.0-beta4', classifier: "android-arm"
  9. compile group: 'org.nd4j', name: 'nd4j-native', version: '1.0.0-beta4', classifier: "android-arm64"
  10. compile group: 'org.nd4j', name: 'nd4j-native', version: '1.0.0-beta4', classifier: "android-x86"
  11. compile group: 'org.nd4j', name: 'nd4j-native', version: '1.0.0-beta4', classifier: "android-x86_64"
  12. compile group: 'org.bytedeco.javacpp-presets', name: 'openblas', version: '0.3.3-1.4.3'
  13. compile group: 'org.bytedeco.javacpp-presets', name: 'openblas', version: '0.3.3-1.4.3', classifier: "android-arm"
  14. compile group: 'org.bytedeco.javacpp-presets', name: 'openblas', version: '0.3.3-1.4.3', classifier: "android-arm64"
  15. compile group: 'org.bytedeco.javacpp-presets', name: 'openblas', version: '0.3.3-1.4.3', classifier: "android-x86"
  16. compile group: 'org.bytedeco.javacpp-presets', name: 'openblas', version: '0.3.3-1.4.3', classifier: "android-x86_64"
  17. compile group: 'org.bytedeco.javacpp-presets', name: 'opencv', version: '3.4.3-1.4.3'
  18. compile group: 'org.bytedeco.javacpp-presets', name: 'opencv', version: '3.4.3-1.4.3', classifier: "android-arm"
  19. compile group: 'org.bytedeco.javacpp-presets', name: 'opencv', version: '3.4.3-1.4.3', classifier: "android-arm64"
  20. compile group: 'org.bytedeco.javacpp-presets', name: 'opencv', version: '3.4.3-1.4.3', classifier: "android-x86"
  21. compile group: 'org.bytedeco.javacpp-presets', name: 'opencv', version: '3.4.3-1.4.3', classifier: "android-x86_64"
  22. compile group: 'org.bytedeco.javacpp-presets', name: 'leptonica', version: '1.76.0-1.4.3'
  23. compile group: 'org.bytedeco.javacpp-presets', name: 'leptonica', version: '1.76.0-1.4.3', classifier: "android-arm"
  24. compile group: 'org.bytedeco.javacpp-presets', name: 'leptonica', version: '1.76.0-1.4.3', classifier: "android-arm64"
  25. compile group: 'org.bytedeco.javacpp-presets', name: 'leptonica', version: '1.76.0-1.4.3', classifier: "android-x86"
  26. compile group: 'org.bytedeco.javacpp-presets', name: 'leptonica', version: '1.76.0-1.4.3', classifier: "android-x86_64"
  27. testCompile 'junit:junit:4.12'

DL4J depends on ND4J, which is a library that offers fast n-dimensional arrays. ND4J in turn depends on a platform-specific native code library called JavaCPP, therefore you must load a version of ND4J that matches the architecture of the Android device. Both -x86 and -arm types can be included to support multiple device processor types.

The above dependencies contain several files with identical names which must be handled with the following exclude parameters to your packagingOptions.

  1. packagingOptions {
  2.     exclude 'META-INF/DEPENDENCIES'
  3.     exclude 'META-INF/DEPENDENCIES.txt'
  4.     exclude 'META-INF/LICENSE'
  5.     exclude 'META-INF/LICENSE.txt'
  6.     exclude 'META-INF/license.txt'
  7.     exclude 'META-INF/NOTICE'
  8.     exclude 'META-INF/NOTICE.txt'
  9.     exclude 'META-INF/notice.txt'
  10.     exclude 'META-INF/INDEX.LIST'
  12. }

After adding the above dependencies and exclusions to the build.gradle file, try syncing Gradle with to see if any other exclusions are needed. The error message will identify the file path that should be added to the list of exclusions. An example error message with file path is: _> More than one file was found with OS independent path ‘org/bytedeco/javacpp/ windows-x86_64/msvp120.dll’_Compiling these dependencies involves a large number of files, thus it is necessary to set multiDexEnabled to true in defaultConfig.

  1. multiDexEnabled true

A conflict in the junit module versions often causes the following error: > Conflict with dependency ‘junit:junit’ in project ‘:app’. Resolved versions for app (4.8.2) and test app (4.12) differ. This can be suppressed by forcing all of the junit modules to use the same version with the following:

  1. configurations.all {
  2.     resolutionStrategy.force 'junit:junit:4.12'
  3. }

Managing Dependencies with ProGuard

The DL4J dependencies compile a large number of files. ProGuard can be used to minimize your APK file size. ProGuard detects and removes unused classes, fields, methods, and attributes from your packaged app, including those from code libraries. You can learn more about using Proguard here.To enable code shrinking with ProGuard, add minifyEnabled true to the appropriate build type in your build.gradle file.

  1. buildTypes {
  2.     release {
  3.         minifyEnabled true
  4.         proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
  5.     }
  6. }

It is recommended to upgrade your ProGuard in the Android SDK to the latest release (5.1 or higher). Note that upgrading the build tools or other aspects of your SDK might cause Proguard to reset to the version shipped with the SDK. In order to force ProGuard to use a version of other than the Android Gradle default, you can include this in the buildscript of build.gradle file:

  1. buildscript {
  2.     configurations.all {
  3.         resolutionStrategy {
  4.             force 'net.sf.proguard:proguard-gradle:5.3.2'
  5.         }
  6.     }
  7. }

Proguard optimizes and reduces the amount of code in your Android application in order to make if smaller and faster. Unfortunately, proguard removes annotations by default, including the @Platform annotation used by javaCV. To make proguard preserve these annotations and keep native methods add the following flags to the progaurd-rules.pro file.

  1. # enable optimization
  2. -optimizations !code/simplification/arithmetic,!code/simplification/cast,!field/*,!class/merging/*
  3. -optimizationpasses 5
  4. -allowaccessmodification
  5. -dontwarn org.apache.lang.**
  6. -ignorewarnings
  8. -keepattributes *Annotation*
  9. # JavaCV
  10. -keep @org.bytedeco.javacpp.annotation interface * {*;}
  11. -keep @org.bytedeco.javacpp.annotation.Platform public class *
  12. -keepclasseswithmembernames class * {@org.bytedeco.* <fields>;}
  13. -keepclasseswithmembernames class * {@org.bytedeco.* <methods>;}
  15. -keepattributes EnclosingMethod
  16. -keep @interface org.bytedeco.javacpp.annotation.*,javax.inject.*
  18. -keepattributes *Annotation*, Exceptions, Signature, Deprecated, SourceFile, SourceDir, LineNumberTable, LocalVariableTable, LocalVariableTypeTable, Synthetic, EnclosingMethod, RuntimeVisibleAnnotations, RuntimeInvisibleAnnotations, RuntimeVisibleParameterAnnotations, RuntimeInvisibleParameterAnnotations, AnnotationDefault, InnerClasses
  19. -keep class org.bytedeco.javacpp.** {*;}
  20. -dontwarn java.awt.**
  21. -dontwarn org.bytedeco.javacv.**
  22. -dontwarn org.bytedeco.javacpp.**
  23. # end javacv
  25. # This flag is needed to keep native methods
  26. -keepclasseswithmembernames class * {
  27.  native <methods>;
  28. }
  30. -keep public class * extends android.view.View {
  31.  public <init>(android.content.Context);
  32.  public <init>(android.content.Context, android.util.AttributeSet);
  33.  public <init>(android.content.Context, android.util.AttributeSet, int);
  34.  public void set*(...);
  35. }
  37. -keepclasseswithmembers class * {
  38.  public <init>(android.content.Context, android.util.AttributeSet);
  39. }
  41. -keepclasseswithmembers class * {
  42.  public <init>(android.content.Context, android.util.AttributeSet, int);
  43. }
  45. -keepclassmembers class * extends android.app.Activity {
  46.  public void *(android.view.View);
  47. }
  49. # For enumeration classes
  50. -keepclassmembers enum * {
  51.  public static **[] values();
  52.  public static ** valueOf(java.lang.String);
  53. }
  55. -keep class * implements android.os.Parcelable {
  56.  public static final android.os.Parcelable$Creator *;
  57. }
  59. -keepclassmembers class **.R$* {
  60.  public static <fields>;
  61. }
  63. -keep class android.support.v7.app.** { *; }
  64. -keep interface android.support.v7.app.** { *; }
  65. -keep class com.actionbarsherlock.** { *; }
  66. -keep interface com.actionbarsherlock.** { *; }
  67. -dontwarn android.support.**
  68. -dontwarn com.google.ads.**
  70. # Flags to keep standard classes
  71. -keep public class * extends android.app.Activity
  72. -keep public class * extends android.app.Application
  73. -keep public class * extends android.app.Service
  74. -keep public class * extends android.content.BroadcastReceiver
  75. -keep public class * extends android.content.ContentProvider
  76. -keep public class * extends android.app.backup.BackupAgent
  77. -keep public class * extends android.preference.Preference
  78. -keep public class * extends android.support.v7.app.Fragment
  79. -keep public class * extends android.support.v7.app.DialogFragment
  80. -keep public class * extends com.actionbarsherlock.app.SherlockListFragment
  81. -keep public class * extends com.actionbarsherlock.app.SherlockFragment
  82. -keep public class * extends com.actionbarsherlock.app.SherlockFragmentActivity
  83. -keep public class * extends android.app.Fragment
  84. -keep public class com.android.vending.licensing.ILicensingService

Testing your app is the best way to check if any errors are being caused by inappropriately removed code; however, you can also inspect what was removed by reviewing the usage.txt output file saved in /build/outputs/mapping/release/.

To fix errors and force ProGuard to retain certain code, add a -keep line in the ProGuard configuration file. For example:

  1. -keep public class MyClass

Memory Management

It may also be advantageous to increase the allocated memory to your app by adding android:largeHeap=”true” to the manifest file. Allocating a larger heap means that you decrease the risk of throwing an OutOfMemoryError during memory intensive operations.

  1. android:largeHeap="true"

As of release 0.9.0, ND4J offers an additional memory-management model: workspaces. Workspaces allow you to reuse memory for cyclic workloads without the JVM Garbage Collector for off-heap memory tracking. D4j Workspace allows for memory to be preallocated before a try / catch block and reused over in over within that block.

If your training process uses workspaces, it is recommended that you disable or reduce the frequency of periodic GC calls prior to your model.fit() call.

  1. // this will limit frequency of gc calls to 5000 milliseconds
  2. Nd4j.getMemoryManager().setAutoGcWindow(5000)
  4. // this will totally disable it
  5. Nd4j.getMemoryManager().togglePeriodicGc(false);

The example below illustrates the use of a Workspace for memory allocation in the AsyncTask of and Android Application. More information concerning ND4J Workspaces can be found here.

  1. import org.nd4j.linalg.api.memory.MemoryWorkspace;
  2. import org.nd4j.linalg.api.memory.conf.WorkspaceConfiguration;
  3. import org.nd4j.linalg.api.memory.enums.AllocationPolicy;
  4. import org.nd4j.linalg.api.memory.enums.LearningPolicy;
  7. private class AsyncTaskRunner extends AsyncTask<String, Integer, INDArray> {
  9.     // Runs in UI before background thread is called
  10.     @Override
  11.     protected void onPreExecute() {
  12.         super.onPreExecute();
  13.     }
  15.     //Runs on background thread, this is where we will initiate the Workspace
  16.     protected INDArray doInBackground(String... params) {
  18.         // we will create configuration with 10MB memory space preallocated
  19.         WorkspaceConfiguration initialConfig = WorkspaceConfiguration.builder()
  20.                 .initialSize(10 * 1024L * 1024L)
  21.                 .policyAllocation(AllocationPolicy.STRICT)
  22.                 .policyLearning(LearningPolicy.NONE)
  23.                 .build();
  25.         INDArray result = null;
  27.         try(MemoryWorkspace ws = Nd4j.getWorkspaceManager().getAndActivateWorkspace(initialConfig, "SOME_ID")) {
  28.         // now, INDArrays created within this try block will be allocated from this workspace pool
  30.             //Load a trained model
  31.             File file = new File(Environment.getExternalStorageDirectory() + "/trained_model.zip");
  32.             MultiLayerNetwork restored = ModelSerializer.restoreMultiLayerNetwork(file);
  34.             // Create input in INDArray
  35.             INDArray inputData = Nd4j.zeros(1, 4);
  37.             inputData.putScalar(new int[]{0, 0}, 1);
  38.             inputData.putScalar(new int[]{0, 1}, 0);
  39.             inputData.putScalar(new int[]{0, 2}, 1);
  40.             inputData.putScalar(new int[]{0, 3}, 0);
  42.             result = restored.output(inputData);
  44.         }
  45.         catch(IOException ex){Log.d("AsyncTaskRunner2 ", "catchIOException = " + ex  );}
  47.         return result;
  48.     }
  50.     protected void onProgressUpdate(Integer... values) {
  51.         super.onProgressUpdate(values);
  52.     }
  54.     protected void onPostExecute(INDArray result) {
  55.         super.onPostExecute(result);
  56.      //Handle results and update UI here.
  57.     }
  59. }

Saving and Loading Networks on Android

Practical considerations regarding performance limits are needed when building Android applications that run neural networks. Training a neural network on a device is possible, but should only be attempted with networks with limited numbers of layers, nodes, and iterations. The first Demo app DL4JIrisClassifierDemo is able to train on a standard device in about 15 seconds.

When training on a device is a reasonable option, the application performance can be improved by saving the trained model on the phone’s external storage once an initial training is complete. The trained model can then be used as an application resource. This approach is useful for training networks with data obtained from user input. The following code illustrates how to train a network and save it on the phone’s external resources.

For API 23 and greater, you will need to include the permissions in your manifest and also programmatically request the read and write permissions in your activity. The required Manifest permissions are:

  1. <manifest>
  2.         <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
  3.         <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
  4.         ...

You need to implement ActivityCompat.OnRequestPermissionsResultCallback in the activity and then check for permission status.

  1. public class MainActivity extends AppCompatActivity
  2.         implements ActivityCompat.OnRequestPermissionsResultCallback {
  4.     private static final int REQUEST_EXTERNAL_STORAGE = 1;
  5.     private static String[] PERMISSIONS_STORAGE = {
  6.             Manifest.permission.READ_EXTERNAL_STORAGE,
  7.             Manifest.permission.WRITE_EXTERNAL_STORAGE
  8.     };
  10.     @Override
  11.     protected void onCreate(Bundle savedInstanceState) {
  12.         super.onCreate(savedInstanceState);
  13.         setContentView(R.layout.activity_main);
  15.         verifyStoragePermission(MainActivity.this);
  16.     //…
  17.     }
  19.     public static void verifyStoragePermission(Activity activity) {
  20.         // Get permission status
  21.         int permission = ActivityCompat.checkSelfPermission(activity, Manifest.permission.WRITE_EXTERNAL_STORAGE);
  22.         if (permission != PackageManager.PERMISSION_GRANTED) {
  23.         // We don't have permission we request it
  24.         ActivityCompat.requestPermissions(
  25.                     activity,
  26.                     PERMISSIONS_STORAGE,
  27.                     REQUEST_EXTERNAL_STORAGE
  28.             );
  29.         }
  30.     }

To save a network after training on the device use a OutputStream within a try catch block.

  1. try {
  2.     File file = new File(Environment.getExternalStorageDirectory() + "/trained_model.zip");
  3.     OutputStream outputStream = new FileOutputStream(file);
  4.     boolean saveUpdater = true;
  5.     ModelSerializer.writeModel(myNetwork, outputStream, saveUpdater);
  7. } catch (Exception e) {
  8.     Log.e("saveToExternalStorage error", e.getMessage());
  9. }

To load the trained network from storage you can use the restoreMultiLayerNetwork method.

  1. try{
  2.     //Load the model
  3.     File file = new File(Environment.getExternalStorageDirectory() + "/trained_model.zip");
  4.     MultiLayerNetwork restored = ModelSerializer.restoreMultiLayerNetwork(file);
  6. } catch (Exception e) {
  7.     Log.e("Load from External Storage error", e.getMessage());
  8. }

For larger or more complex neural networks like Convolutional or Recurrent Neural Networks, training on the device is not a realistic option as long processing times during network training run the risk of generating an OutOfMemoryError and make for a poor user experience. As an alternative, the Neural Network can be trained on the desktop, saved via ModelSerializer, and then loaded as a pre-trained model in the application. Using a pre-trained model in you Android application can be achieved with the following steps:

  • Train the yourModel on desktop and save via modelSerializer.
  • Create a raw resource folder in the res directory of the application.
  • Copy yourModel.zip file into the raw folder.
  • Access it from your resources using an inputStream within a try / catch block.
  1. try {
  2. // Load name of model file (yourModel.zip).
  3.         InputStream is = getResources().openRawResource(R.raw.yourModel);
  5. // Load yourModel.zip.
  6.         MultiLayerNetwork restored = ModelSerializer.restoreMultiLayerNetwork(is);
  8. // Use yourModel.
  9.         INDArray results = restored.output(input)
  10.         System.out.println("Results: "+ results );
  11. // Handle the exception error
  12. } catch(IOException e) {
  13.         e.printStackTrace();
  14.     }

Next Step: Pretrained DL4J Models on Android

An example application which uses a pretrained model can be found here.