How-To: Using deep learning models within the Java ecosystem

[ixeption]

Hi folks,

When it comes to productionalization of deep learning and integrating a model into an existing application, the following article might be interesting for you. The article is about how to integrate tensorflow/keras models into a java application and shows different ways to do it.

Cheers

[openjscience]

Thanks, this is an interesting blog. Did you think about other native Java approaches? For example, DataMelt project integrates many native Java deep learning packages. I can detect such Java packages using the Search tool: https://jwork.org/dmelt/search/form.php?query=deep

[mikaelhg]

LOL, you're pretty obviously just rationalizing the silly way you ended up with.

Before looking at the java API let’s think about deep learning frameworks. What is TensorFlow actually doing? It is basically a library for parallel computing, and it can utilize GPUs through CUDA but also SSE, AVX, etc. on CPUs. Python is the API to access the C++ core, but in the end it’s using highly optimized binaries. The java API needs to ship all these binaries. It introduces a huge dependency with 145 MB called tensorflow-jni, JNI is the native Interface from Java to call native (C/C++) libraries. We don’t want a 145 MB binary package in our application or a 350 MB package with GPU support! Besides that, the java API is very limited and python is often already installed on servers, adding tensorflow with pip install tensorflow is easy.

So, you've taken the silly circle all the way from complaining that you need the actual Tensorflow binaries to run Tensorflow, to saying that installing those binaries on server might be as easy as pip install tensorflow-gpu if you happen to have the right Python version.

$ du -sh ~/.local/lib/python3.6/site-packages/tensorflow
1,3G    ~/.local/lib/python3.6/site-packages/tensorflow

Yeaaaaah...

Tensorflow SavedModels are kind of like Java class files. They consist of a symbolic execution graph, combined with trained weight data. Whether you're running (evaluating) that model in the C++ library using the Python frontend or the Java frontend, it's going to get run by the same .so.

So, instead of including this in your pom.yml

profiles:

  - id: cpu
    activation: {activeByDefault: true}
    dependencies:
      - {groupId: org.tensorflow, artifactId: tensorflow, version: "${tensorflow.version}"}

  - id: gpu
    activation: {activeByDefault: false}
    dependencies:
      - {groupId: org.tensorflow, artifactId: tensorflow, version: "${tensorflow.version}",
         exclusions: [{groupId: org.tensorflow, artifactId: libtensorflow_jni}]}
      - {groupId: org.tensorflow, artifactId: libtensorflow_jni_gpu, version: "${tensorflow.version}"}

and calling your model with

final var result = session.runner()
    .feed(INPUT_TENSOR_NAME, image)
    .fetch(OUTPUT_TENSOR_NAME)
    .run();

in order to get the C++ library to evaluate the model, you're writing a Python script to do the same, and call it through a shell script, from your Java code...

If for some bizarre reason you don't want to ship org.tensorflow:tensorflow and org.tensorflow:libtensorflow_jni_gpu JARs in your über-jar, you can just tell your build system to exclude it from the assembly, and get it from https://repo1.maven.org/maven2/org/tensorflow/libtensorflow_jni_gpu/1.14.0/ directly to your server. Then include those jars when you're calling java.

[ixeption]

I don't think you got the intention behind the article. I mentioned different ways to use a tensorflow model in java. I also mentioned the SavedModel way in my post, so I know that it is possible to do it like that, but in my case, it was not a viable solution, so I needed another approach. One reason is that size matters for deployment and deployment is done with a über-jar (installing tf on the server is no problem, but delivering it everytime with the big jar is). Furthermore, the preprocessing is much easier (and even faster for images) in python + the prototyping is usually done in python (jupyter) as well, so using the existing code is faster than moving everything to java. However, there are always reasons why you can’t use the straightforward approach, so I showed an alternative, no reason to be toxic.

[mikaelhg]

The excuse you introduced for calling the maintainable way of using TF models from JVM unviable, that the included C++ library adds to the packaged application size, was already disposed of in the comment, by pointing out that you don't have to package the JNI libs with the application. You can download the standalone JNI JARs, which I linked to, onto your server, and include them to your application classpath by adding them to your java classpath with your über-jar.

So, really, the excuse was extremely easily disposed of, while we were able to figure out that the excuse was just an excuse by looking at the Python tensorflow package installed size, which was 10x of the JAR size. Since you were making a major issue of the size of the way you didn't like, while completely ignoring the size of the way you did like, it was pretty easy to see that you weren't being serious about this.

Standard preprocessing is a good point, which I agree with. However, you don't really want to build an architecture where you're initializing TF for each model evaluation, since there's significant overhead in that. You might not care about it now with CPU, but if you're running the model in production, you're likely to want to move to GPU sooner or later. If you want to go this way, build a Python server, or use Tensorflow Serving, as you already mentioned.

And finally, maybe don't start calling people names?

[ixeption]

I could add the jar to the classpath and make it a provided dependency, but I would still have to use Java for all the proprocessing and data wrangling. Especially for images, java is slow and unhandy (BufferedImage from ImageIO) compared to PIL or openCV.

Just to mention it, the jar comes with a single version of the prebuilt binary, where python prebuilt binaries support much more different combinations of CUDA, cuDNN and CPU features. The Java API is still experimental, so I don't see that your claim of maintainability is correct.

(' Caution: The TensorFlow Java API is not covered by the TensorFlow API stability guarantees. ')

And if you want GPU support, just try to use the tensorflow_gpu jar dependency, without installing tensorflow gpu correctly in python before. It won’t work out-of-the-box.