6-6 Model Deploying Using tensorflow-serving

There are multiple ways to deploy and run the trained models which saved with the original tensorflow format.

For example:

We can load and run the model in the web browser using javascript through tensorflow-js.

We can load and run the TensorFlow model on mobile and embeded devices through tensorflow-lite.

We can use tensorflow-serving to load the model that providing network interface API service and to acquire the prediction results from the model through sending network requests in arbitrary programming languages.

We can predict using the TensorFlow model in Java or spark (scala) through the TensorFlow for Java port.

This section introduces model deploying by tensorflow serving and using spark (scala) to implement the TensorFlow models.

0. Introduction to model deploying by tensorflow serving

The necessary steps of model deploying using tensorflow serving are:

  • (1) Prepare the protobuf model file.

  • (2) Install the tensorflow serving.

  • (3) Start the tensorflow serving service.

  • (4) Send the request to the API service to obtain the prediction.

You may use the following link for testing (tf_serving, in Chinese) https://colab.research.google.com/drive/1vS5LAYJTEn-H0GDb1irzIuyRB8E3eWc8

  1. %tensorflow_version 2.x
  2. import tensorflow as tf
  3. print(tf.__version__)
  4. from tensorflow.keras import *

1. Prepare the protobuf Model File

Here we train a simple linear regression model with tf.keras and save it as protobuf file.

  1. import tensorflow as tf
  2. from tensorflow.keras import models,layers,optimizers
  4. ## Number of samples
  5. n = 800
  7. ## Generating testing dataset
  8. X = tf.random.uniform([n,2],minval=-10,maxval=10) 
  9. w0 = tf.constant([[2.0],[-1.0]])
  10. b0 = tf.constant(3.0)
  12. Y = X@w0 + b0 + tf.random.normal([n,1],
  13.     mean = 0.0,stddev= 2.0) # @ is matrix multiplication; adding Gaussian noise
  15. ## Modeling
  16. tf.keras.backend.clear_session()
  17. inputs = layers.Input(shape = (2,),name ="inputs") # Set the input name as "inputs"
  18. outputs = layers.Dense(1, name = "outputs")(inputs) # Set the output name as "outputs"
  19. linear = models.Model(inputs = inputs,outputs = outputs)
  20. linear.summary()
  22. ## Training with fit method
  23. linear.compile(optimizer="rmsprop",loss="mse",metrics=["mae"])
  24. linear.fit(X,Y,batch_size = 8,epochs = 100)  
  26. tf.print("w = ",linear.layers[1].kernel)
  27. tf.print("b = ",linear.layers[1].bias)
  29. ## Save the model as pb format
  30. export_path = "../data/linear_model/"
  31. version = "1"       # Version could be used for management of further updates
  32. linear.save(export_path+version, save_format="tf")
  1. # Check the saved model file
  2. !ls {export_path+version}
  1. assets    saved_model.pb    variables
  1. # Check the info of the model file
  2. !saved_model_cli show --dir {export_path+str(version)} --all
  1. MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
  3. signature_def['__saved_model_init_op']:
  4.   The given SavedModel SignatureDef contains the following input(s):
  5.   The given SavedModel SignatureDef contains the following output(s):
  6.     outputs['__saved_model_init_op'] tensor_info:
  7.         dtype: DT_INVALID
  8.         shape: unknown_rank
  9.         name: NoOp
  10.   Method name is: 
  12. signature_def['serving_default']:
  13.   The given SavedModel SignatureDef contains the following input(s):
  14.     inputs['inputs'] tensor_info:
  15.         dtype: DT_FLOAT
  16.         shape: (-1, 2)
  17.         name: serving_default_inputs:0
  18.   The given SavedModel SignatureDef contains the following output(s):
  19.     outputs['outputs'] tensor_info:
  20.         dtype: DT_FLOAT
  21.         shape: (-1, 1)
  22.         name: StatefulPartitionedCall:0
  23.   Method name is: tensorflow/serving/predict
  24. WARNING:tensorflow:From /tensorflow-2.1.0/python3.6/tensorflow_core/python/ops/resource_variable_ops.py:1786: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
  25. Instructions for updating:
  26. If using Keras pass *_constraint arguments to layers.
  28. Defined Functions:
  29.   Function Name: '__call__'
  30.     Option #1
  31.       Callable with:
  32.         Argument #1
  33.           inputs: TensorSpec(shape=(None, 2), dtype=tf.float32, name='inputs')
  34.         Argument #2
  35.           DType: bool
  36.           Value: False
  37.         Argument #3
  38.           DType: NoneType
  39.           Value: None
  40.     Option #2
  41.       Callable with:
  42.         Argument #1
  43.           inputs: TensorSpec(shape=(None, 2), dtype=tf.float32, name='inputs')
  44.         Argument #2
  45.           DType: bool
  46.           Value: True
  47.         Argument #3
  48.           DType: NoneType
  49.           Value: None
  51.   Function Name: '_default_save_signature'
  52.     Option #1
  53.       Callable with:
  54.         Argument #1
  55.           inputs: TensorSpec(shape=(None, 2), dtype=tf.float32, name='inputs')
  57.   Function Name: 'call_and_return_all_conditional_losses'
  58.     Option #1
  59.       Callable with:
  60.         Argument #1
  61.           inputs: TensorSpec(shape=(None, 2), dtype=tf.float32, name='inputs')
  62.         Argument #2
  63.           DType: bool
  64.           Value: True
  65.         Argument #3
  66.           DType: NoneType
  67.           Value: None
  68.     Option #2
  69.       Callable with:
  70.         Argument #1
  71.           inputs: TensorSpec(shape=(None, 2), dtype=tf.float32, name='inputs')
  72.         Argument #2
  73.           DType: bool
  74.           Value: False
  75.         Argument #3
  76.           DType: NoneType
  77.           Value: None

2. Installing tensorflow serving

Two methods for installing tensorflow serving: Using Docker images, or using apt.

Docker image is the simplest way of installation and we recommend it.

Docker is a container that provides independent environment for various programs.

The companies that are using TensorFlow usually use Docker to install tensorflow serving by operation experts, so the algorithm engineers don’t have to worry about the installation.

The installation of Docker on different OS are shown below (in Chinese).

Windows: https://www.runoob.com/docker/windows-docker-install.html

MacOs: https://www.runoob.com/docker/macos-docker-install.html

CentOS: https://www.runoob.com/docker/centos-docker-install.html

After successful installation of Docker, run the following command to load the tensorflow/serving image.

docker pull tensorflow/serving

3. Starting tensorflow serving Service

  1. !docker run -t --rm -p 8501:8501 \
  2.     -v "/Users/.../data/linear_model/" \
  3.     -e MODEL_NAME=linear_model \
  4.     tensorflow/serving & >server.log 2>&1

4. Sending request to the API service

The request could be sent through http function in any kind of the programming languages. We demonstrate request sending using the curl command in Linux and the requests library in Python.

  1. !curl -d '{"instances": [1.0, 2.0, 5.0]}' \
  2.     -X POST http://localhost:8501/v1/models/linear_model:predict
  1. {
  2.     "predictions": [[3.06546211], [5.01313448]
  3.     ]
  4. }
  1. import json,requests
  3. data = json.dumps({"signature_name": "serving_default", "instances": [[1.0, 2.0], [5.0,7.0]]})
  4. headers = {"content-type": "application/json"}
  5. json_response = requests.post('http://localhost:8501/v1/models/linear_model:predict', 
  6.         data=data, headers=headers)
  7. predictions = json.loads(json_response.text)["predictions"]
  8. print(predictions)
  1. [[3.06546211], [6.02843142]]

Please leave comments in the WeChat official account “Python与算法之美” (Elegance of Python and Algorithms) if you want to communicate with the author about the content. The author will try best to reply given the limited time available.

You are also welcomed to join the group chat with the other readers through replying 加群 (join group) in the WeChat official account.

image.png