在Ubuntu上安装Tensorflow Serving

参考https://blog.csdn.net/u010175803/article/details/81333583

1.全局安装grpcio

sudo pip3 install grpcio

2.安装依赖库

sudo apt-get update && sudo apt-get install -y automake build-essential curl libcurl3-dev git libtool libfreetype6-dev libpng12-dev libzmq3-dev pkg-config python3-dev python3-numpy python3-pip software-properties-common swig zip zlib1g-dev

3.安装tensorflow-serving-api，

无需Bazel即可运行Python客户端：

pip3 install tensorflow-serving-api

对于Serving，可以安装二进制文件，也可以从源码安装。此处选择前者。 
TensorFlow Serving ModelServer有两个版本，即tensorflow-model-server和tensorflow-model-server-universal，其中前者针对SSE4和AVX之类的指令集进行了优化，但对老机器支持不好。前者不可行，即处理器不支持AVX指令集，则安装后者。

# 把Serving的发行URI添加为package源

# 把Serving的发行URI添加为package源
echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | sudo tee /etc/apt/sources.list.d/tensorflow-serving.list
curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | sudo apt-key add -
# 安装更新，之后即可通过tensorflow_model_server命令调用
sudo apt-get update && sudo apt-get install tensorflow-model-server

以后可以通过以下方式把ModelServer升级到指定版本：

sudo apt-get upgrade tensorflow-model-server

4.训练部署模型

从https://github.com/tensorflow/serving下载tensorflow serving的源码，复制 serving/tensorflow_serving/example 文件夹到另一处，运行 mnist_saved_model.py 获得测试用模型

python mnist_saved_model.py --training_iteration=1000 --model_version=1 ./test_model

# Copyright 2016 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================

# ! /usr/bin/env python
r"""Train and export a simple Softmax Regression TensorFlow model.

The model is from the TensorFlow "MNIST For ML Beginner" tutorial. This program
simply follows all its training instructions, and uses TensorFlow SavedModel to
export the trained model with proper signatures that can be loaded by standard
tensorflow_model_server.

Usage: mnist_saved_model.py [--training_iteration=x] [--model_version=y] \
    export_dir
"""

from __future__ import print_function

import os
import sys

# This is a placeholder for a Google-internal import.

import tensorflow as tf

import mnist_input_data

tf.app.flags.DEFINE_integer('training_iteration', 1000,
                            'number of training iterations.')
tf.app.flags.DEFINE_integer('model_version', 1, 'version number of the model.')
tf.app.flags.DEFINE_string('work_dir', './mnist_data', 'Working directory.')
FLAGS = tf.app.flags.FLAGS


def main(_):
    if len(sys.argv) < 2 or sys.argv[-1].startswith('-'):
        print('Usage: mnist_saved_model.py [--training_iteration=x] '
              '[--model_version=y] export_dir')
        sys.exit(-1)
    if FLAGS.training_iteration <= 0:
        print('Please specify a positive value for training iteration.')
        sys.exit(-1)
    if FLAGS.model_version <= 0:
        print('Please specify a positive value for version number.')
        sys.exit(-1)

    # Train model
    print('Training model...')
    mnist = mnist_input_data.read_data_sets(FLAGS.work_dir, one_hot=True)

    sess = tf.InteractiveSession()

    x = tf.placeholder('float', shape=[None, 784])
    y_ = tf.placeholder('float', shape=[None, 10])
    w = tf.Variable(tf.zeros([784, 10]))
    b = tf.Variable(tf.zeros([10]))
    sess.run(tf.global_variables_initializer())
    y = tf.nn.softmax(tf.matmul(x, w) + b, name='y')
    cross_entropy = -tf.reduce_sum(y_ * tf.log(y))
    train_step = tf.train.GradientDescentOptimizer(0.01).minimize(cross_entropy)


    for _ in range(FLAGS.training_iteration):
        batch = mnist.train.next_batch(50)
        train_step.run(feed_dict={x: batch[0], y_: batch[1]})
    correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, 'float'))
    print('training accuracy %g' % sess.run(
        accuracy, feed_dict={
            x: mnist.test.images,
            y_: mnist.test.labels
        }))
    print('Done training!')

    # Export model
    # WARNING(break-tutorial-inline-code): The following code snippet is
    # in-lined in tutorials, please update tutorial documents accordingly
    # whenever code changes.

    export_path_base = sys.argv[-1]
    export_path = os.path.join(
        tf.compat.as_bytes(export_path_base),
        tf.compat.as_bytes(str(FLAGS.model_version)))
    print('Exporting trained model to', export_path)
    builder = tf.saved_model.builder.SavedModelBuilder(export_path)
  
    # 生成 输入tensor x 和 输出tensor y的 tensor info
    tensor_info_x = tf.saved_model.utils.build_tensor_info(x)
    tensor_info_y = tf.saved_model.utils.build_tensor_info(y)

    # 生成prediction_signature
    prediction_signature = (
        tf.saved_model.signature_def_utils.build_signature_def(
            # 客户端request的时候 输入的key 要与设置的相同
            inputs={'images': tensor_info_x},
            # 获得的response 结构体中 通过 设置的key('score')字段获得结果
            outputs={'scores': tensor_info_y},
            method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME))
    
    # 定义 meta
    builder.add_meta_graph_and_variables(
        sess, [tf.saved_model.tag_constants.SERVING],
        signature_def_map={
             # 用于客户端request时 的 signature_name
            'predict_images': prediction_signature,
        },
        main_op=tf.tables_initializer(),
        # 向上兼容
        strip_default_attrs=True)

    builder.save()

    print('Done exporting!')


if __name__ == '__main__':
    tf.app.run()

最后一个参数为训练模型保存的地址

运行后 ./test_model 文件夹下目录格式

test_model/
└── 1            
    ├── saved_model.pb
    └── variables
        ├── variables.data-00000-of-00001
        └── variables.index

1是模型的model_version，save_model.pb是一个保存了graph和权重的模型文件

将文件夹 1 复制到 Tensorflow Serving的工作目录  /home/xxx/Serving/model/mnist

开启Serving服务

tensorflow_model_server --port=9000 --model_name=mnist --model_base_path=/home/xxx/Serving/model/mnist

mode_name是模型的名字，调用请求时需要保证request的 ip地址，端口，model_name 与开启的Serving服务一致。

当部署多个模型时，可以让不同模型监听不同端口。

5.调用客户端

运行例子中的 mnist_client.py 

python mnist_client.py --num_tests=100 --server=localhost:9000

request = predict_pb2.PredictRequest()
# serving 开启时定义的 model_name
request.model_spec.name = 'mnist'
# model 训练时定义的 signature_def_map
request.model_spec.signature_name = 'predict_images'
image, label = test_data_set.next_batch(1)

request.inputs['images'].CopyFrom(
            tf.contrib.util.make_tensor_proto(image[0], shape=[1, image[0].size]))
result_counter.throttle()
result_future = stub.Predict.future(request, 5.0)  # 5 seconds
result_future.add_done_callback(_create_rpc_callback(label[0], result_counter))