tensorflow serving slim配置流程

发现国内tensorflow serving方面的博文非常少，包括国外也不多，主要依靠官方文档，但官方文档有些碎片化并包含一些冗余，因此整合为一篇文章方便国人。

之前在我在生产环境下用tensorflow的方式是启动一个thrift服务，load进模型之后不断处理请求，但是在面对大批量数据时有些力不从心，对机器的压力很大，因此考虑使用tensorflow serving的方式部署服务。

服务调用的模型是slim下的inception v3 图像分类模型，对模型进行了迁移学习，得到了新的分类模型，运用该模型对单张图片或者批量图片做预测或者提取特征。

本文假设你已成功安装tensorflow，并利用slim的代码，迁移学习获得了自己的inception v3模型，本文主要阐述如何将该模型部署到tensorflow serving 上。

tensorflow 版本：r.1.3 ， python版本：2.7.8 ，无GPU

Step1：安装tensorflow Serving r1.3

参考官方文档：https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/setup.md

前期准备：

安装bazel

pip install tensorflow-serving-api

pip install grpcio

安装过程：

1.git clone --recurse-submodules https://github.com/tensorflow/serving

cd serving

2. cd tensorflow

   ./configure

You have bazel 0.7.0- (@non-git) installed.
Please specify the location of python. [Default is /data0/home/wang16/local/python/bin/python]:

Found possible Python library paths:
/data0/home/wang16/local/python/lib/python2.7/site-packages/
/data0/install/caffe/python/
/data0/home/wang16/simba/trunk/src/content_analysis/
/data0/home/wang16/local/python/lib/python2.7/site-packages
Please input the desired Python library path to use. Default is [/data0/home/wang16/local/python/lib/python2.7/site-packages/]

Do you wish to build TensorFlow with jemalloc as malloc support? [Y/n]:
jemalloc as malloc support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Google Cloud Platform support? [Y/n]: n
No Google Cloud Platform support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Hadoop File System support? [Y/n]: n
No Hadoop File System support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Amazon S3 File System support? [Y/n]: n
No Amazon S3 File System support will be enabled for TensorFlow.

Do you wish to build TensorFlow with XLA JIT support? [y/N]:
No XLA JIT support will be enabled for TensorFlow.

Do you wish to build TensorFlow with GDR support? [y/N]:
No GDR support will be enabled for TensorFlow.

Do you wish to build TensorFlow with VERBS support? [y/N]:
No VERBS support will be enabled for TensorFlow.

Do you wish to build TensorFlow with OpenCL support? [y/N]:
No OpenCL support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]:
No CUDA support will be enabled for TensorFlow.

Do you wish to build TensorFlow with MPI support? [y/N]:
No MPI support will be enabled for TensorFlow.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:

Add "--config=mkl" to your bazel command to build with MKL support.
Please note that MKL on MacOS or windows is still not supported.
If you would like to use a local MKL instead of downloading, please set the environment variable "TF_MKL_ROOT" every time before build.
Configuration finished

 cd ..

3. bazel build -c opt --copt=-msse4.1 --copt=-msse4.2 --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-O3 tensorflow_serving/...(耗时较长)

这里如果是centos可能会报错：fatal error: stropts.h: No such file or directory

在/usr/include建一个空的文件。命名stropts.h就可以了。

4. bazel test -c opt tensorflow_serving/...

Step2：修改代码以支持新的模型

参考文档：https://gyang274.github.io/docker-tensorflow-serving-slim/0x02b00.slim.inception.v4.html，https://github.com/gyang274/docker-tensorflow-serving-slim

参考文档：https://www.tensorflow.org/serving/serving_inception

这部分是最难的，官方提供的示例代码很少，幸亏找到了一篇相关文档，照猫画虎，才算完成。

1. 回到 serving 主目录，目录下有如下内容

mkdir -p tf_checkpoints/slim/my_inception_v3

拷贝迁移学习训练好的模型到该目录中，拷贝完目录内容如下：

2.参照示例写模型导出代码：tensorflow_serving/example/my_inception_v3_saved_model.py

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

# http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# ==============================================================================

#!/usr/bin/env python2.7

"""Export inception v3 model given existing training checkpoints.

The model is exported as SavedModel with proper signatures that can be loaded by

standard tensorflow_model_server.

"""

import os.path

# This is a placeholder for a Google-internal import.

import tensorflow as tf

slim = tf.contrib.slim

# from preprocessing import inception_preprocessing

from nets import inception

tf.app.flags.DEFINE_string('checkpoint_dir', '/tmp/slim_inception_v3_train',

"""Directory where to read training checkpoints.""")

tf.app.flags.DEFINE_string('output_dir', '/tmp/slim_inception_v3_output',

"""Directory where to export inference model.""")

tf.app.flags.DEFINE_integer('model_version', 1,

"""Version number of the model.""")

tf.app.flags.DEFINE_integer('image_size', 299,

"""Needs to provide same value as in training.""")

FLAGS = tf.app.flags.FLAGS

NUM_CLASSES = 76

NUM_TOP_CLASSES = 5

WORKING_DIR = os.path.dirname(os.path.realpath(__file__))

def preprocess_image(image_buffer):

"""Preprocess JPEG encoded bytes to 3D float Tensor."""

# Decode the string as an RGB JPEG.

# Note that the resulting image contains an unknown height and width

# that is set dynamically by decode_jpeg. In other words, the height

# and width of image is unknown at compile-time.

image = tf.image.decode_jpeg(image_buffer, channels=3)

# After this point, all image pixels reside in [0,1)

# until the very end, when they're rescaled to (-1, 1). The various

# adjust_* ops all require this range for dtype float.

image = tf.image.convert_image_dtype(image, dtype=tf.float32)

# Crop the central region of the image with an area containing 87.5% of

# the original image.

image = tf.image.central_crop(image, central_fraction=0.875)

# Resize the image to the original height and width.

image = tf.expand_dims(image, 0)

image = tf.image.resize_bilinear(

image, [FLAGS.image_size, FLAGS.image_size], align_corners=False)

image = tf.squeeze(image, [0])

# Finally, rescale to [-1,1] instead of [0, 1)

image = tf.subtract(image, 0.5)

image = tf.multiply(image, 2.0)

return image

def export():

with tf.Graph().as_default():

# build inference model

# labels

f = open('/data0/home/wang16/package/serving/tf_checkpoints/slim/my_inception_v3/labels.txt')

lines = f.readlines()

names = {0:'background'}

for line in lines:

i,label = line.split(':')

names[int(i)]=label

names_tensor = tf.constant(

names.values()

)

names_lookup_table = tf.contrib.lookup.index_to_string_table_from_tensor(

names_tensor

)

# input transformation

serialized_tf_example = tf.placeholder(

tf.string, name='tf_example'

)

feature_configs = {

'image/encoded': tf.FixedLenFeature(

shape=[], dtype=tf.string

),

}

tf_example = tf.parse_example(

serialized_tf_example, feature_configs

)

jpegs = tf_example['image/encoded']

images = tf.map_fn(

preprocess_image, jpegs, dtype=tf.float32

)

# run inference

with slim.arg_scope(

inception.inception_v3_arg_scope()

):

# inception v3 models

logits, end_points = inception.inception_v3(

images, num_classes=NUM_CLASSES, is_training=False

)

# logits = tf.Print(logits, [logits])

probs = tf.nn.softmax(logits)

# transform output to topk result

topk_probs, topk_indices = tf.nn.top_k(

probs, NUM_TOP_CLASSES

)

topk_names = names_lookup_table.lookup(

tf.to_int64(topk_indices)

)

init_fn = slim.assign_from_checkpoint_fn(

os.path.join(FLAGS.checkpoint_dir, 'model.ckpt-100000'),

slim.get_model_variables('InceptionV3')

)

# sess config

config = tf.ConfigProto(

# device_count = {

# 'GPU': 0

# },

gpu_options={

'allow_growth': 1,

# 'per_process_gpu_memory_fraction': 0.01

},

allow_soft_placement=True,

log_device_placement=False,

)

with tf.Session(config=config) as sess:

init_fn(sess)

# init on 2017.08.05

# prelogits = sess.graph.get_tensor_by_name(

# 'InceptionV3/Logits/PreLogitsFlatten/Reshape:0'

# )

# update on 2017.10.22

# note: looks like the graphdef is updated for slim-inception-v3

print('Graph Node Tensor Name:')

for node_tensor in tf.get_default_graph().as_graph_def().node:

if str(node_tensor.name).startswith('InceptionV3/Logits'):

print str(node_tensor.name)

prelogits = sess.graph.get_tensor_by_name(

'InceptionV3/Logits/Dropout_1b/Identity:0'

)

# an optional alternative

# prelogits = end_points['PreLogitsFlatten']

# export inference model.

output_path = os.path.join(

tf.compat.as_bytes(FLAGS.output_dir),

tf.compat.as_bytes(str(FLAGS.model_version))

)

print 'Exporting trained model to', output_path

builder = tf.saved_model.builder.SavedModelBuilder(output_path)

# build the signature_def_map.

predict_inputs_tensor_info = tf.saved_model.utils.build_tensor_info(

jpegs

)

classes_output_tensor_info = tf.saved_model.utils.build_tensor_info(

topk_names

)

scores_output_tensor_info = tf.saved_model.utils.build_tensor_info(

topk_probs

)

prelogits_output_tensor_info = tf.saved_model.utils.build_tensor_info(

prelogits

)

prediction_signature = (

tf.saved_model.signature_def_utils.build_signature_def(

inputs={

'images': predict_inputs_tensor_info

},

outputs={

'classes': classes_output_tensor_info,

'scores': scores_output_tensor_info,

'prelogits': prelogits_output_tensor_info

},

method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME

)

legacy_init_op = tf.group(

tf.tables_initializer(), name='legacy_init_op'

)

builder.add_meta_graph_and_variables(

sess, [tf.saved_model.tag_constants.SERVING],

signature_def_map={

'predict_images':

prediction_signature,

},

legacy_init_op=legacy_init_op

)

builder.save()

print 'Successfully exported model to %s' % FLAGS.output_dir

def main(unused_argv=None):

export()

if __name__ == '__main__':

tf.app.run()

参照示例写模型服务代码：tensorflow_serving/example/my_inception_v3_client.py

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

# http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# ==============================================================================

#!/usr/bin/env python2.7

"""Send JPEG image to tensorflow_model_server loaded with inception model.

"""

from __future__ import print_function

import urllib2

# This is a placeholder for a Google-internal import.

from grpc.beta import implementations

import tensorflow as tf

from tensorflow_serving.apis import predict_pb2

from tensorflow_serving.apis import prediction_service_pb2

tf.app.flags.DEFINE_string(

'server', 'localhost:9000', 'PredictionService host:port'

)

tf.app.flags.DEFINE_string(

'image_url', '', 'URL to image in JPEG format'

)

FLAGS = tf.app.flags.FLAGS

def main(_):

host, port = FLAGS.server.split(':')

channel = implementations.insecure_channel(host, int(port))

stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)

# Send request

# See prediction_service.proto for gRPC request/response details.

image_bytes = urllib2.urlopen(FLAGS.image_url).read()

request = predict_pb2.PredictRequest()

request.model_spec.name = 'my_inception_v3'

request.model_spec.signature_name = 'predict_images'

request.inputs['images'].CopyFrom(

tf.contrib.util.make_tensor_proto(

image_bytes, shape=[1]

)

result = stub.Predict(request, 60.0) # 60 secs timeout

print(result)

if __name__ == '__main__':

tf.app.run()

参照参考文献修改：tensorflow_serving/example/BUILD

...

py_binary(

name = "my_inception_v3_saved_model",

srcs = [

"my_inception_v3_saved_model.py",

],

srcs_version = "PY2AND3",

deps = [

"@slim_model//:dataset_factory",

"@slim_model//:preprocessing_factory",

"@slim_model//:nets",

"@org_tensorflow//tensorflow:tensorflow_py",

],

)

py_binary(

name = "my_inception_v3_client",

srcs = [

"my_inception_v3_client.py",

],

srcs_version = "PY2AND3",

deps = [

"//tensorflow_serving/apis:predict_proto_py_pb2",

"//tensorflow_serving/apis:prediction_service_proto_py_pb2",

"@org_tensorflow//tensorflow:tensorflow_py",

],

)

...

参照参考文献修改：/tensorflow_serving/workspace.bzl

...

native.new_local_repository(

name = "slim_model",

path = "tf_models/research/slim",

build_file = "tf_models/research/slim/BUILD",

)

...

修改代码： /tf_models/research/slim/BUILD，注释掉所有 "//tensorflow",

编译代码:

bazel build -c opt --copt=-msse4.1 --copt=-msse4.2 --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-O3 tensorflow_serving/...

提取模型：

bazel-bin/tensorflow_serving/example/my_inception_v3_saved_model \
--checkpoint_dir=tf_checkpoints/slim/my_inception_v3 \
--output_dir=tf_servables/slim/my_inception_v3 \
--model_version=1 \
--image_size=299

提取后的模型目录：

Step3：部署服务

1、部署服务

bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server --model_name=my_inception_v3 --model_base_path=$PWD/tf_servables/slim/my_inception_v3 --port=9000

2、测试服务

python my_inception_v3_client.py

3、返回结果

outputs {
key: "classes"
value {
dtype: DT_STRING
tensor_shape {
dim {
size: 1
}
dim {
size: 5
}
}
string_val: "BeautifulSignt\n"
string_val: "Building\n"
string_val: "Cartoon\n"
string_val: "Society\n"
string_val: "Trip\n"
}
}
outputs {
key: "prelogits"
value {
dtype: DT_FLOAT
tensor_shape {
dim {
size: 1
}
dim {
size: 1
}
dim {
size: 1
}
dim {
size: 2048
}
}
float_val: 0.29251652956
float_val: 0.283589184284
float_val: 0.0797990858555
float_val: 0.427444338799
float_val: 0.147782608867
...
float_val: 0.784989655018
float_val: 0.0607083588839
float_val: 0.256205767393
float_val: 0.177307203412
}
}
outputs {
key: "scores"
value {
dtype: DT_FLOAT
tensor_shape {
dim {
size: 1
}
dim {
size: 5
}
}
float_val: 0.995890200138
float_val: 0.00202985620126
float_val: 0.000434114364907
float_val: 0.000334182434017
float_val: 0.00018374362844
}
}