嵌入式深度学习框架之Tensorflow(四)Tensorflow-Lite运行时环境搭建

参考文档

http://www.cnblogs.com/jojodru/p/7744630.html 在Ubuntu 16.04上使用bazel交叉编译tensorflow
https://docs.bazel.build/versions/master/install-ubuntu.html Ubuntu安装Bazel文档
https://github.com/bazelbuild/bazel/wiki/Building-with-a-custom-toolchain Building with a custom toolchain

本文档涉及到的目标硬件为全志H8(8核Cortex-A7，Armv7架构)，但是对其他Arm芯片也有一定的借鉴意义，只需要更换交叉编译链即可。

本部分详细介绍了Tensorflow针对Arm-Linux操作系统的源码编译，主要内容是参考《在Ubuntu 16.04上使用bazel交叉编译tensorflow》而来。本文档中介绍的编译方法是使用X86_64机器的Ubuntu环境下进行Tensorflow ARM版本的交叉编译，和网上能查到的大部分(例如参考文档3)直接在树莓派3上编译Tensorflow的方式不同。直接在树莓派3(或其他ARM开发板)上编译会避免很多编译错误，但是编译时间非常长(长达数天)，并且由于编译过程中需要消耗大量内存所以还需要额外用TF卡映射交换空间才能保证编译过程不会因为缺少内存而崩溃。另外直接在树莓派3(或其他ARM开发板)上编译Tensorflow会导致编译出来的文件的适用范围比较小，如果你需要评估多款不同ARM开发板，那么你将花费多余的时间针对每块开发板都做一次编译。所以更好的解决方式是通过bazel进行交叉编译。

开发环境介绍

主机操作系统：Ubuntu14.04 64位
目标平台：全志H8 CQA83t
交叉工具链：gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux
Tensorflow版本：1.7.0

下载Tensorflow

git clone https://github.com/tensorflow/tensorflow.git

或者直接从github上下载稳定版本，并解压源代码。

安装Bazel

参考Bazel官方安装文档的建议，推荐使用apt方式安装Bazel。

安装jdk8

对于Ubuntu16.04：

sudo apt-get install openjdk-8-jdk

对于Ubuntu14.04则需要先添加PPA源再进行安装：

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update && sudo apt-get install oracle-java8-installer

添加Bazel安装源

sudo apt-get install curl
echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key add -

安装Bazel

sudo apt-get update
sudo apt-get install bazel

如果安装过程中出现连接googleapi网站失败的问题，可以尝试多试几次。

升级Bazel

建议安装完Bazel之后进行版本升级

sudo apt-get upgrade bazel

设置交叉编译链

# 在/etc/bash.bashrc的最后增加如下指令
# Allwinner H8 CQA83t cross compiler
export ARCH=arm
export PATH=/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/:$PATH
export CROSS_COMPILE=arm-linux-gnueabihf-
export CC=/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-gcc    
export CXX=/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-g++    
export LD=/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-ld
export AR=/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-ar
export AS=/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-as
export RANLIB=/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-ranlib
# 修改完成之后需要重启命令行才能生效
# 你可以通过如下指令来确认交叉编译链是否已经设置好
echo $CC 
# 当显示/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-gcc时表示交叉编译链
# 已经设置好
# 当需要更换为本机编译时屏蔽上面的指令即可

准备交叉编译链

下面列举一下需要新建和修改的交叉编译相关的Bazel配置脚本：

WORKSPACE
arm_compiler
- BUILD
- build_armv8.sh
- build_lite_armv8.sh
- build_armv7.sh
- build_lite_armv7.sh
- cross_toolchain_target_armv8.BUILD
- cross_toolchain_target_armv7.BUILD
- CROSSTOOL

因为全志H8是armv7架构，本文先介绍armv7相关的部分。

修改WORKSPACE

在WORKSPACE添加如下内容：

new_local_repository(
    name ='toolchain_target_armv7',
    path ='/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux',
    build_file = 'arm_compiler/cross_toolchain_target_armv7.BUILD'
)

参数说明(按照实际情况修改)：

toolchain_target_armv7：交叉编译器别名

/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux：交叉编译器路径

arm_compiler/cross_toolchain_target_armv7.BUILD：交叉编译器描述文件

新建交叉编译器描述文件

新建cross_toolchain_target_armv7.BUILD：

package(default_visibility = ['//visibility:public'])

filegroup(
  name = 'gcc',
  srcs = [
    'bin/arm-linux-gnueabihf-gcc',
  ],
)

filegroup(
  name = 'ar',
  srcs = [
    'bin/arm-linux-gnueabihf-ar',
  ],
)

filegroup(
  name = 'ld',
  srcs = [
    'bin/arm-linux-gnueabihf-ld',
  ],
)

filegroup(
  name = 'nm',
  srcs = [
    'bin/arm-linux-gnueabihf-nm',
  ],
)

filegroup(
  name = 'objcopy',
  srcs = [
    'bin/arm-linux-gnueabihf-objcopy',
  ],
)

filegroup(
  name = 'objdump',
  srcs = [
    'bin/arm-linux-gnueabihf-objdump',
  ],
)

filegroup(
  name = 'strip',
  srcs = [
    'bin/arm-linux-gnueabihf-strip',
  ],
)

filegroup(
  name = 'as',
  srcs = [
    'bin/arm-linux-gnueabihf-as',
  ],
)

filegroup(
  name = 'compiler_pieces',
  srcs = glob([
    'arm-linux-gnueabihf/**',
    'libexec/**',
    'lib/gcc/arm-linux-gnueabihf/**',
    'include/**',
  ]),
)

filegroup(
  name = 'compiler_components',
  srcs = [
    ':gcc',
    ':ar',
    ':ld',
    ':nm',
    ':objcopy',
    ':objdump',
    ':strip',
    ':as',
  ],
)

交叉编译器描述文件的具体语法参考：https://github.com/bazelbuild/bazel/wiki/Building-with-a-custom-toolchain

新建CROSSTOOL

CROSSTOOL文件负描述交叉编译器的各种编译选项和链接选项：

major_version: "local"
minor_version: ""
default_target_cpu: "armv7"

default_toolchain {
  cpu: "armv7"
  toolchain_identifier: "arm-linux-gnueabihf"
}

default_toolchain {
  cpu: "k8"
  toolchain_identifier: "local"
}

toolchain {
  abi_version: "gcc"
  abi_libc_version: "glibc_2.21"
  builtin_sysroot: ""
  compiler: "compiler"
  host_system_name: "armv7"
  needsPic: true
  supports_gold_linker: false
  supports_incremental_linker: false
  supports_fission: false
  supports_interface_shared_objects: false
  supports_normalizing_ar: true
  supports_start_end_lib: false
  supports_thin_archives: true
  target_libc: "glibc_2.21"
  target_cpu: "armv7"
  target_system_name: "armv7"
  toolchain_identifier: "arm-linux-gnueabihf"

  tool_path { name: "ar" path: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-ar" }
  tool_path { name: "compat-ld" path: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-ld" }
  tool_path { name: "cpp" path: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-cpp" }
  tool_path { name: "dwp" path: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-dwp" }
  tool_path { name: "gcc" path: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-gcc" }
  tool_path { name: "gcov" path: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-gcov" }
  tool_path { name: "ld" path: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-ld" }
  tool_path { name: "nm" path: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-nm" }
  tool_path { name: "objcopy" path: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-objcopy" }
  objcopy_embed_flag: "-I"
  objcopy_embed_flag: "binary"
  tool_path { name: "objdump" path: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-objdump" }
  tool_path { name: "strip" path: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-strip" }

  compiler_flag: "-nostdinc"
  compiler_flag: "-isystem"
  compiler_flag: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/arm-linux-gnueabihf/libc/usr/include"

  compiler_flag: "-isystem"
  compiler_flag: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/lib/gcc/arm-linux-gnueabihf/4.9.2/include"
  compiler_flag: "-isystem"
  compiler_flag: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/lib/gcc/arm-linux-gnueabihf/4.9.2/include-fixed"
  compiler_flag: "-isystem"
  compiler_flag: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/arm-linux-gnueabihf/include/c++/4.9.2"
  compiler_flag: "-isystem"
  compiler_flag: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/arm-linux-gnueabihf/include/c++/4.9.2/arm-linux-gnueabihf"

  cxx_flag: "-isystem"
  cxx_flag: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/arm-linux-gnueabihf/libc/usr/include"
  cxx_flag: "-isystem"
  cxx_flag: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/lib/gcc/arm-linux-gnueabihf/4.9.2/include"
  cxx_flag: "-isystem"
  cxx_flag: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/lib/gcc/arm-linux-gnueabihf/4.9.2/include-fixed"
  cxx_flag: "-isystem"
  cxx_flag: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/arm-linux-gnueabihf/include/c++/4.9.2"
  cxx_flag: "-isystem"
  cxx_flag: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/arm-linux-gnueabihf/include/c++/4.9.2/arm-linux-gnueabihf"
  cxx_flag: "-std=c++11"

  cxx_builtin_include_directory: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/arm-linux-gnueabihf/libc/usr/include"
  cxx_builtin_include_directory: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/lib/gcc/arm-linux-gnueabihf/4.9.2/include"
  cxx_builtin_include_directory: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/lib/gcc/arm-linux-gnueabihf/4.9.2/include-fixed"
  cxx_builtin_include_directory: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/arm-linux-gnueabihf/include/c++/4.9.2"
  cxx_builtin_include_directory: "/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/arm-linux-gnueabihf/include/c++/4.9.2/arm-linux-gnueabihf"

  linker_flag: "-lstdc++"
  linker_flag: "-L/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/lib"
  linker_flag: "-L/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/arm-linux-gnueabihf/lib"
  linker_flag: "-L/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/arm-linux-gnueabihf/libc/lib"
  linker_flag: "-L/opt/toolchain/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/arm-linux-gnueabihf/libc/usr/lib"
  linker_flag: "-Wl,--dynamic-linker=/lib/ld-linux-aarch64.so.1"

  # Anticipated future default.
  # This makes GCC and Clang do what we want when called through symlinks.
  unfiltered_cxx_flag: "-no-canonical-prefixes"
  linker_flag: "-no-canonical-prefixes"

  # Make C++ compilation deterministic. Use linkstamping instead of these
  # compiler symbols.
  unfiltered_cxx_flag: "-Wno-builtin-macro-redefined"
  unfiltered_cxx_flag: "-D__DATE__=\"redacted\""
  unfiltered_cxx_flag: "-D__TIMESTAMP__=\"redacted\""
  unfiltered_cxx_flag: "-D__TIME__=\"redacted\""

  # Security hardening on by default.
  # Conservative choice; -D_FORTIFY_SOURCE=2 may be unsafe in some cases.
  # We need to undef it before redefining it as some distributions now have
  # it enabled by default.
  compiler_flag: "-U_FORTIFY_SOURCE"
  compiler_flag: "-fstack-protector"
  compiler_flag: "-fPIE"
  linker_flag: "-pie"
  linker_flag: "-Wl,-z,relro,-z,now"

  # Enable coloring even if there's no attached terminal. Bazel removes the
  # escape sequences if --nocolor is specified.
  compiler_flag: "-fdiagnostics-color=always"

  # All warnings are enabled. Maybe enable -Werror as well?
  compiler_flag: "-Wall"
  # Enable a few more warnings that aren't part of -Wall.
  compiler_flag: "-Wunused-but-set-parameter"
  # But disable some that are problematic.
  compiler_flag: "-Wno-free-nonheap-object" # has false positives

  # Keep stack frames for debugging, even in opt mode.
  compiler_flag: "-fno-omit-frame-pointer"

  # Stamp the binary with a unique identifier.
  linker_flag: "-Wl,--build-id=md5"
  linker_flag: "-Wl,--hash-style=gnu"

  compilation_mode_flags {
    mode: DBG
    # Enable debug symbols.
    compiler_flag: "-g"
  }
  compilation_mode_flags {
    mode: OPT

    # No debug symbols.
    # Maybe we should enable https://gcc.gnu.org/wiki/DebugFission for opt or
    # even generally? However, that can't happen here, as it requires special
    # handling in Bazel.
    compiler_flag: "-g0"

    # Conservative choice for -O
    # -O3 can increase binary size and even slow down the resulting binaries.
    # Profile first and / or use FDO if you need better performance than this.
    compiler_flag: "-O3"

    # Disable assertions
    compiler_flag: "-DNDEBUG -mcpu=cortex-a7 -mfpu=neon"

    # Removal of unused code and data at link time
    # Uncomment fllowing flag when software deploy finally
    # compiler_flag: "-ffunction-sections"
    # compiler_flag: "-fdata-sections"
    # linker_flag: "-Wl,--gc-sections"
  }
}

toolchain {
  toolchain_identifier: "local"
  abi_libc_version: "local"
  abi_version: "local"
  builtin_sysroot: ""
  compiler: "compiler"
  compiler_flag: "-U_FORTIFY_SOURCE"
  compiler_flag: "-D_FORTIFY_SOURCE=2"
  compiler_flag: "-fstack-protector"
  compiler_flag: "-Wall"
  compiler_flag: "-Wl,-z,-relro,-z,now"
  compiler_flag: "-B/usr/bin"
  compiler_flag: "-B/usr/bin"
  compiler_flag: "-Wunused-but-set-parameter"
  compiler_flag: "-Wno-free-nonheap-object"
  compiler_flag: "-fno-omit-frame-pointer"
  compiler_flag: "-isystem"
  compiler_flag: "/usr/include"
  cxx_builtin_include_directory: "/usr/include/c++/4.8.5"
  cxx_builtin_include_directory: "/usr/include/c++/4.8"
  cxx_builtin_include_directory: "/usr/lib/gcc/x86_64-linux-gnu/4.8/include"
  cxx_builtin_include_directory: "/usr/include/x86_64-linux-gnu/c++/4.8"
  cxx_builtin_include_directory: "/usr/include/c++/4.8/backward"
  cxx_builtin_include_directory: "/usr/lib/gcc/x86_64-linux-gnu/4.8/include"
  cxx_builtin_include_directory: "/usr/local/include"
  cxx_builtin_include_directory: "/usr/lib/gcc/x86_64-linux-gnu/4.8.5/include-fixed"
  cxx_builtin_include_directory: "/usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed"
  cxx_builtin_include_directory: "/usr/include/x86_64-linux-gnu"
  cxx_builtin_include_directory: "/usr/include"
  cxx_flag: "-std=c++11"
  host_system_name: "local"
  linker_flag: "-lstdc++"
  linker_flag: "-lm"
  linker_flag: "-Wl,-no-as-needed"
  linker_flag: "-B/usr/bin"
  linker_flag: "-B/usr/bin"
  linker_flag: "-pass-exit-codes"
  needsPic: true
  objcopy_embed_flag: "-I"
  objcopy_embed_flag: "binary"
  supports_fission: false
  supports_gold_linker: false
  supports_incremental_linker: false
  supports_interface_shared_objects: false
  supports_normalizing_ar: false
  supports_start_end_lib: false
  supports_thin_archives: false
  target_cpu: "k8"
  target_libc: "local"
  target_system_name: "local"
  unfiltered_cxx_flag: "-fno-canonical-system-headers"
  unfiltered_cxx_flag: "-Wno-builtin-macro-redefined"
  unfiltered_cxx_flag: "-D__DATE__=\"redacted\""
  unfiltered_cxx_flag: "-D__TIMESTAMP__=\"redacted\""
  unfiltered_cxx_flag: "-D__TIME__=\"redacted\""
  tool_path {name: "ar" path: "/usr/bin/ar" }
  tool_path {name: "cpp" path: "/usr/bin/cpp" }
  tool_path {name: "dwp" path: "/usr/bin/dwp" }
  tool_path {name: "gcc" path: "/usr/bin/gcc" }
  tool_path {name: "gcov" path: "/usr/bin/gcov" }
  tool_path {name: "ld" path: "/usr/bin/ld" }
  tool_path {name: "nm" path: "/usr/bin/nm" }
  tool_path {name: "objcopy" path: "/usr/bin/objcopy" }
  tool_path {name: "objdump" path: "/usr/bin/objdump" }
  tool_path {name: "strip" path: "/usr/bin/strip" }

  compilation_mode_flags {
    mode: DBG
    compiler_flag: "-g"
  }
  compilation_mode_flags {
    mode: OPT
    compiler_flag: "-g0"
    compiler_flag: "-O3"
    compiler_flag: "-DNDEBUG"
    compiler_flag: "-ffunction-sections"
    compiler_flag: "-fdata-sections"
    linker_flag: "-Wl,--gc-sections"
  }
  linking_mode_flags { mode: DYNAMIC }
}

新建BUILD

package(default_visibility = ["//visibility:public"])

cc_toolchain_suite(
    name = "toolchain",
    toolchains = {
        "armv7|compiler": ":cc-compiler-armv7",
        "k8|compiler": ":cc-compiler-local",
    },
)

filegroup(
    name = "empty",
    srcs = [],
)

filegroup(
    name = "arm_linux_all_files",
    srcs = [
        "@toolchain_target_armv7//:compiler_pieces",
    ],
)

cc_toolchain(
    name = "cc-compiler-local",
    all_files = ":empty",
    compiler_files = ":empty",
    cpu = "local",
    dwp_files = ":empty",
    dynamic_runtime_libs = [":empty"],
    linker_files = ":empty",
    objcopy_files = ":empty",
    static_runtime_libs = [":empty"],
    strip_files = ":empty",
    supports_param_files = 1,
)

cc_toolchain(
    name = "cc-compiler-armv7",
    all_files = ":arm_linux_all_files",
    compiler_files = ":arm_linux_all_files",
    cpu = "armv7",
    dwp_files = ":empty",
    dynamic_runtime_libs = [":empty"],
    linker_files = ":arm_linux_all_files",
    objcopy_files = "arm_linux_all_files",
    static_runtime_libs = [":empty"],
    strip_files = "arm_linux_all_files",
    supports_param_files = 1,
)

添加nsync的交叉编译支持

因为nsync官方的交叉编译器可能和你自带的交叉编译器不是一致的，所以需要你改动nsync的编译设置。

问题：如何找到nsync的配置代码位置？

答案：~/.cache/bazel/_bazel_root/79bf1a12876b6ceef4e98e2416c50946/external/nsync

79bf1a12876b6ceef4e98e2416c50946是一个自动生成的随机文件夹，如果有多个文件夹的话可以通过文件夹的创建时间来判断。

修改/nsync/BUILD,因为nsync添加一个新的编译器需要添加多个位置的代码，所以分段显示需要修改的地方：

// 添加armv7 & armv8 config_setting
config_setting(
    name = "armv7",
    values = {"cpu": "armeabi-v7a"},
)

config_setting(
    name = "armv8",
    values = {"cpu": "arm64-v8a"},
)

// 
# Compilation options that apply to both C++11 and C.
NSYNC_OPTS_GENERIC = [ # select({
    # Select the CPU architecture include directory.
    # This select() has no real effect in the C++11 build, but satisfies a
    # #include that would otherwise need a #if.
    # ":gcc_linux_x86_32_1": ["-I" + pkg_path_name() + "/platform/x86_32"], 
    # ":gcc_linux_x86_64_1": ["-I" + pkg_path_name() + "/platform/x86_64"],
    # ":gcc_linux_x86_64_2": ["-I" + pkg_path_name() + "/platform/x86_64"],
    # ":gcc_linux_aarch64": ["-I" + pkg_path_name() + "/platform/aarch64"],
    # ":gcc_linux_ppc64": ["-I" + pkg_path_name() + "/platform/ppc64"],
    # ":gcc_linux_s390x": ["-I" + pkg_path_name() + "/platform/s390x"], 
    # ":clang_macos_x86_64": ["-I" + pkg_path_name() + "/platform/x86_64"], 
    # ":freebsd": ["-I" + pkg_path_name() + "/platform/x86_64"],
    # ":ios_x86_64": ["-I" + pkg_path_name() + "/platform/x86_64"],
    # ":armv8": ["-I" + pkg_path_name() + "/platform/aarch64"],
    # ":android_x86_32": ["-I" + pkg_path_name() + "/platform/x86_32"],
    # ":android_x86_64": ["-I" + pkg_path_name() + "/platform/x86_64"],   
    # ":android_armeabi": ["-I" + pkg_path_name() + "/platform/arm"],
    # ":android_arm": ["-I" + pkg_path_name() + "/platform/arm"],
    # ":android_arm64": ["-I" + pkg_path_name() + "/platform/aarch64"],
    # ":msvc_windows_x86_64": ["-I" + pkg_path_name() + "/platform/x86_64"],
# }) + [
    "-I" + pkg_path_name() + "/public",
    "-I" + pkg_path_name() + "/internal",
    "-I" + pkg_path_name() + "/platform/posix",
] + select({
    ":msvc_windows_x86_64": [
    ],
    ":freebsd": ["-pthread"],
    "//conditions:default": [
        "-D_POSIX_C_SOURCE=200809L",  
        "-pthread",
    ],
})  

// 添加armv7 & armv8 NSYNC_OPTS
# Options for C build, rather then C++11 build. 
NSYNC_OPTS = select({
    # Select the OS include directory.
    ":gcc_linux_x86_32_1": ["-I" + pkg_path_name() + "/platform/linux"],
    ":gcc_linux_x86_64_1": ["-I" + pkg_path_name() + "/platform/linux"],
    ":gcc_linux_x86_64_2": ["-I" + pkg_path_name() + "/platform/linux"],
    ":gcc_linux_aarch64": ["-I" + pkg_path_name() + "/platform/linux"],
    ":gcc_linux_ppc64": ["-I" + pkg_path_name() + "/platform/linux"],
    ":gcc_linux_s390x": ["-I" + pkg_path_name() + "/platform/linux"],
    ":clang_macos_x86_64": ["-I" + pkg_path_name() + "/platform/macos"],
    ":freebsd": ["-I" + pkg_path_name() + "/platform/freebsd"],
    ":ios_x86_64": ["-I" + pkg_path_name() + "/platform/macos"],
    ":android_x86_32": ["-I" + pkg_path_name() + "/platform/linux"],
    ":android_x86_64": ["-I" + pkg_path_name() + "/platform/linux"],
    ":android_armeabi": ["-I" + pkg_path_name() + "/platform/linux"],
    ":android_arm": ["-I" + pkg_path_name() + "/platform/linux"],
    ":android_arm64": ["-I" + pkg_path_name() + "/platform/linux"],
    ":armv7": ["-I" + pkg_path_name() + "/platform/linux"],
    ":armv8": ["-I" + pkg_path_name() + "/platform/linux"],
    ":msvc_windows_x86_64": ["-I" + pkg_path_name() + "/platform/win32"],
    "//conditions:default": [],
}) + select({
    # Select the compiler include directory.
    ":gcc_linux_x86_32_1": ["-I" + pkg_path_name() + "/platform/gcc"],
    ":gcc_linux_x86_64_1": ["-I" + pkg_path_name() + "/platform/gcc"],
    ":gcc_linux_x86_64_2": ["-I" + pkg_path_name() + "/platform/gcc"],
    ":gcc_linux_aarch64": ["-I" + pkg_path_name() + "/platform/gcc"],
    ":gcc_linux_ppc64": ["-I" + pkg_path_name() + "/platform/gcc"],
    ":gcc_linux_s390x": ["-I" + pkg_path_name() + "/platform/gcc"],
    ":clang_macos_x86_64": ["-I" + pkg_path_name() + "/platform/clang"],
    ":freebsd": ["-I" + pkg_path_name() + "/platform/clang"],
    ":ios_x86_64": ["-I" + pkg_path_name() + "/platform/clang"],
    ":android_x86_32": ["-I" + pkg_path_name() + "/platform/gcc"],
    ":android_x86_64": ["-I" + pkg_path_name() + "/platform/gcc"],
    ":android_armeabi": ["-I" + pkg_path_name() + "/platform/gcc"],
    ":android_arm": ["-I" + pkg_path_name() + "/platform/gcc"],
    ":android_arm64": ["-I" + pkg_path_name() + "/platform/gcc"],
    ":armv7": ["-I" + pkg_path_name() + "/platform/gcc"],
    ":armv8": ["-I" + pkg_path_name() + "/platform/gcc"],
    ":msvc_windows_x86_64": ["-I" + pkg_path_name() + "/platform/msvc"],
}) + select({
    # Apple deprecated their atomics library, yet recent versions have no
    # working version of stdatomic.h; so some recent versions need one, and
    # other versions prefer the other.  For the moment, just ignore the
    # depreaction.
    ":clang_macos_x86_64": ["-Wno-deprecated-declarations"],
    "//conditions:default": [],
}) + NSYNC_OPTS_GENERIC

// 添加armv7 & armv8 NSYNC_SRC_PLATFORM
NSYNC_SRC_PLATFORM = select({
    ":gcc_linux_x86_32_1": NSYNC_SRC_LINUX,
    ":gcc_linux_x86_64_1": NSYNC_SRC_LINUX,
    ":gcc_linux_x86_64_2": NSYNC_SRC_LINUX,
    ":gcc_linux_aarch64": NSYNC_SRC_LINUX,
    ":gcc_linux_ppc64": NSYNC_SRC_LINUX,
    ":gcc_linux_s390x": NSYNC_SRC_LINUX,
    ":clang_macos_x86_64": NSYNC_SRC_MACOS,
    ":freebsd": NSYNC_SRC_FREEBSD,
    ":ios_x86_64": NSYNC_SRC_MACOS,
    ":android_x86_32": NSYNC_SRC_ANDROID,
    ":android_x86_64": NSYNC_SRC_ANDROID,
    ":android_armeabi": NSYNC_SRC_ANDROID,
    ":android_arm": NSYNC_SRC_ANDROID,
    ":android_arm64": NSYNC_SRC_ANDROID,
    ":armv7": NSYNC_SRC_LINUX,
    ":armv8": NSYNC_SRC_LINUX,
    ":msvc_windows_x86_64": NSYNC_SRC_WINDOWS,
})

// 添加armv7 & armv8 NSYNC_TEST_SRC_PLATFORM
NSYNC_TEST_SRC_PLATFORM = select({
    ":gcc_linux_x86_32_1": NSYNC_TEST_SRC_LINUX,
    ":gcc_linux_x86_64_1": NSYNC_TEST_SRC_LINUX,
    ":gcc_linux_x86_64_2": NSYNC_TEST_SRC_LINUX,
    ":gcc_linux_aarch64": NSYNC_TEST_SRC_LINUX,
    ":gcc_linux_ppc64": NSYNC_TEST_SRC_LINUX,
    ":gcc_linux_s390x": NSYNC_TEST_SRC_LINUX,
    ":clang_macos_x86_64": NSYNC_TEST_SRC_MACOS,
    ":freebsd": NSYNC_TEST_SRC_FREEBSD,
    ":ios_x86_64": NSYNC_TEST_SRC_MACOS,
    ":android_x86_32": NSYNC_TEST_SRC_ANDROID,
    ":android_x86_64": NSYNC_TEST_SRC_ANDROID,
    ":android_armeabi": NSYNC_TEST_SRC_ANDROID,
    ":android_arm": NSYNC_TEST_SRC_ANDROID,
    ":android_arm64": NSYNC_TEST_SRC_ANDROID,
    ":armv7": NSYNC_TEST_SRC_LINUX,
    ":armv8": NSYNC_TEST_SRC_LINUX,
    ":msvc_windows_x86_64": NSYNC_TEST_SRC_WINDOWS,
})

运行configure配置脚本

cd tensorflow-1.7.0
./configure

配置过程中注意以下选项的设置：

python及其库的位置：再安装mxnet的一些python库之后默认的python库位置会出现错误，需要重新设置为/usr/local/lib/python2.7开头的路径

optimization flags：默认为-march=native，对于交叉编译而言需要进行修改，可以根据实际芯片架构进行设置，如-march=armv7-a

其他的选项都选择no即可

交叉编译Tensorflow主体部分

bazel build --copt="-fPIC" --copt="-march=armv7-a" --cxxopt="-fPIC" --cxxopt="-march=armv7-a" --verbose_failures --crosstool_top=//arm_compiler:toolchain --cpu=armv7 --config=opt tensorflow/examples/label_image/...

bazel命令可以通过–jobs选项来指定编译线程数，如果不指定则采用CPU核心数x2的线程数

编译过程中会可能会出现多种错误，下面将一一列举并解决：

(1) pcre编译错误

(2) boringssl编译错误

(3) arm版本python头文件缺失

(4) numpy头文件缺失

(5) tensorflow核心编译错误

在这里值得注意的是，上述错误跟tensorflow的版本有相当大的关系，这里只列举1.7.0版本编译会出现的问题。

完成编译之后将会生成如下目标文件：

bazel-bin/tensorflow/libtensorflow_framework.so
bazel-bin/tensorflow/examples/label_image

交叉编译Tensorflow-Lite

交叉编译过程参考tensorfow/contrib/lite/g3doc/rpi.md

在编译之前先对涉及到多线程的相关代码进行修改，Tensorflow-Lite默认的线程池数量为4：

const Eigen::ThreadPoolDevice& GetThreadPoolDevice() {
  const int thread_count = 4;
  static Eigen::ThreadPool* tp = new Eigen::ThreadPool(thread_count);
  static EigenThreadPoolWrapper* thread_pool_wrapper =
      new EigenThreadPoolWrapper(tp);
  static Eigen::ThreadPoolDevice* device =
      new Eigen::ThreadPoolDevice(thread_pool_wrapper, thread_count);
  return *device;
}

将上述代码修改为：

const Eigen::ThreadPoolDevice& GetThreadPoolDevice() {
  int thread_count = 1;
  const char *val = getenv("OMP_NUM_THREADS");
  if (val != nullptr) {
    thread_count = atoi(val);
  }
  static Eigen::ThreadPool* tp = new Eigen::ThreadPool(thread_count);
  static EigenThreadPoolWrapper* thread_pool_wrapper =
      new EigenThreadPoolWrapper(tp);
  static Eigen::ThreadPoolDevice* device =
      new Eigen::ThreadPoolDevice(thread_pool_wrapper, thread_count);
  return *device;
}

./tensorflow/contrib/lite/download_dependencies.sh
./tensorflow/contrib/lite/build_rpi_lib.sh

download_dependencies.sh只需要执行一次即可，不需要多次执行。

完成编译之后将会生成如下目标文件：

tensorflow/contrib/lite/gen/lib/rpi_armv7/libtensorflow-lite.a
tensorflow/contrib/lite/gen/bin/rpi_armv7/benchmark_model

交叉编译Tensorflow-Lite的label_image测试工具

bazel build --copt="-fPIC" --copt="-march=armv7-a" --cxxopt="-fPIC" --cxxopt="-march=armv7-a" --verbose_failures --crosstool_top=//arm_compiler:toolchain --cpu=armv7 --config=opt //tensorflow/contrib/lite/examples/label_image:label_image

本地编译Toco模型转换工具

Tensorflow-Lite为了最小化运行时环境，使用了Flatbuffer这种轻量级的数据存储格式。所以在使用Tensorflow-Lite进行模型测试前需要使用TOCO工具进行模型转换。

注意：TOCO转换工具和模型预测无关，建议运行在X86机器上

为了不和交叉编译的目录相互影响，拷贝一份tensorflow到另外的目录，并重新进行编译前的配置，并注意设置-march=native选项，其他选项和交叉编译时保持一致。

屏蔽/etc/bash.bashrc中设置的交叉编译链选项，并且重启命令行

bazel build tensorflow/contrib/lite/toco:toco

完成编译之后将会生成如下目标文件：

bazel-bin/tensorflow/contrib/lite/toco/toco

注意：toco工具编译出来之后尽量不要移动位置，推荐原地执行。

至此Tensorflow-Lite的交叉编译过程已经完成，之后将会有一篇ARM端模型部署&测试的文章。