人工智能框架数据集转换C++实现(一)：TFRecord数据集

编程语言 2018-05-20 07:19:39 阅读次数: 5

最近在研究将各种数据集转换为不同ＡI框架的自有数据，这些框架包括Caffe，ＭXNet，Tensorflow等．C++这样一个通用而又强大的语言，却让使用C++的同鞋在AI时代有一个痛点，那就是目前的ＡI框架基本都是底层用C/C++实现，但提供的接口却大部分都是python的接口，而且Python的接口封装的都特别好，MXNet还好，提供im2rec.cc这样的Ｃ/C++源码，而Caffe，尤其是Tensorflow这样的框架，想用C++来转换数据就需要花点功夫了．所以本文首先讲解Tensorflow的数据集格式转换．

１．不同框架的数据分别是怎样的？

MXNet的自有数据集：rec格式

Caffe的自有据集：Lmdb格式

Tensorflow的自有数据集:TFRecord格式

２．什么是TFRecord格式？

关于tensorflow读取数据，官网给出了三种方法：
1、供给数据：在tensorflow程序运行的每一步，让python代码来供给数据
2、从文件读取数据：建立输入管线从文件中读取数据
3、预加载数据：如果数据量不太大，可以在程序中定义常量或者变量来保存所有的数据。

而tfrecord格式是Tensorflow官方推荐的标准格式。tfrecord数据文件是一种将图像数据和标签统一存储的二进制文件，能更好的利用内存，在tensorflow中快速的复制，移动，读取，存储等。

该数据集由一个example.proto文件定义：

syntax = "proto3";


message Example{
    Features features = 1;
};
message Features{
    map<string,Feature> feature = 1;
};

// Containers to hold repeated fundamental values.
message BytesList {
  repeated bytes value = 1;
}
message FloatList {
  repeated float value = 1 [packed = true];
}
message Int64List {
  repeated int64 value = 1 [packed = true];
}

message Feature{
    oneof kind{
        BytesList bytes_list = 1;
        FloatList float_list = 2;
        Int64List int64_list = 3;
    }
};

这是一个protobuf3的格式定义，需要使用以下命令通过该文件生成头文件example.pb.h和cc文件example.pb.cc：

protoc -I=. --cpp_out=./ example.proto

３．自有数据集该准备成什么样？

此处以VOC2007数据集为检测任务的例子讲解，LFW数据集为分类任务讲解．

对于分类任务，数据集统一构建一个这样的列表,该表的构建可以参考Caffe的分类任务列表的构建(文件名和标签中间不是空格，而是\t):

/output/oldFile/1000015_10/wKgB5Fr6WwWAJb7iAAABKohu5Nw109.png   0
/output/oldFile/1000015_10/wKgB5Fr6WwWAEbg6AAABC_mxdD8880.png   0
/output/oldFile/1000015_10/wKgB5Fr6WwWAUGTdAAAA8wVERrQ677.png   0
/output/oldFile/1000015_10/wKgB5Fr6WwWAPJ-lAAABPYAoeuY242.png   0
/output/oldFile/1000015_10/wKgB5Fr6WwWARVIWAAABCK2alGs331.png   0
/output/oldFile/1000015_10/wKgB5Fr6WwWAV3R5AAAA5573dko147.png   0
/output/oldFile/1000015_10/wKgB5Fr6WwaAUjQRAAABIkYxqoY008.png   0
...
/output/oldFile/1000015_10/wKgB5Vr6YF-AALG-AAAA-qStI_Q208.png   1
/output/oldFile/1000015_10/wKgB5Vr6YGCAe1VYAAABN5fz53Y240.png   1
/output/oldFile/1000015_10/wKgB5Vr6YGCAQo7fAAABVFasXJ4223.png   1
/output/oldFile/1000015_10/wKgB5Vr6YGCAL00yAAABJdrU4U0508.png   1
/output/oldFile/1000015_10/wKgB5Vr6YGCAFjTyAAABJVgoCrU242.png   1
/output/oldFile/1000015_10/wKgB5Vr6YGCAKmMMAAABMd1_pJg240.png   1
/output/oldFile/1000015_10/wKgB5Vr6YGCAR2FqAAABFCQ7LRY651.png   1

对于VOC2007数据集，构建的列表如下(文件名和标签中间不是空格，而是\t)：

/home/test/data/VOC2007/JPEGImages/004379.jpg /home/xbx/data/VOC2007/Annotations/004379.xml
/home/test/data/VOC2007/JPEGImages/001488.jpg /home/xbx/data/VOC2007/Annotations/001488.xml
/home/test/data/VOC2007/JPEGImages/004105.jpg /home/xbx/data/VOC2007/Annotations/004105.xml
/home/test/data/VOC2007/JPEGImages/006146.jpg /home/xbx/data/VOC2007/Annotations/006146.xml
/home/test/data/VOC2007/JPEGImages/004295.jpg /home/xbx/data/VOC2007/Annotations/004295.xml
/home/test/data/VOC2007/JPEGImages/001360.jpg /home/xbx/data/VOC2007/Annotations/001360.xml
/home/test/data/VOC2007/JPEGImages/003468.jpg /home/xbx/data/VOC2007/Annotations/003468.xml
．．．

4.数据集转换的流程是怎样的？

数据列表准备好之后，就可以开始分析数据集转换的流程，大体上来说就是对于分类任务，首先初始化一个RecordWriter，然后处理列表中的数据，每一行对应一个Example，每行包含图片路径和相应的标签，使用OPENCV读取图片为Mat后，将其转换为string的格式(为什么不是char*，因为图像中可能存在\0)，保存到Example中的feature中，map名称取为image_raw，并获取图片的宽高通道数，标签等信息，也都保存到Example中的feature中，map名分别为width,height,depth,label等，最后将每行的Example序列化SerializeToString为string,调用writer_->WriteRecord写入．对于检测任务区别则在于增加了对xml文件的解析，并保存bbox信息等．

需要用到的头文件包括:

#include <fcntl.h>
#include <stdio.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>
#include <boost/foreach.hpp>
#include <boost/property_tree/ptree.hpp>
#include <boost/property_tree/xml_parser.hpp>
#include <fstream>
#include <iostream>
#include <map>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <vector>

#include "tensorflow/core/lib/core/status_test_util.h"
#include "tensorflow/core/lib/core/stringpiece.h"
#include "tensorflow/core/lib/io/record_writer.h"

#include <boost/lexical_cast.hpp>
#include "rng.hpp"
using namespace tensorflow::io;
using namespace tensorflow;

主函数的判断：

  if ((dataset_type == "object_detect") && (label_map_file.length() > 0)) {
    //检测任务，其中datalist_file是列表名，label_map_file是标签name和label的转换文件，output_dir是tfrecord需要输出的路径，output_name是tfrecord输出的文件名，samples_pre是tfrecord单个文件保存多少行，Shuffle是是否打乱
    if (!detecteddata_to_tfrecords(datalist_file, label_map_file, output_dir, output_name,
                                   samples_pre, Shuffle)) {
      printf("convert wrong!!!\n");
      return false;
    }
  } else if ((dataset_type == "classification") && (label_width > 0)) {
    //分类任务，其中datalist_file是列表名，output_dir是tfrecord需要输出的路径，output_name是tfrecord输出的文件名，samples_pre是tfrecord单个文件保存多少行，label_width是标签数目，对应单标签还是多标签，Shuffle是是否打乱
    if (!clsdata_to_tfrecords(datalist_file, output_dir, output_name, samples_pre, label_width,
                              Shuffle)) {
      printf("convert wrong!!!\n");
      return false;
    }
  } else {
    printf(
        "dataset type is not object_detect or classification, or label_width [%lu], label_map_file "
        "[%s] is wrong!!!\n",
        label_width, label_map_file.c_str());
    return false;
  }

  // Optional:  Delete all global objects allocated by libprotobuf.清理在各子函数中打开的protobuf资源
  google::protobuf::ShutdownProtobufLibrary();

对于分类任务，代码如下：

bool clsdata_to_tfrecords(string datalist_file, string output_dir, string output_name,
                          int samples_pre, size_t label_width, int Shuffle) {
  std::ifstream infile(datalist_file.c_str());
  std::string line;
  std::vector<std::pair<string, std::vector<int> > > dataset;

  //读取列表文件，并将信息保存到dataset中
  while (getline(infile, line)) {
    vector<string> tmp_str = param_split(line, "\t");
    std::string filename;
    std::vector<int> label_v;
    if (tmp_str.size() != (label_width + 1)) {
      std::cout << "line " << line << "has too many param!!!" << std::endl;
      return false;
    }
    for (size_t i = 0; i < (label_width + 1); ++i) {
      if (i == 0) {
        filename = tmp_str[0];
      } else {
        try {
          int label = boost::lexical_cast<int>(tmp_str[i]);
          label_v.push_back(label);
        } catch (boost::bad_lexical_cast& e) {
          printf("%s\n", e.what());
          return false;
        }
      }
    }
    if (filename.size() > 0) dataset.push_back(std::make_pair(filename, label_v));
  }
　　
　　//打乱数据集，该代码借用caffe中rng.hpp代码
    if (Shuffle) {
    printf("tensorflow task will be shuffled!!!");
    caffe::shuffle(dataset.begin(), dataset.end());
  }

  printf("A total of %lu images.\n", dataset.size());

  // create recordwriter
  std::unique_ptr<WritableFile> file;

  RecordWriterOptions options = RecordWriterOptions::CreateRecordWriterOptions("ZLIB");

  RecordWriter* writer_ = NULL;

  int j = 0, fidx = 0;
  size_t line_id = 0;
  for (line_id = 0; line_id < dataset.size(); ++line_id) {
    if (line_id == 0 || j > samples_pre) {
　　　　　 //如果是第一次或者单个文件的tfrecord记录达到samples_pre上限，则重新初始化一个新的RecordWriter
      if (writer_ != NULL) {
        delete writer_;
        writer_ = NULL;
      }

      char output_file[1024];
      memset(output_file, 0, 1024);

      sprintf(output_file, "%s/%s_%03d.tfrecord", output_dir.c_str(), output_name.c_str(), fidx);
      printf("create new tfrecord file: [%s] \n", output_file);

      Status s = Env::Default()->NewWritableFile((string)output_file, &file);
      if (!s.ok()) {
        printf("create write record file [%s] wrong!!!\n", output_file);
        return false;
      }

      writer_ = new RecordWriter(file.get(), options);
      j = 0;
      fidx += 1;
    }
    //读取图片
    cv::Mat image = ReadImageToCVMat(dataset[line_id].first);
    //将Mat转为string的形式
    std::string image_b = matToBytes(image);
    int height = image.rows;
    int width = image.cols;
    int depth = image.channels();

　　　　//每一条数据对应一个Example
    Example example1;
    Features* features1 = example1.mutable_features();
    ::google::protobuf::Map<string, Feature>* feature1 = features1->mutable_feature();
    Feature feature_tmp;

    feature_tmp.Clear();
    if (!bytes_feature(feature_tmp, image_b)) {
      printf("image: [%s] wrong\n", dataset[line_id].first.c_str());
      continue;
    }

    (*feature1)["image_raw"] = feature_tmp;
    feature_tmp.Clear();
    if (!int64_feature(feature_tmp, height)) {
      printf("image: [%s] , height [%d] wrong\n", dataset[line_id].first.c_str(), height);
      continue;
    }
    (*feature1)["height"] = feature_tmp;

    feature_tmp.Clear();
    if (!int64_feature(feature_tmp, width)) {
      printf("image: [%s] , width [%d] wrong\n", dataset[line_id].first.c_str(), width);
      continue;
    }

    (*feature1)["width"] = feature_tmp;

    feature_tmp.Clear();
    if (!int64_feature(feature_tmp, depth)) {
      printf("image: [%s] , depth [%d] wrong\n", dataset[line_id].first.c_str(), depth);
      continue;
    }
    (*feature1)["depth"] = feature_tmp;

    //此次默认分类数据集的label已经转化为了0,1,2,3,4,5这样的形式，否则此处需要加上name to label的转化代码
    feature_tmp.Clear();
    if (!int64_feature(feature_tmp, dataset[line_id].second)) {
      printf("image: [%s]  wrong\n", dataset[line_id].first.c_str());
      continue;
    }
    (*feature1)["label"] = feature_tmp;
    //将example序列化为string，并写入Writer_
    std::string str;
    example1.SerializeToString(&str);
    writer_->WriteRecord(str);
    ++j;

    if (line_id % 1000 == 0) {
      printf("Processed %lu files.\n", line_id);
    }
  }

  printf("Processed %lu files.\n finished", line_id);

  if (writer_ != NULL) {
    delete writer_;
    writer_ = NULL;
  }

  return true;
}

其中，matToBytes函数定义如下:

std::string matToBytes(cv::Mat image) {
  int size = image.total() * image.elemSize();
  byte* bytes = new byte[size];
  memcpy(bytes, image.data, size * sizeof(byte));

  std::string img_s(bytes, size);
  return img_s;
}

string转feature，或vector<int>转feature等定义如下:

//函数重载，使得int和vector<int>都可以转换为feature
bool int64_feature(Feature& feature, int value) {
  Int64List* i_list1 = feature.mutable_int64_list();
  i_list1->add_value(value);
  return true;
}
bool int64_feature(Feature& feature, std::vector<int> value) {
  if (value.size() < 1) {
    printf("value int64 is wrong!!!");
    return false;
  }
  Int64List* i_list1 = feature.mutable_int64_list();
  for (size_t i = 0; i < value.size(); ++i) i_list1->add_value(value[i]);
  return true;
}

bool float_feature(Feature& feature, std::vector<double> value) {
  if (value.size() < 1) {
    printf("value float is wrong!!!");
    return false;
  }
  FloatList* f_list1 = feature.mutable_float_list();
  for (size_t i = 0; i < value.size(); ++i) f_list1->add_value(value[i]);
  return true;
}

//将图像信息转换为feature
bool bytes_feature(Feature& feature, std::string value) {
  BytesList* b_list1 = feature.mutable_bytes_list();
  //图像中含有0可能会存在问题
  b_list1->add_value(value);
  return true;
}

对于检测任务，大体流程一致，列表读取代码有点差异，另外需要增加对xml文件的格式化处理，可以使用boost的xml解析，大体代码如下:

bool ReadXMLToExapmle(const string& image_file, const string& xmlfile, const int img_height,
                      const int img_width, const std::map<string, int>& name_to_label,
                      RecordWriter* writer_) {
  //图像读取
  cv::Mat image = ReadImageToCVMat(image_file);
  if (!image.data) {
    cout << "Could not open or find file " << image_file;
    return false;
  }
  //将Mat转换为string
  std::string image_b = matToBytes(image);

  Example example1;
  Features* features1 = example1.mutable_features();
  ::google::protobuf::Map<string, Feature>* feature1 = features1->mutable_feature();
  Feature feature_tmp;

  feature_tmp.Clear();
  if (!bytes_feature(feature_tmp, image_b)) {
    printf("image: [%s] wrong\n", image_file.c_str());
    return false;
    ;
  }

  (*feature1)["image/encoded"] = feature_tmp;

  ptree pt;
  read_xml(xmlfile, pt);

  // Parse annotation.
  int width = 0, height = 0, depth = 0;
  try {
    height = pt.get<int>("annotation.size.height");
    width = pt.get<int>("annotation.size.width");
    depth = pt.get<int>("annotation.size.depth");
  } catch (const ptree_error& e) {
    std::cout << "when parsing " << xmlfile << ":" << e.what() << std::endl;
    height = img_height;
    width = img_width;
    return false;
  }

  feature_tmp.Clear();
  feature_tmp.Clear();
  if (!int64_feature(feature_tmp, height)) {
    printf("xml : [%s] 's height wrong\n", xmlfile.c_str());
    return false;
  }

  (*feature1)["image/height"] = feature_tmp;

  feature_tmp.Clear();
  if (!int64_feature(feature_tmp, width)) {
    printf("xml : [%s] 's width wrong\n", xmlfile.c_str());
    return false;
  }

  (*feature1)["image/width"] = feature_tmp;

  feature_tmp.Clear();
  if (!int64_feature(feature_tmp, depth)) {
    printf("xml : [%s] 's depth wrong\n", xmlfile.c_str());
    return false;
  }

  (*feature1)["image/depth"] = feature_tmp;

  std::vector<int> v_label;
  std::vector<int> v_difficult;
  std::vector<double> v_xmin;
  std::vector<double> v_ymin;
  std::vector<double> v_xmax;
  std::vector<double> v_ymax;

  BOOST_FOREACH (ptree::value_type& v1, pt.get_child("annotation")) {
    ptree pt1 = v1.second;
    if (v1.first == "object") {
      bool difficult = false;
      ptree object = v1.second;
      BOOST_FOREACH (ptree::value_type& v2, object.get_child("")) {
        ptree pt2 = v2.second;
        if (v2.first == "name") {
          string name = pt2.data();
          if (name_to_label.find(name) == name_to_label.end()) {
            std::cout << "file : [" << xmlfile << "] Unknown name: " << name << std::endl;
            return true;
          }
          int label = name_to_label.find(name)->second;
          v_label.push_back(label);
        } else if (v2.first == "difficult") {
          difficult = pt2.data() == "1";
          v_difficult.push_back(difficult);
        } else if (v2.first == "bndbox") {
          int xmin = pt2.get("xmin", 0);
          int ymin = pt2.get("ymin", 0);
          int xmax = pt2.get("xmax", 0);
          int ymax = pt2.get("ymax", 0);

          if ((xmin > width) || (ymin > height) || (xmax > width) || (ymax > height) ||
              (xmin < 0) || (ymin < 0) || (xmax < 0) || (ymax < 0)) {
            std::cout << "bounding box exceeds image boundary." << std::endl;
            return false;
          }
          v_xmin.push_back(xmin);
          v_ymin.push_back(ymin);
          v_xmax.push_back(xmax);
          v_ymax.push_back(ymax);
        }
      }
    }
  }
  feature_tmp.Clear();
  if (!int64_feature(feature_tmp, v_label)) {
    printf("xml : [%s]'s label wrong\n", xmlfile.c_str());
    return false; 
  }       
          
  (*feature1)["image/object/bbox/label"] = feature_tmp;
  feature_tmp.Clear();
  if (!int64_feature(feature_tmp, v_difficult)) {
    printf("xml : [%s]'s difficult wrong\n", xmlfile.c_str());
    return false;
  }         
            
  (*feature1)["image/object/bbox/difficult"] = feature_tmp;
  feature_tmp.Clear();
  if (!float_feature(feature_tmp, v_xmin)) {
    printf("xml : [%s]'s v_xmin wrong\n", xmlfile.c_str());
    return false;
  }

  (*feature1)["image/object/bbox/xmin"] = feature_tmp;
  feature_tmp.Clear();
  if (!float_feature(feature_tmp, v_ymin)) {
    printf("xml : [%s]'s v_ymin wrong\n", xmlfile.c_str());
    return false;
  }

  (*feature1)["image/object/bbox/ymin"] = feature_tmp;
  feature_tmp.Clear();
  if (!float_feature(feature_tmp, v_xmax)) {
    printf("xml : [%s]'s v_xmax wrong\n", xmlfile.c_str());
    return false;
  }

  (*feature1)["image/object/bbox/xmax"] = feature_tmp;
  feature_tmp.Clear();
  if (!float_feature(feature_tmp, v_ymax)) {
    printf("xml : [%s]'s v_ymax wrong\n", xmlfile.c_str());
    return false;
  }

  (*feature1)["image/object/bbox/xmax"] = feature_tmp;
  //序列化example并写入writerrecord
  std::string str;
  example1.SerializeToString(&str);
  
  writer_->WriteRecord(str);
  return true;
}

最终编译Makefile如下：

all:
	rm -rf example.pb*
	${PROTOBUF_HOME}/bin/protoc -I=. --cpp_out=./ example.proto		
	${PROTOBUF_HOME}/bin/protoc -I=. --cpp_out=./ label.proto		
	g++ -std=c++11 -o dataset_to_tfrecord dataset_to_tfrecord.cc example.pb.cc common.cpp -I/usr/local/opencv2/include -L/usr/local/opencv2/lib -L. -lopencv_core -lopencv_highgui -lopencv_imgproc -Itensorflow的路径 -Itensorflow的路径/bazel-genfiles -I${PROTOBUF_HOME}/include -I/usr/local/include/eigen3 -L${PROTOBUF_HOME}/lib -Ltensorflow的路径/bazel-bin/tensorflow/ -lprotobuf -ltensorflow_framework -I${JSONCPP_HOME}/include -L${JSONCPP_HOME}/lib -ljson_linux-gcc-5.4.0_libmt


猜你喜欢

转载自blog.csdn.net/u010647109/article/details/80348871
今日推荐
面壁智能发布 Eurux-8x22B 开源大模型 —— 堪称「理科状元」
开源日报 | 谷歌扶持鸿蒙上位；开源Rabbit R1；Docker加持的安卓手机；微软的焦虑和野心；海尔电器把开放平台关了
中国码农的“35岁魔咒”
蘭雅 CorelDRAW 插件 2024.5.1 国际劳动节版，免费下载
Arc Browser for Windows 1.0 正式 GA
90后程序员开发视频搬运软件、不到一年获利超 700 万，结局很刑！
周排行
【转】spring中对控制反转和依赖注入的理解
tms webcore 安装和使用
java程序员进阶相关书籍
SpringMVC接受请求参数、
如何保存训练好的机器学习模型
MyEclipse、Eclipse设置项目JDK的三个地方
商超行业微信小程序开发定制一般多少钱 （行业技术人员解读）
Markdown编辑器语言——30分钟入门到到精通
Linux系统下MongoDB的简单安装与基本操作
Power Strings
每日归档
更多
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)
2024-05-02(0)
2024-05-01(4)
2024-04-30(1)
2024-04-29(40)
2024-04-28(0)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022