mysql initiates get and post requests through udf

mysql initiates HTTP request

I have been studying the company's product data recently, and it is inevitable to write SQL statements. I also encountered some problems in the process. Without a comprehensive understanding of mysql , I always think that I need to write a program to obtain some desired data.

For example: our data needs to be returned through URL and stored in a specific table of the mysql database. The first reaction is to write a program to achieve it, but later we found that the development time is long, the processing efficiency is also slow, and it will be difficult for more data in the future. The requirements may not be completely universal. After carefully searching for information, I found udf and mysql_udf_http.

The most basic development language for UDF is C/C++, which is my old profession. During the process of solving the problem, I found that for those who are new to UDF and have not learned C/C++, the cost It was too high, the development cycle was long, and various environment configurations were cumbersome, so the Go language was finally chosen to achieve the purpose.

The code used in this article comes from : github (as the saying goes, don’t forget the well digger when you are in trouble).
Download the modified code : mysql_udf_http

Usage environment : official docker (mysql8.0.17)

• Configure go environment

  There is definitely no golang environment in docker , so we first need to configure the golang environment. It is actually very simple. The steps are directly posted below.

	1、下载golang官方包
		wget https://dl.google.com/go/go1.10.3.linux-amd64.tar.gz
		
	2、解压缩
		tar -C /usr/local -zxvf go1.10.3.linux-amd64.tar.gz
		
	3、配置环境变量 **vim /etc/profile** //在末尾增加以下配置（注意路径为golang安装路径）
		export GOROOT=/usr/local/go     
		export PATH=$PATH:$GOROOT/bin
		
	4、使用以下命令来保存配置
		source /etc/profile
		
	5、验证（出现正确版本则证明安装成功）
		go version

• Download source code

  git clone https://github.com/RebirthLee/mysql_udf_http_golang.git udf
  

• Steps for usage

  • Note : The author of the original article provides a shell script to generate UDF with one click. If you are interested, you can check out the link given above, but I always feel that the convenient methods are written for others. It does not work for me anyway, so I have found a simple method myself, and I will share it with you here.

  • 1. File introduction
    There is not much to explain. If one-click installation by shell script is not suitable, in fact, you only need to understand the http.go file.

  • 2. Things that need attention and modification.
    You can use the vim editor to modify http.go
    (1) Add the following two lines of code to the head. The first line is the include directory after mysql installation, which needs to be replaced with your own ; the second line It is necessary to define my_bool because neither C language nor Go language itself has my_bool type. If it is not defined, an error will be reported. Code:
    
    

    // #cgo CFLAGS: -I/usr/include/mysql -DMYSQL_DYNAMIC_PLUGIN
    // typedef char my_bool;
    

  (2) I originally thought that the function compiled in this way would be OK, but I did not expect that the data obtained using get would always be missing a few bytes at the end . After some research, I found that the data length when returning is indeed small. A few bytes, so modify the code as follows ( note : all len calculations need to be modified ).
  Before modification: After modification
  uint64(utf8.RuneCountInString(ret))
  
  :
  uint64(len(ret))
  
  (3) After modification, because there is no place to use the " unicode/utf8 " package, just delete it in the import in front of the code , otherwise a compilation error will be reported!
  (4) Compile and put the generated .so file into the plugin folder of the mysql installation directory so that mysql can successfully find the corresponding file when creating udf . The code is as follows, where /usr/lib/mysql is the installation of mysql. Directory needs to be modified according to actual conditions.
  go build -buildmode=c-shared -o /usr/lib/mysql/plugin/http.so http.go

Generate and use udf

• Create UDF

  Http Help
  	CREATE FUNCTION http_help RETURNS STRING SONAME 'http.so';
  Http Get Method
  	CREATE FUNCTION http_get RETURNS STRING SONAME 'http.so';
  Http Post Method
  	CREATE FUNCTION http_post RETURNS STRING SONAME 'http.so';
  

• Use UDF

  SELECT http_get('http://example.com');