Golang XORM distributed link tracking (source code analysis)
Use XORMand Opentracing, let you completely get rid of the cumbersome CRUDshadow, and shift the focus of work to business logic
environment
go version go1.14.3 windows/amd64
xorm.io/xorm 1.0.3
1. ⚠️Reminder
-
XORM version
1.0.2and above (supportsHook钩子函数intrusion into the XORM execution process) -
Must see
Q&A, seeQ&A, seeQ&A! -
In the previous Golang actual combat XORM with OpenTracing+Jaeger link monitoring, SQL execution can be seen at a glance. The method of intrusion into the log module used in the article is
1.0.2the method of intrusion below the version, but this method has concurrency problems and will be lostspan. If it is upgraded to1.0.2a version or The above should be usedHook钩子in the way of intrusion
Two, the process
1. Use Docker to start Opentracing + jaeger service
Mirror used:jaegertracing/all-in-one:1.18
Docker commands
docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14268:14268 -p 14250:14250 -p 9411:9411 jaegertracing/all-in-one:1.18
Browser access localhost:16686, you can see JaegerUIthe interface, as shown below:
So far, the service 内存as a data storage method OpenTracing+Jaegerhas successfully run.
OpentracingThe use of will not be explained in detail here , and interested students can check the official technical documentation
andjaegeruse , which is convenient for us to debug and small-scale online deployment
2. Install Xorm, OpenTracing and Jaeger
Xorm - requires 1.0.2version and above to supportHook钩子函数
go get xorm.io/xorm
OpenTracing and Jaeger - just install Jaeger-Clientand dependOpentracing
go get github.com/uber/jaeger-client-go
3. Initialize Opentracing --> usually uniformly initialized by the service
main.go
package main
func initJaeger() (closer io.Closer, err error) {
// 根据配置初始化Tracer 返回Closer
tracer, closer, err := (&config.Configuration{
ServiceName: "xormWithTracing",
Disabled: false,
Sampler: &config.SamplerConfig{
Type: jaeger.SamplerTypeConst,
// param的值在0到1之间，设置为1则将所有的Operation输出到Reporter
Param: 1,
},
Reporter: &config.ReporterConfig{
// 请注意，如果logSpans没开启就一定没有结果
LogSpans: true,
// 请一定要注意，LocalAgentHostPort填错了就一定没有结果
LocalAgentHostPort: "localhost:6831",
},
}).NewTracer()
if err != nil {
return
}
// 设置全局Tracer - 如果不设置将会导致上下文无法生成正确的Span
opentracing.SetGlobalTracer(tracer)
return
}
3. Define the Hook hook structure
custom_hook.go
package main
type TracingHook struct {
// 注意Hook伴随DB实例的生命周期，所以我们不能在Hook里面寄存span变量
// 否则就会发生并发问题
before func(c *contexts.ContextHook) (context.Context, error)
after func(c *contexts.ContextHook) error
}
// xorm的hook接口需要满足BeforeProcess和AfterProcess函数
func (h *TracingHook) BeforeProcess(c *contexts.ContextHook) (context.Context, error) {
return h.before(c)
}
func (h *TracingHook) AfterProcess(c *contexts.ContextHook) error {
return h.after(c)
}
// 让编译器知道这个是xorm的Hook，防止编译器无法检查到异常
var _ contexts.Hook = &TracingHook{
}
4. Implement the hook function
custom_hook.go
// 使用新定义类型
// context.WithValue方法中注释写到
// 提供的键需要可比性，而且不能是字符串或者任意内建类型，避免不同包之间
// 调用到相同的上下文Key发生碰撞，context的key具体类型通常为struct{}，
// 或者作为外部静态变量（即开头字母为大写）,类型应该是一个指针或者interface类型
type xormHookSpan struct{
}
var xormHookSpanKey = &xormHookSpan{
}
func before(c *contexts.ContextHook) (context.Context, error) {
// 这里一定要注意，不要拿第二个返回值作为上下文进行替换，而是用自己的key
span, _ := opentracing.StartSpanFromContext(c.Ctx, "xorm sql execute")
c.Ctx = context.WithValue(c.Ctx, xormHookSpanKey, span)
return c.Ctx, nil
}
func after(c *contexts.ContextHook) error {
// 自己实现opentracing的SpanFromContext方法，断言将interface{}转换成opentracing的span
sp, ok := c.Ctx.Value(xormHookSpanKey).(opentracing.Span)
if !ok {
// 没有则说明没有span
return nil
}
defer sp.Finish()
// 记录我们需要的内容
if c.Err != nil {
sp.LogFields(tracerLog.Object("err", c.Err))
}
// 使用xorm的builder将查询语句和参数结合，方便后期调试
sql, _ := builder.ConvertToBoundSQL(c.SQL, c.Args)
// 记录
sp.LogFields(tracerLog.String("SQL", sql))
sp.LogFields(tracerLog.Object("args", c.Args))
sp.SetTag("execute_time", c.ExecuteTime)
return nil
}
5. Mount the hook function on the hook
custom_hook.go
func NewTracingHook() *TracingHook {
return &TracingHook{
before: before,
after: after,
}
}
6. Mount the custom Hook when the xorm engine is initialized
main.go
func NewEngineForHook() (engine *xorm.Engine, err error) {
// XORM创建引擎
engine, err = xorm.NewEngine("mysql", "root:root@(localhost:3306)/test?charset=utf8mb4")
if err != nil {
return
}
// 使用我们的钩子函数 <---- 重要
engine.AddHook(NewTracingHook())
return
}
4. Unit test —> If you don’t know how to use it, please take a closer look
main_test.go
// XORM技术文档范例
type User struct {
Id int64
Name string `xorm:"varchar(25) notnull unique 'usr_name' comment('姓名')"`
}
// 新方式进行上下文注入，要求 xorm 1.0.2版本
func TestNewEngineForHook(t *testing.T) {
rand.Seed(time.Now().UnixNano())
// 初始化XORM引擎
engine, err := NewEngineForHook()
if err != nil {
t.Fatal(err)
}
// 初始化Tracer
closer, err := initJaeger()
if err != nil {
t.Fatal(err)
}
defer closer.Close()
// 生成新的Span - 注意将span结束掉，不然无法发送对应的结果
span := opentracing.StartSpan("xorm sync")
defer span.Finish()
// 把生成的Root Span写入到Context上下文，获取一个子Context
ctx := opentracing.ContextWithSpan(context.Background(), span)
// 将子上下文传入Session
session := engine.Context(ctx)
// Sync2同步表结构
if err := session.Sync2(&User{
}); err != nil {
t.Fatal(err)
}
// 插入一条数据
if _, err := session.InsertOne(&User{
Name: fmt.Sprintf("test-%d", rand.Intn(1<<10))}); err != nil {
t.Fatal()
}
}
- Looking at jaeger again, you can see that there are three records in total. Among them, the trace with three spans at the bottom is the record that we initiated the database operation actively, and the two traces above are the database operation initiated by xorm itself.
- Click to enter the trace, expand the sub-span, we can see the SQL statement and parameters executed by xorm, the execution time and the proportion
One, Q&A
1. Jaeger has no records?
- If you visit
jaegerand find that there is no service request at alljaeger, please be sure to checkopentracingwhether the initialization is successful (especially checkLocalAgentHostPortwhether the parameters are filled in correctly) - son
spannofinish
2. Why can't concurrency security be achieved through log module intrusion?
- The xorm session uses the value transfer to transfer the ContextHook in the call log, so the span cannot be stored in the context Ctx in the ContextHook, but can only be temporarily stored in the global log Logger, which also causes the span in the log instance to be concurrent. Unsafe, span will be lost under high concurrency
- The log module intrusion is still
1.0.2the best intrusion method for versions below xorm, but it is not recommended
3. Why are there some inexplicable SQL queries?
- Xorm sometimes optimizes SQL according to special circumstances. We can see that sometimes xorm will query the table structure to facilitate subsequent queries
4. What is the practical use?
- It is convenient to track after the business goes online, which is the actual use of Opentracing
6. Source code analysis (if you are not interested, you can ignore it)
1. Register hook structure
xorm -> engine.go
func (engine *Engine) AddHook(hook contexts.Hook) {
engine.db.AddHook(hook)
}
xorm -> db.go
func (db *DB) AddHook(h ...contexts.Hook) {
db.hooks.AddHook(h...)
}
xorm -> hook.go
type Hook interface {
BeforeProcess(c *ContextHook) (context.Context, error)
AfterProcess(c *ContextHook) error
}
type Hooks struct {
hooks []Hook
}
func (h *Hooks) AddHook(hooks ...Hook) {
h.hooks = append(h.hooks, hooks...)
}
We need to pass in one contexts.Hook, which is an interface type, and we only need to implement two methods to implement this interface
2. Hook context structure
xorm -> hook.go
// ContextHook represents a hook context
type ContextHook struct {
// 开始时间
start time.Time
// 上下文
Ctx context.Context
// SQL语句
SQL string // log content or SQL
// SQL参数
Args []interface{
} // if it's a SQL, it's the arguments
// 查询结果
Result sql.Result
// 执行时间
ExecuteTime time.Duration
// 如果发生错误，会赋值
Err error // SQL executed error
}
ContextHookThe structure is used as the input parameter of the Hook interface call function, including the basic data we need
3. Call the logic when executing the SQL statement
xorm -> db.go
func (db *DB) beforeProcess(c *contexts.ContextHook) (context.Context, error) {
if db.NeedLogSQL(c.Ctx) {
// <-- 重要，这里是将日志上下文转化成值传递
// 所以不能修改context.Context的内容
db.Logger.BeforeSQL(log.LogContext(*c))
}
// Hook是指针传递，所以可以修改context.Context的内容
ctx, err := db.hooks.BeforeProcess(c)
if err != nil {
return nil, err
}
return ctx, nil
}
func (db *DB) afterProcess(c *contexts.ContextHook) error {
// 和beforeProcess同理，日志上下文不能修改context.Context的内容
// 而hook可以
err := db.hooks.AfterProcess(c)
if db.NeedLogSQL(c.Ctx) {
db.Logger.AfterSQL(log.LogContext(*c))
}
return err
}
- This section is the process of calling the log and Hook in the actual SQL query process. From here, it is very obvious that the log module passes in values instead of pointers, which makes us unable to modify the context in the log module to achieve span delivery. Only the global log instance can be used to pass the span, which directly leads to concurrency security issues
- The transfer of Hook uses pointer transfer, and
contexts.ContextHookthe pointer is passed into the execution flow of the hook function, allowing us to directly operateCtx