Golang rule engine principle and actual combat

This article mainly introduces the use of rule engine in golang. It will first introduce the main rule engine framework in golang, and then use golang's native parser to build a simple rule engine to implement basic bool expression parsing.

 background

With the continuous iteration of business code, more and more if-else are born, and the logic in if-else becomes more and more complex, resulting in defects such as complex code logic, poor maintainability, poor readability, and high modification risk.

The complex if-else logic actually corresponds to a rule, and the corresponding operation is executed when the corresponding rule is satisfied, that is, the condition in the if-else is a corresponding bool expression:

|--bool 表达式--|
if a == 1 && b == 2 {
   // do some business
}

A complex logic represents a corresponding rule. After expressing these rules with bool expressions, operations can be performed according to the corresponding rules, greatly reducing the application of if-else:

if 规则 {
   // do some business
}

How to parse these bool expressions is the task that the rule engine needs to complete.

Rule Engine Introduction

The rule engine is a component embedded in the application, which separates the business decision from the application code, and uses the predefined semantic module to write the business decision.

The main rule engine framework implemented by Golang language:

name	address	rule description language	scenes to be used	use complexity
YQL(Yet another-Query-Language)	https://github.com/caibirdme/yql	SQL-like	expression parsing	Low
govaluate	GitHub - Knetic/govaluate: Arbitrary expression evaluation for golang	Golang-like	expression parsing	Low
Gaval	GitHub - PaesslerAG/gval: Expression evaluation in golang	Golang-like	expression parsing	Low
Grule-Rule-Engine	https://github.com/hyperjumptech/grule-rule-engine	Custom DSL (Domain-Specific Language)	rule enforcement	middle
Gina	GitHub - bilibili/gengine	Custom DSL (Domain-Specific Language)	rule enforcement	middle
Common Expression Language	https://github.com/google/cel-go#evaluate	Class C	general expression language	middle
MOUTH	GitHub - dop251/goja: ECMAScript/JavaScript engine in pure Go	JavaScript	rule analysis	middle
GopherLua: VM and compiler for Lua in Go.	https://github.com/yuin/gopher-lua	lua	rule analysis	high

It can be seen that countless people are building rule engines one after another, but these rule engines are a bit heavy for the parsing tasks of some relatively simple logical expressions due to their powerful functions.

For example, if you want to use the rule engine to implement the following rules, for example, the above frameworks to implement parsing will consume a lot of CPU resources, and may become a performance barrier for the system in a system with a large number of requests.

if type == 1 && a = 3{
    //...
}

Therefore, a rule engine with simple, portable and better performance is needed.

Building a rule engine based on the Go parser library

Introduction to the parser library

Go's built-in parser library provides related operations of golang's underlying syntax analysis, and its related APIs are open to users, so you can directly use Go's built-in  parser library  to complete the framework of the above basic rule engine.

Use go's native parser to parse the following regular expressions (the type keyword cannot be used in the rules):

// 使用go语法表示的bool表达式，in_array为函数调用
expr := `a == "3" && b == "0" && in_array(c, []string{"900","1100"}) && d == "0"`

// 使用go parser解析上述表达式，返回结果为一颗ast
parseResult, err := parser.ParseExpr(expr)
if err != nil {
   fmt.Println(err)
   return
}

// 打印该ast
ast.Print(nil, parseResult)

The following results can be obtained (a binary tree):

0  *ast.BinaryExpr {
1  .  X: *ast.BinaryExpr {
2  .  .  X: *ast.BinaryExpr {
3  .  .  .  X: *ast.BinaryExpr {
4  .  .  .  .  X: *ast.Ident {
5  .  .  .  .  .  NamePos: 1
6  .  .  .  .  .  Name: "a"
7  .  .  .  .  }
8  .  .  .  .  OpPos: 12
9  .  .  .  .  Op: ==
10  .  .  .  .  Y: *ast.BasicLit {
11  .  .  .  .  .  ValuePos: 15
12  .  .  .  .  .  Kind: STRING
13  .  .  .  .  .  Value: "\"3\""
14  .  .  .  .  }
15  .  .  .  }
16  .  .  .  OpPos: 19
17  .  .  .  Op: &&
18  .  .  .  Y: *ast.BinaryExpr {
19  .  .  .  .  X: *ast.Ident {
20  .  .  .  .  .  NamePos: 22
21  .  .  .  .  .  Name: "b"
22  .  .  .  .  }
23  .  .  .  .  OpPos: 33
24  .  .  .  .  Op: ==
25  .  .  .  .  Y: *ast.BasicLit {
26  .  .  .  .  .  ValuePos: 36
27  .  .  .  .  .  Kind: STRING
28  .  .  .  .  .  Value: "\"0\""
29  .  .  .  .  }
30  .  .  .  }
31  .  .  }
32  .  .  OpPos: 40
33  .  .  Op: &&
34  .  .  Y: *ast.CallExpr {
35  .  .  .  Fun: *ast.Ident {
36  .  .  .  .  NamePos: 43
37  .  .  .  .  Name: "in_array"
38  .  .  .  }
39  .  .  .  Lparen: 51
40  .  .  .  Args: []ast.Expr (len = 2) {
41  .  .  .  .  0: *ast.Ident {
42  .  .  .  .  .  NamePos: 52
43  .  .  .  .  .  Name: "c"
44  .  .  .  .  }
45  .  .  .  .  1: *ast.CompositeLit {
46  .  .  .  .  .  Type: *ast.ArrayType {
47  .  .  .  .  .  .  Lbrack: 68
48  .  .  .  .  .  .  Elt: *ast.Ident {
49  .  .  .  .  .  .  .  NamePos: 70
50  .  .  .  .  .  .  .  Name: "string"
51  .  .  .  .  .  .  }
52  .  .  .  .  .  }
53  .  .  .  .  .  Lbrace: 76
54  .  .  .  .  .  Elts: []ast.Expr (len = 2) {
55  .  .  .  .  .  .  0: *ast.BasicLit {
56  .  .  .  .  .  .  .  ValuePos: 77
57  .  .  .  .  .  .  .  Kind: STRING
58  .  .  .  .  .  .  .  Value: "\"900\""
59  .  .  .  .  .  .  }
60  .  .  .  .  .  .  1: *ast.BasicLit {
61  .  .  .  .  .  .  .  ValuePos: 83
62  .  .  .  .  .  .  .  Kind: STRING
63  .  .  .  .  .  .  .  Value: "\"1100\""
64  .  .  .  .  .  .  }
65  .  .  .  .  .  }
66  .  .  .  .  .  Rbrace: 89
67  .  .  .  .  .  Incomplete: false
68  .  .  .  .  }
69  .  .  .  }
70  .  .  .  Ellipsis: 0
71  .  .  .  Rparen: 90
72  .  .  }
73  .  }
74  .  OpPos: 92
75  .  Op: &&
76  .  Y: *ast.BinaryExpr {
77  .  .  X: *ast.Ident {
78  .  .  .  NamePos: 95
79  .  .  .  Name: "d"
80  .  .  }
81  .  .  OpPos: 108
82  .  .  Op: ==
83  .  .  Y: *ast.BasicLit {
84  .  .  .  ValuePos: 111
85  .  .  .  Kind: STRING
86  .  .  .  Value: "\"0\""
87  .  .  }
88  .  }
89  }

Create a rule engine based on the parser library

It can be seen that with Golang's native parser, we only need to traverse the binary tree in postorder, and then realize a set of mapping relationship between AST and corresponding data map to realize a simple rule engine.

Among them, the main structure of the implementation code of the mapping relationship between AST and corresponding data map is as follows:

func eval(expr ast.Expr, data map[string]interface{}) interface{} {
   switch expr := expr.(type) {
   case *ast.BasicLit: // 匹配到数据
      return getlitValue(expr)
   case *ast.BinaryExpr: // 匹配到子树
      // 后序遍历
      x := eval(expr.X, data) // 左子树结果
      y := eval(expr.Y, data) // 右子树结果
      if x == nil || y == nil {
         return errors.New(fmt.Sprintf("%+v, %+v is nil", x, y))
      }
      op := expr.Op // 运算符

      // 按照不同类型执行运算
      switch x.(type) {
      case int64:
         return calculateForInt(x, y, op)
      case bool:
         return calculateForBool(x, y, op)
      case string:
         return calculateForString(x, y, op)
      case error:
         return errors.New(fmt.Sprintf("%+v %+v %+v eval failed", x, op, y))
      default:
         return errors.New(fmt.Sprintf("%+v op is not support", op))
      }
   case *ast.CallExpr: // 匹配到函数
      return calculateForFunc(expr.Fun.(*ast.Ident).Name, expr.Args, data)
   case *ast.ParenExpr: // 匹配到括号
      return eval(expr.X, data)
   case *ast.Ident: // 匹配到变量
      return data[expr.Name]
   default:
      return errors.New(fmt.Sprintf("%x type is not support", expr))
   }
}

The complete implementation code is here: gparser

performance comparison

Use the rule engine implemented based on go parser to compare the performance of other common rule engines (YQL, govaluate, gval):

BenchmarkGParser_Match-8         127189   8912     ns/op // 基于 go parser 实现的规则引擎
BenchmarkGval_Match-8            63584    18358    ns/op // gval
BenchmarkGovaluateParser_Match-8 13628    86955    ns/op // govaluate
BenchmarkYqlParser_Match-8       10364    112481   ns/op // yql

Summarize

It can be seen that the rule engine implemented by the native parser has great advantages in performance, but the disadvantage is that it needs to implement a mapping relationship between AST and the corresponding data map, and is limited by the limitations of the native parser library of go. The definition language is relatively cumbersome, which is also the reason why other rule engine frameworks are born, but it is undeniable that the performance of the rule engine based on the native parser library is still good enough, so in some relatively simple rule matching scenarios. Prioritize the use of native parser, which can achieve the effect of cost reduction and efficiency increase with the greatest efficiency.