JMESPath is a query language for JSON.
JMESPath is a JSON query language that can extract and transform elements from JSON documents. When doing an interface automation test project, the most basic step is to obtain various field values to be verified from the response, master the jmespathsyntax, and achieve twice the result with half the effort. After reading the official documents for a day, I recorded what I learned while it was hot.
Install
$ pip install jemspath
The jmespath.py library provides two interfaces:
def compile(expression):
return parser.Parser().parse(expression)
def search(expression, data, options=None):
return parser.Parser().parse(expression).search(data, options=options)
# options 可以通过创建实例来控制 jmespath 表达式的计算方式。
- compile: Similar to the re module, use the compile function to compile expressions and use the parsed expressions to perform repeated searches
- search: receive expressions and data, and return extraction results
basic expression
1. Object value
Returns a value based on the key, or returns null, or the language equivalent to null, if the key does not exist.
>>> from jmespath import search
>>> search("a", {
"a": "foo", "b": "bar", "c": "baz"})
'foo'
>>> search("d", {
"a": "foo", "b": "bar", "c": "baz"})
None
Use a subexpression to get the value, or return a null if the key does not exist.
>>> search("a.b.c", {
"a": {
"b": {
"c": "value"}}})
value
>>> search("a.b.c.d", {
"a": {
"b": {
"c": "value"}}})
None
2. List value
The index expression, starting from 0, returns null if the index is out of bounds; the index can be negative, and -1 is the last element in the list.
>>> search("[1]", ["a", "b", "c"])
"b"
>>> search("[3]", ["a", "b", "c"])
None
>>> search("[-1]", ["a", "b", "c"])
"c"
3. Object nested list
>>> search("a.b[0].c", {
"a": {
"b": [{
"c": 3}, {
"d": 4}]}})
3
4. Slicing
Same as python list slicing, [start:end:step], keep the head and pinch the tail
>>> search("[0:5]", [0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
[0, 1, 2, 3, 4]
>>> search("[5:10]", [0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
[5, 6, 7, 8, 9]
# 步长为 2，为 -1 时翻转列表
>>> search("[::2]", [0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
[0, 2, 4, 6, 8]
5. Projection
# 创建数据
data = {
"student": [
{
"name": "james", "age": 32},
{
"name": "harden", "age": 18},
{
"name": "curry", "age": 13},
{
"test": "ok"}
]
}
- list projection
>>> search("student[*].name", data)
['james', 'harden', 'curry']
Each element in student is collected into an array, which is given to the subsequent expression, which is called projection (noun from Google Translate).
Wildcard *, means to return all elements in the list. When the result expression for a single set of elements is null, the value is omitted from the collected result set .
Therefore, there is no value for in the final output {"test": "ok"}.
- slice projection
>>> search("student[:2].name", data)
['james', 'harden']
- object projection
data = {
"test": {
"funcA": {
"num": 1},
"funcB": {
"num": 2},
"funcC": {
"miss": 3},
}
}
>>> search("test.*.num", data)
[1, 2]
Object projection can be broken down into two parts, left and right:
- Create the initial array to project on the left:
evaluate(test, inputData) -> [
{"num": 1},
{"num": 2},
{"miss": 3}
]
- The right side acts on each element in the array
evaluate(num, {num: 1}) -> 1
evaluate(num, {num: 2}) -> 2
evaluate(num, {miss: 3}) -> null
The third item does not match, and the final result filters null values, so the final expression result is [1, 2].
- flattened projection
List/object projections that preserve the structure of the original document when creating projections in projections.
data = {
"a": [
{
"b": [
{
"name": "a"},
{
"name": "b"}
]
},
{
"b": [
{
"name": "c"},
{
"name": "d"}
]
}
]
}
>>> search("a[*].b[*].name", data)
[
['a', 'b'],
['c', 'd']
]
If you only need all the values under the list and don't care about the so-called hierarchical structure, then you need to obtain the result by flattening the projection.
>>> search("a[].b[].name", data)
['a', 'b', 'c', 'd']
Simply put [*] -> []to flatten the list. It flattens the child list to the parent list, not a recursive relationship, as the following example can illustrate well.
data = [
[0, 1],
2,
[3],
4,
[5, [6, 7]]
]
# 展平一层
>>> search("[]", data)
[0, 1, 2, 3, 4, 5, [6, 7]]
# 展平第二层
>>> search("[][]", data)
[0, 1, 2, 3, 4, 5, 6, 7]
- filter projection
data = {
"book": [
{
"name": "a1", "author": "aa"},
{
"name": "a2", "author": "aa"},
{
"name": "b", "author": "bb"}
]
}
# 获取 aa 写的两本书
>>> search("book[?author=='aa'].name", data)
['a1', 'a2']
Filter expressions are defined for arrays and have a general form of 左侧投影 [? <表达式> <比较器> <表达式>] 右侧投影.
Conditional expressions are supported as follows:
==, tests for equality.!=, tests for inequality.<, less than.<=, less than or equal to.>, greater than.>=, greater than or equal to.
6. Pipeline expressions
A pipeline expression <expression> | <expression>indicating that the projection must be stopped. Pass the result of the current node to the right side of the pipe symbol to continue projection.
>>> search("book[*].name | [0]", data)
"a1"
# 管道符左侧结果为 ['a1', 'a2', 'b']
# [0] 取该结果下标为 0 的数据，得到 a1
7. Multiple choice
Multiple choice list example
data = {
"people": [
{
"name": "a",
"state": {
"name": "up"}
},
{
"name": "b",
"state": {
"name": "down"}
},
{
"name": "c",
"state": {
"name": "up"}
}
]
}
>>> search("people[].[name, state.name]", data)
[
['a', 'up'],
['b', 'down'],
['c', 'up']
]
Extract the required content from the json document and simplify the structure.
[name, state.name]The expression means to create a list with two elements:
- The first element is the result of evaluating the name expression
- The second element is the result of evaluating the state.name expression
Thus, each list element creates a two-element list, and the end result of the entire expression is a list containing two-element lists.
Multiple selection object example
Same idea as a multi-select list, but a hash is created instead of an array.
>>> search("people[].{name: name, state: state.name}", data)
[
{
'name': 'a', 'state': 'up'},
{
'name': 'b', 'state': 'down'},
{
'name': 'c', 'state': 'up'}
]
8. Functions
It is more intuitive to use some functions directly through the test, since the name is self-explanatory, there is nothing to explain.
import jmespath
import pytest
class TestJmesPath:
def setup_class(self):
self.data = {
"book": [
{
"name": "平凡的世界", "author": "路遥", "sort": 3},
{
"name": "围城", "author": "钱钟书", "sort": 2},
{
"name": "围城", "author": "钱钟书", "sort": 2},
{
"name": "活着", "author": "余华", "sort": 1},
{
"name": "麦田里的守望者", "author": "塞林格", "sort": 4},
{
"name": "挪威的森林", "author": "村上春树", "sort": 5}
]
}
def test_keys(self):
"""提取对象的 key
注意：管道符前面是一个对象，所以需要指定下标，不能通过 * 号提取数组后获取 key，否则报错
jmespath.exceptions.JMESPathTypeError:
In function keys(), expected one of: ['object'], received: "array"
"""
result = jmespath.search("book[0].test", self.data)
assert result == ['name', 'author']
def test_values(self):
"""提取对象的 value，不接受数组"""
result = jmespath.search("book[0] | values(@)", self.data)
assert result == ['平凡的世界', '路遥']
def test_sort(self):
"""根据 sort 进行排序"""
result = jmespath.search("book[*].sort | sort(@)", self.data)
assert result == [1, 2, 2, 3, 4, 5]
result = jmespath.search("book[*].author | sort(@) | [join(', ', @)]", self.data)
assert result == ['余华, 塞林格, 村上春树, 路遥, 钱钟书, 钱钟书']
def test_type(self):
result = jmespath.search("book[*].sort | type(@)", self.data)
assert result == "array"
result = jmespath.search("book[0].name | type(@)", self.data)
assert result == "string"
def test_to_string(self):
result = jmespath.search('[].to_string(@)', [1, 2, 3, "number", True])
assert result == ["1", "2", "3", 'number', 'true']
def test_to_number(self):
result = jmespath.search('[].to_number(@)', ["1", "2", "3", "number", True])
assert result == [1, 2, 3]
def test_avg(self):
result = jmespath.search("avg(@)", [10, 15, 20])
assert result == 15.0
with pytest.raises(jmespath.exceptions.JMESPathTypeError):
jmespath.search("avg(@)", [10, False, 20])
def test_contains(self):
result = jmespath.search("contains(`foobar`, `foo`)", {
})
assert result is True
result = jmespath.search("contains(`foobar`, `f123`)", {
})
assert result is False
def test_join(self):
# expected one of: ['array-string']
# @ 为当前节点，得到的结果用逗号加空格分隔，然后放在当前节点下
result = jmespath.search("join(`, `, @)", ["a", "b"])
assert result == "a, b"
result = jmespath.search("join(``, @)", ["a", "b"])
assert result == "ab"
def test_length(self):
result = jmespath.search("length(@)", ["a", "b"])
assert result == 2
def test_max(self):
result = jmespath.search("max(@)", [10, 3, 5, 5, 8])
assert result == 10
def test_min(self):
result = jmespath.search("min(@)", [10, 3, 5, 5, 8])
assert result == 3
9. Custom functions
In the interface test project, there is a need to remove duplicates after obtaining the list data. I have never found the use of related expressions or functions in the official documents. I found this issue in the project, but there is no answer...
Continue to search for information. In jmespath.pythe project, find the method of user-defined functions. According to the documentation, the method of custom sorting and deduplication is implemented as follows:
from jmespath import search
from jmespath import functions
class CustomFunctions(functions.Functions):
""" https://github.com/jmespath/jmespath.py
"""
@functions.signature({
'types': ['string', "array"]})
def _func_uniq(self, arg):
if isinstance(arg, str):
# string of unique
return ''.join(sorted(set(arg)))
if isinstance(arg, list):
# array of unique
return sorted(set(arg))
options = jmespath.Options(custom_functions=CustomFunctions())
>>> search("foo.bar | uniq(@)", {
'foo': {
'bar': 'banana'}}, options=options)
abn
>>> search("foo.bar | uniq(@)", {
'foo': {
'bar': [5, 5, 2, 1]}}, options=options)
[1, 2, 5]
After mastering the above basic grammar, it is very easy to look back at the official multi-extraction expression examples. Using them together can meet most of the data extraction needs in the project. Special needs can also be realized through custom functions, which is very easy to use.