Json格式化输出工具jq使用学习

jq是一款用于处理JSON数据的命令行工具。它可以帮助你将复杂的JSON数据格式化、查询、筛选、编辑等操作，以便更好地处理和分析数据。

一、安装

1.1、yum 包安装

$ yum install jq
No match for argument: jq
Error: Unable to find a match: jq

机器配置yum源中没有找到包

1.2、使用rpm包安装

社区下载包安装：

常用linux 社区包地址：https://centos.pkgs.org/

$ wget http://www6.atomicorp.com/channels/atomic/centos/7/x86_64/RPMS/jq-1.6-2.el7.x86_64.rpm

$ rpm -ivh jq-1.6-2.el7.x86_64.rpm 
warning: jq-1.6-2.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 4520afa9: NOKEY
error: Failed dependencies:
	libonig.so.5()(64bit) is needed by jq-1.6-2.el7.x86_64

安装报错，缺少依赖，安装依赖依然失败：

$ yum localinstall oniguruma-devel-6.8.2-2.el7.x86_64.rpm 
Last metadata expiration check: 0:07:38 ago on Tue 27 Jun 2023 10:15:12 AM CST.
Error: 
 Problem: conflicting requests
  - nothing provides libonig.so.5()(64bit) needed by oniguruma-devel-6.8.2-2.el7.x86_64

使用atomic-release 工具安装：

$ wget http://www6.atomicorp.com/channels/atomic/centos/7/x86_64/RPMS/atomic-release-1.0-23.el7.art.noarch.rpm

$ rpm -Uvh atomic-release-1.0-23.el7.art.noarch.rpm 
warning: atomic-release-1.0-23.el7.art.noarch.rpm: Header V4 RSA/SHA256 Signature, key ID 4520afa9: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:atomic-release-1.0-23.el7.art    ################################# [100%]

# 安装atomic-release实际上是在本地新建atomic.repo yum源文件
$ ll /etc/yum.repos.d/ | grep atomic
-rw-r--r--  1 root root  743 Jan  5  2022 atomic.repo

$ yum install jq
CentOS / Red Hat Enterprise Linux 7 - atomic                                                                                                 80 kB/s | 205 kB     00:02    
Last metadata expiration check: 0:00:00 ago on Tue 27 Jun 2023 10:28:38 AM CST.
Dependencies resolved.
============================================================================================================================================================================
 Package                                   Arch                                   Version                                      Repository                              Size
============================================================================================================================================================================
Installing:
 jq                                        x86_64                                 1.6-2.el7                                    atomic                                 167 k
Installing dependencies:
 oniguruma                                 x86_64                                 6.8.2-1.el7                                  atomic                                 181 k

Transaction Summary
============================================================================================================================================================================
Install  2 Packages

Total download size: 348 k
Installed size: 1.0 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): oniguruma-6.8.2-1.el7.x86_64.rpm                                                                                                      90 kB/s | 181 kB     00:02    
(2/2): jq-1.6-2.el7.x86_64.rpm                                                                                                               83 kB/s | 167 kB     00:02    
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                       102 kB/s | 348 kB     00:03     
warning: /var/cache/dnf/atomic-e5cca8bbd0dce12d/packages/jq-1.6-2.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 4520afa9: NOKEY
Importing GPG key 0x4520AFA9:
 Userid     : "Atomicorp (Atomicorp Official Signing Key) <[email protected]>"
 Fingerprint: 1818 66DF 9DAC A40E 5B42 9B08 FFBD 5D0A 4520 AFA9
 From       : https://www.atomicorp.com/RPM-GPG-KEY.atomicorp.txt
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                    1/1 
  Installing       : oniguruma-6.8.2-1.el7.x86_64                                                                                                                       1/2 
  Running scriptlet: oniguruma-6.8.2-1.el7.x86_64                                                                                                                       1/2 
  Installing       : jq-1.6-2.el7.x86_64                                                                                                                                2/2 
  Running scriptlet: jq-1.6-2.el7.x86_64                                                                                                                                2/2 
  Verifying        : jq-1.6-2.el7.x86_64                                                                                                                                1/2 
  Verifying        : oniguruma-6.8.2-1.el7.x86_64                                                                                                                       2/2 
New leaves:
  jq.x86_64

Installed:
  jq.x86_64 1.6-2.el7                                                              oniguruma.x86_64 6.8.2-1.el7                                                             

Complete!

二、jq的应用

2.1、格式化json数据

使用jq可以将复杂的JSON数据格式化为更加易读的格式
如{"foo": 0} 使用jq格式化后显示为：

$ echo '{"foo": 0}' | jq
{
    
    
  "foo": 0
}

对文件操作：

$ cat test.json 
{
    
    "gender":"man","name":"zhangsan","age":"10","score":{
    
    "math":"85","English":"86","Chinese":"87"}}

$ jq . test.json 
{
    
    
  "gender": "man",
  "name": "zhangsan",
  "age": "10",
  "score": {
    
    
    "math": "85",
    "English": "86",
    "Chinese": "87"
  }
}

# 也可以
$ cat test.json | jq 
{
    
    
  "gender": "man",
  "name": "zhangsan",
  "age": "10",
  "score": {
    
    
    "math": "85",
    "English": "86",
    "Chinese": "87"
  }
}

2.2、查询JSON数据

使用jq可以查询JSON数据中的某个字段或值。

$ cat test.json | jq '.age'
"10"

# 查看math
$ cat test.json | jq '.score.math'
"85"

或者使用：

$ jq '.age' test.json 
"10"

# 查看math
$ jq '.score.math' test.json
"85"

多组数据的查询,数据如下：

$ cat test.json 
[{
    
    "gender":"man","name":"zhangsan","age":10,"score":{
    
    "math":85,"English":86,"Chinese":87}},{
    
    "gender":"man","name":"lisi","age":9,"score":{
    
    "math":80,"English":81,"Chinese":82}},{
    
    "gender":"man","name":"wangwu","age":11,"score":{
    
    "math":85,"English":84,"Chinese":87}},{
    
    "gender":"female","name":"lili","age":11,"score":{
    
    "math":85,"English":88,"Chinese":90}}]

$ cat test.json | jq  .
[
  {
    
    
    "gender": "man",
    "name": "zhangsan",
    "age": 10,
    "score": {
    
    
      "math": 85,
      "English": 86,
      "Chinese": 87
    }
  },
  {
    
    
    "gender": "man",
    "name": "lisi",
    "age": 9,
    "score": {
    
    
      "math": 80,
      "English": 81,
      "Chinese": 82
    }
  },
  {
    
    
    "gender": "man",
    "name": "wangwu",
    "age": 11,
    "score": {
    
    
      "math": 85,
      "English": 84,
      "Chinese": 87
    }
  },
  {
    
    
    "gender": "female",
    "name": "lili",
    "age": 11,
    "score": {
    
    
      "math": 85,
      "English": 88,
      "Chinese": 90
    }
  }
]

$ cat test.json |  jq '.[0]'
{
    
    
  "gender": "man",
  "name": "zhangsan",
  "age": 10,
  "score": {
    
    
    "math": 85,
    "English": 86,
    "Chinese": 87
  }
}

$ cat test.json |  jq '.[1]'
{
    
    
  "gender": "man",
  "name": "lisi",
  "age": 9,
  "score": {
    
    
    "math": 80,
    "English": 81,
    "Chinese": 82
  }
}

2.3、筛选JSON数据

使用jq可以根据某些条件筛选JSON数据中的部分内容

$ cat test.json |  jq '.[] | select(.age > 10)'
{
    
    
  "gender": "man",
  "name": "wangwu",
  "age": 11,
  "score": {
    
    
    "math": 85,
    "English": 84,
    "Chinese": 87
  }
}
{
    
    
  "gender": "female",
  "name": "lili",
  "age": 11,
  "score": {
    
    
    "math": 85,
    "English": 88,
    "Chinese": 90
  }
}

$ cat test.json |  jq '.[] | select(.score.math < 85)'
{
    
    
  "gender": "man",
  "name": "lisi",
  "age": 9,
  "score": {
    
    
    "math": 80,
    "English": 81,
    "Chinese": 82
  }
}


$ cat test.json |  jq '.[] | select(.score.math < 80)'

2.4、编辑JSON数据

使用jq可以编辑JSON数据中的某些字段或值。

$ cat test.json | jq '.[0].age'
10

$ cat test.json | jq '.[0].age += 1'
[
  {
    
    
    "gender": "man",
    "name": "zhangsan",
    "age": 11,
    "score": {
    
    
      "math": 85,
      "English": 86,
      "Chinese": 87
    }
  },
  {
    
    
    "gender": "man",
    "name": "lisi",
    "age": 9,
    "score": {
    
    
      "math": 80,
      "English": 81,
      "Chinese": 82
    }
  },
  {
    
    
    "gender": "man",
    "name": "wangwu",
    "age": 11,
    "score": {
    
    
      "math": 85,
      "English": 84,
      "Chinese": 87
    }
  },
  {
    
    
    "gender": "female",
    "name": "lili",
    "age": 11,
    "score": {
    
    
      "math": 85,
      "English": 88,
      "Chinese": 90
    }
  }
]

$ cat test.json 
[{
    
    "gender":"man","name":"zhangsan","age":10,"score":{
    
    "math":85,"English":86,"Chinese":87}},{
    
    "gender":"man","name":"lisi","age":9,"score":{
    
    "math":80,"English":81,"Chinese":82}},{
    
    "gender":"man","name":"wangwu","age":11,"score":{
    
    "math":85,"English":84,"Chinese":87}},{
    
    "gender":"female","name":"lili","age":11,"score":{
    
    "math":85,"English":88,"Chinese":90}}]

需要注意的是，原文件并没有被需改

三、其他用法

3.1、管道符 |

使用|运算符，我们可以结合两个过滤器。它的工作原理与Unix系统管道符类似。左边的过滤器的输出传递到右边的过滤器。

3.2、keys函数

keys函数获取JSON对象中的所有键（key）。keys函数返回一个列表，其中包含JSON对象中所有键的名称，多组数据获取到的是数据索引，如下：

$ cat test.json | jq '. | keys'
[
  0,
  1,
  2,
  3
]

获取某一组数据keys

$ cat test.json | jq '.[0] | keys'
[
  "age",
  "gender",
  "name",
  "score"
]

$ cat test.json | jq '.[0].score | keys'
[
  "Chinese",
  "English",
  "math"
]

3.3、length函数

返回一个数组或字符串的长度。

$ cat test.json | jq '. | length'
4

$ cat test.json | jq '.[0] | length'
4

$ cat test.json | jq '.[0].score | length'
3

$ cat test.json | jq '.[0].name | length'
8

3.4、map函数

对一个数组中的每个元素执行一个函数，并返回一个新的数组。

$ cat test.json | jq '. | map(keys)'
[
  [
    "age",
    "gender",
    "name",
    "score"
  ],
  [
    "age",
    "gender",
    "name",
    "score"
  ],
  [
    "age",
    "gender",
    "name",
    "score"
  ],
  [
    "age",
    "gender",
    "name",
    "score"
  ]
]

$ cat test.json | jq '. | map(length)'
[
  4,
  4,
  4,
  4
]

3.5、select函数

对一个数组中的每个元素进行筛选，并返回一个新的数组。
参考2.3

3.6、if-then-else语句

if-then-else语句来根据条件执行不同的操作。语法如下：

if CONDITION then EXPRESSION1 else EXPRESSION2 end

CONDITION是一个条件表达式，可以是一个比较、逻辑或其他类型的表达式。如果CONDITION为真，则执行EXPRESSION1，否则执行EXPRESSION2。EXPRESSION1和EXPRESSION2可以是任何jq表达式，包括函数调用、变量引用、数学操作等。

$ cat test.json | jq '.[0].name= if .[0].age >5 then "first" else "second" end'
[
  {
    
    
    "gender": "man",
    "name": "first",
    "age": 10,
    "score": {
    
    
      "math": 85,
      "English": 86,
      "Chinese": 87
    }
  },
  {
    
    
    "gender": "man",
    "name": "lisi",
    "age": 9,
    "score": {
    
    
      "math": 80,
      "English": 81,
      "Chinese": 82
    }
  },
  {
    
    
    "gender": "man",
    "name": "wangwu",
    "age": 11,
    "score": {
    
    
      "math": 85,
      "English": 84,
      "Chinese": 87
    }
  },
  {
    
    
    "gender": "female",
    "name": "lili",
    "age": 11,
    "score": {
    
    
      "math": 85,
      "English": 88,
      "Chinese": 90
    }
  }
]

3.7、join函数

将一个数组中的所有元素拼接成一个字符串。oin函数只能用于字符串数组，不能用于其他类型的数组。
test01.json

{
    
    
  "name": "Alice",
  "hobbies": ["reading", "singing", "dancing"]
}

cat test01.json | jq '.hobbies | join(",")'
"reading,singing,dancing"

3.8、split函数

将一个字符串分割成多个子字符串，并将它们存储在一个数组中。

$ cat test.json | jq '.[0].name | split("")'
[
  "z",
  "h",
  "a",
  "n",
  "g",
  "s",
  "a",
  "n"
]

$ cat test.json | jq '.[0].name | split("n")'
[
  "zha",
  "gsa",
  ""
]

3.9、max和min函数

返回一个数组或一组数字中的最大值或最小值

$ cat test.json | jq '. | map(.score.math) | max'
85

$ cat test.json | jq '. | map(.score.math) | min'
80

$ cat test.json | jq '. | map(.age) | max'
11

3.10、flatten函数

将多维数组转换为一维数组

test02.json

$ cat test02.json 
[
  [1, 2, 3],
  [4, 5, 6],
  [7, 8, 9]
]

$ cat test02.json | jq '. | flatten'
[
  1,
  2,
  3,
  4,
  5,
  6,
  7,
  8,
  9
]

3.11、sort函数

对一个数组中的元素进行排序

$ cat test.json | jq '. | map(.age) | sort'
[
  9,
  10,
  11,
  11
]

$ cat test.json | jq '. | map(.score.Chinese) | sort'
[
  82,
  87,
  87,
  90
]

3.12、group_by函数

根据一个键对一个数组中的元素进行分组

# 依据age分组
$ cat test.json | jq 'group_by(.age)'
[
  [
    {
    
    
      "gender": "man",
      "name": "lisi",
      "age": 9,
      "score": {
    
    
        "math": 80,
        "English": 81,
        "Chinese": 82
      }
    }
  ],
  [
    {
    
    
      "gender": "man",
      "name": "zhangsan",
      "age": 10,
      "score": {
    
    
        "math": 85,
        "English": 86,
        "Chinese": 87
      }
    }
  ],
  [
    {
    
    
      "gender": "man",
      "name": "wangwu",
      "age": 11,
      "score": {
    
    
        "math": 85,
        "English": 84,
        "Chinese": 87
      }
    },
    {
    
    
      "gender": "female",
      "name": "lili",
      "age": 11,
      "score": {
    
    
        "math": 85,
        "English": 88,
        "Chinese": 90
      }
    }
  ]
]

3.13、unique函数

用于删除输入数据中的重复元素

test03.json

$ cat test03.json 
[1, 2, 3, 1, 2, 4, 5, 3]

$ cat test03.json | jq '. | unique'
[
  1,
  2,
  3,
  4,
  5
]

3.14、to_entries和from_entries函数

将一个对象转换为一个包含键值对的数组，或将一个包含键值对的数组转换为一个对象。

$ cat test.json | jq '.[0] |to_entries'
[
  {
    
    
    "key": "gender",
    "value": "man"
  },
  {
    
    
    "key": "name",
    "value": "zhangsan"
  },
  {
    
    
    "key": "age",
    "value": 10
  },
  {
    
    
    "key": "score",
    "value": {
    
    
      "math": 85,
      "English": 86,
      "Chinese": 87
    }
  }
]

test04.json

$ cat test04.json 
[
  {
    
    
    "key": "gender",
    "value": "man"
  },
  {
    
    
    "key": "name",
    "value": "zhangsan"
  },
  {
    
    
    "key": "age",
    "value": 10
  },
  {
    
    
    "key": "score",
    "value": {
    
    
      "math": 85,
      "English": 86,
      "Chinese": 87
    }
  }
]

$ cat test04.json | jq '. |from_entries'
{
    
    
  "gender": "man",
  "name": "zhangsan",
  "age": 10,
  "score": {
    
    
    "math": 85,
    "English": 86,
    "Chinese": 87
  }
}

3.15、contains函数

检查一个数组中是否包含一个特定的元素

$ cat test.json | jq '.[0].name | contains("z")'
true

$ cat test.json | jq '.[0].name | contains("l")'
false

3.16、startswith和endswith函数

检查一个字符串是否以特定的前缀或后缀开头或结束

$ cat test.json | jq '.[0].name | startswith("zh")'
true

$ cat test.json | jq '.[0].name | startswith("li")'
false

$ cat test.json | jq '.[0].name | endswith("san")'
true

$ cat test.json | jq '.[0].name | endswith("hua")'
false

参考文档

1、https://mp.weixin.qq.com/s/U1VwJGFVkrUyv3jMegapCg

2、https://mp.weixin.qq.com/s/45-ztTx2scbNY5u7NQzeIA

3、https://www.cnblogs.com/FunTester/p/14713957.html

4、https://blog.csdn.net/w727655308/article/details/121207345