jqis a command-line tool for manipulating JSON data. It can help you format, query, filter, and edit complex JSON data for better data processing and analysis.
1. Installation
1.1. yum package installation
$ yum install jq
No match for argument: jq
Error: Unable to find a match: jq
No package found in the machine configuration yum source
1.2. Install using rpm package
Community download package installation:
Commonly used linux community package addresses:https://centos.pkgs.org/
$ wget http://www6.atomicorp.com/channels/atomic/centos/7/x86_64/RPMS/jq-1.6-2.el7.x86_64.rpm
$ rpm -ivh jq-1.6-2.el7.x86_64.rpm
warning: jq-1.6-2.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 4520afa9: NOKEY
error: Failed dependencies:
libonig.so.5()(64bit) is needed by jq-1.6-2.el7.x86_64
Installation error, lack of dependencies, installation dependencies still fail:
$ yum localinstall oniguruma-devel-6.8.2-2.el7.x86_64.rpm
Last metadata expiration check: 0:07:38 ago on Tue 27 Jun 2023 10:15:12 AM CST.
Error:
Problem: conflicting requests
- nothing provides libonig.so.5()(64bit) needed by oniguruma-devel-6.8.2-2.el7.x86_64
Install using the atomic-release tool:
$ wget http://www6.atomicorp.com/channels/atomic/centos/7/x86_64/RPMS/atomic-release-1.0-23.el7.art.noarch.rpm
$ rpm -Uvh atomic-release-1.0-23.el7.art.noarch.rpm
warning: atomic-release-1.0-23.el7.art.noarch.rpm: Header V4 RSA/SHA256 Signature, key ID 4520afa9: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:atomic-release-1.0-23.el7.art ################################# [100%]
# 安装atomic-release实际上是在本地新建atomic.repo yum源文件
$ ll /etc/yum.repos.d/ | grep atomic
-rw-r--r-- 1 root root 743 Jan 5 2022 atomic.repo
$ yum install jq
CentOS / Red Hat Enterprise Linux 7 - atomic 80 kB/s | 205 kB 00:02
Last metadata expiration check: 0:00:00 ago on Tue 27 Jun 2023 10:28:38 AM CST.
Dependencies resolved.
============================================================================================================================================================================
Package Arch Version Repository Size
============================================================================================================================================================================
Installing:
jq x86_64 1.6-2.el7 atomic 167 k
Installing dependencies:
oniguruma x86_64 6.8.2-1.el7 atomic 181 k
Transaction Summary
============================================================================================================================================================================
Install 2 Packages
Total download size: 348 k
Installed size: 1.0 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): oniguruma-6.8.2-1.el7.x86_64.rpm 90 kB/s | 181 kB 00:02
(2/2): jq-1.6-2.el7.x86_64.rpm 83 kB/s | 167 kB 00:02
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 102 kB/s | 348 kB 00:03
warning: /var/cache/dnf/atomic-e5cca8bbd0dce12d/packages/jq-1.6-2.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 4520afa9: NOKEY
Importing GPG key 0x4520AFA9:
Userid : "Atomicorp (Atomicorp Official Signing Key) <[email protected]>"
Fingerprint: 1818 66DF 9DAC A40E 5B42 9B08 FFBD 5D0A 4520 AFA9
From : https://www.atomicorp.com/RPM-GPG-KEY.atomicorp.txt
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : oniguruma-6.8.2-1.el7.x86_64 1/2
Running scriptlet: oniguruma-6.8.2-1.el7.x86_64 1/2
Installing : jq-1.6-2.el7.x86_64 2/2
Running scriptlet: jq-1.6-2.el7.x86_64 2/2
Verifying : jq-1.6-2.el7.x86_64 1/2
Verifying : oniguruma-6.8.2-1.el7.x86_64 2/2
New leaves:
jq.x86_64
Installed:
jq.x86_64 1.6-2.el7 oniguruma.x86_64 6.8.2-1.el7
Complete!
2. jqApplication
2.1, format json data
Use jq to format complex JSON data into a more readable format.
For example, {"foo": 0}after using jq to format, it is displayed as:
$ echo '{"foo": 0}' | jq
{
"foo": 0
}
Operation on files:
$ cat test.json
{
"gender":"man","name":"zhangsan","age":"10","score":{
"math":"85","English":"86","Chinese":"87"}}
$ jq . test.json
{
"gender": "man",
"name": "zhangsan",
"age": "10",
"score": {
"math": "85",
"English": "86",
"Chinese": "87"
}
}
# 也可以
$ cat test.json | jq
{
"gender": "man",
"name": "zhangsan",
"age": "10",
"score": {
"math": "85",
"English": "86",
"Chinese": "87"
}
}
2.2. Query JSON data
Use jq to query a certain field or value in JSON data.
$ cat test.json | jq '.age'
"10"
# 查看math
$ cat test.json | jq '.score.math'
"85"
Or use:
$ jq '.age' test.json
"10"
# 查看math
$ jq '.score.math' test.json
"85"
Query for multiple sets of data, the data is as follows:
$ cat test.json
[{
"gender":"man","name":"zhangsan","age":10,"score":{
"math":85,"English":86,"Chinese":87}},{
"gender":"man","name":"lisi","age":9,"score":{
"math":80,"English":81,"Chinese":82}},{
"gender":"man","name":"wangwu","age":11,"score":{
"math":85,"English":84,"Chinese":87}},{
"gender":"female","name":"lili","age":11,"score":{
"math":85,"English":88,"Chinese":90}}]
$ cat test.json | jq .
[
{
"gender": "man",
"name": "zhangsan",
"age": 10,
"score": {
"math": 85,
"English": 86,
"Chinese": 87
}
},
{
"gender": "man",
"name": "lisi",
"age": 9,
"score": {
"math": 80,
"English": 81,
"Chinese": 82
}
},
{
"gender": "man",
"name": "wangwu",
"age": 11,
"score": {
"math": 85,
"English": 84,
"Chinese": 87
}
},
{
"gender": "female",
"name": "lili",
"age": 11,
"score": {
"math": 85,
"English": 88,
"Chinese": 90
}
}
]
$ cat test.json | jq '.[0]'
{
"gender": "man",
"name": "zhangsan",
"age": 10,
"score": {
"math": 85,
"English": 86,
"Chinese": 87
}
}
$ cat test.json | jq '.[1]'
{
"gender": "man",
"name": "lisi",
"age": 9,
"score": {
"math": 80,
"English": 81,
"Chinese": 82
}
}
2.3. Filter JSON data
Use jq to filter some content in JSON data according to certain conditions
$ cat test.json | jq '.[] | select(.age > 10)'
{
"gender": "man",
"name": "wangwu",
"age": 11,
"score": {
"math": 85,
"English": 84,
"Chinese": 87
}
}
{
"gender": "female",
"name": "lili",
"age": 11,
"score": {
"math": 85,
"English": 88,
"Chinese": 90
}
}
$ cat test.json | jq '.[] | select(.score.math < 85)'
{
"gender": "man",
"name": "lisi",
"age": 9,
"score": {
"math": 80,
"English": 81,
"Chinese": 82
}
}
$ cat test.json | jq '.[] | select(.score.math < 80)'
2.4. Edit JSON data
Some fields or values in JSON data can be edited using jq.
$ cat test.json | jq '.[0].age'
10
$ cat test.json | jq '.[0].age += 1'
[
{
"gender": "man",
"name": "zhangsan",
"age": 11,
"score": {
"math": 85,
"English": 86,
"Chinese": 87
}
},
{
"gender": "man",
"name": "lisi",
"age": 9,
"score": {
"math": 80,
"English": 81,
"Chinese": 82
}
},
{
"gender": "man",
"name": "wangwu",
"age": 11,
"score": {
"math": 85,
"English": 84,
"Chinese": 87
}
},
{
"gender": "female",
"name": "lili",
"age": 11,
"score": {
"math": 85,
"English": 88,
"Chinese": 90
}
}
]
$ cat test.json
[{
"gender":"man","name":"zhangsan","age":10,"score":{
"math":85,"English":86,"Chinese":87}},{
"gender":"man","name":"lisi","age":9,"score":{
"math":80,"English":81,"Chinese":82}},{
"gender":"man","name":"wangwu","age":11,"score":{
"math":85,"English":84,"Chinese":87}},{
"gender":"female","name":"lili","age":11,"score":{
"math":85,"English":88,"Chinese":90}}]
It should be noted that the original file has not been changed
3. Other uses
3.1. Pipe character|
Using |operators we can combine two filters. It works similarly to the pipe character on Unix systems. The output of the filter on the left is passed to the filter on the right.
3.2. keysFunction
The keys function gets all the keys in the JSON object. The keys function returns a list containing the names of all keys in the JSON object. The data index obtained for multiple sets of data is as follows:
$ cat test.json | jq '. | keys'
[
0,
1,
2,
3
]
Get a certain set of data keys
$ cat test.json | jq '.[0] | keys'
[
"age",
"gender",
"name",
"score"
]
$ cat test.json | jq '.[0].score | keys'
[
"Chinese",
"English",
"math"
]
3.3. lengthFunction
Returns the length of an array or string.
$ cat test.json | jq '. | length'
4
$ cat test.json | jq '.[0] | length'
4
$ cat test.json | jq '.[0].score | length'
3
$ cat test.json | jq '.[0].name | length'
8
3.4. mapFunction
Execute a function on each element in an array and return a new array.
$ cat test.json | jq '. | map(keys)'
[
[
"age",
"gender",
"name",
"score"
],
[
"age",
"gender",
"name",
"score"
],
[
"age",
"gender",
"name",
"score"
],
[
"age",
"gender",
"name",
"score"
]
]
$ cat test.json | jq '. | map(length)'
[
4,
4,
4,
4
]
3.5. selectFunction
Filters each element in an array and returns a new array.
Reference 2.3
3.6. if-then-elseStatement
if-then-elseStatements to perform different actions based on conditions. The syntax is as follows:
if CONDITION then EXPRESSION1 else EXPRESSION2 end
CONDITION is a conditional expression, which can be a comparison, logical or other type of expression. If CONDITION is true, EXPRESSION1 is executed, otherwise EXPRESSION2 is executed. EXPRESSION1 and EXPRESSION2 can be any jq expression, including function calls, variable references, mathematical operations, etc.
$ cat test.json | jq '.[0].name= if .[0].age >5 then "first" else "second" end'
[
{
"gender": "man",
"name": "first",
"age": 10,
"score": {
"math": 85,
"English": 86,
"Chinese": 87
}
},
{
"gender": "man",
"name": "lisi",
"age": 9,
"score": {
"math": 80,
"English": 81,
"Chinese": 82
}
},
{
"gender": "man",
"name": "wangwu",
"age": 11,
"score": {
"math": 85,
"English": 84,
"Chinese": 87
}
},
{
"gender": "female",
"name": "lili",
"age": 11,
"score": {
"math": 85,
"English": 88,
"Chinese": 90
}
}
]
3.7. joinFunction
Concatenates all elements in an array into a string. The oin function can only be used for string arrays, not other types of arrays.
test01.json
{
"name": "Alice",
"hobbies": ["reading", "singing", "dancing"]
}
cat test01.json | jq '.hobbies | join(",")'
"reading,singing,dancing"
3.8. splitFunction
Split a string into substrings and store them in an array.
$ cat test.json | jq '.[0].name | split("")'
[
"z",
"h",
"a",
"n",
"g",
"s",
"a",
"n"
]
$ cat test.json | jq '.[0].name | split("n")'
[
"zha",
"gsa",
""
]
3.9. max和minFunction
Returns the maximum or minimum value in an array or set of numbers
$ cat test.json | jq '. | map(.score.math) | max'
85
$ cat test.json | jq '. | map(.score.math) | min'
80
$ cat test.json | jq '. | map(.age) | max'
11
3.10. flattenFunction
Convert multidimensional array to one-dimensional array
test02.json
$ cat test02.json
[
[1, 2, 3],
[4, 5, 6],
[7, 8, 9]
]
$ cat test02.json | jq '. | flatten'
[
1,
2,
3,
4,
5,
6,
7,
8,
9
]
3.11. sortFunction
Sort the elements in an array
$ cat test.json | jq '. | map(.age) | sort'
[
9,
10,
11,
11
]
$ cat test.json | jq '. | map(.score.Chinese) | sort'
[
82,
87,
87,
90
]
3.12. group_byFunction
Group elements in an array by a key
# 依据age分组
$ cat test.json | jq 'group_by(.age)'
[
[
{
"gender": "man",
"name": "lisi",
"age": 9,
"score": {
"math": 80,
"English": 81,
"Chinese": 82
}
}
],
[
{
"gender": "man",
"name": "zhangsan",
"age": 10,
"score": {
"math": 85,
"English": 86,
"Chinese": 87
}
}
],
[
{
"gender": "man",
"name": "wangwu",
"age": 11,
"score": {
"math": 85,
"English": 84,
"Chinese": 87
}
},
{
"gender": "female",
"name": "lili",
"age": 11,
"score": {
"math": 85,
"English": 88,
"Chinese": 90
}
}
]
]
3.13. uniqueFunction
Used to remove duplicate elements from input data
test03.json
$ cat test03.json
[1, 2, 3, 1, 2, 4, 5, 3]
$ cat test03.json | jq '. | unique'
[
1,
2,
3,
4,
5
]
3.14. to_entries和from_entriesFunction
Converts an object to an array of key-value pairs, or an array of key-value pairs to an object.
$ cat test.json | jq '.[0] |to_entries'
[
{
"key": "gender",
"value": "man"
},
{
"key": "name",
"value": "zhangsan"
},
{
"key": "age",
"value": 10
},
{
"key": "score",
"value": {
"math": 85,
"English": 86,
"Chinese": 87
}
}
]
test04.json
$ cat test04.json
[
{
"key": "gender",
"value": "man"
},
{
"key": "name",
"value": "zhangsan"
},
{
"key": "age",
"value": 10
},
{
"key": "score",
"value": {
"math": 85,
"English": 86,
"Chinese": 87
}
}
]
$ cat test04.json | jq '. |from_entries'
{
"gender": "man",
"name": "zhangsan",
"age": 10,
"score": {
"math": 85,
"English": 86,
"Chinese": 87
}
}
3.15. containsFunction
Check if an array contains a specific element
$ cat test.json | jq '.[0].name | contains("z")'
true
$ cat test.json | jq '.[0].name | contains("l")'
false
3.16. startswith和endswithFunction
Check if a string starts or ends with a specific prefix or suffix
$ cat test.json | jq '.[0].name | startswith("zh")'
true
$ cat test.json | jq '.[0].name | startswith("li")'
false
$ cat test.json | jq '.[0].name | endswith("san")'
true
$ cat test.json | jq '.[0].name | endswith("hua")'
false
reference documents
1、https://mp.weixin.qq.com/s/U1VwJGFVkrUyv3jMegapCg
2、https://mp.weixin.qq.com/s/45-ztTx2scbNY5u7NQzeIA
3、https://www.cnblogs.com/FunTester/p/14713957.html
4、https://blog.csdn.net/w727655308/article/details/121207345