Use xargs call volume parametric curl command

Short, flat, fast.

background

Sometimes, you need batch processing a plurality of data files.

For example, there is a file file.txt, contains a series of shop ID. There is a ready-made command curl_search can handle documents in a single shop. How, then, batch process the data it?

file.txt

111,222,333

curl_search：

curl -s -H "Content-Type: application/json" -X POST -d '{"source":"xxx", "orderBy":"createTime", "order":"desc", "requestId":"111", "searchParam": {"shopId": 111, "endTime":1583424000,"startTime":1580486400}}' http://127.0.0.1:7001/app/order/search

Curl_search result is:

{"result":true,"code":0,"message":null,"data":{"success":true,"code":200,"message":"successful","requestId":null,"errorData":null,"data":{"orderNos":["E001","E002'],"total":22}}}

I want to get the result is: one per line kdtId total

111 22
222 256
333 1024

solution

Define a script:

curl_m.sh

#!/bin/sh

shopId=$1

curl -s -H "Content-Type: application/json" -X POST -d  '{"source":"xxx", "orderBy":"createTime", "order":"desc", "requestId":"'"${shopId}"'", "searchParam": {"shopId":"'"${shopId}"'", "endTime":1583424000,"startTime":1580486400}}' http://127.0.0.1:7001/trade-manage/order/searchOrderNos | echo $kdtId $(sed -r 's/^.*total.*:([0-9]+)\}.*$/\1/')

Then use:

cat /tmp/file.txt | tr "," "\n" | xargs -I {} sh curl_m.sh {}  > final_orders.txt

Solving process

There are two problems to solve: 1. From the results curl_search command parses the total; 2. call volume curl command, you must go to shopId as a parameter.

Parsing total

Parsing total, regular use sed + capture capability. Regular Expressions basis See: "regular expressions Basics"

Strategy is what will be required by the () enclosed, then replace references. For example, I just total number after the colon will be needed portions :([0-9]+)\}identified and enclosed, then use the reference symbol to get $ n.

Parameterized call curl

ShopId need to spread to curl command. However, both the curl command in double quotes, there are single quotes, which will make variable references fail. May be used " '$ {shopId}'" to incoming shopId.

Command substitution

$ (Command): command itself is replaced with the calculated value of the command, the command value thus calculated can pass additional command as a parameter.

Redirect

> Is the redirection symbol, command execution results can be redirected to a file. When the command applies the result is very large.