In data processing business, sometimes it is necessary to compare the content of two text files which have the same or different data. This article will introduce several situations of text file comparison, such as whole line comparison, key column comparison, and small files at the same time. , Compare large files, and provide code examples written in esProc SPL. esProc is a professional data calculation engine. SPL has a complete set of function libraries in the field of set operations. It is very convenient to compare files and the code written is very concise.
1. Small file comparison
1.1 Whole row comparison
There are two text files, each line of which is a string, and the entire line of the two files must be compared. To deal with this problem, each line of the file can be read as a string to form a set, and then the result can be obtained through the operation of the two sets.
The student ID names of the students who signed up for painting and dancing classes are recorded in paint.txt and dance.txt respectively. Part of the data in
paint.txt is as follows: 20121102-Joan
20121107-Jack
20121113-Mike
1.1.1. Find the same
Find out all the same lines in the two files, that is, find the intersection of the two sets.
Example: Please find all students who have applied for these two interest classes and record them in the p_d.txt file.
The esProc SPL script is as follows:
| A | Comment | |
| 1 | =file("e:/txt/paint.txt").read@n() | Read each line of paint.txt to form a set |
| 2 | =file("e:/txt/dance.txt").read@n() | Read each line of dance.txt to form a set |
| 3 | =file("e:/txt/p_d.txt").write(A1^A2) | Write the intersection of A1 and A2 into the file p_d.txt |
1.1.2. Find the difference
There are two situations to find the difference:
1. Find all the different lines in the two files.
Example: To find out all students who only registered for one interest class, the esProc SPL script is as follows:
| A | Comment | |
| 1 | =file("e:/txt/paint.txt").read@n() | Read each line of paint.txt to form a set |
| 2 | =file("e:/txt/dance.txt").read@n() | Read each line of dance.txt to form a set |
| 3 | =file("e:/txt/p_d.txt").write(A1%A2) | Write the XOR set of A1 and A2 into the file p_d.txt |
2. Find out the lines in one file but not in another file.
Example: To find out the students who only registered for the drawing class and the students who only registered for the dance class, the esProc SPL script is as follows:
| A | Comment | |
| 1 | =file("e:/txt/paint.txt").read@n() | Read each line of paint.txt to form a set |
| 2 | =file("e:/txt/dance.txt").read@n() | Read each line of dance.txt to form a set |
| 3 | =file("e:/txt/p_1.txt").write(A1\A2) | Subtract the difference of drawing class A1 from dance class A2, that is, only the students who reported to the drawing class, write it into the file p_1.txt |
| 4 | =file("e:/txt/d_1.txt").write(A2\A1) | Subtract the difference of the dance class A2 from the drawing class A1, that is, only the students who have reported to the dance class, write it into the file d_1.txt |
1.2 Key column comparison
Two text files have multiple columns of data. The first row is the column name, and the second row is the data record. The content of the key columns in the two files should be compared. To deal with this problem, you can read the file into a data set, take out the record values of the key columns to form a set, and then obtain the result through the operation of the two sets.
There are sales order tables order_2018.txt and order_2019.txt for 2018 and 2019. The two files have the same column structure. Part of the data is as follows:
1.2.1. Find the same
Find out the key column values in the two files that are the same.
Example: Please find out the CustomerId and ProductId of users who have purchased the same product in the past two years and record them in the c_p.txt file.
The esProc SPL script is as follows:
| A | Comment | |
| 1 | =file("e:/txt/order_2018.txt").import@t(CustomerId,ProductId) | Read out the key column data in the 2018 order table |
| 2 | =file("e:/txt/order_2019.txt").import@t(CustomerId,ProductId) | Read out the key column data in the 2019 order table |
| 3 | = [A1, A2] and a .merge @ () | Merge the two-year data, @i means to return the records contained in common |
| 4 | =file("e:/txt/c_p.txt").export@t(A3) | Write the result in A3 to c_p.txt |
Some data in the c_p.txt file are as follows:
CustomerId ProductId
20108 1
20806 3
1.2.2. Find the difference
Example 1: Find out the order status of new customers in 2019 and save it in the file new_c.txt. The esProc SPL script is as follows:
| A | Comment | |
| 1 | =file("e:/txt/order_2018.txt").import@t() | Read the order form for 2018 |
| 2 | =file("e:/txt/order_2019.txt").import@t() | Read the order form for 2019 |
| 3 | =A2.id(CustomerId)\A1.id(CustomerId) | Subtract the 2018 customer ID from all customer IDs in 2019 to get the new customer ID |
| 4 | =A2.select(A3.contain(CustomerId)) | Filter the orders of new customers from the 2019 order table |
| 5 | =file("e:/txt/new_c.txt").export@t(A4) | Write the new customer order into the file new_c.txt |
Some data in new_c.txt are as follows:
Example 2: Find out the IDs of all lost customers in 2019 and save them in the file lost_c.txt. The esProc SPL script is as follows:
| A | Comment | |
| 1 | =file("e:/txt/order_2018.txt").import@t() | Read the order form for 2018 |
| 2 | =file("e:/txt/order_2019.txt").import@t() | Read the order form for 2019 |
| 3 | =A1.id(CustomerId)\A2.id(CustomerId) | Subtract all customer IDs in 2018 from all customer IDs in 2019 to get the customer IDs that were lost |
| 4 | =file("e:/txt/lost_c.txt").write(A3) | Write the lost customer ID to the file lost_c.txt |
2. Large file comparison
大文件数据不能一次性全部装进内存，不能象小文件数据那样全部读出来再进行比对，需要分批读出数据去比较。esProc SPL提供了游标来处理大文件运算，使大文件比对运算也变得十分方便。
2.1 整行比对
有两个大文本文件，其每一行是一个字符串，要对这两个文件中整行内容进行比对。处理此问题要把文件的每一行读成一个字符串，成为游标中的一条记录，然后通过两个游标的运算得出结果。
现有各州房产产权人员登记表大文件，里面记录产权人的身份证及姓名，部分数据如下所示：
510121198802213364-Joan
110113199203259852-Jack
201264197206271113-Mike
2.1.1. 找相同
把两个文件中整行内容相同的行都找出来，即求两文件的交集。
示例：请找出在Washington和New York两个州都有房产的人员，记录在w_n.txt文件中。
esProc SPL脚本如下：
| A | 注释 | |
| 1 | =file("e:/txt/washington.txt").cursor@s().sortx(_1) | 创建Washington数据游标，并排序 |
| 2 | =file("e:/txt/newyork.txt").cursor@s().sortx(_1) | 创建New York的数据游标，并排序 |
| 3 | =[A1,A2].mergex@i() | 对两游标数据进行有序归并，@i表示算出共同包含的数据，即在两州都有房产的人员 |
| 4 | =file("e:/txt/w_n.txt").export(A3) | 将算出的结果写入文件w_n.txt中 |
【注】：若文件中的数据是排好序的，则不再调用sortx函数，以下同
2.1.2. 找不同
示例：现有全国房产产权人员总表all.txt和Washington房产产权人员登记表，请检查总表中是否有遗漏的Washington人员，结果记录在lost_w.txt中。
esProc SPL脚本如下：
| A | 注释 | |
| 1 | =file("e:/txt/washington.txt").cursor@s().sortx(_1) | 创建Washington数据游标，并排序 |
| 2 | =file("e:/txt/all.txt").cursor@s().sortx(_1) | 创建全国的数据游标，并排序 |
| 3 | =[A1,A2].mergex@d() | 对两游标数据进行有序归并，@d表示从A1中去掉A2中的所有人员，即总表中遗漏的Washington人员 |
| 4 | =file("e:/txt/lost_w.txt").export(A3) | 将算出的结果写入文件lost_w.txt中 |
2.2 关键列比对
与小文件比对不同，大文件进行关键列比对时要用游标来操作，先按关键列排序，再对两个游标数据进行有序归并计算出需要的结果。
本节仍用小文件比对时的2018、2019年的销售订单表order_2018.txt和order_2019.txt，只是数据量大大增加到大文件的规模。
2.2.1. 找相同
把两个文件中关键列值相同的都找出来。
示例：请找出这两年都购买了同一种产品的用户CustomerId和产品ProductId，记录在c_p.txt文件中。
esProc SPL脚本如下：
| A | 注释 | |
| 1 | =file("e:/txt/order_2018.txt").cursor@t(CustomerId,ProductId) | 创建2018订单表关键列数据游标 |
| 2 | =file("e:/txt/order_2019.txt").cursor@t(CustomerId,ProductId) | 创建2019订单表关键列数据游标 |
| 3 | =A1.sortx(CustomerId,ProductId) | 对2018数据按关键列排序 |
| 4 | =A2.sortx(CustomerId,ProductId) | 对2019数据按关键列排序 |
| 5 | =[A3,A4].merge@i() | 对两年数据进行归并，@i表示返回共同包含的记录 |
| 6 | =file("e:/txt/c_p.txt").export@t(A5) | 将A5中的结果写入c_p.txt中 |
2.2.2. 找不同
示例1：找出2019年新增客户的订单情况，保存在文件new_c.txt中， esProc SPL脚本如下：
| A | 注释 | |
| 1 | =file("e:/txt/order_2018.txt").cursor@t() | 创建2018年订单表游标 |
| 2 | =file("e:/txt/order_2019.txt").cursor@t() | 创建2019年订单表游标 |
| 3 | =A1.groupx(CustomerId) | 按CustomerId分组取得2018客户Id |
| 4 | =A2.groupx(CustomerId) | 按CustomerId分组取得2019客户Id |
| 5 | =[A4,A3].mergex@d() | 2019减去2018，得到新增客户Id |
| 6 | =A2.reset().sortx(CustomerId) | 2019游标复位后按CustomerId排序 |
| 7 | =joinx(A6:new_order,CustomerId;A5:new_c,CustomerId) | 2019年订单表与新增客户Id连接，筛选出新增客户的订单 |
| 8 | =file("e:/txt/new_c.txt").export@t(A7.(new_order)) | 将新增客户订单写入文件new_c.txt中 |
注：groupx的结果已经按分组字段CustomerId排序。
示例2：找出2019年所有流失的客户Id，保存在文件lost_c.txt中， esProc SPL脚本如下：
| A | 注释 | |
| 1 | =file("e:/txt/order_2018.txt").cursor@t() | 创建2018年订单表游标 |
| 2 | =file("e:/txt/order_2019.txt").cursor@t() | 创建2019年订单表游标 |
| 3 | =A1.groupx(CustomerId) | 按CustomerId分组取得2018客户Id |
| 4 | =A2.groupx(CustomerId) | 按CustomerId分组取得2019客户Id |
| 5 | =[A3,A4].mergex@d() | 2018减去2019，得到流失客户Id |
| 6 | =file("e:/txt/lost_c.txt").export@t(A5) | 将流失客户Id写入文件lost_c.txt中 |
3. 大文件与小文件比对
大文件与小文件对比时，可以把小文件数据全部读进内存，大文件用游标处理，与小文件的数据进行连接计算。
3.1 整行比对
还用上小节的例子，大文件是各州房产产权人员登记表，小文件是某城市房产产权人员登记表。
3.1.1. 找相同
把两个文件中整行内容相同的行都找出来，即求两文件的交集。
示例：请找出在Washington州和New York城都有房产的人员，记录在w_n.txt文件中。
esProc SPL脚本如下：
| A | 注释 | |
| 1 | =file("e:/txt/washington.txt").cursor@s() | 创建Washington数据游标，无需排序 |
| 2 | =file("e:/txt/newyorkcity.txt").read@n() | 读出New York城的数据，@n表示读出整行构成序列 |
| 3 | =A1.join@i(_1,A2) | 将A1游标数据与A2连接，@i表示只保留A2中有的数据 |
| 4 | =file("e:/txt/w_n.txt").export(A3) | 将算出的结果写入文件w_n.txt中 |
3.1.2. 找不同
示例：请检查New York州登记表中是否有遗漏的New York城的人员，结果记录在lost_w.txt中。
esProc SPL脚本如下：
| A | 注释 | |
| 1 | =file("e:/txt/newyork.txt").cursor@s() | 创建Washington数据游标，无需排序 |
| 2 | =file("e:/txt/newyorkcity.txt").read@n() | 读出New York城的数据，@n表示读出整行构成序列 |
| 3 | =A1.join@i(_1,A2) | 将A1游标数据与A2连接，@i表示只保留A2中有的数据 |
| 4 | =A2\A3.(_1) | 从A2中减去共有的数据行，得到遗漏的数据行 |
| 5 | =file("e:/txt/lost_w.txt").export(A4) | 将算出的结果写入文件lost_w.txt中 |
3.2 关键列比对
本节使用2019年的销售订单表order_2019.txt和2019年之前所有年份的订单表order_old.txt，前者是小文件，后者是大文件。
3.2.1. 找相同
示例：请找出2019年及之前某年都购买了同一种产品的用户CustomerId和产品ProductId，记录在c_p.txt文件中。
esProc SPL脚本如下：
| A | 注释 | |
| 1 | =file("e:/txt/order_old.txt").cursor@t(CustomerId,ProductId) | 创建2019之前订单表关键列数据游标 |
| 2 | =file("e:/txt/order_2019.txt").import@t(CustomerId,ProductId) | 读出2019订单表关键列数据 |
| 3 | =A2.group@1(CustomerId,ProductId) | 对2019数据删除重复记录 |
| 4 | =A1.join@i(CustomerId:ProductId,A3:CustomerId:ProductId) | 将A1游标与A3按关键列连接，@i表示只保留能找到的记录 |
| 5 | =A4.fetch().group@1(CustomerId,ProductId) | 取出连接后的结果并删除重复记录 |
| 6 | =file("e:/txt/c_p.txt").export@t(A5) | 将A5中的结果写入c_p.txt中 |
3.2.2. 找不同
示例1：找出2019年新增客户的订单情况，保存在文件new_c.txt中， esProc SPL脚本如下：
| A | 注释 | |
| 1 | =file("e:/txt/order_old.txt").cursor@t() | 创建2019之前订单表数据游标 |
| 2 | =file("e:/txt/order_2019.txt").import@t() | 读出2019订单表数据 |
| 3 | =A1.groupx(CustomerId) | 按CustomerId分组取得2019之前客户Id |
| 4 | =A2.id(CustomerId) | 取得2019年所有不重复的客户Id |
| 5 | =A3.join@i(CustomerId,A4).fetch() | A3与A4连接，@i表示保留A4中包含的客户Id |
| 6 | =A4\A5.(CustomerId) | 2019年客户Id减去A5中的，得到新增的 |
| 7 | =A2.select(A6.contain(CustomerId)) | 筛选出2019新增客户的订单 |
| 8 | =file("e:/txt/new_c.txt").export@t(A7) | 将新增客户订单写入文件new_c.txt中 |
Example 2: Find out the IDs of all lost customers in 2019 and save them in the file lost_c.txt. The esProc SPL script is as follows:
| A | Comment | |
| 1 | =file("e:/txt/order_old.txt").cursor@t(CustomerId) | Create the customer Id cursor in the order table before 2019 |
| 2 | =file("e:/txt/order_2019.txt").import@t(CustomerId) | Read out the customer Id data of the 2019 order form |
| 3 | =A2.id(CustomerId) | Get unique customer Id in 2019 |
| 4 | =A1.select(!A3.contain(CustomerId)) | Filter out customers who are not in 2019 from customers before 2019 |
| 5 | =A4.groupx(CustomerId) | Id delete duplicate customers |
| 6 | =file("e:/txt/lost_c.txt").export@t(A5) | Write lost customer orders into the file lost_c.txt |
"SPL CookBook", there are more related calculation examples.