Python implementation of the txt file de-duplication function example
Updated: July 7, 2018 09:00:36 Author: rice son who I want to comment
This article describes the Python implementation of the txt file de-duplication function, involving Python for reading and writing text files of txt, string traversal, determine the relevant operating skills, need friends can refer to the following
This paper describes examples of txt file Python implementation of deduplication functionality. Share to you for your reference, as follows:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
# -*- coding:utf-8 -*-
#! python2
import
shutil
a
=
0
readDir
=
"/Users/Administrator/Desktop/old.txt"
#old
writeDir
=
"/Users/Administrator/Desktop/new.txt"
#new
# txtDir = "/home/Administrator/Desktop/１"
lines_seen
=
set
()
outfile
=
open
(writeDir,
"w"
)
f
=
open
(readDir,
"r"
)
for
line
in
f:
if
line
not
in
lines_seen:
a
+
=
1
outfile.write(line)
lines_seen.add(line)
print
(a)
print
(
'\n'
)
outfile.close()
print
(
"success"
)
|
Which old.tx as follows:
www.jb51.net
www.baidu.com
www.sina.com.cn
www.jb51.net
www.google.com
www.sohu.com
www.jb51.net
www.163.com
After running new.txt reads as follows:
www.jb51.net
www.baidu.com
www.sina.com.cn
www.google.com
www.sohu.com
www.163.com