About MySQL to handle duplicate data

Statistics duplicate data

The following tables we will repeat the number of records in the first_name and last_name:

mysql> SELECT COUNT(*) as repetitions, last_name, first_name -> FROM person_tbl -> GROUP BY last_name, first_name -> HAVING repetitions > 1;

The above query will return person_tbl table number of duplicate records. Under normal circumstances, the query duplicate values, do the following:

• Determine which column contains values ​​may be repeated.
• In the column selection list using the COUNT (*) those listed in the column.
• Columns listed in the GROUP BY clause.
• HAVING clause sets the number of repetitions is greater than 1.

 

Filter duplicate data

If you need to duplicate data can not be read using the DISTINCT keyword in the SELECT statement to filter duplicate data.

mysql> SELECT DISTINCT last_name, first_name
    -> FROM person_tbl;

You can also use the GROUP BY to read the data in the table does not duplicate data:

mysql> SELECT last_name, first_name
    -> FROM person_tbl -> GROUP BY (last_name, first_name);

 

Delete duplicate data

If you want to dedupe data in the table, you can use the following SQL statement:

mysql> CREATE TABLE tmp SELECT last_name, first_name, sex FROM person_tbl GROUP BY (last_name, first_name, sex); mysql> DROP TABLE person_tbl; mysql> ALTER TABLE tmp RENAME TO person_tbl;

Of course you can also add INDEX (index) and PRIMAY KEY (primary key) in such a simple way to remove duplicate data table records in the table. Methods as below:

mysql> ALTER IGNORE TABLE person_tbl
    -> ADD PRIMARY KEY (last_name, first_name);