For bulk inserts and bulk queries
The thing to watch out for is whether there is data duplication.
I thought it was inserted in batches. The source channel and date of this batch of data are the same.
After a new batch of data arrives, take the channel and date of the first data, go to the database to query, if there is no data, add it; if there is data, it means that it is added repeatedly, do not add it;,
distinct deduplication
select distinct t.fund_code,t.happen_date
from counter.tb_post_table t
where t.fund_code='WEIXIN' and t.happen_date='20170824'