Compilação do método es de armazenamento assíncrono

Sobreviva durante o dia e desenvolva-se durante a noite.
fale para importar biz, mostre seu código perfeito, totalmente ocupado, pule dureza, faça um resultado melhor, aguarde a mudança, desafie Sobreviva.
feliz por hardess para resolver denpendies.

Índice

visão geral

O rastreamento da Web é uma necessidade muito comum.

precisar:

ideias de design

Realizar análise de pensamento

1. Adicionar anotações assíncronas

2. Chamada da camada de controle

Downloader Podemos usar a solicitação HTTP para baixar os elementos URL da interface. A essência é
o downloader.

3. Injetar feijões de primavera

O agendador do rastreador pode usar o mecanismo multithread para rastrear páginas da Web mais rapidamente, como se estivesse agendado.

4. Códigos comuns:

package com.georsoft.search.controller.provider;


import com.alibaba.fastjson.JSONArray;
import com.georsoft.common.core.utils.DateUtils;

import com.georsoft.search.es.entity.ArcDocument;
import com.georsoft.search.index.convert.ArcDocCreateConvert;
import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;

import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.support.WriteRequest;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.xcontent.XContentType;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Component;

import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

@Component
@Slf4j
public class AsyncTaskEs {
    
    

    @Autowired
    private ArcDocCreateConvert arcDocumentConvert;

    @SneakyThrows
    @Async
    public void doTask1(List<Map<String, Object>> addEsDataMapList) {
    
    
        long t1 = System.currentTimeMillis();
        addElasticsearchData(addEsDataMapList);
        long t2 = System.currentTimeMillis();
        log.info("task1 cost {} ms" , t2-t1);

    }


    @Value("${spring.elasticsearch.rest.nodes:10.100.13.105}")
    private String host;
    @Value("${spring.elasticsearch.rest.port:8083}")
    private String port;
    @Value("${spring.elasticsearch.rest.indexName:test}")
    private String arc_document;//索引名称

    /**
     * 创建连接 高级客户端
     */
    public RestHighLevelClient restHighLevelClient() {
    
    
//        RestHighLevelClient client = new RestHighLevelClient(
//                RestClient.builder(new HttpHost(host, Integer.parseInt(port), "http")));
        HttpHost host = new HttpHost("10.100.13.105",  8083, HttpHost.DEFAULT_SCHEME_NAME);

        RestClientBuilder builder = RestClient.builder(host);

        CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("es", "111111"));
        builder.setHttpClientConfigCallback(f -> f.setDefaultCredentialsProvider(credentialsProvider));
        RestHighLevelClient client = new RestHighLevelClient(builder);
        return  client;
    }



        /**
         * Elasticsearch数据导入
         */
        public void addElasticsearchData(List<Map<String, Object>> addEsDataMapList) {
    
    
            //获取连接
            RestHighLevelClient client = restHighLevelClient();
            try {
    
    
                //创建请求
                BulkRequest bulkRequest = new BulkRequest();
                //创建index请求 千万注意，这个写在循环外侧，否则UDP协议会有丢数据的情况，看运气
                IndexRequest requestData = null;
                Map<Object,Object> dataMap=new HashMap<>();
                for (Map<String, Object> addEsDataMap : addEsDataMapList) {
    
    //添加数据
                        for (Map.Entry<String, Object> entry : addEsDataMap.entrySet()) {
    
    
                            dataMap.put(lineToHump(entry.getKey()),  entry.getValue());
                            dataMap.put("description","描述");
                            dataMap.put("year",2008);
                            dataMap.put("trxId","交易ID");
                            dataMap.put("contractNo","12332131");
                            dataMap.put("deadline",12332L);
                        }

                    ArcDocument arcDocument =arcDocumentConvert.convert(dataMap);

                    requestData=new IndexRequest(arc_document, "_doc", dataMap.get("arcId").toString()).source(arcDocument, XContentType.JSON);
                    bulkRequest.add(requestData);
                }
                log.info("es同步数据数量:{}", bulkRequest.numberOfActions());
                //设置索引刷新规则
                bulkRequest.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE);
                //分批次提交，数量控制
                if (bulkRequest.numberOfActions() >= 1) {
    
    
    //                BulkResponse bulkResponse = client.bulk(bulkRequest, RequestOptions.DEFAULT);
    //                log.info("es同步数据结果:{}", bulkResponse.hasFailures());

                    BulkResponse bulkResponse = client.bulk(bulkRequest, RequestOptions.DEFAULT);
                    if(bulkResponse.hasFailures()){
    
    
                        log.info("数据写入失败：{}",bulkResponse.buildFailureMessage());
                    }else {
    
    
                        log.info("实时消息es写入成功");

                    }
                }
            } catch (Exception e) {
    
    
                e.printStackTrace();
                log.error("es同步数据执行失败:{}", addEsDataMapList);
            } finally {
    
    
                try {
    
    
                    client.close();
                } catch (IOException e) {
    
    
                    e.printStackTrace();
                }
            }
        }


    public static String lineToHump(String str) {
    
    
        str = str.toLowerCase();
        Pattern linePattern = Pattern.compile("_(\\w)");
        Matcher matcher = linePattern.matcher(str);
        StringBuffer sb = new StringBuffer();
        while (matcher.find()) {
    
    
            matcher.appendReplacement(sb, matcher.group(1).toUpperCase());
        }
        matcher.appendTail(sb);
        return sb.toString();
    }


}

Teste de parâmetro de desempenho:

Referências e Leitura Recomendada

1. ainda não

Bem-vindo à leitura, ferros velhos, se for útil para você, curta e siga! ~