Zhike travel service system
- Preface
- Logstash is easy to use
- 1. Overview of project requirements
- 2. Technical feasibility
- 3. es+hbase builds search engine
- 4. Transfer data in es to MongoDB
- 5. springboot integrates hadoop
-
- 1. Import hadoop dependencies in the pom.xml file
- 2. Read data from MongoDB as mapreduce input source
- 3. Java calculates the distance between the longitude and latitude of two locations
- 4. SpringBoot encapsulates MapReduce’s WordCount to do character statistics
- 5. Vue+Baidu Map API locates the current location
- springboot+scala development
- vue+echarts builds China map
Preface
通过简单的学习了elasticsearch基础，想做一个简单的、类似于百度京东等搜索项目巩固学习知识。
Logstash is easy to use
下载使用logstash:
Logstash is a lightweight log collection and processing framework that can easily collect scattered and diverse logs, perform customized processing, and then transfer them to a designated location, such as a server or file. Logstash can directly Associated with the database and automatically updates the index based on the data in the database.
https://www.elastic.co/cn/downloads/past-releases/logstash-7-7-0
1.控制台采集数据，控制台输出数据:
logstash -e " intput {stdin {}} output {stdout {}}"
2.控制台采集数据，控制台输出数据（json）
bin/logstash -e "input { stdin { } } output { stdout {codec => json} }"
3.把命令放到文件中使用:
创建demo.conf,添加： intput {stdin {}} output {stdout {}}
启动 bin/logstash -f demo.conf
4.将mysql数据同步到es:
a. Download the mysql driver
b. In the installation directory, create a new configuration file demo2.conf and add the following
input {
# 多张表的同步只需要设置多个jdbc的模块就行了
jdbc {
# mysql 数据库链接
jdbc_connection_string => "jdbc:mysql://localhost:3306/test?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC"
# 用户名和密码
jdbc_user => "root"
jdbc_password => "986836"
# 驱动
jdbc_driver_library => "G:/Download/logstash-7.7.0/data/mysql-connector-java-8.0.27.jar"
# 驱动类名
jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
#是否分页,及分页大小
jdbc_paging_enabled => "true"
jdbc_page_size => "100"
#直接执行sql语句
statement =>"select * from book"
# 执行的sql 文件路径+名称
# statement_filepath => "..../test.sql"
#设置监听间隔 各字段含义（由左至右）分、时、天、月、年，全部为*默认含义为每分钟都更新
schedule => "10 * * * *"
# 索引类型
#type => "jdbc"
}
}
output {
elasticsearch {
#es的ip和端口
hosts => ["http://hadoop:9200"]
#ES索引名称（自己定义的）
index => "books"
#文档类型
document_type => "_doc"
#设置数据的id为数据库中的字段
document_id => "%{isbn}"
}
stdout {
codec => json_lines
}
}
Execution: logstash -f …/data/demo2.conf
1. Overview of project requirements
This system meets the needs of users. It is fully functional, easy to operate, and easy to use. The main function is to implement a system that can recommend nearby/cost-effective food, hotels, leisure and entertainment, foot massage, KTV, attractions and beauties through the user's location/ratings. Clubs and other places to eat, drink and have fun are recommended to users. And the recommended method is not only calculated based on the proximity principle, but also calculated by the merchant rating on the system. Recommend different categories of projects based on the needs of different users. The system not only optimizes the code but also increases the speed of data processing, thus increasing user comfort during development in line with the target user experience.
Considering the issues of development efficiency and build level, the advantages and disadvantages of system performance can be expressed based on the user's visual experience and usage, as well as the responsiveness of the system. The system not only optimizes the code but also increases the speed of data processing, thus increasing user comfort during development in line with the target user experience.
Users can search for desired category items in the system through the search function, and the search function strictly calculates the user's actual distance and store distance as rating items based on LBS. Users can use distance, rating and low price to evaluate the currently searched merchants. Reorder and refine your search functionality.
When users view a merchant, they can comment and rate the merchant's store to improve the accuracy of store recommendations.
Offline and real-time analysis will also be conducted through user comments and ratings information and user click behavior.
2. Technical feasibility
Front-end and back-end separation has become an industry standard method for Internet project development. This system will also adopt the front-end and back-end separation vue+springboot architecture; to implement this system, here I choose Java language as back-end technical support, vue+elementUI+echarts, etc. Front-end architecture support, python crawler, kettle, flume+kafka are used as data collection support, mysql, mongdb are used as data storage support, elasticsearch is used to implement full-text search, etc. Based on the Java language, some middleware is used as support for big data development.
In the world of the Internet, any software with market factors has explosive power. When the number of visits or users of the software increases explosively, it is very important whether the software code is robust and whether some middleware is used to reduce peaks. In the design of this project, redis will be fully used as the cache layer framework to reduce the number of database accesses, as well as data deduplication effects, geographical location calculation and other functions. When there is a large amount of DEMO data or real user data, hadoop (mapreduce) + Spark + scala will be used as a log analysis method, including but not limited to recording the number of page visits, the number of HTTP requests, the number of independent IP visits, and different page visits and refreshes. Quantity etc. Kafka will also be used for traffic peak shaving and asynchronous and real-time transmission of message queues.
In the recommendation algorithm, the als algorithm is used to calculate the merchant's rating of the user. The formula is defined as the pearson correlation coefficient P(x, y) of two continuous variables (X, Y) equal to the covariance cov(X, Y) between them. Divided by the product of their respective standard deviations (σX,σY). The value of the coefficient is always between -1.0 and 1.0. Variables close to 0 are called no correlation, and variables close to 1 or -1 are called strong correlation. Usually, the correlation strength of variables is judged by the following value range: Correlation coefficient 0.8-1.0 Very strong correlation 0.6-0.8 Strong correlation 0.4-0.6 Moderate correlation 0.2-0.4 Weak correlation 0.0-0.2 Very weak correlation or no correlation, use echarts Visualize the data results, and finally integrate various frameworks through springboot to display them to users.
3. es+hbase builds search engine
Specify index library parameters
Next, we manually specify the settings and mapping parameters of the index library.
{
"settings":{
"number_of_shards":5, #分片数
"number_of_replicas":1 #副本数
},
"mappings":{
"dynamic":"strict",
"_source":{"excludes":["content"]},
"properties":{
"title":{"type":"text","analyzer":"ik_max_word"},
"point":{"type":"keyword"},
"price":{"type":"double"},
"level":{"type":"keyword"},
"address":{"type":"text","analyzer":"ik_max_word"}
"describe":{"type":"text","analyzer":"ik_max_word"},
"content":{"type":"text","analyzer":"ik_max_word"}
}
}
}
Explanation:
The dynamic parameter has 4 option values:
1. true is the default, indicating that dynamic mapping is enabled.
2. false indicates that undefined fields are ignored.
3. strict indicates that an exception is thrown when an unknown field is encountered.
4. runtime indicates that an unknown field is encountered. When using a field, treat it as a runtime field. The runtime field was added in the ES7.11 version. The runtime field will not be indexed, but the runtime field content can be obtained from _source, so the runtime can be suitable for public fields that are known. , and want to be compatible with scenarios of unknown extended fields.
Crawl the data and save it to hbase and build es index
package com.niit.data_manager.utils;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.util.Bytes;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.stream.Stream;
/**
* @author brett
* @date 2022-05-08 20:25:43
*
* Jsoup爬取旅游数据
*/
public class mockData {
private static final String[] arr = {
//todo 用alt+光标 选中同时编辑多行
"https://gs.ctrip.com/html5/you/place/1.html",
"https://gs.ctrip.com/html5/you/place/2.html",
"https://gs.ctrip.com/html5/you/place/39.html",
"https://gs.ctrip.com/html5/you/place/38.html",
"https://gs.ctrip.com/html5/you/place/21.html",
"https://gs.ctrip.com/html5/you/place/32.html",
"https://gs.ctrip.com/html5/you/place/61.html",
"https://gs.ctrip.com/html5/you/place/100001.html",
"https://gs.ctrip.com/html5/you/place/100007.html",
"https://gs.ctrip.com/html5/you/place/100008.html",
"https://gs.ctrip.com/html5/you/place/100003.html",
"https://gs.ctrip.com/html5/you/place/100009.html",
"https://gs.ctrip.com/html5/you/place/100076.html",
"https://gs.ctrip.com/html5/you/place/100039.html",
"https://gs.ctrip.com/html5/you/place/100058.html",
"https://gs.ctrip.com/html5/you/place/100053.html",
"https://gs.ctrip.com/html5/you/place/100064.html",
"https://gs.ctrip.com/html5/you/place/100062.html",
"https://gs.ctrip.com/html5/you/place/100065.html",
"https://gs.ctrip.com/html5/you/place/100066.html",
"https://gs.ctrip.com/html5/you/place/100068.html",
"https://gs.ctrip.com/html5/you/place/100056.html",
"https://gs.ctrip.com/html5/you/place/100055.html",
"https://gs.ctrip.com/html5/you/place/100054.html",
"https://gs.ctrip.com/html5/you/place/100051.html",
"https://gs.ctrip.com/html5/you/place/100059.html",
"https://gs.ctrip.com/html5/you/place/100038.html",
"https://gs.ctrip.com/html5/you/place/100052.html",
"https://gs.ctrip.com/html5/you/place/100060.html",
"https://gs.ctrip.com/html5/you/place/100067.html",
"https://gs.ctrip.com/html5/you/place/100063.html",
"https://gs.ctrip.com/html5/you/place/100031.html",
"https://gs.ctrip.com/html5/you/place/100032.html",
"https://gs.ctrip.com/html5/you/place/19.html",
"https://gs.ctrip.com/html5/you/place/14.html",
"https://gs.ctrip.com/html5/you/place/5.html",
"https://gs.ctrip.com/html5/you/place/508.html",
"https://gs.ctrip.com/html5/you/place/23.html"
};
//todo 百度百科爬取数据
public static Scenery searchScenery(Scenery scenery, String key) throws IOException {
ArrayList<String> contentList = new ArrayList<>();
ArrayList<String> imgList = new ArrayList<>();
String url = "https://baike.baidu.com/item/";
Document document = Jsoup.connect(url + key).get(); // 获取页面文档
String describe = document.getElementsByClass("para").eq(0).text();
scenery.setDescribe(describe);
Stream<Element> para = document
.getElementsByClass("para")
.stream().limit(6); // 获取前6条数据
para.forEach(element -> contentList.add(element.text()));
scenery.setContent(contentList);
Stream<Element> img = document.getElementsByClass("lazy-img").stream().limit(6);
img.forEach(element -> imgList.add(element.attr("data-src")));
scenery.setImgs(imgList);
return scenery;
}
//todo 设置es索引并将数据内容保存到hbase中
public static void main(String[] args) throws Exception {
HBaseUtil.createTable("scenery", "info"); //先创建数据表
String url = "https://m.ctrip.com/webapp/you/gspoi/sight/";
for (String s : arr) {
String s2 = s.split("/")[6];
List<String> linkList = new ArrayList<>();
if (s2.length() <= 10) {
linkList.add(s2);
} else {
Document document = Jsoup.connect(s).get();
Elements sightList = document.getElementsByClass("js_list_view");
Document parse = Jsoup.parse(sightList.toString());
Elements js_toJump = parse.getElementsByClass("js_toJump");
js_toJump.forEach(e -> {
String link = e.getElementsByIndexEquals(0).attr("href");
if (link.contains("html")) {
linkList.add(link.split("/")[6]);
}
});
}
for (String s3 : linkList) {
List<Put> sceneryList = new ArrayList<>();
Document document = Jsoup.connect(url + s3).get();
Object[] poiNames = document.getElementsByClass("poiName").stream().limit(5).toArray();
Object[] distanceFields = document.getElementsByClass("distanceField").stream().limit(5).toArray();
Object[] poiLevel = document.getElementsByClass("poiLevel").stream().limit(5).toArray();
Object[] commentScoreNum = document.getElementsByClass("commentScoreNum").stream().limit(5).toArray();
Object[] priceFont = document.getElementsByClass("priceFont").stream().limit(5).toArray();
System.out.println("name:" + poiNames.length + " 地址:" + distanceFields.length + " 等级:" + poiLevel.length + " 评分:" + commentScoreNum.length + " 价格:" + priceFont.length);
if (poiLevel.length == 5 && priceFont.length == 5 && distanceFields.length==5 && commentScoreNum.length==5) {
for (int i = 0; i < 5; i++) {
HashMap<String, String> indexMap = new HashMap<>();
String title = ((Element) poiNames[i]).getElementsByTag("a").attr("title");
String addr = ((Element) distanceFields[i]).text();
String level = ((Element) poiLevel[i]).text();
Double score = new Double(((Element) commentScoreNum[i]).text());
Double price = new Double(((Element) priceFont[i]).text());
Scenery scenery = new Scenery();
scenery.setName(title);
scenery.setAddress(addr);
scenery.setLevel(level);
scenery.setScore(score);
scenery.setTicketPrice(price);
scenery.setPoint(BaiDuApiUtil.getLatAndLngByAddress(title));
Scenery s02 = searchScenery(scenery, title);
// 设置数据在elasticsearch中的索引
indexMap.put("title", s02.getName());
indexMap.put("point", s02.getPoint());
indexMap.put("price", s02.getTicketPrice().toString());
indexMap.put("level", s02.getLevel());
indexMap.put("score", s02.getScore().toString());
indexMap.put("address", s02.getAddress());
indexMap.put("describe", s02.getDescribe());
//向索引库中添加
ESUtil.addIndex("scenery", s02.getName(), indexMap);
Put put = new Put(Bytes.toBytes(s02.getName()));
// 将数据入库到hbase
put.addColumn(Bytes.toBytes("info"), Bytes.toBytes("img"), Bytes.toBytes(s02.getImgs().toString()));
put.addColumn(Bytes.toBytes("info"), Bytes.toBytes("content"), Bytes.toBytes(s02.getContent().toString()));
sceneryList.add(put);
}
HBaseUtil.put2HBaseList("scenery", sceneryList);
try {
Thread.sleep(600);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
}
}
As shown in the figure, hbase saves data successfully!
Index creation successful!
Scenery scenery entity class
package com.example.ssm.domain;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.util.List;
/**
* @author brett
* @date 2022-05-09 15:21:23
*/
@Data
@NoArgsConstructor
@AllArgsConstructor
public class Scenery {
private StringBuilder title; // 景点名
private String point; //经纬度
private Double ticketPrice; //票价
private String level; //景区等级
private Double score; //评分
private String address; //地址
private StringBuilder describe; // 描述
private String firstImg; //封面图
private List<String> content; //景点简介
private List<String> imgs; // 景点图片
}
Find the corresponding longitude and latitude by place name
package com.niit.data_manager.utils;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;
import java.nio.charset.StandardCharsets;
/**
* @author brett
* @date 2022-05-15 20:29:39
*/
public class BaiDuApiUtil {
private static final String AK = "q4WRaidEQFHCqNC7Izof4GxWe8zAEbDj";
public static String getLatAndLngByAddress(String addr) {
String address = "";
String result = "";
try {
address = java.net.URLEncoder.encode(addr, "UTF-8");
} catch (UnsupportedEncodingException e1) {
e1.printStackTrace();
}
String url = "https://api.map.baidu.com/geocoding/v3/?address=" + address +
"&output=json&ak="+AK+"&callback=showLocation ";
URL myURL = null;
URLConnection httpsConn;
//进行转码
try {
myURL = new URL(url);
} catch (MalformedURLException ignored) {
ignored.printStackTrace();
}
try {
assert myURL != null;
httpsConn = myURL.openConnection();
if (httpsConn != null) {
InputStreamReader insr = new InputStreamReader(
httpsConn.getInputStream(), StandardCharsets.UTF_8);
BufferedReader br = new BufferedReader(insr);
String data = br.readLine();
String[] split = data.replace("{", ":").replace("}", "::").split("::");
if (split.length >=3){
result = split[2];
}
insr.close();
}
} catch (IOException ignored) {
ignored.printStackTrace();
}
return result;
}
}
HBaseUtil tool class
package com.example.ssm.utils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.Cell;
import org.apache.hadoop.hbase.CellUtil;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.*;
import org.apache.hadoop.hbase.util.Bytes;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
public class HBaseUtil {
private static Connection connection;
//初始化创建连接
static {
try{
connection = getConnection();
}catch(IOException e){
e.printStackTrace();
}
}
//todo 创建hbase连接
public static Connection getConnection() throws IOException {
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.zookeeper.quorum","192.168.111.137");
conf.set("hbase.zookeeper.property.clientPort","2181");
return ConnectionFactory.createConnection(conf);
}
//TODO 创建hbase表
public static void addTable(String tableName,String... cfs) throws IOException {
//获取admin
Admin admin = connection.getAdmin();
//创建列族builder
List<ColumnFamilyDescriptor> list = new ArrayList<>();
for (String cf:cfs){
ColumnFamilyDescriptor columnFamilyDescriptor = ColumnFamilyDescriptorBuilder.newBuilder(Bytes.toBytes(cf)).build();
list.add(columnFamilyDescriptor);
}
//创建TableBuilder
TableDescriptor tabBuilder = TableDescriptorBuilder.newBuilder(TableName.valueOf(Bytes.toBytes(tableName)))
.setColumnFamilies(list).build();
admin.createTable(tabBuilder);
admin.close();
}
//todo 向hbase中添加一条数据
public static void addData(String tabName,String rowKey,String cfName,String column,String value) throws IOException {
Table table = connection.getTable(TableName.valueOf(Bytes.toBytes(tabName)));
Put put = new Put(Bytes.toBytes(rowKey));
put.addColumn(Bytes.toBytes(cfName),Bytes.toBytes(column),Bytes.toBytes(value));
table.put(put);
table.close();
}
//todo 向hbase中添加一组数据
public static void addListData(String tabName,List<Put> list) throws IOException {
Table table = connection.getTable(TableName.valueOf(tabName));
table.put(list);
table.close();
}
//todo 根据rowKey获取hbase中的数据
public static Map<String,String> getDataByRowkey(String tabName,String rowkey) throws IOException {
Table table = connection.getTable(TableName.valueOf(Bytes.toBytes(tabName)));
Get get = new Get(Bytes.toBytes(rowkey));
Result result = table.get(get);
HashMap<String, String> map = new HashMap<>();
map.put("title",rowkey);
System.out.println("title:"+rowkey);
for (Cell c:result.listCells()){
//获取列信息
byte[] column = CellUtil.cloneQualifier(c);
//获取值
byte[] value = CellUtil.cloneValue(c);
map.put(new String(column),new String(value));
}
return map;
}
}
ESUtil tool class
package com.example.ssm.utils;
import com.example.ssm.domain.Scenery;
import org.apache.http.HttpHost;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.action.search.SearchType;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.text.Text;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.MatchQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.SearchHits;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightField;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
/**
* @author brett
* @date 2022-05-09 18:49:34
*
* es工具类
*/
public class ESUtil {
private ESUtil(){
};
private static RestHighLevelClient client;
private static String host = "192.168.111.137";
private static int port = 9200;
static {
//获取es连接
RestClientBuilder restBuilder = RestClient.builder(new HttpHost(host, port, "http"));
client = new RestHighLevelClient(restBuilder);
}
//获取es连接
public RestHighLevelClient getClient(){
return client;
}
// 建立索引
public static void addIndex(String index, String id, Map<String,String> map) throws IOException {
IndexRequest indexRequest = new IndexRequest(index);
indexRequest.id(id);
indexRequest.source(map);
// 执行
client.index(indexRequest, RequestOptions.DEFAULT);
}
//todo 全文检索功能
public static Map<String,Object> search(String key,String index,int start,int row) throws IOException {
if (start<=1){
start=1;
}
SearchRequest searchRequest = new SearchRequest();
// 指定索引库
searchRequest.indices(index);
//指定searchType
searchRequest.searchType(SearchType.QUERY_THEN_FETCH);
//组装查询条件
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
//如果传递了搜索参数，则拼接查询条件
// if (StringUtils.isNotBlank(key)){ //StringUtils.isNotBlank()判断参数是否为空
// searchSourceBuilder.query(QueryBuilders.multiMatchQuery(key,"title"));
// }
//多条件查询
BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
MatchQueryBuilder q1 = QueryBuilders.matchQuery("title", key);
MatchQueryBuilder q2 = QueryBuilders.matchQuery("describe", key);
boolQuery.should(q1);
boolQuery.should(q2);
searchSourceBuilder.query(boolQuery);
//分页
searchSourceBuilder.from((start-1)*row);
searchSourceBuilder.size(row);
//高亮
//设置高亮字段
HighlightBuilder highlightBuilder = new HighlightBuilder()
.field("title")
.field("describe"); //支持多个高亮字段
//设置高亮字段的前缀后缀
highlightBuilder.preTags("<font color='red'>");
highlightBuilder.postTags("</font>");
searchSourceBuilder.highlighter(highlightBuilder);
//指定查询条件
searchRequest.source(searchSourceBuilder);
//执行查询操作
SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
//存储返回给页面的数据
HashMap<String, Object> map = new HashMap<>();
//获取查询到的返回结果
SearchHits hits = searchResponse.getHits();
//获取数据总量
long numHits = hits.getTotalHits().value;
map.put("count",numHits);
List<Scenery> sceneries = new ArrayList<>();
//获取具体内容
SearchHit[] searchHits = hits.getHits();
//迭代解析具体内容
for (SearchHit hit:searchHits){
Map<String, Object> sourceAsMap = hit.getSourceAsMap();
StringBuilder title = new StringBuilder(sourceAsMap.get("title").toString());
StringBuilder describe = new StringBuilder(sourceAsMap.get("describe").toString());
String firstImg = sourceAsMap.get("firstImg").toString();
String score;
try {
score = sourceAsMap.get("score").toString();
} catch (NullPointerException e) {
score = "3.0";
e.printStackTrace();
}
String address = "";
try {
address = sourceAsMap.get("address").toString();
} catch (NullPointerException e) {
e.printStackTrace();
}
String level="";
try {
level = sourceAsMap.get("level").toString();
} catch (NullPointerException e) {
level = "3A";
e.printStackTrace();
}
String point="";
try {
point = sourceAsMap.get("point").toString();
} catch (NullPointerException e) {
e.printStackTrace();
}
String price="";
try {
price = sourceAsMap.get("price").toString();
} catch (NullPointerException e) {
price = "30.0";
e.printStackTrace();
}
//获取高亮字段
Map<String, HighlightField> highlightFields = hit.getHighlightFields();
//获取title字段的高亮内容
HighlightField titleHighLight = highlightFields.get("title");
if (titleHighLight != null){
title = new StringBuilder();
Text[] fragments = titleHighLight.getFragments();
for (Text t:fragments){
title.append(t);
}
}
//获取describe的高亮字段内容
HighlightField describeHighLight = highlightFields.get("describe");
if (describeHighLight!=null){
describe = new StringBuilder();
Text[] fragments = describeHighLight.getFragments();
for (Text t:fragments){
describe.append(t);
}
}
//把信息封装到Scenery对象中
Scenery scenery = new Scenery();
scenery.setDescribe(describe);
scenery.setTitle(title);
scenery.setDescribe(describe);
scenery.setPoint(point);
scenery.setFirstImg(firstImg);
scenery.setLevel(level);
scenery.setTicketPrice(Double.parseDouble(price));
scenery.setAddress(address);
scenery.setScore(new Double(score));
sceneries.add(scenery);
}
map.put("dataList",sceneries);
return map;
}
}
对应的controller
package com.example.ssm.controller;
import com.example.ssm.utils.ESUtil;
import com.example.ssm.utils.HBaseUtil;
import org.springframework.web.bind.annotation.*;
import java.io.IOException;
import java.util.Map;
import java.util.regex.Pattern;
/**
* @author brett
* @date 2022-05-9 14:54:03
*/
@RestController
@RequestMapping("/ES")
public class ESController {
@GetMapping("/searchData")
public Map<String,Object> getData(@RequestParam String key,@RequestParam int pageNo)
throws IOException {
return ESUtil.search(key,"scenery",pageNo,10);
}
@GetMapping("/selectDataByHbase")
public Map<String,String> getHbaseData(@RequestParam String rowkey)
throws IOException {
String regex = "^[\\u4e00-\\u9fa5]*$"; //匹配中文
StringBuilder key = new StringBuilder();
String[] arr = rowkey.split("");
for (String s:arr){
if (Pattern.matches(regex,s)){
key.append(s);
}
}
return HBaseUtil.getDataByRowkey("scenery", key.toString());
}
}
Note: Elasticsearch, hadoop, and hbase environments should be started in the virtual machine environment before running.
4. Transfer data in es to MongoDB
Download the logstash-output-mongdb plug-in:
https://github.com/logstash-plugins/logstash-output-mongodb
After opening it, be sure to switch the version to 3.1.5, and then download the zip package
Unzip it to the logstash directory.
Modify the Gemfile file in logstash
. Add gem "logstash-output-mongodb", :path => "logstash-output-mongodb-master" to the last line.
Use the administrator to open CMD
and switch to the bin directory of logstash and run: logstash-plugin install --no-verify.
After downloading, check the installation version: logstash-plugin list --verbose
to create the configuration file mongdb-out.conf:
input {
stdin {
}
elasticsearch {
#ESIP地址与端口
hosts => "hadoop:9200"
#ES索引名称（自己定义的）
index => "scenery"
#自增ID编号
# document_id => "%{id}"
#定时字段 各字段含义（由左至右）分、时、天、月、年，全部为*默认含义为每分钟都更新
#schedule => "* * * * *"
#设定ES索引类型
type => "message"
}
}
filter {
json {
source => "message"
remove_field => ["message"]
}
}
#目标mongodb地址信息
output {
stdout {
codec => rubydebug }
mongodb {
#目标mongodb集合
collection => "scenery"
#目标库名称
database => "mydb"
uri => "mongodb://hadoop:27017"
}
}
Execute command: logstash -f .../data/mongodb-out.conf
to import data successfully! ! !
View data in MongoDB:
data import successful
5. springboot integrates hadoop
1. Import hadoop dependencies in the pom.xml file
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.3</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.7.3</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.3</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
</exclusions>
</dependency>
hadoop和springboot冲突的主要有两个，一个是slf4j的日志包，一个是Tomcat 的servlet-api包，去掉这两个依赖就可以成功运行springboot了
Add basic star information in the application.yml configuration file
@Value("${hadoop.name-node}")
private String nameNode;
@Test
public void createDir() throws URISyntaxException, IOException, InterruptedException {
//1获取文件系统
Configuration configuration = new Configuration();
FileSystem fileSystem = FileSystem.get(new URI(nameNode), configuration,"root");
//2执行操作
fileSystem.mkdirs(new Path("/jianxuhui"));
fileSystem.close();
System.out.println("结束！！！");
}
2. Read data from MongoDB as mapreduce input source
Add mongodb-hadoop dependency:
<!-- mongodb整合hadoop,可以用mongodb作为mapreduce数据输入源 -->
<!-- https://mvnrepository.com/artifact/org.mongodb.mongo-hadoop/mongo-hadoop-core -->
<dependency>
<groupId>org.mongodb.mongo-hadoop</groupId>
<artifactId>mongo-hadoop-core</artifactId>
<version>2.0.2</version>
</dependency>
eg:从mongodb中将一个集合中的数据通过mapreduce处理，然后将结果返回到mongodb存储
package com.example.ssm.MR;
import com.mongodb.BasicDBObject;
import com.mongodb.hadoop.MongoInputFormat;
import com.mongodb.hadoop.MongoOutputFormat;
import com.mongodb.hadoop.io.BSONWritable;
import com.mongodb.hadoop.util.MongoConfigUtil;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.bson.BSONObject;
import java.io.IOException;
/**
* @author brett
* @date 2022-05-24 11:46:25
*
* 以mongodb数据作为mapreduce数据输入源
*/
public class mongodbSource {
static class mongodbMapper extends Mapper<Object, BSONObject, Text, BSONWritable>{
@Override
//实现mapper,mongodb读取出来就是一个BSONObject
protected void map(Object key, BSONObject value, Context context) throws IOException, InterruptedException {
System.out.println(key);
context.write(new Text(key.toString()),new BSONWritable(value));
}
}
static class mongodbReducer extends Reducer<Text,BSONWritable,Text,BSONWritable>{
@Override
//不做任何处理，直接将数据读取出来在存入到mongodb中
protected void reduce(Text key, Iterable<BSONWritable> values, Context context) throws IOException, InterruptedException {
for (BSONWritable value:values){
context.write(key,value);
}
}
}
public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {
args = new String[]{
"mongodb://192.168.111.137:27017/mydb.info",
"mongodb://192.168.111.137:27017/mydb.result"};
Configuration configuration = new Configuration();
MongoConfigUtil.setInputURI(configuration,args[0]);
MongoConfigUtil.setOutputURI(configuration,args[1]);
//可以构建条件BasicQuery查询
BasicDBObject query = new BasicDBObject();
MongoConfigUtil.setQuery(configuration,query);
MongoConfigUtil.setCreateInputSplits(configuration,false);
Job job = Job.getInstance(configuration,"Hadoop Mongodb");
job.setJarByClass(mongodbSource.class);
job.setMapperClass(mongodbMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(BSONWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(BSONWritable.class);
job.setReducerClass(mongodbReducer.class);
job.setInputFormatClass(MongoInputFormat.class);
job.setOutputFormatClass(MongoOutputFormat.class);
job.setNumReduceTasks(1);
System.exit(job.waitForCompletion(true)?0:1);
}
}
Result: The data is transferred successfully as shown in the figure!
3. Java calculates the distance between the longitude and latitude of two locations
package com.example.ssm.utils;
/**
* @author brett
* @date 2022-05-26 22:28:51
*
* 通过经纬度计算距离
*/
public class CalculateDistance {
/*
S = acos( sin(第1个点纬度 * ∏ / 180) * sin(第2个点纬度 * ∏ / 180) + cos(第1个点纬度 * ∏ / 180) *
cos(第2个点纬度 * ∏ / 180) * cos(第2个点经度 * ∏ / 180 - 第1个点经度 * ∏ / 180)) * 地球半径
*/
//地球半径，单位km
private static final double EARTH_RADIUS = 6378.137;
public static double getDistance(Double lng1,Double lat1,Double lng2,Double lat2){
//纬度
double lat01 = Math.toRadians(lat1); //Math.toRadians()将数据转化为弧度制
double lat02 = Math.toRadians(lat2);
//经度
double lng01 = Math.toRadians(lng1);
double lng02 = Math.toRadians(lng2);
// 计算两点距离的公式
double s = Math.acos(Math.sin(lat01)*Math.sin(lat02)+Math.cos(lng01)*Math.cos(lng02)*Math.cos(lng02-lng01));
// 弧长乘地球半径, 返回单位: 千米
s = s * EARTH_RADIUS;
return s;
}
}
Find provinces and cities based on latitude and longitude
//传入经纬度, 返回查询的地区, lng: 经度, lat: 纬度
public static String findByLatAndLng(String lat, String lng) {
try {
//移除坐标前后的 空格
/*lng = lng.trim();
lat = lat.trim();*/
CloseableHttpClient httpClient = HttpClients.createDefault();
// url中的ak值要替换成自己的:
String url = "http://api.map.baidu.com/reverse_geocoding/v3/?ak=xeiyKzum1WHj3SE39P216gDI9cnIYbgi&output=json&coordtype=wgs84ll&location=" + lat + "," + lng;
//System.out.println(url);
HttpGet httpGet = new HttpGet(url);
CloseableHttpResponse response = httpClient.execute(httpGet);
HttpEntity httpEntity = response.getEntity();
String json = EntityUtils.toString(httpEntity);
Map<String, Object> result = JSONObject.parseObject(json, Map.class);
if (result.get("status").equals(0)) {
Map<String, Object> resultMap = (Map<String, Object>) result.get("result");
resultMap = (Map<String, Object>) resultMap.get("addressComponent");
String province = (String) resultMap.get("province");
String city = (String) resultMap.get("city");
return province + city;
}
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
4. SpringBoot encapsulates MapReduce’s WordCount to do character statistics
package com.example.ssm.MR;
import org.apache.hadoop.fs.FileSystem;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.net.URI;
/**
* @author brett
* @date 2022-05-28 10:13:21
*
* hadoop配置类
*/
@Configuration
public class HadoopConfig {
@Bean("hdfsConfig")
public org.apache.hadoop.conf.Configuration hdfsChannerl(){
org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
conf.set("dfs.replication","1");
conf.set("dfs.client.use.datanode.hostname","true");
conf.set("mapred.job.tracker", "hdfs://192.168.111.137:8020/");
conf.set("fs.defaultFS", "hdfs://192.168.111.137:8020/");
System.setProperty("HADOOP_USER_NAME","root");
return conf;
}
@Bean("fileSystem")
public FileSystem createFs(@Qualifier("hdfsConfig") org.apache.hadoop.conf.Configuration conf){
FileSystem fs = null;
try {
URI uri = new URI("hdfs://192.168.111.137:8020/");
fs = FileSystem.get(uri,conf);
} catch (Exception e) {
e.printStackTrace();
}
return fs;
}
}
hadoopController class
@RestController
@RequestMapping(value = "/hadoop")
public class HadoopController {
@Qualifier("hdfsConfig")
@Autowired
private org.apache.hadoop.conf.Configuration conf;
@GetMapping(value = "/reduce")
public Boolean reduce() throws IOException, ClassNotFoundException, InterruptedException {
Job job = Job.getInstance(conf);
job.setMapperClass(WordCountMapper.class);
job.setReducerClass(WordCountReducer.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.setInputPaths(job, new Path(template.getNameSpace())); //输入路径
FileOutputFormat.setOutputPath(job, new Path("/statics")); //输出路径
return job.waitForCompletion(true);
}
}
5. Vue+Baidu Map API locates the current location
//js文件导出
export function MP(ak) {
return new Promise(function (resolve, reject) {
window.onload = function () {
resolve(window.BMap)//插入script标签后 会在window上挂一BMap属性,此为跨域获取的数据
};
var script = document.createElement("script");
script.type = "text/javascript";
script.src = "http://api.map.baidu.com/api?v=2.0&ak=" + ak + "&callback=init";
script.onerror = reject;
document.head.appendChild(script);//插入此标签后 会在window上挂一BMap属性,此为跨域获取的数据
})
}
//在vue 生命周期函数created()中引用getCity函数
getCity() {
this.$nextTick(()=>{
MP(this.ak).then(BMap=> {
//在此调用api
var geolocation = new BMap.Geolocation();
geolocation.getCurrentPosition(a=>{
console.log(a)
this.city = a.address.province+a.address.city;
this.latitude = a.latitude;
this.longitude = a.longitude;
})
})
})
}
springboot+scala development
添加依赖：
<!--添加依赖-->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</dependency>
...
<!-- 添加插件 -->
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.1</version>
<executions>
<execution>
<id>compile-scala</id>
<phase>compile</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>test-compile-scala</id>
<phase>test-compile</phase>
<goals>
<goal>add-source</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<recompileMode>incremental</recompileMode>
<scalaVersion>${scala.version}</scalaVersion>
<args>
<arg>-deprecation</arg>
</args>
<jvmArgs>
<jvmArg>-Xms64m</jvmArg>
<jvmArg>-Xmx1024m</jvmArg>
</jvmArgs>
</configuration>
</plugin>
vue+echarts builds China map
下载地图离线包:
<template>
<div class="content">
<div ref="charts" style="width: 70%; height: 800px"></div>
</div>
</template>
<script>
import zhongguo from "@/assets/china.json"
export default {
created () {
this.$nextTick(() => {
this.initCharts();
})
},
methods: {
initCharts() {
const charts = this.$echarts.init(this.$refs["charts"]);
const option = {
title: {
//标题样式
text: '各省份景点访问量',
x: "center",
textStyle: {
fontSize: 18,
color: "black"
},
},
// 背景颜色
backgroundColor: "#eff3e9",
// 提示浮窗样式
tooltip: {
//数据项图形触发
trigger: 'item',
//提示框浮层的背景颜色。 (鼠标悬浮后的提示框背景颜色)
backgroundColor: "white",
//字符串模板(地图): {a}（系列名称），{b}（区域名称），{c}（合并数值）,{d}（无）
formatter: '地区：{b}<br/>客流量：{c}'
},
//视觉映射组件
visualMap: {
top: 'center',
left: 'left',
// 数据的范围
min: 10,
max: 500000,
text: ['High', 'Low'],
realtime: true, //拖拽时，是否实时更新
calculable: true, //是否显示拖拽用的手柄
inRange: {
// 颜色分布
color: ['lightskyblue', 'yellow', 'orangered']
}
},
series: [
{
name: '模拟数据',
type: 'map',
mapType: 'china',
roam: false,//是否开启鼠标缩放和平移漫游
itemStyle: {
//地图区域的多边形 图形样式
normal: {
//是图形在默认状态下的样式
label: {
show: true,//是否显示标签
textStyle: {
color: "black"
}
}
},
zoom: 1.5, //地图缩放比例,默认为1
emphasis: {
//是图形在高亮状态下的样式,比如在鼠标悬浮或者图例联动高亮时
label: {
show: true }
}
},
top: "3%",//组件距离容器的距离
bottom:"0%",
left:"3%",
right:"3%",
data: [
{
name: '北京', value: 350000 },
{
name: '天津', value: 120000 },
{
name: '上海', value: 300000 },
{
name: '重庆', value: 92000 },
{
name: '河北', value: 25000 },
{
name: '河南', value: 20000 },
{
name: '云南', value: 500 },
{
name: '辽宁', value: 3050 },
{
name: '黑龙江', value: 80000 },
{
name: '湖南', value: 2000 },
{
name: '安徽', value: 24580 },
{
name: '山东', value: 40629 },
{
name: '新疆', value: 36981 },
{
name: '江苏', value: 13569 },
{
name: '浙江', value: 24956 },
{
name: '江西', value: 15194 },
{
name: '湖北', value: 41398 },
{
name: '广西', value: 41150 },
{
name: '甘肃', value: 17630 },
{
name: '山西', value: 27370 },
{
name: '内蒙古', value: 27370 },
{
name: '陕西', value: 97208 },
{
name: '吉林', value: 88290 },
{
name: '福建', value: 19978 },
{
name: '贵州', value: 94485 },
{
name: '广东', value: 89426 },
{
name: '青海', value: 35484 },
{
name: '西藏', value: 97413 },
{
name: '四川', value: 54161 },
{
name: '宁夏', value: 56515 },
{
name: '海南', value: 54871 },
{
name: '台湾', value: 48544 },
{
name: '香港', value: 49474 },
{
name: '澳门', value: 34594 }
]
}
]
};
// 地图注册，第一个参数的名字必须和option.geo.map一致
this.$echarts.registerMap("china",zhongguo)
charts.setOption(option);
},
},
};
</script>
result:
Basic use of vuex
cnpm install vuex --save
Import vuex package
import Vuex from 'vuex'
Vue.use(Vuesx)
Create store object
const store = new Vuex.Store({
states:{
count:0
}
})
export default store
Mount the object into the global vue instance
const vue = new Vue({
el:"#app",
rander :h=>h(app),
store
//将创建的共享数据对象，挂载到vue实例中
//所有组件，就可以直接从store中获取全局的数据了
})
Mutation is used to change the data in the Store.
① Store data can only be changed through mutation, and data in the Store cannot be directly manipulated.
②Although the operation is a little more cumbersome in this way, all data changes can be monitored centrally.