[Actual] electricity supplier combat - Tao Tao Mall - cache build (the seventh day)

1. Lesson Plan

Seventh day:

1. solr Cluster Setup
2. Use solrj cluster management solr
3. The search function is switched to the Cluster Edition

 

1. What is SolrCloud

 

SolrCloud (solr cloud) is distributed Solr search program provided using SolrCloud when you require large-scale, fault-tolerant, distributed indexing and retrieval capabilities. When a small amount of index data when the system is not required to use SolrCloud when the index a large volume of search requests concurrent high, then the need to use SolrCloud to meet those needs.

 SolrCloud is distributed Solr-based search program and Zookeeper, its main idea is to use Zookeeper as the center of the cluster configuration information.

It has several features:

1) centralized configuration information

2) automatic fault tolerance

3) near-real-time search

4) automatic load balancing queries

 

1. Solr cluster system architecture

 

1. 1. Physical structure

Solr three examples (examples of which include each two Core), composed of a SolrCloud.

1. 1. Logical structure

Index set comprises two Shard (shard1 and shard2), shard1 shard2 respectively, and three Core, in which one the Replication Leader two, is generated by Leader Election zookeeper, zookeeper consistent control of the three Core index data for each shard, solve the availability problem.

The user initiates the index are available on request shard1 and shard2, to address the high concurrency problems.

 

1. 1. 1. collection

Collection is a complete index structure on a logical sense in SolrCloud cluster. It is often divided into one or more Shard (fragments), they use the same configuration information.

For example: You can create a collection search for product information.

 collection=shard1+shard2+....+shardX

 

1. 1. 1. Core

Each Core Solr is a standalone unit, providing indexing and search services. A shard required by one or more Core Core composition. Since the collection of a plurality of collection so the shard is generally composed of a plurality of core components.

1. 1. 1. Master或Slave

Master is the master node in master-slave configuration (typically said primary server), Slave node from the master-slave structure (typically from a server or said backup server). Shard same data at the master and slave storage is consistent, the purpose of which is to achieve high availability.

1. 1. 1. Shard

Collection of logical fragmentation. Each Shard is into one or more replication, which is determined by the electoral Leader.

 

 

1. 1. The need to achieve solr cluster architecture

 

Zookeeper as a cluster management tool.

1. Cluster management: fault tolerance, load balancing.
2. Centralized management of configuration files
3. Entrance cluster

 

Zookeeper need to achieve high availability. We need to build clusters. Proposal is odd node. It requires three zookeeper server.

 

Solr clusters need to build seven servers.

 

Build a pseudo-distributed:

It requires three nodes zookeeper

Tomcat need four nodes.

 

The contents of the virtual machine is recommended more than 1G.

1. Preparing the Environment

    CentOS-6.5-i386-bin-DVD1.iso

jdk-7u72-linux-i586.tar.gz

    apache-tomcat-7.0.47.tar.gz

    zookeeper-3.4.6.tar.gz

    solr-4.10.3.tgz

1. installation steps
   1. Zookeeper Cluster Setup

The first step: the need to install jdk environment.

Step two: the zookeeper's archive uploaded to the server.

The third step: decompression.

Step four: three copies of the zookeeper.

[root@localhost ~]# mkdir /usr/local/solr-cloud

[root@localhost ~]# cp -r zookeeper-3.4.6 /usr/local/solr-cloud/zookeeper01

[root@localhost ~]# cp -r zookeeper-3.4.6 /usr/local/solr-cloud/zookeeper02

[root@localhost ~]# cp -r zookeeper-3.4.6 /usr/local/solr-cloud/zookeeper03

Step 5: Create a data directory for each zookeeper in the directory.

Step Six: Create a myid file in the data directory, file name called "myid". Content is the id of each instance. For example, 2, 3

[root@localhost data]# echo 1 >> myid

[root@localhost data]# ll

total 4

-rw-r--r--. 1 root root 2 Apr  7 18:23 myid

[root@localhost data]# cat myid

1

Step Seven: modify the configuration file. Zoo_sample.cfg the renamed files in the conf directory zoo.cfg

 

server.1=192.168.25.154:2881:3881 server.2=192.168.25.154:2882:3882 server.3=192.168.25.154:2883:3883

 

Step eight: Start each zookeeper instance.

Start bin / zkServer.sh start

 

View zookeeper status:

bin/zkServer.sh status

 

1. 1. Solr built clusters

Step 1: Create four tomcat instance. Each tomcat running on different ports. 8180,8280,8380,8480

Step Two: Deploying solr war package. Copy the stand-alone version of solr project to cluster in tomcat.

Step 3: Create a corresponding solrhome solr for each instance. To use the single copy of solrhome four.

Step four: solr need to modify the web.xml file. Associate solrhome.

Step Five: Configuring related solrCloud. Under each solrhome has a solr.xml, to which the ip and port number configured.

 

Step Six: Let the zookeeper integrated management configuration file. Need to solrhome / collection1 / conf directory to upload to the zookeeper. Upload any solrhome configuration file.

Use tools to upload the configuration file: /root/solr-4.10.3/example/scripts/cloud-scripts/zkcli.sh

./zkcli.sh -zkhost 192.168.25.154:2181,192.168.25.154:2182,192.168.25.154:2183 -cmd upconfig -confdir /usr/local/solr-cloud/solrhome01/collection1/conf -confname myconf

Check the configuration file on the zookeeper:

Use bin / zkCli.sh zookeeper catalog command in the configuration file on the zookeeper:

[root@localhost bin]# ./zkCli.sh

[zk: localhost:2181(CONNECTED) 0] ls /

[configs, zookeeper]

[zk: localhost:2181(CONNECTED) 1] ls /configs

[myconf]

[zk: localhost:2181(CONNECTED) 2] ls /configs/myconf

[admin-extra.menu-top.html, currency.xml, protwords.txt, mapping-FoldToASCII.txt, _schema_analysis_synonyms_english.json, _rest_managed.json, solrconfig.xml, _schema_analysis_stopwords_english.json, stopwords.txt, lang, spellings.txt, mapping-ISOLatin1Accent.txt, admin-extra.html, xslt, synonyms.txt, scripts.conf, update-script.js, velocity, elevate.xml, admin-extra.menu-bottom.html, clustering, schema.xml]

[zk: localhost:2181(CONNECTED) 3]

drop out:

[zk: localhost:2181(CONNECTED) 3] quit

 

Step Seven: Modify catalina.sh file in the tomcat / bin directory, associated solr and zookeeper.

Add this configuration to the configuration file:

JAVA_OPTS="-DzkHost=192.168.25.154:2181,192.168.25.154:2182,192.168.25.154:2183"

 

 

 

Step eight: Start each tomcat instance. To package zookeeper cluster is activated state.

Step 9: Access cluster

 

Step 10: Create a new Collection fragmented process.

http://192.168.25.154:8180/solr/admin/collections?action=CREATE&name=collection2&numShards=2&replicationFactor=2

 

 

Tenth step: remove unused Collection.

http://192.168.25.154:8180/solr/admin/collections?action=DELETE&name=collection1

 

 

 

1. Using cluster management solrJ
   1. Adding documents

Steps for usage:

The first step: add solrJ jar package related to the project.

Step 2: Create a SolrServer object, you need to use CloudSolrServer subclasses. Argument constructor is the zookeeper's address list.

The third step: the need to set DefaultCollection property.

Step Four: Create a SolrInputDocument object.

Step Five: Add a domain to a document object

Step Six: The document object into the index database.

Step Seven: Submit.

 

@Test public void testSolrCloudAddDocument() throws Exception { // first step: add solrJ jar package related to the project. // Step 2: Create a SolrServer object, you need to use CloudSolrServer subclasses. Argument constructor is the zookeeper 's address list. // parameter is the zookeeper address list, separated by commas CloudSolrServer solrServer = new CloudSolrServer("192.168.25.154:2181,192.168.25.154:2182,192.168.25.154:2183"); // third step: the need to set DefaultCollection property. solrServer.setDefaultCollection("collection2"); // Step Four: Create a SolrInputDocument object. SolrInputDocument document = new SolrInputDocument(); // Step Five: Add a domain to a document object the Document .addField ( "ITEM_TITLE" , "test product" ); document.addField("item_price", "100"); document.addField("id", "test001"); // Step Six: The document object into the index database. solrServer.add(document); // Step Seven: Submit. solrServer.commit();   }

 

1. 1. Query document

Creating a CloudSolrServer objects, consistent processing and other stand-alone version.

 

1. The search function is switched to the Cluster Edition

<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:context="http://www.springframework.org/schema/context" xmlns:p="http://www.springframework.org/schema/p" xmlns:aop="http://www.springframework.org/schema/aop" xmlns:tx="http://www.springframework.org/schema/tx" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans4.2.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context4.2.xsd http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop4.2.xsd http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx4.2.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util4.2.xsd">   <!-- 单机版solr服务配置 --> <!-- <bean id="httpSolrServer" class="org.apache.solr.client.solrj.impl.HttpSolrServer"> <constructor-arg name="baseURL" value="http://192.168.25.154:8080/solr"></constructor-arg> </bean> --> <!-- 集群版solr服务 --> <bean id="cloudSolrServer" class="org.apache.solr.client.solrj.impl.CloudSolrServer"> <constructor-arg name="zkHost" value="192.168.25.154:2181,192.168.25.154:2182,192.168.25.154:2183"></constructor-arg> <property name="defaultCollection" value="collection2"></property> </bean> </beans>