Cloud the Spring
Eureka
to configure the port number and the name and address of the service registry in the configuration file,
and then add comment @EnableEurekaClent configuration similar center on the main startup class
Ribbon
time client load balancing that is when the browser makes a request to the background, customers end to Eureka server reads the list of available service registration information to the server, and then according to the load balancing policy set (ie, not set by default), a decision which sends a request to the server.
In the new class in a master boot method, the return value RestTemplate object and the label will @bean injection method vessel while load balancing function through the opening @LoadBalanced label.
Feign
added first dependency
and add @EnableFeignClients open load process of scanning in the main Feign Clent startup class
defines interfaces and Feign on the interface specified by the service name and annotation @FeignClients url
injection controller interface layer, the interface when the method is called when Feign will generate a RequestTemple for each interface objects
which encapsulates all the information required for the HTTP request includes a parameter name request method;
then generates RequestTemplate requested by the object, and then to process the request to the Client
last Client class is encapsulated LoadBalanceclient this type of binding Ribbon load balancing initiate calls between services.
Hystrix
arranged on the main circuit breaker @EnableCiruitBreaker startup class
Then add annotation @HystrixCommand need to use the method of circuit breaker
filled fallbackMethod in parentheses = pre-prepared return result
Zuul
Zuul Netflix is produced based on a JVM routing and server load balancer.
Can be used to make trans the agents can also be used to filter requests and filter
to filter requests: configure routing rules yml the
nginx server
concept: the nginx web server is a reverse proxy server for HTTP, HTTPS, SMTP, POP3 and IMAP protocols.
Features: Cross-platform, simple configuration, non-blocking, high concurrent connections, memory consumption is small, performance is good
use: a reverse proxy, load balancing (polling, weight, iphash), static and dynamic separation
to achieve high concurrency principle:
a primary process, and more working processes, each worker process can handle multiple requests, each coming in a request, there will be a worker process to deal with. The face of obstruction, to deal with other requests, once the upstream server returns the trigger event and continue processing.
Two processes:
Master process: read and evaluate the configuration and maintenance of
Worker process: process requests
Configuration:
Server {
the listen: port number to listen;
server_name: To monitor the domain name;
/ {LOCATION
proxy_pass need to move the host name and port number;
}
}
configure load balancing policy
upstream name {
host name and port number of the server main steering;
server main steering host name and port number;
server main steering host name and port number;
}
commonly used three strategies (polling, weight, ip_hash)
weight (server host name port weight = n-number +)
IPHash (ip_hash written in the first row;) each fall within a fixed upstream server request, to solve the problem ip on the same server session.
fair (fair at the end of the write line;) configured in the server response time to an upstream allocation request. Short response time priority allocation.
url_hash configuration
mycat middleware
MyCat key features
comply with Mysql native protocol, cross-language, cross-platform, cross-database of generic middleware proxy.
Heart failure based on the automatic switching, reading and writing support separated from the main support MySQL, sub-table support sub-library
based Nio achieve effective management thread, high concurrency issues.
mycat basic configuration (service.xml, rule.xml, scherma.xml):
service.xml main configuration parameters mycat services, such as port number, myact logical database user name and password, etc.
role.xml major routing policy, Split rules;
the schema.xml major physical database configuration information, the relationship between the logical database name and policy routing tables and
sub-sub-table repository
horizontal split (in slicing table size) and the vertical resolution (in lines slicing size)
Redis
supports five data types (string, Hash, List, the Set, zset)
string string:
format: the SET Key value
string type is binary safe. Redis meaning of string can contain any data. Such as jpg image or serialized object.
Redis is the most basic type of string data type can store a maximum 512MB key.
Hash (hash)
format: name hmset VALUE1 key1 key2 value2
the Redis the hash is a key (key => value) pairs.
Redis hash field is a string type and the value of the mapping table, hash is particularly suited for storing objects.
List (list)
the Redis list is a simple list of strings, sort insertion order. You can add an element to the head of the list (on the left) or tails (on the right)
Format: lpush name value
corresponding to the list of head string element is added in key
formats: rpush name value
corresponding to the end of the list to add a string element in the key
format : lrem name index
key corresponding to the list and delete a count value like elements
format: llen name
returns the key corresponding to the length of the list
Set (collection)
Format: Sadd name value
Redis is a string Set the type of unordered collection.
Collection is achieved through a hash table, so add, delete, search complexity is O (1).
zset (sorted set: an ordered collection)
Format: Score Zadd name value
Redis zset and is set as a collection of string type elements, and does not allow duplicate members.
The difference is that a double score will be associated with each type of element. It is to redis from small to large order of collection of member passing score.
zset member is unique, but the score (score) it can be repeated.
What is Redis persistence? Redis, which has several persistent way? What advantages and disadvantages?
Persistence is the memory of the data written to disk to prevent service downtime memory data loss.
Redis persistence provides two ways: RDB (default) and AOF
build redis clusters:
copy multiple redis
start reids group with ruby scripting language
to monitor each node with sentry mechanism to achieve high availability and automatic failover
sentry mechanism:
with heartbeat mechanism to monitor the operation of each node is normal
when there is a problem of a node monitored, Sentinel administrators to send notifications
when a master node does not work, Sentinel will automatically start a failover operation
automatic failover:
when a master node when not working, it will be the master node failure from one node to upgrade to the new master node
and let the failure of the master node from the other node instead copy the new master node;
when a client attempts to connect the master node failure , the cluster will be returned to the client address of the new Master, making the cluster can be used instead of the failed Master Master.
What is Cache penetrate? How to avoid? What is Cache avalanche? Ru avoid?
Cache penetrate the
general caching system, it is in accordance with the key to cache query, if the corresponding value does not exist, they should go to find the back-end systems (such as DB). Some deliberately malicious request query key is not present, a large amount of the request, it will cause a lot of pressure on the back-end systems. This is called caching penetration.
How to avoid?
1: the case where an empty result set also caching and set shorter, or the data key corresponding to the dirty cache after the insert.
2: is not present on certain key filtered. We can put all the possible key into a large Bitmap by the bitmap filter query.
Avalanche cache
when the cache server reboot or a large number of cache concentrated in one time period fail, so that when failure, back-end systems will bring a lot of pressure. Cause the system to crash.
How to avoid?
1: After a cache miss, by locking to control the number of threads or queue database read write cache. For example, a key for allowing only one thread to query the data and write cache, other threads wait.
2: Make secondary cache, cache the original A1, A2 copy is cached, when A1 fails, access to A2, A1 cache expiration time is set short-term, A2 is set for long-term
3: Different different key, set the expiration time for the cache invalidation time points as uniform as possible.
Database Optimization
1.1 as queries based on primary keys
1.2 as single-table queries, the pressure will be accessible to the tomcat server cluster can be solved by the way, greatly reduce the overhead of database
① Select the most efficient sequence table name
resolver database right to left in the order of the processing table name in the fROM clause, written in the fROM clause the table will be the last processed first
②WHERE clause connection sequence
database using the sequence from right to left parsing WHERE clause, according to this principle, the connection between tables must be written in other Zhizuo WHERE conditions, those conditions can filter out the maximum number of records must be written in the WHERE clause of the right hand.
1.3 If the relational query, the data is determined to be early, then relational query, reduce Cartesian product (number associated)
1.4 less database function max / min packet Hive
2. Add indexes and views (Continued)
3. Add buffer ( redis / as MemCache)
3.1 non-relational database
fast read and write speeds 3.2, the throughput per 100,000
4. periodically dump data (not the data stored in the history table (water bank))
the use of high-performance non-relational database storage. (redis / hbase), read and write speed
6. A sub-library sub-table (expensive / O & M)
Database to prevent loss of data?
In archivelog mode (archive mode) as long as the archived log files are not lost, it can effectively prevent data loss.
docker vessel (understand) based on the Go language
you can run on a single machine multiple micro-Docker containers, each container has a micro-micro-service or a standalone application, for example, you can run a Docker Tomcat, and MySQL running in another Docker, both of which may run on the same server or multiple servers.
The project can be directly released for testing in DocKer container above, when the project formally launched, we need to do a good job DocKer can directly image deployment up on the line, DocKer can be deployed in the cloud, Windows, Linux and other environmental
fast start
solr search engine
solr is based Lucence development of enterprise-class search engine technology, and lucence principle is inverted index. So what is the inverted index it? Next we will introduce lucence inverted index principle.
It provides the capability Chinese word.
the schema.xml: Domain configuration information related to
import data from the database:
the corresponding version mysql-connector-java.jar lib package into library directory index file folder solr
arranged in solrconfig.xml file
<requestHandler name = "/ dataimport" class = "org.apache.solr.handler.dataimport.DataImportHandler">
<LST name = "Defaults">
<STR name = "config"> Data-the config.xml </ STR>
</ LST>
</ requestHandler>
Create a data-config.xml file in the current directory
in the file need to import into a database table index
<? XML Version = "1.0" encoding = "UTF-. 8"?>
<DataConfig>
<type the dataSource = "JdbcDataSource"
Driver = "
user="root" password="xuyiqing"/>
<document>
<entity name="user"
query="select * from blog_user" >
<field column="u_id" name="id"></field>
<field column="username" name="username"></field>
<field column="u_password" name="password"></field>
<field column="qq" name="qq"></field>
<field column="avatar" name="avatar"></field>
<field column="article_count" name="count"></field>
</entity>
</document>
</dataConfig>
最后在schema.xml中配置域：
<field name="username" type="text_ik" indexed="true" stored="true"/>
<field name="password" type="text_ik" indexed="false" stored="false"/>
<field name="qq" type="text_ik" indexed="true" stored="true"/>
<field name="avatar" type="string" indexed="false" stored="true"/>
<field name="count" type="float" indexed="true" stored="true"/>
rabbitMQ message queue and kafka
benefits:
can be used to decouple calls between various systems (decoupling)
can be used to ensure that calls between systems instead by a synchronous asynchronous (asynchronous)
when the system is in the treatment of high concurrency, such as spike activity, more than 2,000 requests per second, also the maximum processing capacity of mysql, if that is not the way the message queue, the system will die (clipping)
Cons:
system availability Reduction: the more dependent on external systems introduced, the more easy to hang , you had a system call is three BCD system interfaces like, four people ABCD system properly, nothing issue, you add a partial MQ come in, if MQ hung up zezheng? MQ hung up, the entire system crashes, you do not what would be finished.
Increase system complexity: add MQ abruptly come in, how do you ensure that the message is not repeated consumption? How to deal with the case of message loss? How to ensure that the order of message delivery? The bulk of the big head, a lot of questions, endless pain
consistency: A system to process over direct return is successful, people think that you request will be successful; but the problem is that if the three systems where BCD, BD wrote two systems library is successful, the results of C write library system failed, zezheng? You data is inconsistent.
Message Queue Middleware:
Throughput availability timeliness
the ActiveMQ: ten thousand milliseconds
RabbitMQ: ten thousand lowest delay (microseconds) based master-slave architecture implemented
RocketMQ: 10 ten thousand milliseconds
Kafka: 10 ten thousand (typically with large data system for real-time calculation data, log collection and other scenes) within milliseconds is very high, a plurality of data backup, a small amount of downtime, does not affect
Services Framework Dubbo
Dubbo Alibaba open source high performance RPC-based distributed services framework for Java, has now become the Apache Foundation incubator project.
The difference between Dubbo and Spring Cloud: the
different registries, Dubbo with a strong consistency of Zookeeper, Spring Cloud using a Eureka
Dubbo does not provide a sound breaker, serving gateway functions.
Supported protocols Dubbo:
Dubbo: // (recommended) HTTP: // REST: // Redis: //
Dubbo do not need a Web container
is recommended Zookeeper as a registration center, there Redis, Multicast, Simple registration center, but not recommended.
hadoop ecosystem
Common data structures
Mall project
I mainly to provide some query interface, query the commodity classification and fuzzy query, while calls to other services in the system.
Property Management System
lease management module
main function is to provide real estate information for administrators to track the status of the different states (the owners live, for rent, rented) property presented to the administrator, who can be classified by query and fuzzy query calls query property information. Alert administrators will soon be expiring lease contracts and real estate information.
Administrators need to look at the use of state property, I use to add a field to use state property table, the information can then be classified according to statistics lease state;
administrators need to see which lease contract will expire in the last month, which We need to query, according to the summary information on the property rental Due date field.
Lease Contract Management module
is mainly to lease contract management, contract suspension, renewal, postponed, canceled, change management, etc., can be performed in real-time query statistics.
In fact, on the lease contract table CRUD.
Eureka Client:
First configure the port number and the name and address of the service registry in the configuration file, and then add comment @EnableEurekaClent configuration similar center on the main startup class
Feign:
First add a dependency, and then add @EnableFeignClients on the main startup class open scan Feign Clent the loading process, and the interface is defined Feign specified service name and annotation @FeignClients url used on the interface, the interface controller injection layer, the interface will generate a RequestTemple Feign object for each interface when the method is called, this object encapsulates all the information needed includes a parameter name request HTTP request method; RequestTemplate the object and then by the generation request, and then to process the request to the Client, is encapsulated LoadBalanceclient last Client class, which initiates a service in conjunction with load balancing between Ribbon It calls.
Feign works
in the development of micro-service applications, we will add @EnableFeignClients in the main entrance open for comment Feign Client scan processing load. According to the development of norms Feign Client, the definition of the interface and add @FeignClients comment.
When the program starts, the packet scan, scan all annotations @FeignClients class, and injected into the information Spring IOC container. When the interface definition Feign method is invoked by the agent JDK way to generate specific RequestTemplate. When generating agent, Feign creates an object for each RequetTemplate interface method, the HTTP request object encapsulates all the information needed, such as the request parameter name, the request method and other information is determined in this process.
Is then generated by requestTemplate Request, and then to the Client Request to handle, may be referred to herein Client JDK native URLConnection, Apache may be of the Http Client Okhttp. Finally Client is encapsulated LoadBalanceclient class, launched in conjunction with Ribbon load balancing between service calls.
Synchronous blocking IO (JAVA BIO):
synchronous and blocking, server model to achieve a connection to a thread that has a client connection request on the server side need to start a thread for processing, if this connection will not do anything to cause unnecessary thread overhead, of course, can be improved by a thread pool mechanism.
Synchronous non-blocking IO (Java NIO): non-blocking synchronization, the server requests a mode of realization of a thread, i.e. the connection request sent by the client are registered to the multiplexer, the multiplexer is connected to the polling I when / O request to start a thread for processing. User processes also need time to time to ask whether the IO operation readiness, which requires the user to stop the process of inquiry.
Asynchronous blocking IO (Java NIO):
In this mode refers to the application launched after a IO operation, without waiting for the completion of the core IO operations, such as the kernel will notify the application after the completion of IO operation, which is actually the most crucial synchronous and asynchronous difference , synchronization must wait or take the initiative to ask whether the IO to complete, then why is it blocked? Because at this time is through select system call to complete, and select the function itself is blocking implementation, and use the select function has the advantage that it can monitor handles multiple files at the same time (if viewed from the perspective of UNP, select belong to synchronize operation because then select, the process also need to read and write data), thereby increasing the concurrency of the system!
(Java AIO (NIO.2)) asynchronous non-blocking IO:
In this mode, the user only needs to initiate a process IO operation and then return immediately, and other IO operations after the completion of a true, the application will be notified IO operation is completed, At this time, the user only needs to process the data processed like, does not require the actual IO read and write operations, because the real IO read or write operation has been completed by the kernel.