Number of users: 10000-12000 (the number of users has increased rapidly)
PV: 1000000-5000000 (the total number of visits in 24 hours)
QPS: 50-100* (the number of visits per second) 300
DAU: 2000~*000 (daily active users number)

According to business needs:

User access logs are stored in the web server access.log

Statistics of daily visits, peak visits

Store each access in mongodb, and mongodb is used to filter logs

(2) Simulation operation and maintenance design scheme

According to the above business requirements, the mongodb database is used to store user access logs, and a single server is used

①Access log storage

② Filter and view information in mongo

2. Introduction to Mongodb

(1) Introduction to nosql

Database Ranking: https://db-engines.com/en/ranking

The relational database RDBMS designs the table structure and operates through SQL statements. Linked table relationship

Common relational databases: mysql oracle (commercial) DB2 (IBM) sqlserver (Microsoft) access (Microsoft) sqlite3 (small embedded in APP) postgresql (University of California, Berkeley)

nosql refers to non-relational database storage format key=>value

memcached redis memory cache database

mongodb has more functions and can be applied to most mysql scenarios document store document database

(2) Product Features

1. Storage

It is more suitable for storing large amounts of irregular and unordered data.

Large storage capacity : a single table can store PB-level data

1KB = 1024B

1MB = 1024KB

1GB = 1024MB

1TB = 1024GB

1PB = 1024TB

1EB (Exabyte) = 1024PB,

1ZB (Zettabyte ten trillion terabytes) = 1024EB,

1YB (Yottabyte 1 billion billion bytes) = 1024ZB,

2. Efficiency

Data efficiency refers to storage and read/write speed.

3. Structure

3. Installation and configuration

(1) Installation method introduction

Installation via yum: Install MongoDB Community Edition on Red Hat or CentOS — MongoDB Manual

Manual Universal Installation: Install MongoDB Community Edition on Linux — MongoDB Manual

(2) Binary executable installation

1. Upload the installation package to the server directory

2. Unzip to the installation directory and move

shell > tar xvf mongodb-linux-x86_64-rhel62-3.6.5.tgz
shell > mv mongodb-linux-x86_64-rhel62-3.6.5 /usr/local/mongodb

3. Create data storage directory and log directory

shell > cd /usr/local/mongodb

shell > mkdir data

shell > mkdir logs

4. Start the mongod service

shell > cd /usr/local/mongodb/bin

shell > ./mongod --dbpath=/usr/local/mongodb/data --logpath=/usr/local/mongodb/logs/log.txt --fork

Parameter introduction:

dbpath data storage path

logpath log storage path

fork background start

auth permission enabled

bind_ip specifies the binding network card ip

(3) Command line client operation

4. Data structure type operation CURD

(1) Add data

1. The format document format of data stored in mongodb is in the form of bson format document.

In mongodb, there is no need to create databases and collections, they will be created automatically when used

Create a library devops

2. Syntax: db.collection name.insert (data in bson format)

3. Common data addition

db.goods.insert({name:'huawei01',price:1000,weight:135,number:35})

4. Multidimensional data object addition

db.goods.insert({name:'xiaomi5',price:1999,weight:156,number:45,area:{province:'beijing',city:'beijing'}})

5. Add array information

db.goods.insert({name:'xiaomimax',price:2000,weight:180,number:100,area:{province:'henan',city:'zhengzhou'},color:['black','white','red']})

(2) Query data

1. Grammar:

db.collection name.find(query condition)

db.collection name.findOne(query condition)

2. Query in a general way

No conditional restrictions, all taken out.

findOne will take out the first piece of information that matches the result and return it in a formatted form

3. Conditional restriction query

Conditional formatting, what you see and what you get
db.goods.find({name:'xiaomimax'})
db.goods.findOne({name:'xiaomimax'})

4. Range condition query

mysql < <= > >= !=

mongo $lt $lte $gt $gte $ne

db.goods.find({price:{'$lte':1999}})

db.goods.find({price:{'$lt':1999}})

5. Multiple conditional query

Similar to AND syntax in mysql

db.goods.find({price:{'$lte':1999},number:{'$gte':40}})

6. Multi-dimensional field query

Query by value of multidimensional field

db.goods.find({'area.city':'zhengzhou'});

Note: Multidimensional fields need to be enclosed in quotes

7. Array conditional query

(1) The query can be displayed if it satisfies one of them

db.goods.find({color:'black'})

(2) Only those that meet the query conditions can be displayed

db.collection name.find({field(array):{'$all':[v1,v2]}})

db.goods.insert({name:'iphonex',price:8398,weight:100,number:200,area:{province:'jiangsu',city:'nanjing'},color:['gray','black','white','gold']})

db.goods.find({color:{'$all':['black','gold']}})

8. Restrict query fields

In the actual use environment, there is no need to query and display too many fields. You can choose to set the display.

Syntax: db.collection name.find({query condition},{filter condition})

Displayed as 1, not displayed as 0 If 1 is 1, if 0 is 0 except for _id

db.goods.find({color:{'$all':['black','gold']}},{name:1,_id:0})

Pay attention to the appropriate way to display the content of the query field value

_id is the default primary key id in the collection in the mongodb database, which has index content, and the query speed will be very fast through the primary key query. Do not modify this value arbitrarily, just use the default.

9. $or query

It can be displayed if one of the conditions is met, similar to the or conditional syntax in mysql
select * from goods where price > 5000 or number >= 100

db.goods.find({'$or':[{price:{'$gt':5000}},{number:{'$gte':100}}]})

10. count syntax

The number of results returned

chain operation

db.goods.count()
db.goods.find({price:{'$gt':5000}}).count()
db.goods.count({price:{'$gt':5000}})

11. limit syntax skip syntax

Similar to the limit (skip, length) syntax limit() in mysql, take a few
skip() and skip a few
db.goods.find().limit(1);

db.goods.find().skip(1).limit(1);

Comparison of mongodb syntax and SQL statement

(3) Modify data

1. Grammar:

db.collection name.update({query condition},{modification condition})
db.collection name.updateOne() modify the first matching item
db.collection name.updateMany() modify all matching items

2. There is a modification of $set

db.goods.update({name:'iphonex'},{'$set':{price:8500}})

db. Modified goods.updateOne({name:'xiaomi5'},{'$set':{price:1500}}) without $set

3. No modification of $set

There is no $set keyword syntax, modify the set field, and the unset field will be deleted

db.goods.update({name:'iphonex'},{price:8550})

(4) Delete data

1. Delete records

db.collection name.remove({query condition})

db.collection name.deleteOne() delete the first matching item

db.Collection name.deleteMany() Delete multiple matching items

db.goods.remove({price:8550})

db.goods.deleteMany({price:{'$lte':2000}})

2. Delete field

The operation that can delete a certain field uses the $unset of the update syntax

db.goods.update({name:'huawei01'},{'$unset':{weight:0}})

Give a value to delete

In real business, physical deletion is generally not performed, and an identifier is used to confirm whether the data has been deleted

5. Security settings

https://docs.mongodb.com/manual/tutorial/create-users/

mongodb security incident: https://www.jianshu.com/p/48d17a69e190

(1) Restricted login

1. Use another virtual machine to test using the mongo command line

Remote login method

2. Close mongodb

Do not kill -9 mongod under normal circumstances, use the shutdown command in the mongo command client

3. Add the startup script to start mongod

shell > vim /etc/init.d/mongodb

bind binding is the network card for external network communication --bind_ip_all means binding any address of 0.0.0.0

4. Use remote login mongod service

(2) User rights management

1. Demand

Set up a super administrator account with read and write permissions for all libraries

2. Grammar:

db.createUser({user:"root",pwd:"root",roles:["root"]})

3. Implementation steps:

①Switch the admin library to create users

② Close the mongod service, restart to add permission parameters and start

closure

Add the --auth parameter to the startup script

③Test use

Add users, limit the permissions of the library, read-only

6. Business Examples

(1) PHP extension installation

1. Upload the php extension package

If there is no phpize command, you need yum to install php-devel

2. Unzip, compile and install

shell > yum -y install php-devel #安装phpize
shell > tar xvf mongodb-1.5.3.tgz
shell > cd mongodb-1.5.3
shell > phpize
shell > ./configure && make && make install

3. Add php.ini matching

shell > vim /usr/local/php/etc/php.ini

4. Restart php-fpm to view phpinfo

shell > service php-fpm restart

(2) Use of desktop management

1. The software is a green version, unzip it directly, and click to use it

2. Fill in relevant parameters

3. Test whether it can be connected normally

4. Click to connect

5. View data

(3) Log statistics display

1. Many websites need to count the number of visits to a certain page or function. If the number of visits is relatively large, the components that need to be recorded have fast read and write efficiency. This business can be done with mongodb which is more functional. mongodb has more data flexibility.

2. Information recorded in the website access log:

ip source of user access ip
url address of user access function module page address
time access time record user access time value
user_agent user access client information

3. Use the server

web1（server01）->mycat（server07）->master（server02）->slave（server06）->cache（server08）->mongodb（server10）

4. Tip: The nginx load balancing server solves the problem that the backend service cannot obtain the real user IP

①In the load balancing nginx server, fill in the forwarding of the real user IP

Note that if load balancing is used, the IP obtained by the backend is the IP of the load balancing agent.

If the backend needs to obtain the real IP, the load balancing server must configure parameters to transmit the real IP

②The nginx of the web server needs to be configured to receive the real user IP, and the module --with-http_realip_module needs to be used, which needs to be added when compiling and installing nginx

③Configuration syntax

④Operation implementation steps:

(1) Delete the original file

(2) Upload the modified file

(3) After visiting the page, you can see that the visit has been recorded

The method of obtaining the real IP of the proxy in php, but it must be noted that the proxy server must have the real IP of the fax

MongoDB for enterprise architecture NOSQL database

1. Background description and scheme design

(1) Description of business background

(2) Simulation operation and maintenance design scheme

2. Introduction to Mongodb

(1) Introduction to nosql

(2) Product Features

1. Storage

2. Efficiency

3. Structure

3. Installation and configuration

(1) Installation method introduction

(2) Binary executable installation

1. Upload the installation package to the server directory

2. Unzip to the installation directory and move

3. Create data storage directory and log directory

4. Start the mongod service

(3) Command line client operation

4. Data structure type operation CURD

(1) Add data

1. The format document format of data stored in mongodb is in the form of bson format document.

2. Syntax: db.collection name.insert (data in bson format)

3. Common data addition

4. Multidimensional data object addition

5. Add array information

(2) Query data

1. Grammar:

2. Query in a general way

3. Conditional restriction query

4. Range condition query

5. Multiple conditional query

6. Multi-dimensional field query

7. Array conditional query

(1) The query can be displayed if it satisfies one of them

(2) Only those that meet the query conditions can be displayed

8. Restrict query fields

9. $or query

10. count syntax

11. limit syntax skip syntax

(3) Modify data

1. Grammar:

2. There is a modification of $set

3. No modification of $set

(4) Delete data

1. Delete records

2. Delete field

5. Security settings

(1) Restricted login

1. Use another virtual machine to test using the mongo command line

2. Close mongodb

3. Add the startup script to start mongod

4. Use remote login mongod service

(2) User rights management

1. Demand

2. Grammar:

3. Implementation steps:

①Switch the admin library to create users

② Close the mongod service, restart to add permission parameters and start

③Test use

6. Business Examples

(1) PHP extension installation

1. Upload the php extension package

2. Unzip, compile and install

3. Add php.ini matching

4. Restart php-fpm to view phpinfo

(2) Use of desktop management

1. The software is a green version, unzip it directly, and click to use it

2. Fill in relevant parameters

3. Test whether it can be connected normally

4. Click to connect

5. View data

(3) Log statistics display

2. Information recorded in the website access log:

3. Use the server

4. Tip: The nginx load balancing server solves the problem that the backend service cannot obtain the real user IP

①In the load balancing nginx server, fill in the forwarding of the real user IP

②The nginx of the web server needs to be configured to receive the real user IP, and the module --with-http_realip_module needs to be used, which needs to be added when compiling and installing nginx

③Configuration syntax

④Operation implementation steps:

(1) Delete the original file