大数据项目实战之 --- 某购物平台商品实时推荐系统（三）

一、Centos安装nginx  s100 - s400
---------------------------------------------------------------
   1.在win7安装nginx
      解压即可(任何目录即可)。

   2.在centos上安装nginx
      a)先安装G++
         $>sudo yum install gcc

      b)安装pcre
         方法一、通过yum安装(推荐)
            $>sudo yum install pcre-static.x86_64

         方法二、手动通过源代码编译安装(不使用)
            $>tar -xzvf pcre-8.32.tar.tz -C ~
            $>cd ~/pcre-8.32
            $>sudo ./configure --prefix=/soft/pcre-8.32

   3.安装nginx
         方法一、通过yum安装(无)
            $>sudo yum install nginx

         方法二、手动通过源代码编译安装(使用该项)
            $>tar -xzvf nginx-1.6.3.tar.tz -C ~
            $>cd ~/nginx-1.6.3
            $>sudo ./configure --prefix=/soft/nginx-1.6.3 --without-http_gzip_module
            $>sudo make && make install
            $>sudo ldconfig

   4.配置环境变量
      [/etc/profile]
      ...
      export PATH=$PATH:/soft/nginx-1.6.3/sbin

   5.启动nginx服务器
      $>cd /soft/nginx-1.6.3/sbin
      $>sudo ./nginx             //启动服务器

   6.停止服务器
      $>sudo ./nginx -s stop          //停止服务器
      $>sudo ./nginx -s reload         //重新加载服务器
      $>sudo ./nginx -s reopen         //重新打开服务器
      $>sudo ./nginx -s quit          //退出服务器

   6.通过浏览器访问nginx网页,出现nginx欢迎页面。
      http://s100:80/


二、Ubuntu上安装nginx
---------------------------------------
    1.使用root安装

    2.安装g++编译器
        $>sudo apt-get intsall g++

    3.安装pcre-8.32(root)
        $>tar -xzvf pcre-8.32.tar.gz
        $>sudo ./configure --prefix=/soft/pcre-8.32
        $>sudo make && make install
        $>sudo ldconfig

    4.安装nginx
        $>tar -xzvf nginx-1.63.tar.gz
        $>sudo ./configure --prefix=/soft/nginx-1.6.3 --without-http_gzip_module
        $>sudo make && make install$>sudo ldconfig


三、nginx安装脚本
---------------------
   [nginx_install.sh]
   #!/bin/bash
   sudo yum install -y pcre-static.x86_64

   #
   cp /mnt/hgfs/downloads/nginx-soft/nginx-1.6.3.tar.gz ~
   cd ~
   tar -xzvf nginx-1.6.3.tar.gz -C .
   cd nginx-1.6.3
   sudo ./configure --prefix=/soft/nginx-1.6.3 --without-http_gzip_module
   sudo make && make install
   sudo ldconfig


四、安装tomcat  s100 - s400
------------------------------------------
    略


五、启动s100- s400 的nginx集群
---------------------------------------------
    1.s100切换到root
        $>su root

    2.在usr/bin下创建nginx的符号链接
        $> su root
        $> xcall.sh "ln -s /soft/nginx/sbin/nginx  /usr/local/bin/nginx"
        $> xcall.sh "nginx"


六、配置win7的nginx反向代理服务器
-------------------------------------------
       [nginx/conf/nginx.conf]
    #user  nobody;
    worker_processes  1;

    #error_log  logs/error.log;
    #error_log  logs/error.log  notice;
    #error_log  logs/error.log  info;

    #pid        logs/nginx.pid;


    events {
        worker_connections  1024;
    }


    http {
        include       mime.types;
        default_type  application/octet-stream;

        #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
        #                  '$status $body_bytes_sent "$http_referer" '
        #                  '"$http_user_agent" "$http_x_forwarded_for"';

        #access_log  logs/access.log  main;

        sendfile        on;
        #tcp_nopush     on;

        #keepalive_timeout  0;
        keepalive_timeout  65;

        #gzip  on;
        upstream nginx_servers{
            server 192.168.43.131:80 max_fails=2 fail_timeout=2 weight=1;
            server 192.168.43.132:80 max_fails=2 fail_timeout=2 weight=4;
            server 192.168.43.133:80 max_fails=2 fail_timeout=2 weight=4;
            server 192.168.43.134:80 max_fails=2 fail_timeout=2 weight=4;
        }

        upstream tomcat_servers{
            server 192.168.43.131:9090 weight=1;
            server 192.168.43.132:9090 weight=4;
            server 192.168.43.133:9090 weight=4;
            server 192.168.43.134:9090 weight=4;
        }

        server {
            listen       80;
            server_name  localhost;
            access_log off ;
            location ~* \.(png|html|js|css)$ {
                proxy_pass http://nginx_servers;
            }
            location / {
                proxy_pass http://tomcat_servers;
            }
        }
    }


七、s201-s204 tomat的端口需要修改
----------------------------------------------
   [tomcat/conf/server.xml]共两处.
   8080 --> 9090

八、s201-s204 nginx的servername的需要修改成具体ip
-----------------------------------------------
   [nginx/conf/nginx.conf]
   http{
      ...
        log_format  main  '$remote_addr|||$remote_user|||$time_local|||$request|||'
                          '$status|||$body_bytes_sent|||$http_referer|||'
                          '$http_user_agent|||$http_x_forwarded_for';

      access_log  logs/access.log  main;

      server{
         ...
         server-name 192.168.231.201;
         ...
      }
   }

九、nginx常用命令
--------------------------------------------------
   $> nginx   -t          //检查配置是否正确
   $> nginx            //启动
   $> nginx   -s reload     //重新加载配置文件
   $> nginx   -s stop          //停止服务器


十、生成日志
----------------------------------------------------------
    1.将所有access.log文件的内容清除
        [soft/nginx/logs/access.log]
        $> su root
        $> cd /soft/nginx/logs
        $> xcall.sh "rm -rf /soft/nginx/logs/access.logs"
        $> xcall.sh "touch /soft/nginx/logs/access.logs"


    2.通过浏览器访问192.168.231.201:80,查看日志生成的数据
        [win7]
        http://192.168.231.201:80

    3.win7上启动反向代理nginx服务器，实现负载均衡
        D:\MyProgram\nginx-1.6.3 > nginx.exe
        cmd > netstat -ano | grep 80

十一、部署web模块到tomcat集群
-------------------------------------------------------------
    1.pom.xml中添加打war包命令
    [pom.xml]
    <?xml version="1.0" encoding="UTF-8"?>
    <project xmlns="http://maven.apache.org/POM/4.0.0"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
        <modelVersion>4.0.0</modelVersion>
        <groupId>com.test</groupId>
        <artifactId>eshop</artifactId>
        <version>1.0-SNAPSHOT</version>

        <packaging>war</packaging>
        <build>
            <plugins>
                <plugin>
                    <artifactId>maven-war-plugin</artifactId>
                    <version>2.6</version>
                    <configuration>
                        <warSourceDirectory>web</warSourceDirectory>
                        <failOnMissingWebXml>false</failOnMissingWebXml>
                        <excludes>css/*,images/*,js/*,png/*,phone/*</excludes>
                    </configuration>
                </plugin>
            </plugins>
        </build>

        <dependencies>
            <dependency>
                <groupId>junit</groupId>
                <artifactId>junit</artifactId>
                <version>4.11</version>
            </dependency>
            <dependency>
                <groupId>c3p0</groupId>
                <artifactId>c3p0</artifactId>
                <version>0.9.1.2</version>
            </dependency>
            <dependency>
                <groupId>org.hibernate</groupId>
                <artifactId>hibernate-core</artifactId>
                <version>5.2.2.Final</version>
            </dependency>
            <dependency>
                <groupId>org.springframework</groupId>
                <artifactId>spring-context-support</artifactId>
                <version>4.3.3.RELEASE</version>
            </dependency>
            <dependency>
                <groupId>org.springframework</groupId>
                <artifactId>spring-jdbc</artifactId>
                <version>4.3.3.RELEASE</version>
            </dependency>
            <dependency>
                <groupId>org.springframework</groupId>
                <artifactId>spring-orm</artifactId>
                <version>4.3.3.RELEASE</version>
            </dependency>
            <dependency>
                <groupId>org.springframework</groupId>
                <artifactId>spring-webmvc</artifactId>
                <version>4.3.3.RELEASE</version>
            </dependency>
            <dependency>
                <groupId>org.springframework</groupId>
                <artifactId>spring-tx</artifactId>
                <version>4.3.3.RELEASE</version>
            </dependency>
            <dependency>
                <groupId>org.aspectj</groupId>
                <artifactId>aspectjweaver</artifactId>
                <version>1.8.10</version>
            </dependency>
            <dependency>
                <groupId>mysql</groupId>
                <artifactId>mysql-connector-java</artifactId>
                <version>5.1.17</version>
            </dependency>
            <dependency>
                <groupId>javax.servlet</groupId>
                <artifactId>servlet-api</artifactId>
                <version>2.5</version>
            </dependency>
            <dependency>
                <groupId>javax.servlet</groupId>
                <artifactId>jstl</artifactId>
                <version>1.2</version>
            </dependency>
        </dependencies>
    </project>

    2.修改项目中的jdbc.properties文件的mysql地址。
        [eshop/resources/jdbc.properties]
        jdbc.driverclass=com.mysql.jdbc.Driver
        jdbc.url=jdbc:mysql://192.168.231.1:3306/eshop
        jdbc.username=mysql
        jdbc.password=mysql

        c3p0.pool.size.max=10
        c3p0.pool.size.min=2
        c3p0.pool.size.ini=3
        c3p0.pool.size.increment=2

        hibernate.dialect=org.hibernate.dialect.MySQL5Dialect
        hibernate.show_sql=true

    3.使用maven导出成war包

    4.分发war到s201 - s204四台主机的/soft/tomcat/webapps/eshop.war位置

    5.启动tomcat服务器
        $> xcall.sh "/soft/tomcat/bin/startup.sh"

    6.访问web程序
        http://s100:9090/eshop/admin/userlist        //ok
        http://s100:9090/eshop/admin/userlist --> 编辑   //wrong 404 ->http://s100:9090/eshop/admin/editUser
        //如果出现404的错误，请更改jsp页面上所有的url修改成c:URL标签库
        动态获取上下文

十二、部署静态资源到nginx集群
------------------------------------------
   1.复制web目录到eshop目录

   2.删除jsps和WEB-INF子目录，只留js+css|images|phone

   3.分发eshop文件夹到所有nginx/html/下
      $>cd /soft/nginx/html
      $>xsync.sh eshop            //不要加/

   4.通过浏览器查看
      [win7]
      http://localhost:80/eshop/phone/iphone7.html

   5.查看nginx集群上的log信息


十三、使用ab做压测
--------------------------------------------------------------
   1.安装httpd-2.2.21-win32-x86-no_ssl.msi试软件。

   2.查看ab.exe的帮助.
      cmd>cd ab/bin
      cmd>ab /?

   3.使用ab进行压力测试
      //-c : 设置并发度,同时访问的在线人数
      //-n : 设置总的访问请求数。
      cmd>ab -c 100 -n 100000 http://localhost:80/eshop/phone/huawei.html


十四、cron机制实现nginx的日志滚动--调度
---------------------------------------------------
    1.时钟同步
        $>sudo yum install ntp
        $>sudo ntpdate time.nist.gov

    2.设置时区
       $>sudo cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

    3.配置cron
        a.编写执行脚本
            [/usr/local/bin/testcron.sh]
            #!/bin/bash
            echo hello > /home/centos/1.txt

        b.编辑cron配置
            $>sudo nano /etc/crontab
                [/etc/crontab]
                ...
                PATH=...:/usr/local/bin
                ...
                * * * * * centos testcron.sh

             保存退出。

        c.重启crontab服务器
            $>sudo service crond restart   //重启服务
            $>sudo service crond status       //查看状态
            $>sudo service crond restart   //stop

        d.查看~/1.txt

    4.编写脚本，实现日志滚动
        [/usr/local/bin/rolllog.sh]
        #!/bin/bash
        #
        dataformat=`date +%Y-%m-%d-%H-%M`
        #
        cp /soft/nginx/logs/access.log /soft/nginx/logs/access_$dataformat.log
        host=`hostname`
        sed -i 's/^/'${host}',&/g' /soft/nginx/logs/access_$dataformat.log
        #
        lines=`wc -l < /soft/nginx/logs/access_$dataformat.log`
        # move access-xxx.log flume's spooldir
        mv /soft/nginx/logs/access_$dataformat.log /soft/nginx/logs/flume
        #delete rows
        sed -i '1,'${lines}'d' /soft/nginx/logs/access.log
        #reboot nginx , otherwise log can not roll.
        kill -USR1 `cat /soft/nginx/logs/nginx.pid`

    5.cron定时执行rolllog.sh
        [/etc/crontab]
        min    hour  day of m  month   day of week  user        commond
        *       *       *       *       *           user        commond             //每分钟执行一次
        0       2       *       *       *           ubuntu      /home/xxx.sh        //每天的凌晨2点执行一次
       *表示每一分钟，执行一次
       15表示第15分钟，执行一次
       0表示第0分钟，执行一次
       16-18,20表示第16~18分钟和20分钟，各执行一次。总共执行三次

    6.注意事项
        a.修改nginx/logs/flume文件夹权限为777
           flume可以重命名flume下的文件。
           $>chmod 777 /soft/nginx/logs/flume

        b.权限问题
           crontab文件配置是针对root.
           nginx/logs文件夹以及以下是777权限.(递归)