电信CALL 通话记录hbase kafka flume 学习

可视化:
--------------
   1.
   2.
   3.
   4.
   5.
package com.it18zhang.callloggen;

import java.util.HashMap;
import java.util.Map;

/**
*
*/
public class App {
public static Map<String,String> caller = new HashMap<String, String>();
static{
caller.put("15810092493", "史玉龙");
caller.put("18000696806", "赵贺彪");
caller.put("15151889601", "张倩 ");
caller.put("13269361119", "王世昌");
caller.put("15032293356", "张涛");
caller.put("17731088562", "张阳");
caller.put("15338595369", "李进全");
caller.put("15733218050", "杜泽文");
caller.put("15614201525", "任宗阳");
caller.put("15778423030", "梁鹏");
caller.put("18641241020", "郭美彤");
caller.put("15732648446", "刘飞飞");
caller.put("13341109505", "段光星");
caller.put("13560190665", "唐会华");
caller.put("18301589432", "杨力谋");
caller.put("13520404983", "温海英");
caller.put("18332562075", "朱尚宽");
caller.put("18620192711", "刘能宗");
}

public static void main(String[] args) {
genCallLog();
}

public static void genCallLog(){

}

生成jar包，部署到centos执行
--------------------------------
   1.使用maven生成jar文件
       ...
   2.部署到centos

   3.执行
       $>mkdir /home/centos/calllog
       java -cp xxx.jar com.it18zhang.callloggen.App /home/centos/calllog/calllog.log

   4.创建centos上的执行脚本
       [calllog.sh]
       #!/bin/bash
       java -cp Calllog.jar com.it18zhang.callloggen.App /home/centos/calllog/calllog.log

   5.修改权限
       $>chmod a+x calllog.sh

   6.执行脚本
       $>cd ~/calllog
       $>./calllog.sh

手机号
---------------
   0755-67568979
   186----------

086+0755-67568979
固话
--------------
0755-67568979
0755-67568979

网络电话
-------------
12358757575765656565

启动zk集群[s201 + s202 + s203]
-------------
$>zkServer.sh start

启动kafka集群[s202 + s203 + s204]
-----------------
$>cd /soft/kafka/config
$>kafka-server-start.sh -daemon server.properties

创建kafka主题
-----------------
//创建主题
$>kafka-topics.sh --zookeeper s202:2181 --topic calllog --create --replication-factor 3 --partitions 4

//查看主题列表
$>kafka-topics.sh --zookeeper s202:2181 --list

//启动控制台消费者,消费calllog主题,用于测试.
$>kafka-console-consumer.sh --zookeeper s201:2181 --topic calllog

s201上编写flume配置文件件，实时收集calllog.log日志
--------------------------------------------
   1.配置文件
       [/soft/flume/conf/calllog.conf]
       a1.sources = r1
       a1.sinks = k1
       a1.channels = c1

       a1.sources.r1.type=exec
       #-F 最后10行,如果从头开始收集 -c +0 -F:持续收集后续数据,否则进程停止。
       a1.sources.r1.command=tail -F -c +0 /home/centos/calllog/calllog.log

a1.channels.c1.type=memory

       a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
       a1.sinks.k1.kafka.topic = calllog
       a1.sinks.k1.kafka.bootstrap.servers = s202:9092 s203:9092 s204:9092
       a1.sinks.k1.kafka.flumeBatchSize = 20
       a1.sinks.k1.kafka.producer.acks = 1

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

   2.启动flume收集程序
       $>flume-ng agent -f /soft/flume/conf/calllog.conf -n a1 &

在s202主机安装flume软件
------------------------
...

启动hadoop hdfs集群[s201 ~ s206]
角色划分:
NameNode       //s201,s206
DataNode       //s202,s203,s204,s205
JournalNode       //s202,s203,s204
ZK               //s201,s206
---------------------------------------------
   1.启动hadoop(完全分布式 + HA,s201 + s206作为主备名称节点)
       在s201上启动dfs集群.
       $>start-dfs.sh

   2.webui
       http://s201:50070/

   3.查看节点状态
       $>hdfs haadmin -getServiceState nn1
   4.容灾切换
       $>hdfs haadmin -failover nn1 nn2

启动hbase集群
-------------------------------
   1.[角色划分]
       master           //s201,s204
       regsionServer   //s202,s203,s204

   2.启动hbase集群
       //s201
       $>start-hbase.sh

3.查看进程和webui
http://s201:16010

   4.启动备份master节点
       //s204
       $>hbase-daemon.sh start master

创建hbase名字空间+表
----------------------
   1.创建名字空间
       $>hbase shell                       //进入hbase shell
       $hbase>create_namespace 'ns1'       //创建空间
       $hbase>create 'ns1:calllogs' , 'f1'   //创建表

$hbase>truncate 'ns1:calllogs' //重建表

IBM Power720(210000)
---------------------
   基本规格
   处理器类型POWER7+
   处理器主频3.6GHz
   处理器缓存每个内核256KB二级缓存，4MB三级缓存
   最大处理器个数8
   内存类型DDR3
   标准内存容量32GB
   最大内存容量128GB
   4颗CPU

去IOE
---------
   IBM:       //ibm小型机
   Oracle:       //oracle数据
   EMC           //网络存储设备

戴尔（DELL）13900 * 100 = 140万
-------------------------------
   商品名称：
   戴尔（DELL）R730服务器
   机架式主机   2U
   至强       E5处理器
   内存类型   ECC
   硬盘总容量：8T以上 8T x 100 = 800T = 副本(3) 800/3 = 250 80%(4/5) = 150T
   类型：
   机架服务器电源：冗余
   操作系统：   DOS
   内存       总容量：64G及以上
   硬盘转速：其他
   支持CPU颗数   2颗
   机箱规格   2U机架式
   显存       集成显卡
   硬盘类型   SAS
   Raid卡
   缓存：       无缓存
   处理器：   至强Xeon-E5

创建kafka消费者，订阅calllog主题
-------------------------------
1.设计rowkey
业务数据: caller , callee , date , duration

       分区号号码时间,标记 (对方号码),时长
       regionNo, caller,date , flag,callee,duration

       caller(11) + 通话时间(201701) =
       (后四位) + 201701 = % 100

   2.
   3.
   4.

11 - 4 = 7

hbase(main):001:0> scan 'ns1:calllogs'
ROW COLUMN+CELL
39,15032293356,20170313140201,0,18620192711,297 column=f1:callDuration, timestamp=1491896679156, value=297

39,15032293356,20170313140201,0,18620192711,297 column=f1:callTime, timestamp=1491896679156, value=20170313140201

39,15032293356,20170313140201,0,18620192711,297 column=f1:callee, timestamp=1491896679156, value=18620192711

39,15032293356,20170313140201,0,18620192711,297 column=f1:caller, timestamp=1491896679156, value=15032293356

93,18620192711,20170313140201,1,15032293356,297 column=f1:dummy, timestamp=1491896679184, value=no

使用mvn命令，下载工件的所有依赖软件包
----------------------------------------
   mvn -DoutputDirectory=./lib
       -DgroupId=com.it18zhang
       -DartifactId=CallLogConsumerModule
       -Dversion=1.0-SNAPSHOT
       dependency:copy-dependencies

导入kafka消费者，放置以上面的lib下，使用命令行方式运行
-----------------------------------------------------
java -cp CallLogConsumerModule.jar;./lib/activation-1.1.jar;./lib/apacheds-i18n-2.0.0-M15.jar;./lib/apacheds-kerberos-codec-2.0.0-M15.jar;./lib/api-asn1-api-1.0.0-M20.jar;./lib/api-util-1.0.0-M20.jar;./lib/avro-1.7.4.jar;./lib/commons-beanutils-1.7.0.jar;./lib/commons-beanutils-core-1.8.0.jar;./lib/commons-cli-1.2.jar;./lib/commons-codec-1.9.jar;./lib/commons-collections-3.2.2.jar;./lib/commons-compress-1.4.1.jar;./lib/commons-configuration-1.6.jar;./lib/commons-digester-1.8.jar;./lib/commons-el-1.0.jar;./lib/commons-httpclient-3.1.jar;./lib/commons-io-2.4.jar;./lib/commons-lang-2.6.jar;./lib/commons-logging-1.2.jar;./lib/commons-math3-3.1.1.jar;./lib/commons-net-3.1.jar;./lib/findbugs-annotations-1.3.9-1.jar;./lib/guava-12.0.1.jar;./lib/hadoop-annotations-2.5.1.jar;./lib/hadoop-auth-2.5.1.jar;./lib/hadoop-common-2.5.1.jar;./lib/hadoop-mapreduce-client-core-2.5.1.jar;./lib/hadoop-yarn-api-2.5.1.jar;./lib/hadoop-yarn-common-2.5.1.jar;./lib/hamcrest-core-1.3.jar;./lib/hbase-annotations-1.2.4.jar;./lib/hbase-client-1.2.4.jar;./lib/hbase-common-1.2.4.jar;./lib/hbase-protocol-1.2.4.jar;./lib/htrace-core-3.1.0-incubating.jar;./lib/httpclient-4.2.5.jar;./lib/httpcore-4.2.4.jar;./lib/jackson-core-asl-1.9.13.jar;./lib/jackson-mapper-asl-1.9.13.jar;./lib/jaxb-api-2.2.2.jar;./lib/jcodings-1.0.8.jar;./lib/jdk.tools-1.6.jar;./lib/jetty-util-6.1.26.jar;./lib/jline-0.9.94.jar;./lib/joni-2.1.2.jar;./lib/jopt-simple-4.9.jar;./lib/jsch-0.1.42.jar;./lib/jsr305-1.3.9.jar;./lib/junit-4.12.jar;./lib/kafka-clients-0.10.0.1.jar;./lib/kafka_2.11-0.10.0.1.jar;./lib/log4j-1.2.15.jar;./lib/lz4-1.3.0.jar;./lib/mail-1.4.jar;./lib/metrics-core-2.2.0.jar;./lib/netty-3.7.0.Final.jar;./lib/netty-all-4.0.23.Final.jar;./lib/paranamer-2.3.jar;./lib/protobuf-java-2.5.0.jar;./lib/scala-library-2.11.8.jar;./lib/scala-parser-combinators_2.11-1.0.4.jar;./lib/slf4j-api-1.6.1.jar;./lib/slf4j-log4j12-1.7.21.jar;./lib/snappy-java-1.1.2.6.jar;./lib/stax-api-1.0-2.jar;./lib/xmlenc-0.52.jar;./lib/xz-1.0.jar;./lib/zkclient-0.8.jar;./lib/zookeeper-3.4.6.jar com.it18zhang.calllog.consumer.HbaseConsumer

编写web程序，从hbase中提取所有进行展示，可视化
---------------------------------------
   1.导入ssm项目
       ...
   2.创建CallLog
       ...
   3.创建CallLogService.java + CallLogServiceImpl.java
       ...

   4.

1.按照时间段查询通话记录
   1.hbase交互
       Scan : 设置startRow + endRow
       rowkey : hashcode , + callerid , + callTime, 0 , callee , duration.

2.
3.

编写CallLogController.java
---------------------------
/**
* 进入查询通话记录的页面,form
*/
@RequestMapping("/callLog/toFindCallLogPage")
public String toFindCallLogPage(){
return "callLog/findCallLog" ;
}

@RequestMapping(value = "/callLog/findCallLog",method = RequestMethod.POST)
public String findCallLog(@RequestParam("caller") String caller, @RequestParam("startTime") String startTime, @RequestParam("endTime") String endTime){
Calendar startCalendar = Calendar.getInstance();
Calendar endCalendar = Calendar.getInstance();

return "callLog/callLogList" ;
}

编写jsp
-----------------------------
   <%@ page contentType="text/html;charset=UTF-8" language="java" %>
   <%@ taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c" %>
   <html>
   <head>
       <title>查询通话记录</title>
       <link rel="stylesheet" type="text/css" href="../css/my.css">
   </head>
   <body>
   <form action='<c:url value="/callLog/findCallLog" />' method="post">
       <table>
           <tr>
               <td>电话号码 :</td>
               <td><input type="text" name="caller"></td>
           </tr>
           <tr>
               <td>起始时间 :</td>
               <td><input type="text" name="startTime"></td>
           </tr>
           <tr>
               <td>结束时间:</td>
               <td><input type="text" name="endTime"></td>
           </tr>
           <tr>
               <td colspan="2">
                   <input type="submit" value="查询"/>
               </td>
           </tr>
       </table>
   </form>
   </body>
   </html>

电信CALL 通话记录hbase kafka flume 学习

猜你喜欢