Lambda架构
架构简介
- Lambda架构是大数据架构的一种,其作用无非是存储-处理-展示数据。关于其介绍,网上多篇博客已珠玉在前,这里不再赘述。可参见:
最重要的官网,配合翻译插件食用更佳:http://lambda-architecture.net/
https://blog.csdn.net/lvsaixia/article/details/51778487;
https://blog.csdn.net/brucesea/article/details/45937875;
https://www.jianshu.com/p/b6bf05332c18;
https://www.jianshu.com/p/ef543d6dc15b
https://zhuanlan.zhihu.com/p/20510974;
Lambda架构示意图
架构之组成
1. batch layer
如图可知,优选是Hive和Pig,次选是Spark、Hadoop;
2. speed layer
如图可知,优选是Spark Streaming和Strom,次选是Spring XD;
3. serving layer
如图可知,优选是HBase和Druid,次选是SploutSQL;
由上可知,Lambda架构的组成可以有千百种方式,但其每一个layer的选择却是有限的;只需结合业务需求、然后在上述三幅图中选择即可。
Lambda架构的Demo
(官网给定的5个demo,但目前一个都没看懂-2018年9月4日)
Demo1. 推特标签数量实时分析
batch layer = HDFS;
speed layer = Trident(a high level Storm);
serving layer = Splout SQL;
Demo2. 实时分析数据堆栈
data resource = Kafka;
batch layer = Hadoop;
speed layer = Storm;
serving layer = Druid;
Demo3. 洛杉矶公共交通数据实时分析
data resource = Kafka;
batch layer = Spark;
speed layer = Cassandra;
serving layer = Akka;
Demo4. 基于微软云分析公共会议流程
github项目地址:https://github.com/ANierbeck/BusFloatingData
data resource = ;
batch layer = ;
speed layer = ;
serving layer = ;
Demo5. 推特实时详尽分析
data resource = Kafka/Twitter4j;
batch layer = Spark;
speed layer = Spark-streaming/Cassandra;
serving layer = Akka/Akka-http;
我的Demo
暂时假定,
batch layer = Hive;
speed layer = Storm;
serving layer = HBase;搭建起一个local mode(本地集群)的demo
……
Storm集成HBase:
http://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-hbase.html