Before building a Hadoop cluster with a virtual machine (including pseudo-distributed and fully distributed: Hadoop installation of pseudo-distributed ), but this is too much consumption of resources, and since learned Docker later, something wants to Docker put, this practice one, with Docker to build a Hadoop cluster (at this point my thoughts were still with Docker Spark cluster, follow-up ......), the main Hadoop service here built before the project is not finished: Web logs traffic analysis system (the system is currently virtual machine to achieve the offline analysis module, real-time analysis has not been completed due to resource issues, I think this does not worry) Ali cloud considering the ECS for me personally, the cost is a little too high, but since learned Dokcer later, I no longer afraid, and with the station enough friends, ha ha ha ha ...... start following it! The installation process is longer, so please be patient!
First, prepare the environment
① Ali cloud lightweight application server (student price is very affordable ~), my configuration: 1C, 2G, 40G (enough)
②CentOS7.3, 4.4 kernel (uname -r available to view, comes 3.10.x may cause unstable operation Docker I refer to another blog: From centos7 mirrored to build kubernetes cluster (installation kubeadm way) inside on the upgrade kernel command)
③Docker Version: 19.03.5 (Installation Reference to the official website, I refer to accelerated mirroring another blog: Docker configuration Ali cloud mirror source )
④JDK1.8：https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
⑤Hadoop: http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.ta r.gz (PS: Source is a source package, binary installation package is )
Two, DockerFile construct includes mirroring function ssh
1, create the corresponding directory
centos- mkdir SSH cd CentOS - SSH default name Dockerfile when vim Dockerfile # docker build here
2, write Dockerfile
# Select an existing OS image as the basis FROM centos Author # mirrored MAINTAINER xiedong # install OpenSSH - Server and software packages sudo, and sshd of UsePAM parameter is set to NO RUN yum install -y openssh- Server sudo RUN sed -i ' S / UsePAM Yes / NO UsePAM / G ' / etc / SSH / the sshd_config # mounted OpenSSH - Clients the RUN yum the install -Y openssh- Clients # add a test user root, password root, and add the user to sudoers in the RUN echo " root: root " | chpasswd RUN echo " root ALL = (ALL) ALL " >> / etc / sudoers # Service sshd start and exposed port 22 RUN mkdir / var / RUN / sshd EXPOSE 22 CMD [ " / usr / sbin / sshd " , " -D " ]
3, the mirror Construction
Build -t Docker ' xD / SSH-CentOS ' . Note # Do not forget the end point
Three, DokcerFile construct jdk1.8 mirror (mirror based ssh)
1. Create a directory
2, edit Dockefile (docker build default name Dockerfile)
3, the mirror constructed jdk1.8
Four, DockerFile build hadoop mirror (ssh-based and JDK)
Five, pipework fixed IP
Six, Docker build a Hadoop cluster (pseudo-distributed and fully distributed)
Reference from: https://blog.csdn.net/xu470438000/article/details/50512442 (if infringement, please contact deleted)