linux
安装ssh服务
sudo apt-get install ssh openssh-server
免密登录
$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 600 ~/.ssh/authorized_keys
ECDSA key fingerprint is SHA256:LCAvYsLBFHKF4sT2mNMYC8KjE/LxNxtehw87KWDSuYc.
Hadoop安装包
配置Hadoop中相应的文件
运行Hadoop
初始化HDFS系统
在hadoop目录下使用如下命令：
bin/hdfs namenode -format
运行结果：
开启NameNode和DataNode守护进程
使用如下命令开启：
sbin/start-dfs.sh，成功的截图如下：
报错原因：我使用的是jdk10，应该用jdk8
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.6.4.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
OpenJDK 64-Bit Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
18/05/01 11:10:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
运行WordCount Demo
报错
NameNode和DataNode进程没有开启
重新更换了虚拟机，但NameNode和DataNode进程没有开启
参考链接：https://blog.csdn.net/weixin_30699235/article/details/97423466
ubuntu每次重启都会删除/tmp中的信息，而namenode的格式化信息恰好存在/tmp下，从而导致相应信息丢失
java版本未找到
Command 'java' not found, but can be installed with:
sudo apt install default-jre
sudo apt install openjdk-11-jre-headless
sudo apt install openjdk-8-jre-headless
原因：先进入ssh
ssh localhost
java -version
NativeCodeLoader警告
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
解决方案：在profile文件中添加
export JAVA_LIBRARY_PATH=’/usr/local/sinasrv2/hadoop/hadoop-2.6.4/lib/native
参考链接：https://blog.csdn.net/qq_41541801/article/details/82413776
illegal reflective警告
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/usr/hadoop-2.7.7/share/hadoop/common/lib/hadoop-auth-2.7.7.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
方法：--illegal-access=warn
WARNING: Use --illegal-access=warn to enable warnings of further
illegal reflective access operations
超级管理员文件管理器
sudo nautilus打开具有超级管理员权限的文件管理器
其他命令行
sudo nautilus打开具有超级管理员权限的文件管理器
修改profile
sudo vim /etc/profile
source profile
windows
结果如下：
命令行
查看目录：hdfs dfs -ls /KMeans_m_0
删除文件：hadoop fs -rm /KMeans_m_0
删除文件夹：hadoop fs -rm -r /KMeans_m_0
上传文件：hdfs dfs -put juleiData.txt /
查看文件：hadoop fs -cat /test/out/part-r-00000
修改cmd窗口字符编码为UTF-8，命令行中执行：chcp 65001
切换回中文：chcp 936
IDEA配置
报错
解决Hadoop报错：Failed to locate the winutils binary in the hadoop binary path
https://blog.csdn.net/abc50319/article/details/80284741
Hadoop项目出现No such file or directory错误
https://blog.csdn.net/mercyfreedom/article/details/102961173
乱码问题
使用utf-8格式
参考链接
https://www.cnblogs.com/hello-/articles/9600269.html
https://blog.csdn.net/youthon/article/details/51518772