CentOS8.2 详细安装、配置Spark3.0.0教程(Pyspark)亲自测试

sudo passwd:更改root密码
su:可以直接切换root密码
yum install -y update
yum groupinstall -y 'Development Tools'
yum install -y gcc openssl-devel bzip2-devel libffi-devel
yum -y install gcc pcre pcre-devel zlib zlib-devel openssl openssl-devel
----------------------------------------------------------------------------------

安装sqlite3:
sudo yum install sqlite-devel

----------------------------------------------------------------------------------

安装vim
yum -y install vim*

----------------------------------------------------------------------------------

安装python
下载和编译python3.8.3
(1)在官方网站下载最新版本python3.8.3
wget https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tgz
mkdir /usr/local/python3 
cd Python-3.8.3
./configure --prefix=/usr/local/python3 --enable-optimizations --with-ssl 
make && make install
ln -s /usr/local/python3/bin/python3 /usr/local/bin/python3
ln -s /usr/local/python3/bin/pip3 /usr/local/bin/pip3
ln -s /usr/local/bin/python3 /usr/bin/python
pip3 install --upgrade pip

----------------------------------------------------------------------------------

安装redis6

wget http://download.redis.io/releases/redis-6.2.6.tar.gz
tar -zxvf redis-6.2.6.tar.gz 
cd redis-6.2.6
make PREFIX=/usr/local/soft/redis install
mkdir /usr/local/soft/redis/conf
cp redis.conf /usr/local/soft/redis/conf/
mkdir -p /data/redis6
cd /data/redis6/
mkdir logs
mkdir data
cd /usr/local/soft/redis/conf
vi redis.conf修改配置文件
绑定访问的ip
bind 192.168.159.131(本机IP地址)

设置密码
requirepass 123456

使以daemon方式运行
daemonize yes

扫描二维码关注公众号,回复: 13386400 查看本文章

使用的最大内存数量
maxmemory 128MB

#日志保存目录
logfile "/data/redis6/logs/redis.log"

#数据保存目录
dir /data/redis6/data/

io线程数
io-threads 3

生成供systemd使用的service文件
vi /lib/systemd/system/redis6.service
[Unit]
Description=Redis
After=network.target
[Service]
Type=forking
PIDFile=/var/run/redis_6379.pid
ExecStart=/usr/local/soft/redis/bin/redis-server /usr/local/soft/redis/conf/redis.conf
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

重新加载service文件:systemctl daemon-reload 
启动:systemctl start redis6
停止:systemctl stop redis6
打开端口6379:firewall-cmd --add-port=6379/tcp --permanent
重启防火墙:systemctl restart firewalld.service

/usr/local/soft/redis/bin/redis-cli -h 192.168.159.131 -p 6379 -a 123456

----------------------------------------------------------------------------------

删除mariadb
yum -y remove mariadb*

删除mysql

yum -y remove mysql*

yum list installed mysql*
rpm -qa | grep mysql*

2.下载mysql rpm
yum localinstall https://dev.mysql.com/get/mysql57-community-release-el7-9.noarch.rpm

3.安装
yum install -y mysql-server

4.设置开机启动Mysql
systemctl enable mysqld.service

5.检查是否已经安装了开机自动启动
systemctl list-unit-files | grep mysqld

6.设置开启服务
systemctl start mysqld.service

7.查看临时登录密码
grep 'temporary password' /var/log/mysql/mysqld.log
如果没有就不用输入密码

mysql -u root -p

8.修改root密码并生效
ALTER USER 'root'@'localhost' IDENTIFIED BY 'Sloveb55555..';

9.创建远程登录用户
CREATE USER 'root'@'%' IDENTIFIED BY 'Sloveb55555..';

10.更改用户加密规则
ALTER USER 'root'@'%' IDENTIFIED WITH mysql_native_password BY 'Sloveb55555..';

11.授权用户远程登录权限
grant all on *.* to 'root'@'%';

12.刷新配置
flush privileges;

13.添加3306端口
firewall-cmd --add-port=3306/tcp --permanent

14.重启防火墙
systemctl restart firewalld.service

----------------------------------------------------------------------------------

1、安装mysql8
删除mariadb
yum -y remove mariadb*

删除mysql
yum -y remove mysql*

wget https://repo.mysql.com//mysql80-community-release-el7-1.noarch.rpm
rpm -ivh mysql80-community-release-el7-1.noarch.rpm
yum install mysql-server
启动
systemctl start mysqld.service 
查看mysql状态
systemctl status mysqld.service
mysql -uroot -p(密码是空的直接回车进入)
ALTER USER 'root'@'localhost' IDENTIFIED BY 'newmysql';
创建用户和密码
CREATE USER 'root'@'%' IDENTIFIED BY 'newmysql';
权限赋给root用户,所有ip都能连接
GRANT ALL ON *.* TO 'root'@'%' WITH GRANT OPTION;
更新root密码否则,远程登录会报错
ALTER USER 'root'@'%' IDENTIFIED WITH mysql_native_password BY 'newmysql';
刷新
FLUSH  PRIVILEGES;
加入3306端口
firewall-cmd --permanent --zone=public --add-port=3306/tcp
开启刷新防火墙
firewall-cmd --reload

开机自启动
systemctl enable mysqld
systemctl daemon-reload

----------------------------------------------------------------------------------

PyCharm Pro安装:
到官网下载
执行:tar -xzvf pycharm-professional-2020.1.1.tar.gz -C /usr/local/
执行:cd /usr/local/pycharm-2020.1.1/bin
执行:sh ./pycharm.sh
设置图标
sudo gedit /usr/share/applications/Pycharm.desktop
复制以下代码
[Desktop Entry]
Type=Application
Name=Pycharm
GenericName=pycharm-2020
Comment=Pycharm2020:The Python IDE
Exec="/home/new2020/下载/pycharm-2020.1.1/bin/pycharm.sh" %f
Icon=/home/new2020/下载/pycharm-2020.1.1/bin/pycharm.png
Terminal=pycharm
Categories=Pycharm;

打开以下目录
/usr/share/applications/
复制一份这个pycharm图标到桌面

---------------------------------------------------------------------------------

java自带1.8.0

---------------------------------------------------------------------------------

spark安装:
spark-3.0.0-bin-hadoop2.7.tgz
tar -zxvf spark-3.0.0-bin-hadoop2.7.tgz -C /usr/local/spark
解压到/usr/local/spark目录

配置环境变量并且添加以下配置代码
vim /etc/profile
export SPARK_HOME=/usr/local/spark
export PATH=${PATH}:$SPARK_HOME/bin
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH
保存退出执行生效
source /etc/profile

执行:./bin/pyspark

---------------------------------------------------------------------------------

安装pip
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

---------------------------------------------------------------------------------

安装numpy
pip3 install numpy

---------------------------------------------------------------------------------

安装 py4j
pip3 install py4j

降低log4j级别
进入spark-2.4.0-bin-hadoop2.7下面的conf目录
cp conf/log4j.properties.template conf/log4j.properties
vim conf/log4j.properties
将     log4j.rootCategory=INFO, console  
替换为   log4j.rootCategory=ERROR, console
保存退出

-------------------------------------------------------------------------------

pip3 install redis

猜你喜欢

转载自blog.csdn.net/vlking/article/details/107700693