jenkins 自动打包部署爬虫项目

        最近因工作需要，负责看护公司的生产环境，有16台生产服务节点，所以必须得引入自动化管理工具，不然我可不得累死啊。本文记录一下在使用Jenkins发布过程的整个过程。

1.启动Jenkins服务

2.登陆网页

3.配置

3.1点击选择“配置”

3.2添加项目描述

3.3 添加项目名称

3.4配置SVN路径

3.5配置SVN账户

3.6是否打基线

3.7配置项目列表

3.8配置发布到各节点的路径

3.9配置项目列表格式

4.构建环境

4.1配置SVN密码

4.2在Windows执行py脚本

build_spider.py编译脚本

# -*- coding: utf-8 -*- 
from command import execute
import util
import os
import sys

COMMAMD_SVN_LIST = 'svn list '
COMMAMD_SVN_EXPORT = 'svn export --force '
COMMAMD_SVN_IMPORT = 'svn import -m -F '
COMMAMD_SVN_AUTHORIZATION = ' --username SVN账号 --password 密码***'

COMMAMD_PYTHON_BUILD = 'python setup.py sdist bdist_egg'

EXPORT_CODE_SUCCESS = '已导出版本'
UPLOAD_SUCCESS = '提交后的版本为'
COMPRESS_SUCCESS = '正在添加'
'''
编译spider
'''
class Build:
	def __init__(self):
		paramList = util.getParamList()	
		self.project_name = paramList[1]
		self.project_list = paramList[2]
		self.svn_code_repository = paramList[3]
		self.svn_upload_repository = paramList[4]
		self.svn_username = paramList[5]
		self.svn_password = paramList[6]
		self.svn_auth = ' --username ' + self.svn_username + ' --password ' + self.svn_password
		self.workspace = os.getcwd()
		self.conf_dir = util.getConfDir(self.project_name)
		self.spiders = paramList[8]
		print(self.spiders)

	def getConfList(self):
		print('conf dir: ' + self.conf_dir)
		return os.listdir(self.conf_dir)

	def exportCode2Local(self):
		exportResult = False
		print('export project')
		project = self.project_list
		if self.spiders == 'ALL':
			# 全部的代码
			result = execute(COMMAMD_SVN_EXPORT + util.myUrlJoin(self.svn_code_repository, project) + self.svn_auth)
			print(result)
			if not util.executeResult(EXPORT_CODE_SUCCESS, result):
				return False
			else:
				exportResult = True
		else:
			spider_list = util.getCommaSepratedList(self.spiders)
			for spider in spider_list:
				pass 
		return exportResult

	def getDetsConfFileThroughWar(self, conf):
		project_folder = os.path.join(self.project_name,self.project_name)
#		print(project_folder)
		for fpathe,dirs,fs in os.walk(project_folder):
			for f in fs:
				if(conf == f):
					return(os.path.join(fpathe,f))
		return ''

	def modifyConfig(self):		
		print(' in modifyConfig')
		confList = self.getConfList()
		print(confList)
		for conf in confList:
			dest = self.getDetsConfFileThroughWar(conf)
			if(not len(dest) == 0):	
#				print('copy ' + os.path.join(self.conf_dir, conf) + ' ' + dest + ' ' +'/Y')
				execute('copy ' + os.path.join(self.conf_dir, conf) + ' ' + dest + ' ' +'/Y')
				print(dest)

	def pypiBuild(self):
		buildResult = False
		for project in self.project_list:
			command = COMMAMD_PYTHON_BUILD
			print('build ' + project + ' start')
			os.chdir(os.path.join(self.workspace, project))
			print('plugin path is {}'.format(os.path))
			result = execute(command)
			for r in result:
				print(r)
			print('build ' + project + ' end')
		os.chdir(os.path.join(self.workspace, project)+'\\dist')
		relativeFileList = os.listdir(os.getcwd())
		fileListStr = util.list2Str(relativeFileList)
		print(fileListStr)


	def startWork(self):
		#step1: export code from svn
		exportResult = self.exportCode2Local()
		if exportResult == False:
			print('export fail')
			sys.exit(1)
		else:
			print('export success')

		# modify config
		self.modifyConfig()

		#step2: upload code to svn baseline

		#step3: python build
		# self.pypiBuild()

		#step4: update egg to svn

if __name__ == '__main__':	
	build = Build()
	build.startWork()

4.3通过SSH推送代码到各节点指定路径，有多少节点就配置多少次即可。

4.4推送完代码后，在各节点执行start.sh脚本

这里的einfo-start.sh脚本主要是打包代码，具体代码如下：

sudo kill -9 `ps -ef |grep scrapyd|awk '{print $2}' `
sleep 3
sudo /etc/init.d/scrapyd start
sleep 3
cd /home/test/spiders/eie_nfo
sudo /root/.pyenv/versions/3.6.5/bin/scrapyd-deploy -p eie_info

4.5发布成功与否判断

保存好配置

5.构建发布

如下，控制台输出打印成功！

 

以上，就是我打包发布项目到各个生产服务节点的全部过程。