Article directory
guide
This article will explain the principle of the registration center, self-developed RPC, netty application and related source code
registration center
xxl-job also has its own distributed registration center, its registry uses mysql for storage, and the heartbeat mechanism is implemented through http requests
Service registration in xxl-job needs to be bound to an executor as a carrier. When our service is registered, the registered machine information will be stored in the first, and the 注册表scanning thread will continuously scan the registry, and bind the machine in the registry to the appName configuration according to the appName on the actuator.
First look at the core helper of the registration centercom.xxl.job.admin.core.thread.JobRegistryHelper#start
public void start(){
// 服务注册/删除线程池
registryOrRemoveThreadPool = new ThreadPoolExecutor(
2,
10,
30L,
TimeUnit.SECONDS,
new LinkedBlockingQueue<Runnable>(2000),
new ThreadFactory() {
@Override
public Thread newThread(Runnable r) {
return new Thread(r, "xxl-job, admin JobRegistryMonitorHelper-registryOrRemoveThreadPool-" + r.hashCode());
}
},
new RejectedExecutionHandler() {
@Override
public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
r.run();
logger.warn(">>>>>>>>>>> xxl-job, registry or remove too fast, match threadpool rejected handler(run now).");
}
});
// 注册服务监听器
registryMonitorThread = new Thread(new Runnable() {
@Override
public void run() {
while (!toStop) {
try {
// 获得配置为自动注册的执行器
List<XxlJobGroup> groupList = XxlJobAdminConfig.getAdminConfig().getXxlJobGroupDao().findByAddressType(0);
if (groupList!=null && !groupList.isEmpty()) {
// 将90秒没有心跳的机器移除
List<Integer> ids = XxlJobAdminConfig.getAdminConfig().getXxlJobRegistryDao().findDead(RegistryConfig.DEAD_TIMEOUT, new Date());
if (ids!=null && ids.size()>0) {
// 移除注册表中的机器数据
XxlJobAdminConfig.getAdminConfig().getXxlJobRegistryDao().removeDead(ids);
}
// 归集执行器下的注册机ip地址
HashMap<String, List<String>> appAddressMap = new HashMap<String, List<String>>();
// 获得90秒内有心跳的机器
List<XxlJobRegistry> list = XxlJobAdminConfig.getAdminConfig().getXxlJobRegistryDao().findAll(RegistryConfig.DEAD_TIMEOUT, new Date());
if (list != null) {
for (XxlJobRegistry item: list) {
if (RegistryConfig.RegistType.EXECUTOR.name().equals(item.getRegistryGroup())) {
String appname = item.getRegistryKey();
List<String> registryList = appAddressMap.get(appname);
if (registryList == null) {
registryList = new ArrayList<String>();
}
if (!registryList.contains(item.getRegistryValue())) {
registryList.add(item.getRegistryValue());
}
appAddressMap.put(appname, registryList);
}
}
}
// 将注册表中的数据归档存入到执行器中并入库
for (XxlJobGroup group: groupList) {
List<String> registryList = appAddressMap.get(group.getAppname());
String addressListStr = null;
if (registryList!=null && !registryList.isEmpty()) {
Collections.sort(registryList);
StringBuilder addressListSB = new StringBuilder();
for (String item:registryList) {
addressListSB.append(item).append(",");
}
addressListStr = addressListSB.toString();
addressListStr = addressListStr.substring(0, addressListStr.length()-1);
}
group.setAddressList(addressListStr);
group.setUpdateTime(new Date());
XxlJobAdminConfig.getAdminConfig().getXxlJobGroupDao().update(group);
}
}
} catch (Exception e) {
if (!toStop) {
logger.error(">>>>>>>>>>> xxl-job, job registry monitor thread error:{}", e);
}
}
try {
// 检查周期30秒
TimeUnit.SECONDS.sleep(RegistryConfig.BEAT_TIMEOUT);
} catch (InterruptedException e) {
if (!toStop) {
logger.error(">>>>>>>>>>> xxl-job, job registry monitor thread error:{}", e);
}
}
}
logger.info(">>>>>>>>>>> xxl-job, job registry monitor thread stop");
}
});
registryMonitorThread.setDaemon(true);
registryMonitorThread.setName("xxl-job, admin JobRegistryMonitorHelper-registryMonitorThread");
registryMonitorThread.start();
}
The helper starts a thread pool for registration and removal. This thread will interact with the database. The main purpose of using the thread pool is not to block the scheduling thread. A daemon listening thread is also started to monitor the service heartbeat and bind the executor.
Next, look at the service registration logiccom.xxl.job.admin.core.thread.JobRegistryHelper#registry
public ReturnT<String> registry(RegistryParam registryParam) {
// 参数校验
if (!StringUtils.hasText(registryParam.getRegistryGroup())
|| !StringUtils.hasText(registryParam.getRegistryKey())
|| !StringUtils.hasText(registryParam.getRegistryValue())) {
return new ReturnT<String>(ReturnT.FAIL_CODE, "Illegal Argument.");
}
// 使用线程池执行
registryOrRemoveThreadPool.execute(new Runnable() {
@Override
public void run() {
// 更新注册表的心跳时间
int ret = XxlJobAdminConfig.getAdminConfig().getXxlJobRegistryDao().registryUpdate(registryParam.getRegistryGroup(), registryParam.getRegistryKey(), registryParam.getRegistryValue(), new Date());
if (ret < 1) {
// 更新失败说明不存在该数据，存入注册表
XxlJobAdminConfig.getAdminConfig().getXxlJobRegistryDao().registrySave(registryParam.getRegistryGroup(), registryParam.getRegistryKey(), registryParam.getRegistryValue(), new Date());
// 刷新注册信息，暂时没用到，是个空方法
freshGroupRegistryInfo(registryParam);
}
}
});
return ReturnT.SUCCESS;
}
// 移除注册机
public ReturnT<String> registryRemove(RegistryParam registryParam) {
if (!StringUtils.hasText(registryParam.getRegistryGroup())
|| !StringUtils.hasText(registryParam.getRegistryKey())
|| !StringUtils.hasText(registryParam.getRegistryValue())) {
return new ReturnT<String>(ReturnT.FAIL_CODE, "Illegal Argument.");
}
registryOrRemoveThreadPool.execute(new Runnable() {
@Override
public void run() {
// 移除注册表数据
int ret = XxlJobAdminConfig.getAdminConfig().getXxlJobRegistryDao().registryDelete(registryParam.getRegistryGroup(), registryParam.getRegistryKey(), registryParam.getRegistryValue());
if (ret > 0) {
// 刷新注册信息，暂时没用到，是个空方法
freshGroupRegistryInfo(registryParam);
}
}
});
return ReturnT.SUCCESS;
}
It can be seen that the registration logic is actually not complicated. It only needs to save the heartbeat data and keep updating the heartbeat time to determine the survival of the machine. To remove the registration machine is to directly remove the registry data.
Next, look at the entrance of the service registration
com.xxl.job.admin.controller.JobApiController#api
@RequestMapping("/{uri}")
@ResponseBody
@PermissionLimit(limit=false)
public ReturnT<String> api(HttpServletRequest request, @PathVariable("uri") String uri, @RequestBody(required = false) String data) {
// 请求方法校验
if (!"POST".equalsIgnoreCase(request.getMethod())) {
return new ReturnT<String>(ReturnT.FAIL_CODE, "invalid request, HttpMethod not support.");
}
// 请求uri校验
if (uri==null || uri.trim().length()==0) {
return new ReturnT<String>(ReturnT.FAIL_CODE, "invalid request, uri-mapping empty.");
}
// 请求token校验
if (XxlJobAdminConfig.getAdminConfig().getAccessToken()!=null
&& XxlJobAdminConfig.getAdminConfig().getAccessToken().trim().length()>0
&& !XxlJobAdminConfig.getAdminConfig().getAccessToken().equals(request.getHeader(XxlJobRemotingUtil.XXL_JOB_ACCESS_TOKEN))) {
return new ReturnT<String>(ReturnT.FAIL_CODE, "The access token is wrong.");
}
// 根据请求uri解析参数并调度
if ("callback".equals(uri)) {
List<HandleCallbackParam> callbackParamList = GsonTool.fromJson(data, List.class, HandleCallbackParam.class);
return adminBiz.callback(callbackParamList);
} else if ("registry".equals(uri)) {
RegistryParam registryParam = GsonTool.fromJson(data, RegistryParam.class);
return adminBiz.registry(registryParam);
} else if ("registryRemove".equals(uri)) {
RegistryParam registryParam = GsonTool.fromJson(data, RegistryParam.class);
return adminBiz.registryRemove(registryParam);
} else {
return new ReturnT<String>(ReturnT.FAIL_CODE, "invalid request, uri-mapping("+ uri +") not found.");
}
}
Here is the general entrance of the client scheduling server. You can see that the entrance is an http request, and each business is forwarded according to different uri, including callback (callback), registry (registration/heartbeat), registryRemove (remove registration)
application of netty
Netty is mainly used on the client side to provide a scheduling interface for the server side. xxl-job does not directly use springMVC as the request entry. Its advantage is low coupling, and non-spring applications can also be used. Secondly, netty is used for controllability, stability, and performance. will perform better.
We need to come to XxlJobExecutorthe class first, which is the client instance, and startthe method will be called after startup
start entrycom.xxl.job.core.executor.XxlJobExecutor#start
public void start() throws Exception {
// 初始化日志路径
XxlJobFileAppender.initLogPath(logPath);
// 初始化xxl-job连接信息
initAdminBizList(adminAddresses, accessToken);
// 初始化日志文件切割线程
JobLogFileCleanThread.getInstance().start(logRetentionDays);
// 初始化回调线程
TriggerCallbackThread.getInstance().start();
// 初始化netty服务
initEmbedServer(address, ip, port, appname, accessToken);
}
Enter the method of initializing the netty servicecom.xxl.job.core.executor.XxlJobExecutor#initEmbedServer
private void initEmbedServer(String address, String ip, int port, String appname, String accessToken) throws Exception {
// 选择可用端口号，从9999向下找到未使用过的端口
port = port>0?port: NetUtil.findAvailablePort(9999);
// 获得本机内网ip地址
ip = (ip!=null&&ip.trim().length()>0)?ip: IpUtil.getIp();
// 拼接应用请求地址
if (address==null || address.trim().length()==0) {
String ip_port_address = IpUtil.getIpPort(ip, port); // registry-address：default use address to registry , otherwise use ip:port if address is null
address = "http://{ip_port}/".replace("{ip_port}", ip_port_address);
}
if (accessToken==null || accessToken.trim().length()==0) {
logger.warn(">>>>>>>>>>> xxl-job accessToken is empty. To ensure system security, please set the accessToken.");
}
// 创建server实例并启动
embedServer = new EmbedServer();
embedServer.start(address, port, appname, accessToken);
}
Enter the method of actually creating a netty servicecom.xxl.job.core.server.EmbedServer#start
public void start(final String address, final int port, final String appname, final String accessToken) {
// 创建业务实例
executorBiz = new ExecutorBizImpl();
thread = new Thread(new Runnable() {
@Override
public void run() {
// 用于接受ServerSocketChannel的io请求，再把请求具体执行的回调函数转交给worker group执行
EventLoopGroup bossGroup = new NioEventLoopGroup();
EventLoopGroup workerGroup = new NioEventLoopGroup();
// 创建业务消费线程
ThreadPoolExecutor bizThreadPool = new ThreadPoolExecutor(
0,
200,
60L,
TimeUnit.SECONDS,
new LinkedBlockingQueue<Runnable>(2000),
new ThreadFactory() {
@Override
public Thread newThread(Runnable r) {
return new Thread(r, "xxl-job, EmbedServer bizThreadPool-" + r.hashCode());
}
},
new RejectedExecutionHandler() {
@Override
public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
throw new RuntimeException("xxl-job, EmbedServer bizThreadPool is EXHAUSTED!");
}
});
try {
// 启动netty服务
ServerBootstrap bootstrap = new ServerBootstrap();
bootstrap.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
public void initChannel(SocketChannel channel) throws Exception {
channel.pipeline()
// 空闲检测
.addLast(new IdleStateHandler(0, 0, 30 * 3, TimeUnit.SECONDS)) // beat 3N, close if idle
// http编码处理类
.addLast(new HttpServerCodec())
// post请求参数解析器
.addLast(new HttpObjectAggregator(5 * 1024 * 1024)) // merge request & reponse to FULL
// 自定义业务handler
.addLast(new EmbedHttpServerHandler(executorBiz, accessToken, bizThreadPool));
}
})
.childOption(ChannelOption.SO_KEEPALIVE, true);
// 将端口号绑定到netty服务上
ChannelFuture future = bootstrap.bind(port).sync();
logger.info(">>>>>>>>>>> xxl-job remoting server start success, nettype = {}, port = {}", EmbedServer.class, port);
// 开始服务注册，将服务注册到xxl-job
startRegistry(appname, address);
// 同步阻塞线程
future.channel().closeFuture().sync();
} catch (InterruptedException e) {
if (e instanceof InterruptedException) {
logger.info(">>>>>>>>>>> xxl-job remoting server stop.");
} else {
logger.error(">>>>>>>>>>> xxl-job remoting server error.", e);
}
} finally {
// 执行到此处表示netty服务停止，释放资源
try {
workerGroup.shutdownGracefully();
bossGroup.shutdownGracefully();
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
}
}
});
// 设置守护线程，防止netty线程被回收
thread.setDaemon(true);
// 启动线程
thread.start();
}
It can be seen that when the netty service is started, the service will be registered to xxl-job
com.xxl.job.core.server.EmbedServer#startRegistry
com.xxl.job.core.thread.ExecutorRegistryThread#start
public void start(final String appname, final String address){
// valid
if (appname==null || appname.trim().length()==0) {
logger.warn(">>>>>>>>>>> xxl-job, executor registry config fail, appname is null.");
return;
}
if (XxlJobExecutor.getAdminBizList() == null) {
logger.warn(">>>>>>>>>>> xxl-job, executor registry config fail, adminAddresses is null.");
return;
}
registryThread = new Thread(new Runnable() {
@Override
public void run() {
// 循环心跳机制
while (!toStop) {
try {
// 构建请求参数
RegistryParam registryParam = new RegistryParam(RegistryConfig.RegistType.EXECUTOR.name(), appname, address);
for (AdminBiz adminBiz: XxlJobExecutor.getAdminBizList()) {
try {
// 服务注册/心跳
ReturnT<String> registryResult = adminBiz.registry(registryParam);
if (registryResult!=null && ReturnT.SUCCESS_CODE == registryResult.getCode()) {
registryResult = ReturnT.SUCCESS;
logger.debug(">>>>>>>>>>> xxl-job registry success, registryParam:{}, registryResult:{}", new Object[]{
registryParam, registryResult});
break;
} else {
logger.info(">>>>>>>>>>> xxl-job registry fail, registryParam:{}, registryResult:{}", new Object[]{
registryParam, registryResult});
}
} catch (Exception e) {
logger.info(">>>>>>>>>>> xxl-job registry error, registryParam:{}", registryParam, e);
}
}
} catch (Exception e) {
if (!toStop) {
logger.error(e.getMessage(), e);
}
}
try {
if (!toStop) {
// 心跳时长 30秒
TimeUnit.SECONDS.sleep(RegistryConfig.BEAT_TIMEOUT);
}
} catch (InterruptedException e) {
if (!toStop) {
logger.warn(">>>>>>>>>>> xxl-job, executor registry thread interrupted, error msg:{}", e.getMessage());
}
}
}
// 服务停止后以下代码服务下线通知
try {
RegistryParam registryParam = new RegistryParam(RegistryConfig.RegistType.EXECUTOR.name(), appname, address);
for (AdminBiz adminBiz: XxlJobExecutor.getAdminBizList()) {
try {
ReturnT<String> registryResult = adminBiz.registryRemove(registryParam);
if (registryResult!=null && ReturnT.SUCCESS_CODE == registryResult.getCode()) {
registryResult = ReturnT.SUCCESS;
logger.info(">>>>>>>>>>> xxl-job registry-remove success, registryParam:{}, registryResult:{}", new Object[]{
registryParam, registryResult});
break;
} else {
logger.info(">>>>>>>>>>> xxl-job registry-remove fail, registryParam:{}, registryResult:{}", new Object[]{
registryParam, registryResult});
}
} catch (Exception e) {
if (!toStop) {
logger.info(">>>>>>>>>>> xxl-job registry-remove error, registryParam:{}", registryParam, e);
}
}
}
} catch (Exception e) {
if (!toStop) {
logger.error(e.getMessage(), e);
}
}
logger.info(">>>>>>>>>>> xxl-job, executor registry thread destroy.");
}
});
registryThread.setDaemon(true);
registryThread.setName("xxl-job, executor ExecutorRegistryThread");
registryThread.start();
}
The above code is a service registration, which is also a heartbeat mechanism. When the service stops, the stop variable will become true. At this time, the server will be notified to go offline. It does not matter if the service suddenly goes down. The server has a circular checking mechanism. After receiving the heartbeat from the client, it will automatically go offline.
Next, let's see how the custom netty handler handles server messages
com.xxl.job.core.server.EmbedServer.EmbedHttpServerHandler#channelRead0
protected void channelRead0(final ChannelHandlerContext ctx, FullHttpRequest msg) throws Exception {
// 消息解码
String requestData = msg.content().toString(CharsetUtil.UTF_8);
// 请求uri
String uri = msg.uri();
HttpMethod httpMethod = msg.method();
boolean keepAlive = HttpUtil.isKeepAlive(msg);
String accessTokenReq = msg.headers().get(XxlJobRemotingUtil.XXL_JOB_ACCESS_TOKEN);
// 业务线程执行处理
bizThreadPool.execute(new Runnable() {
@Override
public void run() {
Object responseObj = process(httpMethod, uri, requestData, accessTokenReq);
// 返回对象转换json
String responseJson = GsonTool.toJson(responseObj);
// 返回
writeResponse(ctx, keepAlive, responseJson);
}
});
}
com.xxl.job.core.server.EmbedServer.EmbedHttpServerHandler#process
private Object process(HttpMethod httpMethod, String uri, String requestData, String accessTokenReq) {
// 校验请求方式，token，uri
if (HttpMethod.POST != httpMethod) {
return new ReturnT<String>(ReturnT.FAIL_CODE, "invalid request, HttpMethod not support.");
}
if (uri == null || uri.trim().length() == 0) {
return new ReturnT<String>(ReturnT.FAIL_CODE, "invalid request, uri-mapping empty.");
}
if (accessToken != null
&& accessToken.trim().length() > 0
&& !accessToken.equals(accessTokenReq)) {
return new ReturnT<String>(ReturnT.FAIL_CODE, "The access token is wrong.");
}
try {
// 处理业务
if ("/beat".equals(uri)) {
// 心跳检查
return executorBiz.beat();
} else if ("/idleBeat".equals(uri)) {
// 任务空闲检查
IdleBeatParam idleBeatParam = GsonTool.fromJson(requestData, IdleBeatParam.class);
return executorBiz.idleBeat(idleBeatParam);
} else if ("/run".equals(uri)) {
// 运行一个任务
TriggerParam triggerParam = GsonTool.fromJson(requestData, TriggerParam.class);
return executorBiz.run(triggerParam);
} else if ("/kill".equals(uri)) {
// 杀死一个任务
KillParam killParam = GsonTool.fromJson(requestData, KillParam.class);
return executorBiz.kill(killParam);
} else if ("/log".equals(uri)) {
// 获得客户端记录的日志
LogParam logParam = GsonTool.fromJson(requestData, LogParam.class);
return executorBiz.log(logParam);
} else {
// 无法解析的业务uri，异常返回
return new ReturnT<String>(ReturnT.FAIL_CODE, "invalid request, uri-mapping(" + uri + ") not found.");
}
} catch (Exception e) {
logger.error(e.getMessage(), e);
return new ReturnT<String>(ReturnT.FAIL_CODE, "request error:" + ThrowableUtil.toString(e));
}
}
It can be seen that there is no difference between logical analysis and server-side analysis. They are all judged and executed based on uri. The entire netty scheduling link is xxl-job自研RPC
Link
[xxl-job source code articles 03] xxl-job log system source code interpretation