Table of contents
foreword
The author wrote a long time ago (231 messages) OpenRASP Java application self-protection use_fenglllle's blog-CSDN blog
In fact, many commercial rasp tools are based on the inspiration of OpenRASP. The main method is to inject bytecodes into specific methods through Javaagent technology for core Java classes and perform parameter verification. The core technology is Javaagent, so by analyzing the agent implementation principle of OpenRASP, you can understand the mainstream rasp implementation logic. Optimizing part of the implementation logic on OpenRASP can form a commercial product.
Prepare
Prepare Baidu's admin side, execute ./rasp-cloud -d
get after login
appid、appsecret
Configure the agent and the parameters used by the agent on the Java target machine
Pay attention to Baidu's own one-click injection code jar, the injected agent is before all other agents, and the agent is loaded sequentially.
Source code analysis
1. manifest
agent is the entry defined by mainfest, which can be seen inside the pom plug-in and java package
pom-plugin
jar package
2. Agent analysis
Taking the jvm parameter method as an example, the dynamic injection is similar, and rasp is generally injected when the jvm parameter is started.
com.baidu.openrasp.Agent
public static void premain(String agentArg, Instrumentation inst) {
init(START_MODE_NORMAL, START_ACTION_INSTALL, inst);
}
public static synchronized void init(String mode, String action, Instrumentation inst) {
try {
//添加jar文件到jdk的跟路径下，优先加载
JarFileHelper.addJarToBootstrap(inst);
//读取一些agent的数据
readVersion();
//核心代码
ModuleLoader.load(mode, action, inst);
} catch (Throwable e) {
System.err.println("[OpenRASP] Failed to initialize, will continue without security protection.");
e.printStackTrace();
}
}ModuleLoader.load(mode, action, inst);
/**
* 加载所有 RASP 模块
*
* @param mode 启动模式
* @param inst {@link java.lang.instrument.Instrumentation}
*/
public static synchronized void load(String mode, String action, Instrumentation inst) throws Throwable {
if (Module.START_ACTION_INSTALL.equals(action)) {
if (instance == null) {
try {
//安装
instance = new ModuleLoader(mode, inst);
} catch (Throwable t) {
instance = null;
throw t;
}
} else {
System.out.println("[OpenRASP] The OpenRASP has bean initialized and cannot be initialized again");
}
} else if (Module.START_ACTION_UNINSTALL.equals(action)) {
//卸载
release(mode);
} else {
throw new IllegalStateException("[OpenRASP] Can not support the action: " + action);
}
}Take a look at the new ModuleLoader logic
/**
* 构造所有模块
*
* @param mode 启动模式
* @param inst {@link java.lang.instrument.Instrumentation}
*/
private ModuleLoader(String mode, Instrumentation inst) throws Throwable {
// JBoss参数，实际上就是设置一些系统变量
if (Module.START_MODE_NORMAL == mode) {
setStartupOptionForJboss();
}
//构造对象，加载另外的jar engine.jar 初始化com.baidu.openrasp.EngineBoot
engineContainer = new ModuleContainer(ENGINE_JAR);
//核心逻辑
engineContainer.start(mode, inst);
}new ModuleContainer(ENGINE_JAR);
public ModuleContainer(String jarName) throws Throwable {
try {
//engine jar，openrasp有2个jar，单独加载，方便卸载，agent常用手段
File originFile = new File(baseDirectory + File.separator + jarName);
JarFile jarFile = new JarFile(originFile);
Attributes attributes = jarFile.getManifest().getMainAttributes();
jarFile.close();
this.moduleName = attributes.getValue("Rasp-Module-Name");
// com.baidu.openrasp.EngineBoot
String moduleEnterClassName = attributes.getValue("Rasp-Module-Class");
//用classloader加载jar
if (moduleName != null && moduleEnterClassName != null
&& !moduleName.equals("") && !moduleEnterClassName.equals("")) {
Class moduleClass;
if (ClassLoader.getSystemClassLoader() instanceof URLClassLoader) {
Method method = Class.forName("java.net.URLClassLoader").getDeclaredMethod("addURL", URL.class);
method.setAccessible(true);
method.invoke(moduleClassLoader, originFile.toURI().toURL());
method.invoke(ClassLoader.getSystemClassLoader(), originFile.toURI().toURL());
moduleClass = moduleClassLoader.loadClass(moduleEnterClassName);
//com.baidu.openrasp.EngineBoot对象
module = (Module) moduleClass.newInstance();
} else if (ModuleLoader.isCustomClassloader()) {
moduleClassLoader = ClassLoader.getSystemClassLoader();
Method method = moduleClassLoader.getClass().getDeclaredMethod("appendToClassPathForInstrumentation", String.class);
method.setAccessible(true);
try {
method.invoke(moduleClassLoader, originFile.getCanonicalPath());
} catch (Exception e) {
method.invoke(moduleClassLoader, originFile.getAbsolutePath());
}
moduleClass = moduleClassLoader.loadClass(moduleEnterClassName);
//初始化com.baidu.openrasp.EngineBoot
module = (Module) moduleClass.newInstance();
} else {
throw new Exception("[OpenRASP] Failed to initialize module jar: " + jarName);
}
}
} catch (Throwable t) {
System.err.println("[OpenRASP] Failed to initialize module jar: " + jarName);
throw t;
}
}engineContainer.start(mode, inst);
public void start(String mode, Instrumentation inst) throws Exception {
//帅气的时刻，打印logo
System.out.println("\n\n" +
" ____ ____ ___ _____ ____ \n" +
" / __ \\____ ___ ____ / __ \\/ | / ___// __ \\\n" +
" / / / / __ \\/ _ \\/ __ \\/ /_/ / /| | \\__ \\/ /_/ /\n" +
"/ /_/ / /_/ / __/ / / / _, _/ ___ |___/ / ____/ \n" +
"\\____/ .___/\\___/_/ /_/_/ |_/_/ |_/____/_/ \n" +
" /_/ \n\n");
try {
//载入v8，调用c，实际上可以使用火狐引擎替代，百度的官方文档的图就是rhino
Loader.load();
} catch (Exception e) {
System.out.println("[OpenRASP] Failed to load native library, please refer to https://rasp.baidu.com/doc/install/software.html#faq-v8-load for possible solutions.");
e.printStackTrace();
return;
}
if (!loadConfig()) {
return;
}
//缓存rasp的build信息
Agent.readVersion();
BuildRASPModel.initRaspInfo(Agent.projectVersion, Agent.buildTime, Agent.gitCommit);
// 初始化插件系统 调用v8，支持动态更新插件和js
if (!JS.Initialize()) {
return;
}
//核心代码，加入检查点，就是哪些类的哪些方法需要check
CheckerManager.init();
//字节码替换
initTransformer(inst);
if (CloudUtils.checkCloudControlEnter()) {
CrashReporter.install(Config.getConfig().getCloudAddress() + "/v1/agent/crash/report",
Config.getConfig().getCloudAppId(), Config.getConfig().getCloudAppSecret(),
CloudCacheModel.getInstance().getRaspId());
}
deleteTmpDir();
String message = "[OpenRASP] Engine Initialized [" + Agent.projectVersion + " (build: GitCommit="
+ Agent.gitCommit + " date=" + Agent.buildTime + ")]";
System.out.println(message);
Logger.getLogger(EngineBoot.class.getName()).info(message);
}loadConfig()
private boolean loadConfig() throws Exception {
//处理日志相关
LogConfig.ConfigFileAppender();
//单机模式下动态添加获取删除syslog
if (!CloudUtils.checkCloudControlEnter()) {
LogConfig.syslogManager();
} else {
System.out.println("[OpenRASP] RASP ID: " + CloudCacheModel.getInstance().getRaspId());
}
return true;
}
private void init() {
this.configFileDir = baseDirectory + File.separator + CONFIG_DIR_NAME;
String configFilePath = this.configFileDir + File.separator + CONFIG_FILE_NAME;
try {
//载入靶机的配置文件，里面配置了appid appsecret等
loadConfigFromFile(new File(configFilePath), true);
if (!getCloudSwitch()) {
try {
FileScanMonitor.addMonitor(
baseDirectory, instance);
} catch (JNotifyException e) {
throw new ConfigLoadException("add listener on " + baseDirectory + " failed because:" + e.getMessage());
}
//支持动态刷新文件
addConfigFileMonitor();
}
} catch (FileNotFoundException e) {
handleException("Could not find openrasp.yml, using default settings: " + e.getMessage(), e);
} catch (JNotifyException e) {
handleException("add listener on " + configFileDir + " failed because:" + e.getMessage(), e);
} catch (Exception e) {
handleException("cannot load properties file: " + e.getMessage(), e);
}
String configValidMsg = checkMajorConfig();
if (configValidMsg != null) {
LogTool.error(ErrorType.CONFIG_ERROR, configValidMsg);
throw new ConfigLoadException(configValidMsg);
}
}JS.Initialize(), JS engine initialization, js file monitoring
public synchronized static boolean Initialize() {
try {
//v8初始化检查
if (!V8.Initialize()) {
throw new Exception("[OpenRASP] Failed to initialize V8 worker threads");
}
//log
V8.SetLogger(new com.baidu.openrasp.v8.Logger() {
@Override
public void log(String msg) {
pluginLog(msg);
}
});
//
V8.SetStackGetter(new com.baidu.openrasp.v8.StackGetter() {
@Override
public byte[] get() {
try {
ByteArrayOutputStream stack = new ByteArrayOutputStream();
JsonStream.serialize(StackTrace.getParamStackTraceArray(), stack);
stack.write(0);
return stack.getByteArray();
} catch (Exception e) {
return null;
}
}
});
Context.setKeys();
//是否云端控制，比如js更新，插件更新
if (!CloudUtils.checkCloudControlEnter()) {
//更新插件
UpdatePlugin();
//js文件监听器
InitFileWatcher();
}
return true;
} catch (Exception e) {
e.printStackTrace();
LOGGER.error(e);
return false;
}
}
public synchronized static boolean UpdatePlugin(List<String[]> scripts) {
boolean rst = V8.CreateSnapshot(pluginConfig, scripts.toArray(), BuildRASPModel.getRaspVersion());
if (rst) {
try {
//执行js
String jsonString = V8.ExecuteScript("JSON.stringify(RASP.algorithmConfig || {})",
"get-algorithm-config.js");
Config.getConfig().setConfig(ConfigItem.ALGORITHM_CONFIG, jsonString, true);
} catch (Exception e) {
LogTool.error(ErrorType.PLUGIN_ERROR, e.getMessage(), e);
}
Config.commonLRUCache.clear();
}
return rst;
}CheckerManager.init();
public synchronized static void init() throws Exception {
for (Type type : Type.values()) {
checkers.put(type, type.checker);
}
}The checked class is written in the code, which means that the new class needs to be restarted, but generally the parameter update rule is added, and the frequency of new classes is relatively low
custom converter, retransform
private void initTransformer(Instrumentation inst) throws UnmodifiableClassException {
transformer = new CustomClassTransformer(inst);
transformer.retransform();
}here is the key
public CustomClassTransformer(Instrumentation inst) {
this.inst = inst;
inst.addTransformer(this, true);
//定义了哪些类需要类替换
addAnnotationHook();
}addAnnotationHook();
private void addAnnotationHook() {
//com.baidu.openrasp.hook HookAnnotation注解的类
Set<Class> classesSet = AnnotationScanner.getClassWithAnnotation(SCAN_ANNOTATION_PACKAGE, HookAnnotation.class);
for (Class clazz : classesSet) {
try {
Object object = clazz.newInstance();
if (object instanceof AbstractClassHook) {
addHook((AbstractClassHook) object, clazz.getName());
}
} catch (Exception e) {
LogTool.error(ErrorType.HOOK_ERROR, "add hook failed: " + e.getMessage(), e);
}
}
}addHook((AbstractClassHook) object, clazz.getName());//Actually cache the class that needs to be replaced, and can be used when restoring
The key is still the transform method
public byte[] transform(ClassLoader loader, String className, Class<?> classBeingRedefined,
ProtectionDomain domain, byte[] classfileBuffer) throws IllegalClassFormatException {
if (loader != null) {
DependencyFinder.addJarPath(domain);
}
if (loader != null && jspClassLoaderNames.contains(loader.getClass().getName())) {
jspClassLoaderCache.put(className.replace("/", "."), new SoftReference<ClassLoader>(loader));
}
//关键点，使用javassist字节码增强
for (final AbstractClassHook hook : hooks) {
if (hook.isClassMatched(className)) {
CtClass ctClass = null;
try {
ClassPool classPool = new ClassPool();
addLoader(classPool, loader);
ctClass = classPool.makeClass(new ByteArrayInputStream(classfileBuffer));
if (loader == null) {
hook.setLoadedByBootstrapLoader(true);
}
//字节码增强了，包括Tomcat，jdk等类
classfileBuffer = hook.transformClass(ctClass);
if (classfileBuffer != null) {
checkNecessaryHookType(hook.getType());
}
} catch (IOException e) {
e.printStackTrace();
} finally {
if (ctClass != null) {
ctClass.detach();
}
}
}
}
serverDetector.detectServer(className, loader, domain);
return classfileBuffer;
}Defines a series of buried point class replacements
Take com.baidu.openrasp.hook.file.FileHook as an example
Added a parameter check before calling the list method of File
protected void hookMethod(CtClass ctClass) throws IOException, CannotCompileException, NotFoundException {
String src = getInvokeStaticSrc(FileHook.class, "checkListFiles", "$0", File.class);
insertBefore(ctClass, "list", "()[Ljava/lang/String;", src);
}
The architecture diagram of openrasp does not correspond to the open source code. The v8 engine used in the open source code is Rhino, which is native Java code and does not require JNI.
3. Agent uninstallation logic
release(mode)
public static synchronized void release(String mode) {
try {
if (engineContainer != null) {
System.out.println("[OpenRASP] Start to release OpenRASP");
engineContainer.release(mode);
engineContainer = null;
} else {
System.out.println("[OpenRASP] Engine is initialized, skipped");
}
} catch (Throwable throwable) {
// ignore
}
}engineContainer.release(mode);
public void release(String mode) {
//任务线程停止
CloudManager.stop();
//CPU 监控线程停掉
CpuMonitorManager.release();
if (transformer != null) {
transformer.release(); //类替换还原
}
JS.Dispose(); //JS检查还原
CheckerManager.release(); //java 检查还原
String message = "[OpenRASP] Engine Released [" + Agent.projectVersion + " (build: GitCommit="
+ Agent.gitCommit + " date=" + Agent.buildTime + ")]";
System.out.println(message);
}
public synchronized static void Dispose() {
if (watchId != null) {
boolean oldValue = HookHandler.enableHook.getAndSet(false);
FileScanMonitor.removeMonitor(watchId);
watchId = null;
HookHandler.enableHook.set(oldValue);
}
}4. The process of carrying out the inspection
Take Tomcat Xss as an example, the same is true for the rest, because the hook bytecode interception of Tomcat's request is defined, and the source code at startup has been analyzed
Because it is bytecode interception, the interception level can be defined and the request can be blocked
The specific method of use is defined at startup. For example, http requests have the risk of xss. The specific interception efficiency can greatly improve the performance of the application. For example, if there is no xss, for example, there is no front-end page. Configure xss interception
Finally execute each checker
Checker inspection, and take FileHook code injection inspection as an example
This will also call the checker code, and then check the parameters
Part of the inspection means is JS inspection
Actually the logic is very clear
Bytecode injection checkpoint
Execute the check, according to the logic, you can call the checker through the plug-in, or you can directly check the return
When executing the checker, parameters can be checked through the JS engine (the js file of the js engine can dynamically update the rules)
Summarize
The source code of OpenRASP has been analyzed. In fact, the code is not complicated, but the idea is very good. In the actual use process
To consider the performance impact,
Consider whether to block and intercept the impact on business
It is necessary to consider the use logic of each component, which can be customized and supports dynamic updates. After all, there is still a risk of bytecode replacement, so the data for inspection is generally updated, so the js engine is introduced and updated through js
In the end, we still have to consider the actual needs, such as the intranet environment, after all, the cost is very high.