[Security] Getting Started with Linux Audit

1 Introduction to audit

Audit is an audit mechanism provided by the Linux kernel. Since audit is provided by the kernel, the process of using audit includes kernel space and user space:

• rules: Audit rules, which configure the operations that need to be audited by the audit system
• auditctl: user mode program used to audit rule configuration and configuration changes
• kaudit: kernel space program that records events according to configured audit rules
• auditd: user mode program, obtains audit logs through netlink

Usual usage process:

• Users configure audit rules through auditctl
• After the kernel's kauditd program obtains the audit rules, it records the corresponding audit logs.
• Auditd in user mode obtains the audit log and writes it to the log file.

The main application scenario of audit is security auditing, which detects abnormal behaviors by analyzing logs.

2 Use of auditctl

auditctl is a user-mode control program that can modify the audit configuration and the operation of audit rules.

The options of auditctl can be divided into two categories.

Configuration class:

• -b: configure buffer size
• -e: Set the enabled flag
• -f: Set failure flag
• -s: Returns the overall status
• –backlog_wait_time：设置backlog_wait_time

Audit rule class:

• -a & -A l,a: Add behaviors that need to be recorded to a rule table
• -d: Delete rules from a rule table
• -D: delete all rules
• -F f=v: Set more monitoring conditions
• -l: View rules
• -p: Set permission filtering on file monitoring
• -i: Ignore errors when reading rules from a file
• -c: continue on error
• -r: Set rate_limit, how many messages per second
• -R: Read rules from file
• -S: Set the system call name or system call number to be monitored
• -w: Add monitoring points
• -W: Delete monitoring points

For example, if we want to get events that call the execve system call, we can add the following rules:

auditctl -a always,exit -S execve -F key=123456

Then you can find the log through ausearch:

ausearch -k 123456

If you want to get the events of executing the tail command, you can add rules:

auditctl -w /usr/bin/tail -p x -k 123456

Then use the tail command to view the logs through the ausearch command:

time->Sun Apr 23 15:47:36 2023
type=PROCTITLE msg=audit(1682236056.128:4318964): proctitle=7461696C002D6E0032006C756F2E7368
type=PATH msg=audit(1682236056.128:4318964): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=36969 dev=08:03 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ld_so_t:s0 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1682236056.128:4318964): item=0 name="/usr/bin/tail" inode=100666597 dev=08:03 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1682236056.128:4318964):  cwd="/root"
type=EXECVE msg=audit(1682236056.128:4318964): argc=4 a0="tail" a1="-n" a2="2" a3="luo.sh"
type=SYSCALL msg=audit(1682236056.128:4318964): arch=c000003e syscall=59 success=yes exit=0 a0=20749e0 a1=218ecd0 a2=2179ee0 a3=7fffa4a99460 items=2 ppid=58219 pid=59519 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=956 comm="tail" exe="/usr/bin/tail" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="123456"

You can see that the first line is the event where the event occurred, and the following lines are the event logs generated by executing the tail command. Some logs are very simple, such as CWD, which indicates the current path of the operation, while some logs are very complex. For example, SYSCALL has nearly 30 fields. Each line of log has type field and msg field (before the colon is the timestamp, which can be converted through the date command, and after the colon is the event ID, the same rule The event IDs of the generated events are the same. Therefore, if you do not use ausearch to find the logs generated by a certain rule, you need to first use the key to find the corresponding event ID, and then use the event ID to find all the generated logs).

For the monitoring of the tail command here, we only focus on the above two events:

• EXECVE: The parameters of the call, argc and argv are given here
• SYSCALL: arch (architecture), syscall (system call number, can be viewed through ausyscall --dump), success (whether the call is successful), exit (return code), a0~a3 are the first 4 parameters of the system call, ppid (parent process ID), pid (process ID), comm (executed command), exe (executable file that executes execve)

2 audit configuration and rules

You can see some properties and configurations of the current audit through the auditctl -s command:

• enabled: Indicates whether audit will record events, which can be set by auditctl -e
• Failure: Indicates whether audit will record failure events. Set to 1 to record failure events.
• pid: the pid of the process occupying audit
• rate_limit: The maximum number of messages sent by the kernel per second. If it is 0, it means no limit.
• backlog_limit: cache queue length limit
• lost: The number of records lost due to the cache queue exceeding the limit
• backlog: the number of records waiting to be read in the current cache queue
• backlog_wait_time: waiting time when the cache queue is full

Among them, backlog_wait_time is provided by later versions.

3 Working principle

In addition to the above uses, audit has another feature: exclusivity. The actual audit operation is completed by kauditd in the kernel, and auditd reads the audit log through netlink. Kauditd is only allowed to connect to one user mode process. Therefore, if there is already an auditd process on the system that has established a connection with kauditd, and other processes subsequently preempt it, auditd will be disconnected. So, how to determine which process is currently connected to kautid? It can be judged by the pid in auditctl -s.

Another important point is how kaudit applies configured rules. In the -a <l,a> option of auditctl, the meaning of the option given is: add the rule and the corresponding action to the end of the list. There are 4 types of lists: task, exit, user, and exclude, and 2 types of actions: never and always.

task, exit, and user respectively represent three types of audit events: user events refer to user-related events, such as user login, logout, switching, etc. Task refers to process-related events, such as process creation, exit, switching, etc. Exit refers to events related to system calls. exclude is just a keyword used to exclude files or directories that do not require auditing. Therefore, the event types here are strongly related to some other options:

• -a is used to add rules, and -w is used to monitor files. The two cannot be used at the same time. This means that in terms of implementation, four lists classified by event types are maintained, and a list of files that need to be monitored is also maintained.
• -S specifies the system call number, therefore, it can only be used with -a exit

Configuration and rule changes:

When operating configurations or rules through auditctl, the rules will be sent to the kernel through netlink. After receiving the configuration, the kernel will update the internal configuration or rules
For rules Say, the kernel (4.19.281) maintains 7 linked lists internally:

• AUDIT_FILTER_USER: User generated logs
• AUDIT_FILTER_TASK: process creation
• AUDIT_FILTER_ENTRY: system call entry
• AUDIT_FILTER_WATCH: File system monitoring
• AUDIT_FILTER_EXIT: System call exit
• AUDIT_FILTER_EXCLUDE: Audit log exclusion
• AUDIT_FILTER_FS

4 audit interface call

auditctl uses netlink to interact with the kernel. Therefore, if you want to realize some capabilities of audit, you need to use netlink to implement a set of interactive interfaces. Fortunately, there is already a library that can complete this work: yum install -y audit-libs-devel, then compile with -laudit.

After the installation is complete, you can view the header file /usr/include/libaudit.h to see the methods provided.

4.1 Obtain and modify configuration

#include <iostream>
#include <libaudit.h>

using namespace std;

int main() {

    int fd = audit_open();

    audit_request_status(fd);

    struct audit_reply reply;
    audit_get_reply(fd, &reply, GET_REPLY_BLOCKING, 0);
    struct audit_status *status;
    status = reply.status;

    cout <<"auditctl -s return:" <<endl;
    cout << "enabled=" << status->enabled << endl;
    cout << "failure=" << status->failure << endl;
    cout << "pid=" << status->pid << endl;
    cout << "rate_limit=" << status->rate_limit << endl;
    cout << "backlog_limit=" << status->backlog_limit << endl;
    cout << "lost=" << status->lost << endl;
    cout << "backlog=" << status->backlog << endl;

    return 0;
}

First try audit_request_status() to send a request to the kernel, indicating that you want to obtain configuration information, and then receive data through audit_get_reply(). The data is placed in the structure of struct audit_reply:

// /usr/src/libaudit.h
struct audit_reply {
        int                      type;
        int                      len;
        struct nlmsghdr         *nlh;
        struct audit_message     msg;

        /* Using a union to compress this structure since only one of
         * the following should be valid for any packet. */
        union {
        struct audit_status     *status;
        struct audit_rule_data  *ruledata;
        struct audit_login      *login;
        char                    *message;
        struct nlmsgerr         *error;
        struct audit_sig_info   *signal_info;
        struct daemon_conf      *conf;
#ifdef AUDIT_FEATURE_BITMAP_ALL
        struct audit_features   *features;
#endif
        };
};

If configuration information is obtained, the data is placed in status:

// include/uapi/linux/audit.h
struct audit_status {
        __u32           mask;           /* Bit mask for valid entries */
        __u32           enabled;        /* 1 = enabled, 0 = disabled */
        __u32           failure;        /* Failure-to-log action */
        __u32           pid;            /* pid of auditd process */
        __u32           rate_limit;     /* messages rate limit (per second) */
        __u32           backlog_limit;  /* waiting messages limit */
        __u32           lost;           /* messages lost */
        __u32           backlog;        /* messages waiting in queue */
        union {
                __u32   version;        /* deprecated: audit api version num */
                __u32   feature_bitmap; /* bitmap of kernel audit features */
        };
};

Therefore, just read the above fields in the status in the returned audit_reply. It should be noted that if the third parameter in audit_get_reply() is set to GET_REPLY_NONBLOCKING, the data may not be obtained because fd may not have readable data yet. Therefore, either set it to GET_REPLY_BLOCKING or use select:

#include <iostream>
#include <libaudit.h>

using namespace std;

int main() {

    struct timeval t = {
        .tv_sec = 0, .tv_usec = 500000
    };

    int fd = audit_open();

    audit_request_status(fd);

    fd_set read_mask;
    FD_ZERO(&read_mask);
    FD_SET(fd, &read_mask);
    select(fd+1, &read_mask, NULL, NULL, &t);

    struct audit_reply reply;
    audit_get_reply(fd, &reply, GET_REPLY_NONBLOCKING, 0);
    struct audit_status *status;
    status = reply.status;

    cout <<"auditctl -s return:" <<endl;
    cout << "enabled=" << status->enabled << endl;
    cout << "failure=" << status->failure << endl;
    cout << "pid=" << status->pid << endl;
    cout << "rate_limit=" << status->rate_limit << endl;
    cout << "backlog_limit=" << status->backlog_limit << endl;
    cout << "lost=" << status->lost << endl;
    cout << "backlog=" << status->backlog << endl;

    return 0;
}

For the operation of modifying the configuration, libaudit directly provides the corresponding API function. For example, to set the backlog_limit, you can directly call audit_set_backlog_limit().

4.2 Obtain and modify rules

#include <iostream>
#include <libaudit.h>

using namespace std;

int main() {

    struct timeval t = {
        .tv_sec = 0, .tv_usec = 500000
    };

    int fd = audit_open();

    do {
        audit_request_rules_list_data(fd);

        fd_set read_mask;
        FD_ZERO(&read_mask);
        FD_SET(fd, &read_mask);
        select(fd+1, &read_mask, NULL, NULL, &t);
    
        struct audit_reply reply;
        audit_get_reply(fd, &reply, GET_REPLY_NONBLOCKING, 0);
        if(reply.type == NLMSG_DONE) {
            break;
        }
        struct audit_rule_data *rules;
        rules = reply.ruledata;
    
        cout <<"auditctl -l return:" <<endl;
        cout << audit_flag_to_name(rules->flags) << endl;
        cout << audit_action_to_name(rules->action) << endl;
    } while(true);

    return 0;
}

The difference between obtaining rules and obtaining configurations is only the function that initiates the operation and the data parsing. Obtaining rules uses audit_request_rules_list_data() to initiate the operation. When parsing the data, you need to parse the array of struct audit_rule_data.

#include <iostream>
#include <libaudit.h>
#include <linux/audit.h>

using namespace std;

int main() {

    int fd = audit_open();

    struct audit_rule_data *rule = new(struct audit_rule_data);

    audit_rule_syscall_data(rule, 57);

    audit_add_rule_data(fd, rule, AUDIT_FILTER_EXIT, AUDIT_NEVER);

    return 0;
}

The above code is equivalent toauditctl -a exit,never -S execve.

#include <iostream>
#include <libaudit.h>
#include <linux/audit.h>

using namespace std;

int main() {

    int fd = audit_open();

    struct audit_rule_data *rule = new(struct audit_rule_data);

    audit_add_watch(&rule, "/etc/passwd");

    audit_add_rule_data(fd, rule, AUDIT_FILTER_EXIT, AUDIT_ALWAYS);

    return 0;
}

The above code is equivalent toauditctl -w /etc/passwd -p rwxa.

4.3 Obtain audit logs

To obtain the upgrade log or read it using netlink:

#include <iostream>
#include <libaudit.h>
#include <string.h>
#include <unistd.h>

using namespace std;

int main() {
    int audit_fd = audit_open();
    if (audit_fd < 0) {
	cout << "open audit fail:" << strerror(errno) << endl;
        return -1;
    }

    audit_set_enabled(audit_fd, 1);
    struct audit_reply audit_rep;
    int ret;
    struct timeval t = {
            .tv_sec = 5, .tv_usec = 0
        };
    pid_t cur_pid = getpid();
    ret = audit_set_pid(audit_fd, static_cast<uint32_t>(cur_pid),
                               WAIT_NO);
    if (ret <= 0) {
        cout << "audit_set_pid fail:" << strerror(errno) << endl;
        return -1;
    }
    do {
        fd_set read_mask;
        FD_ZERO(&read_mask);
        FD_SET(audit_fd, &read_mask);
        ret = select(audit_fd + 1, &read_mask, nullptr, nullptr, &t);
        if (ret <= 0) {
            cout << "select fail:" << strerror(errno) << endl;
            continue;
        }
        ret = audit_get_reply(audit_fd, &audit_rep,
                          GET_REPLY_NONBLOCKING, 0);
        if (ret <= 0) {
            cout << "open audit fail:" << strerror(errno) << endl;
        }

        printf("%s %s", __FUNCTION__, audit_rep.msg.data);
        cout << audit_rep.msg.data << endl;
    } while(true);

    return 0;
}

5 problems with audit

If you just use audit normally: configure audit rules and check the audit log, there will be no problem. However, there are still some problems during actual use.

5.1 kernel version

Different versions of the kernel have different implementation mechanisms, so the operating performance and parameter control are also different:

• Kernels less than 3.14 do not provide an interface for setting backlog_wait_time

5.2 Cache queue and disk problems caused by too many audit logs

audit_log_end puts the audit log at the end of the audit_queue queue. If there are too many audit logs, the queue may be very long and the resources occupied may increase. Therefore, the kernel also provides some parameters for control:

• backlog_limit: cache queue length limit
• backlog_wait_time: waiting time for the cache queue to be full

// audit_log_start(linux-4.19.281)
    // auditd_test_task：检查当前进程是否是audit daemon进程
    // audit_ctl_owner_current：检查当前进程是否持有audit_cmd_mutex锁
    // 因此，这里进入if的条件是：当前进程不是audit daemon进程，并且没有持有锁
	if (!(auditd_test_task(current) || audit_ctl_owner_current())) {

		// 获取audit_backlog_wait_time，就是auditctl -s中的backlog_wait_time
		long stime = audit_backlog_wait_time;

		// audit_backlog_limit就是auditctl -s中的backlog_limit，默认值是64
		// 因此，这里进入while的条件是：设置了backlog_limit，并且当前缓存队列的长度大于backlog_limit
		while (audit_backlog_limit &&
		       (skb_queue_len(&audit_queue) > audit_backlog_limit)) {
			// 唤醒kauditd处理队列中的日志
			wake_up_interruptible(&kauditd_wait);

			/* sleep if we are allowed and we haven't exhausted our
			 * backlog wait limit */
		    // 如果当前进程允许休眠，并且backlog_wait_time大于0，则进入if，backlog_wait_time默认是60s
			if (gfpflags_allow_blocking(gfp_mask) && (stime > 0)) {
				// 创建等待队列的节点
				DECLARE_WAITQUEUE(wait, current);

				// 将刚才创建的等待队列的节点wait加入到队列audit_backlog_wait中
				add_wait_queue_exclusive(&audit_backlog_wait,
							 &wait);
				set_current_state(TASK_UNINTERRUPTIBLE);

				// 让当前进程休眠一段时间
				stime = schedule_timeout(stime);

				// 将wait从audit_backlog_wait队列中移除
				remove_wait_queue(&audit_backlog_wait, &wait);
			} else {
				// 如果当前进程没有休眠，则先检查审计日志的生成速度是否超过rate_limit
				if (audit_rate_check() && printk_ratelimit())
					pr_warn("audit_backlog=%d > audit_backlog_limit=%d\n",
						skb_queue_len(&audit_queue),
						audit_backlog_limit);

				// lost自增1，并在审计日志中打印缓存队列超过限制
				audit_log_lost("backlog limit exceeded");
				return NULL;
			}
		}
	}

As can be seen from the above code, when the queue length exceeds backlog_limit, the kernel will sleep for a period of time backlog_wait_time (default 60 seconds). If backlog_limit is 0, it will not sleep, but will print the backlog limit exceeded log.

Therefore, if backlog_wait_time is not 0 and there are too many logs, it may cause the kernel to sleep frequently. In extreme cases, the system will directly freeze.

If you want to solve this problem, you can start from several aspects:

• Configure only the necessary audit rules as much as possible to prevent the generation of a large number of useless audit logs.
• Increase backlog_limit according to machine configuration. For example, backlog_limit can be set to 8193 or greater.
• Backlog_wait_time is set to 0. When there are too many logs, they are discarded directly to prevent them from affecting daily use.
• The consumer of audit logs consumes logs as quickly as possible. If possible, a discarding policy can be added to prevent the accumulation of audit logs.

When there are too many audit logs, it will also cause disk usage problems: When there are too many audit logs, a lot of disk space may be occupied.

It should be noted that even if no audit rules are configured, there may be audit logs, pam authentication, service startup, etc. in the log. The kernel will also generate audit logs in the absence of rules.

At the same time, starting from 3.16.0, the kernel has added multiple consumers, allowing multiple processes to read the audit log at the same time. If there are other processes that also read the audit and then write to the log file, the disk usage problem will be magnified. Therefore, for the problem of disk usage, you can start from the following aspects:

• Are there other processes that also read the audit log?
• Will a large number of logs be generated without configuring audit rules?

5.2 There are differences in the logs of the same command in the container environment

In a container environment, the logs of the same command may be different because the implementation of the command is different. Typically, the vi of some images is redirected to busybox, and some are the same binary files as the host. Then they generate The logs are different, which will cause difficulties in analysis.

6 Reference documents

• RHEL Audit System Reference
• Read the audit log
• Audit framework