Xmake is a lightweight cross-platform build tool based on Lua.
It is very lightweight and has no dependencies because it has a built-in Lua runtime.
It uses xmake.lua to maintain project construction. Compared with makefile/CMakeLists.txt, the configuration syntax is more concise and intuitive, and it is very friendly to novices. It can get started quickly in a short time, allowing users to focus more on actual project development. superior.
We can use it to compile projects directly like Make/Ninja, or generate project files like CMake/Meson. In addition, it has a built-in package management system to help users solve the integrated use of C/C++ dependent libraries.
At present, Xmake is mainly used for the construction of C/C++ projects, but it also supports the construction of other native languages, which can realize mixed compilation with C/C++, and the compilation speed is also very fast, which can be equal to Ninja.
Xmake = Build backend + Project Generator + Package Manager + [Remote|Distributed] Build + Cache
Although not very accurate, we can still understand Xmake in the following way:
Xmake ~= Make/Ninja + CMake/Meson + Vcpkg/Conan + distcc + ccache/sccache
Introduction of new features
In this release, we've added a ton of heavyweight new features:
- Distributed compilation support
- Built-in local compilation cache
- Remote compilation cache support
With these features, we can compile large C/C++ projects much faster. In addition, they are completely cross-platform, support not only gcc/clang but also msvc, and there is no third-party dependency except the compiler, which is very convenient to use.
Therefore, using Xmake is equivalent to using it at the same time . distcc/ccache/sccache
Compared with these third-party tools, Xmake fully supports Windows and msvc, which eliminates platform differences, independent process calls, and the overhead of additional daemon processes.
In addition to these features, the new version of Xmake also adds compiling support for Keil/c51 projects, as well as support for the nvc/nvc++/nvfortran compilers in the nvidia-hpc-sdk toolchain.
Remote compilation supports user authentication
In the last version, we initially supported remote compilation, but did not provide user authentication support, which would bring some security issues. Therefore, in this version, we added user authentication support.
At present, Xmake mainly provides the following authentication mechanisms. In addition, it is also effective for distributed compilation and remote caching.
- Token authentication
- Password authentication
- Trusted Host Verification
Token authentication
This is also the default recommended method, which is more secure, more convenient to configure and connect, and does not need to enter a password every time you connect.
When we execute the command, a server and client configuration file will be generated by default, and a default token will be automatically generated, so the local direct connection does not require any configuration. xmake service
Server Authentication Configuration
The server can be configured with multiple tokens to authorize connections to different user hosts, and of course can share one token.
$ cat ~/.xmake/service/server.conf
{
known_hosts = { },
logfile = "/Users/ruki/.xmake/service/server/logs.txt",
remote_build = {
listen = "0.0.0.0:9691",
workdir = "/Users/ruki/.xmake/service/server/remote_build"
},
tokens = {
"e438d816c95958667747c318f1532c0f"
}
}
Client Authentication Configuration
The client only needs to add the token on the server to the corresponding client configuration.
$ cat ~/.xmake/service/client.conf
{
remote_build = {
connect = "127.0.0.1:9691",
token = "e438d816c95958667747c318f1532c0f"
}
}
Generate new token manually
We can also execute the following command to manually generate a new token and add it to the server configuration ourselves.
$ xmake service --gen-token
New token a7b9fc2d3bfca1472aabc38bb5f5d612 is generated!
Password authentication
We also provide an authorization mode of password authentication. Compared with token authentication, it requires users to enter a password every time they connect, and can only be connected after the verification is passed.
Server Authentication Configuration
For password authentication, we do not need to manually configure the token, just execute the following command to add a user. During the adding process, the user will be prompted to enter a password.
$ xmake service --add-user=ruki
Please input user ruki password:
123456
Add user ruki ok!
Then, xmake will generate a new token through the username and password and add it to the token list configured by the server.
$ cat ~/.xmake/service/server.conf
{
known_hosts = { },
logfile = "/Users/ruki/.xmake/service/server/logs.txt",
remote_build = {
listen = "0.0.0.0:9691",
workdir = "/Users/ruki/.xmake/service/server/remote_build"
},
tokens = {
"e438d816c95958667747c318f1532c0f",
"7889e25402413e93fd37395a636bf942"
}
}
Of course, we can also delete the specified user and password.
$xmake service --rm-user=ruki
Please input user ruki password:
123456
Remove user ruki ok!
Client Authentication Configuration
For the client, we no longer need to set the token of the server. We only need to add the user name to be connected in the connection configuration to enable password authentication. The format is:user@address:port
$ cat ~/.xmake/service/client.conf
{
remote_build = {
connect = "[email protected]:9691"
}
}
!> If the user name is removed and the token is not configured, it is anonymous mode. If the server is not configured with the token, the authentication is completely disabled and the connection is made directly.
Trusted Host Verification
In addition, in order to further improve security, we also provide server-side trusted host verification. If the ip address of the client host that can be connected is configured in the known_hosts list of the server configuration, then only these hosts can successfully connect to this host. The connection of the server and other hosts to it will be prompted as untrusted and refuse the connection, even if the token and password authentication are OK.
$ cat ~/.xmake/service/server.conf
{
logfile = "/Users/ruki/.xmake/service/logs.txt",
server = {
tokens = {
"4b928c7563a0cba10ff4c3f5ca0c8e24"
},
known_hosts = { "127.0.0.1", "xx.xx.xx.xx"}
}
}
connect to a remote server
Next, we only need to enter the root directory of the project that needs to be compiled remotely, execute the command, and connect. xmake service --connect
If it is the token authentication mode, no additional password input is required, and the connection is directly connected.
$ xmake create test
$ cd test
$ xmake service --connect
<remote_build_client>: connect 192.168.56.110:9091 ..
<remote_build_client>: connected!
<remote_build_client>: sync files in 192.168.56.110:9091 ..
Scanning files ..
Comparing 3 files ..
[+]: src/main.cpp
[+]: .gitignore
[+]: xmake.lua
3 files has been changed!
Archiving files ..
Uploading files with 1372 bytes ..
<remote_build_client>: sync files ok!
If it is password authentication, the user will be prompted to enter the password to continue the connection.
$ xmake service --connect
Please input user root password:
000000
<remote_build_client>: connect 127.0.0.1:9691 ..
<remote_build_client>: connected!
<remote_build_client>: sync files in 127.0.0.1:9691 ..
Scanning files ..
Comparing 3 files ..
[+]: xmake.lua
[+]: .gitignore
[+]: src/main.cpp
3 files has been changed!
Archiving files ..
Uploading files with 1591 bytes ..
<remote_build_client>: sync files ok!
If the password is incorrect, an error message will be displayed.
$ xmake service --connect
Please input user root password:
123
<remote_build_client>: connect 127.0.0.1:9691 ..
<remote_build_client>: connect 127.0.0.1:9691 failed, user and password are incorrect!
Distributed compilation support
Xmake provides a built-in distributed compilation service, usually it can cooperate with local compilation cache and remote compilation cache to achieve optimal compilation acceleration.
Also, it is fully cross-platform supported, we not only support gcc/clang, but also Windows and msvc well.
For cross-compilation, as long as the cross-toolchain supports it, we do not require the system environment of the server. Even if the server resources of linux, macOS and Windows are mixed, distributed compilation can be well realized.
Start the service
We can specify a parameter to enable the distributed compilation service. Of course, if this parameter is not specified, xmake will enable all server-configured services by default. --distcc
$ xmake service --distcc
<distcc_build_server>: listening 0.0.0.0:9093 ..
We can also start the service and echo detailed log information.
$ xmake service --distcc -vD
<distcc_build_server>: listening 0.0.0.0:9093 ..
Start the service in Daemon mode
$ xmake service --distcc --start
$ xmake service --distcc --restart
$ xmake service --distcc --stop
Configure the server
We first, run the command, it will automatically generate a default configuration file, store to . xmake service server.conf ~/.xmake/service/server.conf
$ xmake service
generating the config file to /Users/ruki/.xmake/service/server.conf ..
an token(590234653af52e91b9e438ed860f1a2b) is generated, we can use this token to connect service.
generating the config file to /Users/ruki/.xmake/service/client.conf ..
<distcc_build_server>: listening 0.0.0.0:9693 ..
Then, we edit it, fixing the server's listening port (optional).
$ cat ~/.xmake/service/server.conf
{
distcc_build = {
listen = "0.0.0.0:9693",
workdir = "/Users/ruki/.xmake/service/server/distcc_build"
},
known_hosts = { },
logfile = "/Users/ruki/.xmake/service/server/logs.txt",
tokens = {
"590234653af52e91b9e438ed860f1a2b"
}
}
Configure the client
The client configuration file is there , we can configure the server address that the client needs to connect to. ~/.xmake/service/client.conf
We can configure multiple server addresses and corresponding tokens in the hosts list.
!> Distributed compilation, it is recommended to use the token authentication mode, because of the password mode, each server needs to enter the password once when connecting, which is very cumbersome.
$cat ~/.xmake/service/client.conf
{
distcc_build = {
hosts = {
{
connect = "127.0.0.1:9693",
token = "590234653af52e91b9e438ed860f1a2b"
}
}
}
}
connect to the server
After configuring the authentication and server address, you can enter the following command to connect the current project to the configured server.
We need to enter, when connecting , to specify that only the distributed service is connected. --distcc
$ cd projectdir
$ xmake service --connect --distcc
<client>: connect 127.0.0.1:9693 ..
<client>: 127.0.0.1:9693 connected!
We can also connect to multiple services at the same time, such as distributed compilation and remote compilation cache services.
$ xmake service --connect --distcc --ccache
!> If there is no parameter, the default connection is the remote compilation service.
Distributed compilation project
After connecting to the server, we can perform distributed compilation like normal local compilation, for example:
$ xmake
...
[ 93%]: ccache compiling.release src/demo/network/unix_echo_client.c ----> local job
[ 93%]: ccache compiling.release src/demo/network/ipv6.c
[ 93%]: ccache compiling.release src/demo/network/ping.c
[ 93%]: distcc compiling.release src/demo/network/unix_echo_server.c. ----> distcc job
[ 93%]: distcc compiling.release src/demo/network/http.c
[ 93%]: distcc compiling.release src/demo/network/unixaddr.c
[ 93%]: distcc compiling.release src/demo/network/ipv4.c
[ 94%]: distcc compiling.release src/demo/network/ipaddr.c
[ 94%]: distcc compiling.release src/demo/math/fixed.c
[ 94%]: distcc compiling.release src/demo/libm/float.c
[ 95%]: ccache compiling.release src/demo/libm/double.c
[ 95%]: ccache compiling.release src/demo/other/test.cpp
[ 98%]: archiving.release libtbox.a
[ 99%]: linking.release demo
[100%]: build ok!
Among them, the words with distcc are remote compilation tasks, and the others are local compilation tasks. By default, xmake also enables local compilation caching to cache distributed compilation results to avoid frequent requests to the server.
In addition, we can also open the remote compilation cache and share the compilation cache with others to further accelerate the compilation of multi-person collaborative development.
Disconnect
$ xmake service --disconnect --distcc
Specify the number of parallel compilation tasks
Let's briefly introduce the number of parallel tasks currently calculated by default based on the number of host cpu cores:
local default_njob = math.ceil(ncpu * 3 / 2)
Therefore, if distributed compilation is not enabled, the default maximum number of parallel compilation tasks is this default_njob.
If distributed compilation is enabled, the default number of parallel compilation tasks is:
local maxjobs = default_njob + server_count * server_default_njob
Modify the number of local parallel tasks
We can specify the number of local parallel tasks only by passing it, but it will not affect the number of parallel tasks on the server side. -jN
$ xmake -jN
Modify the number of parallel tasks on the server
If you want to modify the number of parallel tasks on the server, you need to modify the configuration file of the client.
$cat ~/.xmake/service/client.conf
{
distcc_build = {
hosts = {
{
connect = "127.0.0.1:9693",
token = "590234653af52e91b9e438ed860f1a2b",
njob = 8 <------- modify here
},
{
connect = "192.168.01:9693",
token = "590234653af52e91b9e438ed860f1a2b",
njob = 4
}
}
}
}
For each server host, add parameter configuration to specify the number of parallel tasks that this server can provide. njob = N
Distributed compilation of Android projects
The distributed compilation service provided by xmake is completely cross-platform and supports Windows, Linux, macOS, Android, iOS and even cross-compilation.
If you want to compile the Android project, you only need to add the toolchain configuration in the server configuration and provide the NDK path. toolchains
$ cat ~/.xmake/service/server.conf
{
distcc_build = {
listen = "0.0.0.0:9693",
toolchains = {
ndk = {
ndk = "~/files/android-ndk-r21e" <------------ here
}
},
workdir = "/Users/ruki/.xmake/service/server/distcc_build"
},
known_hosts = { },
logfile = "/Users/ruki/.xmake/service/server/logs.txt",
tokens = {
"590234653af52e91b9e438ed860f1a2b"
}
}
Then, we can compile the Android project in a distributed manner like normal local compilation, and even configure multiple Windows, macOS, Linux and other different server hosts as resources of the distributed compilation service to compile it.
Just download the NDK of the corresponding platform.
$ xmake f -p android --ndk=~/files/xxxx
$ xmake
Distributed compilation of iOS projects
Compiling iOS projects is easier, because Xmake can usually detect Xcode automatically, so just switch the platform to ios like a normal local.
$ xmake f -p iphoneos
$ xmake
Distributed cross compilation configuration
If we want to distribute cross-compilation, we need to configure the toolchain sdk path on the server side, for example:
$ cat ~/.xmake/service/server.conf
{
distcc_build = {
listen = "0.0.0.0:9693",
toolchains = {
cross = {
sdkdir = "~/files/arm-linux-xxx" <------------ here
}
},
workdir = "/Users/ruki/.xmake/service/server/distcc_build"
},
known_hosts = { },
logfile = "/Users/ruki/.xmake/service/server/logs.txt",
tokens = {
"590234653af52e91b9e438ed860f1a2b"
}
}
Among them, under toolchains, each item corresponds to a toolchain, which is configured as a cross toolchain here . cross = {} toolchain("cross")
In the tool chain, we can configure , , etc., corresponding to the , and other interface configurations inside . sdkdir bindir cross toolchain("cross") set_sdkdir set_bindir set_cross
If the cross toolchain is more standardized, we usually only need to configure , and xmake can automatically detect it. sdkdir
The client-side compilation only needs to specify the sdk directory.
$ xmake f -p cross --sdk=/xxx/arm-linux-xxx
$ xmake
clear server cache
The compilation of each project on the server side will generate some cache files, which are stored according to the project granularity. We can use the following command to clear the cache corresponding to each server for the current project.
$ xmake service --clean --distcc
some internal optimizations
- Cache server-side compilation results to avoid repeated compilation
- Local cache, remote cache optimization, avoid unnecessary server communication
- Server load balancing scheduling, rational allocation of server resources
- Small files are compiled directly locally after preprocessing, which is usually faster
- Real-time compression and transmission of large files, based on lz4 fast compression
- Internal state maintenance, compared to independent tools such as distcc, avoids frequent independent process loading and time-consuming, and avoids additional communication with the daemon process
Local compilation cache support
By default, Xmake will enable the local cache. The version before 2.6.5 uses the external ccache by default, and after 2.6.6, Xmake provides a built-in cross-platform local cache solution.
Compared with third-party independent processes such as ccache, xmake's internal state maintenance is easier to optimize, and it also avoids frequent independent process loading and time-consuming, and avoids additional communication with the daemon process.
In addition, the built-in cache can better support cross-platform, and msvc on Windows can also support it well, while ccache only supports gcc/clang.
Of course, we can also disable the cache with the following command.
$ xmake f --ccache=n
Note: Whether or not the built-in local cache is used, the configuration name means c/c++ build cache, not just the name of the ccache tool. --ccache=
If we want to continue to use other external caching tools, we can also configure it in the following way.
$ xmake f --ccache=n --cxx="ccache gcc" --cc="ccache gcc"
$ xmake
Remote compilation cache support
In addition to the local cache, we also provide a remote cache service, similar to mozilla's sscache, if it is only for personal development, it is usually not used.
However, if a large-scale project is developed collaboratively by multiple people within the company, distributed compilation and local caching alone are not enough. We also need to cache the compiled object files to a separate server for sharing.
In this way, even if other people compile it for the first time, they do not need to compile it distributedly every time, and directly pull the cache from the remote to speed up the compilation.
In addition, the remote cache service provided by Xmake is also supported by all platforms, not only gcc/clang but also msvc.
Start the service
We can specify a parameter to enable the remote compilation cache service. Of course, if this parameter is not specified, xmake will enable all server-configured services by default. --ccache
$ xmake service --ccache
<remote_cache_server>: listening 0.0.0.0:9092 ..
We can also start the service and echo detailed log information.
$ xmake service --ccache -vD
<remote_cache_server>: listening 0.0.0.0:9092 ..
Start the service in Daemon mode
$ xmake service --ccache --start
$ xmake service --ccache --restart
$ xmake service --ccache --stop
Configure the server
We first, run the command, it will automatically generate a default configuration file, store to . xmake service server.conf ~/.xmake/service/server.conf
$ xmake service
generating the config file to /Users/ruki/.xmake/service/server.conf ..
an token(590234653af52e91b9e438ed860f1a2b) is generated, we can use this token to connect service.
generating the config file to /Users/ruki/.xmake/service/client.conf ..
<remote_cache_server>: listening 0.0.0.0:9692 ..
Then, we edit it, fixing the server's listening port (optional).
$ cat ~/.xmake/service/server.conf
{
distcc_build = {
listen = "0.0.0.0:9692",
workdir = "/Users/ruki/.xmake/service/server/remote_cache"
},
known_hosts = { },
logfile = "/Users/ruki/.xmake/service/server/logs.txt",
tokens = {
"590234653af52e91b9e438ed860f1a2b"
}
}
Configure the client
The client configuration file is there , we can configure the server address that the client needs to connect to. ~/.xmake/service/client.conf
We can configure multiple server addresses and corresponding tokens in the hosts list.
$cat ~/.xmake/service/client.conf
{
remote_cache = {
connect = "127.0.0.1:9692,
token = "590234653af52e91b9e438ed860f1a2b"
}
}
}
connect to the server
After configuring the authentication and server address, you can enter the following command to connect the current project to the configured server.
When connecting, we need to enter and specify that only the remote compilation cache service is connected. --ccache
$ cd projectdir
$ xmake service --connect --ccache
<client>: connect 127.0.0.1:9692 ..
<client>: 127.0.0.1:9692 connected!
We can also connect to multiple services at the same time, such as distributed compilation and remote compilation cache services.
$ xmake service --connect --distcc --ccache
!> If there is no parameter, the default connection is the remote compilation service.
Disconnect
$ xmake service --disconnect --ccache
clear server cache
We can also use the following command to clear the cache on the remote server corresponding to the current project.
$ xmake service --clean --ccache
And if we execute it, in the state of connecting to the remote service, it will also automatically clear all caches. xmake clean --all
some internal optimizations
- Pull the snapshot of the remote cache and send it back to the local through bloom filter + lz4, which is used to quickly determine whether the cache exists and avoid frequently querying the server cache information
- With the local cache, you can avoid frequent requests to the remote server and pull the cache.
- Internal state maintenance, compared with independent tools such as sscache, avoids frequent independent process loading and time-consuming, and avoids additional communication with daemons
Keil/C51 Engineering Support
We only need to bind to the c51 toolchain, Xmake can automatically detect the Keil/C51 SDK toolchain environment installed on the system, and then use it to compile.
target("hello")
add_rules("c51.binary")
set_toolchains("c51")
add_files("src/main.c")
Of course, if we don't set the toolchain, we can also switch to the c51 toolchain manually. set_toolchains("c51") xmake f --toolchain=c51
update content
new features
- #2327 : Support nvc/nvc++/nvfortran compiler in nvidia-hpc-sdk toolchain
- Add path instance interface
- #2334 : Add lz4 compression module
- #2349 : Add keil/c51 project support
- #274 : Cross-platform distributed compilation support
- Use the built-in local cache instead of ccache
Improve
- #2309 : Remote compilation supports user authorization verification
- Improve remote compilation, add support for lz4 compression
Bug fixes
- Fix the crash caused by unbalanced lua stack when selecting package version