Article directory

@[toc]

OpenWrt skip tools compilation

foreword

Analysis code

SDK analysis

Top-level `Makefile` analysis

tools/Makefile analysis

subdir.mk analysis

host-build.mk analysis

depends.mk analysis

Hard change the source code

Afterword

OpenWrt skip tools compilation

foreword

It is still time-consuming for OpenWrt to compile tools, so in order to shorten the compilation time, find a way to use the tools that have been compiled in the SDK provided by OpenWrt to use

Analysis code

When analyzing the OpenWrt source code, the more you analyze, the more code you see, and the more knowledge you need to understand. My head is getting bigger. The following will be divided into several modules to find out how to judge whether it should be. The toolsprevious rebuildnonsense There are too many, I don’t want to read nonsense, just fast forward to硬改源码

SDK analysis

In the SDK provided by OpenWrt, this article mainly uses

build_dir
staging_dir

These two folders contain the text toolsandtoolchain

top-level `Makefile`analysis

Here are two important variables first, in case someone is not clear

TOPDIRThis is the top-level path, which is the directory of the source code
CURDIRThis is the current path, which is the directory where the current file is located.

export PATH:=$(TOPDIR)/staging_dir/host/bin:$(PATH)

The following imports a PATH environment variable $(TOPDIR)/staging_dir/host/bin, which is toolsthe tool directory

$(toolchain/stamp-compile): $(tools/stamp-compile) $(if $(CONFIG_BUILDBOT),toolchain_rebuild_check)
$(target/stamp-compile): $(toolchain/stamp-compile) $(tools/stamp-compile) $(BUILD_DIR)/.prepared

worldAt the end, here is the call related to the generation toolchainof toolsthe timestamp stampfile

tools/Makefile analysis

tools_enabled = $(foreach tool,$(sort $(tools-y) $(tools-)),$(if $(filter $(tool),$(tools-y)),y,n))
$(eval $(call stampfile,$(curdir),tools,compile,,_$(subst $(space),,$(tools_enabled)),$(STAGING_DIR_HOST)))
$(eval $(call stampfile,$(curdir),tools,check,$(TMP_DIR)/.build,,$(STAGING_DIR_HOST)))

Here is also called subdir.mkto generatestamp

subdir.mk analysis

ifndef DUMP_TARGET_DB
# Parameters: <subdir> <name> <target> <depends> <config options> <stampfile location>
# 1: subdir
# 2: target
# 3: build type
# 4: build variant
# 5: all variants
define stampfile
  $(1)/stamp-$(3):=$(if $(6),$(6),$(STAGING_DIR))/stamp/.$(2)_$(3)$(5)
  $$($(1)/stamp-$(3)): $(TMP_DIR)/.build $(4)
	@+$(SCRIPT_DIR)/timestamp.pl -n $$($(1)/stamp-$(3)) $(1) $(4) || \
		$(MAKE) $(if $(QUIET),--no-print-directory) $$($(1)lags-$(3)) $(1)/$(3)
	@mkdir -p $$$$(dirname $$($(1)/stamp-$(3)))
	@touch $$($(1)/stamp-$(3))

  $$(if $(call debug,$(1),v),,.SILENT: $$($(1)/stamp-$(3)))

  .PRECIOUS: $$($(1)/stamp-$(3)) # work around a make bug

  $(1)//clean:=$(1)/stamp-$(3)/clean
  $(1)/stamp-$(3)/clean: FORCE
	@rm -f $$($(1)/stamp-$(3))

endef
endif

It is also called timestamp.plto generate stamp, so I probably understand that OpenWrt relies on judgment stampto decide whether to recompile toolsandtoolchain

host-build.mk analysis

HOST_STAMP_PREPARED=$(HOST_BUILD_DIR)/.prepared$(if $(HOST_QUILT)$(DUMP),,$(shell $(call find_md5,${CURDIR} $(PKG_FILE_DEPENDS),))_$(call confvar,CONFIG_AUTOREMOVE $(HOST_PREPARED_DEPENDS)))
HOST_STAMP_CONFIGURED:=$(HOST_BUILD_DIR)/.configured
HOST_STAMP_BUILT:=$(HOST_BUILD_DIR)/.built
HOST_BUILD_PREFIX?=$(if $(IS_PACKAGE_BUILD),$(STAGING_DIR_HOSTPKG),$(STAGING_DIR_HOST))
HOST_STAMP_INSTALLED:=$(HOST_BUILD_PREFIX)/stamp/.$(PKG_NAME)_installed

ifneq ($(if $(HOST_QUILT),,$(CONFIG_AUTOREBUILD)),)
  define HostHost/Autoclean
    $(call rdep,${CURDIR} $(PKG_FILE_DEPENDS),$(HOST_STAMP_PREPARED))
    $(if $(if $(Host/Compile),$(filter prepare,$(MAKECMDGOALS)),1),,$(call rdep,$(HOST_BUILD_DIR),$(HOST_STAMP_BUILT)))
  endef
endif

HOST_BUILD_DIR --> build_dir/host/
HOST_BUILD_PREFIX--> staging_dir/host/, this toolchain-build.mkis defined in

Boldly guess, it should be judged toolswhether it is compiled and installed through several hidden files

.prepared${md5}_${confvar}
.configured
.built
.$(PKG_NAME)_installed

However, there are sum functions and sum variables HOST_STAMP_PREPAREDin it , and it is necessary to understand how to generate them.findmd5confvarPKG_FILE_DEPENDSHOST_PREPARED_DEPENDS

In , the function definition rules.mkis foundconfvar

confvar=$(shell echo '$(foreach v,$(1),$(v)=$(subst ','\'',$($(v))))' | $(MKHASH) md5)

HOST_PREPARED_DEPENDSThis stuff doesn't seem to have appeared anywhere else.
PKG_FILE_DEPENDSpackage/base-files/Makefileis defined in

PKG_FILE_DEPENDS:=$(PLATFORM_DIR)/ $(GENERIC_PLATFORM_DIR)/base-files/

$(PLATFORM_DIR)with GENERIC_PLATFORM_DIRin include/target.mk

PLATFORM_DIR:=$(TOPDIR)/target/linux/$(BOARD)
GENERIC_PLATFORM_DIR := $(TOPDIR)/target/linux/generic

Pass parameter summary

So we find_md5passed in a total of three parameters , namely ${CURDIR}, $(PLATFORM_DIR),$(GENERIC_PLATFORM_DIR)

depends.mk analysis

# define a dependency on a subtree
# parameters:
#	1: directories/files
#	2: directory dependency
#	3: tempfile for file listings
#	4: find options

DEP_FINDPARAMS := -x "*/.svn*" -x ".*" -x "*:*" -x "*\!*" -x "* *" -x "*\\\#*" -x "*/.*_check" -x "*/.*.swp" -x "*/.pkgdir*"

find_md5=find $(wildcard $(1)) -type f $(patsubst -x,-and -not -path,$(DEP_FINDPARAMS) $(2)) -printf "%p%T@\n" | sort | $(MKHASH) md5

define rdep
  .PRECIOUS: $(2)
  .SILENT: $(2)_check

  $(2): $(2)_check
  check-depends: $(2)_check

ifneq ($(wildcard $(2)),)
  $(2)_check::
	$(if $(3), \
		$(call find_md5,$(1),$(4)) > $(3).1; \
		{ [ \! -f "$(3)" ] || diff $(3) $(3).1 >/dev/null; } && \
	) \
	{ \
		[ -f "$(2)_check.1" ] && mv "$(2)_check.1"; \
	    $(TOPDIR)/scripts/timestamp.pl $(DEP_FINDPARAMS) $(4) -n $(2) $(1) && { \
			$(call debug_eval,$(SUBDIR),r,echo "No need to rebuild $(2)";) \
			touch -r "$(2)" "$(2)_check"; \
		} \
	} || { \
		$(call debug_eval,$(SUBDIR),r,echo "Need to rebuild $(2)";) \
		touch "$(2)_check"; \
	}
	$(if $(3), mv $(3).1 $(3))
else
  $(2)_check::
	$(if $(3), rm -f $(3) $(3).1)
	$(call debug_eval,$(SUBDIR),r,echo "Target $(2) not built")
endif

endef

wildcard: Expanded wildcard, here are three paths directly passed in, no processing
patsubst: Replace the wildcard, replace DEP_FINDPARAMSthe middle with-x-and -not -path
$(1): This incoming is the above three paths
$(2): null
Disassemble command parameters
The main command isfind
The second command is sortand md5sum(replace scripts/mkhash.c), calculate findthe md5 value after classification

find {
    
    path1, path2, path3} -type f -and -not -path "*/.svn*" -and -not -path ".*" -and -not -path "*:*" -and -not -path "*\!*" -and -not -path "* *" -and -not -path "*\\\#*" -and -not -path "*/.*_check" -and -not -path "*/.*.swp" -and -not -path "*/.pkgdir" -printf "%p%T@\n" | sort | md5sum | awk '{print $1}'

$(call rdep,${CURDIR} $(PKG_FILE_DEPENDS),$(HOST_STAMP_PREPARED))
$(if $(if $(Host/Compile),$(filter prepare,$(MAKECMDGOALS)),1),,$(call rdep,$(HOST_BUILD_DIR),$(HOST_STAMP_BUILT)))

The above statement corresponds to the following result

$(call rdep,tools/flock,build_dir/host/flock-2.18/.prepared${find_md5}_${confvar}
$(call rdep,build_dir/host/flock-2.18,build_dir/host/flock-2.18/.built)

The analysis is too big, and I can’t do it anymore. I can only adapt the source code to achieve the adaptation effect.

Hard change the source code

Since he uses md5 detection, I also change it to md5 detection, just tools/xxxtake md5 as a comparison according to the modification time of the folder, and change it decisivelyhost-build.mk

source code:

HOST_STAMP_PREPARED=$(HOST_BUILD_DIR)/.prepared$(if $(HOST_QUILT)$(DUMP),,$(shell $(call find_md5,${CURDIR} $(PKG_FILE_DEPENDS),))_$(call confvar,CONFIG_AUTOREMOVE $(HOST_PREPARED_DEPENDS)))

After the change:

HOST_STAMP_PREPARED=$(HOST_BUILD_DIR)/.prepared$(shell stat -c %Y ${CURDIR} | $(MKHASH) md5)

In this way, the verification method becomes what we want

Afterword

After we move SDKthe staging_dirand in the source code directorybuild_dir

mkdir -p staging_dir/host/stamp
tools_name="$(ls -F ./tools/ | grep "/$")"
for a in ${tools_name}; do
    if [ "${a}" != 'include/' ]; then
    PKG_NAME=$(cat "./tools/${a}Makefile" | grep '^PKG_NAME' | cut -d '=' -f 2 | sed 's/^[ \t]*//g')
    PKG_VERSION=$(cat "./tools/${a}Makefile" | grep '^PKG_VERSION' | cut -d '=' -f 2 | sed 's/^[ \t]*//g')

    touch staging_dir/host/stamp/.${PKG_NAME}_installed

    mkdir -p build_dir/host/${PKG_NAME}-${PKG_VERSION}

    prepared_md5=$(stat -c %Y "./tools/${a}" | md5sum | cut -d ' ' -f 1)

    touch build_dir/host/${PKG_NAME}-${PKG_VERSION}/.prepared${prepared_md5}
    touch build_dir/host/${PKG_NAME}-${PKG_VERSION}/.configured
    touch build_dir/host/${PKG_NAME}-${PKG_VERSION}/.built
    
    echo ${PKG_VERSION} >build_dir/host/${PKG_NAME}-${PKG_VERSION}/.version
    echo ${PKG_VERSION} >build_dir/host/${PKG_NAME}-${PKG_VERSION}/.tarball-version

    fi
done

In this way, the compilation can be skipped tools. The easiest way is to tools/Makefileclear all the tools inside

Enjoy it ~~