* [Buildroot] [PATCH 1/1] package/ccache: bump version to 4.6.1
@ 2022-08-04 13:29 Dominik Michael Rauh
2022-08-05 10:24 ` [Buildroot] [PATCH v2] " Dominik Michael Rauh
0 siblings, 1 reply; 9+ messages in thread
From: Dominik Michael Rauh @ 2022-08-04 13:29 UTC (permalink / raw
To: buildroot
Cc: Anders F Björklund, Andrey Smirnov, Samuel Martin,
Dominik Michael Rauh, Thomas Petazzoni, Fabrice Fontaine
Since version 4.0, the following major changes have been made to
"ccache" which affect its integration with Buildroot:
- Changed the default cache directory location to follow the XDG base directory specification.
- Changed compression algorithm from Deflate ("zlib") to Zstandard, enabled by default.
- CMake is now used instead of Autoconf for configuration and building.
Additionally with version 4.4, an optional Redis backend for secondary
storage on any server that supports the Redis protocol (via "hiredis")
has been added.
In contrast to "ccache" versions < 4 that came bundled with "zlib", now
"zstd" can merely be downloaded at compile time, so using the version
that comes with Buildroot is the more sensible option. The same applies
to "hiredis".
Due to this, all of "ccache's" new dependencies ("host-cmake",
"host-hiredis", "host-pkgconf" and "host-zstd") have to be built before
"ccache".
Co-authored-by: Anders F Björklund <anders.f.bjorklund@gmail.com>
Signed-off-by: Dominik Michael Rauh <dmrauh@posteo.de>
---
Config.in | 8 ++++
...cache_dir-through-CCACHE_DIR-env-var.patch | 42 +++++++++++++++++++
package/ccache/ccache.hash | 6 +--
package/ccache/ccache.mk | 42 ++++++++++---------
package/cmake/cmake.mk | 9 +++-
package/hiredis/hiredis.mk | 6 +++
package/pkg-generic.mk | 2 +-
package/pkgconf/pkgconf.mk | 5 +++
package/zstd/zstd.mk | 5 +++
9 files changed, 100 insertions(+), 25 deletions(-)
create mode 100644 package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
diff --git a/Config.in b/Config.in
index 3c57c591a8..5afd06faa8 100644
--- a/Config.in
+++ b/Config.in
@@ -376,6 +376,14 @@ config BR2_CCACHE_USE_BASEDIR
the section "COMPILING IN DIFFERENT DIRECTORIES" in the ccache
manual for more information.
+config BR2_CCACHE_REDIS_BACKEND
+ bool "Enable Redis storage backend"
+ default n
+ select BR2_PACKAGE_HIREDIS
+ help
+ Enable ccache to use the Redis protocol to communicate with a
+ server for the purpose of sharing cache results with others.
+
endif
config BR2_ENABLE_DEBUG
diff --git a/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch b/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
new file mode 100644
index 0000000000..dc2c86ddf8
--- /dev/null
+++ b/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
@@ -0,0 +1,42 @@
+From 09231764c72bad287dd94c0dd67dff07f288b559 Mon Sep 17 00:00:00 2001
+From: Dominik Michael Rauh <dmrauh@posteo.de>
+Date: Thu, 4 Aug 2022 14:46:16 +0200
+Subject: [PATCH] Allow to patch "cache_dir" through "CCACHE_DIR" env var
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+Co-authored-by: Anders F Björklund <anders.f.bjorklund@gmail.com>
+Signed-off-by: Dominik Michael Rauh <dmrauh@posteo.de>
+[Upstream: not applicable due to only being needed by Buildroot]
+---
+ src/Config.cpp | 6 +++++-
+ 1 file changed, 5 insertions(+), 1 deletion(-)
+
+diff --git a/src/Config.cpp b/src/Config.cpp
+index ee4cd9f7..8b40f960 100644
+--- a/src/Config.cpp
++++ b/src/Config.cpp
+@@ -162,7 +162,6 @@ const std::unordered_map<std::string, std::string> k_env_variable_table = {
+ {"DEBUG", "debug"},
+ {"DEBUGDIR", "debug_dir"},
+ {"DEPEND", "depend_mode"},
+- {"DIR", "cache_dir"},
+ {"DIRECT", "direct_mode"},
+ {"DISABLE", "disable"},
+ {"EXTENSION", "cpp_extension"},
+@@ -613,6 +612,11 @@ Config::update_from_environment()
+ throw core::Error("CCACHE_{}{}: {}", negate ? "NO" : "", key, e.what());
+ }
+ }
++
++ const char* const env_ccache_dir = getenv("CCACHE_DIR");
++ if (env_ccache_dir && *env_ccache_dir) {
++ set_item("cache_dir", env_ccache_dir, "DIR", false, "environment");
++ }
+ }
+
+ std::string
+--
+2.30.2
+
diff --git a/package/ccache/ccache.hash b/package/ccache/ccache.hash
index 16cfad602c..980ac46d77 100644
--- a/package/ccache/ccache.hash
+++ b/package/ccache/ccache.hash
@@ -1,4 +1,4 @@
# sha256 computed locally
-sha256 a02f4e8360dc6618bc494ca35b0ae21cea080f804a4898eab1ad3fcd108eb400 ccache-3.7.12.tar.xz
-sha256 ec6b1a326ff93b2cc21df88a697ae470ff6927a55b8929e7e491b315e1563361 GPL-3.0.txt
-sha256 4f63223bcdee822d55a6768cc8399ffe06401d7a03cfe79e1dc305f50dc22c59 LICENSE.adoc
+sha256 e5d47bd3cbb504ada864124690e7c0d28ecb1f9aeac22a9976025aed9633f3d1 ccache-4.6.1.tar.xz
+sha256 80b5112739a423dfac7bed1ca8a1df3cccda3d794425441997d4462b83db4dd5 GPL-3.0.txt
+sha256 c5b8e7e1c59184fd9c0e823e6f019ec261d3ee8e2860dd5e3ad98842b730994a LICENSE.adoc
diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
index 97aa8b1c75..18fec9cfb8 100644
--- a/package/ccache/ccache.mk
+++ b/package/ccache/ccache.mk
@@ -4,27 +4,25 @@
#
################################################################################
-CCACHE_VERSION = 3.7.12
+CCACHE_VERSION = 4.6.1
CCACHE_SITE = https://github.com/ccache/ccache/releases/download/v$(CCACHE_VERSION)
CCACHE_SOURCE = ccache-$(CCACHE_VERSION).tar.xz
CCACHE_LICENSE = GPL-3.0+, others
CCACHE_LICENSE_FILES = LICENSE.adoc GPL-3.0.txt
+HOST_CCACHE_DEPENDENCIES = host-zstd
-# Force ccache to use its internal zlib. The problem is that without
-# this, ccache would link against the zlib of the build system, but we
-# might build and install a different version of zlib in $(O)/host
-# afterwards, which ccache will pick up. This might break if there is
-# a version mismatch. A solution would be to add host-zlib has a
-# dependency of ccache, but it would require tuning the zlib .mk file
-# to use HOSTCC_NOCCACHE as the compiler. Instead, we take the easy
-# path: tell ccache to use its internal copy of zlib, so that ccache
-# has zero dependency besides the C library.
-HOST_CCACHE_CONF_OPTS += --with-bundled-zlib
+HOST_CCACHE_CONF_OPTS += -DENABLE_TESTING=OFF
# We are ccache, so we can't use ccache
-HOST_CCACHE_CONF_ENV = \
- CC="$(HOSTCC_NOCCACHE)" \
- CXX="$(HOSTCXX_NOCCACHE)"
+HOST_CCACHE_CONF_OPTS += \
+ -UCMAKE_C_COMPILER_LAUNCHER \
+ -UCMAKE_CXX_COMPILER_LAUNCHER
+
+ifeq ($(BR2_PACKAGE_HIREDIS),y)
+HOST_CCACHE_DEPENDENCIES += host-hiredis
+else
+HOST_CCACHE_CONF_OPTS += -DREDIS_STORAGE_BACKEND=OFF
+endif
# Patch host-ccache as follows:
# - Use BR_CACHE_DIR instead of CCACHE_DIR, because CCACHE_DIR
@@ -33,13 +31,17 @@ HOST_CCACHE_CONF_ENV = \
# BR2_CCACHE_DIR.
# - Change hard-coded last-ditch default to match path in .config, to avoid
# the need to specify BR_CACHE_DIR when invoking ccache directly.
-# CCache replaces "%s" with the home directory of the current user,
-# So rewrite BR_CACHE_DIR to take that into consideration for SDK purpose
-HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst $(HOME)/%,\%s/%,$(BR_CACHE_DIR))
+# CCache replaces "home_dir" with the home directory of the current user,
+# so rewrite BR_CACHE_DIR to take that into consideration for SDK purpose
+HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst \"$(HOME)/%,home_dir + \"/%,\"$(BR_CACHE_DIR)\")
define HOST_CCACHE_PATCH_CONFIGURATION
- sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/ccache.c
- sed -i 's,"%s/.ccache","$(HOST_CCACHE_DEFAULT_CCACHE_DIR)",' $(@D)/src/conf.c
+ sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/Config.cpp
+ sed -i 's,home_dir + "/.ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
+ sed -i 's,getenv("XDG_CACHE_HOME"),nullptr,' $(@D)/src/Config.cpp
+ sed -i 's,home_dir + "/.cache/ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
+ sed -i 's,getenv("XDG_CONFIG_HOME"),nullptr,' $(@D)/src/Config.cpp
+ sed -i 's,home_dir + "/.config/ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
endef
HOST_CCACHE_POST_PATCH_HOOKS += HOST_CCACHE_PATCH_CONFIGURATION
@@ -62,7 +64,7 @@ endef
HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_DO_INITIAL_SETUP
endif
-$(eval $(host-autotools-package))
+$(eval $(host-cmake-package))
ifeq ($(BR2_CCACHE),y)
ccache-stats: host-ccache
diff --git a/package/cmake/cmake.mk b/package/cmake/cmake.mk
index 053658fad6..eae12c1465 100644
--- a/package/cmake/cmake.mk
+++ b/package/cmake/cmake.mk
@@ -40,9 +40,16 @@ CMAKE_CONF_OPTS = \
HOST_CMAKE_CFLAGS = $(shell echo $(HOST_CFLAGS) | sed -r "s%$(HOST_CPPFLAGS)%%")
HOST_CMAKE_CXXFLAGS = $(shell echo $(HOST_CXXFLAGS) | sed -r "s%$(HOST_CPPFLAGS)%%")
+# we are built before ccache
+HOST_CMAKE_CONFIGURE_OPTS = \
+ $(HOST_CONFIGURE_OPTS) \
+ CC="$(HOSTCC_NOCCACHE)" \
+ GCC="$(HOSTCC_NOCCACHE)" \
+ CXX="$(HOSTCXX_NOCCACHE)"
+
define HOST_CMAKE_CONFIGURE_CMDS
(cd $(@D); \
- $(HOST_CONFIGURE_OPTS) \
+ $(HOST_CMAKE_CONFIGURE_OPTS) \
CFLAGS="$(HOST_CMAKE_CFLAGS)" \
./bootstrap --prefix=$(HOST_DIR) \
--parallel=$(PARALLEL_JOBS) -- \
diff --git a/package/hiredis/hiredis.mk b/package/hiredis/hiredis.mk
index a571951fa3..2cb9991029 100644
--- a/package/hiredis/hiredis.mk
+++ b/package/hiredis/hiredis.mk
@@ -20,4 +20,10 @@ else
HIREDIS_CONF_OPTS += -DENABLE_SSL=OFF
endif
+# we are built before ccache
+HOST_HIREDIS_CONF_OPTS += \
+ -UCMAKE_C_COMPILER_LAUNCHER \
+ -UCMAKE_CXX_COMPILER_LAUNCHER
+
$(eval $(cmake-package))
+$(eval $(host-cmake-package))
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index b233b07548..cba94052a0 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -792,7 +792,7 @@ $(2)_EXTRACT_DEPENDENCIES += \
endif
ifeq ($$(BR2_CCACHE),y)
-ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache,$(1)),)
+ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache host-cmake host-hiredis host-pkgconf host-zstd,$(1)),)
$(2)_DEPENDENCIES += host-ccache
endif
endif
diff --git a/package/pkgconf/pkgconf.mk b/package/pkgconf/pkgconf.mk
index 5d65f69c10..6b187b0238 100644
--- a/package/pkgconf/pkgconf.mk
+++ b/package/pkgconf/pkgconf.mk
@@ -32,6 +32,11 @@ define HOST_PKGCONF_SHARED
$(SED) 's,@STATIC@,,' $(HOST_DIR)/bin/pkg-config
endef
+# we are built before ccache
+HOST_PKGCONF_CONF_ENV = \
+ CC="$(HOSTCC_NOCCACHE)" \
+ CXX="$(HOSTCXX_NOCCACHE)"
+
PKGCONF_POST_INSTALL_TARGET_HOOKS += PKGCONF_LINK_PKGCONFIG
HOST_PKGCONF_POST_INSTALL_HOOKS += HOST_PKGCONF_INSTALL_WRAPPER
diff --git a/package/zstd/zstd.mk b/package/zstd/zstd.mk
index e8f6315222..edc4f1c624 100644
--- a/package/zstd/zstd.mk
+++ b/package/zstd/zstd.mk
@@ -97,6 +97,11 @@ endef
HOST_ZSTD_OPTS += PREFIX=$(HOST_DIR)
+# we are built before ccache
+HOST_ZSTD_OPTS += \
+ CC="$(HOSTCC_NOCCACHE)" \
+ CXX="$(HOSTCXX_NOCCACHE)"
+
define HOST_ZSTD_BUILD_CMDS
$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) $(HOST_ZSTD_OPTS) \
-C $(@D) zstd-release lib-release
--
2.35.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [Buildroot] [PATCH v2] package/ccache: bump version to 4.6.1
2022-08-04 13:29 [Buildroot] [PATCH 1/1] package/ccache: bump version to 4.6.1 Dominik Michael Rauh
@ 2022-08-05 10:24 ` Dominik Michael Rauh
2022-08-05 13:13 ` [Buildroot] [PATCH v3] " Dominik Michael Rauh
0 siblings, 1 reply; 9+ messages in thread
From: Dominik Michael Rauh @ 2022-08-05 10:24 UTC (permalink / raw
To: buildroot
Cc: Anders F Björklund, Andrey Smirnov, Samuel Martin,
Dominik Michael Rauh, Thomas Petazzoni, Fabrice Fontaine
Since version 4.0, the following major changes have been made to ccache,
which affect its integration with Buildroot:
- Changed the default cache directory location to follow the XDG base directory specification.
- Changed compression algorithm from Deflate (zlib) to Zstandard, enabled by default.
- CMake is now used instead of Autoconf for configuration and building.
Additionally with version 4.4, an optional Redis backend for secondary
storage on any server that supports the Redis protocol (by means of
hiredis) has been added.
In contrast to ccache versions < 4, that came bundled with zlib, now
zstd can merely be downloaded at compile time. Thus using the version
that comes with Buildroot is the more sensible option. The same applies
to hiredis.
Due to this, all of ccache's new dependencies (host-cmake, host-hiredis,
host-pkgconf and host-zstd) have to be built before ccache.
Co-authored-by: Anders F Björklund <anders.f.bjorklund@gmail.com>
Signed-off-by: Dominik Michael Rauh <dmrauh@posteo.de>
---
Changes v1 -> v2:
- Improve commmit message
- Disable building host-ccache's docs
Config.in | 8 ++++
...cache_dir-through-CCACHE_DIR-env-var.patch | 42 +++++++++++++++++
package/ccache/ccache.hash | 6 +--
package/ccache/ccache.mk | 45 ++++++++++---------
package/cmake/cmake.mk | 9 +++-
package/hiredis/hiredis.mk | 6 +++
package/pkg-generic.mk | 2 +-
package/pkgconf/pkgconf.mk | 5 +++
package/zstd/zstd.mk | 5 +++
9 files changed, 103 insertions(+), 25 deletions(-)
create mode 100644 package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
diff --git a/Config.in b/Config.in
index 3c57c591a8..5afd06faa8 100644
--- a/Config.in
+++ b/Config.in
@@ -376,6 +376,14 @@ config BR2_CCACHE_USE_BASEDIR
the section "COMPILING IN DIFFERENT DIRECTORIES" in the ccache
manual for more information.
+config BR2_CCACHE_REDIS_BACKEND
+ bool "Enable Redis storage backend"
+ default n
+ select BR2_PACKAGE_HIREDIS
+ help
+ Enable ccache to use the Redis protocol to communicate with a
+ server for the purpose of sharing cache results with others.
+
endif
config BR2_ENABLE_DEBUG
diff --git a/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch b/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
new file mode 100644
index 0000000000..dc2c86ddf8
--- /dev/null
+++ b/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
@@ -0,0 +1,42 @@
+From 09231764c72bad287dd94c0dd67dff07f288b559 Mon Sep 17 00:00:00 2001
+From: Dominik Michael Rauh <dmrauh@posteo.de>
+Date: Thu, 4 Aug 2022 14:46:16 +0200
+Subject: [PATCH] Allow to patch "cache_dir" through "CCACHE_DIR" env var
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+Co-authored-by: Anders F Björklund <anders.f.bjorklund@gmail.com>
+Signed-off-by: Dominik Michael Rauh <dmrauh@posteo.de>
+[Upstream: not applicable due to only being needed by Buildroot]
+---
+ src/Config.cpp | 6 +++++-
+ 1 file changed, 5 insertions(+), 1 deletion(-)
+
+diff --git a/src/Config.cpp b/src/Config.cpp
+index ee4cd9f7..8b40f960 100644
+--- a/src/Config.cpp
++++ b/src/Config.cpp
+@@ -162,7 +162,6 @@ const std::unordered_map<std::string, std::string> k_env_variable_table = {
+ {"DEBUG", "debug"},
+ {"DEBUGDIR", "debug_dir"},
+ {"DEPEND", "depend_mode"},
+- {"DIR", "cache_dir"},
+ {"DIRECT", "direct_mode"},
+ {"DISABLE", "disable"},
+ {"EXTENSION", "cpp_extension"},
+@@ -613,6 +612,11 @@ Config::update_from_environment()
+ throw core::Error("CCACHE_{}{}: {}", negate ? "NO" : "", key, e.what());
+ }
+ }
++
++ const char* const env_ccache_dir = getenv("CCACHE_DIR");
++ if (env_ccache_dir && *env_ccache_dir) {
++ set_item("cache_dir", env_ccache_dir, "DIR", false, "environment");
++ }
+ }
+
+ std::string
+--
+2.30.2
+
diff --git a/package/ccache/ccache.hash b/package/ccache/ccache.hash
index 16cfad602c..980ac46d77 100644
--- a/package/ccache/ccache.hash
+++ b/package/ccache/ccache.hash
@@ -1,4 +1,4 @@
# sha256 computed locally
-sha256 a02f4e8360dc6618bc494ca35b0ae21cea080f804a4898eab1ad3fcd108eb400 ccache-3.7.12.tar.xz
-sha256 ec6b1a326ff93b2cc21df88a697ae470ff6927a55b8929e7e491b315e1563361 GPL-3.0.txt
-sha256 4f63223bcdee822d55a6768cc8399ffe06401d7a03cfe79e1dc305f50dc22c59 LICENSE.adoc
+sha256 e5d47bd3cbb504ada864124690e7c0d28ecb1f9aeac22a9976025aed9633f3d1 ccache-4.6.1.tar.xz
+sha256 80b5112739a423dfac7bed1ca8a1df3cccda3d794425441997d4462b83db4dd5 GPL-3.0.txt
+sha256 c5b8e7e1c59184fd9c0e823e6f019ec261d3ee8e2860dd5e3ad98842b730994a LICENSE.adoc
diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
index 97aa8b1c75..51b3c2ecba 100644
--- a/package/ccache/ccache.mk
+++ b/package/ccache/ccache.mk
@@ -4,27 +4,28 @@
#
################################################################################
-CCACHE_VERSION = 3.7.12
+CCACHE_VERSION = 4.6.1
CCACHE_SITE = https://github.com/ccache/ccache/releases/download/v$(CCACHE_VERSION)
CCACHE_SOURCE = ccache-$(CCACHE_VERSION).tar.xz
CCACHE_LICENSE = GPL-3.0+, others
CCACHE_LICENSE_FILES = LICENSE.adoc GPL-3.0.txt
+HOST_CCACHE_DEPENDENCIES = host-zstd
-# Force ccache to use its internal zlib. The problem is that without
-# this, ccache would link against the zlib of the build system, but we
-# might build and install a different version of zlib in $(O)/host
-# afterwards, which ccache will pick up. This might break if there is
-# a version mismatch. A solution would be to add host-zlib has a
-# dependency of ccache, but it would require tuning the zlib .mk file
-# to use HOSTCC_NOCCACHE as the compiler. Instead, we take the easy
-# path: tell ccache to use its internal copy of zlib, so that ccache
-# has zero dependency besides the C library.
-HOST_CCACHE_CONF_OPTS += --with-bundled-zlib
+HOST_CCACHE_CONF_OPTS += \
+ -DENABLE_DOCUMENTATION=OFF \
+ -DENABLE_TESTING=OFF
# We are ccache, so we can't use ccache
-HOST_CCACHE_CONF_ENV = \
- CC="$(HOSTCC_NOCCACHE)" \
- CXX="$(HOSTCXX_NOCCACHE)"
+HOST_CCACHE_CONF_OPTS += \
+ -UCMAKE_C_COMPILER_LAUNCHER \
+ -UCMAKE_CXX_COMPILER_LAUNCHER
+
+ifeq ($(BR2_PACKAGE_HIREDIS),y)
+HOST_CCACHE_DEPENDENCIES += host-hiredis
+HOST_CCACHE_CONF_OPTS += -DREDIS_STORAGE_BACKEND=ON
+else
+HOST_CCACHE_CONF_OPTS += -DREDIS_STORAGE_BACKEND=OFF
+endif
# Patch host-ccache as follows:
# - Use BR_CACHE_DIR instead of CCACHE_DIR, because CCACHE_DIR
@@ -33,13 +34,17 @@ HOST_CCACHE_CONF_ENV = \
# BR2_CCACHE_DIR.
# - Change hard-coded last-ditch default to match path in .config, to avoid
# the need to specify BR_CACHE_DIR when invoking ccache directly.
-# CCache replaces "%s" with the home directory of the current user,
-# So rewrite BR_CACHE_DIR to take that into consideration for SDK purpose
-HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst $(HOME)/%,\%s/%,$(BR_CACHE_DIR))
+# CCache replaces "home_dir" with the home directory of the current user,
+# so rewrite BR_CACHE_DIR to take that into consideration for SDK purpose
+HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst \"$(HOME)/%,home_dir + \"/%,\"$(BR_CACHE_DIR)\")
define HOST_CCACHE_PATCH_CONFIGURATION
- sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/ccache.c
- sed -i 's,"%s/.ccache","$(HOST_CCACHE_DEFAULT_CCACHE_DIR)",' $(@D)/src/conf.c
+ sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/Config.cpp
+ sed -i 's,home_dir + "/.ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
+ sed -i 's,getenv("XDG_CACHE_HOME"),nullptr,' $(@D)/src/Config.cpp
+ sed -i 's,home_dir + "/.cache/ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
+ sed -i 's,getenv("XDG_CONFIG_HOME"),nullptr,' $(@D)/src/Config.cpp
+ sed -i 's,home_dir + "/.config/ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
endef
HOST_CCACHE_POST_PATCH_HOOKS += HOST_CCACHE_PATCH_CONFIGURATION
@@ -62,7 +67,7 @@ endef
HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_DO_INITIAL_SETUP
endif
-$(eval $(host-autotools-package))
+$(eval $(host-cmake-package))
ifeq ($(BR2_CCACHE),y)
ccache-stats: host-ccache
diff --git a/package/cmake/cmake.mk b/package/cmake/cmake.mk
index 053658fad6..eae12c1465 100644
--- a/package/cmake/cmake.mk
+++ b/package/cmake/cmake.mk
@@ -40,9 +40,16 @@ CMAKE_CONF_OPTS = \
HOST_CMAKE_CFLAGS = $(shell echo $(HOST_CFLAGS) | sed -r "s%$(HOST_CPPFLAGS)%%")
HOST_CMAKE_CXXFLAGS = $(shell echo $(HOST_CXXFLAGS) | sed -r "s%$(HOST_CPPFLAGS)%%")
+# we are built before ccache
+HOST_CMAKE_CONFIGURE_OPTS = \
+ $(HOST_CONFIGURE_OPTS) \
+ CC="$(HOSTCC_NOCCACHE)" \
+ GCC="$(HOSTCC_NOCCACHE)" \
+ CXX="$(HOSTCXX_NOCCACHE)"
+
define HOST_CMAKE_CONFIGURE_CMDS
(cd $(@D); \
- $(HOST_CONFIGURE_OPTS) \
+ $(HOST_CMAKE_CONFIGURE_OPTS) \
CFLAGS="$(HOST_CMAKE_CFLAGS)" \
./bootstrap --prefix=$(HOST_DIR) \
--parallel=$(PARALLEL_JOBS) -- \
diff --git a/package/hiredis/hiredis.mk b/package/hiredis/hiredis.mk
index a571951fa3..2cb9991029 100644
--- a/package/hiredis/hiredis.mk
+++ b/package/hiredis/hiredis.mk
@@ -20,4 +20,10 @@ else
HIREDIS_CONF_OPTS += -DENABLE_SSL=OFF
endif
+# we are built before ccache
+HOST_HIREDIS_CONF_OPTS += \
+ -UCMAKE_C_COMPILER_LAUNCHER \
+ -UCMAKE_CXX_COMPILER_LAUNCHER
+
$(eval $(cmake-package))
+$(eval $(host-cmake-package))
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index b233b07548..cba94052a0 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -792,7 +792,7 @@ $(2)_EXTRACT_DEPENDENCIES += \
endif
ifeq ($$(BR2_CCACHE),y)
-ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache,$(1)),)
+ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache host-cmake host-hiredis host-pkgconf host-zstd,$(1)),)
$(2)_DEPENDENCIES += host-ccache
endif
endif
diff --git a/package/pkgconf/pkgconf.mk b/package/pkgconf/pkgconf.mk
index 5d65f69c10..6b187b0238 100644
--- a/package/pkgconf/pkgconf.mk
+++ b/package/pkgconf/pkgconf.mk
@@ -32,6 +32,11 @@ define HOST_PKGCONF_SHARED
$(SED) 's,@STATIC@,,' $(HOST_DIR)/bin/pkg-config
endef
+# we are built before ccache
+HOST_PKGCONF_CONF_ENV = \
+ CC="$(HOSTCC_NOCCACHE)" \
+ CXX="$(HOSTCXX_NOCCACHE)"
+
PKGCONF_POST_INSTALL_TARGET_HOOKS += PKGCONF_LINK_PKGCONFIG
HOST_PKGCONF_POST_INSTALL_HOOKS += HOST_PKGCONF_INSTALL_WRAPPER
diff --git a/package/zstd/zstd.mk b/package/zstd/zstd.mk
index e8f6315222..edc4f1c624 100644
--- a/package/zstd/zstd.mk
+++ b/package/zstd/zstd.mk
@@ -97,6 +97,11 @@ endef
HOST_ZSTD_OPTS += PREFIX=$(HOST_DIR)
+# we are built before ccache
+HOST_ZSTD_OPTS += \
+ CC="$(HOSTCC_NOCCACHE)" \
+ CXX="$(HOSTCXX_NOCCACHE)"
+
define HOST_ZSTD_BUILD_CMDS
$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) $(HOST_ZSTD_OPTS) \
-C $(@D) zstd-release lib-release
--
2.35.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [Buildroot] [PATCH v3] package/ccache: bump version to 4.6.1
2022-08-05 10:24 ` [Buildroot] [PATCH v2] " Dominik Michael Rauh
@ 2022-08-05 13:13 ` Dominik Michael Rauh
2022-08-05 14:34 ` Thomas Petazzoni via buildroot
2022-08-07 9:38 ` [Buildroot] [PATCH v4] " Dominik Michael Rauh
0 siblings, 2 replies; 9+ messages in thread
From: Dominik Michael Rauh @ 2022-08-05 13:13 UTC (permalink / raw
To: buildroot
Cc: Anders F Björklund, Andrey Smirnov, Samuel Martin,
Dominik Michael Rauh, Thomas Petazzoni, Fabrice Fontaine,
Thomas De Schampheleire
Since version 4.0, the following major changes have been made to ccache,
which affect its integration with Buildroot:
- Changed the default cache directory location to follow the XDG base directory specification.
- Changed compression algorithm from Deflate (zlib) to Zstandard, enabled by default.
- CMake is now used instead of Autoconf for configuration and building.
Additionally with version 4.4, an optional Redis backend for secondary
storage on any server that supports the Redis protocol (by means of
hiredis) has been added.
In contrast to ccache versions < 4, that came bundled with zlib, now
zstd can merely be downloaded at compile time. Thus using the version
that comes with Buildroot is the more sensible option. The same applies
to hiredis.
Due to this, all of ccache's new dependencies (host-cmake, host-hiredis,
host-pkgconf and host-zstd) have to be built before ccache.
Also add a section to Buildroot's ccache support manual on the new
secondary storage backends.
Co-authored-by: Anders F Björklund <anders.f.bjorklund@gmail.com>
Signed-off-by: Dominik Michael Rauh <dmrauh@posteo.de>
---
Changes v2 -> v3:
- Add documentation
Changes v1 -> v2:
- Improve commmit message
- Disable building host-ccache's docs
Config.in | 8 ++++
docs/manual/ccache-support.txt | 17 +++++--
...cache_dir-through-CCACHE_DIR-env-var.patch | 42 +++++++++++++++++
package/ccache/ccache.hash | 6 +--
package/ccache/ccache.mk | 45 ++++++++++---------
package/cmake/cmake.mk | 9 +++-
package/hiredis/hiredis.mk | 6 +++
package/pkg-generic.mk | 2 +-
package/pkgconf/pkgconf.mk | 5 +++
package/zstd/zstd.mk | 5 +++
10 files changed, 116 insertions(+), 29 deletions(-)
create mode 100644 package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
diff --git a/Config.in b/Config.in
index 3c57c591a8..5afd06faa8 100644
--- a/Config.in
+++ b/Config.in
@@ -376,6 +376,14 @@ config BR2_CCACHE_USE_BASEDIR
the section "COMPILING IN DIFFERENT DIRECTORIES" in the ccache
manual for more information.
+config BR2_CCACHE_REDIS_BACKEND
+ bool "Enable Redis storage backend"
+ default n
+ select BR2_PACKAGE_HIREDIS
+ help
+ Enable ccache to use the Redis protocol to communicate with a
+ server for the purpose of sharing cache results with others.
+
endif
config BR2_ENABLE_DEBUG
diff --git a/docs/manual/ccache-support.txt b/docs/manual/ccache-support.txt
index ab4f10be23..1428c81a27 100644
--- a/docs/manual/ccache-support.txt
+++ b/docs/manual/ccache-support.txt
@@ -4,7 +4,7 @@
[[ccache]]
==== Using +ccache+ in Buildroot
-http://ccache.samba.org[ccache] is a compiler cache. It stores the
+https://ccache.dev[ccache] is a compiler cache. It stores the
object files resulting from each compilation process, and is able to
skip future compilation of the same source file (with same compiler
and same arguments) by using the pre-existing object files. When doing
@@ -52,9 +52,9 @@ relative paths in the object file. Therefore, for example, the debugger
will no longer find the file, unless you cd to the output directory
first.
-See https://ccache.samba.org/manual.html#_compiling_in_different_directories[the
-ccache manual's section on "Compiling in different directories"] for
-more details about this rewriting of absolute paths.
+See https://ccache.dev/manual/latest.html#_compiling_in_different_directories[
+the ccache manual's section on "Compiling in different directories"]
+for more details about this rewriting of absolute paths.
When +ccache+ is enabled in Buildroot using the +BR2_CCACHE=y+ option:
@@ -67,3 +67,12 @@ One can override this behavior using the +BR2_USE_CCACHE+ environment
variable: when set to +1+, usage of ccache is enabled (default during
the Buildroot build), when unset or set to a value different from +1+,
usage of ccache is disabled.
+
++ccache+ has support for using the Redis protocol to communicate with
+a server for the purpose of sharing cache results with others e.g. to
+speed up the compilation process in a distributed CI/CD pipeline. For
+this, the option +BR2_CCACHE_REDIS_BACKEND+ of +ccache+ has to be set.
+Please see https://github.com/ccache/ccache/wiki/Redis-storage[ccache's
+Wiki on how to configure this Redis storage backend] and its
+https://ccache.dev/manual/latest.html#_secondary_storage_backends[manual's
+section on "Secondary storage backends"] for general information.
diff --git a/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch b/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
new file mode 100644
index 0000000000..dc2c86ddf8
--- /dev/null
+++ b/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
@@ -0,0 +1,42 @@
+From 09231764c72bad287dd94c0dd67dff07f288b559 Mon Sep 17 00:00:00 2001
+From: Dominik Michael Rauh <dmrauh@posteo.de>
+Date: Thu, 4 Aug 2022 14:46:16 +0200
+Subject: [PATCH] Allow to patch "cache_dir" through "CCACHE_DIR" env var
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+Co-authored-by: Anders F Björklund <anders.f.bjorklund@gmail.com>
+Signed-off-by: Dominik Michael Rauh <dmrauh@posteo.de>
+[Upstream: not applicable due to only being needed by Buildroot]
+---
+ src/Config.cpp | 6 +++++-
+ 1 file changed, 5 insertions(+), 1 deletion(-)
+
+diff --git a/src/Config.cpp b/src/Config.cpp
+index ee4cd9f7..8b40f960 100644
+--- a/src/Config.cpp
++++ b/src/Config.cpp
+@@ -162,7 +162,6 @@ const std::unordered_map<std::string, std::string> k_env_variable_table = {
+ {"DEBUG", "debug"},
+ {"DEBUGDIR", "debug_dir"},
+ {"DEPEND", "depend_mode"},
+- {"DIR", "cache_dir"},
+ {"DIRECT", "direct_mode"},
+ {"DISABLE", "disable"},
+ {"EXTENSION", "cpp_extension"},
+@@ -613,6 +612,11 @@ Config::update_from_environment()
+ throw core::Error("CCACHE_{}{}: {}", negate ? "NO" : "", key, e.what());
+ }
+ }
++
++ const char* const env_ccache_dir = getenv("CCACHE_DIR");
++ if (env_ccache_dir && *env_ccache_dir) {
++ set_item("cache_dir", env_ccache_dir, "DIR", false, "environment");
++ }
+ }
+
+ std::string
+--
+2.30.2
+
diff --git a/package/ccache/ccache.hash b/package/ccache/ccache.hash
index 16cfad602c..980ac46d77 100644
--- a/package/ccache/ccache.hash
+++ b/package/ccache/ccache.hash
@@ -1,4 +1,4 @@
# sha256 computed locally
-sha256 a02f4e8360dc6618bc494ca35b0ae21cea080f804a4898eab1ad3fcd108eb400 ccache-3.7.12.tar.xz
-sha256 ec6b1a326ff93b2cc21df88a697ae470ff6927a55b8929e7e491b315e1563361 GPL-3.0.txt
-sha256 4f63223bcdee822d55a6768cc8399ffe06401d7a03cfe79e1dc305f50dc22c59 LICENSE.adoc
+sha256 e5d47bd3cbb504ada864124690e7c0d28ecb1f9aeac22a9976025aed9633f3d1 ccache-4.6.1.tar.xz
+sha256 80b5112739a423dfac7bed1ca8a1df3cccda3d794425441997d4462b83db4dd5 GPL-3.0.txt
+sha256 c5b8e7e1c59184fd9c0e823e6f019ec261d3ee8e2860dd5e3ad98842b730994a LICENSE.adoc
diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
index 97aa8b1c75..a6b15a6238 100644
--- a/package/ccache/ccache.mk
+++ b/package/ccache/ccache.mk
@@ -4,27 +4,28 @@
#
################################################################################
-CCACHE_VERSION = 3.7.12
+CCACHE_VERSION = 4.6.1
CCACHE_SITE = https://github.com/ccache/ccache/releases/download/v$(CCACHE_VERSION)
CCACHE_SOURCE = ccache-$(CCACHE_VERSION).tar.xz
CCACHE_LICENSE = GPL-3.0+, others
CCACHE_LICENSE_FILES = LICENSE.adoc GPL-3.0.txt
+HOST_CCACHE_DEPENDENCIES = host-zstd
-# Force ccache to use its internal zlib. The problem is that without
-# this, ccache would link against the zlib of the build system, but we
-# might build and install a different version of zlib in $(O)/host
-# afterwards, which ccache will pick up. This might break if there is
-# a version mismatch. A solution would be to add host-zlib has a
-# dependency of ccache, but it would require tuning the zlib .mk file
-# to use HOSTCC_NOCCACHE as the compiler. Instead, we take the easy
-# path: tell ccache to use its internal copy of zlib, so that ccache
-# has zero dependency besides the C library.
-HOST_CCACHE_CONF_OPTS += --with-bundled-zlib
+HOST_CCACHE_CONF_OPTS += \
+ -DENABLE_DOCUMENTATION=OFF \
+ -DENABLE_TESTING=OFF
# We are ccache, so we can't use ccache
-HOST_CCACHE_CONF_ENV = \
- CC="$(HOSTCC_NOCCACHE)" \
- CXX="$(HOSTCXX_NOCCACHE)"
+HOST_CCACHE_CONF_OPTS += \
+ -UCMAKE_C_COMPILER_LAUNCHER \
+ -UCMAKE_CXX_COMPILER_LAUNCHER
+
+ifeq ($(BR2_CCACHE_REDIS_BACKEND),y)
+HOST_CCACHE_DEPENDENCIES += host-hiredis
+HOST_CCACHE_CONF_OPTS += -DREDIS_STORAGE_BACKEND=ON
+else
+HOST_CCACHE_CONF_OPTS += -DREDIS_STORAGE_BACKEND=OFF
+endif
# Patch host-ccache as follows:
# - Use BR_CACHE_DIR instead of CCACHE_DIR, because CCACHE_DIR
@@ -33,13 +34,17 @@ HOST_CCACHE_CONF_ENV = \
# BR2_CCACHE_DIR.
# - Change hard-coded last-ditch default to match path in .config, to avoid
# the need to specify BR_CACHE_DIR when invoking ccache directly.
-# CCache replaces "%s" with the home directory of the current user,
-# So rewrite BR_CACHE_DIR to take that into consideration for SDK purpose
-HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst $(HOME)/%,\%s/%,$(BR_CACHE_DIR))
+# CCache replaces "home_dir" with the home directory of the current user,
+# so rewrite BR_CACHE_DIR to take that into consideration for SDK purpose
+HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst \"$(HOME)/%,home_dir + \"/%,\"$(BR_CACHE_DIR)\")
define HOST_CCACHE_PATCH_CONFIGURATION
- sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/ccache.c
- sed -i 's,"%s/.ccache","$(HOST_CCACHE_DEFAULT_CCACHE_DIR)",' $(@D)/src/conf.c
+ sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/Config.cpp
+ sed -i 's,home_dir + "/.ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
+ sed -i 's,getenv("XDG_CACHE_HOME"),nullptr,' $(@D)/src/Config.cpp
+ sed -i 's,home_dir + "/.cache/ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
+ sed -i 's,getenv("XDG_CONFIG_HOME"),nullptr,' $(@D)/src/Config.cpp
+ sed -i 's,home_dir + "/.config/ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
endef
HOST_CCACHE_POST_PATCH_HOOKS += HOST_CCACHE_PATCH_CONFIGURATION
@@ -62,7 +67,7 @@ endef
HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_DO_INITIAL_SETUP
endif
-$(eval $(host-autotools-package))
+$(eval $(host-cmake-package))
ifeq ($(BR2_CCACHE),y)
ccache-stats: host-ccache
diff --git a/package/cmake/cmake.mk b/package/cmake/cmake.mk
index 053658fad6..eae12c1465 100644
--- a/package/cmake/cmake.mk
+++ b/package/cmake/cmake.mk
@@ -40,9 +40,16 @@ CMAKE_CONF_OPTS = \
HOST_CMAKE_CFLAGS = $(shell echo $(HOST_CFLAGS) | sed -r "s%$(HOST_CPPFLAGS)%%")
HOST_CMAKE_CXXFLAGS = $(shell echo $(HOST_CXXFLAGS) | sed -r "s%$(HOST_CPPFLAGS)%%")
+# we are built before ccache
+HOST_CMAKE_CONFIGURE_OPTS = \
+ $(HOST_CONFIGURE_OPTS) \
+ CC="$(HOSTCC_NOCCACHE)" \
+ GCC="$(HOSTCC_NOCCACHE)" \
+ CXX="$(HOSTCXX_NOCCACHE)"
+
define HOST_CMAKE_CONFIGURE_CMDS
(cd $(@D); \
- $(HOST_CONFIGURE_OPTS) \
+ $(HOST_CMAKE_CONFIGURE_OPTS) \
CFLAGS="$(HOST_CMAKE_CFLAGS)" \
./bootstrap --prefix=$(HOST_DIR) \
--parallel=$(PARALLEL_JOBS) -- \
diff --git a/package/hiredis/hiredis.mk b/package/hiredis/hiredis.mk
index a571951fa3..2cb9991029 100644
--- a/package/hiredis/hiredis.mk
+++ b/package/hiredis/hiredis.mk
@@ -20,4 +20,10 @@ else
HIREDIS_CONF_OPTS += -DENABLE_SSL=OFF
endif
+# we are built before ccache
+HOST_HIREDIS_CONF_OPTS += \
+ -UCMAKE_C_COMPILER_LAUNCHER \
+ -UCMAKE_CXX_COMPILER_LAUNCHER
+
$(eval $(cmake-package))
+$(eval $(host-cmake-package))
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index b233b07548..cba94052a0 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -792,7 +792,7 @@ $(2)_EXTRACT_DEPENDENCIES += \
endif
ifeq ($$(BR2_CCACHE),y)
-ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache,$(1)),)
+ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache host-cmake host-hiredis host-pkgconf host-zstd,$(1)),)
$(2)_DEPENDENCIES += host-ccache
endif
endif
diff --git a/package/pkgconf/pkgconf.mk b/package/pkgconf/pkgconf.mk
index 5d65f69c10..6b187b0238 100644
--- a/package/pkgconf/pkgconf.mk
+++ b/package/pkgconf/pkgconf.mk
@@ -32,6 +32,11 @@ define HOST_PKGCONF_SHARED
$(SED) 's,@STATIC@,,' $(HOST_DIR)/bin/pkg-config
endef
+# we are built before ccache
+HOST_PKGCONF_CONF_ENV = \
+ CC="$(HOSTCC_NOCCACHE)" \
+ CXX="$(HOSTCXX_NOCCACHE)"
+
PKGCONF_POST_INSTALL_TARGET_HOOKS += PKGCONF_LINK_PKGCONFIG
HOST_PKGCONF_POST_INSTALL_HOOKS += HOST_PKGCONF_INSTALL_WRAPPER
diff --git a/package/zstd/zstd.mk b/package/zstd/zstd.mk
index e8f6315222..edc4f1c624 100644
--- a/package/zstd/zstd.mk
+++ b/package/zstd/zstd.mk
@@ -97,6 +97,11 @@ endef
HOST_ZSTD_OPTS += PREFIX=$(HOST_DIR)
+# we are built before ccache
+HOST_ZSTD_OPTS += \
+ CC="$(HOSTCC_NOCCACHE)" \
+ CXX="$(HOSTCXX_NOCCACHE)"
+
define HOST_ZSTD_BUILD_CMDS
$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) $(HOST_ZSTD_OPTS) \
-C $(@D) zstd-release lib-release
--
2.35.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [Buildroot] [PATCH v3] package/ccache: bump version to 4.6.1
2022-08-05 13:13 ` [Buildroot] [PATCH v3] " Dominik Michael Rauh
@ 2022-08-05 14:34 ` Thomas Petazzoni via buildroot
2022-08-07 9:13 ` Dominik Michael Rauh
2022-08-07 9:38 ` [Buildroot] [PATCH v4] " Dominik Michael Rauh
1 sibling, 1 reply; 9+ messages in thread
From: Thomas Petazzoni via buildroot @ 2022-08-05 14:34 UTC (permalink / raw
To: Dominik Michael Rauh
Cc: Anders F Björklund, Andrey Smirnov, Samuel Martin, buildroot,
Fabrice Fontaine, Thomas De Schampheleire
Hello,
Thanks for your work!
I did not do a thorough review, but I spotted something that needs to
be change, see below.
On Fri, 5 Aug 2022 13:13:41 +0000
Dominik Michael Rauh <dmrauh@posteo.de> wrote:
> +config BR2_CCACHE_REDIS_BACKEND
> + bool "Enable Redis storage backend"
> + default n
This is not needed, the default for an option is to be disabled.
> + select BR2_PACKAGE_HIREDIS
This is not correct, as it enables hiredis for the target, which we
don't want. Just don't select anything here.
> +HOST_CCACHE_CONF_OPTS += \
> + -UCMAKE_C_COMPILER_LAUNCHER \
> + -UCMAKE_CXX_COMPILER_LAUNCHER
Just curious, what are these doing?
> diff --git a/package/cmake/cmake.mk b/package/cmake/cmake.mk
> index 053658fad6..eae12c1465 100644
> --- a/package/cmake/cmake.mk
> +++ b/package/cmake/cmake.mk
> @@ -40,9 +40,16 @@ CMAKE_CONF_OPTS = \
> HOST_CMAKE_CFLAGS = $(shell echo $(HOST_CFLAGS) | sed -r "s%$(HOST_CPPFLAGS)%%")
> HOST_CMAKE_CXXFLAGS = $(shell echo $(HOST_CXXFLAGS) | sed -r "s%$(HOST_CPPFLAGS)%%")
>
> +# we are built before ccache
> +HOST_CMAKE_CONFIGURE_OPTS = \
> + $(HOST_CONFIGURE_OPTS) \
> + CC="$(HOSTCC_NOCCACHE)" \
> + GCC="$(HOSTCC_NOCCACHE)" \
> + CXX="$(HOSTCXX_NOCCACHE)"
> +
> define HOST_CMAKE_CONFIGURE_CMDS
> (cd $(@D); \
> - $(HOST_CONFIGURE_OPTS) \
> + $(HOST_CMAKE_CONFIGURE_OPTS) \
It's really a pity that we need to disable ccache support to build
host-cmake, because host-cmake is notoriously slow to build... But
obviously since ccache is now using cmake as its build system, I don't
really see a much better solution.
Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Buildroot] [PATCH v3] package/ccache: bump version to 4.6.1
2022-08-05 14:34 ` Thomas Petazzoni via buildroot
@ 2022-08-07 9:13 ` Dominik Michael Rauh
2022-08-07 12:39 ` Thomas Petazzoni via buildroot
0 siblings, 1 reply; 9+ messages in thread
From: Dominik Michael Rauh @ 2022-08-07 9:13 UTC (permalink / raw
To: buildroot
Cc: anders.f.bjorklund, andrew.smirnov, s.martin49, dmrauh,
thomas.petazzoni, fontaine.fabrice, thomas.de_schampheleire
Hello Thomas,
Thanks for your initial review!
I'm going to fix the two issues you have spotted in the next iteration of my
patch!
On Fri, 5 Aug 2022 16:34:52 +0200
Thomas Petazzoni via buildroot <buildroot@buildroot.org> wrote:
> Just curious, what are these doing?
These two options remove the corresponding CMake CACHE entries that are set at
the beginning of "package/pkg-cmake.mk", when BR2_CCACHE=y (please also see
https://cmake.org/cmake/help/latest/prop_tgt/LANG_COMPILER_LAUNCHER.html). Thus
Buildroot is not trying to compile ccache using ccache :)
> It's really a pity that we need to disable ccache support to build
> host-cmake, because host-cmake is notoriously slow to build... But
> obviously since ccache is now using cmake as its build system, I don't
> really see a much better solution.
Yes, the more external dependencies ccache gains, the less beneficial it may
become. One could think about making ccache a "multi-stage build", where at
first, it is only compiled with all of its essential dependencies and then at a
later stage, all other dependencies are "added". But that's probably more
trouble than it's worth and certainly out of the question for CMake.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Buildroot] [PATCH v4] package/ccache: bump version to 4.6.1
2022-08-05 13:13 ` [Buildroot] [PATCH v3] " Dominik Michael Rauh
2022-08-05 14:34 ` Thomas Petazzoni via buildroot
@ 2022-08-07 9:38 ` Dominik Michael Rauh
2023-01-08 22:37 ` Yann E. MORIN
1 sibling, 1 reply; 9+ messages in thread
From: Dominik Michael Rauh @ 2022-08-07 9:38 UTC (permalink / raw
To: buildroot
Cc: Anders F Björklund, Andrey Smirnov, Samuel Martin,
Dominik Michael Rauh, Thomas Petazzoni, Fabrice Fontaine,
Thomas De Schampheleire
Since version 4.0, the following major changes have been made to ccache,
which affect its integration with Buildroot:
- Changed the default cache directory location to follow the XDG base directory specification.
- Changed compression algorithm from Deflate (zlib) to Zstandard, enabled by default.
- CMake is now used instead of Autoconf for configuration and building.
Additionally with version 4.4, an optional Redis backend for secondary
storage on any server that supports the Redis protocol (by means of
hiredis) has been added.
In contrast to ccache versions < 4, that came bundled with zlib, now
zstd can merely be downloaded at compile time. Thus using the version
that comes with Buildroot is the more sensible option. The same applies
to hiredis.
Due to this, all of ccache's new dependencies (host-cmake, host-hiredis,
host-pkgconf and host-zstd) have to be built before ccache.
Also add a section to Buildroot's ccache support manual on the new
secondary storage backends.
Co-authored-by: Anders F Björklund <anders.f.bjorklund@gmail.com>
Signed-off-by: Dominik Michael Rauh <dmrauh@posteo.de>
---
Changes v3-> v4 (after review by Thomas Petazzoni):
- Rectified "BR2_CCACHE_REDIS_BACKEND" option
Changes v2 -> v3:
- Add documentation
Changes v1 -> v2:
- Improve commmit message
- Disable building host-ccache's docs
Config.in | 6 +++
docs/manual/ccache-support.txt | 17 +++++--
...cache_dir-through-CCACHE_DIR-env-var.patch | 42 +++++++++++++++++
package/ccache/ccache.hash | 6 +--
package/ccache/ccache.mk | 45 ++++++++++---------
package/cmake/cmake.mk | 9 +++-
package/hiredis/hiredis.mk | 6 +++
package/pkg-generic.mk | 2 +-
package/pkgconf/pkgconf.mk | 5 +++
package/zstd/zstd.mk | 5 +++
10 files changed, 114 insertions(+), 29 deletions(-)
create mode 100644 package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
diff --git a/Config.in b/Config.in
index 3c57c591a8..dbc49d0250 100644
--- a/Config.in
+++ b/Config.in
@@ -376,6 +376,12 @@ config BR2_CCACHE_USE_BASEDIR
the section "COMPILING IN DIFFERENT DIRECTORIES" in the ccache
manual for more information.
+config BR2_CCACHE_REDIS_BACKEND
+ bool "Enable Redis storage backend"
+ help
+ Enable ccache to use the Redis protocol to communicate with a
+ server for the purpose of sharing cache results with others.
+
endif
config BR2_ENABLE_DEBUG
diff --git a/docs/manual/ccache-support.txt b/docs/manual/ccache-support.txt
index ab4f10be23..1428c81a27 100644
--- a/docs/manual/ccache-support.txt
+++ b/docs/manual/ccache-support.txt
@@ -4,7 +4,7 @@
[[ccache]]
==== Using +ccache+ in Buildroot
-http://ccache.samba.org[ccache] is a compiler cache. It stores the
+https://ccache.dev[ccache] is a compiler cache. It stores the
object files resulting from each compilation process, and is able to
skip future compilation of the same source file (with same compiler
and same arguments) by using the pre-existing object files. When doing
@@ -52,9 +52,9 @@ relative paths in the object file. Therefore, for example, the debugger
will no longer find the file, unless you cd to the output directory
first.
-See https://ccache.samba.org/manual.html#_compiling_in_different_directories[the
-ccache manual's section on "Compiling in different directories"] for
-more details about this rewriting of absolute paths.
+See https://ccache.dev/manual/latest.html#_compiling_in_different_directories[
+the ccache manual's section on "Compiling in different directories"]
+for more details about this rewriting of absolute paths.
When +ccache+ is enabled in Buildroot using the +BR2_CCACHE=y+ option:
@@ -67,3 +67,12 @@ One can override this behavior using the +BR2_USE_CCACHE+ environment
variable: when set to +1+, usage of ccache is enabled (default during
the Buildroot build), when unset or set to a value different from +1+,
usage of ccache is disabled.
+
++ccache+ has support for using the Redis protocol to communicate with
+a server for the purpose of sharing cache results with others e.g. to
+speed up the compilation process in a distributed CI/CD pipeline. For
+this, the option +BR2_CCACHE_REDIS_BACKEND+ of +ccache+ has to be set.
+Please see https://github.com/ccache/ccache/wiki/Redis-storage[ccache's
+Wiki on how to configure this Redis storage backend] and its
+https://ccache.dev/manual/latest.html#_secondary_storage_backends[manual's
+section on "Secondary storage backends"] for general information.
diff --git a/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch b/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
new file mode 100644
index 0000000000..dc2c86ddf8
--- /dev/null
+++ b/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
@@ -0,0 +1,42 @@
+From 09231764c72bad287dd94c0dd67dff07f288b559 Mon Sep 17 00:00:00 2001
+From: Dominik Michael Rauh <dmrauh@posteo.de>
+Date: Thu, 4 Aug 2022 14:46:16 +0200
+Subject: [PATCH] Allow to patch "cache_dir" through "CCACHE_DIR" env var
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+Co-authored-by: Anders F Björklund <anders.f.bjorklund@gmail.com>
+Signed-off-by: Dominik Michael Rauh <dmrauh@posteo.de>
+[Upstream: not applicable due to only being needed by Buildroot]
+---
+ src/Config.cpp | 6 +++++-
+ 1 file changed, 5 insertions(+), 1 deletion(-)
+
+diff --git a/src/Config.cpp b/src/Config.cpp
+index ee4cd9f7..8b40f960 100644
+--- a/src/Config.cpp
++++ b/src/Config.cpp
+@@ -162,7 +162,6 @@ const std::unordered_map<std::string, std::string> k_env_variable_table = {
+ {"DEBUG", "debug"},
+ {"DEBUGDIR", "debug_dir"},
+ {"DEPEND", "depend_mode"},
+- {"DIR", "cache_dir"},
+ {"DIRECT", "direct_mode"},
+ {"DISABLE", "disable"},
+ {"EXTENSION", "cpp_extension"},
+@@ -613,6 +612,11 @@ Config::update_from_environment()
+ throw core::Error("CCACHE_{}{}: {}", negate ? "NO" : "", key, e.what());
+ }
+ }
++
++ const char* const env_ccache_dir = getenv("CCACHE_DIR");
++ if (env_ccache_dir && *env_ccache_dir) {
++ set_item("cache_dir", env_ccache_dir, "DIR", false, "environment");
++ }
+ }
+
+ std::string
+--
+2.30.2
+
diff --git a/package/ccache/ccache.hash b/package/ccache/ccache.hash
index 16cfad602c..980ac46d77 100644
--- a/package/ccache/ccache.hash
+++ b/package/ccache/ccache.hash
@@ -1,4 +1,4 @@
# sha256 computed locally
-sha256 a02f4e8360dc6618bc494ca35b0ae21cea080f804a4898eab1ad3fcd108eb400 ccache-3.7.12.tar.xz
-sha256 ec6b1a326ff93b2cc21df88a697ae470ff6927a55b8929e7e491b315e1563361 GPL-3.0.txt
-sha256 4f63223bcdee822d55a6768cc8399ffe06401d7a03cfe79e1dc305f50dc22c59 LICENSE.adoc
+sha256 e5d47bd3cbb504ada864124690e7c0d28ecb1f9aeac22a9976025aed9633f3d1 ccache-4.6.1.tar.xz
+sha256 80b5112739a423dfac7bed1ca8a1df3cccda3d794425441997d4462b83db4dd5 GPL-3.0.txt
+sha256 c5b8e7e1c59184fd9c0e823e6f019ec261d3ee8e2860dd5e3ad98842b730994a LICENSE.adoc
diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
index 97aa8b1c75..a6b15a6238 100644
--- a/package/ccache/ccache.mk
+++ b/package/ccache/ccache.mk
@@ -4,27 +4,28 @@
#
################################################################################
-CCACHE_VERSION = 3.7.12
+CCACHE_VERSION = 4.6.1
CCACHE_SITE = https://github.com/ccache/ccache/releases/download/v$(CCACHE_VERSION)
CCACHE_SOURCE = ccache-$(CCACHE_VERSION).tar.xz
CCACHE_LICENSE = GPL-3.0+, others
CCACHE_LICENSE_FILES = LICENSE.adoc GPL-3.0.txt
+HOST_CCACHE_DEPENDENCIES = host-zstd
-# Force ccache to use its internal zlib. The problem is that without
-# this, ccache would link against the zlib of the build system, but we
-# might build and install a different version of zlib in $(O)/host
-# afterwards, which ccache will pick up. This might break if there is
-# a version mismatch. A solution would be to add host-zlib has a
-# dependency of ccache, but it would require tuning the zlib .mk file
-# to use HOSTCC_NOCCACHE as the compiler. Instead, we take the easy
-# path: tell ccache to use its internal copy of zlib, so that ccache
-# has zero dependency besides the C library.
-HOST_CCACHE_CONF_OPTS += --with-bundled-zlib
+HOST_CCACHE_CONF_OPTS += \
+ -DENABLE_DOCUMENTATION=OFF \
+ -DENABLE_TESTING=OFF
# We are ccache, so we can't use ccache
-HOST_CCACHE_CONF_ENV = \
- CC="$(HOSTCC_NOCCACHE)" \
- CXX="$(HOSTCXX_NOCCACHE)"
+HOST_CCACHE_CONF_OPTS += \
+ -UCMAKE_C_COMPILER_LAUNCHER \
+ -UCMAKE_CXX_COMPILER_LAUNCHER
+
+ifeq ($(BR2_CCACHE_REDIS_BACKEND),y)
+HOST_CCACHE_DEPENDENCIES += host-hiredis
+HOST_CCACHE_CONF_OPTS += -DREDIS_STORAGE_BACKEND=ON
+else
+HOST_CCACHE_CONF_OPTS += -DREDIS_STORAGE_BACKEND=OFF
+endif
# Patch host-ccache as follows:
# - Use BR_CACHE_DIR instead of CCACHE_DIR, because CCACHE_DIR
@@ -33,13 +34,17 @@ HOST_CCACHE_CONF_ENV = \
# BR2_CCACHE_DIR.
# - Change hard-coded last-ditch default to match path in .config, to avoid
# the need to specify BR_CACHE_DIR when invoking ccache directly.
-# CCache replaces "%s" with the home directory of the current user,
-# So rewrite BR_CACHE_DIR to take that into consideration for SDK purpose
-HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst $(HOME)/%,\%s/%,$(BR_CACHE_DIR))
+# CCache replaces "home_dir" with the home directory of the current user,
+# so rewrite BR_CACHE_DIR to take that into consideration for SDK purpose
+HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst \"$(HOME)/%,home_dir + \"/%,\"$(BR_CACHE_DIR)\")
define HOST_CCACHE_PATCH_CONFIGURATION
- sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/ccache.c
- sed -i 's,"%s/.ccache","$(HOST_CCACHE_DEFAULT_CCACHE_DIR)",' $(@D)/src/conf.c
+ sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/Config.cpp
+ sed -i 's,home_dir + "/.ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
+ sed -i 's,getenv("XDG_CACHE_HOME"),nullptr,' $(@D)/src/Config.cpp
+ sed -i 's,home_dir + "/.cache/ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
+ sed -i 's,getenv("XDG_CONFIG_HOME"),nullptr,' $(@D)/src/Config.cpp
+ sed -i 's,home_dir + "/.config/ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
endef
HOST_CCACHE_POST_PATCH_HOOKS += HOST_CCACHE_PATCH_CONFIGURATION
@@ -62,7 +67,7 @@ endef
HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_DO_INITIAL_SETUP
endif
-$(eval $(host-autotools-package))
+$(eval $(host-cmake-package))
ifeq ($(BR2_CCACHE),y)
ccache-stats: host-ccache
diff --git a/package/cmake/cmake.mk b/package/cmake/cmake.mk
index 053658fad6..eae12c1465 100644
--- a/package/cmake/cmake.mk
+++ b/package/cmake/cmake.mk
@@ -40,9 +40,16 @@ CMAKE_CONF_OPTS = \
HOST_CMAKE_CFLAGS = $(shell echo $(HOST_CFLAGS) | sed -r "s%$(HOST_CPPFLAGS)%%")
HOST_CMAKE_CXXFLAGS = $(shell echo $(HOST_CXXFLAGS) | sed -r "s%$(HOST_CPPFLAGS)%%")
+# we are built before ccache
+HOST_CMAKE_CONFIGURE_OPTS = \
+ $(HOST_CONFIGURE_OPTS) \
+ CC="$(HOSTCC_NOCCACHE)" \
+ GCC="$(HOSTCC_NOCCACHE)" \
+ CXX="$(HOSTCXX_NOCCACHE)"
+
define HOST_CMAKE_CONFIGURE_CMDS
(cd $(@D); \
- $(HOST_CONFIGURE_OPTS) \
+ $(HOST_CMAKE_CONFIGURE_OPTS) \
CFLAGS="$(HOST_CMAKE_CFLAGS)" \
./bootstrap --prefix=$(HOST_DIR) \
--parallel=$(PARALLEL_JOBS) -- \
diff --git a/package/hiredis/hiredis.mk b/package/hiredis/hiredis.mk
index a571951fa3..2cb9991029 100644
--- a/package/hiredis/hiredis.mk
+++ b/package/hiredis/hiredis.mk
@@ -20,4 +20,10 @@ else
HIREDIS_CONF_OPTS += -DENABLE_SSL=OFF
endif
+# we are built before ccache
+HOST_HIREDIS_CONF_OPTS += \
+ -UCMAKE_C_COMPILER_LAUNCHER \
+ -UCMAKE_CXX_COMPILER_LAUNCHER
+
$(eval $(cmake-package))
+$(eval $(host-cmake-package))
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index b233b07548..cba94052a0 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -792,7 +792,7 @@ $(2)_EXTRACT_DEPENDENCIES += \
endif
ifeq ($$(BR2_CCACHE),y)
-ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache,$(1)),)
+ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache host-cmake host-hiredis host-pkgconf host-zstd,$(1)),)
$(2)_DEPENDENCIES += host-ccache
endif
endif
diff --git a/package/pkgconf/pkgconf.mk b/package/pkgconf/pkgconf.mk
index 5d65f69c10..6b187b0238 100644
--- a/package/pkgconf/pkgconf.mk
+++ b/package/pkgconf/pkgconf.mk
@@ -32,6 +32,11 @@ define HOST_PKGCONF_SHARED
$(SED) 's,@STATIC@,,' $(HOST_DIR)/bin/pkg-config
endef
+# we are built before ccache
+HOST_PKGCONF_CONF_ENV = \
+ CC="$(HOSTCC_NOCCACHE)" \
+ CXX="$(HOSTCXX_NOCCACHE)"
+
PKGCONF_POST_INSTALL_TARGET_HOOKS += PKGCONF_LINK_PKGCONFIG
HOST_PKGCONF_POST_INSTALL_HOOKS += HOST_PKGCONF_INSTALL_WRAPPER
diff --git a/package/zstd/zstd.mk b/package/zstd/zstd.mk
index e8f6315222..edc4f1c624 100644
--- a/package/zstd/zstd.mk
+++ b/package/zstd/zstd.mk
@@ -97,6 +97,11 @@ endef
HOST_ZSTD_OPTS += PREFIX=$(HOST_DIR)
+# we are built before ccache
+HOST_ZSTD_OPTS += \
+ CC="$(HOSTCC_NOCCACHE)" \
+ CXX="$(HOSTCXX_NOCCACHE)"
+
define HOST_ZSTD_BUILD_CMDS
$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) $(HOST_ZSTD_OPTS) \
-C $(@D) zstd-release lib-release
--
2.37.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [Buildroot] [PATCH v3] package/ccache: bump version to 4.6.1
2022-08-07 9:13 ` Dominik Michael Rauh
@ 2022-08-07 12:39 ` Thomas Petazzoni via buildroot
2022-08-07 17:55 ` Dominik Michael Rauh
0 siblings, 1 reply; 9+ messages in thread
From: Thomas Petazzoni via buildroot @ 2022-08-07 12:39 UTC (permalink / raw
To: Dominik Michael Rauh
Cc: anders.f.bjorklund, andrew.smirnov, s.martin49, buildroot,
fontaine.fabrice, thomas.de_schampheleire
On Sun, 7 Aug 2022 09:13:41 +0000
Dominik Michael Rauh <dmrauh@posteo.de> wrote:
> > Just curious, what are these doing?
>
> These two options remove the corresponding CMake CACHE entries that are set at
> the beginning of "package/pkg-cmake.mk", when BR2_CCACHE=y (please also see
> https://cmake.org/cmake/help/latest/prop_tgt/LANG_COMPILER_LAUNCHER.html). Thus
> Buildroot is not trying to compile ccache using ccache :)
OK, makes sense!
> > It's really a pity that we need to disable ccache support to build
> > host-cmake, because host-cmake is notoriously slow to build... But
> > obviously since ccache is now using cmake as its build system, I don't
> > really see a much better solution.
>
> Yes, the more external dependencies ccache gains, the less beneficial it may
> become. One could think about making ccache a "multi-stage build", where at
> first, it is only compiled with all of its essential dependencies and then at a
> later stage, all other dependencies are "added". But that's probably more
> trouble than it's worth and certainly out of the question for CMake.
A possible alternative we could explore is to no longer build ccache
ourselves and rely on ccache being provided by the host distribution.
Of course, raises the question of which ccache version is available on
the host, and how to tweak its configuration to match the needs of
Buildroot.
Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Buildroot] [PATCH v3] package/ccache: bump version to 4.6.1
2022-08-07 12:39 ` Thomas Petazzoni via buildroot
@ 2022-08-07 17:55 ` Dominik Michael Rauh
0 siblings, 0 replies; 9+ messages in thread
From: Dominik Michael Rauh @ 2022-08-07 17:55 UTC (permalink / raw
To: buildroot
Cc: anders.f.bjorklund, andrew.smirnov, s.martin49, dmrauh,
thomas.petazzoni, fontaine.fabrice, thomas.de_schampheleire
Hi Thomas,
> A possible alternative we could explore is to no longer build ccache
> ourselves and rely on ccache being provided by the host distribution.
> Of course, raises the question of which ccache version is available on
> the host, and how to tweak its configuration to match the needs of
> Buildroot.
Another option that comes to mind, is to detect whether ccache is already
installed on the host and use that until Buildroot's ccache can take over.
Best regards,
Dominik
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Buildroot] [PATCH v4] package/ccache: bump version to 4.6.1
2022-08-07 9:38 ` [Buildroot] [PATCH v4] " Dominik Michael Rauh
@ 2023-01-08 22:37 ` Yann E. MORIN
0 siblings, 0 replies; 9+ messages in thread
From: Yann E. MORIN @ 2023-01-08 22:37 UTC (permalink / raw
To: Dominik Michael Rauh
Cc: Anders F Björklund, Andrey Smirnov, Samuel Martin,
James Hilliard, Thomas Petazzoni, buildroot, Fabrice Fontaine,
Thomas De Schampheleire
Dominik, Anders, All,
+James
On 2022-08-07 09:38 +0000, Dominik Michael Rauh spake thusly:
> Since version 4.0, the following major changes have been made to ccache,
> which affect its integration with Buildroot:
>
> - Changed the default cache directory location to follow the XDG base directory specification.
> - Changed compression algorithm from Deflate (zlib) to Zstandard, enabled by default.
> - CMake is now used instead of Autoconf for configuration and building.
>
> Additionally with version 4.4, an optional Redis backend for secondary
> storage on any server that supports the Redis protocol (by means of
> hiredis) has been added.
>
> In contrast to ccache versions < 4, that came bundled with zlib, now
> zstd can merely be downloaded at compile time. Thus using the version
> that comes with Buildroot is the more sensible option. The same applies
> to hiredis.
>
> Due to this, all of ccache's new dependencies (host-cmake, host-hiredis,
> host-pkgconf and host-zstd) have to be built before ccache.
>
> Also add a section to Buildroot's ccache support manual on the new
> secondary storage backends.
>
> Co-authored-by: Anders F Björklund <anders.f.bjorklund@gmail.com>
> Signed-off-by: Dominik Michael Rauh <dmrauh@posteo.de>
Thanks a lot for this original patch! It helped me a lot when I was
reviewing and applying the series from James.
I applied patches from James instead, because they were split into more
manageable patches, rather than a big patch that changed everything at
once and, unless I completely borked and botched those patches when
aplying, could be applied as preparatory patches without actually
requiring bumping ccache.
Sorry that we did not have time to review and apply your patch sp far.
Still, I did refer a lot to this patch to understand parts of the
changes!
Now, we are left with just the need to bump ccache, which should be
slightly easier now.
Regards,
Yann E. MORIN.
> ---
> Changes v3-> v4 (after review by Thomas Petazzoni):
> - Rectified "BR2_CCACHE_REDIS_BACKEND" option
>
> Changes v2 -> v3:
> - Add documentation
>
> Changes v1 -> v2:
> - Improve commmit message
> - Disable building host-ccache's docs
>
> Config.in | 6 +++
> docs/manual/ccache-support.txt | 17 +++++--
> ...cache_dir-through-CCACHE_DIR-env-var.patch | 42 +++++++++++++++++
> package/ccache/ccache.hash | 6 +--
> package/ccache/ccache.mk | 45 ++++++++++---------
> package/cmake/cmake.mk | 9 +++-
> package/hiredis/hiredis.mk | 6 +++
> package/pkg-generic.mk | 2 +-
> package/pkgconf/pkgconf.mk | 5 +++
> package/zstd/zstd.mk | 5 +++
> 10 files changed, 114 insertions(+), 29 deletions(-)
> create mode 100644 package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
>
> diff --git a/Config.in b/Config.in
> index 3c57c591a8..dbc49d0250 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -376,6 +376,12 @@ config BR2_CCACHE_USE_BASEDIR
> the section "COMPILING IN DIFFERENT DIRECTORIES" in the ccache
> manual for more information.
>
> +config BR2_CCACHE_REDIS_BACKEND
> + bool "Enable Redis storage backend"
> + help
> + Enable ccache to use the Redis protocol to communicate with a
> + server for the purpose of sharing cache results with others.
> +
> endif
>
> config BR2_ENABLE_DEBUG
> diff --git a/docs/manual/ccache-support.txt b/docs/manual/ccache-support.txt
> index ab4f10be23..1428c81a27 100644
> --- a/docs/manual/ccache-support.txt
> +++ b/docs/manual/ccache-support.txt
> @@ -4,7 +4,7 @@
> [[ccache]]
> ==== Using +ccache+ in Buildroot
>
> -http://ccache.samba.org[ccache] is a compiler cache. It stores the
> +https://ccache.dev[ccache] is a compiler cache. It stores the
> object files resulting from each compilation process, and is able to
> skip future compilation of the same source file (with same compiler
> and same arguments) by using the pre-existing object files. When doing
> @@ -52,9 +52,9 @@ relative paths in the object file. Therefore, for example, the debugger
> will no longer find the file, unless you cd to the output directory
> first.
>
> -See https://ccache.samba.org/manual.html#_compiling_in_different_directories[the
> -ccache manual's section on "Compiling in different directories"] for
> -more details about this rewriting of absolute paths.
> +See https://ccache.dev/manual/latest.html#_compiling_in_different_directories[
> +the ccache manual's section on "Compiling in different directories"]
> +for more details about this rewriting of absolute paths.
>
> When +ccache+ is enabled in Buildroot using the +BR2_CCACHE=y+ option:
>
> @@ -67,3 +67,12 @@ One can override this behavior using the +BR2_USE_CCACHE+ environment
> variable: when set to +1+, usage of ccache is enabled (default during
> the Buildroot build), when unset or set to a value different from +1+,
> usage of ccache is disabled.
> +
> ++ccache+ has support for using the Redis protocol to communicate with
> +a server for the purpose of sharing cache results with others e.g. to
> +speed up the compilation process in a distributed CI/CD pipeline. For
> +this, the option +BR2_CCACHE_REDIS_BACKEND+ of +ccache+ has to be set.
> +Please see https://github.com/ccache/ccache/wiki/Redis-storage[ccache's
> +Wiki on how to configure this Redis storage backend] and its
> +https://ccache.dev/manual/latest.html#_secondary_storage_backends[manual's
> +section on "Secondary storage backends"] for general information.
> diff --git a/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch b/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
> new file mode 100644
> index 0000000000..dc2c86ddf8
> --- /dev/null
> +++ b/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
> @@ -0,0 +1,42 @@
> +From 09231764c72bad287dd94c0dd67dff07f288b559 Mon Sep 17 00:00:00 2001
> +From: Dominik Michael Rauh <dmrauh@posteo.de>
> +Date: Thu, 4 Aug 2022 14:46:16 +0200
> +Subject: [PATCH] Allow to patch "cache_dir" through "CCACHE_DIR" env var
> +MIME-Version: 1.0
> +Content-Type: text/plain; charset=UTF-8
> +Content-Transfer-Encoding: 8bit
> +
> +Co-authored-by: Anders F Björklund <anders.f.bjorklund@gmail.com>
> +Signed-off-by: Dominik Michael Rauh <dmrauh@posteo.de>
> +[Upstream: not applicable due to only being needed by Buildroot]
> +---
> + src/Config.cpp | 6 +++++-
> + 1 file changed, 5 insertions(+), 1 deletion(-)
> +
> +diff --git a/src/Config.cpp b/src/Config.cpp
> +index ee4cd9f7..8b40f960 100644
> +--- a/src/Config.cpp
> ++++ b/src/Config.cpp
> +@@ -162,7 +162,6 @@ const std::unordered_map<std::string, std::string> k_env_variable_table = {
> + {"DEBUG", "debug"},
> + {"DEBUGDIR", "debug_dir"},
> + {"DEPEND", "depend_mode"},
> +- {"DIR", "cache_dir"},
> + {"DIRECT", "direct_mode"},
> + {"DISABLE", "disable"},
> + {"EXTENSION", "cpp_extension"},
> +@@ -613,6 +612,11 @@ Config::update_from_environment()
> + throw core::Error("CCACHE_{}{}: {}", negate ? "NO" : "", key, e.what());
> + }
> + }
> ++
> ++ const char* const env_ccache_dir = getenv("CCACHE_DIR");
> ++ if (env_ccache_dir && *env_ccache_dir) {
> ++ set_item("cache_dir", env_ccache_dir, "DIR", false, "environment");
> ++ }
> + }
> +
> + std::string
> +--
> +2.30.2
> +
> diff --git a/package/ccache/ccache.hash b/package/ccache/ccache.hash
> index 16cfad602c..980ac46d77 100644
> --- a/package/ccache/ccache.hash
> +++ b/package/ccache/ccache.hash
> @@ -1,4 +1,4 @@
> # sha256 computed locally
> -sha256 a02f4e8360dc6618bc494ca35b0ae21cea080f804a4898eab1ad3fcd108eb400 ccache-3.7.12.tar.xz
> -sha256 ec6b1a326ff93b2cc21df88a697ae470ff6927a55b8929e7e491b315e1563361 GPL-3.0.txt
> -sha256 4f63223bcdee822d55a6768cc8399ffe06401d7a03cfe79e1dc305f50dc22c59 LICENSE.adoc
> +sha256 e5d47bd3cbb504ada864124690e7c0d28ecb1f9aeac22a9976025aed9633f3d1 ccache-4.6.1.tar.xz
> +sha256 80b5112739a423dfac7bed1ca8a1df3cccda3d794425441997d4462b83db4dd5 GPL-3.0.txt
> +sha256 c5b8e7e1c59184fd9c0e823e6f019ec261d3ee8e2860dd5e3ad98842b730994a LICENSE.adoc
> diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
> index 97aa8b1c75..a6b15a6238 100644
> --- a/package/ccache/ccache.mk
> +++ b/package/ccache/ccache.mk
> @@ -4,27 +4,28 @@
> #
> ################################################################################
>
> -CCACHE_VERSION = 3.7.12
> +CCACHE_VERSION = 4.6.1
> CCACHE_SITE = https://github.com/ccache/ccache/releases/download/v$(CCACHE_VERSION)
> CCACHE_SOURCE = ccache-$(CCACHE_VERSION).tar.xz
> CCACHE_LICENSE = GPL-3.0+, others
> CCACHE_LICENSE_FILES = LICENSE.adoc GPL-3.0.txt
> +HOST_CCACHE_DEPENDENCIES = host-zstd
>
> -# Force ccache to use its internal zlib. The problem is that without
> -# this, ccache would link against the zlib of the build system, but we
> -# might build and install a different version of zlib in $(O)/host
> -# afterwards, which ccache will pick up. This might break if there is
> -# a version mismatch. A solution would be to add host-zlib has a
> -# dependency of ccache, but it would require tuning the zlib .mk file
> -# to use HOSTCC_NOCCACHE as the compiler. Instead, we take the easy
> -# path: tell ccache to use its internal copy of zlib, so that ccache
> -# has zero dependency besides the C library.
> -HOST_CCACHE_CONF_OPTS += --with-bundled-zlib
> +HOST_CCACHE_CONF_OPTS += \
> + -DENABLE_DOCUMENTATION=OFF \
> + -DENABLE_TESTING=OFF
>
> # We are ccache, so we can't use ccache
> -HOST_CCACHE_CONF_ENV = \
> - CC="$(HOSTCC_NOCCACHE)" \
> - CXX="$(HOSTCXX_NOCCACHE)"
> +HOST_CCACHE_CONF_OPTS += \
> + -UCMAKE_C_COMPILER_LAUNCHER \
> + -UCMAKE_CXX_COMPILER_LAUNCHER
> +
> +ifeq ($(BR2_CCACHE_REDIS_BACKEND),y)
> +HOST_CCACHE_DEPENDENCIES += host-hiredis
> +HOST_CCACHE_CONF_OPTS += -DREDIS_STORAGE_BACKEND=ON
> +else
> +HOST_CCACHE_CONF_OPTS += -DREDIS_STORAGE_BACKEND=OFF
> +endif
>
> # Patch host-ccache as follows:
> # - Use BR_CACHE_DIR instead of CCACHE_DIR, because CCACHE_DIR
> @@ -33,13 +34,17 @@ HOST_CCACHE_CONF_ENV = \
> # BR2_CCACHE_DIR.
> # - Change hard-coded last-ditch default to match path in .config, to avoid
> # the need to specify BR_CACHE_DIR when invoking ccache directly.
> -# CCache replaces "%s" with the home directory of the current user,
> -# So rewrite BR_CACHE_DIR to take that into consideration for SDK purpose
> -HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst $(HOME)/%,\%s/%,$(BR_CACHE_DIR))
> +# CCache replaces "home_dir" with the home directory of the current user,
> +# so rewrite BR_CACHE_DIR to take that into consideration for SDK purpose
> +HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst \"$(HOME)/%,home_dir + \"/%,\"$(BR_CACHE_DIR)\")
>
> define HOST_CCACHE_PATCH_CONFIGURATION
> - sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/ccache.c
> - sed -i 's,"%s/.ccache","$(HOST_CCACHE_DEFAULT_CCACHE_DIR)",' $(@D)/src/conf.c
> + sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/Config.cpp
> + sed -i 's,home_dir + "/.ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
> + sed -i 's,getenv("XDG_CACHE_HOME"),nullptr,' $(@D)/src/Config.cpp
> + sed -i 's,home_dir + "/.cache/ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
> + sed -i 's,getenv("XDG_CONFIG_HOME"),nullptr,' $(@D)/src/Config.cpp
> + sed -i 's,home_dir + "/.config/ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
> endef
>
> HOST_CCACHE_POST_PATCH_HOOKS += HOST_CCACHE_PATCH_CONFIGURATION
> @@ -62,7 +67,7 @@ endef
> HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_DO_INITIAL_SETUP
> endif
>
> -$(eval $(host-autotools-package))
> +$(eval $(host-cmake-package))
>
> ifeq ($(BR2_CCACHE),y)
> ccache-stats: host-ccache
> diff --git a/package/cmake/cmake.mk b/package/cmake/cmake.mk
> index 053658fad6..eae12c1465 100644
> --- a/package/cmake/cmake.mk
> +++ b/package/cmake/cmake.mk
> @@ -40,9 +40,16 @@ CMAKE_CONF_OPTS = \
> HOST_CMAKE_CFLAGS = $(shell echo $(HOST_CFLAGS) | sed -r "s%$(HOST_CPPFLAGS)%%")
> HOST_CMAKE_CXXFLAGS = $(shell echo $(HOST_CXXFLAGS) | sed -r "s%$(HOST_CPPFLAGS)%%")
>
> +# we are built before ccache
> +HOST_CMAKE_CONFIGURE_OPTS = \
> + $(HOST_CONFIGURE_OPTS) \
> + CC="$(HOSTCC_NOCCACHE)" \
> + GCC="$(HOSTCC_NOCCACHE)" \
> + CXX="$(HOSTCXX_NOCCACHE)"
> +
> define HOST_CMAKE_CONFIGURE_CMDS
> (cd $(@D); \
> - $(HOST_CONFIGURE_OPTS) \
> + $(HOST_CMAKE_CONFIGURE_OPTS) \
> CFLAGS="$(HOST_CMAKE_CFLAGS)" \
> ./bootstrap --prefix=$(HOST_DIR) \
> --parallel=$(PARALLEL_JOBS) -- \
> diff --git a/package/hiredis/hiredis.mk b/package/hiredis/hiredis.mk
> index a571951fa3..2cb9991029 100644
> --- a/package/hiredis/hiredis.mk
> +++ b/package/hiredis/hiredis.mk
> @@ -20,4 +20,10 @@ else
> HIREDIS_CONF_OPTS += -DENABLE_SSL=OFF
> endif
>
> +# we are built before ccache
> +HOST_HIREDIS_CONF_OPTS += \
> + -UCMAKE_C_COMPILER_LAUNCHER \
> + -UCMAKE_CXX_COMPILER_LAUNCHER
> +
> $(eval $(cmake-package))
> +$(eval $(host-cmake-package))
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index b233b07548..cba94052a0 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -792,7 +792,7 @@ $(2)_EXTRACT_DEPENDENCIES += \
> endif
>
> ifeq ($$(BR2_CCACHE),y)
> -ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache,$(1)),)
> +ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache host-cmake host-hiredis host-pkgconf host-zstd,$(1)),)
> $(2)_DEPENDENCIES += host-ccache
> endif
> endif
> diff --git a/package/pkgconf/pkgconf.mk b/package/pkgconf/pkgconf.mk
> index 5d65f69c10..6b187b0238 100644
> --- a/package/pkgconf/pkgconf.mk
> +++ b/package/pkgconf/pkgconf.mk
> @@ -32,6 +32,11 @@ define HOST_PKGCONF_SHARED
> $(SED) 's,@STATIC@,,' $(HOST_DIR)/bin/pkg-config
> endef
>
> +# we are built before ccache
> +HOST_PKGCONF_CONF_ENV = \
> + CC="$(HOSTCC_NOCCACHE)" \
> + CXX="$(HOSTCXX_NOCCACHE)"
> +
> PKGCONF_POST_INSTALL_TARGET_HOOKS += PKGCONF_LINK_PKGCONFIG
> HOST_PKGCONF_POST_INSTALL_HOOKS += HOST_PKGCONF_INSTALL_WRAPPER
>
> diff --git a/package/zstd/zstd.mk b/package/zstd/zstd.mk
> index e8f6315222..edc4f1c624 100644
> --- a/package/zstd/zstd.mk
> +++ b/package/zstd/zstd.mk
> @@ -97,6 +97,11 @@ endef
>
> HOST_ZSTD_OPTS += PREFIX=$(HOST_DIR)
>
> +# we are built before ccache
> +HOST_ZSTD_OPTS += \
> + CC="$(HOSTCC_NOCCACHE)" \
> + CXX="$(HOSTCXX_NOCCACHE)"
> +
> define HOST_ZSTD_BUILD_CMDS
> $(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) $(HOST_ZSTD_OPTS) \
> -C $(@D) zstd-release lib-release
> --
> 2.37.1
>
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2023-01-08 22:38 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-08-04 13:29 [Buildroot] [PATCH 1/1] package/ccache: bump version to 4.6.1 Dominik Michael Rauh
2022-08-05 10:24 ` [Buildroot] [PATCH v2] " Dominik Michael Rauh
2022-08-05 13:13 ` [Buildroot] [PATCH v3] " Dominik Michael Rauh
2022-08-05 14:34 ` Thomas Petazzoni via buildroot
2022-08-07 9:13 ` Dominik Michael Rauh
2022-08-07 12:39 ` Thomas Petazzoni via buildroot
2022-08-07 17:55 ` Dominik Michael Rauh
2022-08-07 9:38 ` [Buildroot] [PATCH v4] " Dominik Michael Rauh
2023-01-08 22:37 ` Yann E. MORIN
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.