* [RFC PATCH v1 0/4] Fix `cargo package` for libgit-sys
@ 2025-03-18 23:24 Josh Steadmon
2025-03-18 23:24 ` [RFC PATCH v1 1/4] libgitpub: move to separate contrib/ directory Josh Steadmon
` (4 more replies)
0 siblings, 5 replies; 21+ messages in thread
From: Josh Steadmon @ 2025-03-18 23:24 UTC (permalink / raw)
To: git; +Cc: brian m. carlson
Fix a variety of issues with `cargo package` for the libgit-sys crate.
Most of these are straightforward, but patch #2 carries an unfortunate
side-effect that building the libgit-sys crate requires cleaning the Git
worktree in the process. Please see the patch description there for
discussion of possible alternatives, and please suggest any other ideas
you might have for that issue.
`cargo package` for the libgit-rs crate is still broken for now, because
it will require publishing the the libgit-sys crate first.
Josh Steadmon (4):
libgitpub: move to separate contrib/ directory
libgit-sys: add symlink to git repo root and clean after build
libgit-sys: parallelize build with Cargo's jobserver
libgit-sys: exclude unnecessary directories in git-src
Makefile | 36 ++++++++++++++-----
contrib/libgit-sys/Cargo.toml | 12 +++++++
contrib/libgit-sys/build.rs | 34 ++++++++++++++++--
contrib/libgit-sys/git-src | 1 +
.../public_symbol_export.c | 2 +-
.../public_symbol_export.h | 0
6 files changed, 72 insertions(+), 13 deletions(-)
create mode 120000 contrib/libgit-sys/git-src
rename contrib/{libgit-sys => libgitpub}/public_symbol_export.c (96%)
rename contrib/{libgit-sys => libgitpub}/public_symbol_export.h (100%)
base-commit: 683c54c999c301c2cd6f715c411407c413b1d84e
--
2.49.0.rc1.451.g8f38331e32-goog
^ permalink raw reply [flat|nested] 21+ messages in thread
* [RFC PATCH v1 1/4] libgitpub: move to separate contrib/ directory
2025-03-18 23:24 [RFC PATCH v1 0/4] Fix `cargo package` for libgit-sys Josh Steadmon
@ 2025-03-18 23:24 ` Josh Steadmon
2025-03-18 23:24 ` [RFC PATCH v1 2/4] libgit-sys: add symlink to git repo root and clean after build Josh Steadmon
` (3 subsequent siblings)
4 siblings, 0 replies; 21+ messages in thread
From: Josh Steadmon @ 2025-03-18 23:24 UTC (permalink / raw)
To: git; +Cc: brian m. carlson
For Cargo packaging purposes, it is better for any dependencies built with Git's
Makefile to live outside the libgit-{sys,rs} crates. This, plus some future
changes to the crates, will allow `cargo package` to treat all of the non-Rust
source as a separate tree apart from the Cargo-managed inputs.
Signed-off-by: Josh Steadmon <steadmon@google.com>
---
Makefile | 16 ++++++++--------
contrib/libgit-sys/build.rs | 4 ++--
.../public_symbol_export.c | 2 +-
.../public_symbol_export.h | 0
4 files changed, 11 insertions(+), 11 deletions(-)
rename contrib/{libgit-sys => libgitpub}/public_symbol_export.c (96%)
rename contrib/{libgit-sys => libgitpub}/public_symbol_export.h (100%)
diff --git a/Makefile b/Makefile
index 7315507381..52eed88dde 100644
--- a/Makefile
+++ b/Makefile
@@ -2764,7 +2764,7 @@ OBJECTS += $(CLAR_TEST_OBJS)
OBJECTS += $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS))
ifdef INCLUDE_LIBGIT_RS
- OBJECTS += contrib/libgit-sys/public_symbol_export.o
+ OBJECTS += contrib/libgitpub/public_symbol_export.o
endif
ifndef NO_CURL
@@ -3762,9 +3762,9 @@ clean: profile-clean coverage-clean cocciclean
$(MAKE) -C Documentation/ clean
$(RM) Documentation/GIT-EXCLUDED-PROGRAMS
$(RM) -r contrib/libgit-sys/target contrib/libgit-rs/target
- $(RM) contrib/libgit-sys/partial_symbol_export.o
- $(RM) contrib/libgit-sys/hidden_symbol_export.o
- $(RM) contrib/libgit-sys/libgitpub.a
+ $(RM) contrib/libgitpub/partial_symbol_export.o
+ $(RM) contrib/libgitpub/hidden_symbol_export.o
+ $(RM) contrib/libgitpub/libgitpub.a
ifndef NO_PERL
$(RM) -r perl/build/
endif
@@ -3937,14 +3937,14 @@ ifdef INCLUDE_LIBGIT_RS
all:: libgit-sys libgit-rs
endif
-LIBGIT_PUB_OBJS += contrib/libgit-sys/public_symbol_export.o
+LIBGIT_PUB_OBJS += contrib/libgitpub/public_symbol_export.o
LIBGIT_PUB_OBJS += libgit.a
LIBGIT_PUB_OBJS += reftable/libreftable.a
LIBGIT_PUB_OBJS += xdiff/lib.a
-LIBGIT_PARTIAL_EXPORT = contrib/libgit-sys/partial_symbol_export.o
+LIBGIT_PARTIAL_EXPORT = contrib/libgitpub/partial_symbol_export.o
-LIBGIT_HIDDEN_EXPORT = contrib/libgit-sys/hidden_symbol_export.o
+LIBGIT_HIDDEN_EXPORT = contrib/libgitpub/hidden_symbol_export.o
$(LIBGIT_PARTIAL_EXPORT): $(LIBGIT_PUB_OBJS)
$(LD) -r $^ -o $@
@@ -3952,5 +3952,5 @@ $(LIBGIT_PARTIAL_EXPORT): $(LIBGIT_PUB_OBJS)
$(LIBGIT_HIDDEN_EXPORT): $(LIBGIT_PARTIAL_EXPORT)
$(OBJCOPY) --localize-hidden $^ $@
-contrib/libgit-sys/libgitpub.a: $(LIBGIT_HIDDEN_EXPORT)
+contrib/libgitpub/libgitpub.a: $(LIBGIT_HIDDEN_EXPORT)
$(AR) $(ARFLAGS) $@ $^
diff --git a/contrib/libgit-sys/build.rs b/contrib/libgit-sys/build.rs
index 3ffd80ad91..e0d979c196 100644
--- a/contrib/libgit-sys/build.rs
+++ b/contrib/libgit-sys/build.rs
@@ -15,7 +15,7 @@ pub fn main() -> std::io::Result<()> {
.current_dir(git_root.clone())
.args([
"INCLUDE_LIBGIT_RS=YesPlease",
- "contrib/libgit-sys/libgitpub.a",
+ "contrib/libgitpub/libgitpub.a",
])
.output()
.expect("Make failed to run");
@@ -26,7 +26,7 @@ pub fn main() -> std::io::Result<()> {
String::from_utf8(make_output.stderr).unwrap()
);
}
- std::fs::copy(crate_root.join("libgitpub.a"), dst.join("libgitpub.a"))?;
+ std::fs::copy(git_root.join("contrib/libgitpub/libgitpub.a"), dst.join("libgitpub.a"))?;
println!("cargo:rustc-link-search=native={}", dst.display());
println!("cargo:rustc-link-lib=gitpub");
println!("cargo:rerun-if-changed={}", git_root.display());
diff --git a/contrib/libgit-sys/public_symbol_export.c b/contrib/libgitpub/public_symbol_export.c
similarity index 96%
rename from contrib/libgit-sys/public_symbol_export.c
rename to contrib/libgitpub/public_symbol_export.c
index dfbb257115..de5f6943c2 100644
--- a/contrib/libgit-sys/public_symbol_export.c
+++ b/contrib/libgitpub/public_symbol_export.c
@@ -6,7 +6,7 @@
#include "git-compat-util.h"
#include "config.h"
-#include "contrib/libgit-sys/public_symbol_export.h"
+#include "contrib/libgitpub/public_symbol_export.h"
#include "version.h"
#pragma GCC visibility push(default)
diff --git a/contrib/libgit-sys/public_symbol_export.h b/contrib/libgitpub/public_symbol_export.h
similarity index 100%
rename from contrib/libgit-sys/public_symbol_export.h
rename to contrib/libgitpub/public_symbol_export.h
--
2.49.0.rc1.451.g8f38331e32-goog
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [RFC PATCH v1 2/4] libgit-sys: add symlink to git repo root and clean after build
2025-03-18 23:24 [RFC PATCH v1 0/4] Fix `cargo package` for libgit-sys Josh Steadmon
2025-03-18 23:24 ` [RFC PATCH v1 1/4] libgitpub: move to separate contrib/ directory Josh Steadmon
@ 2025-03-18 23:24 ` Josh Steadmon
2025-03-19 22:23 ` Josh Steadmon
2025-03-18 23:24 ` [RFC PATCH v1 3/4] libgit-sys: parallelize build with Cargo's jobserver Josh Steadmon
` (2 subsequent siblings)
4 siblings, 1 reply; 21+ messages in thread
From: Josh Steadmon @ 2025-03-18 23:24 UTC (permalink / raw)
To: git; +Cc: brian m. carlson
Unlike `cargo build`, `cargo package` does not get access to the entire Git repo
containing a Rust crate. Instead, it prepares a directory starting from the
crate root (potentially excluding files, such as those not under version
control, or explicity excluded in the Cargo.toml file).
This means that the current method of building the libgit-sys crate does not
work with `cargo package`, as it tries to execute the Makefile from "../.."
relative to the crate root.
Fix this by adding a `git-src` symlink in the crate that points to the Git
repository root. `cargo package` will flatten this to a copy of the Git repo,
excluding non-version-controlled files, any explicitly-excluded files, and trees
that contain a Cargo.toml file (this prevents infinite recursion on the
symlink).
We can then execute the Makefile under the flattened git-src directory from our
build.rs script. However, this exposes a second problem; Cargo will check that
the build script does not add, delete, or modify any source files. This means
that after we copy our libgitpub.a dependency to the output directory, we must
run `make clean` to remove the object files we created during the build process.
Unfortunately, there is not a way to determine from the build.rs script whether
we're running `cargo build` vs. `cargo package`, so now any build of the
libgit-sys crate will result in cleaning the Git worktree.
A potential alternative is to make an additional temporary copy of the worktree
and run the Makefile there. This would avoid removing build artifacts in the
worktree at the cost of copying MBs worth of source files to a temporary
directory. Perhaps hardlinking instead of making a full copy would help here,
but that might be less portable.
Signed-off-by: Josh Steadmon <steadmon@google.com>
---
Makefile | 6 ++++++
contrib/libgit-sys/build.rs | 21 ++++++++++++++++++++-
contrib/libgit-sys/git-src | 1 +
3 files changed, 27 insertions(+), 1 deletion(-)
create mode 120000 contrib/libgit-sys/git-src
diff --git a/Makefile b/Makefile
index 52eed88dde..e7d8786e4e 100644
--- a/Makefile
+++ b/Makefile
@@ -420,6 +420,10 @@ include shared.mak
# Define INCLUDE_LIBGIT_RS if you want `make all` and `make test` to build and
# test the Rust crates in contrib/libgit-sys and contrib/libgit-rs.
#
+# Define PRESERVE_LIBGIT_TARGET to prevent `make clean` from removing the Cargo
+# output directories for libgit-sys and libgit-rs. This is mainly for use in the
+# Cargo build scripts.
+#
# === Optional library: libintl ===
#
# Define NO_GETTEXT if you don't want Git output to be translated.
@@ -3761,7 +3765,9 @@ clean: profile-clean coverage-clean cocciclean
$(RM) $(htmldocs).tar.gz $(manpages).tar.gz
$(MAKE) -C Documentation/ clean
$(RM) Documentation/GIT-EXCLUDED-PROGRAMS
+ifndef PRESERVE_LIBGIT_TARGET
$(RM) -r contrib/libgit-sys/target contrib/libgit-rs/target
+endif
$(RM) contrib/libgitpub/partial_symbol_export.o
$(RM) contrib/libgitpub/hidden_symbol_export.o
$(RM) contrib/libgitpub/libgitpub.a
diff --git a/contrib/libgit-sys/build.rs b/contrib/libgit-sys/build.rs
index e0d979c196..9d586d272d 100644
--- a/contrib/libgit-sys/build.rs
+++ b/contrib/libgit-sys/build.rs
@@ -6,7 +6,7 @@ pub fn main() -> std::io::Result<()> {
ac.emit_has_path("std::ffi::c_char");
let crate_root = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap());
- let git_root = crate_root.join("../..");
+ let git_root = crate_root.join("git-src");
let dst = PathBuf::from(env::var_os("OUT_DIR").unwrap());
let make_output = make_cmd::gnu_make()
@@ -31,5 +31,24 @@ pub fn main() -> std::io::Result<()> {
println!("cargo:rustc-link-lib=gitpub");
println!("cargo:rerun-if-changed={}", git_root.display());
+ let make_output = make_cmd::gnu_make()
+ .env("DEVELOPER", "1")
+ .env_remove("PROFILE")
+ .current_dir(git_root.clone())
+ .args([
+ "INCLUDE_LIBGIT_RS=YesPlease",
+ "PRESERVE_LIBGIT_TARGET=YesPlease",
+ "clean",
+ ])
+ .output()
+ .expect("`make clean` failed to run");
+ if !make_output.status.success() {
+ panic!(
+ "`make clean` failed:\n stdout = {}\n stderr = {}\n",
+ String::from_utf8(make_output.stdout).unwrap(),
+ String::from_utf8(make_output.stderr).unwrap()
+ );
+ }
+
Ok(())
}
diff --git a/contrib/libgit-sys/git-src b/contrib/libgit-sys/git-src
new file mode 120000
index 0000000000..c25bddb6dd
--- /dev/null
+++ b/contrib/libgit-sys/git-src
@@ -0,0 +1 @@
+../..
\ No newline at end of file
--
2.49.0.rc1.451.g8f38331e32-goog
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [RFC PATCH v1 3/4] libgit-sys: parallelize build with Cargo's jobserver
2025-03-18 23:24 [RFC PATCH v1 0/4] Fix `cargo package` for libgit-sys Josh Steadmon
2025-03-18 23:24 ` [RFC PATCH v1 1/4] libgitpub: move to separate contrib/ directory Josh Steadmon
2025-03-18 23:24 ` [RFC PATCH v1 2/4] libgit-sys: add symlink to git repo root and clean after build Josh Steadmon
@ 2025-03-18 23:24 ` Josh Steadmon
2025-03-18 23:24 ` [RFC PATCH v1 4/4] libgit-sys: exclude unnecessary directories in git-src Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 0/5] Fix `cargo package` for libgit-sys Josh Steadmon
4 siblings, 0 replies; 21+ messages in thread
From: Josh Steadmon @ 2025-03-18 23:24 UTC (permalink / raw)
To: git; +Cc: brian m. carlson
Cargo provides GNU Make flags that can be used to connect to its jobserver for
managing parallel builds. They are not automatically passed to any `make`
invocations, so let's add that to our build.rs script.
Signed-off-by: Josh Steadmon <steadmon@google.com>
---
contrib/libgit-sys/build.rs | 3 +++
1 file changed, 3 insertions(+)
diff --git a/contrib/libgit-sys/build.rs b/contrib/libgit-sys/build.rs
index 9d586d272d..16e0d66afb 100644
--- a/contrib/libgit-sys/build.rs
+++ b/contrib/libgit-sys/build.rs
@@ -8,9 +8,11 @@ pub fn main() -> std::io::Result<()> {
let crate_root = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap());
let git_root = crate_root.join("git-src");
let dst = PathBuf::from(env::var_os("OUT_DIR").unwrap());
+ let makeflags = env::var_os("CARGO_MAKEFLAGS").unwrap();
let make_output = make_cmd::gnu_make()
.env("DEVELOPER", "1")
+ .env("MAKEFLAGS", &makeflags)
.env_remove("PROFILE")
.current_dir(git_root.clone())
.args([
@@ -33,6 +35,7 @@ pub fn main() -> std::io::Result<()> {
let make_output = make_cmd::gnu_make()
.env("DEVELOPER", "1")
+ .env("MAKEFLAGS", &makeflags)
.env_remove("PROFILE")
.current_dir(git_root.clone())
.args([
--
2.49.0.rc1.451.g8f38331e32-goog
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [RFC PATCH v1 4/4] libgit-sys: exclude unnecessary directories in git-src
2025-03-18 23:24 [RFC PATCH v1 0/4] Fix `cargo package` for libgit-sys Josh Steadmon
` (2 preceding siblings ...)
2025-03-18 23:24 ` [RFC PATCH v1 3/4] libgit-sys: parallelize build with Cargo's jobserver Josh Steadmon
@ 2025-03-18 23:24 ` Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 0/5] Fix `cargo package` for libgit-sys Josh Steadmon
4 siblings, 0 replies; 21+ messages in thread
From: Josh Steadmon @ 2025-03-18 23:24 UTC (permalink / raw)
To: git; +Cc: brian m. carlson
We can avoid copying tens of megabytes of unnecessary source files by excluding
a few directories which are not needed to compile libgitpub.a. This helps us
stay below crates.io's 10 MB size limit.
Signed-off-by: Josh Steadmon <steadmon@google.com>
---
Makefile | 14 +++++++++++++-
contrib/libgit-sys/Cargo.toml | 12 ++++++++++++
contrib/libgit-sys/build.rs | 8 +++++++-
3 files changed, 32 insertions(+), 2 deletions(-)
diff --git a/Makefile b/Makefile
index e7d8786e4e..c75f1d0208 100644
--- a/Makefile
+++ b/Makefile
@@ -651,6 +651,7 @@ export prefix bindir sharedir sysconfdir perllibdir localedir
CC = cc
AR = ar
RM = rm -f
+RMDIR = rmdir --ignore-fail-on-non-empty
DIFF = diff
TAR = tar
FIND = find
@@ -3477,13 +3478,13 @@ coccicheck-pending: $(COCCICHECK_PATCHES_PENDING_INTREE)
# "Sub"-Makefiles, not really because they can't be run stand-alone,
# only there to contain directory-specific rules and variables
+ifndef NO_GITWEB
## gitweb/Makefile inclusion:
MAK_DIR_GITWEB = gitweb/
include gitweb/Makefile
.PHONY: gitweb
gitweb: $(MAK_DIR_GITWEB_ALL)
-ifndef NO_GITWEB
all:: gitweb
endif
@@ -3763,7 +3764,9 @@ clean: profile-clean coverage-clean cocciclean
$(RM) -r .dist-tmp-dir .doc-tmp-dir
$(RM) $(GIT_TARNAME).tar.gz
$(RM) $(htmldocs).tar.gz $(manpages).tar.gz
+ifndef INCLUDE_LIBGIT_RS
$(MAKE) -C Documentation/ clean
+endif
$(RM) Documentation/GIT-EXCLUDED-PROGRAMS
ifndef PRESERVE_LIBGIT_TARGET
$(RM) -r contrib/libgit-sys/target contrib/libgit-rs/target
@@ -3775,7 +3778,9 @@ ifndef NO_PERL
$(RM) -r perl/build/
endif
$(MAKE) -C templates/ clean
+ifndef INCLUDE_LIBGIT_RS
$(MAKE) -C t/ clean
+endif
ifndef NO_TCLTK
$(MAKE) -C gitk-git clean
$(MAKE) -C git-gui clean
@@ -3798,6 +3803,13 @@ ifdef MSVC
$(RM) compat/vcbuild/MSVC-DEFS-GEN
endif
+# Handle additional cleanup needed for running `cargo package` for libgit-sys.
+ifdef INCLUDE_LIBGIT_RS
+libgit-pkg-clean: clean
+ $(RMDIR) t/helper t/unit-tests/clar t/unit-tests t
+ $(RMDIR) oss-fuzz
+endif
+
.PHONY: all install profile-clean cocciclean clean strip
.PHONY: shell_compatibility_test please_set_SHELL_PATH_to_a_more_modern_shell
.PHONY: FORCE
diff --git a/contrib/libgit-sys/Cargo.toml b/contrib/libgit-sys/Cargo.toml
index e0623022c3..4b8facccfe 100644
--- a/contrib/libgit-sys/Cargo.toml
+++ b/contrib/libgit-sys/Cargo.toml
@@ -7,6 +7,18 @@ links = "gitpub"
rust-version = "1.63" # TODO: Once we hit 1.84 or newer, we may want to remove Cargo.lock from
# version control. See https://lore.kernel.org/git/Z47jgK-oMjFRSslr@tapette.crustytoothpaste.net/
description = "Native bindings to a portion of libgit"
+exclude = [
+ "git-src/.github",
+ "git-src/ci",
+ "git-src/Documentation",
+ "git-src/git-gui",
+ "git-src/gitk-git",
+ "git-src/gitweb",
+ "git-src/oss-fuzz",
+ "git-src/perl",
+ "git-src/po",
+ "git-src/t"
+]
[lib]
path = "src/lib.rs"
diff --git a/contrib/libgit-sys/build.rs b/contrib/libgit-sys/build.rs
index 16e0d66afb..0a72040726 100644
--- a/contrib/libgit-sys/build.rs
+++ b/contrib/libgit-sys/build.rs
@@ -17,6 +17,9 @@ pub fn main() -> std::io::Result<()> {
.current_dir(git_root.clone())
.args([
"INCLUDE_LIBGIT_RS=YesPlease",
+ "NO_GITWEB=YesPlease",
+ "NO_PERL=YesPlease",
+ "NO_TCLTK=YesPlease",
"contrib/libgitpub/libgitpub.a",
])
.output()
@@ -40,8 +43,11 @@ pub fn main() -> std::io::Result<()> {
.current_dir(git_root.clone())
.args([
"INCLUDE_LIBGIT_RS=YesPlease",
+ "NO_GITWEB=YesPlease",
+ "NO_PERL=YesPlease",
+ "NO_TCLTK=YesPlease",
"PRESERVE_LIBGIT_TARGET=YesPlease",
- "clean",
+ "libgit-pkg-clean",
])
.output()
.expect("`make clean` failed to run");
--
2.49.0.rc1.451.g8f38331e32-goog
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [RFC PATCH v1 2/4] libgit-sys: add symlink to git repo root and clean after build
2025-03-18 23:24 ` [RFC PATCH v1 2/4] libgit-sys: add symlink to git repo root and clean after build Josh Steadmon
@ 2025-03-19 22:23 ` Josh Steadmon
2025-03-20 11:10 ` Phillip Wood
0 siblings, 1 reply; 21+ messages in thread
From: Josh Steadmon @ 2025-03-19 22:23 UTC (permalink / raw)
To: git; +Cc: brian m. carlson
On 2025.03.18 16:24, Josh Steadmon wrote:
> Unlike `cargo build`, `cargo package` does not get access to the entire Git repo
> containing a Rust crate. Instead, it prepares a directory starting from the
> crate root (potentially excluding files, such as those not under version
> control, or explicity excluded in the Cargo.toml file).
>
> This means that the current method of building the libgit-sys crate does not
> work with `cargo package`, as it tries to execute the Makefile from "../.."
> relative to the crate root.
>
> Fix this by adding a `git-src` symlink in the crate that points to the Git
> repository root. `cargo package` will flatten this to a copy of the Git repo,
> excluding non-version-controlled files, any explicitly-excluded files, and trees
> that contain a Cargo.toml file (this prevents infinite recursion on the
> symlink).
>
> We can then execute the Makefile under the flattened git-src directory from our
> build.rs script. However, this exposes a second problem; Cargo will check that
> the build script does not add, delete, or modify any source files. This means
> that after we copy our libgitpub.a dependency to the output directory, we must
> run `make clean` to remove the object files we created during the build process.
>
> Unfortunately, there is not a way to determine from the build.rs script whether
> we're running `cargo build` vs. `cargo package`, so now any build of the
> libgit-sys crate will result in cleaning the Git worktree.
>
> A potential alternative is to make an additional temporary copy of the worktree
> and run the Makefile there. This would avoid removing build artifacts in the
> worktree at the cost of copying MBs worth of source files to a temporary
> directory. Perhaps hardlinking instead of making a full copy would help here,
> but that might be less portable.
I'm currently working on an alternate solution where we build the object
files in Cargo's working directory, rather than in the source tree. This
will (mostly) avoid the need to clean after the build. I hope to send a
v2 with this change in the next few days.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC PATCH v1 2/4] libgit-sys: add symlink to git repo root and clean after build
2025-03-19 22:23 ` Josh Steadmon
@ 2025-03-20 11:10 ` Phillip Wood
2025-03-21 19:49 ` Josh Steadmon
0 siblings, 1 reply; 21+ messages in thread
From: Phillip Wood @ 2025-03-20 11:10 UTC (permalink / raw)
To: Josh Steadmon, git, brian m. carlson
Hi Josh
On 19/03/2025 22:23, Josh Steadmon wrote:
> On 2025.03.18 16:24, Josh Steadmon wrote:
>
> I'm currently working on an alternate solution where we build the object
> files in Cargo's working directory, rather than in the source tree. This
> will (mostly) avoid the need to clean after the build. I hope to send a
> v2 with this change in the next few days.
Meson has builtin support for out-of-tree builds which would make
building in Cargo's OUT_DIR trivial. Our meson build is still
experimental though so you may not want to rely on it.
Best Wishes
Phillip
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC PATCH v1 2/4] libgit-sys: add symlink to git repo root and clean after build
2025-03-20 11:10 ` Phillip Wood
@ 2025-03-21 19:49 ` Josh Steadmon
0 siblings, 0 replies; 21+ messages in thread
From: Josh Steadmon @ 2025-03-21 19:49 UTC (permalink / raw)
To: phillip.wood; +Cc: git, brian m. carlson
On 2025.03.20 11:10, Phillip Wood wrote:
> Hi Josh
>
> On 19/03/2025 22:23, Josh Steadmon wrote:
> > On 2025.03.18 16:24, Josh Steadmon wrote:
> >
> > I'm currently working on an alternate solution where we build the object
> > files in Cargo's working directory, rather than in the source tree. This
> > will (mostly) avoid the need to clean after the build. I hope to send a
> > v2 with this change in the next few days.
>
> Meson has builtin support for out-of-tree builds which would make building
> in Cargo's OUT_DIR trivial. Our meson build is still experimental though so
> you may not want to rely on it.
>
> Best Wishes
>
> Phillip
Thanks for the pointer! I have a working Makefile solution which I'll be
sending out soon, but I'll keep meson in mind as a backup if the list
doesn't like my V2.
^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCH v2 0/5] Fix `cargo package` for libgit-sys
2025-03-18 23:24 [RFC PATCH v1 0/4] Fix `cargo package` for libgit-sys Josh Steadmon
` (3 preceding siblings ...)
2025-03-18 23:24 ` [RFC PATCH v1 4/4] libgit-sys: exclude unnecessary directories in git-src Josh Steadmon
@ 2025-03-21 22:14 ` Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 1/5] libgitpub: move to separate contrib/ directory Josh Steadmon
` (4 more replies)
4 siblings, 5 replies; 21+ messages in thread
From: Josh Steadmon @ 2025-03-21 22:14 UTC (permalink / raw)
To: git; +Cc: brian m. carlson, Phillip Wood
Fix `cargo package` for the libgit-sys crate by providing a pristine
copy of the Git source tree at package time, and by building object
files and other generated files outside of this tree. While we're at it,
improve parallel builds by plumbing Cargo's make flags to the make
command invoked in our build.rs script.
We also fix a few problems that would prevent us from publishing the
libgit-sys crate: we add a license and description to Cargo.toml, and we
exclude unnecessary files from the packaged source, to stay below
crates.io's 10 MB limit.
`cargo package` for the libgit-rs crate is still broken for now, because
it will require publishing the libgit-sys crate first.
Changes from V1:
* Reworked patch #2 to build outside of the source tree rather than
running `make clean` after the build.
* Simplified patch #4 now that cleaning is no longer necessary.
* Added patch #5 to add some required Cargo.toml fields.
Josh Steadmon (5):
libgitpub: move to separate contrib/ directory
libgit-sys: add symlink to git repo root and build out of tree
libgit-sys: parallelize build with Cargo's jobserver
libgit-sys: exclude unnecessary directories in git-src
libgit-{sys,rs}: add license and description fields
Makefile | 145 +++++++++++-------
contrib/libgit-rs/Cargo.toml | 2 +
contrib/libgit-sys/Cargo.toml | 13 ++
contrib/libgit-sys/build.rs | 12 +-
contrib/libgit-sys/git-src | 1 +
.../public_symbol_export.c | 2 +-
.../public_symbol_export.h | 0
shared.mak | 5 +
8 files changed, 117 insertions(+), 63 deletions(-)
create mode 120000 contrib/libgit-sys/git-src
rename contrib/{libgit-sys => libgitpub}/public_symbol_export.c (96%)
rename contrib/{libgit-sys => libgitpub}/public_symbol_export.h (100%)
Range-diff against v1:
1: 28e10e1092 = 1: 28e10e1092 libgitpub: move to separate contrib/ directory
2: 58111043a1 < -: ---------- libgit-sys: add symlink to git repo root and clean after build
-: ---------- > 2: 6befc95a2d libgit-sys: add symlink to git repo root and build out of tree
3: 4cf9996096 ! 3: a34e23a83d libgit-sys: parallelize build with Cargo's jobserver
@@ contrib/libgit-sys/build.rs: pub fn main() -> std::io::Result<()> {
let dst = PathBuf::from(env::var_os("OUT_DIR").unwrap());
+ let makeflags = env::var_os("CARGO_MAKEFLAGS").unwrap();
- let make_output = make_cmd::gnu_make()
- .env("DEVELOPER", "1")
-+ .env("MAKEFLAGS", &makeflags)
- .env_remove("PROFILE")
- .current_dir(git_root.clone())
- .args([
-@@ contrib/libgit-sys/build.rs: pub fn main() -> std::io::Result<()> {
-
let make_output = make_cmd::gnu_make()
.env("DEVELOPER", "1")
+ .env("MAKEFLAGS", &makeflags)
4: 9b5d4aa140 < -: ---------- libgit-sys: exclude unnecessary directories in git-src
-: ---------- > 4: 390695ac1f libgit-sys: exclude unnecessary directories in git-src
-: ---------- > 5: 3a87f54693 libgit-{sys,rs}: add license and description fields
base-commit: 683c54c999c301c2cd6f715c411407c413b1d84e
--
2.49.0.395.g12beb8f557-goog
^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCH v2 1/5] libgitpub: move to separate contrib/ directory
2025-03-21 22:14 ` [PATCH v2 0/5] Fix `cargo package` for libgit-sys Josh Steadmon
@ 2025-03-21 22:14 ` Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree Josh Steadmon
` (3 subsequent siblings)
4 siblings, 0 replies; 21+ messages in thread
From: Josh Steadmon @ 2025-03-21 22:14 UTC (permalink / raw)
To: git; +Cc: brian m. carlson, Phillip Wood
For Cargo packaging purposes, it is better for any dependencies built with Git's
Makefile to live outside the libgit-{sys,rs} crates. This, plus some future
changes to the crates, will allow `cargo package` to treat all of the non-Rust
source as a separate tree apart from the Cargo-managed inputs.
Signed-off-by: Josh Steadmon <steadmon@google.com>
---
Makefile | 16 ++++++++--------
contrib/libgit-sys/build.rs | 4 ++--
.../public_symbol_export.c | 2 +-
.../public_symbol_export.h | 0
4 files changed, 11 insertions(+), 11 deletions(-)
rename contrib/{libgit-sys => libgitpub}/public_symbol_export.c (96%)
rename contrib/{libgit-sys => libgitpub}/public_symbol_export.h (100%)
diff --git a/Makefile b/Makefile
index 7315507381..52eed88dde 100644
--- a/Makefile
+++ b/Makefile
@@ -2764,7 +2764,7 @@ OBJECTS += $(CLAR_TEST_OBJS)
OBJECTS += $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS))
ifdef INCLUDE_LIBGIT_RS
- OBJECTS += contrib/libgit-sys/public_symbol_export.o
+ OBJECTS += contrib/libgitpub/public_symbol_export.o
endif
ifndef NO_CURL
@@ -3762,9 +3762,9 @@ clean: profile-clean coverage-clean cocciclean
$(MAKE) -C Documentation/ clean
$(RM) Documentation/GIT-EXCLUDED-PROGRAMS
$(RM) -r contrib/libgit-sys/target contrib/libgit-rs/target
- $(RM) contrib/libgit-sys/partial_symbol_export.o
- $(RM) contrib/libgit-sys/hidden_symbol_export.o
- $(RM) contrib/libgit-sys/libgitpub.a
+ $(RM) contrib/libgitpub/partial_symbol_export.o
+ $(RM) contrib/libgitpub/hidden_symbol_export.o
+ $(RM) contrib/libgitpub/libgitpub.a
ifndef NO_PERL
$(RM) -r perl/build/
endif
@@ -3937,14 +3937,14 @@ ifdef INCLUDE_LIBGIT_RS
all:: libgit-sys libgit-rs
endif
-LIBGIT_PUB_OBJS += contrib/libgit-sys/public_symbol_export.o
+LIBGIT_PUB_OBJS += contrib/libgitpub/public_symbol_export.o
LIBGIT_PUB_OBJS += libgit.a
LIBGIT_PUB_OBJS += reftable/libreftable.a
LIBGIT_PUB_OBJS += xdiff/lib.a
-LIBGIT_PARTIAL_EXPORT = contrib/libgit-sys/partial_symbol_export.o
+LIBGIT_PARTIAL_EXPORT = contrib/libgitpub/partial_symbol_export.o
-LIBGIT_HIDDEN_EXPORT = contrib/libgit-sys/hidden_symbol_export.o
+LIBGIT_HIDDEN_EXPORT = contrib/libgitpub/hidden_symbol_export.o
$(LIBGIT_PARTIAL_EXPORT): $(LIBGIT_PUB_OBJS)
$(LD) -r $^ -o $@
@@ -3952,5 +3952,5 @@ $(LIBGIT_PARTIAL_EXPORT): $(LIBGIT_PUB_OBJS)
$(LIBGIT_HIDDEN_EXPORT): $(LIBGIT_PARTIAL_EXPORT)
$(OBJCOPY) --localize-hidden $^ $@
-contrib/libgit-sys/libgitpub.a: $(LIBGIT_HIDDEN_EXPORT)
+contrib/libgitpub/libgitpub.a: $(LIBGIT_HIDDEN_EXPORT)
$(AR) $(ARFLAGS) $@ $^
diff --git a/contrib/libgit-sys/build.rs b/contrib/libgit-sys/build.rs
index 3ffd80ad91..e0d979c196 100644
--- a/contrib/libgit-sys/build.rs
+++ b/contrib/libgit-sys/build.rs
@@ -15,7 +15,7 @@ pub fn main() -> std::io::Result<()> {
.current_dir(git_root.clone())
.args([
"INCLUDE_LIBGIT_RS=YesPlease",
- "contrib/libgit-sys/libgitpub.a",
+ "contrib/libgitpub/libgitpub.a",
])
.output()
.expect("Make failed to run");
@@ -26,7 +26,7 @@ pub fn main() -> std::io::Result<()> {
String::from_utf8(make_output.stderr).unwrap()
);
}
- std::fs::copy(crate_root.join("libgitpub.a"), dst.join("libgitpub.a"))?;
+ std::fs::copy(git_root.join("contrib/libgitpub/libgitpub.a"), dst.join("libgitpub.a"))?;
println!("cargo:rustc-link-search=native={}", dst.display());
println!("cargo:rustc-link-lib=gitpub");
println!("cargo:rerun-if-changed={}", git_root.display());
diff --git a/contrib/libgit-sys/public_symbol_export.c b/contrib/libgitpub/public_symbol_export.c
similarity index 96%
rename from contrib/libgit-sys/public_symbol_export.c
rename to contrib/libgitpub/public_symbol_export.c
index dfbb257115..de5f6943c2 100644
--- a/contrib/libgit-sys/public_symbol_export.c
+++ b/contrib/libgitpub/public_symbol_export.c
@@ -6,7 +6,7 @@
#include "git-compat-util.h"
#include "config.h"
-#include "contrib/libgit-sys/public_symbol_export.h"
+#include "contrib/libgitpub/public_symbol_export.h"
#include "version.h"
#pragma GCC visibility push(default)
diff --git a/contrib/libgit-sys/public_symbol_export.h b/contrib/libgitpub/public_symbol_export.h
similarity index 100%
rename from contrib/libgit-sys/public_symbol_export.h
rename to contrib/libgitpub/public_symbol_export.h
--
2.49.0.395.g12beb8f557-goog
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree
2025-03-21 22:14 ` [PATCH v2 0/5] Fix `cargo package` for libgit-sys Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 1/5] libgitpub: move to separate contrib/ directory Josh Steadmon
@ 2025-03-21 22:14 ` Josh Steadmon
2025-03-23 1:46 ` Eric Sunshine
2025-03-21 22:14 ` [PATCH v2 3/5] libgit-sys: parallelize build with Cargo's jobserver Josh Steadmon
` (2 subsequent siblings)
4 siblings, 1 reply; 21+ messages in thread
From: Josh Steadmon @ 2025-03-21 22:14 UTC (permalink / raw)
To: git; +Cc: brian m. carlson, Phillip Wood
Unlike `cargo build`, `cargo package` does not get access to the entire Git repo
containing a Rust crate. Instead, it prepares a directory starting from the
crate root (potentially excluding files, such as those not under version
control, or explicity excluded in the Cargo.toml file).
This means that the current method of building the libgit-sys crate does not
work with `cargo package`, as it tries to execute the Makefile from "../.."
relative to the crate root.
Fix this by adding a `git-src` symlink in the crate that points to the Git
repository root. `cargo package` will flatten this to a copy of the Git repo,
excluding non-version-controlled files, any explicitly-excluded files, and trees
that contain a Cargo.toml file (this prevents infinite recursion on the
symlink).
We can then execute the Makefile under the flattened git-src directory from our
build.rs script. However, this exposes a second problem; Cargo will check that
the build script does not add, delete, or modify any files in the source tree.
Without further changes, Cargo complains about the object files and other
generated files created during the build.
To avoid this problem, add a CARGO_OUT_DIR variable to the Makefile. When this
is set, object files and other generated files will be created there, rather
than in the main source tree. This change has only been applied to files created
as part of the libgitpub.a build, to avoid unnecessary churn in the Makefile.
Signed-off-by: Josh Steadmon <steadmon@google.com>
---
Makefile | 135 +++++++++++++++++++++---------------
contrib/libgit-sys/build.rs | 7 +-
contrib/libgit-sys/git-src | 1 +
shared.mak | 5 ++
4 files changed, 91 insertions(+), 57 deletions(-)
create mode 120000 contrib/libgit-sys/git-src
diff --git a/Makefile b/Makefile
index 52eed88dde..91677448ba 100644
--- a/Makefile
+++ b/Makefile
@@ -420,6 +420,9 @@ include shared.mak
# Define INCLUDE_LIBGIT_RS if you want `make all` and `make test` to build and
# test the Rust crates in contrib/libgit-sys and contrib/libgit-rs.
#
+# Define CARGO_OUT_DIR to specify a directory where object files and other files
+# generated during the build of libgitpub.a should be created.
+#
# === Optional library: libintl ===
#
# Define NO_GETTEXT if you don't want Git output to be translated.
@@ -699,6 +702,7 @@ THIRD_PARTY_SOURCES =
UNIT_TEST_PROGRAMS =
UNIT_TEST_DIR = t/unit-tests
UNIT_TEST_BIN = $(UNIT_TEST_DIR)/bin
+CARGO_OUT_DIR =
# Having this variable in your environment would break pipelines because
# you cause "cd" to echo its destination to stdout. It can also take
@@ -917,11 +921,21 @@ export PYTHON_PATH
TEST_SHELL_PATH = $(SHELL_PATH)
-LIB_FILE = libgit.a
-XDIFF_LIB = xdiff/lib.a
-REFTABLE_LIB = reftable/libreftable.a
+### Generated files which may need to live in Cargo output directory
+
+GIT_CFLAGS = $(call maybe_join_path,$(CARGO_OUT_DIR),GIT-CFLAGS)
+GIT_PREFIX = $(call maybe_join_path,$(CARGO_OUT_DIR),GIT-PREFIX)
+GIT_USER_AGENT_FILE = $(call maybe_join_path,$(CARGO_OUT_DIR),GIT-USER-AGENT)
+GIT_VERSION_FILE = $(call maybe_join_path,$(CARGO_OUT_DIR),GIT-VERSION-FILE)
+GIT_VERSION_PATH = $(dir $(GIT_VERSION_FILE))$(notdir $(GIT_VERSION_FILE))
+COMMAND_LIST_H = $(call maybe_join_path,$(CARGO_OUT_DIR),command-list.h)
+VERSION_DEF_H = $(call maybe_join_path,$(CARGO_OUT_DIR),version-def.h)
-GENERATED_H += command-list.h
+LIB_FILE = $(call maybe_join_path,$(CARGO_OUT_DIR),libgit.a)
+XDIFF_LIB = $(call maybe_join_path,$(CARGO_OUT_DIR),xdiff/lib.a)
+REFTABLE_LIB = $(call maybe_join_path,$(CARGO_OUT_DIR),reftable/libreftable.a)
+
+GENERATED_H += $(COMMAND_LIST_H)
GENERATED_H += config-list.h
GENERATED_H += hook-list.h
GENERATED_H += $(UNIT_TEST_DIR)/clar-decls.h
@@ -1472,7 +1486,7 @@ ifdef DEVELOPER
include config.mak.dev
endif
-GIT-VERSION-FILE: FORCE
+$(GIT_VERSION_FILE): FORCE
@OLD=$$(cat $@ 2>/dev/null || :) && \
$(call version_gen,"$(shell pwd)",GIT-VERSION-FILE.in,$@) && \
NEW=$$(cat $@ 2>/dev/null || :) && \
@@ -1482,7 +1496,7 @@ GIT-VERSION-FILE: FORCE
# otherwise any user-provided value for GIT_VERSION would have been overridden
# already.
GIT_VERSION_OVERRIDE := $(GIT_VERSION)
--include GIT-VERSION-FILE
+-include $(GIT_VERSION_FILE)
# what 'all' will build and 'install' will install in gitexecdir,
# excluding programs for built-in commands
@@ -2403,9 +2417,9 @@ endif
GIT_USER_AGENT_SQ = $(subst ','\'',$(GIT_USER_AGENT))
GIT_USER_AGENT_CQ = "$(subst ",\",$(subst \,\\,$(GIT_USER_AGENT)))"
GIT_USER_AGENT_CQ_SQ = $(subst ','\'',$(GIT_USER_AGENT_CQ))
-GIT-USER-AGENT: FORCE
- @if test x'$(GIT_USER_AGENT_SQ)' != x"`cat GIT-USER-AGENT 2>/dev/null`"; then \
- echo '$(GIT_USER_AGENT_SQ)' >GIT-USER-AGENT; \
+$(GIT_USER_AGENT_FILE): FORCE
+ @if test x'$(GIT_USER_AGENT_SQ)' != x"`cat $(GIT_USER_AGENT_FILE) 2>/dev/null`"; then \
+ echo '$(GIT_USER_AGENT_SQ)' >$(GIT_USER_AGENT_FILE); \
fi
ifdef DEFAULT_HELP_FORMAT
@@ -2523,7 +2537,7 @@ strip: $(PROGRAMS) git$X
# dependencies here will not need to change if the force-build
# details change some day.
-git.sp git.s git.o: GIT-PREFIX
+git.sp git.s git.o: $(GIT_PREFIX)
git.sp git.s git.o: EXTRA_CPPFLAGS = \
'-DGIT_HTML_PATH="$(htmldir_relative_SQ)"' \
'-DGIT_MAN_PATH="$(mandir_relative_SQ)"' \
@@ -2533,10 +2547,10 @@ git$X: git.o GIT-LDFLAGS $(BUILTIN_OBJS) $(GITLIBS)
$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
$(filter %.o,$^) $(LIBS)
-help.sp help.s help.o: command-list.h
+help.sp help.s $(call maybe_join_path,$(CARGO_OUT_DIR),help.o): $(COMMAND_LIST_H)
builtin/bugreport.sp builtin/bugreport.s builtin/bugreport.o: hook-list.h
-builtin/help.sp builtin/help.s builtin/help.o: config-list.h GIT-PREFIX
+builtin/help.sp builtin/help.s builtin/help.o: config-list.h $(GIT_PREFIX)
builtin/help.sp builtin/help.s builtin/help.o: EXTRA_CPPFLAGS = \
'-DGIT_HTML_PATH="$(htmldir_relative_SQ)"' \
'-DGIT_MAN_PATH="$(mandir_relative_SQ)"' \
@@ -2545,13 +2559,13 @@ builtin/help.sp builtin/help.s builtin/help.o: EXTRA_CPPFLAGS = \
PAGER_ENV_SQ = $(subst ','\'',$(PAGER_ENV))
PAGER_ENV_CQ = "$(subst ",\",$(subst \,\\,$(PAGER_ENV)))"
PAGER_ENV_CQ_SQ = $(subst ','\'',$(PAGER_ENV_CQ))
-pager.sp pager.s pager.o: EXTRA_CPPFLAGS = \
+pager.sp pager.s $(call maybe_join_path,$(CARGO_OUT_DIR),pager.o): EXTRA_CPPFLAGS = \
-DPAGER_ENV='$(PAGER_ENV_CQ_SQ)'
-version-def.h: version-def.h.in GIT-VERSION-GEN GIT-VERSION-FILE GIT-USER-AGENT
+$(VERSION_DEF_H): version-def.h.in GIT-VERSION-GEN $(GIT_VERSION_FILE) $(GIT_USER_AGENT_FILE)
$(QUIET_GEN)$(call version_gen,"$(shell pwd)",$<,$@)
-version.sp version.s version.o: version-def.h
+version.sp version.s $(call maybe_join_path,$(CARGO_OUT_DIR),version.o): $(VERSION_DEF_H)
$(BUILT_INS): git$X
$(QUIET_BUILT_IN)$(RM) $@ && \
@@ -2564,9 +2578,9 @@ config-list.h: generate-configlist.sh
config-list.h: Documentation/*config.adoc Documentation/config/*.adoc
$(QUIET_GEN)$(SHELL_PATH) ./generate-configlist.sh . $@
-command-list.h: generate-cmdlist.sh command-list.txt
+$(COMMAND_LIST_H): generate-cmdlist.sh command-list.txt
-command-list.h: $(wildcard Documentation/git*.adoc)
+$(COMMAND_LIST_H): $(wildcard Documentation/git*.adoc)
$(QUIET_GEN)$(SHELL_PATH) ./generate-cmdlist.sh \
$(patsubst %,--exclude-program %,$(EXCLUDED_PROGRAMS)) \
. $@
@@ -2589,10 +2603,10 @@ $(SCRIPT_SH_GEN) $(SCRIPT_LIB) : % : %.sh generate-script.sh GIT-BUILD-OPTIONS G
$(QUIET_GEN)./generate-script.sh "$<" "$@+" ./GIT-BUILD-OPTIONS && \
mv $@+ $@
-git.rc: git.rc.in GIT-VERSION-GEN GIT-VERSION-FILE
+git.rc: git.rc.in GIT-VERSION-GEN $(GIT_VERSION_FILE)
$(QUIET_GEN)$(call version_gen,"$(shell pwd)",$<,$@)
-git.res: git.rc GIT-PREFIX
+git.res: git.rc $(GIT_PREFIX)
$(QUIET_RC)$(RC) -i $< -o $@
# This makes sure we depend on the NO_PERL setting itself.
@@ -2626,8 +2640,8 @@ endif
PERL_DEFINES += $(gitexecdir) $(perllibdir) $(localedir)
-$(SCRIPT_PERL_GEN): % : %.perl generate-perl.sh GIT-PERL-DEFINES GIT-PERL-HEADER GIT-VERSION-FILE
- $(QUIET_GEN)$(SHELL_PATH) generate-perl.sh ./GIT-BUILD-OPTIONS ./GIT-VERSION-FILE GIT-PERL-HEADER "$<" "$@+" && \
+$(SCRIPT_PERL_GEN): % : %.perl generate-perl.sh GIT-PERL-DEFINES GIT-PERL-HEADER $(GIT_VERSION_FILE)
+ $(QUIET_GEN)$(SHELL_PATH) generate-perl.sh ./GIT-BUILD-OPTIONS $(GIT_VERSION_PATH) GIT-PERL-HEADER "$<" "$@+" && \
mv $@+ $@
PERL_DEFINES := $(subst $(space),:,$(PERL_DEFINES))
@@ -2691,7 +2705,7 @@ CONFIGURE_RECIPE = sed -e 's/@GIT_VERSION@/$(GIT_VERSION)/g' \
autoconf -o configure configure.ac+ && \
$(RM) configure.ac+
-configure: configure.ac GIT-VERSION-FILE
+configure: configure.ac $(GIT_VERSION_FILE)
$(QUIET_GEN)$(CONFIGURE_RECIPE)
ifdef AUTOCONFIGURED
@@ -2774,6 +2788,14 @@ endif
.PHONY: objects
objects: $(OBJECTS)
+ifdef CARGO_OUT_DIR
+OBJECTS := $(addprefix $(CARGO_OUT_DIR)/,$(OBJECTS))
+LIB_OBJS := $(addprefix $(CARGO_OUT_DIR)/,$(LIB_OBJS))
+REFTABLE_OBJS := $(addprefix $(CARGO_OUT_DIR)/,$(REFTABLE_OBJS))
+XDIFF_OBJS := $(addprefix $(CARGO_OUT_DIR)/,$(XDIFF_OBJS))
+BASIC_CFLAGS += -I$(CARGO_OUT_DIR)
+endif
+
dep_files := $(foreach f,$(OBJECTS),$(dir $f).depend/$(notdir $f).d)
dep_dirs := $(addsuffix .depend,$(sort $(dir $(OBJECTS))))
@@ -2805,10 +2827,15 @@ missing_compdb_dir =
compdb_args =
endif
-$(OBJECTS): %.o: %.c GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
+ifdef CARGO_OUT_DIR
+$(OBJECTS): $(CARGO_OUT_DIR)/%.o: %.c $(GIT_CFLAGS) $(missing_dep_dirs) $(missing_compdb_dir)
+ $(QUIET_CC)$(CC) -o $(CARGO_OUT_DIR)/$*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $<
+else
+$(OBJECTS): %.o: %.c $(GIT_CFLAGS) $(missing_dep_dirs) $(missing_compdb_dir)
$(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $<
+endif
-%.s: %.c GIT-CFLAGS FORCE
+%.s: %.c $(GIT_CFLAGS) FORCE
$(QUIET_CC)$(CC) -o $@ -S $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $<
ifdef USE_COMPUTED_HEADER_DEPENDENCIES
@@ -2829,27 +2856,27 @@ compile_commands.json:
@if test -s $@+; then mv $@+ $@; else $(RM) $@+; fi
endif
-exec-cmd.sp exec-cmd.s exec-cmd.o: GIT-PREFIX
-exec-cmd.sp exec-cmd.s exec-cmd.o: EXTRA_CPPFLAGS = \
+exec-cmd.sp exec-cmd.s $(call maybe_join_path,$(CARGO_OUT_DIR),exec-cmd.o): $(GIT_PREFIX)
+exec-cmd.sp exec-cmd.s $(call maybe_join_path,$(CARGO_OUT_DIR),exec-cmd.o): EXTRA_CPPFLAGS = \
'-DGIT_EXEC_PATH="$(gitexecdir_SQ)"' \
'-DGIT_LOCALE_PATH="$(localedir_relative_SQ)"' \
'-DBINDIR="$(bindir_relative_SQ)"' \
'-DFALLBACK_RUNTIME_PREFIX="$(prefix_SQ)"'
-setup.sp setup.s setup.o: GIT-PREFIX
-setup.sp setup.s setup.o: EXTRA_CPPFLAGS = \
+setup.sp setup.s $(call maybe_join_path,$(CARGO_OUT_DIR),setup.o): $(GIT_PREFIX)
+setup.sp setup.s $(call maybe_join_path,$(CARGO_OUT_DIR),setup.o): EXTRA_CPPFLAGS = \
-DDEFAULT_GIT_TEMPLATE_DIR='"$(template_dir_SQ)"'
-config.sp config.s config.o: GIT-PREFIX
-config.sp config.s config.o: EXTRA_CPPFLAGS = \
+config.sp config.s $(call maybe_join_path,$(CARGO_OUT_DIR),config.o): $(GIT_PREFIX)
+config.sp config.s $(call maybe_join_path,$(CARGO_OUT_DIR),config.o): EXTRA_CPPFLAGS = \
-DETC_GITCONFIG='"$(ETC_GITCONFIG_SQ)"'
-attr.sp attr.s attr.o: GIT-PREFIX
-attr.sp attr.s attr.o: EXTRA_CPPFLAGS = \
+attr.sp attr.s $(call maybe_join_path,$(CARGO_OUT_DIR),attr.o): $(GIT_PREFIX)
+attr.sp attr.s $(call maybe_join_path,$(CARGO_OUT_DIR),attr.o): EXTRA_CPPFLAGS = \
-DETC_GITATTRIBUTES='"$(ETC_GITATTRIBUTES_SQ)"'
-gettext.sp gettext.s gettext.o: GIT-PREFIX
-gettext.sp gettext.s gettext.o: EXTRA_CPPFLAGS = \
+gettext.sp gettext.s $(call maybe_join_path,$(CARGO_OUT_DIR),gettext.o): $(GIT_PREFIX)
+gettext.sp gettext.s $(call maybe_join_path,$(CARGO_OUT_DIR),gettext.o): EXTRA_CPPFLAGS = \
-DGIT_LOCALE_PATH='"$(localedir_relative_SQ)"'
http-push.sp http.sp http-walker.sp remote-curl.sp imap-send.sp: SP_EXTRA_FLAGS += \
@@ -2872,7 +2899,7 @@ compat/nedmalloc/nedmalloc.sp compat/nedmalloc/nedmalloc.o: EXTRA_CPPFLAGS = \
compat/nedmalloc/nedmalloc.sp: SP_EXTRA_FLAGS += -Wno-non-pointer-null
endif
-headless-git.o: compat/win32/headless.c GIT-CFLAGS
+headless-git.o: compat/win32/headless.c $(GIT_CFLAGS)
$(QUIET_CC)$(CC) $(ALL_CFLAGS) $(COMPAT_CFLAGS) \
-fno-stack-protector -o $@ -c -Wall -Wwrite-strings $<
@@ -3116,9 +3143,9 @@ endif
NO_PERL_CPAN_FALLBACKS_SQ = $(subst ','\'',$(NO_PERL_CPAN_FALLBACKS))
endif
-perl/build/lib/%.pm: perl/%.pm generate-perl.sh GIT-BUILD-OPTIONS GIT-VERSION-FILE GIT-PERL-DEFINES
+perl/build/lib/%.pm: perl/%.pm generate-perl.sh GIT-BUILD-OPTIONS $(GIT_VERSION_FILE) GIT-PERL-DEFINES
$(call mkdir_p_parent_template)
- $(QUIET_GEN)$(SHELL_PATH) generate-perl.sh ./GIT-BUILD-OPTIONS ./GIT-VERSION-FILE GIT-PERL-HEADER "$<" "$@"
+ $(QUIET_GEN)$(SHELL_PATH) generate-perl.sh ./GIT-BUILD-OPTIONS $(GIT_VERSION_PATH) GIT-PERL-HEADER "$<" "$@"
perl/build/man/man3/Git.3pm: perl/Git.pm
$(call mkdir_p_parent_template)
@@ -3146,20 +3173,20 @@ cscope: cscope.out
TRACK_PREFIX = $(bindir_SQ):$(gitexecdir_SQ):$(template_dir_SQ):$(prefix_SQ):\
$(localedir_SQ)
-GIT-PREFIX: FORCE
+$(GIT_PREFIX): FORCE
@FLAGS='$(TRACK_PREFIX)'; \
- if test x"$$FLAGS" != x"`cat GIT-PREFIX 2>/dev/null`" ; then \
+ if test x"$$FLAGS" != x"`cat $(GIT_PREFIX) 2>/dev/null`" ; then \
echo >&2 " * new prefix flags"; \
- echo "$$FLAGS" >GIT-PREFIX; \
+ echo "$$FLAGS" >$(GIT_PREFIX); \
fi
TRACK_CFLAGS = $(CC):$(subst ','\'',$(ALL_CFLAGS)):$(USE_GETTEXT_SCHEME)
-GIT-CFLAGS: FORCE
+$(GIT_CFLAGS): FORCE
@FLAGS='$(TRACK_CFLAGS)'; \
- if test x"$$FLAGS" != x"`cat GIT-CFLAGS 2>/dev/null`" ; then \
+ if test x"$$FLAGS" != x"`cat $(GIT_CFLAGS) 2>/dev/null`" ; then \
echo >&2 " * new build flags"; \
- echo "$$FLAGS" >GIT-CFLAGS; \
+ echo "$$FLAGS" >$(GIT_CFLAGS); \
fi
TRACK_LDFLAGS = $(subst ','\'',$(ALL_LDFLAGS))
@@ -3751,7 +3778,7 @@ clean: profile-clean coverage-clean cocciclean
$(RM) $(FUZZ_PROGRAMS)
$(RM) $(SP_OBJ)
$(RM) $(HCC)
- $(RM) version-def.h
+ $(RM) $(VERSION_DEF_H)
$(RM) -r $(dep_dirs) $(compdb_dir) compile_commands.json
$(RM) $(test_bindir_programs)
$(RM) -r po/build/
@@ -3774,8 +3801,8 @@ ifndef NO_TCLTK
$(MAKE) -C gitk-git clean
$(MAKE) -C git-gui clean
endif
- $(RM) GIT-VERSION-FILE GIT-CFLAGS GIT-LDFLAGS GIT-BUILD-OPTIONS
- $(RM) GIT-USER-AGENT GIT-PREFIX
+ $(RM) $(GIT_VERSION_FILE) $(GIT_CFLAGS) GIT-LDFLAGS GIT-BUILD-OPTIONS
+ $(RM) $(GIT_USER_AGENT_FILE) $(GIT_PREFIX)
$(RM) GIT-SCRIPT-DEFINES GIT-PERL-DEFINES GIT-PERL-HEADER GIT-PYTHON-VARS
ifdef MSVC
$(RM) $(patsubst %.o,%.o.pdb,$(OBJECTS))
@@ -3937,14 +3964,14 @@ ifdef INCLUDE_LIBGIT_RS
all:: libgit-sys libgit-rs
endif
-LIBGIT_PUB_OBJS += contrib/libgitpub/public_symbol_export.o
-LIBGIT_PUB_OBJS += libgit.a
-LIBGIT_PUB_OBJS += reftable/libreftable.a
-LIBGIT_PUB_OBJS += xdiff/lib.a
-
-LIBGIT_PARTIAL_EXPORT = contrib/libgitpub/partial_symbol_export.o
+LIBGIT_PUB_OBJS += $(CARGO_OUT_DIR)/contrib/libgitpub/public_symbol_export.o
+LIBGIT_PUB_OBJS += $(LIB_FILE)
+LIBGIT_PUB_OBJS += $(REFTABLE_LIB)
+LIBGIT_PUB_OBJS += $(XDIFF_LIB)
-LIBGIT_HIDDEN_EXPORT = contrib/libgitpub/hidden_symbol_export.o
+LIBGIT_PARTIAL_EXPORT = $(CARGO_OUT_DIR)/contrib/libgitpub/partial_symbol_export.o
+LIBGIT_HIDDEN_EXPORT = $(CARGO_OUT_DIR)/contrib/libgitpub/hidden_symbol_export.o
+GITPUB_LIB = $(call maybe_join_path,$(CARGO_OUT_DIR),contrib/libgitpub/libgitpub.a)
$(LIBGIT_PARTIAL_EXPORT): $(LIBGIT_PUB_OBJS)
$(LD) -r $^ -o $@
@@ -3952,5 +3979,5 @@ $(LIBGIT_PARTIAL_EXPORT): $(LIBGIT_PUB_OBJS)
$(LIBGIT_HIDDEN_EXPORT): $(LIBGIT_PARTIAL_EXPORT)
$(OBJCOPY) --localize-hidden $^ $@
-contrib/libgitpub/libgitpub.a: $(LIBGIT_HIDDEN_EXPORT)
+$(GITPUB_LIB): $(LIBGIT_HIDDEN_EXPORT)
$(AR) $(ARFLAGS) $@ $^
diff --git a/contrib/libgit-sys/build.rs b/contrib/libgit-sys/build.rs
index e0d979c196..19407663f5 100644
--- a/contrib/libgit-sys/build.rs
+++ b/contrib/libgit-sys/build.rs
@@ -6,7 +6,7 @@ pub fn main() -> std::io::Result<()> {
ac.emit_has_path("std::ffi::c_char");
let crate_root = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap());
- let git_root = crate_root.join("../..");
+ let git_root = crate_root.join("git-src");
let dst = PathBuf::from(env::var_os("OUT_DIR").unwrap());
let make_output = make_cmd::gnu_make()
@@ -14,8 +14,9 @@ pub fn main() -> std::io::Result<()> {
.env_remove("PROFILE")
.current_dir(git_root.clone())
.args([
+ &format!("CARGO_OUT_DIR={}", dst.display()),
"INCLUDE_LIBGIT_RS=YesPlease",
- "contrib/libgitpub/libgitpub.a",
+ &format!("{}/contrib/libgitpub/libgitpub.a", dst.display()),
])
.output()
.expect("Make failed to run");
@@ -26,8 +27,8 @@ pub fn main() -> std::io::Result<()> {
String::from_utf8(make_output.stderr).unwrap()
);
}
- std::fs::copy(git_root.join("contrib/libgitpub/libgitpub.a"), dst.join("libgitpub.a"))?;
println!("cargo:rustc-link-search=native={}", dst.display());
+ println!("cargo:rustc-link-search=native={}", dst.join("contrib/libgitpub").display());
println!("cargo:rustc-link-lib=gitpub");
println!("cargo:rerun-if-changed={}", git_root.display());
diff --git a/contrib/libgit-sys/git-src b/contrib/libgit-sys/git-src
new file mode 120000
index 0000000000..c25bddb6dd
--- /dev/null
+++ b/contrib/libgit-sys/git-src
@@ -0,0 +1 @@
+../..
\ No newline at end of file
diff --git a/shared.mak b/shared.mak
index 1a99848a95..0dc611dd90 100644
--- a/shared.mak
+++ b/shared.mak
@@ -127,3 +127,8 @@ GIT_USER_AGENT="$(GIT_USER_AGENT)" \
GIT_VERSION="$(GIT_VERSION_OVERRIDE)" \
$(SHELL_PATH) "$(1)/GIT-VERSION-GEN" "$(1)" "$(2)" "$(3)"
endef
+
+# Apply a path prefix if the prefix is non-empty
+define maybe_join_path
+$(if $(1),$(1)/$(2),$(2))
+endef
--
2.49.0.395.g12beb8f557-goog
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v2 3/5] libgit-sys: parallelize build with Cargo's jobserver
2025-03-21 22:14 ` [PATCH v2 0/5] Fix `cargo package` for libgit-sys Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 1/5] libgitpub: move to separate contrib/ directory Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree Josh Steadmon
@ 2025-03-21 22:14 ` Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 4/5] libgit-sys: exclude unnecessary directories in git-src Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 5/5] libgit-{sys,rs}: add license and description fields Josh Steadmon
4 siblings, 0 replies; 21+ messages in thread
From: Josh Steadmon @ 2025-03-21 22:14 UTC (permalink / raw)
To: git; +Cc: brian m. carlson, Phillip Wood
Cargo provides GNU Make flags that can be used to connect to its jobserver for
managing parallel builds. They are not automatically passed to any `make`
invocations, so let's add that to our build.rs script.
Signed-off-by: Josh Steadmon <steadmon@google.com>
---
contrib/libgit-sys/build.rs | 2 ++
1 file changed, 2 insertions(+)
diff --git a/contrib/libgit-sys/build.rs b/contrib/libgit-sys/build.rs
index 19407663f5..1237c9a37f 100644
--- a/contrib/libgit-sys/build.rs
+++ b/contrib/libgit-sys/build.rs
@@ -8,9 +8,11 @@ pub fn main() -> std::io::Result<()> {
let crate_root = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap());
let git_root = crate_root.join("git-src");
let dst = PathBuf::from(env::var_os("OUT_DIR").unwrap());
+ let makeflags = env::var_os("CARGO_MAKEFLAGS").unwrap();
let make_output = make_cmd::gnu_make()
.env("DEVELOPER", "1")
+ .env("MAKEFLAGS", &makeflags)
.env_remove("PROFILE")
.current_dir(git_root.clone())
.args([
--
2.49.0.395.g12beb8f557-goog
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v2 4/5] libgit-sys: exclude unnecessary directories in git-src
2025-03-21 22:14 ` [PATCH v2 0/5] Fix `cargo package` for libgit-sys Josh Steadmon
` (2 preceding siblings ...)
2025-03-21 22:14 ` [PATCH v2 3/5] libgit-sys: parallelize build with Cargo's jobserver Josh Steadmon
@ 2025-03-21 22:14 ` Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 5/5] libgit-{sys,rs}: add license and description fields Josh Steadmon
4 siblings, 0 replies; 21+ messages in thread
From: Josh Steadmon @ 2025-03-21 22:14 UTC (permalink / raw)
To: git; +Cc: brian m. carlson, Phillip Wood
We can avoid copying tens of megabytes of unnecessary source files by excluding
a few directories which are not needed to compile libgitpub.a. This helps us
stay below crates.io's 10 MB size limit.
Signed-off-by: Josh Steadmon <steadmon@google.com>
---
Makefile | 2 +-
contrib/libgit-sys/Cargo.toml | 12 ++++++++++++
contrib/libgit-sys/build.rs | 3 +++
3 files changed, 16 insertions(+), 1 deletion(-)
diff --git a/Makefile b/Makefile
index 91677448ba..1574fc76e2 100644
--- a/Makefile
+++ b/Makefile
@@ -3500,13 +3500,13 @@ coccicheck-pending: $(COCCICHECK_PATCHES_PENDING_INTREE)
# "Sub"-Makefiles, not really because they can't be run stand-alone,
# only there to contain directory-specific rules and variables
+ifndef NO_GITWEB
## gitweb/Makefile inclusion:
MAK_DIR_GITWEB = gitweb/
include gitweb/Makefile
.PHONY: gitweb
gitweb: $(MAK_DIR_GITWEB_ALL)
-ifndef NO_GITWEB
all:: gitweb
endif
diff --git a/contrib/libgit-sys/Cargo.toml b/contrib/libgit-sys/Cargo.toml
index e0623022c3..1c1efb6211 100644
--- a/contrib/libgit-sys/Cargo.toml
+++ b/contrib/libgit-sys/Cargo.toml
@@ -7,6 +7,18 @@ links = "gitpub"
rust-version = "1.63" # TODO: Once we hit 1.84 or newer, we may want to remove Cargo.lock from
# version control. See https://lore.kernel.org/git/Z47jgK-oMjFRSslr@tapette.crustytoothpaste.net/
description = "Native bindings to a portion of libgit"
+exclude = [
+ "git-src/.github",
+ "git-src/Documentation",
+ "git-src/ci",
+ "git-src/git-gui",
+ "git-src/gitk-git",
+ "git-src/gitweb",
+ "git-src/oss-fuzz",
+ "git-src/perl",
+ "git-src/po",
+ "git-src/t"
+]
[lib]
path = "src/lib.rs"
diff --git a/contrib/libgit-sys/build.rs b/contrib/libgit-sys/build.rs
index 1237c9a37f..7e1244fcd5 100644
--- a/contrib/libgit-sys/build.rs
+++ b/contrib/libgit-sys/build.rs
@@ -17,6 +17,9 @@ pub fn main() -> std::io::Result<()> {
.current_dir(git_root.clone())
.args([
&format!("CARGO_OUT_DIR={}", dst.display()),
+ "NO_GITWEB=YesPlease",
+ "NO_PERL=YesPlease",
+ "NO_TCLTK=YesPlease",
"INCLUDE_LIBGIT_RS=YesPlease",
&format!("{}/contrib/libgitpub/libgitpub.a", dst.display()),
])
--
2.49.0.395.g12beb8f557-goog
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v2 5/5] libgit-{sys,rs}: add license and description fields
2025-03-21 22:14 ` [PATCH v2 0/5] Fix `cargo package` for libgit-sys Josh Steadmon
` (3 preceding siblings ...)
2025-03-21 22:14 ` [PATCH v2 4/5] libgit-sys: exclude unnecessary directories in git-src Josh Steadmon
@ 2025-03-21 22:14 ` Josh Steadmon
4 siblings, 0 replies; 21+ messages in thread
From: Josh Steadmon @ 2025-03-21 22:14 UTC (permalink / raw)
To: git; +Cc: brian m. carlson, Phillip Wood
Add license and description fields for libgit-sys and libgit-rs. These
are required to publish our crates on crates.io.
Signed-off-by: Josh Steadmon <steadmon@google.com>
---
contrib/libgit-rs/Cargo.toml | 2 ++
contrib/libgit-sys/Cargo.toml | 1 +
2 files changed, 3 insertions(+)
diff --git a/contrib/libgit-rs/Cargo.toml b/contrib/libgit-rs/Cargo.toml
index c3289e69db..9747ba3cb4 100644
--- a/contrib/libgit-rs/Cargo.toml
+++ b/contrib/libgit-rs/Cargo.toml
@@ -5,6 +5,8 @@ edition = "2021"
build = "build.rs"
rust-version = "1.63" # TODO: Once we hit 1.84 or newer, we may want to remove Cargo.lock from
# version control. See https://lore.kernel.org/git/Z47jgK-oMjFRSslr@tapette.crustytoothpaste.net/
+description = "Proof-of-concept Rust API for a portion of libgit"
+license = "GPL-2.0-only"
[lib]
diff --git a/contrib/libgit-sys/Cargo.toml b/contrib/libgit-sys/Cargo.toml
index 1c1efb6211..cfbcca9c8c 100644
--- a/contrib/libgit-sys/Cargo.toml
+++ b/contrib/libgit-sys/Cargo.toml
@@ -7,6 +7,7 @@ links = "gitpub"
rust-version = "1.63" # TODO: Once we hit 1.84 or newer, we may want to remove Cargo.lock from
# version control. See https://lore.kernel.org/git/Z47jgK-oMjFRSslr@tapette.crustytoothpaste.net/
description = "Native bindings to a portion of libgit"
+license = "GPL-2.0-only"
exclude = [
"git-src/.github",
"git-src/Documentation",
--
2.49.0.395.g12beb8f557-goog
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree
2025-03-21 22:14 ` [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree Josh Steadmon
@ 2025-03-23 1:46 ` Eric Sunshine
2025-03-24 15:42 ` Junio C Hamano
0 siblings, 1 reply; 21+ messages in thread
From: Eric Sunshine @ 2025-03-23 1:46 UTC (permalink / raw)
To: Josh Steadmon; +Cc: git, brian m. carlson, Phillip Wood
On Fri, Mar 21, 2025 at 6:14 PM Josh Steadmon <steadmon@google.com> wrote:
> Unlike `cargo build`, `cargo package` does not get access to the entire Git repo
> containing a Rust crate. Instead, it prepares a directory starting from the
> crate root (potentially excluding files, such as those not under version
> control, or explicity excluded in the Cargo.toml file).
s/explicity/explicitly/
> diff --git a/contrib/libgit-sys/git-src b/contrib/libgit-sys/git-src
> @@ -0,0 +1 @@
> +../..
> \ No newline at end of file
Meh.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree
2025-03-23 1:46 ` Eric Sunshine
@ 2025-03-24 15:42 ` Junio C Hamano
2025-03-25 17:57 ` Josh Steadmon
0 siblings, 1 reply; 21+ messages in thread
From: Junio C Hamano @ 2025-03-24 15:42 UTC (permalink / raw)
To: Eric Sunshine; +Cc: Josh Steadmon, git, brian m. carlson, Phillip Wood
Eric Sunshine <sunshine@sunshineco.com> writes:
> On Fri, Mar 21, 2025 at 6:14 PM Josh Steadmon <steadmon@google.com> wrote:
>> Unlike `cargo build`, `cargo package` does not get access to the entire Git repo
>> containing a Rust crate. Instead, it prepares a directory starting from the
>> crate root (potentially excluding files, such as those not under version
>> control, or explicity excluded in the Cargo.toml file).
>
> s/explicity/explicitly/
>
>> diff --git a/contrib/libgit-sys/git-src b/contrib/libgit-sys/git-src
>> @@ -0,0 +1 @@
>> +../..
>> \ No newline at end of file
>
> Meh.
https://github.com/git/git/actions/runs/14030831429/job/39278185588#step:3:1
All of the Windows test jobs (not build ones) are broken due to the
presence of ../.. symbolic link.
Is that ugly hack the only way we can make this work?
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree
2025-03-24 15:42 ` Junio C Hamano
@ 2025-03-25 17:57 ` Josh Steadmon
2025-03-25 23:08 ` Junio C Hamano
0 siblings, 1 reply; 21+ messages in thread
From: Josh Steadmon @ 2025-03-25 17:57 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Eric Sunshine, git, brian m. carlson, Phillip Wood
On 2025.03.24 08:42, Junio C Hamano wrote:
> Eric Sunshine <sunshine@sunshineco.com> writes:
>
> > On Fri, Mar 21, 2025 at 6:14 PM Josh Steadmon <steadmon@google.com> wrote:
> >> Unlike `cargo build`, `cargo package` does not get access to the entire Git repo
> >> containing a Rust crate. Instead, it prepares a directory starting from the
> >> crate root (potentially excluding files, such as those not under version
> >> control, or explicity excluded in the Cargo.toml file).
> >
> > s/explicity/explicitly/
> >
> >> diff --git a/contrib/libgit-sys/git-src b/contrib/libgit-sys/git-src
> >> @@ -0,0 +1 @@
> >> +../..
> >> \ No newline at end of file
> >
> > Meh.
>
> https://github.com/git/git/actions/runs/14030831429/job/39278185588#step:3:1
>
> All of the Windows test jobs (not build ones) are broken due to the
> presence of ../.. symbolic link.
>
> Is that ugly hack the only way we can make this work?
It's the only way I know of to accomplish both:
1) creating a packaged crate with `cargo package` and
2) keeping the top-level clean of any Rust code or configuration.
If we're willing to have a Cargo.toml file in the repo root, we could
create a "Cargo workspace", but I'm not sure yet if that avoids the same
problem with accessing sources outside of the crates themselves. I'll be
able to test it out later this week.
If the workspace approach doesn't work, the alternatives are:
1) avoid the issue for now; anyone who wants to experiment with
libgit-rs can do so by building from source (but it will prevent them
from creating their own packaged crates IIUC).
2) move libgit-sys and libgit-rs to separate repos and depend on the Git
source via submodules. This is what I've seen done in other -sys crates
such as zlib-sys (https://github.com/rust-lang/libz-sys).
Of those alternatives, I prefer #1 for now. If we build enough momentum
on libification and expanding the coverage of these crates, then we
could think about switching to #2.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree
2025-03-25 17:57 ` Josh Steadmon
@ 2025-03-25 23:08 ` Junio C Hamano
2025-03-27 18:58 ` Josh Steadmon
0 siblings, 1 reply; 21+ messages in thread
From: Junio C Hamano @ 2025-03-25 23:08 UTC (permalink / raw)
To: Josh Steadmon; +Cc: Eric Sunshine, git, brian m. carlson, Phillip Wood
Josh Steadmon <steadmon@google.com> writes:
> 2) keeping the top-level clean of any Rust code or configuration.
>
> If we're willing to have a Cargo.toml file in the repo root, ...
If it is more like adding a new build configuration file whereever
we have Makefile (or meson.build), and is not like we are adding one
new file per one existing source file, then I see no reason why we
want to avoid adding a few files to the root-level.
> ... we could
> create a "Cargo workspace", but I'm not sure yet if that avoids the same
> problem with accessing sources outside of the crates themselves. I'll be
> able to test it out later this week.
Yeah, that would probably be a reasonable thing to try. Thanks.
> If the workspace approach doesn't work, the alternatives are:
>
> 1) avoid the issue for now; anyone who wants to experiment with
> libgit-rs can do so by building from source (but it will prevent them
> from creating their own packaged crates IIUC).
>
> 2) move libgit-sys and libgit-rs to separate repos and depend on the Git
> source via submodules. This is what I've seen done in other -sys crates
> such as zlib-sys (https://github.com/rust-lang/libz-sys).
>
> Of those alternatives, I prefer #1 for now. If we build enough momentum
> on libification and expanding the coverage of these crates, then we
> could think about switching to #2.
Yeah, or putting it another way, #1 would help us gather enough Rust
minded folks who are familiar enough to come up with ideas and offer
better ways to manage this part of the system.
Thanks.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree
2025-03-25 23:08 ` Junio C Hamano
@ 2025-03-27 18:58 ` Josh Steadmon
2025-03-29 10:46 ` Junio C Hamano
2025-03-31 14:52 ` Johannes Schindelin
0 siblings, 2 replies; 21+ messages in thread
From: Josh Steadmon @ 2025-03-27 18:58 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Eric Sunshine, git, brian m. carlson, Phillip Wood
On 2025.03.25 16:08, Junio C Hamano wrote:
> Josh Steadmon <steadmon@google.com> writes:
>
> > 2) keeping the top-level clean of any Rust code or configuration.
> >
> > If we're willing to have a Cargo.toml file in the repo root, ...
>
> If it is more like adding a new build configuration file whereever
> we have Makefile (or meson.build), and is not like we are adding one
> new file per one existing source file, then I see no reason why we
> want to avoid adding a few files to the root-level.
>
> > ... we could
> > create a "Cargo workspace", but I'm not sure yet if that avoids the same
> > problem with accessing sources outside of the crates themselves. I'll be
> > able to test it out later this week.
>
> Yeah, that would probably be a reasonable thing to try. Thanks.
>
>
> > If the workspace approach doesn't work, the alternatives are:
> >
> > 1) avoid the issue for now; anyone who wants to experiment with
> > libgit-rs can do so by building from source (but it will prevent them
> > from creating their own packaged crates IIUC).
> >
> > 2) move libgit-sys and libgit-rs to separate repos and depend on the Git
> > source via submodules. This is what I've seen done in other -sys crates
> > such as zlib-sys (https://github.com/rust-lang/libz-sys).
> >
> > Of those alternatives, I prefer #1 for now. If we build enough momentum
> > on libification and expanding the coverage of these crates, then we
> > could think about switching to #2.
>
> Yeah, or putting it another way, #1 would help us gather enough Rust
> minded folks who are familiar enough to come up with ideas and offer
> better ways to manage this part of the system.
>
> Thanks.
Unfortunately creating a workspace doesn't provide access to the
top-level source. Symlinks seem to be the only recommended approach [1]
for embedded crates, but since that breaks Windows CI let's just drop
the series for now.
[1] https://users.rust-lang.org/t/including-files-from-parent-directory-in-package/88969
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree
2025-03-27 18:58 ` Josh Steadmon
@ 2025-03-29 10:46 ` Junio C Hamano
2025-03-31 14:52 ` Johannes Schindelin
1 sibling, 0 replies; 21+ messages in thread
From: Junio C Hamano @ 2025-03-29 10:46 UTC (permalink / raw)
To: Josh Steadmon; +Cc: Eric Sunshine, git, brian m. carlson, Phillip Wood
Josh Steadmon <steadmon@google.com> writes:
> Unfortunately creating a workspace doesn't provide access to the
> top-level source. Symlinks seem to be the only recommended approach [1]
> for embedded crates, but since that breaks Windows CI let's just drop
> the series for now.
Understood. Thanks.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree
2025-03-27 18:58 ` Josh Steadmon
2025-03-29 10:46 ` Junio C Hamano
@ 2025-03-31 14:52 ` Johannes Schindelin
1 sibling, 0 replies; 21+ messages in thread
From: Johannes Schindelin @ 2025-03-31 14:52 UTC (permalink / raw)
To: Josh Steadmon
Cc: Junio C Hamano, Eric Sunshine, git, brian m. carlson,
Phillip Wood
Hi Josh,
On Thu, 27 Mar 2025, Josh Steadmon wrote:
> On 2025.03.25 16:08, Junio C Hamano wrote:
> > Josh Steadmon <steadmon@google.com> writes:
> >
> > > 2) keeping the top-level clean of any Rust code or configuration.
> > >
> > > If we're willing to have a Cargo.toml file in the repo root, ...
> >
> > If it is more like adding a new build configuration file whereever
> > we have Makefile (or meson.build), and is not like we are adding one
> > new file per one existing source file, then I see no reason why we
> > want to avoid adding a few files to the root-level.
> >
> > > ... we could
> > > create a "Cargo workspace", but I'm not sure yet if that avoids the same
> > > problem with accessing sources outside of the crates themselves. I'll be
> > > able to test it out later this week.
> >
> > Yeah, that would probably be a reasonable thing to try. Thanks.
> >
> >
> > > If the workspace approach doesn't work, the alternatives are:
> > >
> > > 1) avoid the issue for now; anyone who wants to experiment with
> > > libgit-rs can do so by building from source (but it will prevent them
> > > from creating their own packaged crates IIUC).
> > >
> > > 2) move libgit-sys and libgit-rs to separate repos and depend on the Git
> > > source via submodules. This is what I've seen done in other -sys crates
> > > such as zlib-sys (https://github.com/rust-lang/libz-sys).
> > >
> > > Of those alternatives, I prefer #1 for now. If we build enough momentum
> > > on libification and expanding the coverage of these crates, then we
> > > could think about switching to #2.
> >
> > Yeah, or putting it another way, #1 would help us gather enough Rust
> > minded folks who are familiar enough to come up with ideas and offer
> > better ways to manage this part of the system.
> >
> > Thanks.
>
> Unfortunately creating a workspace doesn't provide access to the
> top-level source. Symlinks seem to be the only recommended approach [1]
> for embedded crates, but since that breaks Windows CI let's just drop
> the series for now.
>
> [1] https://users.rust-lang.org/t/including-files-from-parent-directory-in-package/88969
If you need symbolic linkson Windows in CI, please set
`MSYS=winsymlinks:nativestrict` like it is done here:
https://github.com/git-for-windows/git/blob/4ca71ba5311a8f1bafbf002e97e076f15dcfc15b/t/t2040-checkout-symlink-attr.sh#L8-L10.
It might fail, though, if it requires _Git_ to be able to create symbolic
links because I have not yet managed to upstream the patches to implement
that.
We could fast-track support for `readlink()` without the other parts
(https://github.com/git-for-windows/git/commit/86420a1b84d1), but last
time I checked, Git's test suite did not pass under
`MSYS=winsymlinks:nativestrict` for some reasons I was unable to analyze
for lack of time.
Ciao,
Johannes
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2025-03-31 14:52 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-03-18 23:24 [RFC PATCH v1 0/4] Fix `cargo package` for libgit-sys Josh Steadmon
2025-03-18 23:24 ` [RFC PATCH v1 1/4] libgitpub: move to separate contrib/ directory Josh Steadmon
2025-03-18 23:24 ` [RFC PATCH v1 2/4] libgit-sys: add symlink to git repo root and clean after build Josh Steadmon
2025-03-19 22:23 ` Josh Steadmon
2025-03-20 11:10 ` Phillip Wood
2025-03-21 19:49 ` Josh Steadmon
2025-03-18 23:24 ` [RFC PATCH v1 3/4] libgit-sys: parallelize build with Cargo's jobserver Josh Steadmon
2025-03-18 23:24 ` [RFC PATCH v1 4/4] libgit-sys: exclude unnecessary directories in git-src Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 0/5] Fix `cargo package` for libgit-sys Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 1/5] libgitpub: move to separate contrib/ directory Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree Josh Steadmon
2025-03-23 1:46 ` Eric Sunshine
2025-03-24 15:42 ` Junio C Hamano
2025-03-25 17:57 ` Josh Steadmon
2025-03-25 23:08 ` Junio C Hamano
2025-03-27 18:58 ` Josh Steadmon
2025-03-29 10:46 ` Junio C Hamano
2025-03-31 14:52 ` Johannes Schindelin
2025-03-21 22:14 ` [PATCH v2 3/5] libgit-sys: parallelize build with Cargo's jobserver Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 4/5] libgit-sys: exclude unnecessary directories in git-src Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 5/5] libgit-{sys,rs}: add license and description fields Josh Steadmon
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).