From: Junio C Hamano <gitster@pobox.com>
To: Christian Couder <christian.couder@gmail.com>
Cc: git@vger.kernel.org, John Cai <johncai86@gmail.com>,
Jonathan Tan <jonathantanmy@google.com>,
Jonathan Nieder <jrnieder@gmail.com>,
Taylor Blau <me@ttaylorr.com>, Derrick Stolee <stolee@gmail.com>,
Patrick Steinhardt <ps@pks.im>,
Christian Couder <chriscool@tuxfamily.org>
Subject: Re: [PATCH 8/9] repack: implement `--filter-to` for storing filtered out objects
Date: Thu, 15 Jun 2023 19:21:56 -0700 [thread overview]
Message-ID: <xmqqfs6srvsr.fsf@gitster.g> (raw)
In-Reply-To: <20230614192541.1599256-9-christian.couder@gmail.com> (Christian Couder's message of "Wed, 14 Jun 2023 21:25:40 +0200")
Christian Couder <christian.couder@gmail.com> writes:
> A previous commit has implemented `git repack --filter=<filter-spec>` to
> allow users to filter out some objects from the main pack and move them
> into a new different pack.
>
> It would be nice if this new different pack could be created in a
> different directory than the regular pack. This would make it possible
> to move large blobs into a pack on a different kind of storage, for
> example cheaper storage. Even in a different directory this pack can be
> accessible if, for example, the Git alternates mechanism is used to
> point to it.
Makes sense, I guess, for "in other usecases" scenario. I am not
sure how this would be useful for the originally stated goal of
unbloating a bloated repository with promisor remote(s), though.
> If users want to remove a pack that contains filtered out objects after
> checking that they are all already on a promisor remote, creating the
> pack in a different directory makes it easier to do so.
Care to elaborate? I do not see how a separate directory would make
it easier. After separating the potential cruft into a packfile,
you'd walk its .idx and see if there are any objects that are not
available (yet) at the promisor remotes to check if it is safe to
remove. That can be done regardless of the location of the packfile
that is suspected to be now removable.
> Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
> ---
> Documentation/git-repack.txt | 6 ++++++
> builtin/repack.c | 17 ++++++++++++-----
> t/t7700-repack.sh | 27 +++++++++++++++++++++++++++
> 3 files changed, 45 insertions(+), 5 deletions(-)
>
> diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt
> index aa29c7e648..070dd22610 100644
> --- a/Documentation/git-repack.txt
> +++ b/Documentation/git-repack.txt
> @@ -148,6 +148,12 @@ depth is 4095.
> resulting packfile and put them into a separate packfile. See
> linkgit:git-rev-list[1] for valid `<filter-spec>` forms.
>
> +--filter-to=<dir>::
> + Write the pack containing filtered out objects to the
> + directory `<dir>`. This can be used for putting the pack on a
> + separate object directory that is accessed through the Git
> + alternates mechanism. Only useful with `--filter`.
> +
> -b::
> --write-bitmap-index::
> Write a reachability bitmap index as part of the repack. This
> diff --git a/builtin/repack.c b/builtin/repack.c
> index b13d7196de..8c71e8fd51 100644
> --- a/builtin/repack.c
> +++ b/builtin/repack.c
> @@ -838,7 +838,8 @@ static void prepare_pack_filtered_cmd(struct child_process *cmd,
> }
>
> static void finish_pack_filtered_cmd(struct child_process *cmd,
> - struct string_list *names)
> + struct string_list *names,
> + const char *destination)
> {
> if (cmd->in == -1) {
> /* No packed objects; cmd was never started */
> @@ -848,7 +849,7 @@ static void finish_pack_filtered_cmd(struct child_process *cmd,
>
> close(cmd->in);
>
> - if (finish_pack_objects_cmd(cmd, names, NULL, NULL))
> + if (finish_pack_objects_cmd(cmd, names, destination, NULL))
> die(_("could not finish pack-objects to pack filtered objects"));
> }
>
> @@ -877,6 +878,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
> const char *cruft_expiration = NULL;
> const char *expire_to = NULL;
> struct child_process pack_filtered_cmd = CHILD_PROCESS_INIT;
> + const char *filter_to = NULL;
>
> struct option builtin_repack_options[] = {
> OPT_BIT('a', NULL, &pack_everything,
> @@ -930,6 +932,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
> N_("write a multi-pack index of the resulting packs")),
> OPT_STRING(0, "expire-to", &expire_to, N_("dir"),
> N_("pack prefix to store a pack containing pruned objects")),
> + OPT_STRING(0, "filter-to", &filter_to, N_("dir"),
> + N_("pack prefix to store a pack containing filtered out objects")),
> OPT_END()
> };
>
> @@ -1073,8 +1077,11 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
> strvec_push(&cmd.args, "--incremental");
> }
>
> - if (po_args.filter)
> - prepare_pack_filtered_cmd(&pack_filtered_cmd, &po_args, packtmp);
> + if (po_args.filter) {
> + if (!filter_to)
> + filter_to = packtmp;
> + prepare_pack_filtered_cmd(&pack_filtered_cmd, &po_args, filter_to);
> + }
>
> if (geometry)
> cmd.in = -1;
> @@ -1169,7 +1176,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
> }
>
> if (po_args.filter)
> - finish_pack_filtered_cmd(&pack_filtered_cmd, &names);
> + finish_pack_filtered_cmd(&pack_filtered_cmd, &names, filter_to);
>
> string_list_sort(&names);
>
> diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh
> index 9e7654090f..898f8a01b4 100755
> --- a/t/t7700-repack.sh
> +++ b/t/t7700-repack.sh
> @@ -286,6 +286,33 @@ test_expect_success 'repacking with a filter works' '
> test "$blob_pack2" = "$blob_pack"
> '
>
> +test_expect_success '--filter-to stores filtered out objects' '
> + git -C bare.git repack -a -d &&
> + test_stdout_line_count = 1 ls bare.git/objects/pack/*.pack &&
> +
> + git init --bare filtered.git &&
> + git -C bare.git -c repack.writebitmaps=false repack -a -d \
> + --filter=blob:none \
> + --filter-to=../filtered.git/objects/pack/pack &&
> + test_stdout_line_count = 1 ls bare.git/objects/pack/pack-*.pack &&
> + test_stdout_line_count = 1 ls filtered.git/objects/pack/pack-*.pack &&
> +
> + commit_pack=$(test-tool -C bare.git find-pack HEAD) &&
> + test -n "$commit_pack" &&
> + blob_pack=$(test-tool -C bare.git find-pack HEAD:file1) &&
> + test -z "$blob_pack" &&
> + blob_hash=$(git -C bare.git rev-parse HEAD:file1) &&
> + test -n "$blob_hash" &&
> + blob_pack=$(test-tool -C filtered.git find-pack $blob_hash) &&
> + test -n "$blob_pack" &&
> +
> + echo $(pwd)/filtered.git/objects >bare.git/objects/info/alternates &&
> + blob_pack=$(test-tool -C bare.git find-pack HEAD:file1) &&
> + test -n "$blob_pack" &&
> + blob_content=$(git -C bare.git show $blob_hash) &&
> + test "$blob_content" = "content1"
> +'
> +
> objdir=.git/objects
> midx=$objdir/pack/multi-pack-index
next prev parent reply other threads:[~2023-06-16 2:22 UTC|newest]
Thread overview: 161+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-06-14 19:25 [PATCH 0/9] Repack objects into separate packfiles based on a filter Christian Couder
2023-06-14 19:25 ` [PATCH 1/9] pack-objects: allow `--filter` without `--stdout` Christian Couder
2023-06-21 10:49 ` Taylor Blau
2023-07-05 6:16 ` Christian Couder
2023-06-14 19:25 ` [PATCH 2/9] pack-objects: add `--print-filtered` to print omitted objects Christian Couder
2023-06-15 22:50 ` Junio C Hamano
2023-06-21 10:52 ` Taylor Blau
2023-06-21 11:11 ` Christian Couder
2023-06-21 11:54 ` Taylor Blau
2023-06-14 19:25 ` [PATCH 3/9] t/helper: add 'find-pack' test-tool Christian Couder
2023-06-15 23:32 ` Junio C Hamano
2023-06-21 10:40 ` Christian Couder
2023-06-21 10:54 ` Taylor Blau
2023-06-14 19:25 ` [PATCH 4/9] repack: refactor piping an oid to a command Christian Couder
2023-06-15 23:46 ` Junio C Hamano
2023-06-21 10:55 ` Taylor Blau
2023-06-21 10:56 ` Christian Couder
2023-06-14 19:25 ` [PATCH 5/9] repack: refactor finishing pack-objects command Christian Couder
2023-06-16 0:13 ` Junio C Hamano
2023-06-21 11:06 ` Taylor Blau
2023-06-21 11:19 ` Christian Couder
2023-06-21 11:05 ` Taylor Blau
2023-06-14 19:25 ` [PATCH 6/9] repack: add `--filter=<filter-spec>` option Christian Couder
2023-06-16 0:43 ` Junio C Hamano
2023-06-21 11:20 ` Taylor Blau
2023-06-21 15:04 ` Christian Couder
2023-06-22 11:05 ` Taylor Blau
2023-06-21 14:40 ` Christian Couder
2023-06-21 16:53 ` Junio C Hamano
2023-06-22 8:39 ` Christian Couder
2023-06-22 18:32 ` Junio C Hamano
2023-06-21 11:17 ` Taylor Blau
2023-07-05 7:18 ` Christian Couder
2023-06-14 19:25 ` [PATCH 7/9] gc: add `gc.repackFilter` config option Christian Couder
2023-06-14 19:25 ` [PATCH 8/9] repack: implement `--filter-to` for storing filtered out objects Christian Couder
2023-06-16 2:21 ` Junio C Hamano [this message]
2023-06-21 11:49 ` Taylor Blau
2023-06-21 12:08 ` Christian Couder
2023-06-21 12:25 ` Taylor Blau
2023-06-21 16:44 ` Junio C Hamano
2023-07-05 6:19 ` Christian Couder
2023-06-14 19:25 ` [PATCH 9/9] gc: add `gc.repackFilterTo` config option Christian Couder
2023-06-16 2:54 ` Junio C Hamano
2023-06-14 21:36 ` [PATCH 0/9] Repack objects into separate packfiles based on a filter Junio C Hamano
2023-06-16 3:08 ` Junio C Hamano
2023-07-05 6:08 ` [PATCH v2 0/8] " Christian Couder
2023-07-05 6:08 ` [PATCH v2 1/8] pack-objects: allow `--filter` without `--stdout` Christian Couder
2023-07-05 6:08 ` [PATCH v2 2/8] t/helper: add 'find-pack' test-tool Christian Couder
2023-07-05 6:08 ` [PATCH v2 3/8] repack: refactor finishing pack-objects command Christian Couder
2023-07-05 6:08 ` [PATCH v2 4/8] repack: refactor finding pack prefix Christian Couder
2023-07-05 6:08 ` [PATCH v2 5/8] repack: add `--filter=<filter-spec>` option Christian Couder
2023-07-05 17:53 ` Junio C Hamano
2023-07-24 9:01 ` Christian Couder
2023-07-24 18:28 ` Junio C Hamano
2023-07-25 15:22 ` Christian Couder
2023-07-25 17:25 ` Junio C Hamano
2023-07-25 23:08 ` Junio C Hamano
2023-08-08 8:45 ` Christian Couder
2023-08-09 20:38 ` Taylor Blau
2023-08-09 22:50 ` Junio C Hamano
2023-08-09 23:38 ` Junio C Hamano
2023-08-10 0:10 ` Jeff King
2023-07-05 18:12 ` Junio C Hamano
2023-07-24 9:02 ` Christian Couder
2023-07-05 6:08 ` [PATCH v2 6/8] gc: add `gc.repackFilter` config option Christian Couder
2023-07-05 6:08 ` [PATCH v2 7/8] repack: implement `--filter-to` for storing filtered out objects Christian Couder
2023-07-05 18:26 ` Junio C Hamano
2023-07-24 9:00 ` Christian Couder
2023-07-24 18:18 ` Junio C Hamano
2023-07-25 13:41 ` Robert Coup
2023-07-25 16:50 ` Junio C Hamano
2023-07-25 15:45 ` Christian Couder
2023-07-05 6:08 ` [PATCH v2 8/8] gc: add `gc.repackFilterTo` config option Christian Couder
2023-07-24 8:59 ` [PATCH v3 0/8] Repack objects into separate packfiles based on a filter Christian Couder
2023-07-24 8:59 ` [PATCH v3 1/8] pack-objects: allow `--filter` without `--stdout` Christian Couder
2023-07-25 22:38 ` Taylor Blau
2023-07-25 23:51 ` Junio C Hamano
2023-07-24 8:59 ` [PATCH v3 2/8] t/helper: add 'find-pack' test-tool Christian Couder
2023-07-25 22:44 ` Taylor Blau
2023-08-08 8:28 ` Christian Couder
2023-07-24 8:59 ` [PATCH v3 3/8] repack: refactor finishing pack-objects command Christian Couder
2023-07-25 22:45 ` Taylor Blau
2023-07-24 8:59 ` [PATCH v3 4/8] repack: refactor finding pack prefix Christian Couder
2023-07-25 22:47 ` Taylor Blau
2023-08-08 8:29 ` Christian Couder
2023-07-24 8:59 ` [PATCH v3 5/8] repack: add `--filter=<filter-spec>` option Christian Couder
2023-07-25 23:04 ` Taylor Blau
2023-08-08 8:34 ` Christian Couder
2023-08-09 21:12 ` Taylor Blau
2023-07-24 8:59 ` [PATCH v3 6/8] gc: add `gc.repackFilter` config option Christian Couder
2023-07-25 23:07 ` Taylor Blau
2023-08-08 8:38 ` Christian Couder
2023-08-09 21:15 ` Taylor Blau
2023-07-24 8:59 ` [PATCH v3 7/8] repack: implement `--filter-to` for storing filtered out objects Christian Couder
2023-07-24 8:59 ` [PATCH v3 8/8] gc: add `gc.repackFilterTo` config option Christian Couder
2023-07-25 23:10 ` [PATCH v3 0/8] Repack objects into separate packfiles based on a filter Taylor Blau
2023-08-08 8:26 ` [PATCH v4 " Christian Couder
2023-08-08 8:26 ` [PATCH v4 1/8] pack-objects: allow `--filter` without `--stdout` Christian Couder
2023-08-08 8:26 ` [PATCH v4 2/8] t/helper: add 'find-pack' test-tool Christian Couder
2023-08-09 21:18 ` Taylor Blau
2023-08-08 8:26 ` [PATCH v4 3/8] repack: refactor finishing pack-objects command Christian Couder
2023-08-08 8:26 ` [PATCH v4 4/8] repack: refactor finding pack prefix Christian Couder
2023-08-09 21:20 ` Taylor Blau
2023-08-08 8:26 ` [PATCH v4 5/8] repack: add `--filter=<filter-spec>` option Christian Couder
2023-08-09 21:40 ` Taylor Blau
2023-08-08 8:26 ` [PATCH v4 6/8] gc: add `gc.repackFilter` config option Christian Couder
2023-08-08 8:26 ` [PATCH v4 7/8] repack: implement `--filter-to` for storing filtered out objects Christian Couder
2023-08-08 8:26 ` [PATCH v4 8/8] gc: add `gc.repackFilterTo` config option Christian Couder
2023-08-09 21:45 ` [PATCH v4 0/8] Repack objects into separate packfiles based on a filter Taylor Blau
2023-08-09 21:57 ` Junio C Hamano
2023-08-12 0:12 ` Christian Couder
2023-08-12 0:00 ` [PATCH v5 " Christian Couder
2023-08-12 0:00 ` [PATCH v5 1/8] pack-objects: allow `--filter` without `--stdout` Christian Couder
2023-08-12 0:00 ` [PATCH v5 2/8] t/helper: add 'find-pack' test-tool Christian Couder
2023-08-12 0:00 ` [PATCH v5 3/8] repack: refactor finishing pack-objects command Christian Couder
2023-08-12 0:00 ` [PATCH v5 4/8] repack: refactor finding pack prefix Christian Couder
2023-08-12 0:00 ` [PATCH v5 5/8] repack: add `--filter=<filter-spec>` option Christian Couder
2023-08-12 0:00 ` [PATCH v5 6/8] gc: add `gc.repackFilter` config option Christian Couder
2023-08-12 0:00 ` [PATCH v5 7/8] repack: implement `--filter-to` for storing filtered out objects Christian Couder
2023-08-12 0:00 ` [PATCH v5 8/8] gc: add `gc.repackFilterTo` config option Christian Couder
2023-08-15 0:51 ` [PATCH v5 0/8] Repack objects into separate packfiles based on a filter Junio C Hamano
2023-08-15 21:43 ` Taylor Blau
2023-08-15 22:32 ` Junio C Hamano
2023-08-15 23:09 ` Taylor Blau
2023-08-15 23:18 ` Junio C Hamano
2023-08-16 0:38 ` Taylor Blau
2023-08-16 17:16 ` Junio C Hamano
2023-09-11 15:20 ` Christian Couder
2023-09-11 15:06 ` [PATCH v6 0/9] " Christian Couder
2023-09-11 15:06 ` [PATCH v6 1/9] pack-objects: allow `--filter` without `--stdout` Christian Couder
2023-09-11 15:06 ` [PATCH v6 2/9] t/helper: add 'find-pack' test-tool Christian Couder
2023-09-11 15:06 ` [PATCH v6 3/9] repack: refactor finishing pack-objects command Christian Couder
2023-09-11 15:06 ` [PATCH v6 4/9] repack: refactor finding pack prefix Christian Couder
2023-09-11 15:06 ` [PATCH v6 5/9] pack-bitmap-write: rebuild using new bitmap when remapping Christian Couder
2023-09-11 15:06 ` [PATCH v6 6/9] repack: add `--filter=<filter-spec>` option Christian Couder
2023-09-11 15:06 ` [PATCH v6 7/9] gc: add `gc.repackFilter` config option Christian Couder
2023-09-11 15:06 ` [PATCH v6 8/9] repack: implement `--filter-to` for storing filtered out objects Christian Couder
2023-09-11 15:06 ` [PATCH v6 9/9] gc: add `gc.repackFilterTo` config option Christian Couder
2023-09-25 15:25 ` [PATCH v7 0/9] Repack objects into separate packfiles based on a filter Christian Couder
2023-09-25 15:25 ` [PATCH v7 1/9] pack-objects: allow `--filter` without `--stdout` Christian Couder
2023-09-25 15:25 ` [PATCH v7 2/9] t/helper: add 'find-pack' test-tool Christian Couder
2023-09-25 15:25 ` [PATCH v7 3/9] repack: refactor finishing pack-objects command Christian Couder
2023-09-25 15:25 ` [PATCH v7 4/9] repack: refactor finding pack prefix Christian Couder
2023-09-25 15:25 ` [PATCH v7 5/9] pack-bitmap-write: rebuild using new bitmap when remapping Christian Couder
2023-09-25 15:25 ` [PATCH v7 6/9] repack: add `--filter=<filter-spec>` option Christian Couder
2023-09-25 15:25 ` [PATCH v7 7/9] gc: add `gc.repackFilter` config option Christian Couder
2023-09-25 15:25 ` [PATCH v7 8/9] repack: implement `--filter-to` for storing filtered out objects Christian Couder
2023-09-25 15:25 ` [PATCH v7 9/9] gc: add `gc.repackFilterTo` config option Christian Couder
2023-09-25 19:14 ` [PATCH v7 0/9] Repack objects into separate packfiles based on a filter Junio C Hamano
2023-09-25 22:41 ` Taylor Blau
2023-10-02 16:54 ` [PATCH v8 " Christian Couder
2023-10-02 16:54 ` [PATCH v8 1/9] pack-objects: allow `--filter` without `--stdout` Christian Couder
2023-10-02 16:54 ` [PATCH v8 2/9] t/helper: add 'find-pack' test-tool Christian Couder
2023-10-02 16:54 ` [PATCH v8 3/9] repack: refactor finishing pack-objects command Christian Couder
2023-10-02 16:54 ` [PATCH v8 4/9] repack: refactor finding pack prefix Christian Couder
2023-10-02 16:55 ` [PATCH v8 5/9] pack-bitmap-write: rebuild using new bitmap when remapping Christian Couder
2023-10-02 16:55 ` [PATCH v8 6/9] repack: add `--filter=<filter-spec>` option Christian Couder
2023-10-02 16:55 ` [PATCH v8 7/9] gc: add `gc.repackFilter` config option Christian Couder
2023-10-02 16:55 ` [PATCH v8 8/9] repack: implement `--filter-to` for storing filtered out objects Christian Couder
2023-10-02 16:55 ` [PATCH v8 9/9] gc: add `gc.repackFilterTo` config option Christian Couder
2023-10-02 20:14 ` [PATCH v8 0/9] Repack objects into separate packfiles based on a filter Taylor Blau
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=xmqqfs6srvsr.fsf@gitster.g \
--to=gitster@pobox.com \
--cc=chriscool@tuxfamily.org \
--cc=christian.couder@gmail.com \
--cc=git@vger.kernel.org \
--cc=johncai86@gmail.com \
--cc=jonathantanmy@google.com \
--cc=jrnieder@gmail.com \
--cc=me@ttaylorr.com \
--cc=ps@pks.im \
--cc=stolee@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).