From: Christian Couder <christian.couder@gmail.com>
To: git@vger.kernel.org
Cc: Junio C Hamano <gitster@pobox.com>,
John Cai <johncai86@gmail.com>,
Jonathan Tan <jonathantanmy@google.com>,
Jonathan Nieder <jrnieder@gmail.com>,
Taylor Blau <me@ttaylorr.com>, Derrick Stolee <stolee@gmail.com>,
Christian Couder <christian.couder@gmail.com>
Subject: [PATCH v4 0/3] Implement filtering repacks
Date: Wed, 21 Dec 2022 05:04:43 +0100 [thread overview]
Message-ID: <20221221040446.2860985-1-christian.couder@gmail.com> (raw)
In-Reply-To: <20221122175150.366828-1-christian.couder@gmail.com>
Earlier this year, John Cai sent 2 versions of a patch series to
implement `git repack --filter=<filter-spec>`:
https://lore.kernel.org/git/pull.1206.git.git.1643248180.gitgitgadget@gmail.com/
We tried to "sell" it as a way to use partial clone on a Git server to
offload large blobs to, for example, an http server, while using
multiple promisor remotes on the client side.
Even though it is still our end goal, it seems a bit far fetched for
now and unnecessary as `git repack --filter=<filter-spec>` could be
useful on the client side too.
For example one might want to clone with a filter to avoid too many
space to be taken by some large blobs, and one might realize after
some time that a number of the large blobs have still be downloaded
because some old branches referencing them were checked out. In this
case a filtering repack could remove some of those large blobs.
Some of the comments on the patch series that John sent were related
to the possible data loss and repo corruption that a filtering repack
could cause. It's indeed true that it could be very dangerous, so the
first version of this patch series asked the user to confirm the
command, either by answering 'Y' on the command line or by passing
`--force`.
In the discussion with Junio following that first version though, it
appeared that asking for such confirmation might not be necessary, so
the v2 removed those checks.
Taylor though asked what would happen to the 'remote.<name>.promisor'
and 'remote.<name>.partialclonefilter' config variables when a
filtering repack is run. As it seemed to me that we should just check
that a promisor remote has been configured and fail if that's not the
case, that was implemented in the third version of this patch series.
In the discussions following the first, second and third versions,
Junio commented that `git gc` was a better way for users to launch
filtering repacks then `git repack`, so in this v4 a new
'gc.repackFilter' config option is implemented that allows `git gc` to
perform filtering repacks. When this config option is set to a non
empty string, `git gc` will just add a `--filter=<filter-spec>`
argument to the repack processes it launches, with '<filter-spec>' set
to the value of 'gc.repackFilter'.
So the changes in this v4 compared to v3 are the following:
- rebased on top of 57e2c6ebbe (Start the 2.40 cycle, 2022-12-14) to
avoid a simple conflict,
- simplified the test in patch 2/3 by using `grep -c ...` instead of
`grep ... | wc -l`,
- added patch 3/3 which implements a new 'gc.repackFilter' config
option so that `git gc` can perform filtering repacks.
Thanks to Junio and Taylor for discussing the v1, v2 and v3, to John
Cai, who worked on the previous versions, to Jonathan Nieder, Jonathan
Tan and Taylor, who discussed this with me at the Git Merge and
Contributor Summit, and to Stolee, Taylor, Robert Coup and Junio who
discussed the versions John sent.
Range diff with v3:
1: 1e64cac782 < -: ---------- pack-objects: allow --filter without --stdout
-: ---------- > 1: c2dca82dee pack-objects: allow --filter without --stdout
2: 7216a7bc05 ! 2: 1dcdba4b1d repack: add --filter=<filter-spec> option
@@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix
+ write_promisor_file_1(line.buf);
item->util = populate_pack_exts(item->string);
}
- fclose(out);
+ strbuf_release(&line);
## t/t7700-repack.sh ##
@@ t/t7700-repack.sh: test_expect_success 'auto-bitmaps do not complain if unavailable' '
@@ t/t7700-repack.sh: test_expect_success 'auto-bitmaps do not complain if unavaila
+ git clone --bare --no-local server client &&
+ git -C client config remote.origin.promisor true &&
+ git -C client rev-list --objects --all --missing=print >objects &&
-+ test $(grep "^?" objects | wc -l) = 0 &&
++ test $(grep -c "^?" objects) = 0 &&
+ git -C client -c repack.writebitmaps=false repack -a -d --filter=blob:none &&
+ git -C client rev-list --objects --all --missing=print >objects &&
-+ test $(grep "^?" objects | wc -l) = 1
++ test $(grep -c "^?" objects) = 1
+'
+
objdir=.git/objects
-: ---------- > 3: 6bb98b4b00 gc: add gc.repackFilter config option
Christian Couder (3):
pack-objects: allow --filter without --stdout
repack: add --filter=<filter-spec> option
gc: add gc.repackFilter config option
Documentation/config/gc.txt | 9 +++++++++
Documentation/git-repack.txt | 8 ++++++++
builtin/gc.c | 6 ++++++
builtin/pack-objects.c | 8 ++------
builtin/repack.c | 28 +++++++++++++++++++++-------
t/t6500-gc.sh | 19 +++++++++++++++++++
t/t7700-repack.sh | 15 +++++++++++++++
7 files changed, 80 insertions(+), 13 deletions(-)
--
2.39.0.59.g395bcb85bc.dirty
next prev parent reply other threads:[~2022-12-21 4:05 UTC|newest]
Thread overview: 42+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-10-12 13:51 [PATCH 0/3] Implement filtering repacks Christian Couder
2022-10-12 13:51 ` [PATCH 1/3] pack-objects: allow --filter without --stdout Christian Couder
2022-10-12 13:51 ` [PATCH 2/3] repack: add --filter=<filter-spec> option Christian Couder
2022-10-12 13:51 ` [PATCH 3/3] repack: introduce --force to force filtering Christian Couder
2022-10-14 16:46 ` [PATCH 0/3] Implement filtering repacks Junio C Hamano
2022-10-20 11:23 ` Christian Couder
2022-10-28 19:49 ` Taylor Blau
2022-10-28 20:26 ` Junio C Hamano
2022-11-07 9:12 ` Christian Couder
2022-11-07 9:00 ` Christian Couder
2022-10-25 12:28 ` [PATCH v2 0/2] " Christian Couder
2022-10-25 12:28 ` [PATCH v2 1/2] pack-objects: allow --filter without --stdout Christian Couder
2022-10-25 12:28 ` [PATCH v2 2/2] repack: add --filter=<filter-spec> option Christian Couder
2022-10-28 19:54 ` [PATCH v2 0/2] Implement filtering repacks Taylor Blau
2022-11-07 9:29 ` Christian Couder
2022-11-22 17:51 ` [PATCH v3 " Christian Couder
2022-11-22 17:51 ` [PATCH v3 1/2] pack-objects: allow --filter without --stdout Christian Couder
2022-11-22 17:51 ` [PATCH v3 2/2] repack: add --filter=<filter-spec> option Christian Couder
2022-11-23 0:31 ` [PATCH v3 0/2] Implement filtering repacks Junio C Hamano
2022-12-21 3:53 ` Christian Couder
2022-11-23 0:35 ` Junio C Hamano
2022-12-21 4:04 ` Christian Couder [this message]
2022-12-21 4:04 ` [PATCH v4 1/3] pack-objects: allow --filter without --stdout Christian Couder
2023-01-04 14:56 ` Patrick Steinhardt
2022-12-21 4:04 ` [PATCH v4 2/3] repack: add --filter=<filter-spec> option Christian Couder
2023-01-04 14:56 ` Patrick Steinhardt
2023-01-05 1:39 ` Junio C Hamano
2022-12-21 4:04 ` [PATCH v4 3/3] gc: add gc.repackFilter config option Christian Couder
2023-01-04 14:57 ` Patrick Steinhardt
2024-05-15 13:25 ` [PATCH v2 0/3] upload-pack: support a missing-action Christian Couder
2024-05-15 13:25 ` [PATCH v2 1/3] rev-list: refactor --missing=<missing-action> Christian Couder
2024-05-15 16:16 ` Junio C Hamano
2024-05-15 13:25 ` [PATCH v2 2/3] pack-objects: use the missing action API Christian Couder
2024-05-15 16:46 ` Junio C Hamano
2024-05-24 16:40 ` Christian Couder
2024-05-15 13:25 ` [PATCH v2 3/3] upload-pack: allow configuring a missing-action Christian Couder
2024-05-15 17:08 ` Junio C Hamano
2024-05-24 16:41 ` Christian Couder
2024-05-24 21:51 ` Junio C Hamano
2024-05-28 10:10 ` Christian Couder
2024-05-28 15:54 ` Junio C Hamano
2024-05-15 13:59 ` [PATCH v2 0/3] upload-pack: support " Christian Couder
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20221221040446.2860985-1-christian.couder@gmail.com \
--to=christian.couder@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=johncai86@gmail.com \
--cc=jonathantanmy@google.com \
--cc=jrnieder@gmail.com \
--cc=me@ttaylorr.com \
--cc=stolee@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).