From: Christian Couder <christian.couder@gmail.com>
To: Junio C Hamano <gitster@pobox.com>
Cc: git@vger.kernel.org, John Cai <johncai86@gmail.com>,
Jonathan Tan <jonathantanmy@google.com>,
Jonathan Nieder <jrnieder@gmail.com>,
Taylor Blau <me@ttaylorr.com>, Derrick Stolee <stolee@gmail.com>,
Patrick Steinhardt <ps@pks.im>,
Christian Couder <chriscool@tuxfamily.org>
Subject: Re: [PATCH 6/9] repack: add `--filter=<filter-spec>` option
Date: Thu, 22 Jun 2023 10:39:09 +0200 [thread overview]
Message-ID: <CAP8UFD30Kx_vYfdV4ipPPXNVr76pKshjTUQGcJfkUvG_+KD3zQ@mail.gmail.com> (raw)
In-Reply-To: <xmqqy1kchi3l.fsf@gitster.g>
On Wed, Jun 21, 2023 at 6:53 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Christian Couder <christian.couder@gmail.com> writes:
>
> > It might indicate that we prefer to be safe, do things in different
> > steps and not provide an easy way for users to shoot their own foot.
> > For example it seems pretty safe to do things like this:
> >
> > 1) put all the objects we think should be on the promisor remote in
> > a separate packfile
> > 2) start checking that each object in that packfile is available on
> > the promisor remote
> > 3) if an object in that packfile isn't on the promisor remote, try
> > to send it there
> > 4) if we couldn't send the object, error out
> > 5) if we haven't errored out after checking all the objects in the
> > packfile, it means all these objects are now available from the
> > promisor remote and we can safely delete the packfile
>
> I may be missing something, but to me, the above sound more like a
> tail wagging the dog.
>
> Instead of saying "while repacking, we'll create the new pack with
> the objects that we suspect that we cannot re-fetch from the
> promisor (allowing false positives for safety), and store the rest
> in a backup pack (that can immediately be discarded)",
This might be a good idea, but what if users prefer to send to a
promisor remote the objects that should be on that promisor remote as
soon as possible, instead of keeping them on the local machine where
they take up possibly valuable space for no good reason?
My point is that there are a lot of different strategies that people
operating with a promisor remote could adopt, so it's better to
iteratively give them building blocks that can help them instead of
trying to find and implement right away the best solution for a
special use case or for every use case.
> the above
> says "while repacking, we'll create the new pack with objects that
> match the filter, and store the rest to another pack". But because
> the object selection criteria used in the latter is not something
> with practical/useful meaning, in other words, it does not exactly
> match what we want,
"What we want" depends on the strategy chosen to manage objects on
promisor remotes and I am not sure that the strategy you mention is
always better than the example strategy I talked about. For some users
it might be better for others it might not.
> we fill the gaps between what we want (i.e. sift
> the objects into "refetchable" and "other" bins) and what we
> happened to have implemented (i.e. sift the objects into "match
> filter" and "other" bints) by sending the objects that we _should_
> have included in the new pack (i.e. "not refetchable") to the
> promisor to make them refetchable.
>
> I do not know what to think about that. I do not think there is
> even a way to guarantee that the push done for 3) will always be
> taken and still leave the resulting promisor usable (e.g. we can
> make them connected by coming up with a random new ref to point
> these "we are sending these only because we failed to include them
> in the set of objects we should consider local" objects, but then
> how would we avoid bloating the refs at the promisor remote side
> (which now has become a "dumping ground", rather than holding the
> objects needed for histories that project participants care about).
There are some configurations where users never want to delete any git
object. In those cases it doesn't matter if the promisor remote is a
"dumping ground". Users might just want a promisor remote to keep all
the large files that have ever been pushed into the repo, to save more
precious space on the machine hosting the regular repo.
There are configurations where users can have garanties that the push
done for 3) will work with a very high probability so that the example
strategy I talked about can work reliably enough.
The example strategy I talked about is just one example where having
repack --filter work like it does in this patch series can be useful
and safe. I don't pretend that it is always the best strategy and that
some users might not prefer another better strategy for them. If
that's the case perhaps they can implement another different option
that just checks that filtered out objects are indeed available from a
promisor remote and then just omit these objects from the resulting
pack. In fact I would have nothing against such an option, and I might
even implement it myself one day (no promise though).
Right now they have nearly no helpful command (except perhaps using
pack-objects directly), so even if this is not the best possible help
in all use cases, I am just saying that this can be useful in _some_
cases.
> As an argument to salvage this series as (one of the possible
> ingredients to) a solution to "slim down a bloated lazy clone"
> problem, it sounds a bit weak.
I don't quite agree, but anyway I will use this argument less in
version 2 of this patch series, and I will talk more about the
argument that `repack --filter=...` allows users to put packfiles
containing some kinds of objects (like large blobs or all the blobs)
on cheaper disks (when not using a promisor remote).
Thanks.
next prev parent reply other threads:[~2023-06-22 8:39 UTC|newest]
Thread overview: 161+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-06-14 19:25 [PATCH 0/9] Repack objects into separate packfiles based on a filter Christian Couder
2023-06-14 19:25 ` [PATCH 1/9] pack-objects: allow `--filter` without `--stdout` Christian Couder
2023-06-21 10:49 ` Taylor Blau
2023-07-05 6:16 ` Christian Couder
2023-06-14 19:25 ` [PATCH 2/9] pack-objects: add `--print-filtered` to print omitted objects Christian Couder
2023-06-15 22:50 ` Junio C Hamano
2023-06-21 10:52 ` Taylor Blau
2023-06-21 11:11 ` Christian Couder
2023-06-21 11:54 ` Taylor Blau
2023-06-14 19:25 ` [PATCH 3/9] t/helper: add 'find-pack' test-tool Christian Couder
2023-06-15 23:32 ` Junio C Hamano
2023-06-21 10:40 ` Christian Couder
2023-06-21 10:54 ` Taylor Blau
2023-06-14 19:25 ` [PATCH 4/9] repack: refactor piping an oid to a command Christian Couder
2023-06-15 23:46 ` Junio C Hamano
2023-06-21 10:55 ` Taylor Blau
2023-06-21 10:56 ` Christian Couder
2023-06-14 19:25 ` [PATCH 5/9] repack: refactor finishing pack-objects command Christian Couder
2023-06-16 0:13 ` Junio C Hamano
2023-06-21 11:06 ` Taylor Blau
2023-06-21 11:19 ` Christian Couder
2023-06-21 11:05 ` Taylor Blau
2023-06-14 19:25 ` [PATCH 6/9] repack: add `--filter=<filter-spec>` option Christian Couder
2023-06-16 0:43 ` Junio C Hamano
2023-06-21 11:20 ` Taylor Blau
2023-06-21 15:04 ` Christian Couder
2023-06-22 11:05 ` Taylor Blau
2023-06-21 14:40 ` Christian Couder
2023-06-21 16:53 ` Junio C Hamano
2023-06-22 8:39 ` Christian Couder [this message]
2023-06-22 18:32 ` Junio C Hamano
2023-06-21 11:17 ` Taylor Blau
2023-07-05 7:18 ` Christian Couder
2023-06-14 19:25 ` [PATCH 7/9] gc: add `gc.repackFilter` config option Christian Couder
2023-06-14 19:25 ` [PATCH 8/9] repack: implement `--filter-to` for storing filtered out objects Christian Couder
2023-06-16 2:21 ` Junio C Hamano
2023-06-21 11:49 ` Taylor Blau
2023-06-21 12:08 ` Christian Couder
2023-06-21 12:25 ` Taylor Blau
2023-06-21 16:44 ` Junio C Hamano
2023-07-05 6:19 ` Christian Couder
2023-06-14 19:25 ` [PATCH 9/9] gc: add `gc.repackFilterTo` config option Christian Couder
2023-06-16 2:54 ` Junio C Hamano
2023-06-14 21:36 ` [PATCH 0/9] Repack objects into separate packfiles based on a filter Junio C Hamano
2023-06-16 3:08 ` Junio C Hamano
2023-07-05 6:08 ` [PATCH v2 0/8] " Christian Couder
2023-07-05 6:08 ` [PATCH v2 1/8] pack-objects: allow `--filter` without `--stdout` Christian Couder
2023-07-05 6:08 ` [PATCH v2 2/8] t/helper: add 'find-pack' test-tool Christian Couder
2023-07-05 6:08 ` [PATCH v2 3/8] repack: refactor finishing pack-objects command Christian Couder
2023-07-05 6:08 ` [PATCH v2 4/8] repack: refactor finding pack prefix Christian Couder
2023-07-05 6:08 ` [PATCH v2 5/8] repack: add `--filter=<filter-spec>` option Christian Couder
2023-07-05 17:53 ` Junio C Hamano
2023-07-24 9:01 ` Christian Couder
2023-07-24 18:28 ` Junio C Hamano
2023-07-25 15:22 ` Christian Couder
2023-07-25 17:25 ` Junio C Hamano
2023-07-25 23:08 ` Junio C Hamano
2023-08-08 8:45 ` Christian Couder
2023-08-09 20:38 ` Taylor Blau
2023-08-09 22:50 ` Junio C Hamano
2023-08-09 23:38 ` Junio C Hamano
2023-08-10 0:10 ` Jeff King
2023-07-05 18:12 ` Junio C Hamano
2023-07-24 9:02 ` Christian Couder
2023-07-05 6:08 ` [PATCH v2 6/8] gc: add `gc.repackFilter` config option Christian Couder
2023-07-05 6:08 ` [PATCH v2 7/8] repack: implement `--filter-to` for storing filtered out objects Christian Couder
2023-07-05 18:26 ` Junio C Hamano
2023-07-24 9:00 ` Christian Couder
2023-07-24 18:18 ` Junio C Hamano
2023-07-25 13:41 ` Robert Coup
2023-07-25 16:50 ` Junio C Hamano
2023-07-25 15:45 ` Christian Couder
2023-07-05 6:08 ` [PATCH v2 8/8] gc: add `gc.repackFilterTo` config option Christian Couder
2023-07-24 8:59 ` [PATCH v3 0/8] Repack objects into separate packfiles based on a filter Christian Couder
2023-07-24 8:59 ` [PATCH v3 1/8] pack-objects: allow `--filter` without `--stdout` Christian Couder
2023-07-25 22:38 ` Taylor Blau
2023-07-25 23:51 ` Junio C Hamano
2023-07-24 8:59 ` [PATCH v3 2/8] t/helper: add 'find-pack' test-tool Christian Couder
2023-07-25 22:44 ` Taylor Blau
2023-08-08 8:28 ` Christian Couder
2023-07-24 8:59 ` [PATCH v3 3/8] repack: refactor finishing pack-objects command Christian Couder
2023-07-25 22:45 ` Taylor Blau
2023-07-24 8:59 ` [PATCH v3 4/8] repack: refactor finding pack prefix Christian Couder
2023-07-25 22:47 ` Taylor Blau
2023-08-08 8:29 ` Christian Couder
2023-07-24 8:59 ` [PATCH v3 5/8] repack: add `--filter=<filter-spec>` option Christian Couder
2023-07-25 23:04 ` Taylor Blau
2023-08-08 8:34 ` Christian Couder
2023-08-09 21:12 ` Taylor Blau
2023-07-24 8:59 ` [PATCH v3 6/8] gc: add `gc.repackFilter` config option Christian Couder
2023-07-25 23:07 ` Taylor Blau
2023-08-08 8:38 ` Christian Couder
2023-08-09 21:15 ` Taylor Blau
2023-07-24 8:59 ` [PATCH v3 7/8] repack: implement `--filter-to` for storing filtered out objects Christian Couder
2023-07-24 8:59 ` [PATCH v3 8/8] gc: add `gc.repackFilterTo` config option Christian Couder
2023-07-25 23:10 ` [PATCH v3 0/8] Repack objects into separate packfiles based on a filter Taylor Blau
2023-08-08 8:26 ` [PATCH v4 " Christian Couder
2023-08-08 8:26 ` [PATCH v4 1/8] pack-objects: allow `--filter` without `--stdout` Christian Couder
2023-08-08 8:26 ` [PATCH v4 2/8] t/helper: add 'find-pack' test-tool Christian Couder
2023-08-09 21:18 ` Taylor Blau
2023-08-08 8:26 ` [PATCH v4 3/8] repack: refactor finishing pack-objects command Christian Couder
2023-08-08 8:26 ` [PATCH v4 4/8] repack: refactor finding pack prefix Christian Couder
2023-08-09 21:20 ` Taylor Blau
2023-08-08 8:26 ` [PATCH v4 5/8] repack: add `--filter=<filter-spec>` option Christian Couder
2023-08-09 21:40 ` Taylor Blau
2023-08-08 8:26 ` [PATCH v4 6/8] gc: add `gc.repackFilter` config option Christian Couder
2023-08-08 8:26 ` [PATCH v4 7/8] repack: implement `--filter-to` for storing filtered out objects Christian Couder
2023-08-08 8:26 ` [PATCH v4 8/8] gc: add `gc.repackFilterTo` config option Christian Couder
2023-08-09 21:45 ` [PATCH v4 0/8] Repack objects into separate packfiles based on a filter Taylor Blau
2023-08-09 21:57 ` Junio C Hamano
2023-08-12 0:12 ` Christian Couder
2023-08-12 0:00 ` [PATCH v5 " Christian Couder
2023-08-12 0:00 ` [PATCH v5 1/8] pack-objects: allow `--filter` without `--stdout` Christian Couder
2023-08-12 0:00 ` [PATCH v5 2/8] t/helper: add 'find-pack' test-tool Christian Couder
2023-08-12 0:00 ` [PATCH v5 3/8] repack: refactor finishing pack-objects command Christian Couder
2023-08-12 0:00 ` [PATCH v5 4/8] repack: refactor finding pack prefix Christian Couder
2023-08-12 0:00 ` [PATCH v5 5/8] repack: add `--filter=<filter-spec>` option Christian Couder
2023-08-12 0:00 ` [PATCH v5 6/8] gc: add `gc.repackFilter` config option Christian Couder
2023-08-12 0:00 ` [PATCH v5 7/8] repack: implement `--filter-to` for storing filtered out objects Christian Couder
2023-08-12 0:00 ` [PATCH v5 8/8] gc: add `gc.repackFilterTo` config option Christian Couder
2023-08-15 0:51 ` [PATCH v5 0/8] Repack objects into separate packfiles based on a filter Junio C Hamano
2023-08-15 21:43 ` Taylor Blau
2023-08-15 22:32 ` Junio C Hamano
2023-08-15 23:09 ` Taylor Blau
2023-08-15 23:18 ` Junio C Hamano
2023-08-16 0:38 ` Taylor Blau
2023-08-16 17:16 ` Junio C Hamano
2023-09-11 15:20 ` Christian Couder
2023-09-11 15:06 ` [PATCH v6 0/9] " Christian Couder
2023-09-11 15:06 ` [PATCH v6 1/9] pack-objects: allow `--filter` without `--stdout` Christian Couder
2023-09-11 15:06 ` [PATCH v6 2/9] t/helper: add 'find-pack' test-tool Christian Couder
2023-09-11 15:06 ` [PATCH v6 3/9] repack: refactor finishing pack-objects command Christian Couder
2023-09-11 15:06 ` [PATCH v6 4/9] repack: refactor finding pack prefix Christian Couder
2023-09-11 15:06 ` [PATCH v6 5/9] pack-bitmap-write: rebuild using new bitmap when remapping Christian Couder
2023-09-11 15:06 ` [PATCH v6 6/9] repack: add `--filter=<filter-spec>` option Christian Couder
2023-09-11 15:06 ` [PATCH v6 7/9] gc: add `gc.repackFilter` config option Christian Couder
2023-09-11 15:06 ` [PATCH v6 8/9] repack: implement `--filter-to` for storing filtered out objects Christian Couder
2023-09-11 15:06 ` [PATCH v6 9/9] gc: add `gc.repackFilterTo` config option Christian Couder
2023-09-25 15:25 ` [PATCH v7 0/9] Repack objects into separate packfiles based on a filter Christian Couder
2023-09-25 15:25 ` [PATCH v7 1/9] pack-objects: allow `--filter` without `--stdout` Christian Couder
2023-09-25 15:25 ` [PATCH v7 2/9] t/helper: add 'find-pack' test-tool Christian Couder
2023-09-25 15:25 ` [PATCH v7 3/9] repack: refactor finishing pack-objects command Christian Couder
2023-09-25 15:25 ` [PATCH v7 4/9] repack: refactor finding pack prefix Christian Couder
2023-09-25 15:25 ` [PATCH v7 5/9] pack-bitmap-write: rebuild using new bitmap when remapping Christian Couder
2023-09-25 15:25 ` [PATCH v7 6/9] repack: add `--filter=<filter-spec>` option Christian Couder
2023-09-25 15:25 ` [PATCH v7 7/9] gc: add `gc.repackFilter` config option Christian Couder
2023-09-25 15:25 ` [PATCH v7 8/9] repack: implement `--filter-to` for storing filtered out objects Christian Couder
2023-09-25 15:25 ` [PATCH v7 9/9] gc: add `gc.repackFilterTo` config option Christian Couder
2023-09-25 19:14 ` [PATCH v7 0/9] Repack objects into separate packfiles based on a filter Junio C Hamano
2023-09-25 22:41 ` Taylor Blau
2023-10-02 16:54 ` [PATCH v8 " Christian Couder
2023-10-02 16:54 ` [PATCH v8 1/9] pack-objects: allow `--filter` without `--stdout` Christian Couder
2023-10-02 16:54 ` [PATCH v8 2/9] t/helper: add 'find-pack' test-tool Christian Couder
2023-10-02 16:54 ` [PATCH v8 3/9] repack: refactor finishing pack-objects command Christian Couder
2023-10-02 16:54 ` [PATCH v8 4/9] repack: refactor finding pack prefix Christian Couder
2023-10-02 16:55 ` [PATCH v8 5/9] pack-bitmap-write: rebuild using new bitmap when remapping Christian Couder
2023-10-02 16:55 ` [PATCH v8 6/9] repack: add `--filter=<filter-spec>` option Christian Couder
2023-10-02 16:55 ` [PATCH v8 7/9] gc: add `gc.repackFilter` config option Christian Couder
2023-10-02 16:55 ` [PATCH v8 8/9] repack: implement `--filter-to` for storing filtered out objects Christian Couder
2023-10-02 16:55 ` [PATCH v8 9/9] gc: add `gc.repackFilterTo` config option Christian Couder
2023-10-02 20:14 ` [PATCH v8 0/9] Repack objects into separate packfiles based on a filter Taylor Blau
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAP8UFD30Kx_vYfdV4ipPPXNVr76pKshjTUQGcJfkUvG_+KD3zQ@mail.gmail.com \
--to=christian.couder@gmail.com \
--cc=chriscool@tuxfamily.org \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=johncai86@gmail.com \
--cc=jonathantanmy@google.com \
--cc=jrnieder@gmail.com \
--cc=me@ttaylorr.com \
--cc=ps@pks.im \
--cc=stolee@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).