From: Taylor Blau <me@ttaylorr.com>
To: Jeff King <peff@peff.net>
Cc: git@vger.kernel.org, Chris Torek <chris.torek@gmail.com>,
Derrick Stolee <derrickstolee@github.com>,
Junio C Hamano <gitster@pobox.com>
Subject: Re: [PATCH v3 2/2] builtin/pack-objects.c: introduce `pack.recentObjectsHook`
Date: Mon, 15 May 2023 16:38:17 -0400 [thread overview]
Message-ID: <ZGKYOeMGs4fGO5bz@nand.local> (raw)
In-Reply-To: <20230512212456.GA2495860@coredump.intra.peff.net>
On Fri, May 12, 2023 at 05:24:56PM -0400, Jeff King wrote:
> I think this description suffers a bit from being adapted from the
> original patch which was targeting cruft packs. It's not clear to me
> what "the caller" means here. And really, I think this is getting into
> the details before giving an overview and motivation.
>
> I'd expect something the rationale to be something like:
Re-reading it myself, I tend to agree with you. I modified it quite a
bit, and I'm much happier with the result. Thanks for mentioning it.
> One option I don't see here is: update the mtime on the objects you want
> to salvage.
>
> Why would we want this patch instead of just having the caller update
> the mtimes of objects (or in a cruft-pack world, call a command that
> rewrites the .mtimes file with new values)?
>
> I can think of some possible arguments against it (you might want to
> retain the old mtimes, or you might find it a hassle to have to
> continually update them before gc kills them). But I think the commit
> message should probably make those arguments.
I agree with everything you wrote here.
> > We then add those as tips to another reachability traversal (along with
> > any recent objects, if pruning), marking every object along the way
> > (either adding it to the cruft pack, or writing it out as a loose
> > object).
>
> I didn't understand this "if pruning" comment. If we are not pruning at
> all, wouldn't we skip the extra traversal entirely, since we know we are
> saving everything?
I was talking about the rescuing traversal for generating a cruft pack.
But I ended up dropping this whole paragraph anyway, since I don't think
it's adding anything in the context of the new patch message.
> > @@ -126,8 +198,14 @@ static int want_recent_object(struct recent_data *data,
> > const struct object_id *oid)
> > {
> > if (data->ignore_in_core_kept_packs &&
> > - has_object_kept_pack(oid, IN_CORE_KEEP_PACKS))
> > + has_object_kept_pack(oid, IN_CORE_KEEP_PACKS)) {
> > + if (!data->extra_recent_oids_loaded)
> > + load_pack_recent_objects(data);
> > + if (oidset_contains(&data->extra_recent_oids, oid))
> > + return 1;
> > +
> > return 0;
> > + }
> > return 1;
> > }
>
> This hunk I'm less sure about. The purpose of this function is that the
> caller has told us about some packs which are "special", and we avoid
> adding their objects to the traversal.
>
> This kicks in for cruft packs, when the git-repack caller says "I just
> made pack xyz.pack; do not bother saving anything in it to the cruft
> pack, since xyz.pack is here to stay". So if a hook says "you should
> keep object X", why would we want to override that check? It is already
> a reachable object that has been packed into xyz.pack, so we know there
> is no point in even considering its recency.
Yup, you're absolutely right here. Thanks for catching it.
> > --- a/t/t5329-pack-objects-cruft.sh
> > +++ b/t/t5329-pack-objects-cruft.sh
> > @@ -739,4 +739,175 @@ test_expect_success 'cruft objects are freshend via loose' '
> > )
> > '
> >
> > +test_expect_success 'additional cruft tips may be specified via pack.extraCruftTips' '
>
> This title (and others below) seems out of date. :)
Thanks for noticing, fixed.
> > diff --git a/t/t7701-repack-unpack-unreachable.sh b/t/t7701-repack-unpack-unreachable.sh
> > index ebb267855f..d2eea6e754 100755
> > --- a/t/t7701-repack-unpack-unreachable.sh
> > +++ b/t/t7701-repack-unpack-unreachable.sh
> > @@ -113,6 +113,28 @@ test_expect_success 'do not bother loosening old objects' '
> > test_must_fail git cat-file -p $obj2
> > '
> >
> > +test_expect_success 'extra recent tips are kept regardless of age' '
> > + obj1=$(echo one | git hash-object -w --stdin) &&
> > + obj2=$(echo two | git hash-object -w --stdin) &&
> > + pack1=$(echo $obj1 | git pack-objects .git/objects/pack/pack) &&
> > + pack2=$(echo $obj2 | git pack-objects .git/objects/pack/pack) &&
> > + git prune-packed &&
> > +
> > + git cat-file -p $obj1 &&
> > + git cat-file -p $obj2 &&
> > +
> > + write_script extra-tips <<-EOF &&
> > + echo $obj2
> > + EOF
> > + git config pack.recentObjectsHook ./extra-tips &&
> > +
> > + test-tool chmtime =-86400 .git/objects/pack/pack-$pack2.pack &&
> > + git repack -A -d --unpack-unreachable=1.hour.ago &&
> > +
> > + git cat-file -p $obj1 &&
> > + git cat-file -p $obj2
> > +'
>
> And this is the new test in this iteration covering the "repack -A"
> case.
>
> It is checking that $obj2, which our hook mentions, is saved. It also
> checks that $obj1 is saved because it is still recent. But there are two
> other possibly interesting cases:
>
> - an object that is too old and is _not_ saved. It seems useful to
> confirm that the new patch does not simply break the ability to drop
> objects. ;)
>
> - an object that is reachable from $obj2 is also saved. From a
> white-box perspective this is less interesting, because we should
> already test elsewhere that this works for recent objects, and we
> know the new feature is implemented by faking recency. But it might
> be worth it for completeness, and because it's easy to do (making
> $obj2 a tag pointing to a blob should work).
All very good cases to check for. Here's a patch on top (which I'll
obviously squash into my new version, but figured I'd send it as a
response to you directly, too):
--- 8< ---
S
diff --git a/t/t7701-repack-unpack-unreachable.sh b/t/t7701-repack-unpack-unreachable.sh
index d2eea6e754..fa2df6016b 100755
--- a/t/t7701-repack-unpack-unreachable.sh
+++ b/t/t7701-repack-unpack-unreachable.sh
@@ -116,23 +116,33 @@ test_expect_success 'do not bother loosening old objects' '
test_expect_success 'extra recent tips are kept regardless of age' '
obj1=$(echo one | git hash-object -w --stdin) &&
obj2=$(echo two | git hash-object -w --stdin) &&
+ obj3=$(echo three | git hash-object -w --stdin) &&
pack1=$(echo $obj1 | git pack-objects .git/objects/pack/pack) &&
pack2=$(echo $obj2 | git pack-objects .git/objects/pack/pack) &&
+ pack3=$(echo $obj3 | git pack-objects .git/objects/pack/pack) &&
git prune-packed &&
git cat-file -p $obj1 &&
git cat-file -p $obj2 &&
+ git cat-file -p $obj3 &&
- write_script extra-tips <<-EOF &&
- echo $obj2
+ git tag -a -m tag obj2-tag $obj2 &&
+ obj2_tag="$(git rev-parse obj2-tag)" &&
+
+
+ write_script precious-objects <<-EOF &&
+ echo $obj2_tag
EOF
- git config pack.recentObjectsHook ./extra-tips &&
+ git config pack.recentObjectsHook ./precious-objects &&
test-tool chmtime =-86400 .git/objects/pack/pack-$pack2.pack &&
+ test-tool chmtime =-86400 .git/objects/pack/pack-$pack3.pack &&
git repack -A -d --unpack-unreachable=1.hour.ago &&
git cat-file -p $obj1 &&
- git cat-file -p $obj2
+ git cat-file -p $obj2 &&
+ git cat-file -p $obj2_tag &&
+ test_must_fail git cat-file -p $obj3
'
test_expect_success 'keep packed objects found only in index' '
--- >8 ---
Thanks,
Taylor
next prev parent reply other threads:[~2023-05-15 20:38 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-11 23:20 [PATCH v3 0/2] pack-objects: introduce `pack.extraRecentObjectsHook` Taylor Blau
2023-05-11 23:20 ` [PATCH v3 1/2] reachable.c: extract `obj_is_recent()` Taylor Blau
2023-05-11 23:20 ` [PATCH v3 2/2] builtin/pack-objects.c: introduce `pack.recentObjectsHook` Taylor Blau
2023-05-12 4:58 ` Jeff King
2023-05-15 20:15 ` Taylor Blau
2023-05-12 21:24 ` Jeff King
2023-05-12 21:36 ` Taylor Blau
2023-05-12 21:46 ` Jeff King
2023-05-12 21:45 ` Jeff King
2023-05-12 22:01 ` Jeff King
2023-05-12 23:21 ` Junio C Hamano
2023-05-13 0:11 ` Jeff King
2023-05-13 0:11 ` Jeff King
2023-05-15 20:49 ` Taylor Blau
2023-05-15 20:38 ` Taylor Blau [this message]
2023-05-11 23:23 ` [PATCH v3 0/2] pack-objects: introduce `pack.extraRecentObjectsHook` Taylor Blau
2023-05-11 23:39 ` Junio C Hamano
2023-05-11 23:48 ` Taylor Blau
2023-05-16 0:23 ` [PATCH v4 0/2] gc: introduce `gc.recentObjectsHook` Taylor Blau
2023-05-16 0:24 ` [PATCH v4 1/2] reachable.c: extract `obj_is_recent()` Taylor Blau
2023-05-16 0:24 ` [PATCH v4 2/2] gc: introduce `gc.recentObjectsHook` Taylor Blau
2023-05-24 23:21 ` Glen Choo
2023-06-07 22:56 ` Taylor Blau
2023-06-07 22:58 ` [PATCH v5 0/2] " Taylor Blau
2023-06-07 22:58 ` [PATCH v5 1/2] reachable.c: extract `obj_is_recent()` Taylor Blau
2023-06-07 22:58 ` [PATCH v5 2/2] gc: introduce `gc.recentObjectsHook` Taylor Blau
2023-06-09 23:33 ` Glen Choo
2023-06-12 21:14 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZGKYOeMGs4fGO5bz@nand.local \
--to=me@ttaylorr.com \
--cc=chris.torek@gmail.com \
--cc=derrickstolee@github.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=peff@peff.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).