From: John Cai <johncai86@gmail.com>
To: Taylor Blau <me@ttaylorr.com>
Cc: John Cai via GitGitGadget <gitgitgadget@gmail.com>,
git@vger.kernel.org, Junio C Hamano <gitster@pobox.com>,
Christian Couder <christian.couder@gmail.com>
Subject: Re: [PATCH v3 2/4] pack-refs: teach --exclude option to exclude refs from being packed
Date: Fri, 12 May 2023 17:11:34 -0400 [thread overview]
Message-ID: <5D8B0ED5-F122-4E96-9ED4-5D21C0A392A1@gmail.com> (raw)
In-Reply-To: <7664500D-8C64-46DD-8567-F3AAB532B7EF@gmail.com>
On 12 May 2023, at 8:53, John Cai wrote:
> Hey Taylor,
>
> On 11 May 2023, at 20:00, Taylor Blau wrote:
>
>> On Thu, May 11, 2023 at 06:10:32PM +0000, John Cai via GitGitGadget wrote:
>>> From: John Cai <johncai86@gmail.com>
>>>
>>> At GitLab, we have a system that creates ephemeral internal refs that
>>> don't live long before getting deleted. Having an option to exclude
>>> certain refs from a packed-refs file allows these internal references to
>>> be deleted much more efficiently.
>>>
>>> Add an --exclude option to the pack-refs builtin, and use the ref
>>> exclusions API to exclude certain refs from being packed into the final
>>> packed-refs file
>>>
>>> Signed-off-by: John Cai <johncai86@gmail.com>
>>> ---
>>> Documentation/git-pack-refs.txt | 12 +++++++++++-
>>> builtin/pack-refs.c | 20 ++++++++++++++++----
>>> refs.c | 4 ++--
>>> refs.h | 7 ++++++-
>>> refs/debug.c | 4 ++--
>>> refs/files-backend.c | 16 ++++++++++------
>>> refs/packed-backend.c | 2 +-
>>> refs/refs-internal.h | 3 ++-
>>> revision.h | 2 +-
>>> t/helper/test-ref-store.c | 3 ++-
>>> t/t3210-pack-refs.sh | 16 ++++++++++++++++
>>> 11 files changed, 69 insertions(+), 20 deletions(-)
>>>
>>> diff --git a/Documentation/git-pack-refs.txt b/Documentation/git-pack-refs.txt
>>> index e011e5fead3..c0f7426e519 100644
>>> --- a/Documentation/git-pack-refs.txt
>>> +++ b/Documentation/git-pack-refs.txt
>>> @@ -8,7 +8,7 @@ git-pack-refs - Pack heads and tags for efficient repository access
>>> SYNOPSIS
>>> --------
>>> [verse]
>>> -'git pack-refs' [--all] [--no-prune]
>>> +'git pack-refs' [--all] [--no-prune] [--exclude <pattern>]
>>>
>>> DESCRIPTION
>>> -----------
>>> @@ -60,6 +60,16 @@ interests.
>>> The command usually removes loose refs under `$GIT_DIR/refs`
>>> hierarchy after packing them. This option tells it not to.
>>>
>>> +--exclude <pattern>::
>>> +
>>> +Do not pack refs matching the given `glob(7)` pattern. Repetitions of this option
>>> +accumulate exclusion patterns. Use `--no-exclude` to clear and reset the list of
>>> +patterns. If a ref is already packed, including it with `--exclude` will not
>>> +unpack it.
>>> +
>>> +When used with `--all`, it will use the difference between the set of all refs,
>>> +and what is provided to `--exclude`.
>>> +
>>
>> I think this last paragraph could be simplified, though feel free to
>> discard my suggestion if you think it makes things less clear.
>>
>> When used with `--all`, pack only loose refs which do not match any of
>> the provided `--exclude` patterns.
>
> I like the wording here, thanks
>
>>
>>> int cmd_pack_refs(int argc, const char **argv, const char *prefix)
>>> {
>>> unsigned int flags = PACK_REFS_PRUNE;
>>> + static struct ref_exclusions excludes = REF_EXCLUSIONS_INIT;
>>> + struct pack_refs_opts pack_refs_opts = {.exclusions = &excludes, .flags = flags};
>>> + static struct string_list option_excluded_refs = STRING_LIST_INIT_NODUP;
>>> + struct string_list_item *item;
>>
>> Since this list does not appear to be sensitive to its order, have you
>> considered using the strvec API instead of the string_list one?
>
> Thanks for this suggestion--you're right in that the order doesn't matter here.
> The only thing is, the only option parsing macro I could find that operates on
> strvec is OPT_PASSTHRU_ARGV. I tried it out, and it seem to work just fine.
Actually I had a bug in my test that made it seem like it worked. I realize that
OPT_PASSTHRU_ARGV passes through the entire option flag eg
--include=refs/tags/dont_pack* so we would have to strip the flag name. It's not
as user friendly as OPT_STRING_LIST that is just plug and play.
Unless there is a significant performance improvement in using strvec, I'm
thinking maybe I'll just stick to string_list.
thanks
John
>
>>
>>> diff --git a/refs.c b/refs.c
>>> index d2a98e1c21f..881a0da65cf 100644
>>> --- a/refs.c
>>> +++ b/refs.c
>>> @@ -2132,9 +2132,9 @@ void base_ref_store_init(struct ref_store *refs, struct repository *repo,
>>> }
>>>
>>> /* backend functions */
>>> -int refs_pack_refs(struct ref_store *refs, unsigned int flags)
>>> +int refs_pack_refs(struct ref_store *refs, struct pack_refs_opts *opts)
>>> {
>>> - return refs->be->pack_refs(refs, flags);
>>> + return refs->be->pack_refs(refs, opts);
>>> }
>>>
>>> int peel_iterated_oid(const struct object_id *base, struct object_id *peeled)
>>> diff --git a/refs.h b/refs.h
>>> index 123cfa44244..46020bd335c 100644
>>> --- a/refs.h
>>> +++ b/refs.h
>>> @@ -63,6 +63,11 @@ struct worktree;
>>> #define RESOLVE_REF_NO_RECURSE 0x02
>>> #define RESOLVE_REF_ALLOW_BAD_NAME 0x04
>>>
>>> +struct pack_refs_opts {
>>> + unsigned int flags;
>>> + struct ref_exclusions *exclusions;
>>
>> I think this would be OK to include directly in the struct instead of
>> via a pointer, but either is fine.
>>
>>> @@ -1175,15 +1176,18 @@ static void prune_refs(struct files_ref_store *refs, struct ref_to_prune **refs_
>>> */
>>> static int should_pack_ref(const char *refname,
>>> const struct object_id *oid, unsigned int ref_flags,
>>> - unsigned int pack_flags)
>>> + struct pack_refs_opts *opts)
>>> {
>>> /* Do not pack per-worktree refs: */
>>> if (parse_worktree_ref(refname, NULL, NULL, NULL) !=
>>> REF_WORKTREE_SHARED)
>>> return 0;
>>>
>>> + if (opts->exclusions && ref_excluded(opts->exclusions, refname))
>>> + return 0;
>>
>> Looks good, here is where we throw out refs that we don't want. I wonder
>> if ref_excluded() does the right thing with a zero-initialized argument
>> (i.e. that it behaves as if nothing matches).
>
> Yes, I think we can skip checking if opt->exclusions is not null. Junio had
> feedback around this as well.
>
>>
>> I wonder if it's possible to skip over certain loose references by
>> avoiding traversal into the sub-directories for simple prefixes. That
>> may be a premature optimization, though, so I don't think you
>> necessarily need to worry about it in this round.
>>
>>> +test_expect_success 'test excluded refs are not packed' '
>>> + git branch dont_pack1 &&
>>> + git branch dont_pack2 &&
>>> + git branch pack_this &&
>>> + git pack-refs --all --exclude "refs/heads/dont_pack*" &&
>>> + test -f .git/refs/heads/dont_pack1 &&
>>> + test -f .git/refs/heads/dont_pack2 &&
>>> + ! test -f ./git/refs/heads/pack_this'
>>> +
>>> +test_expect_success 'test --no-exclude refs clears excluded refs' '
>>> + git branch dont_pack3 &&
>>> + git branch dont_pack4 &&
>>> + git pack-refs --all --exclude "refs/heads/dont_pack*" --no-exclude &&
>>> + ! test -f .git/refs/heads/dont_pack3 &&
>>> + ! test -f .git/refs/heads/dont_pack4'
>>
>> Tests look good. The trailing quote is a little odd to be placed on the
>> last line of the test instead of off on its own, but I suppose that is
>> imitating existing style, which is OK.
>
> thanks for the feedback!
> John
>>
>> Thanks,
>> Taylor
next prev parent reply other threads:[~2023-05-12 21:14 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-04 15:48 [PATCH] pack-refs: teach --exclude option to exclude refs from being packed John Cai via GitGitGadget
2023-05-04 16:48 ` Junio C Hamano
2023-05-04 21:26 ` John Cai
2023-05-09 19:18 ` [PATCH v2 0/3] pack-refs: Teach " John Cai via GitGitGadget
2023-05-09 19:18 ` [PATCH v2 1/3] docs: clarify git-pack-refs --all will pack all refs John Cai via GitGitGadget
2023-05-09 19:18 ` [PATCH v2 2/3] pack-refs: teach --exclude option to exclude refs from being packed John Cai via GitGitGadget
2023-05-09 21:04 ` Junio C Hamano
2023-05-09 19:18 ` [PATCH v2 3/3] pack-refs: teach pack-refs --include option John Cai via GitGitGadget
2023-05-09 21:25 ` Junio C Hamano
2023-05-10 19:52 ` John Cai
2023-05-11 18:10 ` [PATCH v3 0/4] pack-refs: allow users control over which refs to pack John Cai via GitGitGadget
2023-05-11 18:10 ` [PATCH v3 1/4] docs: clarify git-pack-refs --all will pack all refs John Cai via GitGitGadget
2023-05-11 23:53 ` Taylor Blau
2023-05-11 18:10 ` [PATCH v3 2/4] pack-refs: teach --exclude option to exclude refs from being packed John Cai via GitGitGadget
2023-05-11 19:34 ` Junio C Hamano
2023-05-12 0:00 ` Taylor Blau
2023-05-12 12:53 ` John Cai
2023-05-12 21:11 ` John Cai [this message]
2023-05-11 18:10 ` [PATCH v3 3/4] revision: modify ref_exclusions to handle inclusions John Cai via GitGitGadget
2023-05-11 19:54 ` Junio C Hamano
2023-05-12 14:56 ` John Cai
2023-05-11 18:10 ` [PATCH v3 4/4] pack-refs: teach pack-refs --include option John Cai via GitGitGadget
2023-05-11 20:06 ` Junio C Hamano
2023-05-12 14:48 ` John Cai
2023-05-12 19:03 ` John Cai
2023-05-12 21:34 ` [PATCH v4 0/3] pack-refs: allow users control over which refs to pack John Cai via GitGitGadget
2023-05-12 21:34 ` [PATCH v4 1/3] docs: clarify git-pack-refs --all will pack all refs John Cai via GitGitGadget
2023-05-12 21:34 ` [PATCH v4 2/3] pack-refs: teach --exclude option to exclude refs from being packed John Cai via GitGitGadget
2023-05-12 21:34 ` [PATCH v4 3/3] pack-refs: teach pack-refs --include option John Cai via GitGitGadget
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5D8B0ED5-F122-4E96-9ED4-5D21C0A392A1@gmail.com \
--to=johncai86@gmail.com \
--cc=christian.couder@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitgitgadget@gmail.com \
--cc=gitster@pobox.com \
--cc=me@ttaylorr.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).