From: Jeff King <peff@peff.net>
To: Teng Long <dyroneteng@gmail.com>
Cc: avarab@gmail.com, derrickstolee@github.com, git@vger.kernel.org,
gitster@pobox.com, me@ttaylorr.com, tenglong.tl@alibaba-inc.com
Subject: Re: [PATCH v3 1/2] pack-bitmap.c: remove unnecessary "open_pack_index()" calls
Date: Mon, 14 Nov 2022 17:03:56 -0500 [thread overview]
Message-ID: <Y3K7TEpB8EzczjTb@coredump.intra.peff.net> (raw)
In-Reply-To: <aaeb021538cdfeb83dc6004fe7b3ac35a23aef49.1668063122.git.dyroneteng@gmail.com>
On Thu, Nov 10, 2022 at 03:10:11PM +0800, Teng Long wrote:
> It's likely that we'll end up relying on objects in that pack later
> in the process (in which case we're doing the work of opening the
> pack index optimistically), but not guaranteed.
>
> For instance, consider a repository with a large number of small
> packs, and one large pack with a bitmap. If we see that bitmap pack
> last in our loop which calls open_pack_bitmap_1(), the current code
> will have opened *all* pack index files in the repository. If the
> request can be served out of the bitmapped pack alone, then the time
> spent opening these idx files was wasted.S
By the way, I wondered if it was possible to measure a slowdown in this
case. It is, but you have to try pretty hard. Something like this:
# one bitmapped pack
git repack -adb
# and then a bunch of other packs
git rev-list HEAD |
head -10000 |
while read commit; do
echo $commit | git pack-objects .git/objects/pack/pack
done
# make the bitmapped one newest, since otherwise our non-bitmap lookup
# of the initial traversal commit causes us to open all the other
# packs first!
bitmap=$(echo .git/objects/pack/pack-*.bitmap)
touch ${bitmap%.bitmap}.*
hyperfine -L v old,new './git.{v} rev-list --count --use-bitmap-index HEAD'
where "new" and "old" are builds with and without this patch. I get:
Benchmark 1: ./git.old rev-list --count --use-bitmap-index HEAD
Time (mean ± σ): 117.9 ms ± 1.8 ms [User: 26.9 ms, System: 90.0 ms]
Range (min … max): 113.4 ms … 120.3 ms 25 runs
Benchmark 2: ./git.new rev-list --count --use-bitmap-index HEAD
Time (mean ± σ): 71.8 ms ± 2.6 ms [User: 21.2 ms, System: 50.5 ms]
Range (min … max): 67.0 ms … 75.1 ms 41 runs
Summary
'./git.new rev-list --count --use-bitmap-index HEAD' ran
1.64 ± 0.06 times faster than './git.old rev-list --count --use-bitmap-index HEAD'
which implies to me two things:
- this probably isn't helping anybody much in the real world, as
evidenced by the contortions I had to go through to set up the
situation (and which would be made much better by repacking, which
would also speed up non-bitmap operations).
- it's worth doing anyway. Even if it only shaves off microseconds,
the existing call is just pointless.
-Peff
next prev parent reply other threads:[~2022-11-14 22:04 UTC|newest]
Thread overview: 67+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-08-26 7:09 [PATCH 0/1] pack-bitmap.c: avoid exposing absolute paths Teng Long
2022-08-26 7:09 ` [PATCH 1/1] " Teng Long
2022-08-26 16:34 ` [PATCH 0/1] " Junio C Hamano
2022-08-29 2:48 ` Teng Long
2022-10-26 21:42 ` Taylor Blau
2022-10-26 23:19 ` Ævar Arnfjörð Bjarmason
2022-10-31 13:20 ` Teng Long
2022-10-27 20:45 ` Jeff King
2022-10-30 18:42 ` Taylor Blau
2022-10-31 12:22 ` [PATCH 0/1] pack-bitmap.c: avoid exposing absolute paths Taylor Blau <me@ttaylorr.com> writes: Teng Long
2022-11-02 5:37 ` [PATCH 0/1] pack-bitmap.c: avoid exposing absolute paths Teng Long
2022-11-02 7:54 ` Jeff King
2022-11-02 13:52 ` Teng Long
2022-10-31 13:13 ` Teng Long
2022-11-03 1:00 ` Taylor Blau
2022-11-02 9:20 ` Ævar Arnfjörð Bjarmason
2022-11-02 13:04 ` Teng Long
2022-11-02 12:56 ` [PATCH v2 " Teng Long
2022-11-02 12:56 ` [PATCH v2 1/1] " Teng Long
2022-11-03 1:16 ` Taylor Blau
2022-11-03 9:35 ` Teng Long
2022-11-05 0:35 ` Taylor Blau
2022-11-03 1:21 ` [PATCH v2 0/1] " Taylor Blau
2022-11-03 8:42 ` Teng Long
2022-11-04 3:17 ` [PATCH v3 0/2] " Teng Long
2022-11-04 3:17 ` [PATCH v3 1/2] " Teng Long
2022-11-04 22:11 ` Taylor Blau
2022-11-04 3:17 ` [PATCH v3 2/2] pack-bitmap.c: remove unnecessary "open_pack_index()" calls Teng Long
2022-11-04 22:09 ` Taylor Blau
2022-11-04 22:13 ` [PATCH v3 0/2] pack-bitmap.c: avoid exposing absolute paths Taylor Blau
2022-11-10 7:10 ` Teng Long
2022-11-10 7:10 ` [PATCH v3 1/2] pack-bitmap.c: remove unnecessary "open_pack_index()" calls Teng Long
2022-11-14 22:03 ` Jeff King [this message]
2022-11-14 22:14 ` Taylor Blau
2022-11-14 22:31 ` Jeff King
2022-11-14 22:50 ` Taylor Blau
2022-11-10 7:10 ` [PATCH v3 2/2] pack-bitmap.c: avoid exposing absolute paths Teng Long
2022-11-11 22:26 ` [PATCH v3 0/2] " Taylor Blau
2022-11-14 22:23 ` Jeff King
2022-11-17 14:19 ` Teng Long
2022-11-17 15:03 ` Jeff King
2022-11-17 21:57 ` Taylor Blau
2022-11-21 3:27 ` Teng Long
2022-11-21 12:16 ` [PATCH v4 0/4] " Teng Long
2022-11-21 12:16 ` [PATCH v4 1/4] pack-bitmap.c: remove unnecessary "open_pack_index()" calls Teng Long
2022-11-21 12:16 ` [PATCH v4 2/4] pack-bitmap.c: avoid exposing absolute paths Teng Long
2022-11-21 12:16 ` [PATCH v4 3/4] pack-bitmap.c: break out of the bitmap loop early if not tracing Teng Long
2022-11-21 23:27 ` Junio C Hamano
2022-11-28 13:09 ` Teng Long
2022-11-21 12:16 ` [PATCH v4 4/4] pack-bitmap.c: trace bitmap ignore logs when midx-bitmap is found Teng Long
2022-11-21 19:09 ` Jeff King
2022-11-21 23:29 ` Junio C Hamano
2022-11-28 12:29 ` Teng Long
2022-11-28 12:37 ` Teng Long
2022-11-29 1:27 ` Jeff King
2022-11-29 13:14 ` Teng Long
2022-11-21 19:04 ` [PATCH v4 0/4] pack-bitmap.c: avoid exposing absolute paths Jeff King
2022-11-28 12:48 ` Teng Long
2022-11-28 14:09 ` [PATCH v5 " Teng Long
2022-11-28 14:09 ` [PATCH v5 1/4] pack-bitmap.c: remove unnecessary "open_pack_index()" calls Teng Long
2022-11-28 14:09 ` [PATCH v5 2/4] pack-bitmap.c: avoid exposing absolute paths Teng Long
2022-11-28 14:09 ` [PATCH v5 3/4] pack-bitmap.c: break out of the bitmap loop early if not tracing Teng Long
2022-11-28 23:26 ` Taylor Blau
2022-11-29 13:17 ` Teng Long
2022-11-28 14:09 ` [PATCH v5 4/4] pack-bitmap.c: trace bitmap ignore logs when midx-bitmap is found Teng Long
2022-11-28 23:30 ` [PATCH v5 0/4] pack-bitmap.c: avoid exposing absolute paths Taylor Blau
2022-11-29 13:21 ` Teng Long
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y3K7TEpB8EzczjTb@coredump.intra.peff.net \
--to=peff@peff.net \
--cc=avarab@gmail.com \
--cc=derrickstolee@github.com \
--cc=dyroneteng@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=me@ttaylorr.com \
--cc=tenglong.tl@alibaba-inc.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).