Git Mailing List Archive mirror
 help / color / mirror / Atom feed
From: Jonathan Tan <jonathantanmy@google.com>
To: Junio C Hamano <gitster@pobox.com>
Cc: Jonathan Tan <jonathantanmy@google.com>,
	Derrick Stolee <derrickstolee@github.com>,
	git@vger.kernel.org, me@ttaylorr.com
Subject: Re: [PATCH 2/2] commit-graph: fix murmur3, bump filter ver. to 2
Date: Wed, 24 May 2023 14:26:57 -0700	[thread overview]
Message-ID: <20230524212657.1348621-1-jonathantanmy@google.com> (raw)
In-Reply-To: <xmqqv8gihahz.fsf@gitster.g>

Junio C Hamano <gitster@pobox.com> writes:
> I may be misremembering the original discussion, but wasn't the
> conclusion that v1 data is salvageable in the sense that it can
> still reliably say that, given a pathname without bytes with
> high-bit set, it cannot possibly belong to the set of changed paths,
> even though, because the filter is contaminated with "signed" data,
> its false-positive rate may be higher than using "unsigned" version?
> And based on that observation, we can still read v1 data but only
> use the Bloom filters when the queried paths have no byte with
> high-bit set.

There are at least 3 ways of salvaging the data that we've discussed:

- Enumerating all of a repo's paths and if none of them have a high bit,
  retain the existing filters.
- Walking all of a repo's trees (so that we know which tree corresponds
  to which commit) and if for a commit, all its trees have no high bit,
  retain the filter for that tree (otherwise recompute it).
- Keep using a version 1 filter but only when the sought-for path has no
  high bit (as you describe here).

(The first 2 is my interpretation of what Taylor described [1].)

I'm not sure if we want to keep version 1 filters around at all -
this would work with Git as long as it is not compiled with different
signedness of char, but would not work with other implementations of
Git (unless they replicate the hashing bug). There is also the issue of
how we're going to indicate that in a commit graph file, some filters
are version 1 and some filters are version 2 (unless the plan is to
completely rewrite the filters in this case, but then we'll run into
the issue that computing these filters en-masse is expensive, as Taylor
describes also in [1]).

> Also if we are operating in such an environment then would it be
> possible to first compute as if we were going to generate v2 data,
> but write it as v1 after reading all the path and realizing there
> are no problematic paths?  

I think in this case, we would want to write it as v2 anyway, because
there's no way to distinguish a v1 that has high bits and is written
incorrectly versus a v1 that happens to have no high bits and therefore
is identical under v2.

> IOW, even if the version of Git is
> capable of writing and reading v2, it does not have to use v2,
> right?  To put it the other way around, we will have to and can keep
> supporting data that is labeled as v1, no?

I think this is the main point - whether we want to continue supporting
data labeled as v1. I personally think that we should migrate to v2
as quickly as possible. But if the consensus is that we should support
both, at least for a few releases of Git, I'll go with that.

[1] https://lore.kernel.org/git/ZF116EDcmAy7XEbC@nand.local/

  reply	other threads:[~2023-05-24 21:27 UTC|newest]

Thread overview: 116+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-05-22 21:48 [PATCH 0/2] Changed path filter hash fix and version bump Jonathan Tan
2023-05-22 21:48 ` [PATCH 1/2] t4216: test wrong bloom filter version rejection Jonathan Tan
2023-05-22 21:48 ` [PATCH 2/2] commit-graph: fix murmur3, bump filter ver. to 2 Jonathan Tan
2023-05-23 13:00   ` Derrick Stolee
2023-05-23 23:00     ` Jonathan Tan
2023-05-23 23:51     ` Junio C Hamano
2023-05-24 21:26       ` Jonathan Tan [this message]
2023-05-26 13:19         ` Derrick Stolee
2023-05-30 17:26           ` Jonathan Tan
2023-05-23  4:42 ` [PATCH 0/2] Changed path filter hash fix and version bump Junio C Hamano
2023-05-31 23:12 ` [PATCH v2 0/3] " Jonathan Tan
2023-05-31 23:12   ` [PATCH v2 1/3] t4216: test changed path filters with high bit paths Jonathan Tan
2023-05-31 23:12   ` [PATCH v2 2/3] repo-settings: introduce commitgraph.changedPathsVersion Jonathan Tan
2023-05-31 23:12   ` [PATCH v2 3/3] commit-graph: new filter ver. that fixes murmur3 Jonathan Tan
2023-06-03  1:01   ` [PATCH v2 0/3] Changed path filter hash fix and version bump Junio C Hamano
2023-06-03  2:24     ` Junio C Hamano
2023-06-07 16:30       ` Jonathan Tan
2023-06-07 21:37         ` Jonathan Tan
2023-06-08 19:21 ` [PATCH v3 0/4] " Jonathan Tan
2023-06-08 19:21   ` [PATCH v3 1/4] gitformat-commit-graph: describe version 2 of BDAT Jonathan Tan
2023-06-08 19:52     ` Ramsay Jones
2023-06-12 21:26       ` Junio C Hamano
2023-06-08 19:21   ` [PATCH v3 2/4] t4216: test changed path filters with high bit paths Jonathan Tan
2023-06-08 19:21   ` [PATCH v3 3/4] repo-settings: introduce commitgraph.changedPathsVersion Jonathan Tan
2023-06-08 19:21   ` [PATCH v3 4/4] commit-graph: new filter ver. that fixes murmur3 Jonathan Tan
2023-06-08 19:50   ` [PATCH v3 0/4] Changed path filter hash fix and version bump Ramsay Jones
2023-06-09  0:08     ` Jonathan Tan
2023-06-12 21:31     ` Junio C Hamano
2023-06-13 17:16       ` Jonathan Tan
2023-06-13 17:29         ` [PATCH] CodingGuidelines: use octal escapes, not hex Jonathan Tan
2023-06-13 18:16           ` Eric Sunshine
2023-06-13 18:43             ` Jonathan Tan
2023-06-13 19:15               ` Eric Sunshine
2023-06-13 19:29                 ` Junio C Hamano
2023-06-13 19:16         ` [PATCH v3 0/4] Changed path filter hash fix and version bump Junio C Hamano
2023-06-13 17:39 ` [PATCH v4 " Jonathan Tan
2023-06-13 17:39   ` [PATCH v4 1/4] gitformat-commit-graph: describe version 2 of BDAT Jonathan Tan
2023-06-13 21:58     ` Junio C Hamano
2023-06-20 13:22       ` Derrick Stolee
2023-06-21 12:08       ` Taylor Blau
2023-06-22 22:26         ` Jonathan Tan
2023-06-23 13:05           ` Derrick Stolee
2023-06-13 17:39   ` [PATCH v4 2/4] t4216: test changed path filters with high bit paths Jonathan Tan
2023-06-13 17:39   ` [PATCH v4 3/4] repo-settings: introduce commitgraph.changedPathsVersion Jonathan Tan
2023-06-20 13:28     ` Derrick Stolee
2023-06-21 12:14     ` Taylor Blau
2023-06-13 17:39   ` [PATCH v4 4/4] commit-graph: new filter ver. that fixes murmur3 Jonathan Tan
2023-06-20 13:39     ` Derrick Stolee
2023-06-20 18:37       ` Junio C Hamano
2023-06-13 19:21   ` [PATCH v4 0/4] Changed path filter hash fix and version bump Junio C Hamano
2023-06-20 13:43     ` Derrick Stolee
2023-06-20 21:56       ` Jonathan Tan
2023-06-21 12:19       ` Taylor Blau
2023-06-21 17:53         ` Derrick Stolee
2023-06-22 22:27           ` Jonathan Tan
2023-06-23 13:18             ` Derrick Stolee
2023-07-13 21:42 ` [PATCH v5 " Jonathan Tan
2023-07-13 21:42   ` [PATCH v5 1/4] gitformat-commit-graph: describe version 2 of BDAT Jonathan Tan
2023-07-19 17:25     ` Taylor Blau
2023-07-20 20:20       ` Jonathan Tan
2023-07-21  1:38         ` Taylor Blau
2023-07-13 21:42   ` [PATCH v5 2/4] t4216: test changed path filters with high bit paths Jonathan Tan
2023-07-13 22:50     ` Junio C Hamano
2023-07-19 17:27       ` Taylor Blau
2023-07-19 17:55         ` [PATCH 0/4] commit-graph: avoid looking at Bloom filter data directly Taylor Blau
2023-07-19 17:55           ` [PATCH 1/4] t/helper/test-read-graph.c: extract `dump_graph_info()` Taylor Blau
2023-07-19 17:55           ` [PATCH 2/4] bloom.h: make `load_bloom_filter_from_graph()` public Taylor Blau
2023-07-19 17:55           ` [PATCH 3/4] t/helper/test-read-graph: implement `bloom-filters` mode Taylor Blau
2023-07-19 17:55           ` [PATCH 4/4] fixup! t4216: test changed path filters with high bit paths Taylor Blau
2023-07-19 19:24           ` [PATCH 0/4] commit-graph: avoid looking at Bloom filter data directly Junio C Hamano
2023-07-20 20:22           ` Jonathan Tan
2023-07-13 21:42   ` [PATCH v5 3/4] repo-settings: introduce commitgraph.changedPathsVersion Jonathan Tan
2023-07-19 18:10     ` Taylor Blau
2023-07-20 20:42       ` Jonathan Tan
2023-07-20 21:02         ` Taylor Blau
2023-07-13 21:42   ` [PATCH v5 4/4] commit-graph: new filter ver. that fixes murmur3 Jonathan Tan
2023-07-19 18:24     ` Taylor Blau
2023-07-20 21:27       ` Jonathan Tan
2023-07-26 23:32         ` Taylor Blau
2023-07-13 22:16   ` [PATCH v5 0/4] Changed path filter hash fix and version bump Junio C Hamano
2023-07-13 22:59     ` Junio C Hamano
2023-07-14 18:48     ` Jonathan Tan
2023-07-20 21:46 ` [PATCH v6 0/7] " Jonathan Tan
2023-07-20 21:46   ` [PATCH v6 1/7] gitformat-commit-graph: describe version 2 of BDAT Jonathan Tan
2023-07-20 21:46   ` [PATCH v6 2/7] t/helper/test-read-graph.c: extract `dump_graph_info()` Jonathan Tan
2023-07-26 23:26     ` Taylor Blau
2023-07-20 21:46   ` [PATCH v6 3/7] bloom.h: make `load_bloom_filter_from_graph()` public Jonathan Tan
2023-07-20 21:46   ` [PATCH v6 4/7] t/helper/test-read-graph: implement `bloom-filters` mode Jonathan Tan
2023-07-20 21:46   ` [PATCH v6 5/7] t4216: test changed path filters with high bit paths Jonathan Tan
2023-07-26 23:28     ` Taylor Blau
2023-07-20 21:46   ` [PATCH v6 6/7] repo-settings: introduce commitgraph.changedPathsVersion Jonathan Tan
2023-07-20 21:46   ` [PATCH v6 7/7] commit-graph: new filter ver. that fixes murmur3 Jonathan Tan
2023-07-25 20:52   ` [PATCH v6 0/7] Changed path filter hash fix and version bump Junio C Hamano
2023-07-26 20:39   ` Junio C Hamano
2023-07-27  0:17     ` Taylor Blau
2023-07-27  0:49       ` Junio C Hamano
2023-07-27 17:39         ` Jonathan Tan
2023-07-27 17:56           ` Taylor Blau
2023-07-27 20:53             ` Jonathan Tan
2023-08-01 18:08               ` Taylor Blau
2023-08-01 18:52                 ` Jonathan Tan
2023-08-03  0:01                 ` Taylor Blau
2023-08-03 13:18                   ` Derrick Stolee
2023-08-03 18:45                     ` Taylor Blau
2023-07-27 18:44         ` Junio C Hamano
2023-08-01 18:41 ` [PATCH v7 " Jonathan Tan
2023-08-01 18:41   ` [PATCH v7 1/7] gitformat-commit-graph: describe version 2 of BDAT Jonathan Tan
2023-08-01 18:41   ` [PATCH v7 2/7] t/helper/test-read-graph.c: extract `dump_graph_info()` Jonathan Tan
2023-08-01 18:41   ` [PATCH v7 3/7] bloom.h: make `load_bloom_filter_from_graph()` public Jonathan Tan
2023-08-01 18:41   ` [PATCH v7 4/7] t/helper/test-read-graph: implement `bloom-filters` mode Jonathan Tan
2023-08-01 18:41   ` [PATCH v7 5/7] t4216: test changed path filters with high bit paths Jonathan Tan
2023-08-01 18:41   ` [PATCH v7 6/7] repo-settings: introduce commitgraph.changedPathsVersion Jonathan Tan
2023-08-01 18:41   ` [PATCH v7 7/7] commit-graph: new filter ver. that fixes murmur3 Jonathan Tan
2023-08-01 18:44   ` [PATCH v7 0/7] Changed path filter hash fix and version bump Junio C Hamano
2023-08-01 20:58     ` Taylor Blau
2023-08-01 21:07       ` Junio C Hamano

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20230524212657.1348621-1-jonathantanmy@google.com \
    --to=jonathantanmy@google.com \
    --cc=derrickstolee@github.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=me@ttaylorr.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).