From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on dcvr.yhbt.net X-Spam-Flag: YES X-Spam-Level: **** X-Spam-ASN: AS206264 45.139.122.0/24 X-Spam-Status: Yes, score=4.7 required=3.0 tests=AWL,BAYES_00, RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_MSPIKE_BL,RCVD_IN_MSPIKE_ZBI, RCVD_IN_SBL_CSS,RCVD_IN_XBL,RDNS_NONE,SPF_FAIL,SPF_HELO_FAIL shortcircuit=no autolearn=no autolearn_force=no version=3.4.6 X-Spam-Report: * 1.3 RCVD_IN_BL_SPAMCOP_NET RBL: Received via a relay in * bl.spamcop.net * [Blocked - see ] * 3.3 RCVD_IN_SBL_CSS RBL: Received via a relay in Spamhaus SBL-CSS * [45.139.122.241 listed in zen.spamhaus.org] * 0.4 RCVD_IN_XBL RBL: Received via a relay in Spamhaus XBL * -3.0 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 SPF_HELO_FAIL SPF: HELO does not match SPF record (fail) * [SPF failed: Rejected by SPF record] * 0.0 SPF_FAIL SPF: sender does not match SPF record (fail) * [SPF failed: Rejected by SPF record] * 0.8 RDNS_NONE Delivered to internal network by a host with no rDNS * 0.0 RCVD_IN_MSPIKE_BL Mailspike blocklisted * 2.7 RCVD_IN_MSPIKE_ZBI No description available. * -0.9 AWL AWL: Adjusted score from AWL reputation of From: address Received: from 80x24.org (unknown [45.139.122.241]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id DA1451F43D for ; Wed, 15 Mar 2023 10:12:41 +0000 (UTC) From: Eric Wong To: spew@80x24.org Subject: [PATCH 10/10] cindex: read-only xdb shards fixup Date: Wed, 15 Mar 2023 10:11:05 +0000 Message-Id: <20230315101105.3651530-10-e@80x24.org> In-Reply-To: <20230315101105.3651530-1-e@80x24.org> References: <20230315101105.3651530-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: --- lib/PublicInbox/CodeSearchIdx.pm | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/lib/PublicInbox/CodeSearchIdx.pm b/lib/PublicInbox/CodeSearchIdx.pm index 105c76d2..02c9ed84 100644 --- a/lib/PublicInbox/CodeSearchIdx.pm +++ b/lib/PublicInbox/CodeSearchIdx.pm @@ -275,6 +275,7 @@ sub prep_repo ($$) { my $shard = $repo->{shard} = bless { %$self, shard => $n }, ref($self); delete @$shard{qw(lockfh lock_path)}; my $xdb = $XDB_SHARDS_FLAT[$n] // die "BUG: shard[$n] undef"; + $xdb->reopen; my @docids = docids_by_postlist({ xdb => $xdb }, 'P'.$git_dir); my $docid = shift(@docids) // return get_roots($self, $git); if (@docids) { @@ -298,14 +299,16 @@ sub partition_refs ($$$) { sysseek($refs, 0, SEEK_SET) or die "seek: $!"; # for rev-list --stdin my $fh = $git->popen(qw(rev-list --stdin), undef, { 0 => $refs }); close $refs or die "close: $!"; - my ($seen, $nchange, $nshard) = (0, 0, $self->{nshard}); - my @shard_in; - for (0..($nshard - 1)) { - open $shard_in[$_], '+>', undef or die "open: $!"; - } + my ($seen, $nchange) = (0, 0); + my @shard_in = map { + $_->reopen; + open my $fh, '+>', undef or die "open: $!"; + $fh; + } @XDB_SHARDS_FLAT; + while (defined(my $cmt = <$fh>)) { chomp $cmt; - my $n = hex(substr($cmt, 0, 8)) % $nshard; + my $n = hex(substr($cmt, 0, 8)) % scalar(@XDB_SHARDS_FLAT); if (seen($XDB_SHARDS_FLAT[$n], 'Q'.$cmt)) { last if ++$seen > $SEEN_MAX; } else {