From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.2 required=3.0 tests=ALL_TRUSTED,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id F0EB01F55F for ; Sun, 1 Oct 2023 02:43:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=80x24.org; s=selector1; t=1696128204; bh=pw6PjGJOm0NfQmRjyS6dhRf74/vn6m+7PTPIN1X5YlU=; h=From:To:Subject:Date:From; b=jhWczOpVWwRHeFK9jH7cWYHg9sCWjMPsccZnf4fBA9N9WNTXjMMV2KfFYQfbT9yvm n3Q0yEl+vOB9HT6X6oTx9aohYmy+h/iGaOGB1FiT/tePdzXqQ85xs8NCBmM+6JyDmI HAbUv5ANDEWhZIf7+V7xUkrl64a25VQSXcgLOr1w= From: Eric Wong To: spew@80x24.org Subject: [PATCH 01/16] git: decouple cat_async_retry from POSIX pipe semantics Date: Sun, 1 Oct 2023 02:43:08 +0000 Message-ID: <20231001024323.1960491-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: While pipes guarantee writes of <= 512 bytes to be atomic, Unix stream sockets (or TCP sockets) have no such guarantees. Removing the pipe assumption will make it possible for us to switch to bidirectional Unix stream sockets and save FDs with `git cat-file' processes as we have with Gcf2Client. The performance benefit of larger pipe buffers over stream sockets isn't irrelevant when interacting with git as it is with SearchIdx shards. --- lib/PublicInbox/Git.pm | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-) diff --git a/lib/PublicInbox/Git.pm b/lib/PublicInbox/Git.pm index eb88aa48..8ac40d2b 100644 --- a/lib/PublicInbox/Git.pm +++ b/lib/PublicInbox/Git.pm @@ -223,25 +223,21 @@ sub my_readline ($$) { } sub cat_async_retry ($$) { - my ($self, $inflight) = @_; + my ($self, $old_inflight) = @_; # {inflight} may be non-existent, but if it isn't we delete it # here to prevent cleanup() from waiting: delete $self->{inflight}; cleanup($self); + batch_prepare($self, my $new_inflight = []); - batch_prepare($self, $inflight); - my $buf = ''; - for (my $i = 0; $i < @$inflight; $i += 3) { - $buf .= "$inflight->[$i]\n"; + while (my ($oid, $cb, $arg) = splice(@$old_inflight, 0, 3)) { + write_all($self, $self->{out}, $oid."\n", + \&cat_async_step, $new_inflight); + $oid = \$oid if !@$new_inflight; # to indicate oid retried + push @$new_inflight, $oid, $cb, $arg; } - $self->{out}->blocking(1); # brand new pipe, should never block - print { $self->{out} } $buf or $self->fail("write error: $!"); - $self->{out}->blocking(0); - my $req = shift @$inflight; - unshift(@$inflight, \$req); # \$ref to indicate retried - - cat_async_step($self, $inflight); # take one step + cat_async_step($self, $new_inflight); # take one step } # returns true if prefetch is successful