From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS22989 209.51.188.0/24 X-Spam-Status: No, score=-3.9 required=3.0 tests=AWL,BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id EEE911F463 for ; Fri, 20 Dec 2019 01:40:22 +0000 (UTC) Received: from localhost ([::1]:49636 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ii7HC-0002w7-36 for e@80x24.org; Thu, 19 Dec 2019 20:40:22 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:37421) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ii7GW-0002lU-5d for dtas-all@nongnu.org; Thu, 19 Dec 2019 20:39:42 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ii7GU-00020w-7h for dtas-all@nongnu.org; Thu, 19 Dec 2019 20:39:40 -0500 Received: from dcvr.yhbt.net ([64.71.152.64]:49682) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ii7GT-0001ot-Rc for dtas-all@nongnu.org; Thu, 19 Dec 2019 20:39:38 -0500 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id D30D21F46D for ; Fri, 20 Dec 2019 01:39:18 +0000 (UTC) From: Eric Wong To: dtas-all@nongnu.org Subject: [PATCH v2 3/5] buffer: replace sleepy_penguin with fiddle Date: Fri, 20 Dec 2019 01:39:15 +0000 Message-Id: <20191220013917.17212-4-e@80x24.org> In-Reply-To: <20191220013917.17212-1-e@80x24.org> References: <20191220013917.17212-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 64.71.152.64 X-BeenThere: dtas-all@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: duct tape audio suite List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dtas-all-bounces+e=80x24.org@nongnu.org Sender: "dtas-all" Fiddle exists on all Ruby 1.9.2+ installations and seems alright. Since splice is a Linux-only API, we don't need to worry about the values of constants changing (and they're architecture-independent). --- lib/dtas/buffer.rb | 15 ++- lib/dtas/buffer/fiddle_splice.rb | 216 +++++++++++++++++++++++++++++++ lib/dtas/buffer/read_write.rb | 4 +- lib/dtas/buffer/splice.rb | 2 + test/test_buffer.rb | 10 +- 5 files changed, 234 insertions(+), 13 deletions(-) create mode 100644 lib/dtas/buffer/fiddle_splice.rb diff --git a/lib/dtas/buffer.rb b/lib/dtas/buffer.rb index 39070d7..e0919d4 100644 --- a/lib/dtas/buffer.rb +++ b/lib/dtas/buffer.rb @@ -8,12 +8,15 @@ class DTAS::Buffer # :nodoc: begin raise LoadError, "no splice with _DTAS_POSIX" if ENV["_DTAS_POSIX"] - require 'sleepy_penguin' # splice is only in Linux for now... - SleepyPenguin.respond_to?(:splice) or - raise LoadError, 'sleepy_penguin 3.5+ required for splice', [] - require_relative 'buffer/splice' - include DTAS::Buffer::Splice - rescue LoadError + # splice is only in Linux for now + begin + require_relative 'buffer/splice' + include DTAS::Buffer::Splice + rescue LoadError + require_relative 'buffer/fiddle_splice' + include DTAS::Buffer::FiddleSplice + end + rescue LoadError, StandardError require_relative 'buffer/read_write' include DTAS::Buffer::ReadWrite end diff --git a/lib/dtas/buffer/fiddle_splice.rb b/lib/dtas/buffer/fiddle_sp= lice.rb new file mode 100644 index 0000000..543b3e0 --- /dev/null +++ b/lib/dtas/buffer/fiddle_splice.rb @@ -0,0 +1,216 @@ +# Copyright (C) 2013-2019 all contributors +# License: GPL-3.0+ +# frozen_string_literal: true +require 'io/nonblock' +require 'fiddle' # require_relative caller should expect LoadError +require_relative '../../dtas' +require_relative '../pipe' + +# Used by -player on Linux systems with the "splice" syscall +module DTAS::Buffer::FiddleSplice # :nodoc: + MAX_AT_ONCE =3D 4096 # page size in Linux + MAX_AT_ONCE_1 =3D 65536 + F_MOVE =3D 1 + F_NONBLOCK =3D 2 + + Splice =3D Fiddle::Function.new(DTAS.libc['splice'], [ + Fiddle::TYPE_INT, # int fd_in, + Fiddle::TYPE_VOIDP, # loff_t *off_in + Fiddle::TYPE_INT, # int fd_out + Fiddle::TYPE_VOIDP, # loff_t *off_out + Fiddle::TYPE_SIZE_T, # size_t len + Fiddle::TYPE_INT, # unsigned int flags + ], + Fiddle::TYPE_SSIZE_T) # ssize_t + + Tee =3D Fiddle::Function.new(DTAS.libc['tee'], [ + Fiddle::TYPE_INT, # int fd_in, + Fiddle::TYPE_INT, # int fd_out + Fiddle::TYPE_SIZE_T, # size_t len + Fiddle::TYPE_INT, # unsigned int flags + ], + Fiddle::TYPE_SSIZE_T) # ssize_t + + def _syserr(s, func) + raise "BUG: we should not encounter EOF on #{func}" if s =3D=3D 0 + case errno =3D Fiddle.last_error + when Errno::EAGAIN::Errno + return :EAGAIN + when Errno::EPIPE::Errno + raise Errno::EPIPE.exception + when Errno::EINTR::Errno + return nil + else + raise SystemCallError, "#{func} error: #{errno}" + end + end + + def splice(src, dst, len, flags) + begin + s =3D Splice.call(src.fileno, nil, dst.fileno, nil, len, flags) + return s if s > 0 + sym =3D _syserr(s, 'splice') and return sym + end while true + end + + def tee(src, dst, len, flags =3D 0) + begin + s =3D Tee.call(src.fileno, dst.fileno, len, flags) + return s if s > 0 + sym =3D _syserr(s, 'tee') and return sym + end while true + end + + def buffer_size + @to_io.pipe_size + end + + # nil is OK, won't reset existing pipe, either... + def buffer_size=3D(bytes) + @to_io.pipe_size =3D bytes if bytes + @buffer_size =3D bytes + end + + # be sure to only call this with nil when all writers to @wr are done + def discard(bytes) + splice(@to_io, DTAS.null, bytes, 0) + end + + def broadcast_one(targets, limit =3D nil) + # single output is always non-blocking + limit ||=3D MAX_AT_ONCE_1 + s =3D splice(@to_io, targets[0], limit, F_MOVE|F_NONBLOCK) + if Symbol =3D=3D=3D s + targets # our one and only target blocked on write + else + @bytes_xfer +=3D s + :wait_readable # we want to read more from @to_io soon + end + rescue Errno::EPIPE, IOError =3D> e + __dst_error(targets[0], e) + targets.clear + nil # do not return error here, we already spewed an error message + end + + def __tee_in_full(src, dst, bytes) + rv =3D 0 + while bytes > 0 + s =3D tee(src, dst, bytes) + bytes -=3D s + rv +=3D s + end + rv + end + + def __splice_in_full(src, dst, bytes, flags) + rv =3D 0 + while bytes > 0 + s =3D splice(src, dst, bytes, flags) + rv +=3D s + bytes -=3D s + end + rv + end + + # returns the largest value we teed + def __broadcast_tee(blocked, targets, chunk_size) + most_teed =3D 0 + targets.delete_if do |dst| + begin + t =3D (dst.nonblock? || most_teed =3D=3D 0) ? + tee(@to_io, dst, chunk_size, F_NONBLOCK) : + __tee_in_full(@to_io, dst, chunk_size) + if Integer =3D=3D=3D t + if t > most_teed + chunk_size =3D t if most_teed =3D=3D 0 + most_teed =3D t + end + else + blocked << dst + end + false + rescue IOError, Errno::EPIPE =3D> e + __dst_error(dst, e) + true + end + end + most_teed + end + + def broadcast_inf(targets, limit =3D nil) + if targets.all?(&:ready_write_optimized?) + blocked =3D [] + elsif targets.none?(&:nonblock?) + # if all targets are blocking, don't start until they're all writa= ble + r =3D IO.select(nil, targets, nil, 0) or return targets + blocked =3D targets - r[1] + + # tell DTAS::UNIXServer#run_once to wait on the blocked targets + return blocked if blocked[0] + + # all writable, yay! + else + blocked =3D [] + end + + # don't pin too much on one target + bytes =3D limit || MAX_AT_ONCE + last =3D targets.pop # we splice to the last one, tee to the rest + + # this may return zero if all targets were non-blocking + most_teed =3D __broadcast_tee(blocked, targets, bytes) + + # don't splice more than the largest amount we successfully teed + bytes =3D most_teed if most_teed > 0 + + begin + targets << last + if last.nonblock? || most_teed =3D=3D 0 + s =3D splice(@to_io, last, bytes, F_MOVE|F_NONBLOCK) + if Symbol =3D=3D=3D s + blocked << last + + # we accomplished nothing! + # If _all_ writers are blocked, do not discard data, + # stay blocked on :wait_writable + return blocked if most_teed =3D=3D 0 + + # the tees targets win, drop data intended for last + if most_teed > 0 + discard(most_teed) + @bytes_xfer +=3D most_teed + # do not watch for writability of last, last is non-blocking + return :wait_readable + end + end + else + # the blocking case is simple + s =3D __splice_in_full(@to_io, last, bytes, F_MOVE) + end + @bytes_xfer +=3D s + + # if we can't splice everything + # discard it so the early targets do not get repeated data + if s < bytes && most_teed > 0 + discard(bytes - s) + end + :wait_readable + rescue IOError, Errno::EPIPE =3D> e # last failed, drop it + __dst_error(last, e) + targets.pop # we're no longer a valid target + + if most_teed =3D=3D 0 + # nothing accomplished, watch any targets + return blocked if blocked[0] + else + # some progress, discard the data we could not splice + @bytes_xfer +=3D most_teed + discard(most_teed) + end + + # stop decoding if we're completely errored out + # returning nil will trigger close + return targets[0] ? :wait_readable : nil + end + end +end diff --git a/lib/dtas/buffer/read_write.rb b/lib/dtas/buffer/read_write.r= b index 04856c7..e2001b6 100644 --- a/lib/dtas/buffer/read_write.rb +++ b/lib/dtas/buffer/read_write.rb @@ -6,8 +6,8 @@ require_relative '../pipe' require_relative '../nonblock' =20 -# compatibility code for systems lacking "splice" support via the -# "sleepy_penguin" 3.5+ RubyGem. Used only by -player +# compatibility code for non-Linux systems lacking "splice" support. +# Used only by -player module DTAS::Buffer::ReadWrite # :nodoc: MAX_AT_ONCE =3D 512 # min PIPE_BUF value in POSIX attr_accessor :buffer_size diff --git a/lib/dtas/buffer/splice.rb b/lib/dtas/buffer/splice.rb index 2e86d0a..b234a57 100644 --- a/lib/dtas/buffer/splice.rb +++ b/lib/dtas/buffer/splice.rb @@ -5,6 +5,8 @@ require 'sleepy_penguin' require_relative '../../dtas' require_relative '../pipe' +SleepyPenguin.respond_to?(:splice) or + raise LoadError, 'sleepy_penguin 3.5+ required for splice', [] =20 # Used by -player on Linux systems with the "sleepy_penguin" RubyGem ins= talled module DTAS::Buffer::Splice # :nodoc: diff --git a/test/test_buffer.rb b/test/test_buffer.rb index 8f5d8b5..1773ca3 100644 --- a/test/test_buffer.rb +++ b/test/test_buffer.rb @@ -49,14 +49,14 @@ def test_set_buffer_size buf =3D new_buffer buf.buffer_size =3D @@max_size assert_equal @@max_size, buf.buffer_size - end if defined?(SleepyPenguin::F_GETPIPE_SZ) + end if defined?(DTAS::Pipe::F_GETPIPE_SZ) =20 def test_buffer_size buf =3D new_buffer assert_operator buf.buffer_size, :>, 128 buf.buffer_size =3D @@max_size assert_equal @@max_size, buf.buffer_size - end if defined?(SleepyPenguin::F_GETPIPE_SZ) + end if defined?(DTAS::Pipe::F_GETPIPE_SZ) =20 def test_broadcast_1 buf =3D new_buffer @@ -108,7 +108,7 @@ def test_broadcast assert_equal "HELLO", a[0].read(5) assert_equal "HELLO", b[0].read(5) =20 - return unless defined?(SleepyPenguin::F_GETPIPE_SZ) + return unless defined?(DTAS::Pipe::F_GETPIPE_SZ) =20 b[1].nonblock =3D true b[1].write('*' * pipe_size(b[1])) @@ -167,7 +167,7 @@ def test_broadcast_all_full buf.wr.write "HELLO" assert_equal tmp, buf.broadcast(tmp) assert_equal [a[1], b[1]], tmp - end if defined?(SleepyPenguin::F_GETPIPE_SZ) + end if defined?(DTAS::Pipe::F_GETPIPE_SZ) =20 def test_serialize buf =3D new_buffer @@ -206,6 +206,6 @@ def test_load_size end =20 def pipe_size(io) - io.fcntl(SleepyPenguin::F_GETPIPE_SZ) + io.fcntl(DTAS::Pipe::F_GETPIPE_SZ) end end