[PATCH v2 0/5] support fiddle for Linux-only syscalls

[PATCH v2 0/5] support fiddle for Linux-only syscalls

[Eric Wong]

fiddle can be used as an alternative to sleepy_penguin, since
sleepy_penguin requires development headers and a compiler to
install, and fiddle is bundled as part of Ruby.  sleepy_penguin
can still be faster, though, so we continue supporting
sleepy_penguin.

Eric Wong (5):
  pipe: avoid loading sleepy_penguin
  provide fiddle-based eventfd implementation
  buffer: replace sleepy_penguin with fiddle
  watchable: use fiddle for inotify support
  doc: remove most recommendations for sleepy_penguin

 Documentation/dtas-sinkedit.pod   |   2 +-
 Documentation/dtas-sourceedit.pod |   2 +-
 INSTALL                           |  16 +--
 lib/dtas.rb                       |   8 ++
 lib/dtas/buffer.rb                |  15 ++-
 lib/dtas/buffer/fiddle_splice.rb  | 216 ++++++++++++++++++++++++++++++
 lib/dtas/buffer/read_write.rb     |   4 +-
 lib/dtas/buffer/splice.rb         |   2 +
 lib/dtas/pipe.rb                  |  20 +--
 lib/dtas/sigevent.rb              |   7 +-
 lib/dtas/sigevent/efd.rb          |   2 +
 lib/dtas/sigevent/fiddle_efd.rb   |  38 ++++++
 lib/dtas/sigevent/pipe.rb         |   2 +-
 lib/dtas/watchable.rb             | 115 ++++++++--------
 lib/dtas/watchable/fiddle_ino.rb  |  78 +++++++++++
 lib/dtas/watchable/inotify.rb     |  13 ++
 test/test_buffer.rb               |  10 +-
 test/test_sigevent.rb             |  20 +++
 test/test_sink_pipe_size.rb       |  27 ++--
 19 files changed, 486 insertions(+), 111 deletions(-)
 create mode 100644 lib/dtas/buffer/fiddle_splice.rb
 create mode 100644 lib/dtas/sigevent/fiddle_efd.rb
 create mode 100644 lib/dtas/watchable/fiddle_ino.rb
 create mode 100644 lib/dtas/watchable/inotify.rb
 create mode 100644 test/test_sigevent.rb

[PATCH v2 1/5] pipe: avoid loading sleepy_penguin

[Eric Wong]

The values of F_{GET,SET}PIPE_SZ are architecture-independent
and stable in Linux (unlike Ruby :P), so we won't need to bother
loading an extra .so here for two constants.
---
 lib/dtas/pipe.rb            | 20 ++++++++++++--------
 test/test_sink_pipe_size.rb | 27 ++++++++++++---------------
 2 files changed, 24 insertions(+), 23 deletions(-)

diff --git a/lib/dtas/pipe.rb b/lib/dtas/pipe.rb
index 58d926c..4c3203d 100644
--- a/lib/dtas/pipe.rb
+++ b/lib/dtas/pipe.rb
@@ -1,10 +1,6 @@
 # Copyright (C) 2013-2019 all contributors <dtas-all@nongnu.org>
 # License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
 # frozen_string_literal: true
-begin
-  require 'sleepy_penguin'
-rescue LoadError
-end
 require_relative '../dtas'
 require_relative 'writable_iter'
 require_relative 'nonblock'
@@ -14,6 +10,11 @@ class DTAS::Pipe < DTAS::Nonblock # :nodoc:
   include DTAS::WritableIter
   attr_accessor :sink
 
+  if RUBY_PLATFORM =~ /linux/i && File.readable?('/proc/sys/fs/pipe-max-size')
+    F_SETPIPE_SZ = 1031
+    F_GETPIPE_SZ = 1032
+  end
+
   def self.new
     _, w = rv = pipe
     w.writable_iter_init
@@ -21,13 +22,16 @@ def self.new
   end
 
   def pipe_size=(nr)
-    defined?(SleepyPenguin::F_SETPIPE_SZ) and
-      fcntl(SleepyPenguin::F_SETPIPE_SZ, nr)
+    fcntl(F_SETPIPE_SZ, nr) if defined?(F_SETPIPE_SZ)
+  rescue Errno::EINVAL # old kernel
+  rescue Errno::EPERM
+    # resizes fail if Linux is close to the pipe limit for the user
+    # or if the user does not have permissions to resize
   end
 
   def pipe_size
-    fcntl(SleepyPenguin::F_GETPIPE_SZ)
-  end if defined?(SleepyPenguin::F_GETPIPE_SZ)
+    fcntl(F_GETPIPE_SZ)
+  end if defined?(F_GETPIPE_SZ)
 
   # avoid syscall, we never change IO#nonblock= directly
   def nonblock?
diff --git a/test/test_sink_pipe_size.rb b/test/test_sink_pipe_size.rb
index 1b6db72..bbe2884 100644
--- a/test/test_sink_pipe_size.rb
+++ b/test/test_sink_pipe_size.rb
@@ -1,20 +1,17 @@
 # Copyright (C) 2013-2019 all contributors <dtas-all@nongnu.org>
 # License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
 # frozen_string_literal: true
-begin
-  require 'sleepy_penguin'
-  require './test/player_integration'
-  class TestSinkPipeSizeIntegration < Testcase
-    include PlayerIntegration
+require './test/player_integration'
+class TestSinkPipeSizeIntegration < Testcase
+  include PlayerIntegration
 
-    def test_sink_pipe_size_integration
-      s = client_socket
-      default_sink_pid(s)
-      s.req_ok("sink ed default pipe_size=0x1000")
-      s.req_ok("sink ed default pipe_size=0x10000")
-      s.req_ok("sink ed default pipe_size=")
-      s.req_ok("sink ed default pipe_size=4096")
-    end if SleepyPenguin.const_defined?(:F_SETPIPE_SZ)
+  def test_sink_pipe_size_integration
+    s = client_socket
+    default_sink_pid(s)
+    s.req_ok("sink ed default pipe_size=0x1000")
+    s.req_ok("sink ed default pipe_size=0x10000")
+    s.req_ok("sink ed default pipe_size=")
+    s.req_ok("sink ed default pipe_size=4096")
   end
-rescue LoadError
-end
+end if RUBY_PLATFORM =~ /linux/i &&
+      File.readable?('/proc/sys/fs/pipe-max-size')

[PATCH v2 2/5] provide fiddle-based eventfd implementation

[Eric Wong]

sleepy_penguin requires a compiler and development headers to
install, so it could be a PITA to install for users on
distro-provided Ruby.  Allow fiddle since it's part of the Ruby
standard library since 1.9.2 and users won't have to install
anything else.
---
 lib/dtas.rb                     |  8 +++++++
 lib/dtas/sigevent.rb            |  7 ++++--
 lib/dtas/sigevent/efd.rb        |  2 ++
 lib/dtas/sigevent/fiddle_efd.rb | 38 +++++++++++++++++++++++++++++++++
 lib/dtas/sigevent/pipe.rb       |  2 +-
 test/test_sigevent.rb           | 20 +++++++++++++++++
 6 files changed, 74 insertions(+), 3 deletions(-)
 create mode 100644 lib/dtas/sigevent/fiddle_efd.rb
 create mode 100644 test/test_sigevent.rb

diff --git a/lib/dtas.rb b/lib/dtas.rb
index 7e9c0d5..ae2f815 100644
--- a/lib/dtas.rb
+++ b/lib/dtas.rb
@@ -25,6 +25,14 @@ def self.null # :nodoc:
     @null ||= File.open('/dev/null', 'r+')
   end
 
+  @libc = nil
+  def self.libc
+    @libc ||= begin
+      require 'fiddle'
+      Fiddle.dlopen(nil)
+    end
+  end
+
   # String#-@ will deduplicate strings when Ruby 2.5 is released (Dec 2017)
   # https://bugs.ruby-lang.org/issues/13077
   if RUBY_VERSION.to_f >= 2.5
diff --git a/lib/dtas/sigevent.rb b/lib/dtas/sigevent.rb
index d4a96d7..74d22df 100644
--- a/lib/dtas/sigevent.rb
+++ b/lib/dtas/sigevent.rb
@@ -3,8 +3,11 @@
 # frozen_string_literal: true
 begin
   raise LoadError, "no eventfd with _DTAS_POSIX" if ENV["_DTAS_POSIX"]
-  require 'sleepy_penguin'
-  require_relative 'sigevent/efd'
+  begin
+    require_relative 'sigevent/efd'
+  rescue LoadError
+    require_relative 'sigevent/fiddle_efd'
+  end
 rescue LoadError
   require_relative 'sigevent/pipe'
 end
diff --git a/lib/dtas/sigevent/efd.rb b/lib/dtas/sigevent/efd.rb
index 4be2c84..22772ee 100644
--- a/lib/dtas/sigevent/efd.rb
+++ b/lib/dtas/sigevent/efd.rb
@@ -3,6 +3,8 @@
 
 # used in various places for safe wakeups from IO.select via signals
 # This requires a modern Linux system and the "sleepy_penguin" RubyGem
+require 'sleepy_penguin'
+
 class DTAS::Sigevent < SleepyPenguin::EventFD # :nodoc:
   def self.new
     super(0, :CLOEXEC)
diff --git a/lib/dtas/sigevent/fiddle_efd.rb b/lib/dtas/sigevent/fiddle_efd.rb
new file mode 100644
index 0000000..e29f6ca
--- /dev/null
+++ b/lib/dtas/sigevent/fiddle_efd.rb
@@ -0,0 +1,38 @@
+# Copyright (C) 2013-2019 all contributors <dtas-all@nongnu.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# frozen_string_literal: true
+
+# used in various places for safe wakeups from IO.select via signals
+# This requires a modern GNU/Linux system with eventfd(2) support
+require_relative '../nonblock'
+require 'fiddle'
+class DTAS::Sigevent # :nodoc:
+
+  EventFD = Fiddle::Function.new(DTAS.libc['eventfd'],
+    [ Fiddle::TYPE_INT, Fiddle::TYPE_INT ], # initval, flags
+    Fiddle::TYPE_INT) # fd
+
+  attr_reader :to_io
+  ONE = [ 1 ].pack('Q').freeze
+
+  def initialize
+    fd = EventFD.call(0, 02000000|00004000) # EFD_CLOEXEC|EFD_NONBLOCK
+    raise "eventfd failed: #{Fiddle.last_error}" if fd < 0
+    @to_io = DTAS::Nonblock.for_fd(fd)
+    @buf = ''.b
+  end
+
+  def signal
+    @to_io.syswrite(ONE)
+  end
+
+  def readable_iter
+    @to_io.read_nonblock(8, @buf, exception: false)
+    yield self, nil # calls DTAS::Process.reaper
+    :wait_readable
+  end
+
+  def close
+    @to_io.close
+  end
+end
diff --git a/lib/dtas/sigevent/pipe.rb b/lib/dtas/sigevent/pipe.rb
index 921a5b3..0ea9d31 100644
--- a/lib/dtas/sigevent/pipe.rb
+++ b/lib/dtas/sigevent/pipe.rb
@@ -3,7 +3,7 @@
 # frozen_string_literal: true
 
 # used in various places for safe wakeups from IO.select via signals
-# A fallback for non-Linux systems lacking the "sleepy_penguin" RubyGem
+# A fallback for non-Linux systems lacking the "splice" syscall
 require_relative '../nonblock'
 class DTAS::Sigevent # :nodoc:
   attr_reader :to_io
diff --git a/test/test_sigevent.rb b/test/test_sigevent.rb
new file mode 100644
index 0000000..6cfe528
--- /dev/null
+++ b/test/test_sigevent.rb
@@ -0,0 +1,20 @@
+# Copyright (C) 2019 all contributors <dtas-all@nongnu.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# frozen_string_literal: true
+require_relative 'helper'
+require 'dtas'
+require 'dtas/sigevent'
+
+class TestSigevent < Testcase
+  def test_sigevent
+    io = DTAS::Sigevent.new
+    io.signal
+    assert IO.select([io]), 'IO.select returns'
+    res = io.readable_iter do |f,arg|
+      assert_same io, f
+      assert_nil arg
+    end
+    assert_equal :wait_readable, res
+    assert_nil io.close
+  end
+end

[PATCH v2 3/5] buffer: replace sleepy_penguin with fiddle

[Eric Wong]

Fiddle exists on all Ruby 1.9.2+ installations and seems
alright.  Since splice is a Linux-only API, we don't need to
worry about the values of constants changing (and they're
architecture-independent).
---
 lib/dtas/buffer.rb               |  15 ++-
 lib/dtas/buffer/fiddle_splice.rb | 216 +++++++++++++++++++++++++++++++
 lib/dtas/buffer/read_write.rb    |   4 +-
 lib/dtas/buffer/splice.rb        |   2 +
 test/test_buffer.rb              |  10 +-
 5 files changed, 234 insertions(+), 13 deletions(-)
 create mode 100644 lib/dtas/buffer/fiddle_splice.rb

diff --git a/lib/dtas/buffer.rb b/lib/dtas/buffer.rb
index 39070d7..e0919d4 100644
--- a/lib/dtas/buffer.rb
+++ b/lib/dtas/buffer.rb
@@ -8,12 +8,15 @@
 class DTAS::Buffer # :nodoc:
   begin
     raise LoadError, "no splice with _DTAS_POSIX" if ENV["_DTAS_POSIX"]
-    require 'sleepy_penguin' # splice is only in Linux for now...
-    SleepyPenguin.respond_to?(:splice) or
-      raise LoadError, 'sleepy_penguin 3.5+ required for splice', []
-    require_relative 'buffer/splice'
-    include DTAS::Buffer::Splice
-  rescue LoadError
+    # splice is only in Linux for now
+    begin
+      require_relative 'buffer/splice'
+      include DTAS::Buffer::Splice
+    rescue LoadError
+      require_relative 'buffer/fiddle_splice'
+      include DTAS::Buffer::FiddleSplice
+    end
+  rescue LoadError, StandardError
     require_relative 'buffer/read_write'
     include DTAS::Buffer::ReadWrite
   end
diff --git a/lib/dtas/buffer/fiddle_splice.rb b/lib/dtas/buffer/fiddle_splice.rb
new file mode 100644
index 0000000..543b3e0
--- /dev/null
+++ b/lib/dtas/buffer/fiddle_splice.rb
@@ -0,0 +1,216 @@
+# Copyright (C) 2013-2019 all contributors <dtas-all@nongnu.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# frozen_string_literal: true
+require 'io/nonblock'
+require 'fiddle' # require_relative caller should expect LoadError
+require_relative '../../dtas'
+require_relative '../pipe'
+
+# Used by -player on Linux systems with the "splice" syscall
+module DTAS::Buffer::FiddleSplice # :nodoc:
+  MAX_AT_ONCE = 4096 # page size in Linux
+  MAX_AT_ONCE_1 = 65536
+  F_MOVE = 1
+  F_NONBLOCK = 2
+
+  Splice = Fiddle::Function.new(DTAS.libc['splice'], [
+      Fiddle::TYPE_INT, # int fd_in,
+      Fiddle::TYPE_VOIDP, # loff_t *off_in
+      Fiddle::TYPE_INT, # int fd_out
+      Fiddle::TYPE_VOIDP, # loff_t *off_out
+      Fiddle::TYPE_SIZE_T, # size_t len
+      Fiddle::TYPE_INT, # unsigned int flags
+    ],
+    Fiddle::TYPE_SSIZE_T) # ssize_t
+
+  Tee = Fiddle::Function.new(DTAS.libc['tee'], [
+      Fiddle::TYPE_INT, # int fd_in,
+      Fiddle::TYPE_INT, # int fd_out
+      Fiddle::TYPE_SIZE_T, # size_t len
+      Fiddle::TYPE_INT, # unsigned int flags
+    ],
+    Fiddle::TYPE_SSIZE_T) # ssize_t
+
+  def _syserr(s, func)
+    raise "BUG: we should not encounter EOF on #{func}" if s == 0
+    case errno = Fiddle.last_error
+    when Errno::EAGAIN::Errno
+      return :EAGAIN
+    when Errno::EPIPE::Errno
+      raise Errno::EPIPE.exception
+    when Errno::EINTR::Errno
+      return nil
+    else
+      raise SystemCallError, "#{func} error: #{errno}"
+    end
+  end
+
+  def splice(src, dst, len, flags)
+    begin
+      s = Splice.call(src.fileno, nil, dst.fileno, nil, len, flags)
+      return s if s > 0
+      sym = _syserr(s, 'splice') and return sym
+    end while true
+  end
+
+  def tee(src, dst, len, flags = 0)
+    begin
+      s = Tee.call(src.fileno, dst.fileno, len, flags)
+      return s if s > 0
+      sym = _syserr(s, 'tee') and return sym
+    end while true
+  end
+
+  def buffer_size
+    @to_io.pipe_size
+  end
+
+  # nil is OK, won't reset existing pipe, either...
+  def buffer_size=(bytes)
+    @to_io.pipe_size = bytes if bytes
+    @buffer_size = bytes
+  end
+
+  # be sure to only call this with nil when all writers to @wr are done
+  def discard(bytes)
+    splice(@to_io, DTAS.null, bytes, 0)
+  end
+
+  def broadcast_one(targets, limit = nil)
+    # single output is always non-blocking
+    limit ||= MAX_AT_ONCE_1
+    s = splice(@to_io, targets[0], limit, F_MOVE|F_NONBLOCK)
+    if Symbol === s
+      targets # our one and only target blocked on write
+    else
+      @bytes_xfer += s
+      :wait_readable # we want to read more from @to_io soon
+    end
+  rescue Errno::EPIPE, IOError => e
+    __dst_error(targets[0], e)
+    targets.clear
+    nil # do not return error here, we already spewed an error message
+  end
+
+  def __tee_in_full(src, dst, bytes)
+    rv = 0
+    while bytes > 0
+      s = tee(src, dst, bytes)
+      bytes -= s
+      rv += s
+    end
+    rv
+  end
+
+  def __splice_in_full(src, dst, bytes, flags)
+    rv = 0
+    while bytes > 0
+      s = splice(src, dst, bytes, flags)
+      rv += s
+      bytes -= s
+    end
+    rv
+  end
+
+  # returns the largest value we teed
+  def __broadcast_tee(blocked, targets, chunk_size)
+    most_teed = 0
+    targets.delete_if do |dst|
+      begin
+        t = (dst.nonblock? || most_teed == 0) ?
+              tee(@to_io, dst, chunk_size, F_NONBLOCK) :
+              __tee_in_full(@to_io, dst, chunk_size)
+        if Integer === t
+          if t > most_teed
+            chunk_size = t if most_teed == 0
+            most_teed = t
+          end
+        else
+          blocked << dst
+        end
+        false
+      rescue IOError, Errno::EPIPE => e
+        __dst_error(dst, e)
+        true
+      end
+    end
+    most_teed
+  end
+
+  def broadcast_inf(targets, limit = nil)
+    if targets.all?(&:ready_write_optimized?)
+      blocked = []
+    elsif targets.none?(&:nonblock?)
+      # if all targets are blocking, don't start until they're all writable
+      r = IO.select(nil, targets, nil, 0) or return targets
+      blocked = targets - r[1]
+
+      # tell DTAS::UNIXServer#run_once to wait on the blocked targets
+      return blocked if blocked[0]
+
+      # all writable, yay!
+    else
+      blocked = []
+    end
+
+    # don't pin too much on one target
+    bytes = limit || MAX_AT_ONCE
+    last = targets.pop # we splice to the last one, tee to the rest
+
+    # this may return zero if all targets were non-blocking
+    most_teed = __broadcast_tee(blocked, targets, bytes)
+
+    # don't splice more than the largest amount we successfully teed
+    bytes = most_teed if most_teed > 0
+
+    begin
+      targets << last
+      if last.nonblock? || most_teed == 0
+        s = splice(@to_io, last, bytes, F_MOVE|F_NONBLOCK)
+        if Symbol === s
+          blocked << last
+
+          # we accomplished nothing!
+          # If _all_ writers are blocked, do not discard data,
+          # stay blocked on :wait_writable
+          return blocked if most_teed == 0
+
+          # the tees targets win, drop data intended for last
+          if most_teed > 0
+            discard(most_teed)
+            @bytes_xfer += most_teed
+            # do not watch for writability of last, last is non-blocking
+            return :wait_readable
+          end
+        end
+      else
+        # the blocking case is simple
+        s = __splice_in_full(@to_io, last, bytes, F_MOVE)
+      end
+      @bytes_xfer += s
+
+      # if we can't splice everything
+      # discard it so the early targets do not get repeated data
+      if s < bytes && most_teed > 0
+        discard(bytes - s)
+      end
+      :wait_readable
+    rescue IOError, Errno::EPIPE => e # last failed, drop it
+      __dst_error(last, e)
+      targets.pop # we're no longer a valid target
+
+      if most_teed == 0
+        # nothing accomplished, watch any targets
+        return blocked if blocked[0]
+      else
+        # some progress, discard the data we could not splice
+        @bytes_xfer += most_teed
+        discard(most_teed)
+      end
+
+      # stop decoding if we're completely errored out
+      # returning nil will trigger close
+      return targets[0] ? :wait_readable : nil
+    end
+  end
+end
diff --git a/lib/dtas/buffer/read_write.rb b/lib/dtas/buffer/read_write.rb
index 04856c7..e2001b6 100644
--- a/lib/dtas/buffer/read_write.rb
+++ b/lib/dtas/buffer/read_write.rb
@@ -6,8 +6,8 @@
 require_relative '../pipe'
 require_relative '../nonblock'
 
-# compatibility code for systems lacking "splice" support via the
-# "sleepy_penguin" 3.5+ RubyGem.  Used only by -player
+# compatibility code for non-Linux systems lacking "splice" support.
+# Used only by -player
 module DTAS::Buffer::ReadWrite # :nodoc:
   MAX_AT_ONCE = 512 # min PIPE_BUF value in POSIX
   attr_accessor :buffer_size
diff --git a/lib/dtas/buffer/splice.rb b/lib/dtas/buffer/splice.rb
index 2e86d0a..b234a57 100644
--- a/lib/dtas/buffer/splice.rb
+++ b/lib/dtas/buffer/splice.rb
@@ -5,6 +5,8 @@
 require 'sleepy_penguin'
 require_relative '../../dtas'
 require_relative '../pipe'
+SleepyPenguin.respond_to?(:splice) or
+  raise LoadError, 'sleepy_penguin 3.5+ required for splice', []
 
 # Used by -player on Linux systems with the "sleepy_penguin" RubyGem installed
 module DTAS::Buffer::Splice # :nodoc:
diff --git a/test/test_buffer.rb b/test/test_buffer.rb
index 8f5d8b5..1773ca3 100644
--- a/test/test_buffer.rb
+++ b/test/test_buffer.rb
@@ -49,14 +49,14 @@ def test_set_buffer_size
     buf = new_buffer
     buf.buffer_size = @@max_size
     assert_equal @@max_size, buf.buffer_size
-  end if defined?(SleepyPenguin::F_GETPIPE_SZ)
+  end if defined?(DTAS::Pipe::F_GETPIPE_SZ)
 
   def test_buffer_size
     buf = new_buffer
     assert_operator buf.buffer_size, :>, 128
     buf.buffer_size = @@max_size
     assert_equal @@max_size, buf.buffer_size
-  end if defined?(SleepyPenguin::F_GETPIPE_SZ)
+  end if defined?(DTAS::Pipe::F_GETPIPE_SZ)
 
   def test_broadcast_1
     buf = new_buffer
@@ -108,7 +108,7 @@ def test_broadcast
     assert_equal "HELLO", a[0].read(5)
     assert_equal "HELLO", b[0].read(5)
 
-    return unless defined?(SleepyPenguin::F_GETPIPE_SZ)
+    return unless defined?(DTAS::Pipe::F_GETPIPE_SZ)
 
     b[1].nonblock = true
     b[1].write('*' * pipe_size(b[1]))
@@ -167,7 +167,7 @@ def test_broadcast_all_full
     buf.wr.write "HELLO"
     assert_equal tmp, buf.broadcast(tmp)
     assert_equal [a[1], b[1]], tmp
-  end if defined?(SleepyPenguin::F_GETPIPE_SZ)
+  end if defined?(DTAS::Pipe::F_GETPIPE_SZ)
 
   def test_serialize
     buf = new_buffer
@@ -206,6 +206,6 @@ def test_load_size
   end
 
   def pipe_size(io)
-    io.fcntl(SleepyPenguin::F_GETPIPE_SZ)
+    io.fcntl(DTAS::Pipe::F_GETPIPE_SZ)
   end
 end

[PATCH v2 4/5] watchable: use fiddle for inotify support

[Eric Wong]

We have String#unpack at our disposal for working with "struct
inotify_event", so use it instead of depending on an extension
which requires a compiler and development headers to install.
---
 lib/dtas/watchable.rb            | 115 ++++++++++++++++---------------
 lib/dtas/watchable/fiddle_ino.rb |  78 +++++++++++++++++++++
 lib/dtas/watchable/inotify.rb    |  13 ++++
 3 files changed, 149 insertions(+), 57 deletions(-)
 create mode 100644 lib/dtas/watchable/fiddle_ino.rb
 create mode 100644 lib/dtas/watchable/inotify.rb

diff --git a/lib/dtas/watchable.rb b/lib/dtas/watchable.rb
index d0f37af..2502b7c 100644
--- a/lib/dtas/watchable.rb
+++ b/lib/dtas/watchable.rb
@@ -1,71 +1,72 @@
 # Copyright (C) 2013-2019 all contributors <dtas-all@nongnu.org>
 # License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
 # frozen_string_literal: true
+require_relative '../dtas'
+require_relative 'nonblock'
 begin
-require 'sleepy_penguin'
+  module DTAS::Watchable # :nodoc:
+    module InotifyCommon # :nodoc:
+      FLAGS = 8 | 128 # IN_CLOSE_WRITE | IN_MOVED_TO
 
-# used to restart DTAS::Source::SplitFX processing in dtas-player
-# if the YAML file is edited
-module DTAS::Watchable # :nodoc:
-  class InotifyReadableIter < SleepyPenguin::Inotify # :nodoc:
-    def self.new
-      super(:CLOEXEC)
-    end
-
-    FLAGS = CLOSE_WRITE | MOVED_TO
-
-    def readable_iter
-      or_call = false
-      while event = take(true) # drain the buffer
-        w = @watches[event.wd] or next
-        if (event.mask & FLAGS) != 0 && w[event.name]
-          or_call = true
+      def readable_iter
+        or_call = false
+        while event = take(true) # drain the buffer
+          w = @watches[event.wd] or next
+          if (event.mask & FLAGS) != 0 && w[event.name]
+            or_call = true
+          end
+        end
+        if or_call
+          @on_readable.call
+          :delete
+        else
+          :wait_readable
         end
       end
-      if or_call
-        @on_readable.call
-        :delete
-      else
-        :wait_readable
-      end
-    end
 
-    # we must watch the directory, since
-    def watch_files(paths, blk)
-      @watches = {} # wd -> { basename -> true }
-      @on_readable = blk
-      @dir2wd = {}
-      Array(paths).each do |path|
-        watchdir, watchbase = File.split(File.expand_path(path))
-        begin
-          wd = @dir2wd[watchdir] ||= add_watch(watchdir, FLAGS)
-          m = @watches[wd] ||= {}
-          m[watchbase] = true
-        rescue SystemCallError => e
-          warn "#{watchdir.dump}: #{e.message} (#{e.class})"
+      # we must watch the directory, since
+      def watch_files(paths, blk)
+        @watches = {} # wd -> { basename -> true }
+        @on_readable = blk
+        @dir2wd = {}
+        Array(paths).each do |path|
+          watchdir, watchbase = File.split(File.expand_path(path))
+          begin
+            wd = @dir2wd[watchdir] ||= add_watch(watchdir, FLAGS)
+            m = @watches[wd] ||= {}
+            m[watchbase] = true
+          rescue SystemCallError => e
+            warn "#{watchdir.dump}: #{e.message} (#{e.class})"
+          end
         end
       end
-    end
-  end
+    end # module InotifyCommon
 
-  def watch_begin(blk)
-    @ino = InotifyReadableIter.new
-    @ino.watch_files(@watch_extra << @infile, blk)
-    @ino
-  end
+    begin
+      require_relative 'watchable/inotify'
+    rescue LoadError
+      # TODO: support kevent
+      require_relative 'watchable/fiddle_ino'
+    end
 
-  def watch_extra(paths)
-    @ino.watch_extra(paths)
-  end
+    def watch_begin(blk)
+      @ino = DTAS::Watchable::InotifyReadableIter.new
+      @ino.watch_files(@watch_extra << @infile, blk)
+      @ino
+    end
 
-  # Closing the inotify descriptor (instead of using inotify_rm_watch)
-  # is cleaner because it avoids EINVAL on race conditions in case
-  # a directory is deleted: https://lkml.org/lkml/2007/7/9/3
-  def watch_end(srv)
-    srv.wait_ctl(@ino, :delete)
-    @ino = @ino.close
-  end
-end
+    def watch_extra(paths)
+      @ino.watch_extra(paths)
+    end
 
-rescue LoadError
-end
+    # Closing the inotify descriptor (instead of using inotify_rm_watch)
+    # is cleaner because it avoids EINVAL on race conditions in case
+    # a directory is deleted: https://lkml.org/lkml/2007/7/9/3
+    def watch_end(srv)
+      srv.wait_ctl(@ino, :delete)
+      @ino = @ino.close
+    end
+  end # module DTAS::Watchable
+rescue LoadError, StandardError => e
+  warn "#{e.message} (#{e.class})"
+end # begin
diff --git a/lib/dtas/watchable/fiddle_ino.rb b/lib/dtas/watchable/fiddle_ino.rb
new file mode 100644
index 0000000..3b9d668
--- /dev/null
+++ b/lib/dtas/watchable/fiddle_ino.rb
@@ -0,0 +1,78 @@
+# Copyright (C) 2013-2019 all contributors <dtas-all@nongnu.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# frozen_string_literal: true
+require 'fiddle'
+
+# used to restart DTAS::Source::SplitFX processing in dtas-player
+# if the YAML file is edited
+class DTAS::Watchable::InotifyReadableIter # :nodoc:
+  include DTAS::Watchable::InotifyCommon
+
+  Inotify_init = Fiddle::Function.new(DTAS.libc['inotify_init1'],
+    [ Fiddle::TYPE_INT ],
+    Fiddle::TYPE_INT)
+
+  Inotify_add_watch = Fiddle::Function.new(DTAS.libc['inotify_add_watch'],
+    [ Fiddle::TYPE_INT, Fiddle::TYPE_VOIDP, Fiddle::TYPE_INT ],
+    Fiddle::TYPE_INT)
+
+  # IO.select compatibility
+  attr_reader :to_io  #:nodoc:
+
+  def initialize # :nodoc:
+    fd = Inotify_init.call(02000000 | 04000) # CLOEXEC | NONBLOCK
+    raise "inotify_init failed: #{Fiddle.last_error}" if fd < 0
+    @to_io = DTAS::Nonblock.for_fd(fd)
+    @buf = ''.b
+    @q = []
+  end
+
+  # struct inotify_event {
+  #     int      wd;       /* Watch descriptor */
+  #     uint32_t mask;     /* Mask describing event */
+  #     uint32_t cookie;   /* Unique cookie associating related
+  #                           events (for rename(2)) */
+  #     uint32_t len;      /* Size of name field */
+  #     char     name[];   /* Optional null-terminated name */
+  InotifyEvent = Struct.new(:wd, :mask, :cookie, :len, :name) # :nodoc:
+
+  def take(nonblock) # :nodoc:
+    event = @q.pop and return event
+    case rv = @to_io.read_nonblock(16384, @buf, exception: false)
+    when :wait_readable, nil
+      return
+    else
+      until rv.empty?
+        hdr = rv.slice!(0,16)
+        name = nil
+        wd, mask, cookie, len = res = hdr.unpack('iIII')
+        wd && mask && cookie && len or
+          raise "bogus inotify_event #{res.inspect} hdr=#{hdr.inspect}"
+        if len > 0
+          name = rv.slice!(0, len)
+          name.size == len or raise "short name #{name.inspect} != #{len}"
+          name.sub!(/\0+\z/, '') or
+            raise "missing: `\\0', inotify_event.name=#{name.inspect}"
+          name = DTAS.dedupe_str(name)
+        end
+        ie = InotifyEvent.new(wd, mask, cookie, len, name)
+        if event
+          @q << ie
+        else
+          event = ie
+        end
+      end # /until rv.empty?
+      return event
+    end while true
+  end
+
+  def add_watch(watchdir, flags)
+    wd = Inotify_add_watch.call(@to_io.fileno, watchdir, flags)
+    raise "inotify_add_watch failed: #{Fiddle.last_error}" if wd < 0
+    wd
+  end
+
+  def close
+    @to_io = @to_io.close if @to_io
+  end
+end
diff --git a/lib/dtas/watchable/inotify.rb b/lib/dtas/watchable/inotify.rb
new file mode 100644
index 0000000..eface0e
--- /dev/null
+++ b/lib/dtas/watchable/inotify.rb
@@ -0,0 +1,13 @@
+# Copyright (C) 2013-2019 all contributors <dtas-all@nongnu.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# frozen_string_literal: true
+require 'sleepy_penguin'
+
+# used to restart DTAS::Source::SplitFX processing in dtas-player
+# if the YAML file is edited
+class DTAS::Watchable::InotifyReadableIter < SleepyPenguin::Inotify # :nodoc:
+  include DTAS::Watchable::InotifyCommon
+  def self.new
+    super(:CLOEXEC)
+  end
+end

[PATCH v2 5/5] doc: remove most recommendations for sleepy_penguin

[Eric Wong]

sleepy_penguin is still a hair faster than libffi+fiddle,
but most users probably won't notice.  So stop documenting
it, but just don't introduce performance regressions for
existing users.
---
 Documentation/dtas-sinkedit.pod   |  2 +-
 Documentation/dtas-sourceedit.pod |  2 +-
 INSTALL                           | 16 +++-------------
 3 files changed, 5 insertions(+), 15 deletions(-)

diff --git a/Documentation/dtas-sinkedit.pod b/Documentation/dtas-sinkedit.pod
index ee2d5b7..f572684 100644
--- a/Documentation/dtas-sinkedit.pod
+++ b/Documentation/dtas-sinkedit.pod
@@ -13,7 +13,7 @@ dtas-sinkedit SINKNAME
 dtas-sinkedit spawns an editor to allow editing of a sink as a YAML file.
 See L<dtas-player_protocol(7)> for details on SINKARGS.
 
-On Linux machines with the sleepy_penguin RubyGem installed, L<inotify(7)>
+On Linux machines,  L<inotify(7)>
 is used to monitor the file for changes while the text editor is running.
 Each time a user finishes saving a file, changes are committed immediately.
 This behavior may be disabled by using the -N or --no-watch command-line
diff --git a/Documentation/dtas-sourceedit.pod b/Documentation/dtas-sourceedit.pod
index ee88e8f..d8313b4 100644
--- a/Documentation/dtas-sourceedit.pod
+++ b/Documentation/dtas-sourceedit.pod
@@ -16,7 +16,7 @@ a pipe or file, it is parsed as YAML and fed to the L<dtas-player(1)> instance
 non-interactively.  This is useful for loading various profiles from the
 filesystem.
 
-On Linux machines with the sleepy_penguin RubyGem installed, L<inotify(7)>
+On Linux machines, L<inotify(7)>
 is used to monitor the file for changes while the text editor is running.
 Each time a user finishes saving a file, changes are committed immediately.
 This behavior may be disabled by using the -N or --no-watch command-line
diff --git a/INSTALL b/INSTALL
index d0b8a24..0ca2e43 100644
--- a/INSTALL
+++ b/INSTALL
@@ -16,18 +16,7 @@ Debian 7+ users can install dependencies easily:
 
     sudo apt-get install sox libsox-fmt-all mp3gain flac ruby-dev
 
-# installing dtas RubyGem on GNU/Linux (Linux kernel 2.6.32+)
-
-Be sure to have Ruby development headers and a working C compiler.
-This will pull in the sleepy_penguin RubyGems for minor
-speedups.  If you cannot be bothered to have a development
-environment, just use "gem install dtas".
-
-    sudo gem install dtas-linux
-
-This should pull in the "sleepy_penguin" RubyGems
-
-For future upgrades of dtas (upgrades to dtas-linux will be infrequent)
+For future upgrades of dtas
 
     sudo gem update dtas
 
@@ -45,7 +34,8 @@ Grab the latest tarball from our HTTPS site:
     $ cd dtas-0.17.0
     $ sudo ruby setup.rb
 
-GNU/Linux users may optionally install the "sleepy_penguin" package:
+GNU/Linux users may optionally install the "sleepy_penguin" RubyGem
+for a tiny speed improvement over the default "fiddle" RubyGem.
 
     * sleepy_penguin - https://bogomips.org/sleepy_penguin/