* [PATCH v2 1/5] pipe: avoid loading sleepy_penguin
2019-12-20 1:39 [PATCH v2 0/5] support fiddle for Linux-only syscalls Eric Wong
@ 2019-12-20 1:39 ` Eric Wong
2019-12-20 1:39 ` [PATCH v2 2/5] provide fiddle-based eventfd implementation Eric Wong
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2019-12-20 1:39 UTC (permalink / raw)
To: dtas-all
The values of F_{GET,SET}PIPE_SZ are architecture-independent
and stable in Linux (unlike Ruby :P), so we won't need to bother
loading an extra .so here for two constants.
---
lib/dtas/pipe.rb | 20 ++++++++++++--------
test/test_sink_pipe_size.rb | 27 ++++++++++++---------------
2 files changed, 24 insertions(+), 23 deletions(-)
diff --git a/lib/dtas/pipe.rb b/lib/dtas/pipe.rb
index 58d926c..4c3203d 100644
--- a/lib/dtas/pipe.rb
+++ b/lib/dtas/pipe.rb
@@ -1,10 +1,6 @@
# Copyright (C) 2013-2019 all contributors <dtas-all@nongnu.org>
# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
# frozen_string_literal: true
-begin
- require 'sleepy_penguin'
-rescue LoadError
-end
require_relative '../dtas'
require_relative 'writable_iter'
require_relative 'nonblock'
@@ -14,6 +10,11 @@ class DTAS::Pipe < DTAS::Nonblock # :nodoc:
include DTAS::WritableIter
attr_accessor :sink
+ if RUBY_PLATFORM =~ /linux/i && File.readable?('/proc/sys/fs/pipe-max-size')
+ F_SETPIPE_SZ = 1031
+ F_GETPIPE_SZ = 1032
+ end
+
def self.new
_, w = rv = pipe
w.writable_iter_init
@@ -21,13 +22,16 @@ def self.new
end
def pipe_size=(nr)
- defined?(SleepyPenguin::F_SETPIPE_SZ) and
- fcntl(SleepyPenguin::F_SETPIPE_SZ, nr)
+ fcntl(F_SETPIPE_SZ, nr) if defined?(F_SETPIPE_SZ)
+ rescue Errno::EINVAL # old kernel
+ rescue Errno::EPERM
+ # resizes fail if Linux is close to the pipe limit for the user
+ # or if the user does not have permissions to resize
end
def pipe_size
- fcntl(SleepyPenguin::F_GETPIPE_SZ)
- end if defined?(SleepyPenguin::F_GETPIPE_SZ)
+ fcntl(F_GETPIPE_SZ)
+ end if defined?(F_GETPIPE_SZ)
# avoid syscall, we never change IO#nonblock= directly
def nonblock?
diff --git a/test/test_sink_pipe_size.rb b/test/test_sink_pipe_size.rb
index 1b6db72..bbe2884 100644
--- a/test/test_sink_pipe_size.rb
+++ b/test/test_sink_pipe_size.rb
@@ -1,20 +1,17 @@
# Copyright (C) 2013-2019 all contributors <dtas-all@nongnu.org>
# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
# frozen_string_literal: true
-begin
- require 'sleepy_penguin'
- require './test/player_integration'
- class TestSinkPipeSizeIntegration < Testcase
- include PlayerIntegration
+require './test/player_integration'
+class TestSinkPipeSizeIntegration < Testcase
+ include PlayerIntegration
- def test_sink_pipe_size_integration
- s = client_socket
- default_sink_pid(s)
- s.req_ok("sink ed default pipe_size=0x1000")
- s.req_ok("sink ed default pipe_size=0x10000")
- s.req_ok("sink ed default pipe_size=")
- s.req_ok("sink ed default pipe_size=4096")
- end if SleepyPenguin.const_defined?(:F_SETPIPE_SZ)
+ def test_sink_pipe_size_integration
+ s = client_socket
+ default_sink_pid(s)
+ s.req_ok("sink ed default pipe_size=0x1000")
+ s.req_ok("sink ed default pipe_size=0x10000")
+ s.req_ok("sink ed default pipe_size=")
+ s.req_ok("sink ed default pipe_size=4096")
end
-rescue LoadError
-end
+end if RUBY_PLATFORM =~ /linux/i &&
+ File.readable?('/proc/sys/fs/pipe-max-size')
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH v2 2/5] provide fiddle-based eventfd implementation
2019-12-20 1:39 [PATCH v2 0/5] support fiddle for Linux-only syscalls Eric Wong
2019-12-20 1:39 ` [PATCH v2 1/5] pipe: avoid loading sleepy_penguin Eric Wong
@ 2019-12-20 1:39 ` Eric Wong
2019-12-20 1:39 ` [PATCH v2 3/5] buffer: replace sleepy_penguin with fiddle Eric Wong
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2019-12-20 1:39 UTC (permalink / raw)
To: dtas-all
sleepy_penguin requires a compiler and development headers to
install, so it could be a PITA to install for users on
distro-provided Ruby. Allow fiddle since it's part of the Ruby
standard library since 1.9.2 and users won't have to install
anything else.
---
lib/dtas.rb | 8 +++++++
lib/dtas/sigevent.rb | 7 ++++--
lib/dtas/sigevent/efd.rb | 2 ++
lib/dtas/sigevent/fiddle_efd.rb | 38 +++++++++++++++++++++++++++++++++
lib/dtas/sigevent/pipe.rb | 2 +-
test/test_sigevent.rb | 20 +++++++++++++++++
6 files changed, 74 insertions(+), 3 deletions(-)
create mode 100644 lib/dtas/sigevent/fiddle_efd.rb
create mode 100644 test/test_sigevent.rb
diff --git a/lib/dtas.rb b/lib/dtas.rb
index 7e9c0d5..ae2f815 100644
--- a/lib/dtas.rb
+++ b/lib/dtas.rb
@@ -25,6 +25,14 @@ def self.null # :nodoc:
@null ||= File.open('/dev/null', 'r+')
end
+ @libc = nil
+ def self.libc
+ @libc ||= begin
+ require 'fiddle'
+ Fiddle.dlopen(nil)
+ end
+ end
+
# String#-@ will deduplicate strings when Ruby 2.5 is released (Dec 2017)
# https://bugs.ruby-lang.org/issues/13077
if RUBY_VERSION.to_f >= 2.5
diff --git a/lib/dtas/sigevent.rb b/lib/dtas/sigevent.rb
index d4a96d7..74d22df 100644
--- a/lib/dtas/sigevent.rb
+++ b/lib/dtas/sigevent.rb
@@ -3,8 +3,11 @@
# frozen_string_literal: true
begin
raise LoadError, "no eventfd with _DTAS_POSIX" if ENV["_DTAS_POSIX"]
- require 'sleepy_penguin'
- require_relative 'sigevent/efd'
+ begin
+ require_relative 'sigevent/efd'
+ rescue LoadError
+ require_relative 'sigevent/fiddle_efd'
+ end
rescue LoadError
require_relative 'sigevent/pipe'
end
diff --git a/lib/dtas/sigevent/efd.rb b/lib/dtas/sigevent/efd.rb
index 4be2c84..22772ee 100644
--- a/lib/dtas/sigevent/efd.rb
+++ b/lib/dtas/sigevent/efd.rb
@@ -3,6 +3,8 @@
# used in various places for safe wakeups from IO.select via signals
# This requires a modern Linux system and the "sleepy_penguin" RubyGem
+require 'sleepy_penguin'
+
class DTAS::Sigevent < SleepyPenguin::EventFD # :nodoc:
def self.new
super(0, :CLOEXEC)
diff --git a/lib/dtas/sigevent/fiddle_efd.rb b/lib/dtas/sigevent/fiddle_efd.rb
new file mode 100644
index 0000000..e29f6ca
--- /dev/null
+++ b/lib/dtas/sigevent/fiddle_efd.rb
@@ -0,0 +1,38 @@
+# Copyright (C) 2013-2019 all contributors <dtas-all@nongnu.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# frozen_string_literal: true
+
+# used in various places for safe wakeups from IO.select via signals
+# This requires a modern GNU/Linux system with eventfd(2) support
+require_relative '../nonblock'
+require 'fiddle'
+class DTAS::Sigevent # :nodoc:
+
+ EventFD = Fiddle::Function.new(DTAS.libc['eventfd'],
+ [ Fiddle::TYPE_INT, Fiddle::TYPE_INT ], # initval, flags
+ Fiddle::TYPE_INT) # fd
+
+ attr_reader :to_io
+ ONE = [ 1 ].pack('Q').freeze
+
+ def initialize
+ fd = EventFD.call(0, 02000000|00004000) # EFD_CLOEXEC|EFD_NONBLOCK
+ raise "eventfd failed: #{Fiddle.last_error}" if fd < 0
+ @to_io = DTAS::Nonblock.for_fd(fd)
+ @buf = ''.b
+ end
+
+ def signal
+ @to_io.syswrite(ONE)
+ end
+
+ def readable_iter
+ @to_io.read_nonblock(8, @buf, exception: false)
+ yield self, nil # calls DTAS::Process.reaper
+ :wait_readable
+ end
+
+ def close
+ @to_io.close
+ end
+end
diff --git a/lib/dtas/sigevent/pipe.rb b/lib/dtas/sigevent/pipe.rb
index 921a5b3..0ea9d31 100644
--- a/lib/dtas/sigevent/pipe.rb
+++ b/lib/dtas/sigevent/pipe.rb
@@ -3,7 +3,7 @@
# frozen_string_literal: true
# used in various places for safe wakeups from IO.select via signals
-# A fallback for non-Linux systems lacking the "sleepy_penguin" RubyGem
+# A fallback for non-Linux systems lacking the "splice" syscall
require_relative '../nonblock'
class DTAS::Sigevent # :nodoc:
attr_reader :to_io
diff --git a/test/test_sigevent.rb b/test/test_sigevent.rb
new file mode 100644
index 0000000..6cfe528
--- /dev/null
+++ b/test/test_sigevent.rb
@@ -0,0 +1,20 @@
+# Copyright (C) 2019 all contributors <dtas-all@nongnu.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# frozen_string_literal: true
+require_relative 'helper'
+require 'dtas'
+require 'dtas/sigevent'
+
+class TestSigevent < Testcase
+ def test_sigevent
+ io = DTAS::Sigevent.new
+ io.signal
+ assert IO.select([io]), 'IO.select returns'
+ res = io.readable_iter do |f,arg|
+ assert_same io, f
+ assert_nil arg
+ end
+ assert_equal :wait_readable, res
+ assert_nil io.close
+ end
+end
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH v2 3/5] buffer: replace sleepy_penguin with fiddle
2019-12-20 1:39 [PATCH v2 0/5] support fiddle for Linux-only syscalls Eric Wong
2019-12-20 1:39 ` [PATCH v2 1/5] pipe: avoid loading sleepy_penguin Eric Wong
2019-12-20 1:39 ` [PATCH v2 2/5] provide fiddle-based eventfd implementation Eric Wong
@ 2019-12-20 1:39 ` Eric Wong
2019-12-20 1:39 ` [PATCH v2 4/5] watchable: use fiddle for inotify support Eric Wong
2019-12-20 1:39 ` [PATCH v2 5/5] doc: remove most recommendations for sleepy_penguin Eric Wong
4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2019-12-20 1:39 UTC (permalink / raw)
To: dtas-all
Fiddle exists on all Ruby 1.9.2+ installations and seems
alright. Since splice is a Linux-only API, we don't need to
worry about the values of constants changing (and they're
architecture-independent).
---
lib/dtas/buffer.rb | 15 ++-
lib/dtas/buffer/fiddle_splice.rb | 216 +++++++++++++++++++++++++++++++
lib/dtas/buffer/read_write.rb | 4 +-
lib/dtas/buffer/splice.rb | 2 +
test/test_buffer.rb | 10 +-
5 files changed, 234 insertions(+), 13 deletions(-)
create mode 100644 lib/dtas/buffer/fiddle_splice.rb
diff --git a/lib/dtas/buffer.rb b/lib/dtas/buffer.rb
index 39070d7..e0919d4 100644
--- a/lib/dtas/buffer.rb
+++ b/lib/dtas/buffer.rb
@@ -8,12 +8,15 @@
class DTAS::Buffer # :nodoc:
begin
raise LoadError, "no splice with _DTAS_POSIX" if ENV["_DTAS_POSIX"]
- require 'sleepy_penguin' # splice is only in Linux for now...
- SleepyPenguin.respond_to?(:splice) or
- raise LoadError, 'sleepy_penguin 3.5+ required for splice', []
- require_relative 'buffer/splice'
- include DTAS::Buffer::Splice
- rescue LoadError
+ # splice is only in Linux for now
+ begin
+ require_relative 'buffer/splice'
+ include DTAS::Buffer::Splice
+ rescue LoadError
+ require_relative 'buffer/fiddle_splice'
+ include DTAS::Buffer::FiddleSplice
+ end
+ rescue LoadError, StandardError
require_relative 'buffer/read_write'
include DTAS::Buffer::ReadWrite
end
diff --git a/lib/dtas/buffer/fiddle_splice.rb b/lib/dtas/buffer/fiddle_splice.rb
new file mode 100644
index 0000000..543b3e0
--- /dev/null
+++ b/lib/dtas/buffer/fiddle_splice.rb
@@ -0,0 +1,216 @@
+# Copyright (C) 2013-2019 all contributors <dtas-all@nongnu.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# frozen_string_literal: true
+require 'io/nonblock'
+require 'fiddle' # require_relative caller should expect LoadError
+require_relative '../../dtas'
+require_relative '../pipe'
+
+# Used by -player on Linux systems with the "splice" syscall
+module DTAS::Buffer::FiddleSplice # :nodoc:
+ MAX_AT_ONCE = 4096 # page size in Linux
+ MAX_AT_ONCE_1 = 65536
+ F_MOVE = 1
+ F_NONBLOCK = 2
+
+ Splice = Fiddle::Function.new(DTAS.libc['splice'], [
+ Fiddle::TYPE_INT, # int fd_in,
+ Fiddle::TYPE_VOIDP, # loff_t *off_in
+ Fiddle::TYPE_INT, # int fd_out
+ Fiddle::TYPE_VOIDP, # loff_t *off_out
+ Fiddle::TYPE_SIZE_T, # size_t len
+ Fiddle::TYPE_INT, # unsigned int flags
+ ],
+ Fiddle::TYPE_SSIZE_T) # ssize_t
+
+ Tee = Fiddle::Function.new(DTAS.libc['tee'], [
+ Fiddle::TYPE_INT, # int fd_in,
+ Fiddle::TYPE_INT, # int fd_out
+ Fiddle::TYPE_SIZE_T, # size_t len
+ Fiddle::TYPE_INT, # unsigned int flags
+ ],
+ Fiddle::TYPE_SSIZE_T) # ssize_t
+
+ def _syserr(s, func)
+ raise "BUG: we should not encounter EOF on #{func}" if s == 0
+ case errno = Fiddle.last_error
+ when Errno::EAGAIN::Errno
+ return :EAGAIN
+ when Errno::EPIPE::Errno
+ raise Errno::EPIPE.exception
+ when Errno::EINTR::Errno
+ return nil
+ else
+ raise SystemCallError, "#{func} error: #{errno}"
+ end
+ end
+
+ def splice(src, dst, len, flags)
+ begin
+ s = Splice.call(src.fileno, nil, dst.fileno, nil, len, flags)
+ return s if s > 0
+ sym = _syserr(s, 'splice') and return sym
+ end while true
+ end
+
+ def tee(src, dst, len, flags = 0)
+ begin
+ s = Tee.call(src.fileno, dst.fileno, len, flags)
+ return s if s > 0
+ sym = _syserr(s, 'tee') and return sym
+ end while true
+ end
+
+ def buffer_size
+ @to_io.pipe_size
+ end
+
+ # nil is OK, won't reset existing pipe, either...
+ def buffer_size=(bytes)
+ @to_io.pipe_size = bytes if bytes
+ @buffer_size = bytes
+ end
+
+ # be sure to only call this with nil when all writers to @wr are done
+ def discard(bytes)
+ splice(@to_io, DTAS.null, bytes, 0)
+ end
+
+ def broadcast_one(targets, limit = nil)
+ # single output is always non-blocking
+ limit ||= MAX_AT_ONCE_1
+ s = splice(@to_io, targets[0], limit, F_MOVE|F_NONBLOCK)
+ if Symbol === s
+ targets # our one and only target blocked on write
+ else
+ @bytes_xfer += s
+ :wait_readable # we want to read more from @to_io soon
+ end
+ rescue Errno::EPIPE, IOError => e
+ __dst_error(targets[0], e)
+ targets.clear
+ nil # do not return error here, we already spewed an error message
+ end
+
+ def __tee_in_full(src, dst, bytes)
+ rv = 0
+ while bytes > 0
+ s = tee(src, dst, bytes)
+ bytes -= s
+ rv += s
+ end
+ rv
+ end
+
+ def __splice_in_full(src, dst, bytes, flags)
+ rv = 0
+ while bytes > 0
+ s = splice(src, dst, bytes, flags)
+ rv += s
+ bytes -= s
+ end
+ rv
+ end
+
+ # returns the largest value we teed
+ def __broadcast_tee(blocked, targets, chunk_size)
+ most_teed = 0
+ targets.delete_if do |dst|
+ begin
+ t = (dst.nonblock? || most_teed == 0) ?
+ tee(@to_io, dst, chunk_size, F_NONBLOCK) :
+ __tee_in_full(@to_io, dst, chunk_size)
+ if Integer === t
+ if t > most_teed
+ chunk_size = t if most_teed == 0
+ most_teed = t
+ end
+ else
+ blocked << dst
+ end
+ false
+ rescue IOError, Errno::EPIPE => e
+ __dst_error(dst, e)
+ true
+ end
+ end
+ most_teed
+ end
+
+ def broadcast_inf(targets, limit = nil)
+ if targets.all?(&:ready_write_optimized?)
+ blocked = []
+ elsif targets.none?(&:nonblock?)
+ # if all targets are blocking, don't start until they're all writable
+ r = IO.select(nil, targets, nil, 0) or return targets
+ blocked = targets - r[1]
+
+ # tell DTAS::UNIXServer#run_once to wait on the blocked targets
+ return blocked if blocked[0]
+
+ # all writable, yay!
+ else
+ blocked = []
+ end
+
+ # don't pin too much on one target
+ bytes = limit || MAX_AT_ONCE
+ last = targets.pop # we splice to the last one, tee to the rest
+
+ # this may return zero if all targets were non-blocking
+ most_teed = __broadcast_tee(blocked, targets, bytes)
+
+ # don't splice more than the largest amount we successfully teed
+ bytes = most_teed if most_teed > 0
+
+ begin
+ targets << last
+ if last.nonblock? || most_teed == 0
+ s = splice(@to_io, last, bytes, F_MOVE|F_NONBLOCK)
+ if Symbol === s
+ blocked << last
+
+ # we accomplished nothing!
+ # If _all_ writers are blocked, do not discard data,
+ # stay blocked on :wait_writable
+ return blocked if most_teed == 0
+
+ # the tees targets win, drop data intended for last
+ if most_teed > 0
+ discard(most_teed)
+ @bytes_xfer += most_teed
+ # do not watch for writability of last, last is non-blocking
+ return :wait_readable
+ end
+ end
+ else
+ # the blocking case is simple
+ s = __splice_in_full(@to_io, last, bytes, F_MOVE)
+ end
+ @bytes_xfer += s
+
+ # if we can't splice everything
+ # discard it so the early targets do not get repeated data
+ if s < bytes && most_teed > 0
+ discard(bytes - s)
+ end
+ :wait_readable
+ rescue IOError, Errno::EPIPE => e # last failed, drop it
+ __dst_error(last, e)
+ targets.pop # we're no longer a valid target
+
+ if most_teed == 0
+ # nothing accomplished, watch any targets
+ return blocked if blocked[0]
+ else
+ # some progress, discard the data we could not splice
+ @bytes_xfer += most_teed
+ discard(most_teed)
+ end
+
+ # stop decoding if we're completely errored out
+ # returning nil will trigger close
+ return targets[0] ? :wait_readable : nil
+ end
+ end
+end
diff --git a/lib/dtas/buffer/read_write.rb b/lib/dtas/buffer/read_write.rb
index 04856c7..e2001b6 100644
--- a/lib/dtas/buffer/read_write.rb
+++ b/lib/dtas/buffer/read_write.rb
@@ -6,8 +6,8 @@
require_relative '../pipe'
require_relative '../nonblock'
-# compatibility code for systems lacking "splice" support via the
-# "sleepy_penguin" 3.5+ RubyGem. Used only by -player
+# compatibility code for non-Linux systems lacking "splice" support.
+# Used only by -player
module DTAS::Buffer::ReadWrite # :nodoc:
MAX_AT_ONCE = 512 # min PIPE_BUF value in POSIX
attr_accessor :buffer_size
diff --git a/lib/dtas/buffer/splice.rb b/lib/dtas/buffer/splice.rb
index 2e86d0a..b234a57 100644
--- a/lib/dtas/buffer/splice.rb
+++ b/lib/dtas/buffer/splice.rb
@@ -5,6 +5,8 @@
require 'sleepy_penguin'
require_relative '../../dtas'
require_relative '../pipe'
+SleepyPenguin.respond_to?(:splice) or
+ raise LoadError, 'sleepy_penguin 3.5+ required for splice', []
# Used by -player on Linux systems with the "sleepy_penguin" RubyGem installed
module DTAS::Buffer::Splice # :nodoc:
diff --git a/test/test_buffer.rb b/test/test_buffer.rb
index 8f5d8b5..1773ca3 100644
--- a/test/test_buffer.rb
+++ b/test/test_buffer.rb
@@ -49,14 +49,14 @@ def test_set_buffer_size
buf = new_buffer
buf.buffer_size = @@max_size
assert_equal @@max_size, buf.buffer_size
- end if defined?(SleepyPenguin::F_GETPIPE_SZ)
+ end if defined?(DTAS::Pipe::F_GETPIPE_SZ)
def test_buffer_size
buf = new_buffer
assert_operator buf.buffer_size, :>, 128
buf.buffer_size = @@max_size
assert_equal @@max_size, buf.buffer_size
- end if defined?(SleepyPenguin::F_GETPIPE_SZ)
+ end if defined?(DTAS::Pipe::F_GETPIPE_SZ)
def test_broadcast_1
buf = new_buffer
@@ -108,7 +108,7 @@ def test_broadcast
assert_equal "HELLO", a[0].read(5)
assert_equal "HELLO", b[0].read(5)
- return unless defined?(SleepyPenguin::F_GETPIPE_SZ)
+ return unless defined?(DTAS::Pipe::F_GETPIPE_SZ)
b[1].nonblock = true
b[1].write('*' * pipe_size(b[1]))
@@ -167,7 +167,7 @@ def test_broadcast_all_full
buf.wr.write "HELLO"
assert_equal tmp, buf.broadcast(tmp)
assert_equal [a[1], b[1]], tmp
- end if defined?(SleepyPenguin::F_GETPIPE_SZ)
+ end if defined?(DTAS::Pipe::F_GETPIPE_SZ)
def test_serialize
buf = new_buffer
@@ -206,6 +206,6 @@ def test_load_size
end
def pipe_size(io)
- io.fcntl(SleepyPenguin::F_GETPIPE_SZ)
+ io.fcntl(DTAS::Pipe::F_GETPIPE_SZ)
end
end
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH v2 4/5] watchable: use fiddle for inotify support
2019-12-20 1:39 [PATCH v2 0/5] support fiddle for Linux-only syscalls Eric Wong
` (2 preceding siblings ...)
2019-12-20 1:39 ` [PATCH v2 3/5] buffer: replace sleepy_penguin with fiddle Eric Wong
@ 2019-12-20 1:39 ` Eric Wong
2019-12-20 1:39 ` [PATCH v2 5/5] doc: remove most recommendations for sleepy_penguin Eric Wong
4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2019-12-20 1:39 UTC (permalink / raw)
To: dtas-all
We have String#unpack at our disposal for working with "struct
inotify_event", so use it instead of depending on an extension
which requires a compiler and development headers to install.
---
lib/dtas/watchable.rb | 115 ++++++++++++++++---------------
lib/dtas/watchable/fiddle_ino.rb | 78 +++++++++++++++++++++
lib/dtas/watchable/inotify.rb | 13 ++++
3 files changed, 149 insertions(+), 57 deletions(-)
create mode 100644 lib/dtas/watchable/fiddle_ino.rb
create mode 100644 lib/dtas/watchable/inotify.rb
diff --git a/lib/dtas/watchable.rb b/lib/dtas/watchable.rb
index d0f37af..2502b7c 100644
--- a/lib/dtas/watchable.rb
+++ b/lib/dtas/watchable.rb
@@ -1,71 +1,72 @@
# Copyright (C) 2013-2019 all contributors <dtas-all@nongnu.org>
# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
# frozen_string_literal: true
+require_relative '../dtas'
+require_relative 'nonblock'
begin
-require 'sleepy_penguin'
+ module DTAS::Watchable # :nodoc:
+ module InotifyCommon # :nodoc:
+ FLAGS = 8 | 128 # IN_CLOSE_WRITE | IN_MOVED_TO
-# used to restart DTAS::Source::SplitFX processing in dtas-player
-# if the YAML file is edited
-module DTAS::Watchable # :nodoc:
- class InotifyReadableIter < SleepyPenguin::Inotify # :nodoc:
- def self.new
- super(:CLOEXEC)
- end
-
- FLAGS = CLOSE_WRITE | MOVED_TO
-
- def readable_iter
- or_call = false
- while event = take(true) # drain the buffer
- w = @watches[event.wd] or next
- if (event.mask & FLAGS) != 0 && w[event.name]
- or_call = true
+ def readable_iter
+ or_call = false
+ while event = take(true) # drain the buffer
+ w = @watches[event.wd] or next
+ if (event.mask & FLAGS) != 0 && w[event.name]
+ or_call = true
+ end
+ end
+ if or_call
+ @on_readable.call
+ :delete
+ else
+ :wait_readable
end
end
- if or_call
- @on_readable.call
- :delete
- else
- :wait_readable
- end
- end
- # we must watch the directory, since
- def watch_files(paths, blk)
- @watches = {} # wd -> { basename -> true }
- @on_readable = blk
- @dir2wd = {}
- Array(paths).each do |path|
- watchdir, watchbase = File.split(File.expand_path(path))
- begin
- wd = @dir2wd[watchdir] ||= add_watch(watchdir, FLAGS)
- m = @watches[wd] ||= {}
- m[watchbase] = true
- rescue SystemCallError => e
- warn "#{watchdir.dump}: #{e.message} (#{e.class})"
+ # we must watch the directory, since
+ def watch_files(paths, blk)
+ @watches = {} # wd -> { basename -> true }
+ @on_readable = blk
+ @dir2wd = {}
+ Array(paths).each do |path|
+ watchdir, watchbase = File.split(File.expand_path(path))
+ begin
+ wd = @dir2wd[watchdir] ||= add_watch(watchdir, FLAGS)
+ m = @watches[wd] ||= {}
+ m[watchbase] = true
+ rescue SystemCallError => e
+ warn "#{watchdir.dump}: #{e.message} (#{e.class})"
+ end
end
end
- end
- end
+ end # module InotifyCommon
- def watch_begin(blk)
- @ino = InotifyReadableIter.new
- @ino.watch_files(@watch_extra << @infile, blk)
- @ino
- end
+ begin
+ require_relative 'watchable/inotify'
+ rescue LoadError
+ # TODO: support kevent
+ require_relative 'watchable/fiddle_ino'
+ end
- def watch_extra(paths)
- @ino.watch_extra(paths)
- end
+ def watch_begin(blk)
+ @ino = DTAS::Watchable::InotifyReadableIter.new
+ @ino.watch_files(@watch_extra << @infile, blk)
+ @ino
+ end
- # Closing the inotify descriptor (instead of using inotify_rm_watch)
- # is cleaner because it avoids EINVAL on race conditions in case
- # a directory is deleted: https://lkml.org/lkml/2007/7/9/3
- def watch_end(srv)
- srv.wait_ctl(@ino, :delete)
- @ino = @ino.close
- end
-end
+ def watch_extra(paths)
+ @ino.watch_extra(paths)
+ end
-rescue LoadError
-end
+ # Closing the inotify descriptor (instead of using inotify_rm_watch)
+ # is cleaner because it avoids EINVAL on race conditions in case
+ # a directory is deleted: https://lkml.org/lkml/2007/7/9/3
+ def watch_end(srv)
+ srv.wait_ctl(@ino, :delete)
+ @ino = @ino.close
+ end
+ end # module DTAS::Watchable
+rescue LoadError, StandardError => e
+ warn "#{e.message} (#{e.class})"
+end # begin
diff --git a/lib/dtas/watchable/fiddle_ino.rb b/lib/dtas/watchable/fiddle_ino.rb
new file mode 100644
index 0000000..3b9d668
--- /dev/null
+++ b/lib/dtas/watchable/fiddle_ino.rb
@@ -0,0 +1,78 @@
+# Copyright (C) 2013-2019 all contributors <dtas-all@nongnu.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# frozen_string_literal: true
+require 'fiddle'
+
+# used to restart DTAS::Source::SplitFX processing in dtas-player
+# if the YAML file is edited
+class DTAS::Watchable::InotifyReadableIter # :nodoc:
+ include DTAS::Watchable::InotifyCommon
+
+ Inotify_init = Fiddle::Function.new(DTAS.libc['inotify_init1'],
+ [ Fiddle::TYPE_INT ],
+ Fiddle::TYPE_INT)
+
+ Inotify_add_watch = Fiddle::Function.new(DTAS.libc['inotify_add_watch'],
+ [ Fiddle::TYPE_INT, Fiddle::TYPE_VOIDP, Fiddle::TYPE_INT ],
+ Fiddle::TYPE_INT)
+
+ # IO.select compatibility
+ attr_reader :to_io #:nodoc:
+
+ def initialize # :nodoc:
+ fd = Inotify_init.call(02000000 | 04000) # CLOEXEC | NONBLOCK
+ raise "inotify_init failed: #{Fiddle.last_error}" if fd < 0
+ @to_io = DTAS::Nonblock.for_fd(fd)
+ @buf = ''.b
+ @q = []
+ end
+
+ # struct inotify_event {
+ # int wd; /* Watch descriptor */
+ # uint32_t mask; /* Mask describing event */
+ # uint32_t cookie; /* Unique cookie associating related
+ # events (for rename(2)) */
+ # uint32_t len; /* Size of name field */
+ # char name[]; /* Optional null-terminated name */
+ InotifyEvent = Struct.new(:wd, :mask, :cookie, :len, :name) # :nodoc:
+
+ def take(nonblock) # :nodoc:
+ event = @q.pop and return event
+ case rv = @to_io.read_nonblock(16384, @buf, exception: false)
+ when :wait_readable, nil
+ return
+ else
+ until rv.empty?
+ hdr = rv.slice!(0,16)
+ name = nil
+ wd, mask, cookie, len = res = hdr.unpack('iIII')
+ wd && mask && cookie && len or
+ raise "bogus inotify_event #{res.inspect} hdr=#{hdr.inspect}"
+ if len > 0
+ name = rv.slice!(0, len)
+ name.size == len or raise "short name #{name.inspect} != #{len}"
+ name.sub!(/\0+\z/, '') or
+ raise "missing: `\\0', inotify_event.name=#{name.inspect}"
+ name = DTAS.dedupe_str(name)
+ end
+ ie = InotifyEvent.new(wd, mask, cookie, len, name)
+ if event
+ @q << ie
+ else
+ event = ie
+ end
+ end # /until rv.empty?
+ return event
+ end while true
+ end
+
+ def add_watch(watchdir, flags)
+ wd = Inotify_add_watch.call(@to_io.fileno, watchdir, flags)
+ raise "inotify_add_watch failed: #{Fiddle.last_error}" if wd < 0
+ wd
+ end
+
+ def close
+ @to_io = @to_io.close if @to_io
+ end
+end
diff --git a/lib/dtas/watchable/inotify.rb b/lib/dtas/watchable/inotify.rb
new file mode 100644
index 0000000..eface0e
--- /dev/null
+++ b/lib/dtas/watchable/inotify.rb
@@ -0,0 +1,13 @@
+# Copyright (C) 2013-2019 all contributors <dtas-all@nongnu.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# frozen_string_literal: true
+require 'sleepy_penguin'
+
+# used to restart DTAS::Source::SplitFX processing in dtas-player
+# if the YAML file is edited
+class DTAS::Watchable::InotifyReadableIter < SleepyPenguin::Inotify # :nodoc:
+ include DTAS::Watchable::InotifyCommon
+ def self.new
+ super(:CLOEXEC)
+ end
+end
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH v2 5/5] doc: remove most recommendations for sleepy_penguin
2019-12-20 1:39 [PATCH v2 0/5] support fiddle for Linux-only syscalls Eric Wong
` (3 preceding siblings ...)
2019-12-20 1:39 ` [PATCH v2 4/5] watchable: use fiddle for inotify support Eric Wong
@ 2019-12-20 1:39 ` Eric Wong
4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2019-12-20 1:39 UTC (permalink / raw)
To: dtas-all
sleepy_penguin is still a hair faster than libffi+fiddle,
but most users probably won't notice. So stop documenting
it, but just don't introduce performance regressions for
existing users.
---
Documentation/dtas-sinkedit.pod | 2 +-
Documentation/dtas-sourceedit.pod | 2 +-
INSTALL | 16 +++-------------
3 files changed, 5 insertions(+), 15 deletions(-)
diff --git a/Documentation/dtas-sinkedit.pod b/Documentation/dtas-sinkedit.pod
index ee2d5b7..f572684 100644
--- a/Documentation/dtas-sinkedit.pod
+++ b/Documentation/dtas-sinkedit.pod
@@ -13,7 +13,7 @@ dtas-sinkedit SINKNAME
dtas-sinkedit spawns an editor to allow editing of a sink as a YAML file.
See L<dtas-player_protocol(7)> for details on SINKARGS.
-On Linux machines with the sleepy_penguin RubyGem installed, L<inotify(7)>
+On Linux machines, L<inotify(7)>
is used to monitor the file for changes while the text editor is running.
Each time a user finishes saving a file, changes are committed immediately.
This behavior may be disabled by using the -N or --no-watch command-line
diff --git a/Documentation/dtas-sourceedit.pod b/Documentation/dtas-sourceedit.pod
index ee88e8f..d8313b4 100644
--- a/Documentation/dtas-sourceedit.pod
+++ b/Documentation/dtas-sourceedit.pod
@@ -16,7 +16,7 @@ a pipe or file, it is parsed as YAML and fed to the L<dtas-player(1)> instance
non-interactively. This is useful for loading various profiles from the
filesystem.
-On Linux machines with the sleepy_penguin RubyGem installed, L<inotify(7)>
+On Linux machines, L<inotify(7)>
is used to monitor the file for changes while the text editor is running.
Each time a user finishes saving a file, changes are committed immediately.
This behavior may be disabled by using the -N or --no-watch command-line
diff --git a/INSTALL b/INSTALL
index d0b8a24..0ca2e43 100644
--- a/INSTALL
+++ b/INSTALL
@@ -16,18 +16,7 @@ Debian 7+ users can install dependencies easily:
sudo apt-get install sox libsox-fmt-all mp3gain flac ruby-dev
-# installing dtas RubyGem on GNU/Linux (Linux kernel 2.6.32+)
-
-Be sure to have Ruby development headers and a working C compiler.
-This will pull in the sleepy_penguin RubyGems for minor
-speedups. If you cannot be bothered to have a development
-environment, just use "gem install dtas".
-
- sudo gem install dtas-linux
-
-This should pull in the "sleepy_penguin" RubyGems
-
-For future upgrades of dtas (upgrades to dtas-linux will be infrequent)
+For future upgrades of dtas
sudo gem update dtas
@@ -45,7 +34,8 @@ Grab the latest tarball from our HTTPS site:
$ cd dtas-0.17.0
$ sudo ruby setup.rb
-GNU/Linux users may optionally install the "sleepy_penguin" package:
+GNU/Linux users may optionally install the "sleepy_penguin" RubyGem
+for a tiny speed improvement over the default "fiddle" RubyGem.
* sleepy_penguin - https://bogomips.org/sleepy_penguin/
^ permalink raw reply related [flat|nested] 6+ messages in thread