* [PATCH] use sleepy_penguin 3.5+ for splice and tee support
@ 2019-01-02 20:42 4% Eric Wong
0 siblings, 0 replies; 7+ results
From: Eric Wong @ 2019-01-02 20:42 UTC (permalink / raw)
To: dtas-all
Eliminate loading of the io_splice RubyGem to reduce memory overhead.
Extra DSOs are wasteful and io_splice is being phased oiut for
sleepy_penguin, which encapsulates more Linux-specific functionality
anyways.
cf. https://udrepper.livejournal.com/8790.html
---
INSTALL | 8 +++-----
dtas-linux.gemspec | 5 ++---
lib/dtas/buffer.rb | 4 +++-
lib/dtas/buffer/read_write.rb | 2 +-
lib/dtas/buffer/splice.rb | 20 +++++++++++---------
lib/dtas/pipe.rb | 13 ++++++++-----
test/test_buffer.rb | 22 +++++++++++++---------
test/test_sink_pipe_size.rb | 4 ++--
8 files changed, 43 insertions(+), 35 deletions(-)
diff --git a/INSTALL b/INSTALL
index 96cfcdb..b07b3c7 100644
--- a/INSTALL
+++ b/INSTALL
@@ -19,13 +19,13 @@ Debian 7+ users can install dependencies easily:
# installing dtas RubyGem on GNU/Linux (Linux kernel 2.6.32+)
Be sure to have Ruby development headers and a working C compiler.
-This will pull in the io_splice and sleepy_penguin RubyGems for minor
+This will pull in the sleepy_penguin RubyGems for minor
speedups. If you cannot be bothered to have a development
environment, just use "gem install dtas".
sudo gem install dtas-linux
-This should pull in the "io_splice" and "sleepy_penguin" RubyGems
+This should pull in the "sleepy_penguin" RubyGems
For future upgrades of dtas (upgrades to dtas-linux will be infrequent)
@@ -45,10 +45,8 @@ Grab the latest tarball from our HTTPS site:
$ cd dtas-0.15.0
$ sudo ruby setup.rb
-GNU/Linux users may optionally install "io_splice" and
-"sleepy_penguin" packages:
+GNU/Linux users may optionally install the "sleepy_penguin" package:
- * io_splice - https://bogomips.org/ruby_io_splice/
* sleepy_penguin - https://bogomips.org/sleepy_penguin/
# CONTACT
diff --git a/dtas-linux.gemspec b/dtas-linux.gemspec
index ca1be89..0c1d97e 100644
--- a/dtas-linux.gemspec
+++ b/dtas-linux.gemspec
@@ -13,8 +13,7 @@
s.email = %q{e@80x24.org}
s.files = []
s.homepage = 'https://80x24.org/dtas/'
- s.add_dependency(%q<dtas>)
- s.add_dependency(%q<io_splice>, '~> 4')
- s.add_dependency(%q<sleepy_penguin>, '~> 3')
+ s.add_dependency(%q<dtas>, '~> 0.16')
+ s.add_dependency(%q<sleepy_penguin>, '~> 3.5')
s.licenses = 'GPL-3.0+'
end
diff --git a/lib/dtas/buffer.rb b/lib/dtas/buffer.rb
index 6f43dfa..c3d8ee2 100644
--- a/lib/dtas/buffer.rb
+++ b/lib/dtas/buffer.rb
@@ -8,7 +8,9 @@
class DTAS::Buffer # :nodoc:
begin
raise LoadError, "no splice with _DTAS_POSIX" if ENV["_DTAS_POSIX"]
- require 'io/splice' # splice is only in Linux for now...
+ require 'sleepy_penguin' # splice is only in Linux for now...
+ SleepyPenguin.respond_to?(:splice) or
+ raise LoadError, 'sleepy_penguin 3.5+ required for splice', []
require_relative 'buffer/splice'
include DTAS::Buffer::Splice
rescue LoadError
diff --git a/lib/dtas/buffer/read_write.rb b/lib/dtas/buffer/read_write.rb
index 5845309..06947a5 100644
--- a/lib/dtas/buffer/read_write.rb
+++ b/lib/dtas/buffer/read_write.rb
@@ -7,7 +7,7 @@
require_relative '../nonblock'
# compatibility code for systems lacking "splice" support via the
-# "io-splice" RubyGem. Used only by -player
+# "sleepy_penguin" 3.5+ RubyGem. Used only by -player
module DTAS::Buffer::ReadWrite # :nodoc:
MAX_AT_ONCE = 512 # min PIPE_BUF value in POSIX
attr_accessor :buffer_size
diff --git a/lib/dtas/buffer/splice.rb b/lib/dtas/buffer/splice.rb
index 1cb0fbc..cd00bbb 100644
--- a/lib/dtas/buffer/splice.rb
+++ b/lib/dtas/buffer/splice.rb
@@ -2,15 +2,17 @@
# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
# frozen_string_literal: true
require 'io/nonblock'
-require 'io/splice'
+require 'sleepy_penguin'
require_relative '../../dtas'
require_relative '../pipe'
-# Used by -player on Linux systems with the "io-splice" RubyGem installed
+# Used by -player on Linux systems with the "sleepy_penguin" RubyGem installed
module DTAS::Buffer::Splice # :nodoc:
MAX_AT_ONCE = 4096 # page size in Linux
MAX_AT_ONCE_1 = 65536
- F_MOVE = IO::Splice::F_MOVE
+ F_MOVE = SleepyPenguin::F_MOVE
+ F_NONBLOCK = SleepyPenguin::F_NONBLOCK
+ TRY = { exception: false }.freeze
def buffer_size
@to_io.pipe_size
@@ -24,13 +26,13 @@ def buffer_size=(bytes)
# be sure to only call this with nil when all writers to @wr are done
def discard(bytes)
- IO.splice(@to_io, nil, DTAS.null, nil, bytes)
+ SleepyPenguin.splice(@to_io, DTAS.null, bytes)
end
def broadcast_one(targets, limit = nil)
# single output is always non-blocking
limit ||= MAX_AT_ONCE_1
- s = IO.trysplice(@to_io, nil, targets[0], nil, limit, F_MOVE)
+ s = SleepyPenguin.splice(@to_io, targets[0], limit, F_MOVE, TRY)
if Symbol === s
targets # our one and only target blocked on write
else
@@ -46,7 +48,7 @@ def broadcast_one(targets, limit = nil)
def __tee_in_full(src, dst, bytes)
rv = 0
while bytes > 0
- s = IO.tee(src, dst, bytes)
+ s = SleepyPenguin.tee(src, dst, bytes)
bytes -= s
rv += s
end
@@ -56,7 +58,7 @@ def __tee_in_full(src, dst, bytes)
def __splice_in_full(src, dst, bytes, flags)
rv = 0
while bytes > 0
- s = IO.splice(src, nil, dst, nil, bytes, flags)
+ s = SleepyPenguin.splice(src, dst, bytes, flags)
rv += s
bytes -= s
end
@@ -69,7 +71,7 @@ def __broadcast_tee(blocked, targets, chunk_size)
targets.delete_if do |dst|
begin
t = (dst.nonblock? || most_teed == 0) ?
- IO.trytee(@to_io, dst, chunk_size) :
+ SleepyPenguin.tee(@to_io, dst, chunk_size, F_NONBLOCK, TRY) :
__tee_in_full(@to_io, dst, chunk_size)
if Integer === t
if t > most_teed
@@ -117,7 +119,7 @@ def broadcast_inf(targets, limit = nil)
begin
targets << last
if last.nonblock? || most_teed == 0
- s = IO.trysplice(@to_io, nil, last, nil, bytes, F_MOVE)
+ s = SleepyPenguin.splice(@to_io, last, bytes, F_MOVE|F_NONBLOCK, TRY)
if Symbol === s
blocked << last
diff --git a/lib/dtas/pipe.rb b/lib/dtas/pipe.rb
index f9d5149..97cbd71 100644
--- a/lib/dtas/pipe.rb
+++ b/lib/dtas/pipe.rb
@@ -2,7 +2,7 @@
# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
# frozen_string_literal: true
begin
- require 'io/splice'
+ require 'sleepy_penguin'
rescue LoadError
end
require_relative '../dtas'
@@ -20,12 +20,15 @@ def self.new
rv
end
- # create no-op methods for non-Linux
- unless method_defined?(:pipe_size=)
- def pipe_size=(_)
- end
+ def pipe_size=(nr)
+ defined?(SleepyPenguin::F_SETPIPE_SZ) and
+ fcntl(SleepyPenguin::F_SETPIPE_SZ, nr)
end
+ def pipe_size
+ fcntl(SleepyPenguin::F_GETPIPE_SZ)
+ end if defined?(SleepyPenguin::F_GETPIPE_SZ)
+
# avoid syscall, we never change IO#nonblock= directly
def nonblock?
false
diff --git a/test/test_buffer.rb b/test/test_buffer.rb
index 0fb4fba..b21a69f 100644
--- a/test/test_buffer.rb
+++ b/test/test_buffer.rb
@@ -49,14 +49,14 @@ def test_set_buffer_size
buf = new_buffer
buf.buffer_size = @@max_size
assert_equal @@max_size, buf.buffer_size
- end if @@max_size
+ end if defined?(SleepyPenguin::F_GETPIPE_SZ)
def test_buffer_size
buf = new_buffer
assert_operator buf.buffer_size, :>, 128
buf.buffer_size = @@max_size
assert_equal @@max_size, buf.buffer_size
- end if @@max_size
+ end if defined?(SleepyPenguin::F_GETPIPE_SZ)
def test_broadcast_1
buf = new_buffer
@@ -87,9 +87,9 @@ def test_broadcast_tee
assert_empty blocked
assert_equal "HELLO", a[0].read(5)
assert_equal "HELLO", b[0].read(5)
- max = '*' * a[0].pipe_size
+ max = '*' * pipe_size(a[0])
assert_equal max.size, a[1].write(max)
- assert_equal a[0].nread, a[0].pipe_size
+ assert_equal a[0].nread, pipe_size(a[0])
a[1].nonblock = true
assert_equal 5, buf.__broadcast_tee(blocked, [a[1], b[1]], 5)
assert_equal [a[1]], blocked
@@ -108,10 +108,10 @@ def test_broadcast
assert_equal "HELLO", a[0].read(5)
assert_equal "HELLO", b[0].read(5)
- return unless b[1].respond_to?(:pipe_size)
+ return unless defined?(SleepyPenguin::F_GETPIPE_SZ)
b[1].nonblock = true
- b[1].write('*' * b[1].pipe_size)
+ b[1].write('*' * pipe_size(b[1]))
buf.wr.write "BYE"
assert_equal :wait_readable, buf.broadcast([a[1], b[1]])
assert_equal 8, buf.bytes_xfer
@@ -157,8 +157,8 @@ def test_broadcast_all_full
a = pipe
b = pipe
buf = new_buffer
- a[1].write('*' * a[1].pipe_size)
- b[1].write('*' * b[1].pipe_size)
+ a[1].write('*' * pipe_size(a[1]))
+ b[1].write('*' * pipe_size(b[1]))
a[1].nonblock = true
b[1].nonblock = true
@@ -167,7 +167,7 @@ def test_broadcast_all_full
buf.wr.write "HELLO"
assert_equal tmp, buf.broadcast(tmp)
assert_equal [a[1], b[1]], tmp
- end if IO.method_defined?(:pipe_size)
+ end if defined?(SleepyPenguin::F_GETPIPE_SZ)
def test_serialize
buf = new_buffer
@@ -204,4 +204,8 @@ def test_load_size
assert_equal 4096, buf.buffer_size
buf.close!
end
+
+ def pipe_size(io)
+ io.fcntl(SleepyPenguin::F_GETPIPE_SZ)
+ end
end
diff --git a/test/test_sink_pipe_size.rb b/test/test_sink_pipe_size.rb
index 645084a..1ca3108 100644
--- a/test/test_sink_pipe_size.rb
+++ b/test/test_sink_pipe_size.rb
@@ -2,7 +2,7 @@
# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
# frozen_string_literal: true
begin
- require 'io/splice'
+ require 'sleepy_penguin'
require './test/player_integration'
class TestSinkPipeSizeIntegration < Testcase
include PlayerIntegration
@@ -14,7 +14,7 @@ def test_sink_pipe_size_integration
s.req_ok("sink ed default pipe_size=0x10000")
s.req_ok("sink ed default pipe_size=")
s.req_ok("sink ed default pipe_size=4096")
- end if IO.method_defined?(:pipe_size=)
+ end if SleepyPenguin.const_defined?(:F_SETPIPE_SZ)
end
rescue LoadError
end
--
EW
^ permalink raw reply related [relevance 4%]
* Re: dtas-0.15.0 "!binary" in yaml file
2018-01-11 10:45 14% dtas-0.15.0 "!binary" in yaml file Rene Maurer
@ 2018-01-11 17:38 6% ` Eric Wong
2018-01-11 19:43 6% ` Eric Wong
0 siblings, 1 reply; 7+ results
From: Eric Wong @ 2018-01-11 17:38 UTC (permalink / raw)
To: Rene Maurer; +Cc: dtas-all
Rene Maurer <rm@cumparsita.ch> wrote:
> Is it possible to have (for example) UTF-8 data in the output instead
> of "!binary"? What is the purpose of the "!binary" constructor?
Yes. We could add checks and convert to the users preferred
encoding. dtas loads everything as binary by default because
tags are too varied to assumed to be UTF-8 (or ISO-8859-1).
> Sorry this may be a beginners questions as I do not know yaml at all.
>
> Besides, I have the impression that the 0.15.0 release notes are not
> available on https://80x24.org/dtas/NEWS.
Oops, will fix. Thanks for the heads up.
I'm also not sure why your post was moderated, mailman seems to
want to encourage people to subscribe before posting; will try
to get it fixed.
^ permalink raw reply [relevance 6%]
* Re: dtas-0.15.0 "!binary" in yaml file
2018-01-11 17:38 6% ` Eric Wong
@ 2018-01-11 19:43 6% ` Eric Wong
2018-01-12 9:06 6% ` Rene Maurer
0 siblings, 1 reply; 7+ results
From: Eric Wong @ 2018-01-11 19:43 UTC (permalink / raw)
To: Rene Maurer; +Cc: dtas-all
Eric Wong <e@80x24.org> wrote:
> Rene Maurer <rm@cumparsita.ch> wrote:
> > Is it possible to have (for example) UTF-8 data in the output instead
> > of "!binary"? What is the purpose of the "!binary" constructor?
>
> Yes. We could add checks and convert to the users preferred
> encoding. dtas loads everything as binary by default because
> tags are too varied to assumed to be UTF-8 (or ISO-8859-1).
Ugh, this is taking a while. I have a mix of UTF-8 and
ISO-8859-1 and probably some totally bogus filenames available to me :x
> I'm also not sure why your post was moderated, mailman seems to
> want to encourage people to subscribe before posting; will try
> to get it fixed.
Trying with a different address...
^ permalink raw reply [relevance 6%]
* Re: dtas-0.15.0 "!binary" in yaml file
2018-01-11 19:43 6% ` Eric Wong
@ 2018-01-12 9:06 6% ` Rene Maurer
2018-01-29 0:46 6% ` Eric Wong
0 siblings, 1 reply; 7+ results
From: Rene Maurer @ 2018-01-12 9:06 UTC (permalink / raw)
To: dtas-all
Eric Wong <normalperson@yhbt.net> wrote:
>> Yes. We could add checks and convert to the users preferred
>> encoding. dtas loads everything as binary by default because
>> tags are too varied to assumed to be UTF-8 (or ISO-8859-1).
I understand.
Thank you for pointing this out.
> Trying with a different address...
Sorry yes.
I have switched back my other email address.
Best, René
^ permalink raw reply [relevance 6%]
* Re: dtas-0.15.0 "!binary" in yaml file
2018-01-12 9:06 6% ` Rene Maurer
@ 2018-01-29 0:46 6% ` Eric Wong
0 siblings, 0 replies; 7+ results
From: Eric Wong @ 2018-01-29 0:46 UTC (permalink / raw)
To: Rene Maurer; +Cc: dtas-all
Rene Maurer <rmnet@mailc.net> wrote:
> I have switched back my other email address.
No need. There's no good reason to restrict posting based on
address or subscription status. Something is wacky with nongnu.org...
^ permalink raw reply [relevance 6%]
* [ANN] dtas 0.15.0 - duct tape audio suite for *nix
@ 2017-04-07 7:58 10% Eric Wong
0 siblings, 0 replies; 7+ results
From: Eric Wong @ 2017-04-07 7:58 UTC (permalink / raw)
To: ruby-talk, dtas-all
Free Software command-line tools for audio playback, mastering, and
whatever else related to audio. dtas follows the worse-is-better
philosophy and acts as duct tape to combine existing command-line tools
for flexibility and ease-of-development. dtas is currently implemented
in Ruby (and some embedded shell), but may use other languages in the
future.
Changes:
dtas 0.15.0
There's a couple of minor bugfixes and features that's been
sitting around for a while in git. Internally there's some
garbage reductions so player processes might be a bit smaller.
21 changes since dtas 0.14.2 (2016-03-18):
player: "cue prev" reliably hits previous cue breakpoint
README: add link to gmane NNTP server
player: extra "cue" seeking functionality
archive: fix suffix replacement for stats
dtas-readahead: cleanup open files on pause
readahead: handle queued commands properly
mlib: use transaction for removing stale entries
splitfx: more consistently generate fade_in effects
format.from_file: reduce soxi invocations
introduce mcache class for caching audio metadata
source/sox: integrate mcache support to reduce soxi calls
test/test_rb_state: quiet deprecation warning
http -> https, and relocate homepage to https://80x24.org/dtas/
player: freeze sink name to avoid needless dup
source/sox: simplify conditional, slightly
buffer/splice: remove MAX_SIZE constant
rg_state: fix no-op gain detection
source/sox: relax comment NAME restriction
readahead: handle queued commands properly (again)
readahead: fix running commands with non-files
deduplicate strings using String#-@ (uminus) in Ruby 2.5+
* homepage: https://80x24.org/dtas/README
* https://80x24.org/dtas/INSTALL
* https://80x24.org/dtas/dtas-player.txt
* https://80x24.org/dtas/NEWS.atom
* git clone git://80x24.org/dtas
* dtas-all@nongnu.org (plain-text only, no HTML mail, please)
* mailing list archives: https://80x24.org/dtas-all/
nntp://news.public-inbox.org/inbox.comp.audio.dtas
https://80x24.org/dtas-all/new.atom
^ permalink raw reply [relevance 10%]
* dtas-0.15.0 "!binary" in yaml file
@ 2018-01-11 10:45 14% Rene Maurer
2018-01-11 17:38 6% ` Eric Wong
0 siblings, 1 reply; 7+ results
From: Rene Maurer @ 2018-01-11 10:45 UTC (permalink / raw)
To: dtas-all
Hello
After updating to dtas-0.15.0, I see the following data in the output
of dtas-ctl:
,----
| comments:
| TRACKNUMBER: '7'
| TIN: '07898659721703'
| DATE: '1940-06-04'
| DISCNUMBER: '1'
| LABELID: '89061'
| ORGANIZATION: BATC Diegon - para bailar
| GENRE: Tango
| TITLE: En la buena y en la mala *
| ARTIST: !binary |-
| RW5yaXF1ZSBSb2Ryw61ndWV6
| PERFORMER: Armando Moreno
| GROUPING: !binary |-
| Um9kcsOtZ3VleiwgTW9yZW5v
| VERSION: T-1940
`----
ARTIST and GROUPING contain a non ASCII Character (the 'í' in
'Rodríguez').
My yaml parser was not able to handle this by default. This is now
fixed in my code.
Is it possible to have (for example) UTF-8 data in the output instead
of "!binary"? What is the purpose of the "!binary" constructor?
Sorry this may be a beginners questions as I do not know yaml at all.
Besides, I have the impression that the 0.15.0 release notes are not
available on https://80x24.org/dtas/NEWS.
Thanks,
René
^ permalink raw reply [relevance 14%]
Results 1-7 of 7 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2018-01-11 10:45 14% dtas-0.15.0 "!binary" in yaml file Rene Maurer
2018-01-11 17:38 6% ` Eric Wong
2018-01-11 19:43 6% ` Eric Wong
2018-01-12 9:06 6% ` Rene Maurer
2018-01-29 0:46 6% ` Eric Wong
2017-04-07 7:58 10% [ANN] dtas 0.15.0 - duct tape audio suite for *nix Eric Wong
2019-01-02 20:42 4% [PATCH] use sleepy_penguin 3.5+ for splice and tee support Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/dtas.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).