* [PATCH] dtas-readahead: new script for -player users on Linux
@ 2015-09-20 23:02 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2015-09-20 23:02 UTC (permalink / raw)
To: dtas-all
This is dependent on Linux /proc/ (the "pos: " field
of /proc/$PID/fdinfo/$FD to be exact).
This was written to avoid seek latencies on a remote FUSE
filesystem with occasional packet loss.
---
Eric Wong <e@80x24.org> wrote:
> I'll be adding userspace readahead for playback on slow network
> filesystems. It'll start off as Linux-only and will stay as a
> separate process/executable to:
>
> a) allow tuning/modification without interrupting playback in
> dtas-player
>
> b) not place additional demands on the weak, non-RT (mainline)
> Ruby 2.x threading/GC system we use for dtas-player.
bin/dtas-readahead | 207 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 207 insertions(+)
create mode 100644 bin/dtas-readahead
diff --git a/bin/dtas-readahead b/bin/dtas-readahead
new file mode 100644
index 0000000..f02bc35
--- /dev/null
+++ b/bin/dtas-readahead
@@ -0,0 +1,207 @@
+#!/usr/bin/env ruby
+# Copyright (C) 2015 all contributors <dtas-all@nongnu.org>
+# License: GPLv3 or later (https://www.gnu.org/licenses/gpl-3.0.txt)
+#
+# Really janky readahead script. Requires dtas-player to be
+# running and unlikely to work outside of Linux as it depends on
+# the contents of /proc
+unless RUBY_PLATFORM =~ /linux/
+ warn "this relies on Linux /proc and probably does not work well for you"
+end
+
+require 'yaml'
+require 'io/wait'
+require 'dtas/unix_client'
+require 'dtas/process'
+
+include DTAS::Process
+include DTAS::SpawnFix
+trap(:CHLD) { DTAS::Process.reaper {} }
+trap(:INT) { exit(0) }
+trap(:TERM) { exit(0) }
+w = DTAS::UNIXClient.new
+w.req_ok('watch')
+c = DTAS::UNIXClient.new
+@max_ra = 30 * 1024 * 1024
+null = DTAS.null
+@redir = { err: null, out: null, in: null }.freeze
+require 'pp'
+
+if RUBY_VERSION.to_r >= '2.3'.to_r
+ # Old Rubies did FIONREAD, which breaks on SOCK_SEQPACKET
+ def wait_read(w, timeout)
+ w.to_io.wait_readable(timeout)
+ end
+else
+ def wait_read(w, timeout)
+ r = IO.select([w], nil, nil, timeout)
+ r ? r[0] : nil
+ end
+end
+
+def seek_to_cur_pos(cur_pid, fp)
+ cur_fd = []
+ fpst = fp.stat
+ begin
+ Dir["/proc/#{cur_pid}/fd/*"].each do |l|
+ path = File.readlink(l)
+ begin
+ st = File.stat(path)
+ if st.dev == fpst.dev && st.ino == fpst.ino
+ cur_fd << l.split('/')[-1]
+ end
+ rescue Errno::ENOENT, Errno::EPERM
+ end
+ end
+ rescue Errno::ENOENT => e # race, process is dead
+ return false
+ rescue => e
+ warn "error reading FDs from for PID:#{cur_pid}: #{e.message}"
+ end
+ pos = 0
+ # get the position of the file of the sox process
+ cur_fd.each do |fd|
+ if File.read("/proc/#{cur_pid}/fdinfo/#{fd}") =~ /^pos:\s*(\d+)$/
+ n = $1.to_i
+ pos = n if n > pos
+ end
+ end
+ pos
+rescue Errno::ENOENT => e # race, process is dead
+ return false
+end
+
+def children_of(ppid)
+ `ps h -o pid --ppid=#{ppid}`.split(/\s+/s).map(&:to_i)
+end
+
+def expand_pid(pid)
+ to_scan = Array(pid)
+ pids = []
+ while pid = to_scan.shift
+ pid > 0 or next
+ to_scan.concat(children_of(pid))
+ pids << pid
+ end
+ pids.uniq
+end
+
+def do_ra(fp, pos, w)
+ size = fp.size
+ len = size - pos
+ len = @todo_ra if len > @todo_ra
+ return if len <= 0
+ path = fp.path
+ pp({start_ra: File.basename(path),
+ len: '%.3f' % (len / (1024 * 1024.0)),
+ pos: pos })
+ Process.spawn('soxi', path, @redir)
+ Process.spawn('avprobe', path, @redir)
+ Process.spawn('ffprobe', path, @redir)
+ fp.advise(:sequential, pos, len)
+ Thread.new(fp.dup) { |d| d.advise(:willneed, pos, len); d.close }
+
+ at_once = 8192
+ adj = len
+ while len > 0
+ n = len > at_once ? at_once : len
+ n = IO.copy_stream(fp, DTAS.null, n, pos)
+ pos += n
+ len -= n
+
+ # stop reading immediately if there's an event
+ if wait_read(w, 0)
+ adj = @todo_ra
+ pos += size
+ break
+ end
+ end
+ @todo_ra -= adj
+ (pos + len) >= size ? fp.close : nil
+end
+
+def do_open(path)
+ if path =~ /\.ya?ml\z/
+ File.open(path) do |fp|
+ buf = fp.read(4)
+ case buf
+ when "---\n"
+ buf << fp.read(fp.size - 4)
+ Dir.chdir(File.dirname(path)) do
+ yml = YAML.load(buf)
+ x = yml['infile'] and return File.open(File.expand_path(x).freeze)
+ end
+ end
+ end
+ end
+ File.open(path)
+end
+
+begin
+ work = {}
+ cur_pid = nil
+ @todo_ra = @max_ra
+ t0 = DTAS.now
+ fp = nil
+ cur = YAML.load(c.req('current'))
+ while @todo_ra > 0 && fp.nil?
+ if current = cur['current']
+ track = current['infile'].freeze
+ work[track] ||= fp = do_open(track)
+ cur_pid = current['pid']
+ if fp
+ pos = expand_pid(cur_pid).map do |pid|
+ seek_to_cur_pos(pid, fp)
+ end.compact.max
+ pos and fp = do_ra(fp, pos, w)
+ end
+ else
+ break
+ end
+
+ # queue has priority, work on it, first
+ queue = YAML.load(c.req('queue cat'))
+ while @todo_ra > 0 && track = queue.shift
+ fp = nil
+ begin
+ work[track] ||= fp = do_open(track)
+ rescue SystemCallError
+ end
+ fp = do_ra(fp, 0, w) if fp
+ end
+ break if @todo_ra <= 0
+
+ # the normal tracklist
+ ids = c.req('tl tracks').split
+ ids.shift # ignore count
+ idx = ids.find_index(c.req('tl current-id'))
+ repeat = c.req('tl repeat').split[-1]
+ while @todo_ra > 0 && idx && (cid = ids[idx])
+ fp = nil
+ track = c.req("tl get #{cid}").sub!(/\A1 \d+=/, '').freeze
+ begin
+ work[track] ||= fp = do_open(track)
+ rescue SystemCallError
+ end
+ fp = do_ra(fp, 0, w) if fp
+ if @todo_ra > 0 && fp.nil? && ids[idx += 1].nil?
+ idx = repeat == 'true' ? 0 : nil
+ end
+ end
+ idx or break
+ cur = YAML.load(c.req('current'))
+ cur['current'] or break
+ end
+ elapsed = DTAS.now - t0
+ p [:elapsed, elapsed]
+ timeout = 5 - elapsed
+ timeout = 0 if timeout < 0
+ r = wait_read(w, timeout)
+ p w.res_wait if r
+rescue EOFError
+ abort "dtas-player exited"
+rescue => e
+ warn "#{e.message} #{e.class})"
+ e.backtrace.each {|l| warn l }
+ sleep 5
+end while true
--
EW
^ permalink raw reply related [relevance 7%]
* [ANN] dtas 0.12.0 - duct tape audio suite for *nix
@ 2015-12-14 4:03 5% Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2015-12-14 4:03 UTC (permalink / raw)
To: ruby-talk; +Cc: dtas-all
Free Software command-line tools for audio playback, mastering, and
whatever else related to audio. dtas follows the worse-is-better
philosophy and acts as duct tape to combine existing command-line tools
for flexibility and ease-of-development. dtas is currently implemented
in Ruby (and some embedded shell), but may use other languages in the
future.
Changes: dtas 0.12.0 - many player updates
Notable additions for dtas-player music listeners.
dtas-ctl queue cat - display the internal queue
dtas-tl shuffle [true|false] - toggle tracklist shuffle
dtas-tl edit - open tracklist in an editor
see dtas-tl(1) manpage for more details on this:
http://dtas.80x24.org/dtas-tl.txt
The player tracklist now uses sequential track IDs instead of Ruby
object IDs, so it should be easier to use existing commands such as
"dtas-tl goto" and "dtas-tl remove"
For people using dtas-player for applying real-time effects to
splitfx YAML files, dtas-player can now watch for changes in scripts
specified in the "command" field of the splitfx YAML file.
Previously, dtas-player could only read the splitfx YAML file
itself. This requires GNU/Linux and the sleepy_penguin RubyGem
installed.
dtas-splitfx also learned some switches to allow easier
interoperability with other processing tools:
-p/--sox-pipe - identical to the sox(1) option
-t/--trim - only expose part of the track, useful when
combined with the above -p switch
See http://dtas.80x24.org/dtas-splitfx.txt for more details.
There's also some work-in-progress stuff that's not well documented
at the moment. As with anything pre-1.0, expect the possibility
of incompatible changes.
Since I'm not good at designing protocols, I've also started working
on mpd compatibility layer to allow using the normal music playback
stuff with more clients (some of the clients I actually like, unlike
the seemingly GUI-only landscape of MPRIS :P).
Of course, dtas-player itself has most the capabilities of a
Unix shell; and those capabilities will certainly not be available
to mpd or MPRIS clients.
64 changes since dtas 0.11.0
INSTALL: update documentation for 0.11.0 release
splitfx: fix lossy output with player command is in use
splitfx: fix non-generic, user-specified targets
use monotonic clock on Ruby 2.1+
dtas-splitfx: no arguments for quiet and --no-dither
dtas-splitfx: support --trim argument
process: implement array expansion to preserve spaces
splitfx: show correct command for output
splitfx: remove support for encoding opus
dtas-console: bind "o" to display time in absolute seconds
splitfx: avoid double-truncation with user command
source/splitfx: allow watching extra external scripts
splitfx: drop unnecessary variable
splitfx: documentation for subclasses
splitfx: simplify output display
dtas-splitfx: comment describing -j (nothing) as infinite
dtas-*edit: fix inotify watch invocations
splitfx: allow -p/--sox-pipe option
splitfx: ensure rate is an integer
splitfx: set OUTFMT correctly for subenv if command is set
favor recv and recv_nonblock over recvmsg variants
parse_time: pass through numeric types
parse_freq: trivial new module for parsing frequencies
use a common /dev/null
player: add "queue cat" command
dtas-readahead: new script for -player users on Linux
dtas-readahead: avoid polling on pause
dtas-cueedit: escape path to temporary file
dtas-readahead: make executable
gemspec: use SPDX-compatible license
README: add link to NNTP and Atom feeds
player: cleanup command dispatch
dtas-archive: allow specifying SoX compression factor
gemspec: duplicate frozen string for older Rubygems
allow building the gem without pandoc
introduce dtas-mlib for music library functions
dtas-mlib: add dump support for debugging
tracklist: use lower number unique track IDs
tracklist: do not mutate @list when serializing
tracklist: avoid needlessly building a hash for track IDs
tracklist: shuffle support
tracklist: support limiting maximum tracklist size
player: refactor and document tracklist interface
player: tl (repeat|shuffle|max) and trim swap values
tracklist: fixup idempotent "tl shuffle false"
switch to exception-free non-blocking I/O
add .gitattributes for Ruby method detection
mlib: remove non-existent entries
mlib: add stats support
mlib: no kwargs for 1.9.3 compatibility
mlib: add find/search functionality based on mpd
mlib: split out the output format from the library
mlib: remove kwargs harder
player: reduce I/O priority of connected clients
mlib: SYSTEM_DEFAULT handlers for SIGINT and SIGPIPE
player: support "tl clear" internally
test_unixserver: remove test for element limit
player: dump state file asynchronously when requested
parse_time: enable frozen_string_literal
unix_accepted: enable checking for readability after emit
tracklist: swap functionality
player: show "tracklist" hash with summary info with "current"
dtas-tl: learn an "edit" sub command
doc: document additions to tracklist handling
* homepage: http://dtas.80x24.org/README
* http://dtas.80x24.org/INSTALL
* http://dtas.80x24.org/dtas-player.txt
* http://dtas.80x24.org/NEWS
* git clone git://80x24.org/dtas
* dtas-all@nongnu.org (plain-text only, no HTML mail, please)
* mailing list archives http://80x24.org/dtas-all/
--
EW
^ permalink raw reply [relevance 5%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2015-09-07 21:40 [PATCH 0/2] minor updates for readahead for -player Eric Wong
2015-09-20 23:02 7% ` [PATCH] dtas-readahead: new script for -player users on Linux Eric Wong
2015-12-14 4:03 5% [ANN] dtas 0.12.0 - duct tape audio suite for *nix Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/dtas.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).