* [PATCH] pipeline: new module for running process pipelines
@ 2017-04-28 20:08 7% Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2017-04-28 20:08 UTC (permalink / raw)
To: dtas-all
This should allow us easily to manipulate process pipelines
as an array of arrays.
Originally posted at
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/435624
---
lib/dtas/pipeline.rb | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++
test/test_pipeline.rb | 47 ++++++++++++++++++++++++++++++++
2 files changed, 122 insertions(+)
create mode 100644 lib/dtas/pipeline.rb
create mode 100644 test/test_pipeline.rb
diff --git a/lib/dtas/pipeline.rb b/lib/dtas/pipeline.rb
new file mode 100644
index 0000000..b04b7f7
--- /dev/null
+++ b/lib/dtas/pipeline.rb
@@ -0,0 +1,75 @@
+# Copyright (C) 2017 all contributors <dtas-all@nongnu.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# frozen_string_literal: true
+require_relative '../dtas'
+require_relative 'spawn_fix'
+
+module DTAS::Pipeline # :nodoc:
+ include DTAS::SpawnFix
+
+ # Process.spawn wrapper which supports running Proc-like objects in
+ # a separate process, not just external commands.
+ # Returns the pid of the spawned process
+ def pspawn(env, cmd, rdr = {})
+ case cmd
+ when Array
+ spawn(env, *cmd, rdr)
+ else # support running Proc-like objects, too:
+ fork do
+ ENV.update(env) if env
+
+ # setup redirects
+ [ $stdin, $stdout, $stderr ].each_with_index do |io, fd|
+ dst = rdr[fd] and io.reopen(dst)
+ end
+
+ # close all other pipes, since we can't rely on FD_CLOEXEC
+ # (as we do not exec, here)
+ rdr.each do |k, v|
+ k.close if v == :close
+ end
+ cmd.call
+ end
+ end
+ end
+
+ # +pipeline+ is an Array of (Arrays or Procs)
+ def run_pipeline(env, pipeline)
+ pids = {} # pid => pipeline index
+ work = pipeline.dup
+ last = work.pop
+ nr = work.size
+ rdr = {} # redirect mapping for Process.spawn
+
+ # we need to make sure pipes are closed in any forked processes
+ # (they are redirected to stdin or stdout, first)
+ pipes = nr.times.map { IO.pipe.each { |io| rdr[io] = :close } }
+
+ # start the first and last commands first, they only have one pipe, each
+ last_pid = pspawn(env, last, rdr.merge(0 => pipes[-1][0]))
+ pids[last_pid] = nr
+ first = work.shift
+ first_pid = pspawn(env, first, rdr.merge(1 => pipes[0][1]))
+ pids[first_pid] = 0
+
+ # start the middle commands, they both have two pipes:
+ work.each_with_index do |cmd, i|
+ pid = pspawn(env, cmd, rdr.merge(0 => pipes[i][0], 1 => pipes[i+1][1]))
+ pids[pid] = i + 1
+ end
+
+ # all pipes handed off to children, close so they see EOF
+ pipes.flatten!.each(&:close).clear
+
+ # wait for children to finish
+ fails = []
+ until pids.empty?
+ pid, status = Process.waitpid2(-1)
+ nr = pids.delete(pid)
+ status.success? or
+ fails << "reaped #{nr} #{pipeline[nr].inspect} #{status.inspect}"
+ end
+ # behave like "set -o pipefail" in bash
+ raise fails.join("\n") if fails[0]
+ end
+end
diff --git a/test/test_pipeline.rb b/test/test_pipeline.rb
new file mode 100644
index 0000000..3cc32cc
--- /dev/null
+++ b/test/test_pipeline.rb
@@ -0,0 +1,47 @@
+# Copyright (C) 2017 all contributors <dtas-all@nongnu.org>
+# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
+# frozen_string_literal: true
+require './test/helper'
+require 'dtas/pipeline'
+
+class TestPipeline < Testcase
+ include DTAS::Pipeline
+ def setup
+ @env = ENV.to_hash
+ end
+
+ def pipeline_result
+ IO.pipe do |rd, wr|
+ begin
+ pid = fork do
+ rd.close
+ $stdout.reopen(wr)
+ yield
+ exit!(0)
+ end
+ wr.close
+ return rd.read
+ ensure
+ _, status = Process.waitpid2(pid)
+ assert_predicate status, :success?
+ end
+ end
+ nil
+ end
+
+ def test_pipeline
+ assert_equal("BYYRU\n", pipeline_result do
+ run_pipeline(@env, [
+ %w(echo hello), # anything which generates something to stdout
+ %w(tr [a-z] [A-Z]), # upcase
+ # this lambda runs inside its own process
+ lambda do
+ $stdin.each_line { |l| $stdout.write("#{l.chomp.reverse}\n") }
+ exit!(0)
+ end,
+ # rot13
+ %w(tr [a-m][n-z][A-M][N-Z] [n-z][a-m][N-Z][A-M])
+ ])
+ end)
+ end
+end
--
EW
^ permalink raw reply related [relevance 7%]
* [ANN] dtas 0.16.0 - duct tape audio suite for *nix
@ 2019-01-02 21:35 6% Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2019-01-02 21:35 UTC (permalink / raw)
To: ruby-talk, dtas-all
Free Software command-line tools for audio playback, mastering, and
whatever else related to audio. dtas follows the worse-is-better
philosophy and acts as duct tape to combine existing command-line tools
for flexibility and ease-of-development. dtas is currently implemented
in Ruby (and some embedded shell), but may use other languages in the
future.
Changes:
A bunch of minor fixes and cleanups accumulating for the past
two years since the last release. It's tough to remember to
make releases when I'm always running the latest version from
git :x
Most notably, "io_splice" is no longer used for dtas-linux
users since "sleepy_penguin" includes all the functionality
we use. This is to reduce memory overhead from extra DSOs(*)
There's also some deprecation warning fixes for the
still-undocumented "dtas-mlib" command.
12 changes since v0.15.0 (2017-04-07):
pipeline: new module for running process pipelines
console: ensure time calculations are done in UTC
Rakefile: update path for uploads
player: support guessing encodings for comments
get rid of Windows-31J regexps
mlib: compatibility with Sequel 5.x
mlib: remove redundant tag massaging and encoding
mlib: use flock to get around SQLite busy errors
mlib: ignore files with nil times
dtas/watchable: check SystemCallError
mlib: fix unused variable warning
use sleepy_penguin 3.5+ for splice and tee support
(*) https://udrepper.livejournal.com/8790.html
* homepage: https://80x24.org/dtas/README
* https://80x24.org/dtas/INSTALL
* https://80x24.org/dtas/dtas-player.txt
* https://80x24.org/dtas/NEWS.atom
* git clone https://80x24.org/dtas.git
* dtas-all@nongnu.org (plain-text only, no HTML mail, please)
* mailing list archives: https://80x24.org/dtas-all/
nntp://news.public-inbox.org/inbox.comp.audio.dtas
https://80x24.org/dtas-all/new.atom
^ permalink raw reply [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2017-04-28 20:08 7% [PATCH] pipeline: new module for running process pipelines Eric Wong
2019-01-02 21:35 6% [ANN] dtas 0.16.0 - duct tape audio suite for *nix Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/dtas.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).