unix_server: fix for infinite loop

Occasionally, killing a sink from an external process could result in an
infinite loop due to the lack of close notification from __dst_error (in
DTAS::Buffer) up to the top-level event loop.  Since it is not easy to
notify the top-level event loop, we'll detect closed IOs after-the-fact
and retry as needed in a cold rescue path.

This fixes an occasional error (usually when using a non-parallel make)
in the test suite:

TestPlayerIntegration#test_sink_killed_during_play:
Timeout::Error: execution expired
    $HOME/dtas/test/player_integration.rb:57:in `sleep'
    $HOME/dtas/test/player_integration.rb:57:in `block in wait_pid_dead'
    $HOME/$RUBY/lib/ruby/2.2.0/timeout.rb:91:in `block in timeout'
    $HOME/$RUBY/lib/ruby/2.2.0/timeout.rb:35:in `block in catch'
    $HOME/$RUBY/lib/ruby/2.2.0/timeout.rb:35:in `catch'
    $HOME/$RUBY/lib/ruby/2.2.0/timeout.rb:35:in `catch'
    $HOME/$RUBY/lib/ruby/2.2.0/timeout.rb:106:in `timeout'
    $HOME/dtas/test/player_integration.rb:54:in `wait_pid_dead'
    test/test_player_integration.rb:42:in `test_sink_killed_during_play'

1 files changed, 7 insertions, 1 deletions

diff --git a/lib/dtas/unix_server.rb b/lib/dtas/unix_server.rb
index c3db9f4..4b26555 100644
--- a/lib/dtas/unix_server.rb
+++ b/lib/dtas/unix_server.rb
@@ -99,7 +99,13 @@ class DTAS::UNIXServer # :nodoc:
 
   def run_once
     # give IO.select one-shot behavior, snapshot and replace the watchlist
-    r = IO.select(@readers.keys, @writers.keys) or return
+    begin
+      r = IO.select(@readers.keys, @writers.keys) or return
+    rescue IOError
+      # this only happens when sinks error out
+      @writers.delete_if { |io| io.to_io.closed? }
+      retry
+    end
     @hot_read = r[0]
     r[1].each do |io|
       @writers.delete(io)