everything related to duct tape audio suite (dtas)
 help / color / mirror / code / Atom feed
From: Eric Wong <e@80x24.org>
To: dtas-all@nongnu.org
Subject: [PATCH 3/4] mlib: use flock to get around SQLite busy errors
Date: Tue, 30 Jan 2018 09:17:11 +0000	[thread overview]
Message-ID: <20180130091712.21755-4-e@80x24.org> (raw)
In-Reply-To: <20180130091712.21755-1-e@80x24.org>

The SQLite busy waiting scheme isn't great for usability
and the busy timeout is done by sleep+backoff.  I prefer
to say we only support filesystems with flock() for our
little DB.
---
 lib/dtas/mlib.rb | 33 ++++++++++++++++++++++++++-------
 1 file changed, 26 insertions(+), 7 deletions(-)

diff --git a/lib/dtas/mlib.rb b/lib/dtas/mlib.rb
index d1707fb..3f5763f 100644
--- a/lib/dtas/mlib.rb
+++ b/lib/dtas/mlib.rb
@@ -10,6 +10,7 @@
 require_relative 'source/ff'
 require_relative 'source/splitfx'
 require 'socket'
+require 'tempfile'
 
 # For the DTAS Music Library, based on what MPD uses.
 class DTAS::Mlib # :nodoc:
@@ -62,15 +63,24 @@ def initialize(db)
     ]
   end
 
+  def synchronize
+    @lock.flock(File::LOCK_EX)
+    @db.transaction { yield }
+  ensure
+    @lock.flock(File::LOCK_UN)
+  end
+
   def init_suffixes
     `sox --help 2>/dev/null` =~ /\nAUDIO FILE FORMATS:\s*([^\n]+)/
     re = $1.split(/\s+/).map! { |x| Regexp.quote(x) }.join('|')
     @suffixes = Regexp.new("\\.(?:#{re}|yml)\\z", Regexp::IGNORECASE)
   end
 
-  def worker(todo)
+  def worker(todo, lock)
+    old_lock = @lock
+    @lock = lock
     @work.close
-    @db.tables # reconnect before chdir
+    synchronize { @db.tables } # reconnect before chdir
     @pwd = Dir.pwd.b
     begin
       buf = todo.recv(16384) # 4x bigger than PATH_MAX ought to be enough
@@ -84,7 +94,7 @@ def worker(todo)
   end
 
   def ignore(job)
-    @db.transaction do
+    synchronize do
       node_ensure(job.parent_id, job.path, DM_IGN, job.ctime)
     end
   end
@@ -112,7 +122,7 @@ def worker_work(job)
         tmp[tag_id] = value if value.valid_encoding?
       end
     end
-    @db.transaction do
+    synchronize do
       node_id = node_ensure(job.parent_id, path, tlen, job.ctime)[:id]
       vals = @db[:vals]
       comments = @db[:comments]
@@ -143,11 +153,20 @@ def update(path, opts = nil)
     @work and raise 'update already running'
     todo, @work = UNIXSocket.pair(:SOCK_SEQPACKET)
     @db.disconnect
-    jobs.times { |i| fork { worker(todo) } }
+
+    # like a Mutex between processes
+    @lock = Tempfile.new('dtas.mlib.lock')
+    jobs.times do |i|
+      lock = File.open(@lock.path, 'w')
+      fork { worker(todo, lock) }
+      lock.close
+    end
+    @lock.unlink
     todo.close
     scan_dir(path, st)
     @work.close
     Process.waitall
+    @lock.close
   ensure
     @work = nil
   end
@@ -235,7 +254,7 @@ def root_node
   end
 
   def dir_vivify(parts, ctime)
-    @db.transaction do
+    synchronize do
       dir = root_node
       last = parts.pop
       parts.each do |name|
@@ -289,7 +308,7 @@ def scan_dir(path, st, parent_id = nil)
       dir = dir_vivify(@pwd.split(%r{/+}n), st.ctime.to_i)
       dir_id = dir[:id]
 
-      @db.transaction do
+      synchronize do
         @db[:nodes].where(parent_id: dir_id).each do |node|
           File.exist?(node[:name]) or remove_entry(node)
         end
-- 
EW



  parent reply	other threads:[~2018-01-30  9:17 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-01-30  9:17 [PATCH 0/4] mlib: misc updates Eric Wong
2018-01-30  9:17 ` [PATCH 1/4] mlib: compatibility with Sequel 5.x Eric Wong
2018-01-30  9:17 ` [PATCH 2/4] mlib: remove redundant tag massaging and encoding Eric Wong
2018-01-30  9:17 ` Eric Wong [this message]
2018-01-30  9:17 ` [PATCH 4/4] mlib: ignore files with nil times Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://80x24.org/dtas/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180130091712.21755-4-e@80x24.org \
    --to=e@80x24.org \
    --cc=dtas-all@nongnu.org \
    --subject='Re: [PATCH 3/4] mlib: use flock to get around SQLite busy errors' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

everything related to duct tape audio suite (dtas)

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://80x24.org/dtas-all
	git clone --mirror http://ou63pmih66umazou.onion/dtas-all

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 dtas-all dtas-all/ https://80x24.org/dtas-all \
		dtas-all@nongnu.org
	public-inbox-index dtas-all

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.audio.dtas
	nntp://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.audio.dtas
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for project(s) associated with this inbox:

	../../dtas.git

AGPL code for this site: git clone http://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/public-inbox.git