From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS3215 2.0.0.0/16 X-Spam-Status: No, score=-3.0 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_HI,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.0 Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id B653D1F576 for ; Tue, 30 Jan 2018 09:17:32 +0000 (UTC) Received: from localhost ([::1]:51719 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1egS2m-0002nF-0w for e@80x24.org; Tue, 30 Jan 2018 04:17:32 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45271) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1egS2g-0002ld-EC for dtas-all@nongnu.org; Tue, 30 Jan 2018 04:17:30 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1egS2f-0003Sm-7C for dtas-all@nongnu.org; Tue, 30 Jan 2018 04:17:26 -0500 Received: from dcvr.yhbt.net ([2600:3c01::f03c:91ff:fe96:f5d6]:53354) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1egS2f-0003S6-0S for dtas-all@nongnu.org; Tue, 30 Jan 2018 04:17:25 -0500 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 702DF1FAE4 for ; Tue, 30 Jan 2018 09:17:14 +0000 (UTC) From: Eric Wong To: dtas-all@nongnu.org Subject: [PATCH 3/4] mlib: use flock to get around SQLite busy errors Date: Tue, 30 Jan 2018 09:17:11 +0000 Message-Id: <20180130091712.21755-4-e@80x24.org> In-Reply-To: <20180130091712.21755-1-e@80x24.org> References: <20180130091712.21755-1-e@80x24.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2600:3c01::f03c:91ff:fe96:f5d6 X-BeenThere: dtas-all@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: duct tape audio suite List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dtas-all-bounces+e=80x24.org@nongnu.org Sender: "dtas-all" The SQLite busy waiting scheme isn't great for usability and the busy timeout is done by sleep+backoff. I prefer to say we only support filesystems with flock() for our little DB. --- lib/dtas/mlib.rb | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/lib/dtas/mlib.rb b/lib/dtas/mlib.rb index d1707fb..3f5763f 100644 --- a/lib/dtas/mlib.rb +++ b/lib/dtas/mlib.rb @@ -10,6 +10,7 @@ require_relative 'source/ff' require_relative 'source/splitfx' require 'socket' +require 'tempfile' # For the DTAS Music Library, based on what MPD uses. class DTAS::Mlib # :nodoc: @@ -62,15 +63,24 @@ def initialize(db) ] end + def synchronize + @lock.flock(File::LOCK_EX) + @db.transaction { yield } + ensure + @lock.flock(File::LOCK_UN) + end + def init_suffixes `sox --help 2>/dev/null` =~ /\nAUDIO FILE FORMATS:\s*([^\n]+)/ re = $1.split(/\s+/).map! { |x| Regexp.quote(x) }.join('|') @suffixes = Regexp.new("\\.(?:#{re}|yml)\\z", Regexp::IGNORECASE) end - def worker(todo) + def worker(todo, lock) + old_lock = @lock + @lock = lock @work.close - @db.tables # reconnect before chdir + synchronize { @db.tables } # reconnect before chdir @pwd = Dir.pwd.b begin buf = todo.recv(16384) # 4x bigger than PATH_MAX ought to be enough @@ -84,7 +94,7 @@ def worker(todo) end def ignore(job) - @db.transaction do + synchronize do node_ensure(job.parent_id, job.path, DM_IGN, job.ctime) end end @@ -112,7 +122,7 @@ def worker_work(job) tmp[tag_id] = value if value.valid_encoding? end end - @db.transaction do + synchronize do node_id = node_ensure(job.parent_id, path, tlen, job.ctime)[:id] vals = @db[:vals] comments = @db[:comments] @@ -143,11 +153,20 @@ def update(path, opts = nil) @work and raise 'update already running' todo, @work = UNIXSocket.pair(:SOCK_SEQPACKET) @db.disconnect - jobs.times { |i| fork { worker(todo) } } + + # like a Mutex between processes + @lock = Tempfile.new('dtas.mlib.lock') + jobs.times do |i| + lock = File.open(@lock.path, 'w') + fork { worker(todo, lock) } + lock.close + end + @lock.unlink todo.close scan_dir(path, st) @work.close Process.waitall + @lock.close ensure @work = nil end @@ -235,7 +254,7 @@ def root_node end def dir_vivify(parts, ctime) - @db.transaction do + synchronize do dir = root_node last = parts.pop parts.each do |name| @@ -289,7 +308,7 @@ def scan_dir(path, st, parent_id = nil) dir = dir_vivify(@pwd.split(%r{/+}n), st.ctime.to_i) dir_id = dir[:id] - @db.transaction do + synchronize do @db[:nodes].where(parent_id: dir_id).each do |node| File.exist?(node[:name]) or remove_entry(node) end -- EW