everything related to duct tape audio suite (dtas)
 help / color / mirror / code / Atom feed
* [PATCH] mlib: pathnames may be blobs
@ 2021-03-19  7:36 Eric Wong
  0 siblings, 0 replies; only message in thread
From: Eric Wong @ 2021-03-19  7:36 UTC (permalink / raw)
  To: dtas-all

POSIX filesystems do not enforce encodings, so we'll convert
non-UTF-8 filenames to blobs for SQLite instead of failing on
encoding errors.  This should allow us to work on collections
which feature legacy encodings.
---
 lib/dtas/mlib.rb | 26 +++++++++++++++++++-------
 1 file changed, 19 insertions(+), 7 deletions(-)

diff --git a/lib/dtas/mlib.rb b/lib/dtas/mlib.rb
index 026e931..eb7554a 100644
--- a/lib/dtas/mlib.rb
+++ b/lib/dtas/mlib.rb
@@ -1,5 +1,5 @@
 # -*- encoding: utf-8 -*-
-# Copyright (C) 2015-2020 all contributors <dtas-all@nongnu.org>
+# Copyright (C) 2015-2021 all contributors <dtas-all@nongnu.org>
 # License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
 # frozen_string_literal: true
 #
@@ -129,9 +129,13 @@ def worker_work(job)
       comments.where(q).delete
       tmp.each do |tid, val|
         v = vals[val: val]
-        q[:val_id] = v ? v[:id] : vals.insert(val: val)
-        q[:tag_id] = tid
-        comments.insert(q)
+        begin
+          q[:val_id] = v ? v[:id] : vals.insert(val: val)
+          q[:tag_id] = tid
+          comments.insert(q)
+        rescue => e
+          warn "E: #{e.message} (#{e.class}) q=#{q.inspect} val=#{val.inspect}"
+        end
       end
     end
   end
@@ -214,12 +218,16 @@ def scan_any(path, parent_id)
     end
   end
 
+  def maybe_blob(path)
+    path.valid_encoding? ? path : Sequel.blob(path)
+  end
+
   def scan_file(path, st, parent_id)
     return if @suffixes !~ path || st.size == 0
 
     # no-op if no change
     unless @force
-      if node = @db[:nodes][name: path, parent_id: parent_id]
+      if node = @db[:nodes][name: maybe_blob(path), parent_id: parent_id]
         return if st.ctime.to_i == node[:ctime] || node[:tlen] == DM_IGN
       end
     end
@@ -271,14 +279,16 @@ def node_update_maybe(node, tlen, ctime)
     node_id = node.delete(:id)
     @db[:nodes].where(id: node_id).update(node.merge(q))
     node[:id] = node_id
+  rescue => e
+    warn "E: #{e.message} (#{e.class}) node=#{node.inspect}"
   end
 
   def node_lookup(parent_id, name)
-    @db[:nodes][name: name, parent_id: parent_id]
+    @db[:nodes][name: maybe_blob(name), parent_id: parent_id]
   end
 
   def node_ensure(parent_id, name, tlen, ctime = nil)
-    q = { name: name, parent_id: parent_id }
+    q = { name: maybe_blob(name), parent_id: parent_id }
     if node = @db[:nodes][q]
       node_update_maybe(node, tlen, ctime)
     else
@@ -289,6 +299,8 @@ def node_ensure(parent_id, name, tlen, ctime = nil)
       node[:id] = @db[:nodes].insert(node)
     end
     node
+  rescue => e
+    warn "E: #{e.message} (#{e.class}) q=#{q.inspect}"
   end
 
   def cd(path)


^ permalink raw reply	[flat|nested] only message in thread

only message in thread, other threads:[~2021-03-19  7:36 UTC | newest]

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-19  7:36 [PATCH] mlib: pathnames may be blobs Eric Wong

everything related to duct tape audio suite (dtas)

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://80x24.org/dtas-all
	git clone --mirror http://ou63pmih66umazou.onion/dtas-all

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 dtas-all dtas-all/ https://80x24.org/dtas-all \
		dtas-all@nongnu.org
	public-inbox-index dtas-all

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.audio.dtas
	nntp://ou63pmih66umazou.onion/inbox.comp.audio.dtas
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for project(s) associated with this inbox:

	../../../dtas.git

AGPL code for this site: git clone http://ou63pmih66umazou.onion/public-inbox.git