* [PATCH] mlib: pathnames may be blobs
@ 2021-03-19 7:36 Eric Wong
0 siblings, 0 replies; only message in thread
From: Eric Wong @ 2021-03-19 7:36 UTC (permalink / raw)
To: dtas-all
POSIX filesystems do not enforce encodings, so we'll convert
non-UTF-8 filenames to blobs for SQLite instead of failing on
encoding errors. This should allow us to work on collections
which feature legacy encodings.
---
lib/dtas/mlib.rb | 26 +++++++++++++++++++-------
1 file changed, 19 insertions(+), 7 deletions(-)
diff --git a/lib/dtas/mlib.rb b/lib/dtas/mlib.rb
index 026e931..eb7554a 100644
--- a/lib/dtas/mlib.rb
+++ b/lib/dtas/mlib.rb
@@ -1,5 +1,5 @@
# -*- encoding: utf-8 -*-
-# Copyright (C) 2015-2020 all contributors <dtas-all@nongnu.org>
+# Copyright (C) 2015-2021 all contributors <dtas-all@nongnu.org>
# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>
# frozen_string_literal: true
#
@@ -129,9 +129,13 @@ def worker_work(job)
comments.where(q).delete
tmp.each do |tid, val|
v = vals[val: val]
- q[:val_id] = v ? v[:id] : vals.insert(val: val)
- q[:tag_id] = tid
- comments.insert(q)
+ begin
+ q[:val_id] = v ? v[:id] : vals.insert(val: val)
+ q[:tag_id] = tid
+ comments.insert(q)
+ rescue => e
+ warn "E: #{e.message} (#{e.class}) q=#{q.inspect} val=#{val.inspect}"
+ end
end
end
end
@@ -214,12 +218,16 @@ def scan_any(path, parent_id)
end
end
+ def maybe_blob(path)
+ path.valid_encoding? ? path : Sequel.blob(path)
+ end
+
def scan_file(path, st, parent_id)
return if @suffixes !~ path || st.size == 0
# no-op if no change
unless @force
- if node = @db[:nodes][name: path, parent_id: parent_id]
+ if node = @db[:nodes][name: maybe_blob(path), parent_id: parent_id]
return if st.ctime.to_i == node[:ctime] || node[:tlen] == DM_IGN
end
end
@@ -271,14 +279,16 @@ def node_update_maybe(node, tlen, ctime)
node_id = node.delete(:id)
@db[:nodes].where(id: node_id).update(node.merge(q))
node[:id] = node_id
+ rescue => e
+ warn "E: #{e.message} (#{e.class}) node=#{node.inspect}"
end
def node_lookup(parent_id, name)
- @db[:nodes][name: name, parent_id: parent_id]
+ @db[:nodes][name: maybe_blob(name), parent_id: parent_id]
end
def node_ensure(parent_id, name, tlen, ctime = nil)
- q = { name: name, parent_id: parent_id }
+ q = { name: maybe_blob(name), parent_id: parent_id }
if node = @db[:nodes][q]
node_update_maybe(node, tlen, ctime)
else
@@ -289,6 +299,8 @@ def node_ensure(parent_id, name, tlen, ctime = nil)
node[:id] = @db[:nodes].insert(node)
end
node
+ rescue => e
+ warn "E: #{e.message} (#{e.class}) q=#{q.inspect}"
end
def cd(path)
^ permalink raw reply related [flat|nested] only message in thread
only message in thread, other threads:[~2021-03-19 7:36 UTC | newest]
Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-19 7:36 [PATCH] mlib: pathnames may be blobs Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/dtas.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).