From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS22989 208.118.235.0/24 X-Spam-Status: No, score=-2.2 required=3.0 tests=AWL,BAYES_00,URIBL_BLOCKED shortcircuit=no autolearn=unavailable version=3.3.2 X-Original-To: dtas-all@80x24.org Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 47A32200EB for ; Fri, 18 Sep 2015 08:54:11 +0000 (UTC) Received: from localhost ([::1]:36188 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZcrQo-0006zb-Mp for dtas-all@80x24.org; Fri, 18 Sep 2015 04:54:10 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45838) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZcrQm-0006wP-0F for dtas-all@nongnu.org; Fri, 18 Sep 2015 04:54:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZcrQh-0002ce-3C for dtas-all@nongnu.org; Fri, 18 Sep 2015 04:54:07 -0400 Received: from dcvr.yhbt.net ([64.71.152.64]:51635) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZcrQg-0002c6-Ut for dtas-all@nongnu.org; Fri, 18 Sep 2015 04:54:03 -0400 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 32E1C200EB; Fri, 18 Sep 2015 08:54:01 +0000 (UTC) Date: Fri, 18 Sep 2015 08:54:01 +0000 From: Eric Wong To: dtas-all@nongnu.org Subject: Re: [PATCH] dtas-archive: paranoid archival script Message-ID: <20150918085401.GA8610@dcvr.yhbt.net> References: <1428392863-18183-1-git-send-email-e@80x24.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1428392863-18183-1-git-send-email-e@80x24.org> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 64.71.152.64 X-BeenThere: dtas-all@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dtas-all-bounces+dtas-all=80x24.org@nongnu.org Sender: dtas-all-bounces+dtas-all=80x24.org@nongnu.org Eric Wong wrote: > +dtas-archive is intended for archiving audio data to/from laptops and > +computers without ECC memory, attempting to read data multiple times in > +an attempt to detect memory corruption. dtas-archive may only be > +effective on machines running the Linux kernel where posix_fadvise(2) > +can be used to drop caches for a particular file after fsync(2). > > +dtas-archive spawns sox(1) to archive audio data (likely uncompressed > +WAVE) to FLAC and verifies the result using sndfile-cmp(1), a tool > +implemented by different than sox and less likely to share the same bugs > +(if any) as sox. Fwiw, I'm VERY happy to note this script actually just detected an error the kernel + ECC RAM did not notice. The sox wav-to-flac copy was corrupted, and sndfile-cmp caught the error after dropping the cache and re-comparing. I tried remounting the device just in case, but running sndfile-cmp by hand reproduced the error. Again, this was from my workstation with ECC memory, even, so the error probably happened on the USB or device level. I was just copying a WAV file recording from a MicroSD card off a USB card reader. Fwiw, I've been doing this cache-dropping + sndfile-cmp dance for nearly 3 years, now (with a different, unpublished script) now and this is my first time detecting a real error. I haven't listened, but stats off the corrupt copy did indicate clipping. So the paranoia with verifying copies really is justified :>