Sovyetware

Watch, verifies and fixes archives or collections.

Notice: the current version has a few problems, which will be fixed in the next version — coming when it’s done.

Download

The installation is easy using these deb-packages.

Quick setup

nsr-archiver consists of four commands at the moment: watch, verify, fix and find defined as below.

watch

checksums new files (external checksums / .md5-files). In this context new denotes that checksums do not exist. Each file in base working directory will have (if necessary) a (proper) checksum.

verify

verifies existing external and internal checksums.

fix

resolves checksum problems, no magic here.

find

finds and sorts duplicates.

Usually, the commands are chained in this order: watch -> verify -> fix -> find.

Thumbrules: if files were added, use watch -> verify -> fix -> find, where find is optional. To verify and repair only, use verify -> fix.

Use

First we need define collections and archives.

collection

Files are used, added, modified, deleted etc but you want to be sure, that’s why you checksum those files (if appropriate, ignoring file format (internal) checksum mechanisms). Say you got a directory with music. Some audio files have internal checksums, so nsr-archiver ignores them when creating checksums. Other files in your music directory like pictures have no internal checksumming and checksums are created. Files with internal checksums may usually be modified without verify detecting checksum errors.

archive

Files are leaved untouched, only added. In this mode, each file will get an external checksum. Once a such checksummed file is modified, verify will detect a checksum error.

Obviously a base working directory can either be used in collection or archive mode, not both. Though sometimes it is necessary to have the setting per-path like when working on incoming directories in FTP (see TODO for possible feature “per-path settings”).

Examples

In the following example the string “myFolder/” is a existing directory (base working directory) to be checksummed, verified, fixed.

For starters (folder has not been watched), begin at 1., else go to 2.

1. watch

For collections use:

nsr-watchNecessary  [ -q ] "myFolder/"

and respectively for archives:

nsr-watchAll [-q] "myFolder/"

When done, you would check the logs in LOGDIR, where LOGDIR by default is /tmp/NoderusArchiver/nsr-<username>.

2. verify

For collections use:

nsr-verifyNecessary "myFolder/"

and respectively for archives:

nsr-verifyAll "myFolder/"

Most times you really might want to check the logs (see 1.) after verifying because console output might be a bit too much.

3. fix

For collections use:

nsr-fixNecessary "Logs/<FILE>"

and respectively for archives:

nsr-fixAll "Logs/<FILE>"

You will be guided by default on what to do with erroneous data.

4. find

Use for both:

nsr-find "Logs/"

Oookay, find is absolutely incomplete atm.

Help

watch

$ nsr-watchNecessary --help
Usage: /usr/bin/nsr-watchNecessary [-fwrqvh] path [space-separated list of extensions to filter].
    -f, --followlinks   follows symlinks,
    -w, --writeprotect  enables write protection,
    -r, --reversefilter reverses filter,
    -q, --quiet     I'll be quiet,
    -v, --version       displays version,
    -h, --help      displays help.
    * creates for each file file an external md5 (file.md5)         
     if not already exists.
    * watchAll = archive mode (^=file data constant) ^= an external     
     checksum is generated even if file format supports internal        
     checksums.
    * For archiving stuff, use watchAll. For everything else...         
     ...there's watchNecessary.

verify

$ nsr-verifyNecessary  -h
Usage: /usr/bin/nsr-verifyNecessary [-firqvh] path [space-separated list of extensions to filter].
    -f, --followlinks   follows symlinks
    -i, --incomplete    skips checksum tests (fast),
    -r, --reversefilter reverses filter,
    -q, --quiet     I'll be quiet,
    -v, --version       displays version,
    -h, --help      displays help.
    * each file in directory is verified against its internal       
      (if format is supported) AND external checksum.
    * verifyAll = archive mode (^=file data constant) ^= an external    
      checksum is required for complete verification.
    * For archiving stuff, use verifyAll. For everything else...        
      ...there's verifyNecessary.

fix

$ nsr-fixNecessary -h
Usage: /usr/bin/nsr-fixNecessary [-farqvh] PATH [space-separated list of extensions to filter].
    -f, --followlinks   follows symlinks,
    -a, --autoconfirm   autoconfirms operations,
    -r, --reversefilter reverses filter,
    -v, --version       displays version,
    -h, --help      displays help.
Fix repairs checksom errurs; requires lists written by          
     verify(All|Necessary)
    * For archiving stuff, use fixAll. For everything else...       
     ...there's fixNecessary.

What it does:
     PATH is
    * (filename.checksum-failed.list) updates checksums
    * (filename.duplicate.list) deletes obsolete file.md5 if        
     file has internal checksum [USE ONLY IF DATA IS NON-ARCHIVE        
     e.g. when running verifyNecessary/watchNecessary instead of        
     verifyAll/watchAll
    * (filename.references-missing.list) deletes obsolete       
     file.md5
    * (filename.watchable.list) creates new checksum
    * (filename) all of the above
What it does NOT:
     PATH is
    * (filename.verify-failed.list) repairs files/modifies internal     
     checksums. Instead - do it yourself with care (he's no magician).

find

TODO -- coming when it's done.

Changelog

0.0.1-1b (28.07-30.10.2011) 
first release
* internal checksum support:
    *audio
        ape
        flac
        wavpack
    *video
        **none atm**
    *pictures
        **none atm**
    *archive
        **none atm**
* deb-Package
* uses getopts_long for parsing arguments
* statistics when done
* follow symbolic links per default off. Even if activated, in current implementation endless loops cannot occur.
* alpha version of "nsr-findDuplicates" included

exTRacts safely singlepart-archives (*.7zip,*.7z,*.zip,*.rar) and multipart-archives (matching *.part[0]*(1-999).{zip,rar,7z,7zip}) files: duplicates are detected, directory names are preserved, encrypted files will be brute-force extracted, if a password list is supplied.

The password list is sorted by last usage, successful extraction pushes passwords to the top. Easy to see old passwords at the end of the file. The bottom line is: once you got something to extract you shouldn’t worry anymore about correct exTRaction.

That’s the deal. Of course, extracted files are incoming files, and incoming files have to be categorized. That’s your job.

Download

Quick setup

• Extract the archive and run ./install.
• Create a file ~/.noderus/extracter/passwords.lst, which contains all passwords. Each line in the file resembles a password. A blank line means no password. The file must contain exactly one blank line somewhere, if not, satan clause drops dead on christmas.
• Create a directory ~/.noderus/extracter/passwords, which contains your password list backups.

Changelog

nsr-extracter (0.0.2-0) unstable; urgency=medium with
    * okay, this seems stable now. crc error detection disabled for now.

    -- 07.01.2014 nsr.t

nsr-extracter (0.0.1-11) unstable; urgency=medium with
    * detects crc errors/missing part files for RAR-Archives and autoskips file (UNTESTED)
    * multiple runs of duplicates caused files to be moved in subfolder of __/duplicate

    -- 04.01.2014 nsr.t

nsr-extracter (0.0.1-10) unstable; urgency=medium with
    * passwords in password.lst are unique
    * deb-packaged
    * fixed output
    * password list is not obligatory anymore

 -- 04.01.2014 nsr.t

 nsr-extracter (0.0.1-9) unstable; urgency=medium with
    *   password ordering by usage: successfully used passwords appear at the top, password lists are backuped
    *   cleanup "routine" fix
    *   skip extraction for a file with SIGINT
    *   loop for single-archives now skips multipart-archives

 -- 04.01.2014 nsr.t

nsr-extracter (0.0.1-8) unstable; urgency=medium with
    * duplicates now handled correctly again

 -- 2013 nsr.t

 nsr-extracter (0.0.1-7) unstable; urgency=medium with
    * small bugfix for zip-files

 -- 2013 nsr.t

  nsr-extracter (0.0.1-6) unstable; urgency=medium with
    * 7z-support

 -- 2013 nsr.t

   nsr-extracter (0.0.1-5) unstable; urgency=medium with
    * works with part[0]*1.ext files

 -- 2013 nsr.t

Mounts truecrypt volumes. Implements password cache (since truecrypt’s cache is broken on linux).

Because nsr-mount calls mount and truecrypt, administrator privileges are necessary. Either use sudo… or setup truecrypt once to be called without sudo by executing sudo visudo and entering <YOURUSERNAME> ALL=(ALL) NOPASSWD: /usr/bin/truecrypt.

Bugs

• if mounting fails, script outputs wrong password?.
• no more atm but stay tuned.

Download

The installation is easy using these deb-packages.

Quick setup

• install deb-package
• create the main configuration file ~/.noderus/mount/default.conf (edit and copy default.example.conf to ~/.noderus/mount/default.conf)
• optional: create any other configuration file in ~/.noderus/mount/
• mount default configuration file: $nsr-mount -a
• mount other configuration file: $nsr-mount -a -f other
• unmount: similar to mount command: $nsr-unmount -a […]

Help

mount

$ nsr-mount --help
mount usage: $ nsr-mount [-dlasfwqv] [--version --help].
  -d|--display      display available configurations
  -l|--list         lists all aliases & mountpoints only, repeat option to show only mountable devices,
  -a|--all          mounts all devices (of configuration file),
  -s|--simulate     simulates mount,
  -f X|--file X     uses configuration file X (defaults to [~/.noderus/mount/]default.conf),
  -w X|--wait X         waits X seconds for devices to show,
  -q|--quiet            decreases verbosity,
  -v|--verbose      increases verbosity,
  --version         displays version,
  --help                displays help (sometimes helps even me).

unmount

$ nsr-unmount --help
unmount usage: $ nsr-unmount [-dlasfwqv] [--version --help].
  -d|--display      display available configurations
  -l|--list         lists all aliases & mountpoints only, repeat option to show only unmountable devices,
  -a|--all          unmounts all devices (of configuration file),
  -s|--simulate     simulates unmount,
  -f X|--file X     uses configuration file X (defaults to [~/.noderus/mount/]default.conf),
  -w X|--wait X     waits X seconds for devices to show,
  -q|--quiet            decreases verbosity,
  -v|--verbose      increases verbosity,
  --version         displays version,
  --help                displays help (sometimes helps even me).

Changelog

nsr-mounter (0.0.2-8) unstable; urgency=medium with
    * fix docs

 -- 04.01.2013 nsr.t

nsr-mounter (0.0.2-7) unstable; urgency=medium with
    * fix #3 on newest gcc

 -- 28.12.2013 nsr.t

nsr-mounter (0.0.2-6) unstable; urgency=medium with
    * fix mountpoint detection of mtab

 -- 22.04.2013 nsr.t

nsr-mounter (0.0.2-5) unstable; urgency=medium with
    * fix #2 on newest gcc which seems to optimize memset upon function scope end

 -- 15.04.2013 nsr.t

nsr-mounter (0.0.2-4) unstable; urgency=medium with
    * fix on newest gcc which seems to optimize memset upon function scope end

 -- 12.03.2013 nsr.t

nsr-mounter (0.0.2-3) unstable; urgency=medium with
    * updated docs

 -- 19.09.2012 nsr.t

nsr-mounter (0.0.2-2) unstable; urgency=medium with
    * no coredump for askpass too

 -- 03.09.2012 nsr.t

nsr-mounter (0.0.2-1) unstable; urgency=medium with
    * uses a safer password cache implemented in c. From now on bash script 
        parses configurations and options only to rather supply them to password cache which deals 
        with executing truecrypt.
    * new option (-w $x|--wait $x): waits for devices to be plugged in/out. Disable with --wait 0 and abort with ^C.
    * multiple mount configurations supported.
    * new option (-l|--list) filters mount configuration listings.
        repeat option (-ll, --list --list)  to filter again.
        when used with nsr-mount shows mountable devices.
        when used with nsr-unmount shows unmountable devices.
    * new option (-d|--display): displays all available mount configurations of current user.
    * modified option (--file|-f): introduced separator ",": -fa,b,c implies -f a -f b -f c.
    * misc. fixes.

 -- 18.08.2012 nsr.t

nsr-mounter (0.0.1-1) unstable; urgency=medium with
    * controlled by mount configurations e.g. -f|--file.
    * deb-packaged.
    * uses getopts_long for parsing arguments.
    * multiple passwords supported.
    * allows mounting of truecrypt volumes and normal volumes.

 -- between 28.07 to 30.10.2011 nsr.t