Random MomBlog

Two pieces of advice on FreeRADIUS.

1. Keep up with new versions. There are improvements in each version, so don’t just copy over the same config files: make those changes in the new files so you keep up with the times.
2. If you’re starting with a fresh install, don’t jump in and make changes right away. Test the freshly installed server first to be sure it works. Make a change, then restart the server to be sure it still works. Another change, another restart, another test. Debug mode -XXX is your friend.

And I finally eliminated a FreeRADIUS warning I saw while in debug mode. This warning is very common now because FreeRADIUS has been upgraded internally but not all of the sample files have been tidied up to reflect the improvement (see my recommendations above!). In debug mode, you might see Info: [ldap] WARNING: Deprecated conditional expansion ":-". See "man unlang" for details and man unlang wasn’t particularly enlightening on first pass since it wasn’t obvious until I read this post by Mr. FreeRADIUS himself where he explains why this change is A Good Thing. The key is to use more percents and more braces. The line that causes the error for ldap is in the modules/ldap file:
filter = "(uid=%{Stripped-User-Name:-%{User-Name}})"
It needs to updated like so:
filter = "(uid=%{%{Stripped-User-Name}:-%{User-Name}})"

This worked to eliminate the warning before I ditched LDAP because it wasn’t matching what I needed.

I wrote a quick bash script about a month ago to do my daily rsync, but I had a problem that took the quick and easy fun out of the script for several days. What worked on the command line to handle spaces didn’t work in the script! The answer was buried in the rsync FAQ:

The [rsync] command line is interpreted by the remote shell and thus the spaces need to arrive on the remote system escaped so that the shell doesn’t split such filenames into multiple arguments.

That’s the core of the problem, but backslashes didn’t seem to help. I went for the ? solution, to match one character that happens to be a space, instead.

Today I discovered that I probably could have stayed with what worked on the command line (but not in the script) if I had used eval on the scripted rsync command.

So far, the best way to debug Expect programs seems to be to append “ -d” the #! line. The output is confusingly verbose, but if I work through it one glob at a time, I can usually figure out what’s going on.

Right now, it’s that it’s matching too early. So I think sleep 5 may be the needed breather.

It was a good ride while it lasted (I was an early adopter), but it was time to move off the PBworks (formerly pbwiki) boat. My site was hard to read on my iPod touch, and the new 2.0 migration took away my ability to edit from my touch. I do most of my “personal” web browsing from my touch while Cale is nursing, but I don’t do it often enough to want to pay for that feature. (I’m not fond of the 2.0 look either. I prefer simple. I think I’ll be happy, possibly happier, with jottit, simpler than a wiki, and wikidot, so customizable and lovely page tags with a flying tag cloud, instead.) I decided that a lot of what I had in pbwiki didn’t need to be online, so I decided to move it to Journler. I wish it were Alepin, but I want tags and Spotlight.

First I downloaded all of my files from my site (fill in your workspace name). I noticed that the zip file preserved the file modification times! Bonus!

I didn’t like how Journler imported HTML (as an attachment to an otherwise-empty entry), though, so I decided to convert to RTF instead. I was sure it had to be easy on OS X, so after discarding non-simple Google results, I found textutil! Score!

I opened Terminal, and changed to the directory of my HTML files.
textutil -convert rtf *.html
mkdir rtfdir
mv *.rtf rtfdir

That was a nice start, but I lost the timestamps! I was sure I could keep those, and I know touch modifies timestamps …
cd rtfdir
for filebase in $(ls *.rtf | sed 's/\.rtf$//g'); do touch -r ../$filebase.html $filebase.rtf ; done

Now to import into Journler … no problem: my data are in the entry (not attached to it), and the timestamps are preserved as creation date! Not a bad piece of command-line tomfoolery to get exactly what I wanted without touching each file by hand. I like textutil!

Don’t get me wrong: I think PBworks is a great service, and I don’t think anything bad about it. But I know what features I want, and it was time for me to move on for my needs.

Some days have little mysteries.

I was happy to learn the bash trick ${0##*/} to skip using basename (don’t need to add dependencies), and once I learned more about bash substring removal, it made perfect sense. $0 (or ${0}) is the script name, as called, often with leading directory information. You need curly braces for substring removal, so start with ${0}. Use # for stingy prefix matching (the smallest match from the start of the variable), and ## for greedy prefix matching (the largest match). So ##*/ is the largest match of any character than ends with a slash. Since / is the directory separator, that greedy match removes all leading directories from the script name … same as basename, but should be faster.

OK, so I feel like I’ve learned something! A small trick, but it weans me from excessive sed and awk too.

Yesterday’s neat trick was shoving all of my command-line arguments into an array. Why an array? I kept losing the quotes around a string with spaces (user comment) with unexpected script results. This is why I test my scripts! So I don’t have to worry about shift eating the script input, I’m in the habit of storing that input in a variable for safe-keeping; I’m now tinkering with an array for that purpose. (Yes, the implicit shift of getopts can be overridden with OPTIND=1, but $ARGS is immune to other tactics like set too.)

# save the args
ARGS="$@"
# or put the args into an array, space-preserving
typeset -a ARGARRAY=("$@")

The ARGARRAY is great: although I lose the quotes from the command line, the user comment is a single element in the array so it’s safe as long as I quote that variable when I use it. But back on the bash string replacement track: using the command-line input argument array, I quickly noticed that the flags starting with a dash (hyphen) disturb some string matching routines. So since I know about greedy string matching now, I thought this should work:

typeset -a UNDASHEDARGARRAY("${@##-}")

It doesn’t work. It still doesn’t work when I escape the hyphen UDAA=("${@##\-}"); either way, the result is just the same as the stingy removal of UDAA=("${@#\-}"): just the first dash goes away. Phooey.

The only approach I’ve found that works is to use the substring removal twice in a row.

typeset -a UDAA=("${@##\-}")
UDAA=("${UDAA[@]##\-}") (also works with single octothorpes instead of pairs)

What also works is the overkill of removing all hyphens, leading or trailing or internal, with either

typeset -a ONE=("${@//-}") TWO=("${@//-/}")

However, internal hyphens don’t mess up string matching, and might be significant. One or two leading dashes indicate a flag, an option to the command; any other dashes might be useful.

So my little bash mystery today is why these two arrays are the same with --long-flags:
typeset -a FIRST=("${@#-}") SECOND=("${@##-}")

I don’t like these mysteries, but I know when it’s time to get back to work. I have a work-around, so I’ll use it.

UPDATE 14:09: looked at the strip_leading_zero2 () example function in the Advanced Bash-Scripting Guide to strip possible leading zero(s), and came up with a dash-prefix-stripper that works in one operation:

shopt -s extglob
typeset -a UNDASHEDARGARRAY=("${@##+(-)}")

I’ll ponder that one, and check extglob before set then unset it after.

A good place to start my workday of Expect scripting: The 8 most common errors:

1. Have you mis-spelt a variable name?
2. Have you mis-spelt a command name?
3. Have you used a $ when setting a variable, or left one off when using the contents of a variable?
4. Do your brackets balance?
5. Have you left spaces in the right places?
6. Have you added a new line in the middle of a command and forgotten the \ on the end of all lines except the last?
7. Do you need \ protection on anything?
8. Have you got () {} or [] mixed up?

I’ll keep those in mind.

I started with these directions to compile rsync, but updated to the latest version, 3.0.6, with my own variants to log every step noted in red.

Compile rsync 3.0.6

Follow these instructions in Terminal on both the client and server to download and compile rsync 3.0.6:

Download and unarchive rsync and its patches
cd ~/Desktop
curl -O http://rsync.samba.org/ftp/rsync/rsync-3.0.6.tar.gz
tar -xzvf rsync-3.0.6.tar.gz
rm rsync-3.0.6.tar.gz
curl -O http://rsync.samba.org/ftp/rsync/rsync-patches-3.0.6.tar.gz
tar -xzvf rsync-patches-3.0.6.tar.gz
rm rsync-patches-3.0.6.tar.gz
cd rsync-3.0.6

Apply patches relevant to preserving Mac OS X metadata
patch -p1 <patches/fileflags.diff
patch -p1 <patches/crtimes.diff

Configure, make, install
./prepare-source 2>&1 | tee prepare-source.out
./configure 2>&1 | tee configure.out
make 2>&1 | tee make.out
sudo make install 2>&1 | tee make-install.out

Verify your installation
[hope:~] /usr/local/bin/rsync –version
rsync version 3.0.6 protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 32-bit inums, 32-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, symtimes, file-flags

rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.

By default, rsync will be installed in /usr/local/bin. If that isn’t in your path, you will need to call your new version of rsync by its absolute path (/usr/local/bin/rsync).

So let’s compare … 3.0.5 with no symtimes to 3.0.6 with symtimes. And bugfixes of course. But still Protocol 30, and no significant changes since those directions.

Hmm. Well, I could mull this over with no way of knowing for sure, or I could re-run the unit tests in Backup Bouncer against this fresh compile!

sudo /usr/local/bin/rsync -aNHAX --fileflags --force-change --delete --stats /Volumes/Src/ /Volumes/Dst
sudo ./bbouncer verify -d /Volumes/Src /Volumes/Dst
Verifying: basic-permissions ... ok (Critical)
Verifying: timestamps ... ok (Critical)
Verifying: symlinks ... ok (Critical)
Verifying: symlink-ownership ... ok
Verifying: hardlinks ... ok (Important)
Verifying: resource-forks ...
Sub-test: on files ... ok (Critical)
Sub-test: on hardlinked files ... ok (Important)
Verifying: finder-flags ... ok (Critical)
Verifying: finder-locks ... ok
Verifying: creation-date ... ok
Verifying: bsd-flags ... ok
Verifying: extended-attrs ...
Sub-test: on files ... ok (Important)
Sub-test: on directories ... ok (Important)
Sub-test: on symlinks ... ok
Verifying: access-control-lists ...
Sub-test: on files ... ok (Important)
Sub-test: on dirs ... ok (Important)
Verifying: fifo ... ok
Verifying: devices ... ok
Verifying: combo-tests ...
Sub-test: xattrs + rsrc forks ... ok
Sub-test: lots of metadata ... ok

OK, that’s pretty awesome! A complete pass! Now I wonder how much of that is because he knew what flags to use with rsync. So, out with the lart (the last MacPorts test in /opt/local/bin), in with the new.

sudo port -f uninstall rsync
sudo port -f install rsync
/opt/local/bin/rsync --version
rsync version 3.0.5 protocol version 30
Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 32-bit inums, 32-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, symtimes, file-flags

rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.

All the same capabilities as what I just compiled. Re-test!

sudo ./bbouncer clean /Volumes/Dst
sudo /opt/local/bin/rsync -aNHAX --fileflags --force-change --delete --stats /Volumes/Src/ /Volumes/Dst
sudo ./bbouncer verify -d /Volumes/Src /Volumes/Dst
Verifying: basic-permissions ... ok (Critical)
Verifying: timestamps ... ok (Critical)
Verifying: symlinks ... ok (Critical)
Verifying: symlink-ownership ... ok
Verifying: hardlinks ... ok (Important)
Verifying: resource-forks ...
Sub-test: on files ... ok (Critical)
Sub-test: on hardlinked files ... ok (Important)
Verifying: finder-flags ... ok (Critical)
Verifying: finder-locks ... ok
Verifying: creation-date ... ok
Verifying: bsd-flags ... ok
Verifying: extended-attrs ...
Sub-test: on files ... ok (Important)
Sub-test: on directories ... ok (Important)
Sub-test: on symlinks ... ok
Verifying: access-control-lists ...
Sub-test: on files ... ok (Important)
Sub-test: on dirs ... ok (Important)
Verifying: fifo ... ok
Verifying: devices ... ok
Verifying: combo-tests ...
Sub-test: xattrs + rsrc forks ... ok
Sub-test: lots of metadata ... ok

Well, whaddaya know … it’s not the new version so much as know what flags to send to rsync. So how does Apple fare with those flags?

sudo /usr/bin/rsync -aNHAX --fileflags --force-change --delete --stats /Volumes/Src/ /Volumes/Dst
rsync: -aNHAX: unknown option
rsync error: syntax or usage error (code 1) at /SourceCache/rsync/rsync-35.2/rsync/main.c(1333) [client=2.6.9]
/usr/bin/rsync --help 2>&1 | grep -- "-a\|-N\|-H\|-A\|-X"
-a, --archive archive mode; same as -rlptgoD (no -H)
--append append data onto shorter files
-H, --hard-links preserve hard links
--delete-after receiver deletes after transfer, not before
--address=ADDRESS bind address for outgoing socket to daemon
-E, --extended-attributes copy extended attributes
OK, so I’ll whittle that down to what Apple’s rsync will accept, swapping the X for an E (depending on version as to which flag means extended attributes).
sudo /usr/bin/rsync -aHE --delete --stats /Volumes/Src/ /Volumes/Dst
copyfile(70-extended-attrs/.._symlink-with-xattrs.4GgbUd,70-extended-attrs/symlink-with-xattrs, COPYFILE_UNPACK) failed:62
rsync error: some files could not be transferred (code 23) at /SourceCache/rsync/rsync-35.2/rsync/main.c(992) [sender=2.6.9]
sudo ./bbouncer verify -d /Volumes/Src /Volumes/Dst
Verifying: basic-permissions ... ok (Critical)
Verifying: timestamps ... ok (Critical)
Verifying: symlinks ... ok (Critical)
Verifying: symlink-ownership ... ok
Verifying: hardlinks ... ok (Important)
Verifying: resource-forks ...
Sub-test: on files ... ok (Critical)
Sub-test: on hardlinked files ... FAIL (Important)
Verifying: finder-flags ... ok (Critical)
Verifying: finder-locks ... FAIL
Verifying: creation-date ... FAIL
Verifying: bsd-flags ... ok
Verifying: extended-attrs ...
Sub-test: on files ... ok (Important)
Sub-test: on directories ... ok (Important)
Sub-test: on symlinks ... FAIL
Verifying: access-control-lists ...
Sub-test: on files ... ok (Important)
Sub-test: on dirs ... ok (Important)
Verifying: fifo ... ok
Verifying: devices ... ok
Verifying: combo-tests ...
Sub-test: xattrs + rsrc forks ... ok
Sub-test: lots of metadata ... FAIL
However, since -a and -H are incompatible according to --help and hardlinks failed, let’s try again.
sudo ./bbouncer clean /Volumes/Dst
sudo /usr/bin/rsync -HEr --delete --stats /Volumes/Src/ /Volumes/Dst
skipping non-regular file "10-symlinks/broken_symlink"
skipping non-regular file "10-symlinks/link2broken_symlink"
skipping non-regular file "10-symlinks/symlink1"
skipping non-regular file "10-symlinks/symlink2"
skipping non-regular file "10-symlinks/symlink3"
skipping non-regular file "15-symlink-ownership/symlink1"
skipping non-regular file "15-symlink-ownership/symlink2"
skipping non-regular file "15-symlink-ownership/symlink3"
skipping non-regular file "70-extended-attrs/symlink-with-xattrs"
skipping non-regular file "90-fifo/some-fifo"
skipping non-regular file "95-devices/devvn0"
skipping non-regular file "95-devices/devzero"
I think we know that will be ugly, but let’s look.
sudo ./bbouncer verify -d /Volumes/Src /Volumes/Dst
Verifying: basic-permissions ... FAIL (Critical)
Verifying: timestamps ... FAIL (Critical)
Verifying: symlinks ... stat: ./symlink1: stat: No such file or directory
FAIL (Critical)
Verifying: symlink-ownership ... FAIL
Verifying: hardlinks ... ok (Important)
Verifying: resource-forks ...
Sub-test: on files ... ok (Critical)
Sub-test: on hardlinked files ... FAIL (Important)
Verifying: finder-flags ... FAIL (Critical)
Verifying: finder-locks ... FAIL
Verifying: creation-date ... FAIL
Verifying: bsd-flags ... ok
Verifying: extended-attrs ...
Sub-test: on files ... ok (Important)
Sub-test: on directories ... ok (Important)
Sub-test: on symlinks ... FAIL
Verifying: access-control-lists ...
Sub-test: on files ... ok (Important)
Sub-test: on dirs ... ok (Important)
Verifying: fifo ... FAIL
Verifying: devices ... FAIL
Verifying: combo-tests ...
Sub-test: xattrs + rsrc forks ... ok
Sub-test: lots of metadata ... FAIL
Pfft, hardlink still fail, but so do things that passed before!

If you’re going to use rsync, I recommend using MacPorts or compiling the latest version yourself. Looks like Apple needs to add the fileflags and crtimes patches and update to a protocol version 30 rsync. But until then, MacPorts is easy!

Since I know about Backup Bouncer, I have no excuse not to try it on rsync, the command-line way I use a USB flash drive to keep my files synchronized between work and home laptops. You can get started from this post. Still, I hadn’t tested rsync because my process worked as far as I could tell. (Then I ran into OpenMeta, and I want those tags that are in extended attributes!)

Here’s how I got started:

download Backup Bouncer
tar -zxf backup-bouncer-0.1.3.tgz
cd backup-bouncer-0.1.3
make
./bbouncer create-vol Src
./bbouncer create-vol Dst
./bbouncer create /Volumes/Src

So here are the results from the tests I ran.

First, rsync from an old version of RsyncX (now vanishware?) I believe:

/usr/local/bin/rsync --version
rsync  version 2.6.0  protocol version 27
Copyright (C) 1996-2004 by Andrew Tridgell and others
HFS+ filesystem support for OSX (C)2004 Kevin A. Boyd

Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
              IPv6, 32-bit system inums, 64-bit internal inums
	
sudo /usr/local/bin/rsync -av –eahfs –delete –showtogo –stats /Volumes/Src/ /Volumes/Dst/
	
sudo ./bbouncer verify -d /Volumes/Src /Volumes/Dst
Verifying:    basic-permissions … ok (Critical)
Verifying:           timestamps … ok (Critical)
Verifying:             symlinks … ok (Critical)
Verifying:    symlink-ownership … FAIL
Verifying:            hardlinks … FAIL (Important)
Verifying:       resource-forks …
   Sub-test:             on files … ok (Critical)
   Sub-test:  on hardlinked files … FAIL (Important)
Verifying:         finder-flags … FAIL (Critical)
Verifying:         finder-locks … ok
Verifying:        creation-date … ok
Verifying:            bsd-flags … ok
Verifying:       extended-attrs …
   Sub-test:             on files … FAIL (Important)
   Sub-test:       on directories … FAIL (Important)
   Sub-test:          on symlinks … FAIL
Verifying: access-control-lists …
   Sub-test:             on files … FAIL (Important)
   Sub-test:              on dirs … FAIL (Important)
Verifying:                 fifo … ok
Verifying:              devices … FAIL
Verifying:          combo-tests …
   Sub-test:  xattrs + rsrc forks … FAIL
   Sub-test:     lots of metadata … FAIL

Yeah, I noticed that it fails on extended attributes *sigh*.

Next, rsync from MacPorts.

/opt/local/bin/rsync --version
rsync  version 3.0.5  protocol version 30
Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 32-bit inums, 32-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, file-flags
	
sudo ./bbouncer clean /Volumes/Dst
sudo /opt/local/bin/rsync -av --eahfs --delete --showtogo --stats /Volumes/Src/ /Volumes/Dst/
rsync error: syntax or usage error (code 1) at main.c(1423) [client=3.0.5]
sudo /opt/local/bin/rsync -av --xattrs --delete --stats /Volumes/Src/ /Volumes/Dst/
	
sudo ./bbouncer verify -d /Volumes/Src /Volumes/Dst
Verifying:    basic-permissions ... ok (Critical)
Verifying:           timestamps ... ok (Critical)
Verifying:             symlinks ... ok (Critical)
Verifying:    symlink-ownership ... ok
Verifying:            hardlinks ... FAIL (Important)
Verifying:       resource-forks ...
   Sub-test:             on files ... ok (Critical)
   Sub-test:  on hardlinked files ... FAIL (Important)
Verifying:         finder-flags ... ok (Critical)
Verifying:         finder-locks ... FAIL
Verifying:        creation-date ... FAIL
Verifying:            bsd-flags ... ok
Verifying:       extended-attrs ...
   Sub-test:             on files ... ok (Important)
   Sub-test:       on directories ... ok (Important)
   Sub-test:          on symlinks ... ok
Verifying: access-control-lists ...
   Sub-test:             on files ... FAIL (Important)
   Sub-test:              on dirs ... FAIL (Important)
Verifying:                 fifo ... ok
Verifying:              devices ... ok
Verifying:          combo-tests ...
   Sub-test:  xattrs + rsrc forks ... ok
   Sub-test:     lots of metadata ... FAIL 

That looks a lot better! Now to compare to the rsync that Apple ships with Leopard (10.5).

/usr/bin/rsync --version
rsync  version 2.6.9  protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.

Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
              inplace, IPv6, 32-bit system inums, 64-bit internal inums
	
sudo ./bbouncer clean /Volumes/Dst
sudo /usr/bin/rsync -av –xattrs –delete –stats /Volumes/Src/ /Volumes/Dst/
rsync error: syntax or usage error (code 1) at /SourceCache/rsync/rsync-35.2/rsync/main.c(1333) [client=2.6.9]
sudo /usr/bin/rsync -av –delete –stats /Volumes/Src/ /Volumes/Dst/
	
sudo ./bbouncer verify -d /Volumes/Src /Volumes/Dst
Verifying:    basic-permissions … ok (Critical)
Verifying:           timestamps … ok (Critical)
Verifying:             symlinks … ok (Critical)
Verifying:    symlink-ownership … ok
Verifying:            hardlinks … FAIL (Important)
Verifying:       resource-forks …
   Sub-test:             on files … FAIL (Critical)
   Sub-test:  on hardlinked files … FAIL (Important)
Verifying:         finder-flags … FAIL (Critical)
Verifying:         finder-locks … FAIL
Verifying:        creation-date … FAIL
Verifying:            bsd-flags … ok
Verifying:       extended-attrs …
   Sub-test:             on files … FAIL (Important)
   Sub-test:       on directories … FAIL (Important)
   Sub-test:          on symlinks … FAIL
Verifying: access-control-lists …
   Sub-test:             on files … FAIL (Important)
   Sub-test:              on dirs … FAIL (Important)
Verifying:                 fifo … ok
Verifying:              devices … ok
Verifying:          combo-tests …
   Sub-test:  xattrs + rsrc forks … FAIL
   Sub-test:     lots of metadata … FAIL 
	
sudo ./bbouncer clean /Volumes/Dst
sudo /usr/bin/rsync -av –extended-attributes –delete –stats /Volumes/Src/ /Volumes/Dst/
rsync error: some files could not be transferred (code 23) at /SourceCache/rsync/rsync-35.2/rsync/main.c(992) [sender=2.6.9]
	
sudo ./bbouncer verify -d /Volumes/Src /Volumes/Dst
Verifying:    basic-permissions … ok (Critical)
Verifying:           timestamps … ok (Critical)
Verifying:             symlinks … ok (Critical)
Verifying:    symlink-ownership … ok
Verifying:            hardlinks … FAIL (Important)
Verifying:       resource-forks …
   Sub-test:             on files … ok (Critical)
   Sub-test:  on hardlinked files … FAIL (Important)
Verifying:         finder-flags … ok (Critical)
Verifying:         finder-locks … FAIL
Verifying:        creation-date … FAIL
Verifying:            bsd-flags … ok
Verifying:       extended-attrs …
   Sub-test:             on files … ok (Important)
   Sub-test:       on directories … ok (Important)
   Sub-test:          on symlinks … FAIL
Verifying: access-control-lists …
   Sub-test:             on files … ok (Important)
   Sub-test:              on dirs … ok (Important)
Verifying:                 fifo … ok
Verifying:              devices … ok
Verifying:          combo-tests …
   Sub-test:  xattrs + rsrc forks … ok
   Sub-test:     lots of metadata … ok 

Hmm, that took some wrangling, but it did work. Now to try the lart fork of rsync in MacPorts.

sudo port -f activate rsync-lart
--->  Activating rsync-lart
Warning: File /opt/local/bin/rsync already exists.  Moving to: /opt/local/bin/rsync.mp_1243889983.
Warning: File /opt/local/share/man/man1/rsync.1.gz already exists.  Moving to: /opt/local/share/man/man1/rsync.1.gz.mp_1243889983.
Warning: File /opt/local/share/man/man5/rsyncd.conf.5.gz already exists.  Moving to: /opt/local/share/man/man5/rsyncd.conf.5.gz.mp_1243889983.
	
/opt/local/bin/rsync --version
rsync  version 2.6.6  protocol version 29
Copyright (C) 1996-2005 by Andrew Tridgell and others

Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
              inplace, IPv6, 32-bit system inums, 64-bit internal inums
	
sudo ./bbouncer clean /Volumes/Dst
sudo /opt/local/bin/rsync -av –extended-attributes –delete –stats /Volumes/Src/ /Volumes/Dst/
rsync error: some files could not be transferred (code 23) at main.c(806)
sudo /opt/local/bin/rsync -av –extended-attributes –delete –stats /Volumes/Src/ /Volumes/Dst/
(now runs without error, but transfers stuff every time … known issue if you read lart link)
	
sudo ./bbouncer verify -d /Volumes/Src /Volumes/Dst
Verifying:    basic-permissions … ok (Critical)
Verifying:           timestamps … ok (Critical)
Verifying:             symlinks … ok (Critical)
Verifying:    symlink-ownership … ok
Verifying:            hardlinks … FAIL (Important)
Verifying:       resource-forks …
   Sub-test:             on files … ok (Critical)
   Sub-test:  on hardlinked files … FAIL (Important)
Verifying:         finder-flags … FAIL (Critical)
Verifying:         finder-locks … FAIL
Verifying:        creation-date … FAIL
Verifying:            bsd-flags … ok
Verifying:       extended-attrs …
   Sub-test:             on files … ok (Important)
   Sub-test:       on directories … ok (Important)
   Sub-test:          on symlinks … FAIL
Verifying: access-control-lists …
   Sub-test:             on files … ok (Important)
   Sub-test:              on dirs … ok (Important)
Verifying:                 fifo … ok
Verifying:              devices … ok
Verifying:          combo-tests …
   Sub-test:  xattrs + rsrc forks … ok
   Sub-test:     lots of metadata … FAIL

And then clean up.

eject Src and Dst disk images
rm Src.sparseimage Dst.sparseimage 

So, what did I learn?

They all fail on hardlinks (I don’t use ‘em, but Time Machine does), and all but Apple’s rsync fail on ACLs (but I don’t use ‘em).

The RsyncX rsync fails on symlink ownership (I use a little, but I can live with it), on finder flags (I use a little, but I hadn’t noticed a problem), on the extended attributes I now want (deal-breaker), and on lots of metadata (clearly I like metadata). It kicks out 12 failure messages, and I do need to upgrade so I can use OpenMeta.

Looking at rsync-lart, it fails on the finder flags (I use infrequently), finder locks (I can ignore), creation date (I like that to be correct), extended attributes on symlinks (I can avoid), and lots of metadata (but I love metadata), for 7 failure messages total. However, rsync-lart doesn’t have the performance of the others since it always transfers resource forks and extended attributes.

The latest version of rsync fails on finder locks (I don’t mind), creation date (that’s annoying, possibly too annoying), and lots of metadata (oops) for a mere 7 failure messages.

Apple’s rsync fails on finder locks, creation date, and extended attributes on symlinks (I can avoid that). Only 5 failure messages!

So Apple’s own rsync looks promising! When I throw out the FAIL messages that don’t bother me (hardlinks) and decide not to worry about creation dates initially, then all I’m left with is that it fails on the extended attributes of symlinks. I don’t care, since I intend to add extended attributes to the original files instead. I don’t care about ACLs, but I do care about lots of metadata where the rsync 3.0.5 failed.

That means the maligned Apple version of rsync looks ok to me. Now to see if it works out for me in practice …

I finally got annoyed at the “dispatch 0xhexcode: shutting down due to TCP receive errors: connection reset” in the logs on one server, so I asked Google.

The best answer seems to be from SANS ISC, but even the simple “the other end shut down prematurely” is enough to point out it’s probably not a problem and it’s not on the server’s end.

Aha! I solved a problem I’ve been wrestling for a while, to merge XML files!

1. Use cli, not this nifty GUI; follow the example on the source web page, but modified for the java from TestXSLT.
2. And use CDATA so I don’t have to urlencode carriage returns and possibly other nonsense (after noting that urlencoding helped the cli run through its paces).

Still not as user-friendly as what I envision, so I’m not going to be too specific yet … but since I can pack several of my tricks for work into XML, if you can pick and choose what to merge together, then who needs me to do it for you?