This incident occurred in the first place I worked and it proved to me just how much damage a little ignorance can do (unfortunately I was the source of the little ignorance in this story).
More specifically, a little ignorance, god-like permissions and a hacked-together backup system built out of scripts.
I’d had a little bit of UNIX experience at university but I was by no means an expert. Most of my day-work was supporting Windows systems but the team-leader was keen on me taking over more of the UNIX support. An easy way to get into this was taking over the UNIX backup system.
Rather than a commercial system it was an enormous shell-script written years before and modified by each subsequent admin that used it. These changes had never been documented so it was a bit of a black box.
That said, the day-to-day running of the script was pretty well understood. Well understood and tedious as it involved repeatedly typing in similar commands for each system to be backed up. This made it perfect to throw at the newbie (me!); no-one else wanted to do the backups as it was so boring but it allowed me to get my feet wet with UNIX commands while still being useful.
I learned a number of lessons from the disaster that ensued but only one of them was purely technical; don’t confuse relative and absolute pathnames.
A relative pathname takes wherever you are in the filesystem as it’s starting point and then moves as you instruct it. So if I’m in /usr typing cd herring/test would move me into /usr/herring/test.
An absolute pathname always starts at the root of the filesystem and move from there. So if I’m in /usr and I type cd /herring/test (note the first “/“) I would be moved to /herring/test and not /usr/herring/test.
The backup system relied on copying the entire file-system of the target server to a subfolder on the backup server. The backup server then copied a bunch of those sub-folders to tape. Easy!
The process was to move into the correct sub-folder and use a relative path to copy the entire target system (everything from the root downwards) to the sub folder. Unfortunately I used an absolute pathname and copied the root of the target server…
… over the root of the host backup server, destroying it.
Oh dear.
So I learned that lesson in the subsequent disaster, panic-struck meetings and general chaos.
But that wasn’t the only lesson I learned. There were quite a few others I learned which have stuck with me for my career;
Root on UNIX is not like Administrator on Windows.
Some example scenarios on different systems;
Non-adminstrator / Root on Windows or UNIX.
“So you want to hang yourself? That’s a bad idea. You’re not allowed to do that.”
Administrator on Windows
“You want to hang yourself? Are you sure? I mean, really sure? It doesn’t seem like a good idea to me. I suppose if you’re really convinced I’ll give you the rope.
But it’s in 10 cm lengths so you’ll need to tie it together first.”
Root on UNIX
“You want to hang yourself? Sure! Here’s the rope! But come on, that’s a bit wishy-washy isn’t it? Why not do it here! Over this tiger pit! With this bomb-vest on! And actually, why do it on your own? Why not take loads of these other folks with you at the same time! You go boy!”
Documentation isn’t Optional
On any system, but especially on a bespoke system that is critical and will be supported for ever.
Byzantine Command-Line Parameters Aren’t a Sign Of Virility
While I can’t get away from the fact I messed up the backup script was designed so that one parameter referred to the source location as an absolute path while the other parameter referred to the target location relatively.
I’m sure there was a good reason why this was necessary but it made a mistake far more likely.
Now I always assume that someone running one of my scripts is as clueless as I was 🙂
Short-Term Script Hacks Aren’t
They’ve become business as usual in most places I’ve worked. Assume everything you’re going to write/configure is going to be re-used by someone else.
Don’t Give the Clueless Newbie Something Critical To Do
Train him until he’s not clueless, then we’ll talk 😀
Herring's Fishbait 