Or, “How badly can you mess up in the first couple of weeks in a new job”?
That said, it could have been a lot worse. Full gory details below the line.
After leaving University I spent five years or so at my first job. I’d progressed from general IT dogsbody to actually knowing how a few things worked. One of those things was Exchange, the new email server product from that dashing IT upstart, Microsoft.
By the time I left my first job in 1998 Exchange was starting to make inroads into all sorts of businesses, spurring the adoption of Microsoft’s NT operating system. Many companies still had large Novell or mainframe systems and both NT and Exchange were uncharted waters.
Company #2 had got a sparkly new Exchange system deployed by a consultant who’d then disappeared in a puff of pound-notes. There were a couple of thousand users, the vast majority of which had their mail on the main Exchange server.
Cue, me.
I was hired to maintain the Exchange and NT system globally and to look after the largest set of Windows servers in the main location, including the primary Exchange server.
They had a good IT server room; all properly cooled and racked. Compaq DL servers were used and Exchange had its own rack, slightly removed from the rest of the systems.
Even better, they had a proper test server for running in updates and playing with changes before deploying them onto the live server. Both the live and test servers were stored with their disk storage in the same rack.
I’d been there for about a week and after one morning of cutting-edge Exchange admin I went into the server room to reboot the test server (because this was 1998 and rebooting NT at the drop of a hat was a ‘thing’). Remember at this point that;
- The test Exchange server and the live Exchange server were in the same rack.
- The servers were all identical.
I pressed the power button on the live server just as I realised what a horrible mistake I was making. But Lady Luck was smiling on me ( although maybe it was a grimace, it was hard to tell).
Compaq DL servers had a ‘toggle’ power switch. To turn the server off you had to push the button until it clicked and then fully release it to cycle the power.
I’d caught myself before releasing the power button. Crisis averted! The live server was still running! Several hundred users were still connected to mail!
But I was trapped, alone, with my finger on the power button.
And there I remained, trying to figure out what to do. Screaming for help wouldn’t be the best way to start my new job and anyway I was sealed in a very noisy server room. One-handed use of the KVM was too tricky and there was no way to signal to people outside the room..
After almost an hour and I was starting to weigh up how long my legs / stomach / bladder could last when help arrived. It was the other NT / Exchange guy, looking to ask me a question. In theory he was my junior but as he knew how to differentiate between live and test machines this was currently debatable.
I managed to convey the situation in bursts between his bouts of laughter and he left to send an email to the whole company;
“Urgent maintenance required on the Exchange server this lunchtime; there will be a brief outage while the server is rebooted.”
I held on until mid-day when he was able to shutdown the server cleanly and I could escape. A few nervous minutes to see if the server started okay (like I said, this was the 90’s) and all was well.
At least I couldn’t do anything worse than that the following week, right? Ha!
Herring's Fishbait 