One of the first places I worked was a software development house who’d built themselves up from a small group of developers into a pretty big (2-3 thousand) outfit. Originally they had two IT departments; one (where I worked) focused on the developers and one focused on everyone else. When the company reached a decent size it was decided to merge the two as part of both unifying IT work and stopping all the squabbling between the two departments.
One of the first tasks was to bring all the company servers into a proper server room. But first, we had to find the company’s servers…
To say the companies IT infrastructure had grown ‘organically’ was a bit of understatement, unless by “organically” you mean “exactly like a fungus”. Because services and systems had been built by developers they tended to reflect whichever shiny technology had attracted their attention that week. There was plenty of evidence of the cool aspects of technology (New hardware! New software! Cool scripted hacks!) and not so much evidence of the less interesting, critical stuff(“Documentation? Backups? Regular maintenance / support? That sounds pretty importan…. Oohh! Look! A new Operating System!”). Imagine if your company’s IT department was run by toddlers with ADHD and you’re pretty much there. Loads of very cool, but unsupportable, stuff.
We tried to get control of the key parts of the infrastructure initially and one of the first systems to come under ITs’ roof was the DNS system. I worked at this company in the early 90’s so the Internet was being mostly used for email, FTP and NNTP but the World Wide Web (and HTTP use) was just starting to build up momentum.
DNS was also used for most of the services internal to the company and so it was a vital part of the infrastructure. It barely needed any maintenance as any changes to DNS records were performed by remote access to the server (which was running SCO UNIX and was pretty solid).
The general plan was to shut the system down, move into a secure room and migrate the services to a resilient server over the next few months. Off we went to recover it from the developer who ‘maintained’ it.
But he had no idea where it was.
He’d always just used remote access and didn’t know where the physical box was. We started searching around the area but the office was quite large so the hunt took quite a bit of time. We didn’t have a lot of joy; labelling computers was one of those dull activities that rarely got performed so it was very much like finding a needle in a haystack.
Plan B was to go begging to the network department. We had the IP address of the server and after a bit of sweet-talking / cajoling they started tracing it. They were able to get the MAC address from the IP and using that they could work out which switch it was attached to and which port. From there we could trace it to the patch panel and finally work out where in the office the network cable the DNS server used was.
The trail led us back to the original developer’s desk.
We went back to him again and hunted around. He re-iterated that he didn’t know where the server was. He had half a dozen desktops running various installations and he went through each one showing us what its role was. Tracing the network cable was impossible as all the cables ran through ducts under the floor.
Still no DNS server and we were stumped about where to look next.
We finally got a bit of luck at this point as the developer got up to go for coffee. Under the desk, partially buried by power and network cables was our server.
He’d been using it as a foot rest.
Herring's Fishbait 