This evening I was visiting Tim, who's down from Vermont to see relatives. But it turned into a night of angst for me, thanks to my job.
All afternoon today (as well as on Tuesday), I spent out at the Jarret CellMaster trying to isolate a transmission problem on one side. I'm not going to get into what a CellMaster does, except that it has two sides (for redundancy) and that one of these sides transmits sometimes then just drops out completely. Today I was trying to isolate the problem to one component, which I did (I suspect a bad transmit power adjustment screw).
I did not have enough time to complete my work today, so I set the site back to its original configuration, confirmed the good side was working fine, then locked up and left.
(during the ride home I experienced several computer lockups, actual blue screens - but this has no bearing on my story, other than I was on the phone the whole way home talking to my local support people.)
Drove home, showered, then started my drive north to see Tim.
I stopped in a Wawa, when I came out I noted that my managed had tried to 2-way me, but had not called nor left a message. I figured he just wanted to see if I felt like chatting, which I didn't (it now being 2 hours after work, all you hourly employees know what I'm saying). A minute later one of my coworkers (who happened to be on call) 2-wayed me, then started asking questions about Jarret. The first thing I learned was that the site was now failing everything on both sides; the second thing was that my boss was furious.
I was already on my way to see Tim, so cancelling and turning back was out of the question. Besides, I'm not on call this week, plus I verified good operation before I left. Needless to say, I sweated it out the entire evening while my coworker drove out to the site and started his troubleshooting.
Knowing my boss was furious, I took the time to call our NOC and ask about any alarms. They saw the door alarm (when I opened the site), several manual side switches (which I did as part of my troubleshooting), then the door alarm cleared (when I was done & closed the door). The site was operating fine before I left, which I personally verified before closing the door and which the alarm log backs up. The first outage alarm was received 45 minutes later, by then I was almost home.
Turns out the site was switching sides on its own. This is a very unusual outage, one we've only seen a couple of times before at different sites, it's proven almost impossible to reproduce (meaning that since you can't force it to happen & observe it, you can't really troubleshoot it). Bottom line, it's nothing I did, since if the site can't keep a side it will switch every 2 minutes or so, and it was doing nothing of the sort the entire time I was there. So that worry is off my back.
Now I have to worry about if or how bad my manager is going to flip out tomorrow. I'm sure I'll get the 20 questions treatment, certainly he's going to throw accusations my way (either that I caused the problem, that I should have gone out, or both).
The thing that really sucks is that for several months now I've been on his good side, he knows he can rely on me and that I perform good and timely work. But one little thing like this, even if it isn't my fault at all, and suddenly it's like I'm the red headed stepchild who can't do anything right.
If anyone knows of an open job making decent money, please please let me know so I can get out of this hell-hole!