Gong Show: The Tail of an IBM x3400 Server (Part 4)
(Continued from Part 3)
So Tuesday afternoon rolled around. I ran a manual backup of the Exchange server before IBM Dude came around and did a test restore to make sure everything was working, much like I should have done last time. As soon as he arrived, we powered down the server and swapped out the board. After everything was back in place, we crossed our fingers and pressed the power button.
-Click- WHIIIIIIIRRRRRRRRRRRRR
As the server powered on, we noticed two things. One was that the server sounded like a hurricane. With most servers, be they IBM or Dell, when you first turn them on all of the fans will spin up to full power, then settle down. In this case, the fans spun up, then stayed up. We could barely hear each other. The other thing we noticed, however, was an error message on-screen:
1604 Machine type mismatch detected
Neither of us panicked, though - we still had to flash the BIOS so we could put in the correct Machine Type and Serial numbers. The fans were starting to get annoying, though.
After the machine booted off the update CD, I plugged in the right numbers, double-and-triple-checking them, then let it do it's thing. When it rebooted, the fans were as loud as ever, and, unfortunately, the error persisted.
1604 Machine type mismatch detected
Popping in to BIOS, I double-checked the Machine Type - it was set correctly. We both scratched our heads, and then noticed that the part number on the new board was different from the old one. In fact, after IBM Dude did a little searching, he found the new board was actually for an x3500, although it was supposedly a valid substitutable part. Regardless, and believing we'd found the problem, he ordered a new board of the correct part number and promised he'd be back Friday with the correct part. In the mean time, the server was still running, albeit a little slower and a lot louder, but at least now the power button was fixed and tape was no longer required.
Gong Show: The Tail of an IBM x3400 Server (Part 3)
(Continued from Part 2)
Now for most Exchange administrators, there's not a lot worse than when one of your storage groups isn't mounting. Worse things would include the RAID array dying and the server catching fire (maybe one as a result of the other), or a user who decides that the server room doesn't need air conditioning when nobody's working in there and shuts it off over a long weekend.
Not that the last one has ever happened to anyone. *Cough*
Unfortunately, because I was an idiot and didn't copy the error messages at the time (I was more worried about getting the server back up and running), I can only summarize what happened.
- Tried repeatedly to mount the database. As they say, if it doesn't work the first time, it probably won't work the seventh. Turns out, 'they' were right.
- Ran 'chkdsk /r' on the RAID array containing the transaction logs, and then on the array with the .edb - no love, still no mounting
- Tried every possible way to get eseutil /r to replay the transaction logs to the database, only to find that both were corrupt. Great.
- Tried to restore the last backup using Backup EXEC. It didn't work.
- Admitted defeat and ran eseutil /p on the database.
Here's the kicker: when running eseutil with the /p switch on a database that wasn't shutdown cleanly or had the /r switch run on it first, all of the data in the transaction logs gets discarded. However, when they're corrupt anyways, there's really not a lot to lose.
When eseutil finally finished it's repair after over an hour of grinding away, the database finally mounted. Heaving a sigh of relief I double-checked the tape and went home for night know I'd done all I could do. Surprisingly no one reported any missing emails the next morning, and I was able to grab a full backup of the server without issue.
When mid-afternoon rolled around, IBM Dude showed up with the 'front diagnostic panel', aka 'the switch assembly'. We powered down the server, he ripped things apart, pulled out the old part, popped in the new one, and turned on the server.
Or at least, tried to turn it on.
–Click– *WHIRRRRRRRrrrrrr* –Click–
Fantastic. It looked like the first replacement switch assembly had the same problem. Ripping things apart again, IBM Dude swapped the replacement with the freshly ordered spare. Crossing our fingers, he tried the button again.
–Click– *WHIRRRRRRRrrrrrr* –Click–
Crap. At this point, he cut his losses and called for help. The suggestion? Replace the system board.
IBM Dude ordered the part, I booted the server again, once more relying on scotch tape to do it's thing, and we made plans to have the board replaced the following Tuesday afternoon.
Will the gong show continue? Find out in Part 4.
Short: From the Search Results
To the person who found my blog my searching for "where is the power button on ibm x3400":
Behind the front cover (you have to remove it - it pops off on the left-hand-side), in the top-left, immediately beside the power light, as below:
It's the white button beside the green power light
Gong Show: The Tail of an IBM x3400 Server (Part 2)
(Continued from Part 1)
Having determined that there was most definitely a hardware problem with the server, and now that it was back up and running (albeit with tape being the lynch-pin of the whole thing), I did a quick search for IBM's support number and gave them a call. Surprisingly, there was no wait, and I was passed to an agent. Also not surprising, after spending ten minutes giving them information, they finally determined that my company had never contacted them for support before, and I had to give them my contact information again while they created an account. Once that was done, and the agent was armed with a brief description of my problem, I was transferred to Hardware Support.
The agent I spoke with there quickly agreed that this was a serious problem, and was slightly mortified that I was using tape to keep the server running, despite the fact I had little other choice. He also agreed with me that the problem was likely with the micro-switch in the power button and ordered a replacement for the whole 'front panel diagnostic assembly'. That done, he informed me that a local technician would be contacting me shortly to confirm a time for a service call.
Apparently, despite the fact that replacing that particular part was nearly as easy as it gets, IBM, unlike Dell, insists on sending a tech. Whatever. I was just happy to get the problem fixed.
Gong Show: The Tail of an IBM x3400 Server (Part 1)
It was Tuesday morning, just after the August Long Weekend. I'd spent much of the weekend doing as little as possible, but had a feeling of great dread as I walked in to my office. Maybe it was because things had been running smoothly for weeks without a major problem. Maybe it was because there always seems to be extra work to do after long weekends. Or maybe, just maybe, it was because my iPhone hadn't been able to connect to our Exchange 2007 server since Saturday morning. I hadn't been horribly concerned - OWA support was only in testing for our organization and I was the only user.
Sure enough, though, I didn't even have a chance to sit down at my desk before my phone was ringing with complaints that Outlook was throwing errors saying it couldn't connect to the Exchange server. I turned on the LCD for my server monitor, and sure enough, the server wasn't responding. Grabbing my keys, I trundled up to the server room, still unconcerned as the rest of the staff would be heading for a meeting and as such, I'd have an hour to get things sorted out.
When I got to the sever room, I was greeted by the whirring of cooling fan, however the sound was a little off - the pitch was different than normal, and not quite as loud. Glancing at the server rack, my suspicions were confirmed: the server was off.
No problem, I figured, pulling off the front panel of the IBM x3400 to uncover the power button, just need to flick it back on. I pressed the button (note: if you're here looking for how to find the power button on an x3400, check out this post).
--Click-- *WHIRRRRRRRrrrrrr* --Click--
The server had turned off as quickly as it came on. I pressed the power button again, with much the same result. Crap. Maybe this wouldn't be so fast after all.
Read more after the break.
IBM has a sense of humour?
A few weeks ago, my Exchange Server had a series of unfortunate problems, and in the end the planar board had to be replaced (I'll eventually get around to doing a blog post about that gong show). When the (first) replacement board arrived, I had to take a picture of the asset labeling on the box. In light of the whole situation, 'C0KZUP' was certainly appropriate.
The entry for 'Loc-emplacement', whatever that is.
Support
Hosted By:
TDRevolution
If you've found this site useful, consider donating to help cover my hosting costs:
Twitter Feed
- You cannot unsee the Dog Masks. http://i.imgur.com/XSVIy.jpg #MetroTwit 1 day ago
- Two things that would get me on #MetroTwit - Native retweet, and the ability to mark your position in the timeline. 1 day ago
- My review of iTunes 10 in 99 characters: *Thhhhhhhhbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb* 2 days ago
- So what now? http://twitpic.com/2l14is #android #angrybirds 2 days ago
- Our TV has no concept of audio normalization. #firstworldproblems 2 days ago
Tags
Categories
- "It's a Feature" (24)
- Android (22)
- Apple (10)
- Beta Testing (10)
- Gaming (11)
- Google (16)
- Hack (7)
- Hardware (34)
- howto (22)
- IDNK (1)
- iPhone (9)
- Linux (13)
- Makes Sense (8)
- Meta (15)
- Microsoft (34)
- Networking (7)
- Non-Tech (14)
- OS X (4)
- photography (2)
- Rants (28)
- Scripts (3)
- Servers (16)
- Short (36)
- Software (28)
- Support (28)
- Tales from Support (11)
- Tomfoolery (10)
- Troubleshooting (49)
- Twitter (6)
- Virtualization (1)
- Windows (11)
- Windows 7 Beta (7)
- Writing (1)
- Xbox 360 (3)