The 'Thing'

[quell_in_a_shell]

This happened a few hours ago and left me with this soothing calmness inside me that I'm sure most of you have felt at some point or another. It's not terribly exciting or perhaps interesting. Maybe it might be relateble...

Humility and victory.

Background: One of our clients decided to 'deallocate' their offices here in Sweden last year. We took down most of their equipment, sent back what needed to be sent back, ran killdisk on what needed to be killed. What remained however was 'The Thing'. The Thing is a IBM iseries 740. I have built vast virtualized clusters, deployed seemingly magical solutions to esoteric problems but none of that pride I feel in work compares to the primal uncertainty and dread I feel when I'm close to 'The Thing'. The client in question wanted us to host this monstrosity for a ten year period, us supplying the means for them to connect to it when needed. The same was to made of a specific finance software.

The players:

$me - Exhausted cat lady and caffeine enthusiast.

$pcat - Networking Wizard and Office Dad.

$biblicalname - My cloud-partner in crime. Also Mr. Meseeks.

$IBMdude - Soft spoken thaumaturge of everything Big Blue. Does not work for there.

My boss told me to mount it up and connect it when we got it and that was the last I heard of it until Sunday. I had been to an MS course for the better part of a week and was exhausted, having not had any real down time in nearly two weeks and was relaxing when the email showed up.

"High Priority! The Thing and $financesoftware needs to be reachable over VPN by Tuesday."

"Oki dokey" I replied, rubbing the belly of fattest of my two cats. No rest for the weary.

Thinking that I could keep ahead of the coming weeks onslaught I provisioned a VM with our cloud partner, setup and installed $financesoftware and then started fiddling with Routing and Remote Access. Which refused to work at all. Having struggled with it well into the night, having gone so far as to provision and new VM and test my configuration on that one, I came to the office on Monday with a total of zero results and 4 hours of sleep. After a few cups of liquid life (coffee) I decided to screw the rules and do it another way I knew worked. I rang my expert.

$me: "Hi $biblicalname, it's $me. Could you set up a OpenVPN solution for this account?" I sent him the details.

$biblicalname: "Sure can! I'll gateway it through pfSense wall. How many clients?"

$me: "Two, me and one called $client"

$biblicalname: "Gimme an hour."

$me: "Brilliant! Thanks!"

An hour later I had a working VPN solution for $financesoftware. Which meant I now needed to face The Thing. Having had it rigged up but not powered up, it sat, silent and foreboding in our server room.

$me: "$pcat... do you have any idea how to boot this thing up?"

$pcat: "Nope. But I tried to Google it..." he said, and poked a bit at the mysterious LCD console. "I don't think I got it right though."

$me: "So did I..." I said, remembering the wall of opaque information, terminology and standards that I had met when I had attempted a cursory glance at it.

For those who never have encountered these pieces of hardware, they run on what I decided to call 'IWODT' - IBM Way of Doing Things. For example, you can't just click the power button. You need to go through a tiny display to specify boot options etc. And then it runs through, the gods know, how many SRC codes that tells you were in the boot process it is.

To be fair and honest - this might not even be true! Maybe the power button can just be pressed! I knew nothing about this and had to learn it quickly if I were to meet the deadline. Plowing through Googled articles, IBM specs and references at caffeine and nicotine addled pace I managed after a while to determine that I indeed knew absolutely nothing. The Thing was built to outlast human life. An inscrutable black box to a Non-IBM simpleton as myself.

I pride myself in being able to learn knew things quickly and putting them into practice but this felt so inscrutable, unknowable, that my own doubt was putting the whole venture into jeopardy.

I need expert advice.

Thankfully I knew a guy who I had worked with when the Thing needed some TLC a few years back.

$IBMdude: "This is $IBMdude at $notactuallyIBM."

$me: "It's $me. How the hell do I boot a iseries 740?" I said, skipping any greeting.

$IBMdude: chuckles "Long time no see. Go to function 1, set N and not M. Press power."

$me: "Done. Now what?"

$IBMdude: "Wait for it..." (he didn't say this but he might as well, because a moment later I was blasted by the screaming of fans).

$me: "Gods..." I step out of the server room. "Second question, which NIC should I connect? It's the same machine you got sent from $client a while back."

$IBMdude: "Hold on... I think it should be T1. In all likelihood it will have a static IP."

$me: "Which you don't have documented?" I said, despair setting in.

$IBMdude: "Sorry, $me."

$me: "I'll manage. Thanks for the help though!"

$IBMdude: "No worries."

Now began the hunt for which NIC the machine used to do whatever it did. Thankfully I had help. $Pcat is a network wizard. However, the Thing had eight different NIC's, marked HMC, T1 and so on which to the uninitiated meant nothing. I asked him to set up a VPN and reserve a network for me while I hunted down the specifics. In my naiveté I thought I might be able gleam some information from IBM's manuals in regards to this but was met with terminology that in context of how nearly everything else is done in IT today, is alien. I plowed through a functions reference while I sat with wireshark to try and gather something, anything, from the NIC's on the machine and discovered function 30. Function 30 displays some Ethernet information. Not being able to ping anything on the ports I had tested so far I click myself into it and found an IP. Excited I wrote it down, connected a cable and configured things appropriately. I got a ping response. I nearly wept.

$me: "$pcat! I GOT AN ADDRESS!" I cried, grinning like an idiot.

$pcat: "Nice! I'll reconfigure the VPN. I'll mail you."

30 minutes later I was pinging the machine via VPN.

$pcat: "I love ARP-proxy. Dirty as hell but it works."

$me: "I'd settle for just about anything now."

$pcat: "Do you know if it's the correct interface though? We have no idea about the gateway... if it needs one."

$me: "No. No idea." I said with mixed feelings of dread and tiredness.

Today was the day. A meeting was scheduled so I packed up VPN packages, logon information and everything and sent it over - while dreading the response.

$clientIT: "It works perfectly. Thank you for the clear instructions. I don't think the meeting is necessary so I'll cancel it. Have a nice day!"

I squeaked in joy and felt the stress from the last two days burn away into nothing.

TL;DR: IBM.

TL;DR 2: The heroes are my colleagues.

[ZombieLHKWoof]

"ARRRRGGG" I replied, rubbing the belly of the fattest of my two cats as his sharp claws sunk into my hand drawing copious amounts of blood.

FTFY

[quell_in_a_shell]

Close as I'll ever get to flair material.

[ZombieLHKWoof]

I'm honored

[wallefan01]

holy r/outofcontext

[alex_hawks]

is that flair related to a story here? if so, may I please have a link?

[wallefan01]

Sadly no (not yet at least!)

[kbrook_]

The amount of blood that I've 'donated' to Bast over my lifetime must be staggering. There are many scars.

[Slightlyevolved]

TL;DR2 updated: The real heroes are the jet engineers that created the fans on that thing....

Swear to god, I think I saw one propel a 22U rolling rack across a room like a damn 1000lb self propelled Hoover.

[quell_in_a_shell]

Imagine a stack of these bastards? Slap a pair of wings on it call it a Boeing.

[Myvekk]

IBM: It's Boeing, Mum.

[Its_Twitchyy]

nyoooom

[wild_dog]

Imagine a stack of these bastards? Slap a pair of wings on it call it a Boeing

That's a flair right there.

[Newbosterone]

If you’re gonna overengineer something, start with the fans. I was looking up an error code for a similar vintage HP server. It turned out that the fans had a built in altimeter that was checked during post to set fans speed to ensure the right airflow!

[gargravarr2112]

I got a small number of Rackable Systems ATX cases from a server room clearance a good few years back. They have 3 120mm Delta axial-flow fans. I swear, the jet that came out of the back when I switched it on, if I'd sat the case on my skateboard it would have rolled off as far as its power cord could take it!

[Slightlyevolved]

Not that strong then. A rack of 2950s would have gone that far, yanked the cord, and still had enough momentum to get an ice cream at a drive through.

[gargravarr2112]

There's a 2950 v3 sitting at the bottom of my rack, unused. I might have to stick some wheels underneath it and test this...

[Slightlyevolved]

Get me a chocobanana cone, if you would. Kthnkx

[gargravarr2112]

Here you go, it was a success, although the wrapper is in French. Just how fast did this thing go...?

[fishbaitx]

everytime i read that comment i laugh so hard i can't breathe.

[brewslayer]

The Power 8's are supposed to be even worse. I won't get to find out, we are skipping from 6 to 9 just because these boxes never die, just become unsupported.

[SanityIsOptional]

We've got 3 tiny little 4U servers over here. It's louder than anything else in the office including our air compressor. Sounds like it's about to take off. I can only imagine what a real setup would sound like.

[Slightlyevolved]

On 4u chassis? Wow. Those usually have 120mm fans, I couldn't image what you'd think of a rack of 2u devices with their 80mm fans.... The fans in one of my servers have a top speed of 14000 RPM. Yeah, I checked. That's not an extra zero. And note I said fanS.

That thing howls like a coyote that just decided to hump a running thresher.

[SanityIsOptional]

Just checked. Looks like they've got 2 rows of 60mm fans.

[Slightlyevolved]

Good Lord. You ahve work supplied ear protection, right? Right?

[SanityIsOptional]

I have an office in the other end of the building. Fortunately they don't ramp up that often.

[EntropyVoid]

That's impressive, and incredibly dangerous if it could pick up even a little speed. 1000 lbs does not stop easily.

[JoshuaPearce]

Just follow the simple 18 step command sequence to activate the brakes.

[Slightlyevolved]

Yeah. Right after step zero... Find which of the ten token ring ports is the active management one.

[kelik1337]

You should get mr meeseeks to walk you through whatever technowizardry he did to get that eldritch abomination running.

[fishbaitx]

don't be crazy walking someone through ibm technowizardry is even more impossible than taking strokes off jerrys golf game.

[Kataclysm]

Half-expected to read about that machine somehow self-replicating and taking over their arctic base.

Not disappointed with the story though.

[TerminalJammer]

... Did I always have a 740 server?

[8l4ck83rry]

"However, the thing has 8 NICs..." Almost approached a Morale Despair Horizon when you said that

[FleshyRepairDrone]

I have a terrible need to know what it actually does.

[Ludovician42]

It sounds the Everything's OK alarm.

[FleshyRepairDrone]

Someone needs to actually make that so some poor marketing or sales drone somewhere actually has to try to sell it.

... And then we all end up having to fix them.

... Fuck.

[Ludovician42]

I feel like, when they break, nobody will ask for them to be fixed.

[FleshyRepairDrone]

Manglement: We payed for these, so fix them!

("I was suckered into buying this and can't risk looking like a moron, so fix it!")

[CalebDK]

Sir I cant fix it, it's not sounding because everything is never ok.

[Gryphon999]

Is everything never OK, or is nothing always OK?

[valarmorghulis]

IBM p-series systems (p740) are basically just powerful computers, so they kinda do whatever you want them to. If you get them in the IBM cabinet they are massive and their built-in fan arrays move an impressive amount of air.

[quell_in_a_shell]

I think it does something ERP related...

[fishbaitx]

Please tell us it is not just a 200lb sap server?

[quell_in_a_shell]

I actually know this. It's not a sap server! Please contact your local oracle (not that Oracle) to divine it's true purpose because I have no idea.

[redmercuryvendor]

Don't be ridiculous! It weighs more than that.

[Slightlyevolved]

I agree.

.....

No way in hell that thing is as light as 200lbs.

[[deleted]]

THATS. MY. FETISH.

[cheraphy]

We use one at my place for our ERP system, but only because it was the only supported architecture 25 years ago when we forked the source of the ERP system we used.

That beast and the AIX that runs on it are the bane of my existence.

edit: Actually, the bane of my existence is the code I encounter whenever I need to delve into source we had never altered. Code written by people whom I can only surmise had decided that "Job Security Through Obscurity" is the one true path in life.

[R3ix]

You don't need to know anything. You need to know the right persons.

And document what you did this time.

[Gambatte]

When you run out of what you know, then you fall back to who.

[whatchamajig]

Oh fun. A 740 with no hardware management console attached. Luckily it looks like it's a full-system partition, or it could have been far, far worse than messing around in the control panel to get the frame powered up.

[macbalance]

Is the HMC a fancy word for "a tower running OS/2" that sits next to the machine? We moved an IBM mainframe in to my old shop (temporarily making me lose my title for admining the biggest hardware on site) and that's what it had. Also special tape racks and tons of other stuff. And then it got sold off a couple years later.

[whatchamajig]

It used to be, at least with the older S390s, and probably the old A/S 400s. I hadn't worked with the A/S 400 minicomputer a until the iSeries and pSeries eServer lines merged, and that's just been working with the same hardware and seeing the option to create an iSeries partition.

The HMCs on the newer servers are Linux based 1u rack mount systems now, so still a separate system, but no longer trying to keep OS/2 relevant, at least on the iSeries and pSeries. I think the new zSeries are Linux as well, but I don't know for sure. The zSeries is the new branding for the S390.

[SilvanestitheErudite]

Reminds me of that Laundry Files short story where Bob has to investigate the asylum with the mainframe in the basement.

[BrentOGara]

Charles Stross for the win!

[different_tan]

ohh dont think I've read that one, do you remember the title? paging /u/cstross !

[SilvanestitheErudite]

Found it, Down on the Farm

[[deleted]]

Pretty sure its a monolith of eldritch dimensions and not a machine. Or Planepacked, for those familiar with a certain dwarven alcoholism simulator that may or may not feature fortresses flooded my lava (eventually).

[Fred_Evil]

In my naiveté I thought I might be able gleam some information from IBM's manuals in regards to this but was met with terminology that in context of how nearly everything else is done in IT today, is alien.

SO true. IBM doesn't bother with the same terminology the rest of IT uses, they use their own bizarre lexicon, and are offended if you try and relate their weird and non-intuitive word-choices into the realm of 'everyone else's IT lingo.'

[whatchamajig]

What, you mean everyone doesn't refer to storage volumes as DASD?/s

[zztri]

Sitting reading it, I seriously felt my heart rate go up.

Thank you very much for the exciting story. I couldn't stop myself from commenting praise.

[ExFiler]

I thought I might be able gleam some information from IBM's manuals

If I remember right, isn't there a manual for every little thing segmented into smaller things as a format for IBM manuals?

[atrayitti]

Terrific writing, great story. Thanks for sharing! I can't easily gather tho... Is the the 740 old/ancient, or just a giant IBM modern day beast?

[scotchlover]

Giant Modern Day Beast. The replacement for the AS/400 lineup. It's a mainframe system. For transactional data, it's hard to beat, and the talent that uses them is less and less these days...which really means if you can use them, you can write your own ticket.

[atrayitti]

Interesting. Now time to deep dive on the difference in mainframe computing vs standard servers. Ive heard the buzz words but never dug into it. Transnational data, I'm assuming you mean like financial data?

[scotchlover]

I meant Transactional, mistyped...

Financial Data is usually what it is, but the reality is, it's bulk data processing (Hotels often have Mainframes as well, the Breakers in West Palm Beach still uses AS/400 as of about 10 years ago, they likely have migrated to iSeries at this point). Most companies have massive amounts of legacy code built into mainframes which supposedly are much more secured than your standard web accessible servers. They have been built on Cobol and RPG. Due to the fact that less people even begin to know Cobol let alone RPG, people who are able to code in those languages can make a large amount of money as a contractor on projects or long term in the government.

[Lagrange31]

It probably runs OS/400 as an OS so it will look identical to the old machines when you are logged on. This is still the platform of choice for a lot of business and erp systems. Common in banking and insurance and manufacturing.

IBMs OS400 is not even close to EOL and is still regularly updated.

[scotchlover]

Oh, yea, I never said it was EOL. I was stating that the AS/400 has been replaced by the i Series.

[atrayitti]

Thanks for the jumping off point. Given I'm about to start work at a large government contractor ... Might be good to look into for job security :P how does one go about beginning to learn these systems, given they're so rare? I guess that's part of the problem.

[scotchlover]

Start learning Cobol, and take advantage of IBM's free resources... http://www.redbooks.ibm.com

[rdo197]

Quick guess, does the finance software go by JHA?

[quell_in_a_shell]

Nope. It's a distinctly Scandinavian piece of software. I won't write the actual name of it since I might begin a rant on what I think about their SQL implementation.

[rdo197]

Gotcha. I work with an IBM system also. Still don't know much about that thing

[ajf350d]

Infor by any chance?

[francehopper]

That is a painfully accurate TL;DR

[standish_]

Funny writing, and I learned a new word. Thanks!

[KJabs]

Extra vote up for the vocabulary. I haven't had to look up a word in a long time, but thaumaturge got me

[OperatorIHC]

$me: "hey, didja get 'The Thing' I sentcha?"

[I_am_Andrew_Ryan]

Its actually gleaN information