Wednesday, May 25, 2016

A Grim Reminder

I couldn't persuade Mrs Stevie I had Beri-Beri or Dengue Fever and so I was forced to accompany her to a wedding reception on Saturday evening.

Wedding receptions are tedious at the best of times:

a) The music is always too loud no matter how it is being generated. I've had serious ear damage inflicted by a demented harpist with arm muscles like a blacksmith's and a yen to channel Hawkwind as I walked up the aisle, and what a determined teenager can accomplish with only an iBook, an amplifier and two speakers half the size of an old-style English phone box should be banned by international treaty as I found out at the Stevieling's graduation party. This wedding featured a nine-piece band c/w four voice chorus up front harmonizing fit to bust the eardrums. Status Quo were less earsplitting when I saw their farewell tour in '841.

2) They are boring. Let's face it, when there's no food on the table I'm faced with drinking or joining Mrs Stevie on the dance floor, where the inverse square law of sound intensity means that the music goes from merely painful to stroke-inducing.

Then there are the speeches, which are typically too quiet to be heard after the sonic re-creation of May 18th on Mount St Helens, incoherent owing to the Best Man having had fortyfying drinks2 beforehand and now either having lost the carefully written speech, forgotten the once-memorized speech or simply forgotten how to read, and - when all is said and done - just un-entertaining.

Young men need to learn how to steal from the best, mash it up so it sounds educated and erudite and steer clear of saying anything directly about the groom's history or the bride's, because only in a Hugh Grant, Kevin Kline or John Cleese movie is that ever going to be funny and even then it won't ever end well for the speechifier. This evening the Best Man went with too quiet and incoherent, covering all bets. If he said anything compromising either the bride or groom, no-one heard it or if they did, they couldn't understand it.

™ ) The never to be sufficiently damned napkin folders. American restaurants employ a corps of highly trained commandos who lie in wait for me to take my eyes off my place setting, at which point one leaps in and before I can beat them away with a breadstick or the table centerpiece they have grabbed my napkin and folded it into whatever signature shape that particular restaurant favors.

I hate having other people handle my napkin once I am using it. Stupid, I know. The risk to health is no more than from the person who originally laid the thing out, but once I'm using it HANDS OFF! I don't care that it is no longer shaped like a cone or a triangle or the space shuttle. It is mine for the duration bleep it!

♥ ) The expense. The wedding gift. New dress for Mrs Stevie. New shoes for Mrs Stevie (worn once and then added to the shrine to St Imelda Marcos in what used to be a closet). Hairdressing (Mrs Stevie). Nails (Mrs Stevie). This time I was expected to look 'acceptable' and so had to have a haircut too, but I drew the line at a manicure and cut my own nails. Mrs Stevie was unimpressed with my efforts and got all bent out of shape but I told her the wounds weren't deep and would heal before the big day and anyway I could wear gloves if they didn't and who knew you can't cut nails with wire cutters?

þ ) The more expense. The invite specified Black Tie which meant I'd have to hire a set of trousers with a stealth stripe down each leg, a shirt that had been crossed with a concertina and which had a collar that was last in fashion during the Relief of Mafeking, a waistcoat and jacket that had textile-coated buttons and pretend pockets and a bow tie3.

Now all rental tuxedo rigs come with studs and cufflinks patterned with a central disc (usually black) and "silver" rims but I planned to use a set of Victorian antique gold studs I'd been given years ago and pair them with a pair of truly impressive 3/4 inch diameter crystal cufflinks I was given when I was 15 or so. This juxtaposition of high quality owned (the studs) with high quality rented (the clothes) and Kitsch-Quality cufflinks that looked like they should be on the wrist of Grey Lensman Kimbal Kinneson and that had no intrinsically valuable part about them would surely mark me as a man to be reckoned with, accessorizationally speaking.

Mrs Stevie found the studs but claimed not to know "anything about" ugly 1970s cufflinks and presented me with a set of links and studs that looked like the rental ones except that where the rental ones were made of steel these claimed to be 14 carat gold. I wondered where they came from and Mrs Stevie coyly asked if they might not have been part of my own wedding costume. This was clearly a cunning ambush ploy so I countered with a fake attack of sudden gastric distress and hid in the bathroom until she found more urgent things to be annoyed at.

And so it came to pass that we rolled up at the wedding dressed to the nines, where it turned out there were about eight of us dressed in Black Tie rigs not counting the wedding party itself. Everyone else not in a dress or carrying some sort of musical weapon was wearing a perfectly ordinary suit and tie. All evening I puzzled this and eventually came to the only possible answer: that these days "Black Tie" was code for "Don't turn up in Bermuda Shorts, sandals and an Ozzie Osborn T-shirt". Which happens, because I've seen it.

Mrs Stevie insisted I dance with her and I put up a spirited and animated argument against this, but since we were on the dance floor at the time it apparently just looked like I had capitulated and was doing some sort of lame old person's dance from like 1990 or something.

"It reminds me a bit of our own wedding reception" I said once the band had stopped for a rest and the ringing in my ears had subsided.

"It should!" snarled Mrs Stevie. "We had our own reception here 29 years ago next August!"

"We did?" I said, brightening somewhat. "Well that explains the pall of dread and horror that has been hanging over the affair all night."

Shortly thereafter Mrs Stevie drank five cups of the lethally strong Espresso they were serving at the Viennese Hour4, got one of her migraines and demanded I drive her home. The trip was, for once, a quiet affair devoid of the usual critique of my shortcomings. Indeed, possibly due to some confluence of the Lutheran calendar, Mrs Stevie seems to have taken a vow of silence, not speaking a word since that night.

Bonus.

  1. First in a continuing series
  2. As in forty fbleep drinks
  3. I have plenty of bow ties but they are all real bow ties and all too small for my neck these days
  4. An over the top dessert affair. Look it up.

Tuesday, May 03, 2016

Anatomy Of A Fiasco

Last week I was the epicenter of a confluence of screw-up the likes of which the world hasn't seen since it got biffed by another planet and the moon was spat out.

It started innocently enough, oozing good intentions and only a whiff of brimstone™ in the air.

For the last year I've been husbanding a set of computer jobs that were written by a since-retired employee to some scheme only he knows. Operational issues arising from user-requested expansion of scope meant that the first job had been belted with the bodge hammer so hard and so often it was now a rat's nest of scripts scheduled all over the clock which worked to do one job - mostly.

The scripts had been amended often, sometimes by someone with only a tenuous grasp of what the update would mean1. They also had to be de-scheduled occasionally. The retiree had done this by hand but I wrote a script to do it automatically2 because I was going on vacation during one of the de-scheduling windows.

Later, I was asked to come up with a filewatcher, a database thingy that reacts to files arriving in a directory. I could not get the blasted thing working, so I wrote one in perl. It worked, indeed is working as I type, very well indeed. Not only that, I realized that for once I had designed something with so much flexibility that it could be used to replace the Rat's Nest code.

The Rat's Nest jobs essentially look for a file that gets transferred into that computer and do things with it. It adopts a technique of checking for the file by name and sleeping for half an hour if it doesn't see it. Problems arise in that the file has a date component as part of its name and files with the wrong date must not get processed - unless they must3.

Crossing midnight is especially nasty and causes all sorts of bugs to appear. I've been denying requests for longer run windows on all the other runs that do the same sort of thing for months because I don't want to inherit the same problems as the retiree did. Almost sorting out this issue actually had the Retiree making three copies of RatsNest.sh with slightly different code and running them in different time windows, which (nearly) covers all the bases and traps the incoming files. Sometimes more than once, which is a problem.

By replacing RatsNest.sh with SteviesSpiffyFilewatcher.sh all would be well. The umpteen different ways of doing the same job would become one thing. It was tested. What could go wrong?

I set up a sort of fake run to run alongside RatsNest.sh in order to see what could go wrong. Nothing did5.

One fly in the ointment was that before I could replace any late-night process, I would have to have the long-ago requested but yet-to-put-in-an-appearance remote computer access facility. This is a set of credentials that would allow me to use bleep to bleep and access the work computers from my laptop. I had been waiting for months for the go-ahead. It arrived at the end of last year. I tested it and it was, after a few teething troubles, dead good

Other projects got in the way of my implementing Project Unrestrained Genius, but last week H hour, D day, S script was decided upon and I adjusted the scheduler to turn off all copies of RatsNest.sh and switch on PureGenius.sh6 and I went home.

A sad mistake.

Naturally, I had made a spelling mistake that I hadn't picked up on. I learned a long time ago that if I don't see a problem in fifteen minutes of looking for it I never will, and have adopted review procedures to negate this7, but this was a small error I didn't know was there and missing iit was easy.

But I had planned for just such a screw-up and would be there with my trusty laptop to fix things on-the-fly should All not Be Well. I detected things not being well around 9:30pm (an expected email did not arrive) and activated remote access.

The remote access software announced it was going to update itself, and promptly did so, and that was it for my remote access. I struggled with the software in a World Gone Mad for two hours but couldn't figure out what had gone wrong.

All things being equal, I would then have done what I used to do in 1986 - jump an the next train west and work at my desk to fix the issue - but our office management has sent many emails telling me that unless I add my name to a special list ahead of time every time I need to get in I won't be allowed in after office hours.

I think you can see things were escalating nicely

I went to bed determined to takle the first train in that would arrive around 7 am, but was so worried that I couldn't sleep. I got up at 4:30, showered and ran for a train, just missing one and having to wait until 5:40 for the next one. I finally got into the office, fixed the typo and ran the scripts. I mailed out to everyone I could think of about the situation, who was to blame and why, what had been done and when and grabbed a cup of tea. It was about 7:20 am.

At around 7:40 am I became aware that my email client had disconnected from the email service, and that my explanatory mea culpa was still sitting in my outbox. A frantic series of phone calls revealed the ugly truth that everyone switched to the new cloud-hosted Office 3658 service a few days before was now working as they did back in 1986, sans email.

I began another frantic series of phone calls to the people I'd been mailing, but no-one was picking up. Turns out that the process I had screwed up was an essential part of the early morning processing and everyone had been up since three trying to figure out what was wrong.

I gnashed my teeth (again) at the paucity of documentation left me by the retiree, and ran upstairs to try and find anyone in the affected user group to tell face-to-face what was happening. Since none of them had been migrated to the failing cloud email system, they found my tales of dropped email service unconvincing, only reluctantly coming around when I suggested their quiet morning didn't mean no-one had problems, just that they couldn't tell anyone about them. Once I had them convinced I had to stand around for five minutes so they could all shout at me.

It transpired that another part of the organization had pushed out a "patch" that had nobbled the network connectivity to the Office 365 cloud, but I didn't find out about that until it was all over bar the punching.

On returning to my desk, one of my colleagues mentioned that he was getting emails to his phone, and an idea formed9. I would send out the mea culpa email from my personal account and see who yelled back at me.

Unfortunately my personal email account's name is one of those Fluffybunnystevie.net sort of names and a good 50% of the recipients would simply bin the incoming mail even though I sent it with the subject header AS YOU LOVE LIFE DO NOT BIN THIS EMAIL10.

It worked, sorta. Two of the upper management wrote back to say vile things in SHOUTY CAPS about me and my extreme incompetence, and mock me for my stupid email account name. About three hours later the cloud email service came back online and my original email went out, causing a reprise of SHOUTY CAPS and a stream of "me too" Replies to All that gave the cloud service a good workout.

So, all things considered, not my best work.

I have at least six more scripts that need replacing with the new style code, so I redesigned my new changes rollout checklist such that in the event I lost remote access nothing happens11 and poked around, eventually discovering the secret extra step needed to provoke my laptop to connect to the remote service. It seemed that all the Incompetence Demons had fled and gone away.

I can't wait to have another go.

  1. My favorite being the calculation of yesterday's date that would occasionally result in the nothingth of any given month
  2. And was shouted at by the retiree's Luddite colleagues when confronted by the new code
  3. All computer programmers4 recognize this requirement. Users supply it disappointingly frequently
  4. Now termed "developers"
  5. Naturally this roused my paranoia to new levels, but weeks of testing provoked no anti-programmer demons to manifest
  6. The original name was too long
  7. By grabbing someone who doesn't know what I'm doing and explaining the code to them. Works every time
  8. Now Office 364.5 since it took until lunch time to fix things
  9. I never learn
  10. Now I come to read it back, perhaps the subject header could have been better worded
  11. Which is what I should have done in the first place; turn off the old processes and turn on the new ones from home using the remote access service instead of doing that at work and planning on rolling it back from home