Last Day of Summer 2022

Many years ago (it feels like a lifetime ago) when I graduated from the Ball State ICS program, I took a job at my undergraduate alma mater as a systems analyst. It was my first full-time job and of course I had a world to learn yet about being a professional technologist but one of the distinctives about working for a small liberal arts college is that often resources are so scant that you’ll find yourself entrusted with responsibilities that are significantly beyond what you’d expect in other places of employment. This can be a tremendous boon in terms of getting valuable experience rather quickly, but the obvious downside is that, as a novice IT worker, you may face challenges that overmatch you.

I wasn’t in the job for more a few months when I was conscripted as the system administrator for the school’s primary online learning software, Moodle. It’s worth noting that this all occurred before online learning resources played such a top-to-bottom role in education, but it was still a highly critical role to be filled by someone with virtually no administrative experience. As it happened, I wasn’t in the role very long at all when my inexperience proved costly.

Early one morning I rolled out of bed to run a scheduled update on the SuSE Linux server that hosted our Moodle installation.  Groggy from want of sleep, I fumbled with my computer to connect to the school’s VPN, fire up the console app, and Telnet into the Linux box. For those of who have been in administration positions over technologies with a wide userbase, you’ll know that the reason for performing updates at an inconvenient hour is to minimize the impact that you’ll have on active sessions while the system is unavailable. For those of you who have worked with Unix-like command line interfaces before, you’ll know that running a shutdown and reboot command looks something like this: shutdown -r . In retrospect, I must have omitted the “-r” argument which indicates that the system should come back up once it has finished shutting down because after running the final command of the update script the Moodle server went down and stayed down.

Now very much awake and in a state of panic, I hurriedly got dressed, gathered my essentials and made the half an hour commute to the campus. I ran to the server room as quickly as I could and turned the server back on with a push of a button. It was now about 7:00 AM. “Whew!” thought I, “No harm done!” After all, Moodle had been down for less than an hour at a time when most of the student body would have surely been asleep. Perhaps nobody even noticed that the site was unavailable?

All throughout the day my boss received emails from inquisitive professors who said that their students were claiming that that they were unable to submit homework because Moodle had been unavailable that morning.  The students were angry, the professors were angry, and I think my boss was somewhat displeased as well. While this all amounted to a rather unpleasant few days for me, it was indeed a valuable learning experience and I have reflected on it several times in the intervening years. On balance, I think I would still say that this early hiccup still represents the most acutely stressful episode of my professional life but early last fall a strong contender manifested from the ether.

On October 11, 2021, I began my first day’s labor with a new employer. I had nominally been hired to fill the role of “Director of IT” but I was made to understand that for the near-to-mid future my obligations would effectively be those of a senior software engineer. Still, when you are operating as a member of a very small team, it is common to be called on to handle functions that do not fall neatly within your core responsibilities.

It was in just this fashion I was advised that my employer’s promotional site, the one that sits on the company's top-level domain, needed a little updating ahead of a significant marketing campaign. The site was purportedly a simple WordPress affair and the requested changes seemed fairly modest: the profile of the previous Director of IT was to be removed along with the publish dates of a few testimonials. Given the limited scope of my brief and WordPress’ reputation for being relatively beginner-friendly, I was not overly concerned about being able to effect the desired updates but the hard-won lessons of failure and mishap have compelled me to be cautious even in the face of near-certain success. With this in mind, I purposed to commit no changes to the site upon my first login; I would just verify that the credentials I’d found in the Excel document on the PC that I’d inherited were correct and then I’d watch a lengthy video on WordPress administration.

My initial efforts were not exactly met with unadulterated success because while the login credentials in the aforementioned Excel document ultimately did prove to be correct, the link to the admin portal was not. A cursory internet search advised me as to some of the more likely locations for the admin login screen and one of them turned out to be accurate.  Once I was logged in, I clicked around just enough to find out where the testimonials page was, always being very careful to not alter any of the settings. Once I had found the testimonials, I noticed that there was an enticing red button that read “EDIT” in bold, sesame letters in the lower right-hand corner of the screen. “I may as well take a look at the editing screen” I reasoned, still committed to the prospect of not actually saving any changes until I’d become better acquainted with WordPress generally.

With only mild trepidation, I clicked the edit button and the testimonials page remained much the same, save for a few more controls popping up on the sides and bottom of the screen. I right-clicked on the article date of one (which was to be removed since it had become a bit… dated) and a context menu popped up offering to, among other things, delete the selected object. I clicked ahead and deleted the indicated text just to get an idea for how the testimonial icon would look without it. I thought that it looked just fine, but it occurred to me that I should compare it side-by-side with the live site so I fired up a different browser and navigated to the promotional site once again. I clicked on the testimonials page and was greeted by the familiar header of “What People Are Saying” followed by… nothing.

A slight sting of panic shot through me as I noted that the very page I was in the middle of editing on one browser was displaying a big hole where content had formerly been on the other. I thought that perhaps having that page open in the editor had perhaps “locked” the content such that it wasn’t currently viewable and so perhaps closing out of the editor would restore the testimonials. I backed out of the editor on the one browser and then refreshed the page on the other browser. No improvement; the testimonials were still obstinately non-extant.

I hastily re-opened the page editor only to find that the testimonials were all missing from there as well. This was particularly baffling because, as I mentioned, I had just deleted a single date from the page, not all the dates and certainly not the articles themselves. It was at about this time that I regarded one of the buttons on the side of the editor menu that read “RESTORE”. That seemed relevant at the time, so I clicked it and up popped a few different modification dates from which to choose, none of which were recent. It seemed to me that what the situation demanded was action over certainty, so I clicked the most recent restore date and was then prompted with an ominous alert box that read “Are you sure you want to make this change?” I hadn’t recalled being asked about my certainty when I evidently deleted an entire page’s worth of content but I tried not to let that bother me as I clicked “Yes.”

Very unsurprisingly (for longtime readers of This Blog and You), restoring the earlier save point for that page did little to improve things so I had to think of other possible solutions. As a personal confession, I might mention that when it comes to software development, I have a bit of a hacker inclination. To be clear, that’s not “hacker” as in “Let’s hack into the FBI database” that’s “hacker” as in “I don’t know how to do this the right way so let’s try to hack it together some other way.” Given that inclination and given the strong instruction received from Professor Miller back in the Intro to Internet Algorithms class I had taken nearly 20 years prior, I figured I could paste in some raw HTML and CSS that might look better than the gaping hole that currently dominated the testimonials page.

At the close of business that day, all my efforts had redounded to a mountain of stress and slightly degraded testimonials page. The HTML/CSS hack had worked somewhat and though it was clearly worse than what preceded it, I had no choice but to call it quits for the day and my wife and I were hosting a social function in our home later that evening.

At said social function, my ordinarily unparalleled hosting prowess was blunted somewhat by my preoccupation with how to undo the damage that I’d visited upon the company’s promotional site. My guess was that my boss had not yet checked on the page to verify that the requested work had been done and I was not looking forward to explaining how I’d so badly bungled my very first assignment. To be frank, I was not even sure that I knew how I’d bungled it. All I knew was that I’d opened the WordPress editor and the page’s contents just vanished. Wistfully, I reflected that if only I had left some browser tabs open of the page as it existed before I’d modified it, I could just recreate the contents and styling from the code rendered on that page.

Just then, inspiration struck! I excused myself to the computer room and quickly conducted a Google search for the company’s testimonials page whereupon I clicked the three vertical dots to the right of the first result and then clicked on “Cached.” Eureka! The unborked testimonials page was there in all its pristine glory, implying (if not promising) a path forward for restoration.

Feeling much better about my prospects of overall success, I rejoined the party and enjoyed myself for the rest of the evening.

The next day, I strolled purposefully into my office, fired up my state-of-the-art computational device, and repeated the steps from the previous evening. Disappointingly, I found that the Google caching routine was more proactive than I would’ve guessed: the cached version of the page now reflected the same brokenness that the live page exhibited. I felt my heart sink into my stomach. I quickly tried a few other search engines (as well as the Wayback Machine) only to find that none of them were suitable for the task at hand. Just then, as I veered toward the precipice of despair, it occurred to me that I had left the Google cache tab from last night open on my home computer. I quickly drove home, hoping that the contents of that tab was still intact and that it wouldn’t try to reload when I clicked on it. Of course, it would make for a more dramatic (or more tedious?) story if the tab indeed had reloaded but mercifully it did not. The intact testimonials page was still pulled up from which I was able to get the code I needed to recreate the required testimonials (sans the publishing dates).

It was at this juncture that I recalled that I had also been tasked with removing the previous Director of IT’s profile from the site. The stakes were a bit higher with this change since the employee profiles sat on the main page and any corruption of that page would be immediately obvious by anyone visiting the site. Because I am such a formidably clever boy, I first opened a dozen tabs of the company site so that I would have ample failsafe measures in place should I have to recreate the entire page out of whole cloth HTML. After I had done that I once again logged into the admin portal and gingerly navigated my way to the employee profiles section. With my brow sweaty and hand trembling, I once again clicked the now foreboding “EDIT” button. I glanced around at the ensuing edit screen and scrolled up and down, ascertaining the location of all the constituent elements. Then, without so much as clicking a single button, I opened yet another tab and navigated to the company site only to discover that, comically enough, it was now completely blank. “I didn’t even do anything!” I remember audibly shouting at my monitor before remembering that it likely didn’t care.

This was real pretty pickle. Yes, I did have the open tabs of the home page from which I could, with time, reconstitute the now-corrupted home page. The main issue with that (aside from the unseemly amount of work that it represented) was that someone would most certainly visit the home page before I had finished restoring it and not owning up to my own bungling was job #1 as far as I was concerned.

This blog post is running real long and I’m not sure that it’s that interesting of story to begin with, so here’s the Reader’s Digest version: I found in our Azure portal (we deploy and host our apps in Azure) a nightly backup routine from which I was able to restore the site in its entirety. Did that discovery undercut the Herculean exploits involved in restoring the testimonials page? Yes, certainly.  Incidentally, I later pieced together that the editor we were using on our WordPress site was corrupted and upon opening it automatically injected the page with its own CSS class which had the presumably unintended consequence of hiding everything.

--------------

So it’s another Last Day of Summer and a beautiful one at that! The past year has truly been wonderful. In February, we celebrated the arrival of our second bundle of joy and happily, her arrival did not precipitate in the near exsanguination of my wife. Being a parent of two is actually enormously fun right now, if you don’t factor in the debilitating sleep loss. I, for one, am looking forward to autumn with exceptional keenness this year. I would also like to write on here a little bit more than once a year but writing is hard and it takes altogether too long for an undisciplined chap such as myself. Speaking of which, please forgive the many typos and writing errors that I’m sure have made their way into the preceding paragraphs; I didn’t proofread it at all.

Keep your stick on the ice,

-Ryan Wolfgang