My name is Macalister Scrum, IT Detective, and troubleshooting is my business. I was on assignment with Big City National Bank. The home mortgage division just deployed a loan processing software release on Friday, and it was the following Tuesday. I always rested easier when we could get past the Wednesday following a Friday release without the kind of trouble that causes every manager from Development and Operations to sound off like fraternity pledges taking turns with a stadium horn.
That morning at the IT campus the sun was just bathing the tops of the Sycamore trees in yellow light, like a teased hairdo obstructing the beam of a conference room projector. Avoiding the swarm of coworkers later in the day, the buffalo grass and sage bushes were all I passed as I walked the dirt pathway to one of the 5-story concrete and steel buildings surrounding the quad. All was quiet except for the white-noise from the nearby freeway.
I wanted to get an early start that day because my virtual inbox was stacked higher than a space elevator.
I took the automatic elevator to the 4th floor, and surveyed the large open floorplan, empty except for the regular early-arrivers. Operations Manager Hope Forbes-Best was at her standing desk, appearing taller than her already above-average 5’7” thanks to feathered mules and a teased #D93E14 red hairdo. Kent Keepfocus, the Loan Processor, was in early as usual, his #A1CAF1 blue eyes behind a pair of pince-nez followed my tall, trench-coated figure as I made the long walk to my work zone. It doesn’t take a detective to know the only reason Kent came in early each day was to grab the choice complementary snacks before they were gone. Instant oatmeal and corn nuts were all that were left by 9am.
At my 3’ x 5’ flexible workspace I was booting up and before I could hang my Borsalino, Mr. Keepfocus from Loan Processing appeared next to me, pulling up a green cushioned stool. Without an invitation or morning pleasantries he handed me his tablet and said, “Your release broke my loan.”
It doesn’t take a detective to know the only reason Kent came in early each day was to grab the choice complementary snacks before they were gone.
The way his pear-shaped physique perched awkwardly on the ancillary stool, it was obvious he accepted too many cookies of the non-virtual kind. Glancing at his work tablet, he had more windows open than a high-rise apartment building during a summer blackout.
Before I could sit down and open my mouth he added, “See, the interest rate is a point too high. It was fine on Friday before the release, but last night when I was ready to print the final forms, the rate jumped a whole percentage point. This is a very important customer…”
He droned on like that for a while. I’ll spare you the details because they’re not important. What’s really important is Mac Scrum’s first rule of troubleshooting:
Get it in writing
“We’ve been through this before, Kent. Did you submit an incident with this detail, or are you going to rely on my short-term memory, which is about as dependable as an oblong bowling ball?”
“I’m telling you; I confirmed this borrower’s credit score is just high enough to qualify…”
“Look Keepfocus, I don’t tell you how to process loans so don’t tell me how to do my job! Troubleshooting is my business, and the next time you bellyache to me about your broken loan before I’ve had my cup of joe, you’d better have an incident opened with screenshots, loan detail, and a full description of the issue, or you’ll have to drink that lukewarm common-fridge apple juice out of a paper straw!”
I didn’t say that last part out loud. The actual words I used expressed the same information with a more cordial attitude. HR would be proud! (As you may have guessed, Mac Scrum has anger issues which are out of favor in most modern workplaces, but with mandatory sensitivity training I’m making good progress.)
Kent did what he was told, and I only had a moment to glance at the incident before my calendar’s automatic notification alerted the morning Stand-up meeting was about to begin. According to Big City National’s Service Management system, this borrower was flagged VIP. That meant Kent wasn’t just beating a tattoo when he said the customer was important. It also meant my virtual inbox wasn’t getting any smaller until I solved The Case of the Unexpected High Interest Rate first.
Hope Forbes-Best led the Stand-up with practiced efficiency, enforcing brevity with each attendee, without sparing everyone her pedantic comments along the way. She was only slightly put off stride when she arrived at Kent Keepfocus, who was not part of the operations team, but was making a guest appearance in the meeting.
“Well hello Kent.” She asked, “To what do we owe the honor of your presence?”
I knew what was coming next and I wasn’t going to like it. In this business, perception is more important than reality, and the IT Detectives pay the price when perception goes off the rails. I knew the perception of this release was about to get wrecked worse than the Congressional Limited at Frankford Junction.
Kent gave an encore performance of his “Your release broke my loan” monologue from earlier, demanding to know when we were going to move heaven and earth to correct the injustice. Hope looked my way with tired eyes that pleaded, “Please don’t let this be another release gone bad!”
In this business, perception is more important than reality, and the IT Detectives pay the price when perception goes off the rails.
Back at my desk I sported noise-cancelling headphones, queued up my favorite playlist of all the popular hits from Arlen to Ziegfeld, and proceeded with Mac Scrum’s 2nd rule of troubleshooting:
Check if we’ve seen this before
I searched the Knowledge Management System first, but couldn’t find any clews. Next, I reviewed past incidents and got a partial match: another loan processor reported the wrong interest rate on the same loan product, but it turned out the borrower didn’t meet the minimum credit score requirement. Kent swore his borrower’s credit score was high enough. I checked the screen capture on the incident, and confirmed the score was 801, just above the 800 minimum. Time to move on to rule #3:
Confirm the issue wasn’t caused by a recent release
This was the rule Kent Keepfocus sunk his teeth into like his daily yogurt fruit bar. I can’t say I blame him for jumping to that conclusion based on our less-than-stellar Release Management track record. It’s always a good idea to check the release notes to see if any changes could have caused the issue. I didn’t have to check, because Mac Scrum knows his release notes like a Gen Xer knows the words to the Gilligan’s Island song. There were no changes to the interest rate calculation or loan program, so with another dead end it was time for rule #4:
Verify the issue can be replicated
With the loan detail available in the incident, I was able to create a similar loan in our test environment in less time than it takes to replace the lemon meringue at the local Automat. My test loan didn’t exhibit the issue. I tried the new build, and the build from before Friday’s release. Same result: no issue. I was down to the last tool in my bag of tricks:
Trust only what you see with your own eyes
It was time for Mac Scrum to take a dose of Bromo Seltzer and dig into the data. Instead of relying on Kent’s screen shots, I accessed the loan in the production application, reviewed the borrower data, checked the credit score…then it happened: that moment of clarity, like when a high def photo renders on a 4K monitor. Mystery solved.
* * *
“Kent, while you gave us good detail in the incident, you left out one important fact.” The stars must have aligned that morning because I miraculously found an available conference room for 30 minutes when both Kent and Hope were free. Kent’s problem loan was projected from my laptop.
“You can see that a credit report was run last month, then again yesterday.”
“Yes.” Kent Keepfocus interrupted. “We do that all the time. It had been more than 30 days since the first one was run.”
“And which credit report had the 801 score, the one from last month, or Monday?”
The loan processor opened his mouth to speak, but didn’t have anything to say for the first time today.
Without waiting for an answer, I accessed the borrower’s information page showing a credit score of 798, below the 800 required for the discounted interest rate.
The pince-nez perched on Kent’s nose slipped off and his complexion went from #F7E1D3 to #EEB4B0 before the spectacles clattered on the conference room table.
“You see Kent, while you sat on this VIP loan for 30 days, the borrower must have shopped at other lenders, and they ran credit reports. Even the most junior loan processor knows a borrower’s credit score drops a little with every credit inquiry. If you didn’t have so many windows open on your tablet, you would have seen the lower score on the latest credit report. Fortunately, your boss can click this override button to apply the discounted rate, because the credit score is within 10 points of the minimum required. If you need the loan origination intern to drop in to explain these basics to you, I’ll ping her now.”
Okay I didn’t say that last part out loud. I gave the same explanation but with much kinder words. (Thanks sensitivity training!) I also thanked him for reporting the issue, and advised that the Knowledge Management System was updated with the scenario, and an enhancement request was submitted to make it easier to catch on future loans.
Hope Forbes-Best smiled, then turned to Kent, “You’re welcome in our morning stand-up anytime, and I expect you’ll drop in tomorrow morning to let everyone know what a great release this has been, and like Detective Scrum has demonstrated, it definitely did not break your loan.”
Again, Kent Keepfocus was speechless, and Mac Scrum, for once, was looking forward to a stand-up on a Wednesday after a Friday release, even if my virtual inbox reached the moon by then.