That's also the day the proverbial "you-know-what" hit the fan at my work. I'm a software developer. Technically, I'm not considered operational support. However, in small company land, everyone has that title especially when your company is a 24/7 operation. As a result of this, I can get called into operational issues at almost any time, day or night.
We were having issues all day on Friday. Something just wasn't quite right. We had rolled out a software release earlier that week. We also rolled out a bunch of new cloud servers (geek speak...think of it as virtual hardware out there on the internet). We turned on the systems on Friday and the data just wasn't right so we were rolling back different components of the system to figure out where the problem was. When you are dealing with a very large volume of data (I can't say how much data we have running through our system but we rank way up there on the charts), finding the problem is a literal needle in a massive haystack. Nothing was getting us back to where we knew the data was supposed to be at.
I left work with my cell phone glued to my ear because I had to go take that instructor test. I was giving as much info as I could about what changes we had made in the software to support the new functionality. We were trying various rollbacks of software pieces trying to isolate the problem. Nothing was working. I was literally in my dobak (uniform) outside the testing room with the cell phone to my ear. Master Lee had to call me in to the room. I literally dropped out of the phone call mid-discussion.
One thing that my martial arts training has developed in me is the ability to focus on the task at hand. This is especially true in forms. Memorizing the forms and performing them to the degree expected by Master Lee really teaches you to focus on what you are doing right now. I believe this learned focus is the only way I was able to mentally switch gears to my instructor test so quickly. I was told I did well with just a minor adjustment to how I personally interact with the students (more walking among and interacting with the students rather than instructing from the front). My work mates also told me later that they felt really bad about everything and were hoping I would be able to focus on the test.
After the test, I left the room and remembered I had to dial back in to the phone call. Within about 2 minutes of being on the call, the light bulb went off in the back of my head. I headed back to the office, sat down, and investigated the illumination in the code base. Bingo! Found it. Problem solved. The talk at that point was to just roll everything back while I tried to diagnose it, reconvene in the morning and essentially blow our Saturday and quite possibly our whole weekend. Luckily, the code change was trivial. I did a quick before and after test on sample production data and knew we had it licked. A quick roll out followed by a verification of real data in our production systems and we were all able to get a solid night's sleep. More importantly the weekend was free*.
I wish I could say I learned something new that night but honestly I can't. Everyone knows you need to step away from a problem sometimes to solve it. That stepping away place for me right now is Taekwondo. Focus on something else for a bit and it's amazing what your brain will do when you come back to it.
* We ended up rolling out other functionality that weekend and I ended up calling in during my lunch breaks during the test. No issues though. It was at least stress-free!
No comments:
Post a Comment