Burning Out, Almost.

I have finally done all urgent tasks that I am supposed to do. Finnish the last task today yesterday noon. It's not fun to have to go back to office on Saturday. I was all alone in the office, which actually was pretty good. I could get my task done without any interruption from others.

The software was finnally successfully installed on the PDA for our marketing team to be brought to Shanghai for the exhibition next week. They are flying off on Sunday, that's why I gotta finnish it by Saturday. Why I didn't do it earlier? Well, one thing is because I was busy on the troubleshooting. The most important thing is the software development have been outsourced, and was only sent to me on late afternoon Friday. The worst thing is the software wasn't work. I feed back to them, and after some troubleshooting, another troubleshoot again *sigh*, I fed back the problem to them. By almost 11 pm, after some failed attempt with few fixes, I gave up and went out drinking with my colleagues.

This has gotta to be among one of the longest weeks month I have ever gone through. Ya, exactly one month since the Doraemon production system is being migrated to our latest software. I have finally manage to catch the culprits that causes the instability in the system on Thursday night.

If it is not because of my stupidity and cocky-ness, I could have found the problem earlier. The support manager has earlier ask me to check the possibility of bug in our program, but I told him I don't think so because I gone through all the log file, and I should have seen some error there if there is a database connection leaks. I've seen that error on the staging system, so my logic is if there is to be a connection leak problem in the production systems, I should be able to see the same error in the log file of the production system. Little did I know; and being over confident cocky arse hole that I am, fail to check the database connection configuration on the production system and notice that it is different from the one in the staging system, which prevent the error from being log into the log file.

It would be very convinient for me to blame this on other people, but I think in the end it just doesn't reflex good on me either. I am supposed to take full responsibilities for whatever damage that has been done. Afterall it's my own fault too. I have asked one of our developer to check the code and find if there is any connection leaks and he said no more. I even ask him to do that at least twice, and on both occasion he told me there is no connection leak anymore. Because of my laziness, I just trusted him and didn't audit the codes myself. The price of my laziness? One month of restless investigation and troubleshooting, and of course our reputation with the customer is at stake, which added to my stress. In fact I was almost completely burn out this week from this problem.

Yet, not all is lost, I have a better understanding of the system now. Have a better and more understanding on low level TCP stack inner working. In fact, the customer complained about a newly observed intermitten problem with the system and this was brought to my attention by the support manager on Friday noon. Based on what I have observed on the system throughout this one whole month, I can immediately point to him where to start investigation and by Friday late afternoon we have managed to identify the pattern of the problem, and I think we have found a solution for it. But only next week I will apply the solution because I want to monitor the system for the database connection leak over the weekend. If there is no problem by this Monday morning, I am pretty confident that there will be no more stability problem with the system.

I should feel better now, but the problem is I have start to feel better yet. Problem it's the stress and burning out. Thanks goodness that I manage to finally found the culprits, otherwise I am almost certain I will be completely knocked out next week, what more with the CNY around the corner. I don't want this problem to persist till then. Being stubborn is not a bad thing after all. If not because of my stubborness, I would have give up long ago.

My mood is still pretty low, and I think I owe myself a good rest. Maybe the reality have not sink in just yet. If it is not because CNY is around the corner, I might even give myself a few day off and just take a good rest. I will have catch up on issues on the rest of other project that is being undertook by the rest of my team members, and aside from that I will just lepak in office, with all the festive mood building up, it will be hard not too. Yay!