For the longest time I had no idea what a router was despite configuring Linux for years. This is not about a hipster who is configuring his WIFI at Starbucks so he can watch the latest news while sipping on an Espresso. This is about me, a different sort of hipster, and about the '90s when Linux was all the rage, unlike these days when it's so passe that even my grandma knows how to somewhat wield it - and I did teach my grandma, I kid you not. At least how to make new folders in GNOME and so forth. But back then I had no clue what a router was even though I was configuring IP Masquareading and adding routes to the routing table. Reason being, nobody had more than one computer in a home at a time hooked up to the modem, nobody! I still remember how elite it felt when I bought my first 4-port Ethernet NIC - I thought I was God or something. Like wow man, I can make a Linux box that can serve computers in other rooms of our family home? This was a mind blowing radical idea. Today it seems so childish and many a modern "geekstresses" might mock and bully me for this moment but it is a fact. I love 4-port NICs. It's a fetish of mine I suppose. And then one day I got a job with billion dollar equipment that serves thousands of ports of Gigabits a second.
For years I worked in a NOC and handled all manner of crisis and outage and maintenance duty. It was not unusual to spend a double shift at times, handling a hundred phone calls an hour even. Rarely did anyone swear, argue, fight, or toss about cruelly phrased insults. We hung out at pubs and other places after work and had a blast of a time. But when people made mistakes, and sometimes we humans do make mistakes believe it or not, regardless of how well meaning we are, money was lost and people got upset. But at no point did anyone belittle, ridicule, nor even let go of an employee for it. Usually it was a very serious training issue and people resolved it promptly. All working there were responsible adults and we all took great care with every keystroke we pushed as even the wrong digit could cause a corporate entity tremendous harm. I suppose you can linken it to the medical realm prescribing 5.0g instead of 50mg for a heart condition. Small difference, right? All you who claim grammar is not important, think on these concepts for a while, even meditate about them and grasp that not all of us are made equal, despite all of your eastern guru philosophizing. Some of us make more mistakes in some places while others make less. Some play piano poorly, some make love poorly, some are poor bean counters, others are tremendously efficient at Kung Fu. These differences are inherent in training, experiences, and psyches. The longer we insist we are all equals, the longer we as a whole will suffer. This all ties into IT quite well.
I was sitting in my cubicle working away when suddenly I hear a colleague sighing and as we worked for years I knew this was a big deal. So I looked over and said "are you dead?". And he nodded, so I came around the giant cubicle wall of democracy, and saw what he had done. Instead of shutting a subinterface on an ATM switch he shut down the whole switch with hundreds of customers. It took no time to correct, but the damage had been done. As he executed the command a while ago thinking he shut down the correct part to isolate the problem and was doing other things, he caused a hundred companies to lose connectivity for quite a while. So when we looked at our phone queue we saw just how busy our lunch hours would have become. So we did not do what many government and police forces do and try to pretend it was somebody else's problem. We did the right thing and proactively called as many of these businesses and alerted them of the fact that the problem is on our end and that it was corrected. The management acted quickly in a meeting and they assembled a compensation strategy and paid back part of the service fee and then some. This is what responsible governance looks like no matter where you are and no matter whom you work for and it is how I also govern myself personally. If you do not, or they do not, then something's quite wrong. The corporation was so large, we could have just blamed it on the vendor or a third-party and all would have trusted us and we would have saved quite a lot of money. But instead, we bit the bullet, corrected the employee, and kept the trust with all our customers. In fact, maybe it even gave us future business. This is what those in authority need to learn from the corporate world. Stop burying your problems under the carpet. When you have wronged a citizen, confess and move forward. Stop punishing the innocent civilians. But sadly, only a few entities even in the corporate realm understand this fine simple human principle called honesty, integrity and respect,. even here in Canada.
Now lately I begun listening to a most amazing podcast called Packet Pushers. And it's incredible how many topics they discuss and how pleasing they are to listen to. One topic in particular caught me by unawares. The host discussed Network Operating Systems or NOS. A fine example is of a router from Cisco that runs IOS (as opposed to the iPhone's iOS, note the capitalizations). IOS governs how much of the Internet is configured. There are others obviously, much like not all phones run iOS but some run Android. Those who create NOS systems like the IOS are apparently very rare breed of coder. And it does make sense. If you've ever looked at a Google Play store app, they are bloated, huge, ugly messes of code. A simple tic tac toe game can be over 50 megabytes in size! Granted I understand why, but this is ridiculous. If a router for email was this poorly written, I'm sorry to say but none of you would ever use the Internet due to how slow it would be. NOS systems have to be built with tremendous grasp of code, and even the compiler toolchain. They are optimized on such a level that every byte matters. The larger the executable the slower it runs. So Cisco in this case, as the makers of IOS, takes a lot of pride and effort in hiring the best talent. And yet this talent is selected by current employees. So the worst the HR team the worst the IOS code. As well, the worse the coding in the real world is taught the worst the offerings to Cisco's HR team is. So the code in IOS depends on several factors. The HR teams ability to select top talent and the luck in top talent choosing to apply to Cisco and not being discouraged by third parties, and universities and self-starters learning proper and decent coding techniques on their own. When these factors are properly set only then is the IOS code tight and compact. If the script kiddies who make 50MB tic-tac-toes were hired for the IOS team we could call Cisco incompetent then, but not before.
Now the Packet Pushers episode called Cisco incompetent before this point and that is what I have a beef with. There is a lot of security vulnerabilities in IOS code that rears its ugly head here and there. But the problem is calling Cisco incompetent when 99% of the code works flawlessly is a bit silly and imprecise. It's difficult to explain without simplifying. Let's suppose there are only 100 routers in all the world. Since Cisco has market space dominance let's say 90 are Cisco's and 10 are of competitors. If hackers were to start hacking as hackers often do like to hack since not hacking for a hacker is living like a hack, we must conclude that hackers will spend more time hacking the 90 Cisco devices than the 10 of the others. As such the statistical probability that more vulnerabilities will be found in the IOS are much larger than in the other 10 routers. But here's another catch, since Cisco has market dominance that means the most important devices run Cisco and not others and as such when hackers do hack and select targets they will be trying and focusing mostly on Cisco's devices. So there are more of their devices and they are in the spotlight more than others. This is twice more attention than other devices. Now let's suppose that Cisco's code is just as secure as of the competition. The above example would yield quite a lot more found bugs in the IOS than in other's code simply by the number of larger quantity of attempts at hacking it. Someone whom analyzes things improperly might conclude thusly that since the quantity of problems found in IOS is 50 and in others is 5 that Cisco must suck! "50 is greater than 5 man, Cisco is incompetent!". But this is far from the truth. I started off saying that NOS programmers were hard to find. Fixing IOS bugs requires NOS programmers. If they're hard to find then fixing bugs will be difficult will it not? Therefore, since Cisco has the largest offering of devices and platforms fixing problems takes a bit longer than for smaller competitors who might be leaner due to a smaller selection. Juniper might be able to fix a bug in an hour, but Cisco might take a day. Some even look at these numbers on their surface and imply that Cisco is neglecting the severity of vulnerabilities since Juniper solves it 23 hours sooner! A fact no doubt. But a misinterpreted and wrongly analyzed one at that. Cisco might have to fix a bug in 5 platforms affecting a 1,000 different models, and take care not to introduce new problems along the way, while Juniper may only need to address 1 platform affecting 20 models. Perhaps now you see the difference?
This is why I had a beef with Packet Pushers calling Cisco incompetent in their show. Having more vulnerabilities than the other guys and taking longer is not a sign of incompetence whatsoever. In fact, it could be a sign of utter competence, and top notch quality at work. Juniper might have twenty times more bugs, but since hackers focus on Cisco, and since script kiddies only zombie around what others have discovered, even a noob like me might be able to crash a Juniper but not a Cisco. And that perhaps is the real truth of whom is doing what job. As far as I go, and without being an expert, although I agree that vulnerabilities are a problem, calling them incompetent merely due to a quantity and severity without explaining what I just did is irresponsible and I disagree with it. There are many governments, including Canada's, that use a similar model of accusations to even convict people, based merely on a remark or an opinion without any evidence. I sincerely hope the tech world steers clear of those poor examples of finger pointing. We can do better than that guys. As for Packet Pushers I will keep listening, as not only is it the best techcast out there, but the phrasing makes me laugh much of the time.
No comments:
Post a Comment