Front page
Silflay Hraka?

Bigwig is a systems administrator at a public university
Hrairoo is the proprietor of a quality used bookstore
Kehaar is the head web developer for a regional newspaper
Woundwort is a professor of counseling at a private university

The Hraka RSS feed

bigwig AT

Friends of Hraka
Daily Pundit
cut on the bias
Meryl Yourish
This Blog Is Full Of Crap
Winds of Change
A Small Victory
Silent Running
Dr. Weevil
Little Green Footballs
Fragments from Floyd
The Feces Flinging Monkey
Dean's World
Little Tiny Lies
The Redsugar Muse
Natalie Solent
From the Mrs.
The Anti-Idiotarian Rottweiler
On the Third Hand
Public Nuisance
Not a Fish
Electric Venom
Skippy, The Bush Kangaroo
Common Sense and Wonder
Neither Here Nor There
The Greatest Jeneration
Ipse Dixit
Blog On the Run
Redwood Dragon
Greeblie Blog
Have A Cuppa Tea
A Dog's Life
Iberian Notes
Midwest Conservative Journal
A Voyage to Arcturus
Trojan Horseshoes
In Context
The People's Republic of Seabrook
Country Store
Blog Critics
Chicago Boyz
Hippy Hill News
Kyle Still Free Press
The Devil's Excrement
The Fat Guy
War Liberal
Assume the Position
Balloon Juice
Iron Pen In A Velvet Glove
Freedom Lives
Where Worlds Collide
Knot by Numbers
How Appealing
South Knox Bubba
Heretical Ideas
The Kitchen Cabinet
Bo Cowgill
Raving Atheist
The Short Strange Trip
Shark Blog
Ron Bailey's Weblog
Cornfield Commentary
Northwest Notes
The Blog from the Core
The Talking Dog
WTF Is It Now??
Blue Streak
Smarter Harper's Index
nikita demosthenes
Bloviating Inanities
Sneakeasy's Joint
Ravenwood's Universe
The Eleven Day Empire
World Wide Rant
All American
The Rant
The Johnny Bacardi Show
The Head Heeb
Viking Pundit
Oscar Jr. Was Here
Just Some Poor Schmuck
Katy & Bruce Loebrich
But How's The Coffee?
Roscoe Ellis
Sasha Castel
Susskins Central Dispatch
Josh Heit
Aaron's Rantblog
As I was saying...
Blog O' Dob
Dr. Frank's Blogs Of War
Betsy's Page
A Knob for Brightness
Fresh Bilge
The Politburo Diktat
Drumwaster's rants
Curt's Page
The Razor
An Unsealed Room
The Legal Bean
Helloooo chapter two!
As I Was Saying...
SkeptiLog AGOG!
Tong family blog
Vox Beth
I was thinking
Judicious Asininity
This Woman's Work
Fragrant Lotus
Single Southern Guy
Jay Solo's Verbosity
Snooze Button Dreams
You Big Mouth, You!
From the Inside looking Out
Night of the Lepus
No Watermelons Allowed
From The Inside Looking Out
Lies, Damn Lies, and Statistics
Suburban Blight
The SmarterCop
Dog of Flanders
From Behind the Wall of Sleep
Beaker's Corner
Bad State of Gruntledness
Who Tends The Fires
Granny Rant
Elegance Against Ignorance
Say What?
Blown Fuse
Wait 'til Next Year
The Pryhills
The Whomping Willow
The National Debate
The Skeptician
Zach Everson
Geekward Ho
Life in New Orleans
Rotten Miracles
The Biomes Blog
See What You Share
Blog d’Elisson
Your Philosophy Sucks
Watauga Rambler
Socialized Medicine
Verging on Pertinence
Read My Lips
The Flannel Avenger
Butch Howard's WebLog
Castle Argghhh!
Andrew Hofer
Moron Abroad
White Pebble
Darn Floor
Pajama Pundits
Goddess Training 101
A & W

January 06, 2003

Hey, Baby

The biggest waste of money I've ever seen was when my old dot-com, net32, gave several million dollars to IXL for the development of a web portal application, back in the late nineties. Several months later, IXL delivered a supposedly customized app that ran slowly, crashed constantly and wrote out customer names and credit card information to a plain text log file. Right after delivery, the senior programmer, database administrator and project manager all left for presumably greener pastures with the company that produced the unwieldy application, Broadvision, so no one at the company that we had paid to develop and administer the program had any experience with it. We were one of a number of Broadvision end customers in North Carolina that year, and a year later, all of them had gone out of business. We almost made it. A month after the app was delivered, we hired an entire technical staff to do 2 things; patch the holes in Broadvision long enough to code an app to replace it, and produce the functionality we had supposedly gotten from Broadvision and IXL ourselves. A year later, we had done both, for a fraction of what we had paid when we originally outsourced the project, but it was too late. Like damn near every other dot-com, we had run out of money. We came within an inch of being bought by Colgate before they backed out. Rumor had it at the time that the head of their sales division had torpedoed the sale to protect his staff from Internet sales.

The original founder bought out net32 for a song, hired a skeleton staff and deployed the new code. It's still running.

I thought about that today when I read this story in the NYT about possible security vulnerabilities that arise when companies outsource their software development. I especially liked this quote from the president of an Indian software consulting company

Mr. Gupta assured his clients that his company used exacting background checks and multiple reviews of company-written software based on industry standards. "With all these in place, we can guarantee, basically, that the code we deliver will be bug-free and will perform to specifications and will not have holes in it," he said.

The only question your mind should not be whether Mr. Gupta is lying or not, but whether or not Mr. Gupta knows he is lying. There is no such thing as bug-free code at the level his company and those he competes with function at, ever. No plan of battle ever survives contact with the enemy, no application survives contact with the user. What's worse about code is that, while most battles have only one enemy, applications have multiple users. Marketing wants one thing, the web-publishers want another, and the end-users always behave in an entirely different way than initially expected.

The only thing that a company can do to make all of the users happy with an application is to fix bugs, fast, when their reported, and outsourcing companies suck at that, even when they try not to. Let's say you're a user of Application A, from Company C, and you find a bug in an application built for them by Outsourcer O. You call in, and by some miracle get the receptionist, who directs you to the help desk. The help desk confirms your bug, then passes it to the somebody in administration. If they're good, and they've been there a while, then they'll pass you to the person in administration who has a relationship with the company that built the app. Otherwise your bug will bounce around until it gets to him, or her. Then, when they can get around to it, they call the company that built the app or who administers the app. They don't have to be the same company, and if they're not, that bug is going to live for almost forever. Even if they are the same company, odds are that development and support are different divisions. The admin person at Company C has more valuable things to do than waste time playing phone tag with geeks, so they call the person at Outsourcer O that sold them on outsourcing in the first place.

That's right, your bug report just went to Marketing, when they will ignore it, because they've already got the money they wanted from Company C. They're busy wining and dining new clients to pass the message along, so it takes even longer for your bug to get in front of someone who can fix it. Odds are by the time it does get there, the person in charge of it is not the person who originally coded the part of the application producing the bug, so they have to get themselves up to speed on the application before they can even attempt to fix it. Once they've got a fix, they can't just push it out, either. It's got to go through multiple reviews of company-written software based on industry standards.

Even at the best, fastest and most conscientious outsourcers, bug fixes take a month of Sundays. The only way Company C is going to be at all responsive to a bug report within a reasonable time frame is if their own in-house IT staff built and maintains their code. Even then the fix has to go through some reviews. The process is sped up because people who write and maintain their own code have a better idea of what changes will do to that code, but it's still going to tested first, and likely will be rolled out as part of a build with other changes. But the time saved in the process can likely be measured in weeks.

The problem with having an in-house IT staff is that an in-house IT staff does a lot of sitting around when they're not developing. Hell, when they are developing a lot of them aren't going to come in till noon, and they may leave at 4. IT staff is notorious for measuring productivity in things done, in goals achieved, rather than in hours spent doing it. It's not their fault if the project you asked them to code got done in a day. You asked them to do it, they did it, so now they're going home.

Management hates that. Management wants people that are busy all day long, as if productivity is something that can be correlated to calories burned. In fact, management hates that so much that management will pay more money than it takes to employ those workers to an outside company so that they don't have to look at them. That's how outsourcers work, and the result is software bugs take a long time to discover, and a longer time to fix.

You don't fix bugs by throwing money at them. You fix bugs by throwing knowledge at them, and the person with the most knowledge of a software application is the person who built it, or maintains it. With an inhouse staff, that person is right there, in the building. With outsourcers, the person who maintains the application is almost certainly not the person who wrote the code. The person who did write the code has almost certainly been focused on a new project, if they're still with the company at all, and his management will be loath to redirect him from a project that's currently bringing in money to one that they've already gotten money from.

In short, outsourcers will fuck you, promise to call, and forget you. It's the nature of the beast. Even the good ones, the ones who intend to call, who want to respect you in the morning, probably won't.

Here's another story, about a website at UNC. The site was developed by a fairly well thought of local company, one that gets a little shout-out from NPR each day during All Things Considered because they're a supporter of public radio. A law student organization contracted with them to build a site for them where they could publish a little online magazine each month. The developer had worked with the particular environment it was built in before, and was a pretty decent coder overall. But, right after the initial application was delivered, she jumped to another company. A week later, the application crashed, and kept crashing every time it was brought back up. The consulting company put a new developer in charge of the application, who was wasn't familiar with it or the environment it was running in. (Tech aside: Tomcat 4.0.3 with Cocoon 2.0.1). He futzed around, confirmed for the fourth or fifth time that yes, the site couldn't handle a load, and asked if maybe the O/S needed a patch, despite the fact that several other apps using the same environment on the same machine handled a load just fine.

At some point a check was cut, and once the company got its money, he was gone. I'd say like a thief in the night if that wasn't wholly inadequate in describing the swiftness of his departure.

I have lots of stories like that. I have some that are even worse, and almost all involve software development by outside parties. There will always be bugs in software, always, and the nature of the software consulting business practically assures that the bugs developed under that system will not only be more critical, they will take longer to fix than those from an application built by inhouse staff. Farming out development may be cheaper in the short run, but it almost never is in the long run.

Posted by Bigwig at January 6, 2003 10:13 PM | TrackBack
First time visitor to House Hraka? Wondering if everything we produce could possibly be as brilliant/stupid/evil/pedantic/insipid/inspired as the post you just read? Check out the Hraka Essentials, the (mostly) reader-selected guide to Hraka's best posts, and decide for yourself.
Post a comment Note: Comments with more than two dashes per line will be blocked as spam.

Remember personal info?