Front page
Silflay Hraka?

Bigwig is a systems administrator at a public university
Hrairoo is the proprietor of a quality used bookstore
Kehaar is.
Woundwort is a professor of counseling at a private university

The Hraka RSS feed

bigwig AT

Friends of Hraka
Daily Pundit
cut on the bias
Meryl Yourish
This Blog Is Full Of Crap
Winds of Change
A Small Victory
Silent Running
Dr. Weevil
Little Green Footballs
Fragments from Floyd
The Feces Flinging Monkey
the skwib
Dean's World
Little Tiny Lies
The Redsugar Muse
Natalie Solent
From the Mrs.
The Anti-Idiotarian Rottweiler
On the Third Hand
Public Nuisance
Not a Fish
Electric Venom
Skippy, The Bush Kangaroo
Common Sense and Wonder
Neither Here Nor There
The Greatest Jeneration
Ipse Dixit
Blog On the Run
Redwood Dragon
Greeblie Blog
Have A Cuppa Tea
A Dog's Life
Iberian Notes
Midwest Conservative Journal
A Voyage to Arcturus
Trojan Horseshoes
In Context
The People's Republic of Seabrook
Country Store
Blog Critics
Chicago Boyz
Hippy Hill News
Kyle Still Free Press
The Devil's Excrement
The Fat Guy
War Liberal
Assume the Position
Balloon Juice
Iron Pen In A Velvet Glove
Freedom Lives
Where Worlds Collide
Knot by Numbers
How Appealing
South Knox Bubba
Heretical Ideas
The Kitchen Cabinet
Bo Cowgill
Raving Atheist
The Short Strange Trip
Shark Blog
Ron Bailey's Weblog
Cornfield Commentary
Northwest Notes
The Blog from the Core
The Talking Dog
WTF Is It Now??
Blue Streak
Smarter Harper's Index
nikita demosthenes
Bloviating Inanities
Sneakeasy's Joint
Ravenwood's Universe
The Eleven Day Empire
World Wide Rant
All American
The Rant
The Johnny Bacardi Show
The Head Heeb
Viking Pundit
Oscar Jr. Was Here
Just Some Poor Schmuck
Katy & Bruce Loebrich
But How's The Coffee?
Roscoe Ellis
Sasha Castel
Susskins Central Dispatch
Josh Heit
Aaron's Rantblog
As I was saying...
Blog O' Dob
Dr. Frank's Blogs Of War
Betsy's Page
A Knob for Brightness
Fresh Bilge
The Politburo Diktat
Drumwaster's rants
Curt's Page
The Razor
An Unsealed Room
The Legal Bean
Helloooo chapter two!
As I Was Saying...
SkeptiLog AGOG!
Tong family blog
Vox Beth
I was thinking
Judicious Asininity
This Woman's Work
Fragrant Lotus
Single Southern Guy
Jay Solo's Verbosity
Snooze Button Dreams
You Big Mouth, You!
From the Inside looking Out
Night of the Lepus
No Watermelons Allowed
From The Inside Looking Out
Lies, Damn Lies, and Statistics
Suburban Blight
The SmarterCop
Dog of Flanders
From Behind the Wall of Sleep
Beaker's Corner
Bad State of Gruntledness
Who Tends The Fires
Granny Rant
Elegance Against Ignorance
Say What?
Blown Fuse
Wait 'til Next Year
The Pryhills
The Whomping Willow
The National Debate
The Skeptician
Zach Everson
Geekward Ho
Life in New Orleans
Rotten Miracles
The Biomes Blog
See What You Share
Blog d’Elisson
Your Philosophy Sucks
Watauga Rambler
Socialized Medicine
Verging on Pertinence
Read My Lips
The Flannel Avenger
Butch Howard's WebLog
Castle Argghhh!
Andrew Hofer
Moron Abroad
White Pebble
Darn Floor
Pajama Pundits
Goddess Training 101
A & W
Medical Madhouse
Slowly Going Sane
The Oubliette
American Future
Right Side Redux
See The Donkey
Newbie Trucker
The Right Scale
Running Scared
Ramblings Journal
Focus On Reality
Wyatt's Torch

July 28, 2004

Fingernails On A....

Over the past two days I've been participating in an effort to upgrade our Blackboard software from version 6.0 to version 6.1. I say "participating" because any upgrade of Blackboard is a massive effort, not to be undertaken by any team that is small in number or faint of heart. Our team is composed of two sysadmins, two testers, and one dba. Frankly, we could use more testers.

This is, at least so far, our second successful migration of Blackboard in just under a year. That's out of four attempts. The Blackboard software and tech support being what it is, we basically have to suffer through a major migration failure in order to learn enough to achieve a successful one later on.

Aside: A while back, during a teleconference, a Blackboard tech support guy once warned me that a server configuration we wished to try was "unsupported." I asked him how exactly we would be able to tell the difference.

The first migration, from 5.5. to 6.0, took two weeks just to complete, and the new system wasn't really ready for users for another month afterwards--though thanks to the demands of our schedule there were users aplenty less than two days after the migration finished. It was an exciting time to wear a pager.

This migration only took 12 hours--at least the major part of it did. Technically it's still ongoing, but thanks to our server setup, we can put the new 6.1 Blackboard service itself back up before all the machines supporting it have been upgraded. It also helps that the university is still in summer session. We wouldn't be able to do this with a typical fall load. Also, the migration from 6.0 to 6.1 isn't nearly as complex as the migration from 5.5 to 6.0. Had it been we'd have simply not done it. After last year, we basically told Blackboard that we'd never agree to another multi-day migration process.

Our Blackboard pool of servers is made up of four RedHat Linux servers running behind an Alteon load balancer and connecting to an Oracle DB server. Content shared between the four servers is NFS-mounted from a Sun StorEdge A1000 RAID.

Asides for Mom: A load balancer takes incoming requests and shares them out among multiple computers in situations where the sheer number of requests would crash one computer. Experience with load balancers is a pre-requisite for obtaining a job with most porn sites.

DB means "database." "Oracle" is a type of database that attempts to predict the future based on the records it has stored within it

NFS stands for Network File System. Basically it means that I can share files across multiple computers without having to store them in more than one place. It's a replacement for the old Sneaker Net protocol.

A RAID is a big group of hard drives that acts as One, thanks to its proprietary Highlander operating system.

Our migration process involved upgrading one server, testing it, then upgrading the rest of the servers one at a time, adding them back into the server pool as each was successfully upgraded. This is even a supported procedure according to Blackboard, which has two different installation procedures, one for the first server to be migrated, and one for all the others.

The actual migration itself only takes about two hours now, much better than the two weeks we endured last year. The rest of the time was taken up by the creation of various backups in case the migration fails, re-installing necessary customizations to the app once it has been migrated, as they are wiped out by the migration process, and testing. We could use more testers because every time we do perform an extensive Q/A check of blackboard we find something else that needs to be added to the list--usually because it breaks.

Yesterday we managed to add two servers to the pool before we brought it back up for the users. Today, when I attempted to upgrade the third server, the original two crashed. They worked fine right up until the install process started, and they worked just fine afterward. During, however, was an entirely story. Both wrote this error into the logs numerous times during the outage

"[error] Can't call method "mtime" on an undefined value at blackboard/system/lib/perl/CI/ line 64."

So, in the triumph of hope over experience, off the error and a description of the events went to Blackboard tech support. With them went one question "Does the addition of extra application servers to a load-balanced server pool necessitate extra downtime?"

What I got back from tech support was a suggestion that I check the file ownership of an entirely separate section of the Blackboard file system. This was annoying for a number of reasons. One, because their documentation specifically states that "On UNIX platforms, ownership of all Blackboard Learning System files is changed to 'bbuser' after the update." The request to check file ownership is basically an admission by tech support that their software doesn't work as documented.

Yes, big surprise. Welcome to the world of Technology, primitive wanderer from the past! It's still annoying.

Two, the file system they wanted me to check was in one of the NFS mounts, which the particular upgrade process I was using is not supposed to touch. Yet touch something it obviously does, else my other servers would not have crashed.

Third, all the file ownerships were as they should have been, meaning that I have to go back to tech support, inform them of this, and then wait for the next set of random, unexplained instructions to come trickling* out of them.

So, you ask, why don't ya'll jump ship, take up with some better online courseware?

Here's the sad truth. Given our size, and we are friggin' huge, and the faculty's desire for a simple interface, and they are friggin' simple, there is none better. If we could, we'd have jumped ship long ago.

So, in conclusion--I am Sisyphus, and this is my rock--or something like a rock, at least.

Update: There is this, however.

* By "trickling" I really mean "oozing," in the sense that, given enough time, it might be possible to see sap ooze from a tree in the dead of winter once you drill a hole in it.

Posted by Bigwig at July 28, 2004 04:45 PM | TrackBack
First time visitor to House Hraka? Wondering if everything we produce could possibly be as brilliant/stupid/evil/pedantic/insipid/inspired as the post you just read? Check out the Hraka Essentials, the (mostly) reader-selected guide to Hraka's best posts, and decide for yourself.

That reminds me of Websphere 3.0 - major sections of the documentation were simply pipedreams. Vapor.

Unfortunately I had a VP who never quite seemed to believe me when I pointed out "we can't do that, even IBM says those features don't work yet". Happy days.

Posted by: Greg at July 28, 2004 08:27 PM

I remember one of our server guys almost had to be physically restrained on a conference call with our main software vendor. Their materials management product is an utter piece of crap that is barely better after about 6 years of paying big money to be their beta tester. A couple of years ago when this call took place, they told us our server (which we purchased based on the specs they gave us, at no small cost I might add) that we ran that product on was totally inadequate. We had been struggling with this for quite some time and it was the last straw for Bill- if someone hadn't basically put their hand over his mouth the vendor would have learned several new words, none of them complimentary.

Posted by: Kevin at July 28, 2004 10:21 PM

BlackBoard wouldn't happen to be a Rational product, would it? Because that sounds very similar to the "upgrade" process for ClearCase employed at work. This process consists of:
Send email informing development staff of impending breakage. This works best if the "upgrade" is going to occur imedeately before a delivery deadline.

Reassure development staff that "everthing will Work As Expected, Because We Have Rational Experts On Call".

Perform "upgrade".

Scamble to find adequate cover and fix system as entire development staff is idled, usualy for a day or more, since the upgrade Encountered Unexpected Difficulty. Note that the Rational Experts are nowhere in evidence at this point.

Finally return system to working status. Often by rolling back to the previous configuration.

Endure harsh mocking from the development staff and the ire of the Powers That Be.


Posted by: Matt Navarre at July 28, 2004 11:34 PM

Hey, I found your problem. You need to *rinse* and *lather* before you repeat. :)

Posted by: Greg at July 29, 2004 05:43 AM
Post a comment Note: Comments with more than two dashes per line will be blocked as spam.

Remember personal info?