Front page
Archive
Silflay Hraka?


Bigwig is a systems administrator at a public university
Hrairoo is the proprietor of a quality used bookstore
Kehaar is.
Woundwort is a professor of counseling at a private university

The Hraka RSS feed

Email
bigwig AT nc.rr.com

Friends of Hraka
InstaPundit
Daily Pundit
cut on the bias
Meryl Yourish
This Blog Is Full Of Crap
Winds of Change
A Small Victory
Silent Running
Dr. Weevil
Little Green Footballs
ColdFury
Oceanguy
Fragments from Floyd
VodkaPundit
Allah
The Feces Flinging Monkey
Dean's World
Little Tiny Lies
The Redsugar Muse
Sperari
Natalie Solent
From the Mrs.
ErosBlog
The Anti-Idiotarian Rottweiler
On the Third Hand
Public Nuisance
Not a Fish
Rantburg
AMCGLTD
WeckUpToThees!
Electric Venom
Skippy, The Bush Kangaroo
Common Sense and Wonder
Neither Here Nor There
Wizbang!
Bogieblog
ObscuroRant
RocketJones
The Greatest Jeneration
Ravenwolf
Ipse Dixit
TarHeelPundit
Blog On the Run
blogatron
Redwood Dragon
Notables
Greeblie Blog
Have A Cuppa Tea
A Dog's Life
IMAO
Zonitics.com
Iberian Notes
Midwest Conservative Journal
A Voyage to Arcturus
HokiePundit
Trojan Horseshoes
In Context
dcthornton.blog
The People's Republic of Seabrook
Country Store
Blog Critics
Chicago Boyz
Hippy Hill News
Kyle Still Free Press
The Devil's Excrement
The Fat Guy
War Liberal
Assume the Position
Balloon Juice
Iron Pen In A Velvet Glove
IsraPundit
Freedom Lives
Where Worlds Collide
Knot by Numbers
How Appealing
South Knox Bubba
Heretical Ideas
The Kitchen Cabinet
Dustbury.com
tonecluster
Bo Cowgill
mtpolitics.net
Raving Atheist
The Short Strange Trip
Shark Blog
Hoplites
Jimspot
Ron Bailey's Weblog
Cornfield Commentary
Testify!
Northwest Notes
pseudorandom
The Blog from the Core
Ain'tNoBadDude
CroMagnon
The Talking Dog
WTF Is It Now??
Blue Streak
Smarter Harper's Index
nikita demosthenes
Bloviating Inanities
Sneakeasy's Joint
Ravenwood's Universe
The Eleven Day Empire
World Wide Rant
All American
Pdawwg
The Rant
The Johnny Bacardi Show
The Head Heeb
Viking Pundit
Mercurial
Oscar Jr. Was Here
Just Some Poor Schmuck
Katy & Bruce Loebrich
But How's The Coffee?
Roscoe Ellis
Foolsblog
Sasha Castel
Dodgeblogium
Susskins Central Dispatch
DoggerelPundit
Josh Heit
Attaboy
Aaron's Rantblog
MojoMark
As I was saying...
Blog O' Dob
Dr. Frank's Blogs Of War
Betsy's Page
A Knob for Brightness
Fresh Bilge
The Politburo Diktat
Drumwaster's rants
Curt's Page
The Razor
An Unsealed Room
The Legal Bean
Helloooo chapter two!
As I Was Saying...
SkeptiLog AGOG!
Tong family blog
Vox Beth
Velociblog
I was thinking
Judicious Asininity
This Woman's Work
Fragrant Lotus
DaGoddess
Single Southern Guy
Caerdroia
GrahamLester.Com
Jay Solo's Verbosity
TacJammer
Snooze Button Dreams
Horologium
You Big Mouth, You!
From the Inside looking Out
Night of the Lepus
No Watermelons Allowed
From The Inside Looking Out
Lies, Damn Lies, and Statistics
Suburban Blight
Aimless
The SmarterCop
Dog of Flanders
From Behind the Wall of Sleep
Beaker's Corner
Bad State of Gruntledness
Who Tends The Fires
Granny Rant
Elegance Against Ignorance
Moxie.nu
Eccentricity
Say What?
Blown Fuse
Wait 'til Next Year
The Pryhills
The Whomping Willow
The National Debate
The Skeptician
Zach Everson
MonkeyWatch
Geekward Ho
Argghhh!!!
Life in New Orleans
Rotten Miracles
Fringe
The Biomes Blog
illinigirl
See What You Share
Truthprobe
Blog d’Elisson
Your Philosophy Sucks
Watauga Rambler
Socialized Medicine
Consternations
Verging on Pertinence
Read My Lips
ambivablog
Soccerdad
The Flannel Avenger
Butch Howard's WebLog
Castle Argghhh!
Andrew Hofer
kschlenker.com
Moron Abroad
White Pebble
Darn Floor
Wizblog
tweedler
Pajama Pundits
BabyTrollBlog
Cadmusings
Goddess Training 101
A & W
Medical Madhouse
Slowly Going Sane
The Oubliette
American Future
Right Side Redux
See The Donkey
Newbie Trucker
The Right Scale
Running Scared
Ramblings Journal
Focus On Reality
Wyatt's Torch

February 04, 2004

Fixing Internal Service Errors In Movable Type

Spent good chunks of Monday and yesterday trying to track down odd errors on a couple of UNC Movable Type blogs--and love is not the easy thing and Kevin's Blog2EndAllBlogs Blog.

Both were seeing an Error 500: Internal Service Error message when they tried to rebuild their sites, even though the rebuilds appeared to have completed successfully.

Sorry, we encountered an internal error or misconfiguration and we were unable to complete your request--this is not normal. This is usually caused by some configuration error or the author is using some invalid options in their .htaccess file.

Visitors to alintet also saw the error message when they posted a comment, though again if the page was refreshed their message appeared.

Initially, most of my time was spent trying to find the messages that should have been written to the server logs when this happened. I eventually found them.

Here's one from an Apache access log:

152.2.21.46 - [02/Feb/2004:16:13:17 -0500] "POST /usr-bin/pennock/MT-2.64/mt-comments.cgi HTTP/1.1" 500 3663 "" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) Opera 7.11 [en]" 4

152.2.21.46 is the ip address and domain of the computer browsing the site Then there's a timestamp, the action being attempted, a POST using the comments.cgi script and the HTTP/1.1 protocol. The "500" is the error code generated when this was attempted. "3663" is the number of bytes transferred to the page, in this case

http://www.unc.edu/usr-bin/pennock/MT-2.64/mt-comments.cgi?entry_id=42

The rest is browser and computer information about the machine at 152.2.21.46 It's running Windows NT 5.0 and using an english language version of the Opera browser to surf the net. The terminal "4" is the number of seconds in real time the process took to complete.

It's a lot of information, but by itself it means nothing. The timestamp and ip address do match those of an message from the webserver's error logs, however.

[Mon Feb 2 16:13:17 2004] [error] [client 152.2.21.46] Premature end of script headers: /afs/fakedpath/morefakedpath/yetmorefakery/kwrapper

Now that means something, though it contains less actual information than the entry above.

In order for a UNC student or faculty member to run an MT blog they must first sign up for a service called Personal CGI Access. The service creates a special directory in the requestor's personal space for use as a cgi-bin. All the scripts in this directory, including the MT scripts, run under a custom cgi-wrapper called kwrap, for security reasons.

That's what the long path beginning with /afs/ is. I've munged the actual path on general principles. The "Premature end of script headers" means that somehow the MT blogs are breaking the kwrap application.

There are a number of ways a user can break kwrap that will produce a log entry like the one above. One of the most common is to specify an incorrect path to perl or php. That wasn't the case in the present circumstances. Nor did I manage to trace the outage to any of the other possibilities.

Fortunately the guy who wrote kwrap works right down the hall from me. Between us we figured out what was going on.

There's a configuration file for the kwrap on each www.unc.edu server that restricts, among other things, the number of seconds of CPU time alloted for a script to run. The default setting is to allow 2 seconds for script execution. We set it to 2 seconds for no particular reason when kwrap was first installed, over a year ago. Never had any problem with it

However, It appears that Movable type blogs, once they reach a certain size, bump up against this limit, first during a site rebuild, then when comments are submitted.

It turns out this is a known problem in the Movable Type world, not that we knew this at the time.

In the end, our solution was the same one offered by one of the posters in the link above. We kicked up the CPU time alloted for script execution, first to 4 seconds, then to 5. Voila! No more Internal Service Errors.

However, kicking the time allotted up brings problems of its own, in that increases the potential load on the servers. If over the next week we notice that unacceptable load spikes are occuring more frequently, then we'll either have to back the script execution time allowed back down or add another server to the www.unc.edu pool.

We don't have any more servers, at the moment. However, the load does not appear to have gone up appreciably in the hours since I made the change, so we might be ok.

Posted by Bigwig at February 4, 2004 10:56 AM | TrackBack
Postscript:
First time visitor to House Hraka? Wondering if everything we produce could possibly be as brilliant/stupid/evil/pedantic/insipid/inspired as the post you just read? Check out the Hraka Essentials, the (mostly) reader-selected guide to Hraka's best posts, and decide for yourself.
Comments


thanks for fixing my blog. you rock!

- kevin

Posted by: kevin at March 31, 2004 11:03 PM
Post a comment Note: Comments with more than two dashes per line will be blocked as spam.









Remember personal info?