Front page
Silflay Hraka?

Bigwig is a systems administrator at a public university
Hrairoo is the proprietor of a quality used bookstore
Kehaar is.
Woundwort is a professor of counseling at a private university

The Hraka RSS feed

bigwig AT

Friends of Hraka
Daily Pundit
cut on the bias
Meryl Yourish
This Blog Is Full Of Crap
Winds of Change
A Small Victory
Silent Running
Dr. Weevil
Little Green Footballs
Fragments from Floyd
The Feces Flinging Monkey
Dean's World
Little Tiny Lies
The Redsugar Muse
Natalie Solent
From the Mrs.
The Anti-Idiotarian Rottweiler
On the Third Hand
Public Nuisance
Not a Fish
Electric Venom
Skippy, The Bush Kangaroo
Common Sense and Wonder
Neither Here Nor There
The Greatest Jeneration
Ipse Dixit
Blog On the Run
Redwood Dragon
Greeblie Blog
Have A Cuppa Tea
A Dog's Life
Iberian Notes
Midwest Conservative Journal
A Voyage to Arcturus
Trojan Horseshoes
In Context
The People's Republic of Seabrook
Country Store
Blog Critics
Chicago Boyz
Hippy Hill News
Kyle Still Free Press
The Devil's Excrement
The Fat Guy
War Liberal
Assume the Position
Balloon Juice
Iron Pen In A Velvet Glove
Freedom Lives
Where Worlds Collide
Knot by Numbers
How Appealing
South Knox Bubba
Heretical Ideas
The Kitchen Cabinet
Bo Cowgill
Raving Atheist
The Short Strange Trip
Shark Blog
Ron Bailey's Weblog
Cornfield Commentary
Northwest Notes
The Blog from the Core
The Talking Dog
WTF Is It Now??
Blue Streak
Smarter Harper's Index
nikita demosthenes
Bloviating Inanities
Sneakeasy's Joint
Ravenwood's Universe
The Eleven Day Empire
World Wide Rant
All American
The Rant
The Johnny Bacardi Show
The Head Heeb
Viking Pundit
Oscar Jr. Was Here
Just Some Poor Schmuck
Katy & Bruce Loebrich
But How's The Coffee?
Roscoe Ellis
Sasha Castel
Susskins Central Dispatch
Josh Heit
Aaron's Rantblog
As I was saying...
Blog O' Dob
Dr. Frank's Blogs Of War
Betsy's Page
A Knob for Brightness
Fresh Bilge
The Politburo Diktat
Drumwaster's rants
Curt's Page
The Razor
An Unsealed Room
The Legal Bean
Helloooo chapter two!
As I Was Saying...
SkeptiLog AGOG!
Tong family blog
Vox Beth
I was thinking
Judicious Asininity
This Woman's Work
Fragrant Lotus
Single Southern Guy
Jay Solo's Verbosity
Snooze Button Dreams
You Big Mouth, You!
From the Inside looking Out
Night of the Lepus
No Watermelons Allowed
From The Inside Looking Out
Lies, Damn Lies, and Statistics
Suburban Blight
The SmarterCop
Dog of Flanders
From Behind the Wall of Sleep
Beaker's Corner
Bad State of Gruntledness
Who Tends The Fires
Granny Rant
Elegance Against Ignorance
Say What?
Blown Fuse
Wait 'til Next Year
The Pryhills
The Whomping Willow
The National Debate
The Skeptician
Zach Everson
Geekward Ho
Life in New Orleans
Rotten Miracles
The Biomes Blog
See What You Share
Blog d’Elisson
Your Philosophy Sucks
Watauga Rambler
Socialized Medicine
Verging on Pertinence
Read My Lips
The Flannel Avenger
Butch Howard's WebLog
Castle Argghhh!
Andrew Hofer
Moron Abroad
White Pebble
Darn Floor
Pajama Pundits
Goddess Training 101
A & W
Medical Madhouse
Slowly Going Sane
The Oubliette
American Future
Right Side Redux
See The Donkey
Newbie Trucker
The Right Scale
Running Scared
Ramblings Journal
Focus On Reality
Wyatt's Torch

December 16, 2003

Moving Tomcat and Cocoon Logs

Got paged at 6:30 this morning. The /usr filesystem on one of the servers was at 90% of allocated space and growing. Bad things happen if a filesystem uses up 100% of its allocated space, so we set the system to page us when it hits 90%.

If you think of a filesystem as a hard drive, then you'll have a useful metaphor for understanding what was going on. It's an incorrect description, but metaphors do not require a hard base in reality in order to successfully convey information.

As per usual, the culprit turned out to be the log files (after a ship's log. Most everything the application does is written to a file. It can be configured so that more or less information is written.) One of the Tomcat/Cocoon instances on that machine had become log-mad all of a sudden, filling it with reams of information. This happens fairly regularly, once every other month or so. I deleted the logs for each, which by default are in two separate locations, figured it was too late to go back to bed, and started the coffee and biscuits instead.

After a while, as I sat at the kitchen table, I decided once every other month was too damn often.

I spent the rest of the morning reconfiguring the logs on that machine* so that they write to another file system. It will eventually fill up as well, but there is more free space there to begin with, and should there be another crisis on that machine, they will be easier to find--all in the same place, rather than scattered about. Setting up a system-wide log rotation for them is a little easier, as well. I suspect I can alter the command we use to rotate the older apache logs out to do that in about five minutes.

Here's a picture of the file system on the machine in question, taken after I deleted the logs this morning.

Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/md3 1035596 130640 852348 13% /
none 642840 0 642840 0% /dev/shm
/dev/md1 1035596 32908 950080 3% /tmp
/dev/md2 4127040 2520372 1397024 64% /usr
/dev/md0 4127040 1579180 2338216 40% /var
AFS 9000000 0 9000000 0% /afs

/usr was at 68% when I got in this morning, so it was growing at a rate of 2% an hour. Had it kept up that pace, I would have gotten another page sometime tonight. There's obviously something else going on here, I'll have to see what it might be.

For a typical Tomcat instance, logs are written to two different places. On my server they can be found at /usr/local/tomcat/logs and /usr/local/tomcat/webapps/cocoon/WEB_INF/logs.

Two locations, but (typically) three different files need to be changed in order make sure that all the logs written to each are moved. They are;

/usr/local/tomcat/conf/server.xml and

The creates the /usr/local/tomcat/logs/catalina.out file. I wanted catalina.out to go to /var/log/tomcat/ instead, as I did all the logs. In order to do this, I had to replace the bolded code in the snippet below with /usr/local/tomcat/logs/catalina.out

touch "$CATALINA_BASE"/logs/catalina.out
if [ "$1" = "-security" ] ; then
echo "Using Security Manager"
-Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" -classpath "$CLASSPATH" \ \"$CATALINA_BASE"/conf/catalina.policy \
-Dcatalina.base="$CATALINA_BASE" \
-Dcatalina.home="$CATALINA_HOME" \"$CATALINA_TMPDIR" \
org.apache.catalina.startup.Bootstrap "$@" start \
>> "$CATALINA_BASE"/logs/catalina.out 2>&1 &
-Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" -classpath "$CLASSPATH" \
-Dcatalina.base="$CATALINA_BASE" \
-Dcatalina.home="$CATALINA_HOME" \"$CATALINA_TMPDIR" \
org.apache.catalina.startup.Bootstrap "$@" start \
>> "$CATALINA_BASE"/logs/catalina.out 2>&1 &

Simple enough. The /usr/local/tomcat/conf/server.xml was slightly more annoying, in that it configured multiple logs two different ways. One config would look like this:
< Logger className="org.apache.catalina.logger.FileLogger"

The other, like this;
< Valve className="org.apache.catalina.valves.AccessLogValve"

I've bolded the essential difference. Both use what is called a relative path to tell the application where to write the logs, in that the location where the log is written is relative to where the config file is, rather than an absolute path which is relative to the unchanging root directory. I need to change the relative paths in the snippets above to absolute paths.

Once that is done, they'll look like this;
< Logger className="org.apache.catalina.logger.FileLogger"

< Valve className="org.apache.catalina.valves.AccessLogValve"

Once again, simple if you know what to do and where to do it. The trick lies in learning those two things. Like the joke says

A guy called the computer repairman to come and fix his computer. The repairman showed up, pushed a button and the computer started working. He then handed the guy a bill for $250.49.

"$250.49! What's that for? All you did was push one button!"

The repairman replied, "The $0.49 is for pushing the button. The $250 is for knowing which button to push."

Instead of relative paths, the /usr/local/tomcat/webapps/cocoon/WEB_INF/logkit.xconf uses absolute paths with a variable embedded. The entries (four of them--one each for the error, core, sitemap and access logs) are fairly straightforward, though, appearing something like this;

< filename>${context-root}/WEB-INF/logs/core.log </filename>

${context-root} is the variable, referring to the filesystem path that exists above WEB-INF/logs. Whatever the program has decided the path should be is what gets written there. I don't have to worry about what the app thinks--I'm just going to delete everything in front of "core.log", so that the entry will read

< filename>/var/log/tomcat/core.log </filename>

Then I'll spend...let's see.....three hours, about three times as long as it took to do the steps above, describing them.

*From eight separate Tomcat instances, in case you're wondering why it's taken me so long. And yes, I realize I could roll all of them into one instance and save myself trouble. The problem with doing that is that each of these Tomcats is used by a different department or group within the university, including:

The Robertson Scholars
Human Resources
The Carolina Center for Public Service
The North Carolina Law Review
and part of the UNC Portal

Were they all served by a single Tomcat instance, then when it broke or was taken down for maintenance, none of the services it offers would be available. So each group has an instance of its very own.

02/03/04 Update: For the moment, this post is the #1 result for "catalina.out rotate every day" on Google. It's also pretty highly ranked for "cocoon logs," which means 4 or 5 people a day stop by in search of help.

In case ya'll out there are just looking for a quick way to reduce the size of the catalina.out without impacting the tomcat app running it, try this command.

#cat /dev/null > catalina.out.

That should erase the contents of the log without interrupting the parent Tomcat process, or preventing it from writing to the file in future.

Posted by Bigwig at December 16, 2003 01:30 PM | TrackBack
First time visitor to House Hraka? Wondering if everything we produce could possibly be as brilliant/stupid/evil/pedantic/insipid/inspired as the post you just read? Check out the Hraka Essentials, the (mostly) reader-selected guide to Hraka's best posts, and decide for yourself.

Have you looked into logrotate, or similar software?

Might be a useful additional prophylactic (since it's obviously too late to save you the effort of moving the logs).

Posted by: Sigivald at December 16, 2003 04:00 PM

I found the following link which might be useful to you

I'm a home user of tomcat and i had problems with
the catalina.out file being to big.

I deleted it but for some reason the space being
used by the log has not become free any suggestions?

Posted by: Daniel Thomas at April 9, 2004 05:55 AM
Post a comment Note: Comments with more than two dashes per line will be blocked as spam.

Remember personal info?