Front page
Archive
Silflay Hraka?


Bigwig is a systems administrator at a public university
Hrairoo is the proprietor of a quality used bookstore
Kehaar is.
Woundwort is a professor of counseling at a private university

The Hraka RSS feed

Email
bigwig AT nc.rr.com

Friends of Hraka
InstaPundit
Daily Pundit
cut on the bias
Meryl Yourish
This Blog Is Full Of Crap
Winds of Change
A Small Victory
Silent Running
Dr. Weevil
Little Green Footballs
ColdFury
Oceanguy
Fragments from Floyd
VodkaPundit
Allah
The Feces Flinging Monkey
Dean's World
Little Tiny Lies
The Redsugar Muse
Sperari
Natalie Solent
From the Mrs.
ErosBlog
The Anti-Idiotarian Rottweiler
On the Third Hand
Public Nuisance
Not a Fish
Rantburg
AMCGLTD
WeckUpToThees!
Electric Venom
Skippy, The Bush Kangaroo
Common Sense and Wonder
Neither Here Nor There
Wizbang!
Bogieblog
ObscuroRant
RocketJones
The Greatest Jeneration
Ravenwolf
Ipse Dixit
TarHeelPundit
Blog On the Run
blogatron
Redwood Dragon
Notables
Greeblie Blog
Have A Cuppa Tea
A Dog's Life
IMAO
Zonitics.com
Iberian Notes
Midwest Conservative Journal
A Voyage to Arcturus
HokiePundit
Trojan Horseshoes
In Context
dcthornton.blog
The People's Republic of Seabrook
Country Store
Blog Critics
Chicago Boyz
Hippy Hill News
Kyle Still Free Press
The Devil's Excrement
The Fat Guy
War Liberal
Assume the Position
Balloon Juice
Iron Pen In A Velvet Glove
IsraPundit
Freedom Lives
Where Worlds Collide
Knot by Numbers
How Appealing
South Knox Bubba
Heretical Ideas
The Kitchen Cabinet
Dustbury.com
tonecluster
Bo Cowgill
mtpolitics.net
Raving Atheist
The Short Strange Trip
Shark Blog
Hoplites
Jimspot
Ron Bailey's Weblog
Cornfield Commentary
Testify!
Northwest Notes
pseudorandom
The Blog from the Core
Ain'tNoBadDude
CroMagnon
The Talking Dog
WTF Is It Now??
Blue Streak
Smarter Harper's Index
nikita demosthenes
Bloviating Inanities
Sneakeasy's Joint
Ravenwood's Universe
The Eleven Day Empire
World Wide Rant
All American
Pdawwg
The Rant
The Johnny Bacardi Show
The Head Heeb
Viking Pundit
Mercurial
Oscar Jr. Was Here
Just Some Poor Schmuck
Katy & Bruce Loebrich
But How's The Coffee?
Roscoe Ellis
Foolsblog
Sasha Castel
Dodgeblogium
Susskins Central Dispatch
DoggerelPundit
Josh Heit
Attaboy
Aaron's Rantblog
MojoMark
As I was saying...
Blog O' Dob
Dr. Frank's Blogs Of War
Betsy's Page
A Knob for Brightness
Fresh Bilge
The Politburo Diktat
Drumwaster's rants
Curt's Page
The Razor
An Unsealed Room
The Legal Bean
Helloooo chapter two!
As I Was Saying...
SkeptiLog AGOG!
Tong family blog
Vox Beth
Velociblog
I was thinking
Judicious Asininity
This Woman's Work
Fragrant Lotus
DaGoddess
Single Southern Guy
Caerdroia
GrahamLester.Com
Jay Solo's Verbosity
TacJammer
Snooze Button Dreams
Horologium
You Big Mouth, You!
From the Inside looking Out
Night of the Lepus
No Watermelons Allowed
From The Inside Looking Out
Lies, Damn Lies, and Statistics
Suburban Blight
Aimless
The SmarterCop
Dog of Flanders
From Behind the Wall of Sleep
Beaker's Corner
Bad State of Gruntledness
Who Tends The Fires
Granny Rant
Elegance Against Ignorance
Moxie.nu
Eccentricity
Say What?
Blown Fuse
Wait 'til Next Year
The Pryhills
The Whomping Willow
The National Debate
The Skeptician
Zach Everson
MonkeyWatch
Geekward Ho
Argghhh!!!
Life in New Orleans
Rotten Miracles
Fringe
The Biomes Blog
illinigirl
See What You Share
Truthprobe
Blog d’Elisson
Your Philosophy Sucks
Watauga Rambler
Socialized Medicine
Consternations
Verging on Pertinence
Read My Lips
ambivablog
Soccerdad
The Flannel Avenger
Butch Howard's WebLog
Castle Argghhh!
Andrew Hofer
kschlenker.com
Moron Abroad
White Pebble
Darn Floor
Wizblog
tweedler
Pajama Pundits
BabyTrollBlog
Cadmusings
Goddess Training 101
A & W
Medical Madhouse
Slowly Going Sane
The Oubliette
American Future
Right Side Redux
See The Donkey
Newbie Trucker
The Right Scale
Running Scared
Ramblings Journal
Focus On Reality
Wyatt's Torch

March 07, 2005

A Procedure For Replacing A Bad Drive On A Sun Netra AC200

I realize just how exciting a title that is for Hraka's non-technical audience, so I've moved the bulk of the post to the "Continue Reading" section

An SOP, in which is described procedure to follow in case of a hard drive failure on a Sun Netra AC200 system, such as the one that greeted me this morning upon my arrival at the office. Wherever one sees a "DO NOT," assume that I've done that as some point in my life.

And yes, it's a fairly elderly machine. Four years or so, I think.

If a single drive fails, the system should continue to operate normally without any intervention. The Netra's Online DiskSuite software will automatically disable failed submirrors and write a message to syslog.

In this example, we have two hard drives in a RAID 1 configuration. There are four mirrors, each with two sub-mirrors.

Use the "metastat " command to get a complete look at the metadevice configuration and status of all components (state, disk slices, labels, etc.) Here's an example of that output. The bad submirrors are in the "Needs maintenance" state.

# metastat
d10: Mirror
Submirror 0: d11
State: Needs maintenance
Submirror 1: d12
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 4198392 blocks

d11: Submirror of d10
State: Needs maintenance
Invoke: metareplace d10 c0t0d0s0
Size: 4198392 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c0t0d0s0 0 No Maintenance

d12: Submirror of d10
State: Okay
Size: 4198392 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c0t1d0s0 0 No Okay

d20: Mirror
Submirror 0: d21
State: Okay
Submirror 1: d22
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 2101552 blocks

d21: Submirror of d20
State: Okay
Size: 2101552 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c0t0d0s1 0 No Okay

d22: Submirror of d20
State: Okay
Size: 2101552 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c0t1d0s1 0 No Okay

d30: Mirror
Submirror 0: d31
State: Needs maintenance
Submirror 1: d32
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 8392072 blocks

d31: Submirror of d30
State: Needs maintenance
Invoke: metareplace d30 c0t0d0s3
Size: 8392072 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c0t0d0s3 0 No Maintenance

d32: Submirror of d30
State: Okay
Size: 8392072 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c0t1d0s3 0 No Okay

d40: Mirror
Submirror 0: d41
State: Needs maintenance
Submirror 1: d42
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 8392072 blocks

d41: Submirror of d40
State: Needs maintenance
Invoke: metareplace d40 c0t0d0s4
Size: 8392072 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c0t0d0s4 0 No Maintenance

d42: Submirror of d40
State: Okay
Size: 8392072 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c0t1d0s4 0 No Okay

Step 1. Detach bad submirrors from mirrors.

Command: `metadetach -f d10 d11`.
The -f option is required to force the detachment due to the submirror being in an errored state. "d10" is the mirror. "d11" is the submirror. Repeat this step for each bad submirror.

2. Clear all bad metadevices.

Command:`metaclear d11`.
Repeat this step for each bad submirror.

3. Clear out all bad metadatabases.

Command: `metadb -d cxtxd0sx` where x corresponds to the respective controller, target and slice of the bad section of disk that has old metadatabases. If you do not already know that information, then running the command `metadb -i` will give it to you.

4. Remove bad disk from system.

After taking note of the proper drive number, physically remove the drive from the system. In our case, those drives are labeled from left to right, starting with drive 0. Typically c0t0d0 refers to drive 0 and c0t1d0 refers to drive 1. These drives are hot-swappable, meaning that one does not have to shut down the server in order to replace the hard drive.

Make sure that you don't remove the good drive. That would be bad.

5. Install new drive. It should slide right in. Test by attempting to pull it back out. If it's jammed, then you shouldn't have used the hammer to install it in the first place.

6. Verify that new disk drive is operational.

Command: 'format'
This will ensure that all disk drivess are visible. Here's output from that command for a badly seated or broken drive, in this case, "0."

# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
0. c0t0d0
/pci@1f,0/pci@1/scsi@8/sd@0,0
1. c0t1d0
/pci@1f,0/pci@1/scsi@8/sd@1,0

Try reseating the new disk drive if you see the above.

7. Create identical slices identical to old drive on the new one. Display the partition layout on the current (good) drive, using the format command. Format the new disk to match the old disk, cylinder for cylinder. Remember, this is a mirror. The partitions must be identical.

EXAMPLE OUTPUT (from this morning, actually) Comments are in bold.

# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
0. c0t0d0
/pci@1f,0/pci@1/scsi@8/sd@0,0 Note the difference from the above format command
1. c0t1d0
/pci@1f,0/pci@1/scsi@8/sd@1,0
Specify disk (enter its number): 1
Get the information we need from the good disk
selecting c0t1d0
[disk formatted]

FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
save - save new disk/partition definitions
inquiry - show vendor, product and revision
volname - set 8-character volume name
! - execute , then return
quit
format> current
Current Disk = c0t1d0

/pci@1f,0/pci@1/scsi@8/sd@1,0

format> partition

PARTITION MENU:
0 - change `0' partition
1 - change `1' partition
2 - change `2' partition
3 - change `3' partition
4 - change `4' partition
5 - change `5' partition
6 - change `6' partition
7 - change `7' partition
select - select a predefined table
modify - modify a predefined partition table
name - name the current table
print - display the current table
label - write partition map and label to the disk
! - execute , then return
quit
partition> print
Current partition table (original):
Total disk cylinders available: 7506 + 2 (reserved cylinders)

Yes, we have more slices than we do mirrors.

Part Tag Flag Cylinders Size Blocks
0 unassigned wm 0 - 890 2.00GB (891/0/0) 4198392
1 unassigned wm 891 - 1336 1.00GB (446/0/0) 2101552
2 backup wm 0 - 7505 16.86GB (7506/0/0) 35368272
3 unassigned wm 1337 - 3117 4.00GB (1781/0/0) 8392072
4 unassigned wm 3118 - 4898 4.00GB (1781/0/0) 8392072
5 unassigned wm 4904 - 5349 1.00GB (446/0/0) 2101552
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 4899 - 4903 11.50MB (5/0/0) 23560

partition> quit

format> disk

AVAILABLE DISK SELECTIONS:
0. c0t0d0
/pci@1f,0/pci@1/scsi@8/sd@0,0
1. c0t1d0
/pci@1f,0/pci@1/scsi@8/sd@1,0
Specify disk (enter its number)[1]: 0

Now that we have the information we need from the good disk, we apply it to the new one.
selecting c0t0d0
[disk formatted]
format> partition


PARTITION MENU:
0 - change `0' partition
1 - change `1' partition
2 - change `2' partition
3 - change `3' partition
4 - change `4' partition
5 - change `5' partition
6 - change `6' partition
7 - change `7' partition
select - select a predefined table
modify - modify a predefined partition table
name - name the current table
print - display the current table
label - write partition map and label to the disk
! - execute , then return
quit
partition> 0
Part Tag Flag Cylinders Size Blocks
0 root wm 0 - 55 128.84MB (56/0/0) 263872

Enter partition id tag[root]: un unassigned
Enter partition permission flags[wm]: Accepting the default
Enter new starting cyl[0]: 0
Enter partition size[263872b, 56c, 128.84mb, 0.13gb]: 891c "c" is for cylinders, unless you're Cookie Monster
partition> 1
Part Tag Flag Cylinders Size Blocks
1 swap wu 56 - 111 128.84MB (56/0/0) 263872

Enter partition id tag[swap]: un
Enter partition permission flags[wu]: wm Note we did NOT accept the default here
Enter new starting cyl[56]: 891
Enter partition size[263872b, 56c, 128.84mb, 0.13gb]: 446c
partition> 3
Part Tag Flag Cylinders Size Blocks
3 unassigned wm 0 0 (0/0/0) 0

Enter partition id tag[unassigned]: un
Enter partition permission flags[wm]:
Enter new starting cyl[0]: 1337
Enter partition size[0b, 0c, 0.00mb, 0.00gb]: 1781c
partition> 4
Part Tag Flag Cylinders Size Blocks
4 unassigned wm 0 0 (0/0/0) 0

Enter partition id tag[unassigned]: un
Enter partition permission flags[wm]:
Enter new starting cyl[0]: 3118
Enter partition size[0b, 0c, 0.00mb, 0.00gb]: 1781c
partition> 5
Part Tag Flag Cylinders Size Blocks
5 unassigned wm 0 0 (0/0/0) 0

Enter partition id tag[unassigned]: un
Enter partition permission flags[wm]:
Enter new starting cyl[0]: 4904
Enter partition size[0b, 0c, 0.00mb, 0.00gb]: 446c
partition> 6
Part Tag Flag Cylinders Size Blocks
6 usr wm 112 - 7505 16.61GB (7394/0/0) 34840528

Sun throws me a curve here, with some unexpected defaults. I ignore them.
Enter partition id tag[usr]: un
Enter partition permission flags[wm]:
Enter new starting cyl[112]: 0
Enter partition size[34840528b, 7394c, 17011.98mb, 16.61gb]: 0
partition> 7
Part Tag Flag Cylinders Size Blocks
7 unassigned wm 0 0 (0/0/0) 0

Enter partition id tag[unassigned]: un
Enter partition permission flags[wm]:
Enter new starting cyl[0]: 4899
Enter partition size[0b, 0c, 0.00mb, 0.00gb]: 5c
partition>
partition> print
Current partition table (unnamed):
Total disk cylinders available: 7506 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 unassigned wm 0 - 890 2.00GB (891/0/0) 4198392
1 unassigned wm 891 - 1336 1.00GB (446/0/0) 2101552
2 backup wu 0 - 7505 16.86GB (7506/0/0) 35368272
3 unassigned wm 1337 - 3117 4.00GB (1781/0/0) 8392072
4 unassigned wm 3118 - 4898 4.00GB (1781/0/0) 8392072
5 unassigned wm 4904 - 5349 1.00GB (446/0/0) 2101552
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 4899 - 4903 11.50MB (5/0/0) 23560

partition> label
Ready to label disk, continue? y

partition> quit

8. Add metadatabases to new disk.

Execute `metadb -a -c 2 c0t0d0s7`. This creates 2 metadatabases on the selected drive in slice 7, where they are typically located.

9. Initialize new metadevices.

Execute `metainit d12 1 1 cxtxd0sx`, where x corresponds to the respective controller, target and slice on the new disk.

Repeat this step for each submirror, like so;

# metainit d11 1 1 c0t0d0s0
d11: Concat/Stripe is setup
# metainit d31 1 1 c0t0d0s3
d31: Concat/Stripe is setup
# metainit d41 1 1 c0t0d0s4
d41: Concat/Stripe is setup

10. Attach new submirrors to mirrors.

Execute `metattach d10 d11`.

DO NOT repeat this step for each new submirror, as it will cause the disk to undergo excessive thrashing. DO steps 11 and 12 first!

11. Verify progress of synchronization.

Execute `metastat d10` and monitor the progress measured in % of resync. Wait for this to reach 100% before moving on to the next metadevice.

12. Repeat steps 10 and 12 for each metadevice.

Posted by Bigwig at March 7, 2005 12:57 PM | TrackBack
Postscript:
First time visitor to House Hraka? Wondering if everything we produce could possibly be as brilliant/stupid/evil/pedantic/insipid/inspired as the post you just read? Check out the Hraka Essentials, the (mostly) reader-selected guide to Hraka's best posts, and decide for yourself.
Comments

Oh how I miss Solaris. I work in a mostly HP-UX shop.

I used to do stuff like that, now I can barely remember the DiskSuite commands. I sort of remember, though, that if you have several disks with identical layouts (or you want to duplicate an existing layout on a clean disk) you might want to look into 'prtvtoc' and 'fmthard'.

Posted by: Greg at March 11, 2005 03:44 AM
Post a comment Note: Comments with more than two dashes per line will be blocked as spam.









Remember personal info?