Trucluster (5.1a pk6) CDSL problems on cluster_root

Jack Estes · *nix forums beginner Joined: 11 Oct 2005 Posts: 14

Eric, while I completely agree with you, unfortunately as an instrument
of policy, I haven't the authority to command an immediate change. I
have made known the tragic flaws in the previous configuration and have
taken some steps to prevent a recurrence.

In the end, there was just no saving the cluster with all the damage
done to the CDSLs and member boot disks. Since the data domains were
safe and sound on the SAN, I just made a completely new 5.1B-3 build
disk, created a brand new cluster, and reattached the old data domains.
All of the Oracle configuration files were in these domains so there
were no special tweaks that had to be recreated on the new cluster. I
was able to remove about 95% of the user accounts (this is a database
server and really only requires sysadmin accounts and a service account
for the app servers to log on with). I was able to can telnetd, allow
only SSH2, and implement tcp wrappers for all network communications.
That should go pretty far since the system is completely segregated
from the rest of the intranet and has no connection to the Internet.

My thanks to Tom Smith @ HP and Adam Price for valuable assistance.

Eric de Redelijkheid · *nix forums addict Joined: 29 Mar 2005 Posts: 55

Anno Domini 11-10-2005 7:21, Adam Price sprak aldus:

Adam Price · *nix forums beginner Joined: 31 May 2005 Posts: 23

On 10 Oct 2005 18:53:06 -0700, Jack Estes wrote:

Jack Estes · *nix forums beginner Joined: 11 Oct 2005 Posts: 14

I think I'm screwed on this one, but I thought I'd post to the group to
see if anyone else has seen this.

I have a two node trucluster serving an oracle 8i database and an
application originally written as single-instance so CAA manages it.
The cluster interconnect is memory channel and storage is fiber
channel-connected dual HSG80s into 3 StorageWorks SCSI enclosures.

For reasons passing understanding, about 40 people know the root
password on this cluster and by department policy, root is allowed to
login remotely. I'm assuming because this cluster is on a private
network serving only this department and is physically disconnected
from anything else, they think this is a good idea.

On to my problem: A user reluctantly admitted to me that he forgot his
user account password so he went ahead and logged in as root from his
PC. Thinking he was in his home directory, he executed an mv on /
thinking he would only move the files in his home directory to another
NFS mounted drive he used as a backup device. Shortly into this
process, I think he figured out he was getting the wrong stuff, killed
the command, then tried to mv the stuff back to /. In doing so, he
obviously destroyed all the CDSLs for both nodes on the cluster /,
/usr, /etc, and /var filesystems. Within about 10 minutes of this
catastrophe, a thermal breaker blew in the cabinet and uncerimoniously
shut off power to both nodes. Neither node can boot now because, among
other things, the cluster database is unavailable to them and neither
node has a unique identity (there's just one /etc/* between them).

I CAN boot the first member via the old build disk on the internal bus
and see down the HSGs to the AdvFS domains on the other side, and
they're all good. In fact, I mounted them read-only by monkeying
around in /etc/fdmns on the build disk to make temp domains and
filesets so I'm not worried about. That's how I found out there were no
more CDSLs on the cluster filesystem. The quorum disk out there is
perfectly intact as well. The entire application and oracle database
are on some filesets out on the SAN and the startup scripts are
manually executed so I'm not worried about recreating anything other
than users (I know the old UIDs to make home directories match up right
away, etc...)

I'm wondering if there's a reliable way to recreate the links and put
things back where they belong (um, like the correct /etc/sysconfigtab)
maybe with some combination of clu_delete_member and clu_add_member or
if I'm better off time-wise, to just finally upgrade to 5.1B-3 (i'm
licensed and have the media) by doing a fresh install and remounting
the AdvFS data domains on the SAN. LSM was not managing any disks.

Any help is graciously accepted. Thanks!

Jack
________________________
Jack M. Estes II, Ph.D.
Cinergy Corporation
1000 Main Street
Plainfield, Indiana 46168
PHONE #s STRIPPED FOR USENET

Google