Prompt.cz

1.1. Install "pcs" and "fence-agents-all" on each cluster node:

# yum install -y pcs fence-agents-all

1.2. Enable and start the "pcsd" service on each cluster node:

# systemctl enable pcsd
# systemctl start pcsd

1.3. If needed, enable cluster communication through the firewall on each cluster node:

# firewall-cmd --permanent --add-service=high-availability
# firewall-cmd --reload

1.4. Change the password of the "hacluster" user to "H4clust3r" on each cluster node:

# echo "H4clust3r" | passwd --stdin hacluster

1.5. Authenticate all cluster nodes on any cluster node:

[root@node1 ~]# pcs cluster auth node1 node2 node3
Username: hacluster
Password: H4clust3r
node1: Authorized
node2: Authorized
node3: Authorized

1.6. Configure and start a three-node cluster named "cluster01" on any cluster node:

[root@node1 ~]# pcs cluster setup --start --name cluster01 node1 node2 node3

1.7. Enable automatic startup of the cluster on all configured cluster nodes on any cluster node:

[root@node1 ~]# pcs cluster enable --all

1.8. Verify that the cluster is running and all configured cluster nodes have joined the cluster on any cluster node:

[root@node1 ~]# pcs status
Cluster name: cluster01
WARNING: no stonith devices and stonith-enabled is not false
Last updated: Mon Sep 15 05:41:18 2014
Last change: Mon Sep 15 05:41:03 2014 via crmd on node1
Stack: corosync
Current DC: node1 (1) - partition with quorum
Version: 1.1.10-29.el7-368c726
3 Nodes configured
0 Resources configured

Online: [ node1 node2 node3 ]

Full list of resources:

PCSD Status:
node1: Online
node2: Online
node3: Online

Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled

1.9. Add the fence devices for the virtual machines "node1", "node2" and "node3" to the cluster on any cluster node:

[root@node1 ~]# pcs stonith create fence_node1 fence_ipmilan pcmk_host_list="node1" ipaddr="172.24.6.1" login="node1" passwd="N0d31" lanplus=1 cipher=3
[root@node1 ~]# pcs stonith create fence_node2 fence_ipmilan pcmk_host_list="node2" ipaddr="172.24.6.1" login="node2" passwd="N0d32" lanplus=1 cipher=3
[root@node1 ~]# pcs stonith create fence_node3 fence_ipmilan pcmk_host_list="node3" ipaddr="172.24.6.1" login="node3" passwd="N0d33" lanplus=1 cipher=3

1.10. Verify that the fence devices have been added correctly to the cluster on any cluster node:

[root@node1 ~]# pcs stonith show
fence_node1 (stonith:fence_ipmilan): Started
fence_node2 (stonith:fence_ipmilan): Started
fence_node3 (stonith:fence_ipmilan): Started

2.1. Install "pcs" and "fence-agents-all" on the new cluster node:

[root@node4 ~]# yum install -y pcs fence-agents-all

2.2. Enable and start the "pcsd" service on the new cluster node:

[root@node4 ~]# systemctl enable pcsd; systemctl start pcsd

2.3. If needed, allow cluster communications to pass through the firewall on the new cluster node:

[root@node4 ~]# firewall-cmd --permanent --add-service=high-availability; firewall-cmd --reload

2.4. Change the password of the "hacluster" user to "H4clust3r" on the new cluster node:

[root@node4 ~]# echo "H4clust3r" | passwd --stdin hacluster

2.5. Authenticate the new node on the existing cluster on any cluster node:

[root@node1 ~]# pcs cluster auth -u hacluster -p H4clust3r node4

2.6. Add the new node to the cluster on any cluster node:

[root@node1 ~]# pcs cluster node add node4
nodea1: Corosync updated
node2: Corosync updated
node3: Corosync updated
node4: Succeeded

2.7. Authenticate "node4" with the other nodes on the new cluster node:

[root@node4 ~]# pcs cluster auth -u hacluster -p H4clust3r
node1: Authorized
node2: Authorized
node3: Authorized
node4: Already authorized

2.8. Add a fence device for the new node to the cluster on any cluster node:

[root@node1 ~]# pcs stonith create fence_node4 fence_ipmilan pcmk_host_list="node4" ipaddr="172.24.6.1" login="node4" passwd="N0d34" lanplus=1 cipher=3

2.9. Verify that the fence device has been added correctly to the cluster on any cluster node:

[root@node1 ~]# pcs stonith show
fence_node1 (stonith:fence_ipmilan): Started
fence_node2 (stonith:fence_ipmilan): Started
fence_node3 (stonith:fence_ipmilan): Started
fence_node4 (stonith:fence_ipmilan): Started

2.10. Enable automatic startup of the cluster on the new cluster node:

[root@node4 ~]# pcs cluster enable

2.11. Start the cluster on the new cluster node:

[root@node4 ~]# pcs cluster start
Starting Cluster...

2.12. Verify that the cluster is running and all configured cluster nodes have joined the cluster on any cluster node:

[root@node1 ~]# pcs status
Cluster name: cluster01
Last updated: Tue Sep 24 05:41:18 2014
Last change: Tue Sep 24 05:41:03 2014 via crmd on node1
Stack: corosync
Current DC: node1 (1) - partition with quorum
Version: 1.1.10-29.el7-368c726
4 Nodes configured
4 Resources configured

Online: [ node1 node2 node3 node4 ]

Full list of resources:

fence_node1 (stonith:fence_ipmilan): Started node1
fence_node2 (stonith:fence_ipmilan): Started node2
fence_node3 (stonith:fence_ipmilan): Started node3
fence_node4 (stonith:fence_ipmilan): Started node4

PCSD Status:
node1: Online
node2: Online
node3: Online
node4: Online

Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled

3.1. Remove the cluster node from the cluster on any cluster node:

[root@node1 ~]# pcs cluster node remove node4

3.2. Remove the fence device from the cluster on any cluster node:

[root@node1 ~]# pcs stonith delete fence_node4

4.1. Stop the cluster on all nodes on any cluster node:

[root@node1 ~]# pcs cluster stop --all

4.2. Add the "auto_tie_breaker" option in the "quorum" section of /etc/corosync/corosync.conf on any cluster node:

[root@node1 ~]# vi /etc/corosync/corosync.conf
quorum {
provider: corosync_votequorum
auto_tie_breaker: 1
auto_tie_breaker_node: lowest
}

4.3. Synchronize /etc/corosync/corosync.conf from the current node to all other nodes in the cluster:

[root@node1 ~]# pcs cluster sync

4.4. Start the cluster on all nodes on any cluster node:

[root@node1 ~]# pcs cluster start --all

5.1. Install "iscsi-initiator-utils" on each cluster node:

# yum install -y iscsi-initiator-utils

5.2. Create a unique IQN for the iSCSI initiator by modifying the "InitiatorName" setting in /etc/iscsi/initiatorname.iscsi on each cluster node:

# vi /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2020-02.com.domain:nodeX

5.3. Enable and start the "iscsi" service on each cluster node:

# systemctl enable iscsi; systemctl start iscsi

5.4. Discover the configured iSCSI target(s) provided by the iSCSI target server portal(s) on each cluster node:

# iscsiadm -m discovery -t st -p 192.168.1.2
192.168.1.2:3390,1 iqn.2020-02.com.domain:cluster
# iscsiadm -m discovery -t st -p 192.168.1.3
192.168.1.3:3390,1 iqn.2020-02.com.domain:cluster

5.5. Log into the presented iSCSI targets on each cluster node:

# iscsiadm -m node -T iqn.2020-02.com.domain:cluster -p 192.168.1.2 -l
# iscsiadm -m node -T iqn.2020-02.com.domain:cluster -p 192.168.1.3 -l

5.6. Verify that the new block devices created by the iSCSI target logins are available on each cluster node:

# lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda      8:0    0  20G  0 disk 
sdb      8:16   0  20G  0 disk 
vda    253:0    0  10G  0 disk 
└─vda1 253:1    0  10G  0 part /

5.7. Optionally display detailed information about the available iSCSI targets on any cluster node:

# iscsiadm -m session -P 3

6.1. Install "device-mapper-multipath" on each cluster node:

# yum install -y device-mapper-multipath

6.2. Enable multipath configuration on each cluster node:

# mpathconf --enable

6.3. Edit the "blacklist" section in /etc/multipath.conf to ignore the local disks on each cluster node:

# vi /etc/multipath.conf
blacklist {
devnode "^vd[a-z]"
}

6.4. Enable and start the "multipathd" service on each cluster node:

# systemctl enable multipathd; systemctl start multipathd

6.5. Verify that the multipath device "mpatha" is available on each cluster node:

# multipath -ll
mpatha (360014053bd9ea2a35914e39a556051cf) dm-0 LIO-ORG ,clusterstor     
size=20.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 2:0:0:0 sda 8:0  active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 3:0:0:0 sdb 8:16 active ready running

7.1. Install "dlm" and "lvm2-cluster" on each cluster node:

# yum install -y dlm lvm2-cluster

7.2. Set the LVM "locking_type" to "3" (enable cluster locking) on each cluster node:

# lvmconf --enable-cluster

7.3. Stop the lvmetad service to match the modified LVM configuration on each cluster node:

# systemctl stop lvm2-lvmetad

7.4. Set the global Pacemaker parameter "no_quorum_policy" to "freeze" on any cluster node:

[root@node1 ~]# pcs property set no-quorum-policy=freeze

7.5. Create a cloned "dlm" resource (required dependency for clvmd and GFS2) using the "controld" resource agent on any cluster node:

[root@node1 ~]# pcs resource create dlm controld op monitor interval=30s on-fail=fence clone interleave=true ordered=true

7.6. Create a cloned "clvmd" resource using the "clvm" resource agent on any cluster node:

[root@node1 ~]# pcs resource create clvmd clvm op monitor interval=30s on-fail=fence clone interleave=true ordered=true

7.7. Create resource constraints to control "dlm" and "clvmd" start up order and enforce them to run on the same node on any cluster node:

[root@node1 ~]# pcs constraint order start dlm-clone then clvmd-clone
[root@node1 ~]# pcs constraint colocation add clvmd-clone with dlm-clone

7.8. Create an LVM physical volume "/dev/mapper/mpatha" on any cluster node:

[root@node1 ~]# pvcreate /dev/mapper/mpatha

7.9. Create a clustered volume group "clustervg" on any cluster node:

[root@node1 ~]# vgcreate -Ay -cy clustervg /dev/mapper/mpatha

7.10. Create a 10 GB logical volume "clusterlv" on any cluster node:

[root@node1 ~]# lvcreate -L 10G -n clusterlv clustervg

8.1. Install "gfs2-utils" on each cluster node:

# yum install -y gfs2-utils

8.2. Format the logical volume with a GFS2 file system with 3 journals on any cluster node:

[root@node1 ~]# mkfs.gfs2 -j3 -p lock_dlm -t cluster01:web /dev/clustervg/clusterlv

9.1. List all available resource agents (scripts managing cluster resources) on any cluster node:

[root@node1 ~]# pcs resource list
ocf:heartbeat:CTDB - CTDB Resource Agent
ocf:heartbeat:Delay - Waits for a defined timespan
ocf:heartbeat:Dummy - Example stateless resource agent
ocf:heartbeat:Filesystem - Manages filesystem mounts
...

9.2. Display a particular resource agent parameters on any cluster node:

[root@node1 ~]# pcs resource describe Filesystem
ocf:heartbeat:Filesystem - Manages filesystem mounts
...
Resource options:
device (required): The name of block device for the filesystem, or -U, -L
options for mount, or NFS mount specification.
directory (required): The mount point for the filesystem.
fstype (required): The type of filesystem to be mounted.
options: Any extra options to be given as -o options to mount. For bind mounts,
add "bind" here and set fstype to "none". We will do the right thing for options
such as "bind,ro".
...

9.3. Create a "clusterfs" cloned resource using the "Filesystem" resource agent to manage the GFS2 file system and automatically mount it on all three nodes on cluster startup on any cluster node:

[root@node1 ~]# pcs resource create clusterfs Filesystem device="/dev/clustervg/clusterlv" directory="/var/www" fstype="gfs2" options="noatime" op monitor interval=10s on-fail=fence clone interleave=true

9.4. Create resource constraints to control "clvmd" and "clusterfs" start up order and enforce them to run on the same node on any cluster node:

[root@node1 ~]# pcs constraint order start clvmd-clone then clusterfs-clone
[root@node1 ~]# pcs constraint colocation add clusterfs-clone with clvmd-clone

9.5. Create a "webip" resource using the "IPaddr2" resource agent for the IP address 172.15.1.10/24 as part of the "web_rg" resource group on any cluster node:

[root@node1 ~]# pcs resource create webip IPaddr2 ip=172.15.1.10 cidr_netmask=24 --group web_rg

9.6. Create a "webserver" resource using the "apache" resource agent with default settings as part of the "web_rg" resource group on any cluster node:

[root@node1 ~]# pcs resource create webserver apache --group web_rg

9.7. Configure the "web_rg" resource group to start after the "clusterfs" resource on any cluster node:

[root@node1 ~]# pcs constraint order start clusterfs-clone then web_rg

9.8. Configure the "web_rg" resource group to preferably start on "node1" on any cluster node:

[root@node1 ~]# pcs constraint location web_rg prefers node1

9.9. Display all configured cluster resource constraints on any cluster node:

[root@node1 ~]# pcs constraint show

9.10. Display all configured cluster resources on any cluster node:

[root@node1 ~]# pcs resource show

9.11. Display details about a configured cluster resource on any cluster node:

[root@node1 ~]# pcs resource show webserver

10.1. Remove the "webserver" resource from the "web_rg" resource group on any cluster node:

[root@node1 ~]# pcs resource group remove web_rg webserver

10.2. Remove the "web_rg" resource group including all resources that it contains on any cluster node:

[root@node1 ~]# pcs resource delete web_rg