While building high-availability Linux computing clusters with Corosync, Pacemaker and OpenSUSE I’ve assembled a collection of useful notes. I’ll share them here, and add to them as time permits.
Terms and abbreviations
Corosync allows any number of servers to be part of the cluster using any number of fault-tolerant configurations (active/passive, active/active, N+1, etc.)
Corosync provides messaging between servers within the same cluster.
Pacemaker manages the resources and applications on a node within the cluster.
OpenAIS can be thought of as the API between Corosync and Pacemaker, as well as between Corosync and other plug in components.
The ClusterLabs FAQ explains it this way: “Originally Corosync and OpenAIS were the same thing. Then they split into two parts… the core messaging and membership capabilities are now called Corosync, and OpenAIS retained the layer containing the implementation of the AIS standard. Pacemaker itself only needs the Corosync piece in order to function, however some of the applications it can manage (such as OCFS2 and GFS2) require the OpenAIS layer as well.”
“DC” stands for “Designated Co-ordinator”, the node with final say on the cluster’s current state.
“dlm” is the “distributed lock manager” used for lock-update-unlock data updates across the cluster.
Pacemaker Daemons (managed by Corosync):
- lrmd – local resource manager daemon
- crmd – cluster resource manager daemon
- cib – cluster information base (database of cluster information)
- pengine – policy engine
- stonithd – “shoot the other node in the head” daemon
- attrd – I would guess “attribute daemon”, but I haven’t found a formal definition yet of what this does. Need to check the source code and see what it says
- mgmtd – management daemon. I haven’t found a formal definition yet of what this does. Need to check the source code and see what it says
To start Corosync on an OpenSUSE server:
# rcopenais start
To stop Corosync on an OpenSUSE server:
# rcopenais stop
To get the current cluster status:
# crm status
Text-based cluster monitoring tool:
To list all of the resources in a cluster:
# crm resource status
To view the current pacemaker config:
# crm configure show # crm configure show xml
To verify the current pacemaker config:
# crm_verify -L -V
Force the ClusterIP resource (and all related resources based on your policy settings) to move from server0 to server1:
# crm resource move ClusterIP server1
Note that to transfer the resources this actually sets the preferred node to server1. To remove the preference and return control to the cluster:
# crm resource unmove ClusterIP
Making a new shadow (backup) copy of a working CIB called “working”:
# crm configure crm(live)configure# cib new working INFO: working shadow CIB created
Updating the “working” backup copy of the live CIB:
# crm configure crm(live)configure# cib reset working INFO: copied live CIB to working
Setting the “working” CIB to act as the “live” configuration:
# crm configure crm(live)configure# cib commit working INFO: commited 'working' shadow CIB to the cluster
Saving the configuration to a file, modifying the file, and loading just the updates:
# crm configure show > /tmp/crm.xml # vi /tmp/crm.xml # crm configure load update /tmp/crm.xml
To get a list of all of the OCF resource agents provided by Pacemaker and Heartbeat:
# crm ra list ocf heartbeat # crm ra list ocf pacemaker
To see which host a resource is running on:
# crm resource status ClusterIP resource ClusterIP is running on: server0
To clear the fail count on a WebSite resource:
# crm resource cleanup WebSite
On OpenSUSE servers OCF resource definitions can be found in /usr/lib/ocf/resource.d/.
Pacemaker resources are configured using the crm tool.
To disable STONITH on a two-node cluster:
# crm configure property stonith-enabled=false # crm_verify -L -V
Adding a second IP address resource to the cluster:
# crm configure primitive ClusterIP ocf:heartbeat:IPaddr2 \ params ip="192.168.1.97" cidr_netmask="24" \ op monitor interval="30" timeout="25" start-delay="0"
To turn the quorum policy OFF for two-node clusters:
# crm configure property no-quorum-policy=ignore # crm configure show
Adding an Apache resource with https enabled:
# crm configure primitive WebSite ocf:heartbeat:apache \ params configfile="/etc/apache2/httpd.conf" options="-DSSL" \ operations $id="WebSite-operations" \ op start interval="0" timeout="40" \ op stop interval="0" timeout="60" \ op monitor interval="60" timeout="120" start-delay="0" \ statusurl="http://127.0.0.1/server-status/" \ meta target-role="Started"
Note that the Apache mod_status module must be loaded and configured so that http://127.0.0.1/server-status/ will return the server status page, otherwise Pacemaker cannot tell that the server is actually running and it will attempt to fail over to the other node, then fail again.
Adding a DRBD resource (based on http://www.drbd.org/users-guide-emb/s-pacemaker-crm-drbd-backed-service.html):
# crm configure primitive VolumeDRBD ocf:linbit:drbd \ params drbd_resource="drbd_resource_name" \ operations $id="VolumeDRBD-operations" \ op start interval="0" timeout="240" \ op promote interval="0" timeout="90" \ op demote interval="0" timeout="90" \ op stop interval="0" timeout="100" \ op monitor interval="40" timeout="60" start-delay="0" \ op notify interval="0" timeout="90" \ meta target-role="started" # crm configure primitive FileSystemDRBD ocf:heartbeat:Filesystem \ params device="/dev/drbd0" directory="/srv/src" fstype="ext3" \ operations $id="FileSystemDRBD-operations" \ op start interval="0" timeout="60" \ op stop interval="0" timeout="60" fast_stop="no" \ op monitor interval="40" timeout="60" start-delay="0" \ op notify interval="0" timeout="60" # crm configure ms MasterDRBD VolumeDRBD \ meta clone-max="2" notify="true" target-role="started"
Note that the resource name drbd_resource="drbd_resource_name" must match the name listed in /etc/drbd.conf and that drbd should NOT be started by init — you want to let Pacemaker control starting and stopping drbd. To make sure DRBD isn’t being started by init use:
# chkconfig drbd off
Telling Pacemaker that the floating IP address, DRBD primary, and Apache all have to be on the same node:
# crm configure group Cluster ClusterIP FileSystemDRBD WebSite \ meta target-role="Started" # crm configure colocation WebServerWithIP inf: Cluster MasterDRBD:Master # crm configureorder StartFileSystemFirst inf: MasterDRBD:promote Cluster:startFollow earlruby