Cluster Aware AIX

in AIX

Cluster Aware AIX (CAA)

The Cluster Aware function of AIX was introduced with AIX 7.1 and AIX 6.1 TL6. This new technology builds clustering technologies into the AIX base operating system. Cluster Aware provides the AIX kernel with heartbeating, health management and monitoring capabilities.

Using Cluster Aware AIX you can easily create a cluster of AIX nodes with the following capabilities:

  • Clusterwide event management - The AIX Event Infrastructure allows event propagation across the cluster so that applications can monitor events from any node in the cluster.
    • Communication and storage events
      • Node UP and node DOWN
      • Network adapter UP and DOWN
      • Network address change
      • Point-of-contact UP and DOWN
      • Disk UP and DOWN
    • Predefined and user-defined events
  • Clusterwide storage naming service - When a cluster is defined or modified, the AIX interfaces automatically create a consistent shared device view across the cluster. A global device name, such as cldisk5, would refer to the same physical disk from any node in the cluster.
  • Clusterwide command distribution - The clcmd command provides a facility to distribute a command to a set of nodes that are members of a cluster. For example, the command clcmd date returns the output of the date command from each of the nodes in the cluster.
  • Clusterwide communication making use of networking and storage communication

Cluster Aware AIX is not a high availability solution taken alone and does not replace existing high availability products. It can be seen as a set of commands and services that the cluster software can exploit to provide high availability and disaster recovery support to external applications. CAA  does not provide the application monitoring and resource failover capabilities that PowerHA  provides. In fact, IBM PowerHA SystemMirror 7.1 and even (RSCT) Reliable Scalable Cluster Technology use the built-in AIX clustering capabilities. The reason for the introduction of cluster built-in functions of AIX was to simplify the configuration and management of high availability clusters. It also lays a foundation for future AIX capabilities and the next generation of PowerHA SystemMirror.

Note: Cluster Aware AIX capability is included in AIX 7.1 Standard or Enterprise Editions, but is not included in AIX 7.1 Express Edition.

Creating the cluster

Before creating the cluster there are some things to consider.

CAA uses IP based network communications and storage interface communication through Fibre Channel and SAS adapters. When using both type of communication, all nodes in the cluster can always communicate with any other nodes in the cluster configuration and thus eliminating "split brain" incidents.

Network

A multicast address is used for cluster communications between the nodes in the cluster. Therefore, you need to ensure proper network configuration on each node. Each node must have at least one IP address configured on its network interface. The IP address is used as a basis for creating an IP multicast address, which the cluster uses for internal communications. Check also if entries of the IP addresses exist in every node’s /etc/hosts file.

Storage

Each node of the cluster should have common storage devices available, either SAN or SAS disks. These storage devices are used for the cluster repository disk and for any clustered shared disks. If Fibre Channel devices will be used, the following procedure must be followed before creating the cluster (SAS adapters do not require special setup):

  1. Run the following command:
rmdev -Rl fcsX

Note: X is the number of your adapter. If you booted from the Fibre Channel adapter, you do not need to complete this step.

  1. Run the following command:
chdev -l fcsX -a tme=yes 

Note: If you booted from the Fibre Channel adapter, add the -P flag. The target mode enabled (TME) attribute is needed in order FC adapter to be supported.

  1. Run the following command:
chdev -l fscsiX -a dyntrk=yes -a fc_err_recov=fast_fail
  1. Run the cfgmgr command.

Note: If you booted from the Fibre Channel adapter and used the -P flag, you must reboot.

  1. Verify the configuration changes by running the following command:
lsdev -C | grep sfwcom

The following is an example of the output displayed from the lsdev -C | grep sfwcom command:

lsdev -C | grep sfwcom
sfwcomm0       Available 01-00-02-FF Fiber Channel Storage Framework Comm
sfwcomm1       Available 01-01-02-FF Fiber Channel Storage Framework Comm

The above procedure has to be performed on all nodes of the cluster.

The cluster repository disk is used as the central repository for the cluster configuration data. A disk size of 10 GB is recommended. The minimum is 1 GB.

 

The following commands can be used for creating and managing the cluster:

lscluster       Used to list cluster configuration information.

mkcluster       Used to create a cluster.

chcluster       Used to change a cluster configuration.

rmcluster       Used to remove a cluster configuration.

clcmd           Used to distribute a command to a set of nodes that are members of a cluster.

See Man pages for different options.

 

Create the cluster:

Create the cluster with mkcluster command:

# mkcluster -n mycluster -m nodeA,nodeB -r hdisk1 -d hdisk2,hdisk3

mkcluster: Cluster shared disks are automatically renamed to names such as

        cldisk1, [cldisk2, ...] on all cluster nodes.  However, this cannot

        take place while a disk is busy or on a node which is down or not

        reachable.  If any disks cannot be renamed now, they will be renamed

        later by the clconfd daemon, when the node is available and the disks

        are not busy.

The name of the cluster is mycluster, the nodes are nodeA and nodeB, the repository disk is hdisk1 and the shared disks are hdisk2 and hdisk3. Note that repository disk and shared disks will be automatically renamed as caa_private0 and cldisk1 and cldisk2 respectively. These names will be the same over both nodes, no matter what was their initial hdisk number (which could be different on each node).

Before mkcluster command:

# lspv

hdisk0        0050187a43833dc5             rootvg        active

hdisk1        0050187a8de3af7d             None          active

hdisk2        0050187a70d8c6cf             None

hdisk3        none                         None

After mkcluster command:

# lspv

hdisk0              0050187a43833dc5            rootvg              active

caa_private0        0050187a8de3af7d            caavg_private       active

cldisk1             0050187a70d8c6cf            None

cldisk2             none                        None

When the cluster is ready a special volume group (caavg_private), new logical volumes and filesystems are created.

When you create a cluster with mkcluster command the following actions are performed (taken from http://pic.dhe.ibm.com/infocenter/aix/v7r1/index.jsp?topic=%2Fcom.ibm.aix.clusteraware%2Fclaware_architecture.htm):

  • The cluster is created using the mkcluster command.
  • The cluster configuration is written to the raw section of the cluster repository disk.
  • Special volume groups and logical volumes are created on the cluster repository disk.
  • Cluster file systems are created on the special volume group.
  • Cluster services are made available to other functions in the operating system, such as Reliable Scalable Cluster Technology (RSCT) and PowerHA® SystemMirror.
  • Storage framework register lists are created on the cluster repository disk.
  • A global device namespace is created and interaction with LVM starts for handling associated volume group events.
  • A clusterwide multicast address is established.
  • The node discovers all of the available communication interfaces.
  • The cluster interface monitoring starts.
  • The cluster interacts with Autonomic Health Advisory File System (AHAFS) for clusterwide event distribution.
  • The cluster exports cluster messaging and cluster socket services to other functions in the operating system, such as Reliable Scalable Cluster Technology (RSCT) and PowerHA SystemMirror.

 

To check the status of the cluster use lscluster with the following options:

-c     Lists the cluster configuration.

-d     Lists the cluster storage interfaces.

-i     Lists the cluster configuration interfaces on the local node.

-m     Lists the cluster node configuration information.

 

To check if the cluster is operating properly execute any clusterwide command:

#clcmd date

-------------------------------

NODE nodeA

-------------------------------

Wed Jun  6 11:19:44 EEST 2012

-------------------------------

NODE nodeB

-------------------------------

Wed Jun  6 11:19:44 EEST 2012

 

To remove the cluster just type:

# rmcluster -n mycluster

 rmcluster: Removed cluster shared disks are automatically renamed to names such

        as hdisk10, [hdisk11, ...] on all cluster nodes.  However, this cannot

        take place while a disk is busy or on a node which is down or not

        reachable.  If any disks cannot be renamed now, you must manually

        rename them by removing them from the ODM database and then running

        the cfgmgr command to recreate them with default names.  For example:

        rmdev -l cldisk1 -d

        rmdev -l cldisk2 -d

        cfgmgr

Cluster Aware AIX helps to create very easily cluster with minimum set of commands and user intervention. In our opinion, one of the best features it provides is the common disk names used in all the participating nodes in the cluster.


tags: CAA, cluster, mkcluster


Comments:

Deane

2013-09-22 18:51:55
How to determine what the Multicast address shoudl be: Should it be the floating address ? How to determine if multicat is working while syning I get the following error: ERROR: Multicast communication with node utaixha6 using mping failed. Errors encountered verifying communications between nodes.

Leave a Comment: