Linux – A LVS configuration on CentOS based on Tunnel IP Setup with Single Ethernet.

>***A LVS configuration on CentOS based on Tunnel IP Setup with Single Ethernet.***
#Installation

The following RPM needs to be installed to avoid any dependency

perl-Net-IMAP-Simple-SSL-1.3-1.el5.centos.noarch.rpm
perl-Crypt-SSLeay-debuginfo-0.51-11.i386.rpm
heartbeat-stonith-2.1.3-3.el5.centos.i386.rpm
heartbeat-2.1.3-3.el5.centos.i386.rpm
perl-Net-IMAP-Simple-1.17-1.el5.centos.noarch.rpm
perl-MailTools-1.77-1.el5.centos.noarch.rpm
perl-LDAP-0.33-3.noarch.rpm
perl-Authen-Radius-0.13-1.el5.centos.noarch.rpm
perl-TimeDate-1.16-1.el5.centos.noarch.rpm
perl-Mail-POP3Client-2.17-1.el5.centos.noarch.rpm
perl-Data-HexDump-0.02-1.el5.centos.noarch.rpm
perl-Crypt-SSLeay-0.51-11.i386.rpm
perl-Convert-ASN1-0.20-1.1.noarch.rpm
heartbeat-pils-2.1.3-3.el5.centos.i386.rpm
heartbeat-ldirectord-2.1.3-3.el5.centos.i386.rpm

heartbeat/ldirectord installation
/******************************************************************************************************************/
[root@vmlinux3 LVS]# rpm -ivh heartbeat-pils-2.1.3-3.el5.centos.i386.rpm
warning: heartbeat-pils-2.1.3-3.el5.centos.i386.rpm: Header V3 DSA signature: NOKEY, key ID e8562897
Preparing… ########################################### [100%]
1:heartbeat-pils ########################################### [100%]
[root@vmlinux3 LVS]# rpm -ivh heartbeat-stonith-2.1.3-3.el5.centos.i386.rpm
warning: heartbeat-stonith-2.1.3-3.el5.centos.i386.rpm: Header V3 DSA signature: NOKEY, key ID e8562897
Preparing… ########################################### [100%]
1:heartbeat-stonith ########################################### [100%]
[root@vmlinux3 LVS]# rpm -ivh heartbeat-2.1.3-3.el5.centos.i386.rpm
warning: heartbeat-2.1.3-3.el5.centos.i386.rpm: Header V3 DSA signature: NOKEY, key ID e8562897
Preparing… ########################################### [100%]
1:heartbeat ########################################### [100%]
[root@vmlinux3 LVS]# rpm -ivh perl-Data-HexDump-0.02-1.el5.centos.noarch.rpm
warning: perl-Data-HexDump-0.02-1.el5.centos.noarch.rpm: Header V3 DSA signature: NOKEY, key ID e8562897
Preparing… ########################################### [100%]
1:perl-Data-HexDump ########################################### [100%]
[root@vmlinux3 LVS]# rpm -ivh perl-Authen-Radius-0.13-1.el5.centos.noarch.rpm
warning: perl-Authen-Radius-0.13-1.el5.centos.noarch.rpm: Header V3 DSA signature: NOKEY, key ID e8562897
Preparing… ########################################### [100%]
1:perl-Authen-Radius ########################################### [100%]
[root@vmlinux3 LVS]# rpm -ivh perl-TimeDate-1.16-1.el5.centos.noarch.rpm
warning: perl-TimeDate-1.16-1.el5.centos.noarch.rpm: Header V3 DSA signature: NOKEY, key ID e8562897
Preparing… ########################################### [100%]
1:perl-TimeDate ########################################### [100%]
[root@vmlinux3 LVS]# rpm -ivh perl-MailTools-1.77-1.el5.centos.noarch.rpm
warning: perl-MailTools-1.77-1.el5.centos.noarch.rpm: Header V3 DSA signature: NOKEY, key ID e8562897
Preparing… ########################################### [100%]
1:perl-MailTools ########################################### [100%]
[root@vmlinux3 LVS]# rpm -ivh perl-Net-IMAP-Simple-1.17-1.el5.centos.noarch.rpm
warning: perl-Net-IMAP-Simple-1.17-1.el5.centos.noarch.rpm: Header V3 DSA signature: NOKEY, key ID e8562897
Preparing… ########################################### [100%]
1:perl-Net-IMAP-Simple ########################################### [100%]
[root@vmlinux3 LVS]# rpm -ivh perl-Net-IMAP-Simple-SSL-1.3-1.el5.centos.noarch.rpm
warning: perl-Net-IMAP-Simple-SSL-1.3-1.el5.centos.noarch.rpm: Header V3 DSA signature: NOKEY, key ID e8562897
Preparing… ########################################### [100%]
1:perl-Net-IMAP-Simple-SS########################################### [100%]
[root@vmlinux3 LVS]# rpm -ivh perl-Mail-POP3Client-2.17-1.el5.centos.noarch.rpm
warning: perl-Mail-POP3Client-2.17-1.el5.centos.noarch.rpm: Header V3 DSA signature: NOKEY, key ID e8562897
Preparing… ########################################### [100%]
1:perl-Mail-POP3Client ########################################### [100%]
[root@vmlinux3 LVS]# rpm -ivh heartbeat-ldirectord-2.1.3-3.el5.centos.i386.rpm
warning: heartbeat-ldirectord-2.1.3-3.el5.centos.i386.rpm: Header V3 DSA signature: NOKEY, key ID e8562897
Preparing… ########################################### [100%]
1:heartbeat-ldirectord ########################################### [100%]

#********************************************************************************************************************
# Configuration

Copy the following files inside /etc/ha.d for heartbeat and ldirectord if they are not created

root@vmlinux3 LVS]# cp /usr/share/doc/heartbeat-2.1.3/authkeys /etc/ha.d/
[root@vmlinux3 LVS]# cp /usr/share/doc/heartbeat-2.1.3/ha.cf /etc/ha.d/
[root@vmlinux3 LVS]# cp /usr/share/doc/heartbeat-2.1.3/haresources /etc/ha.d/
[root@vmlinux3 LVS]# cp /usr/share/doc/heartbeat-2.1.3/ldir /etc/ha.d/
[root@vmlinux3 LVS]# cp /usr/share/doc/heartbeat-ldirectord-2.1.3/ldirectord.cf /etc/ha.d/

Issue ‘ls -tlr’ to check if all the files are placed properly inside /etc/ha.d

[root@vmlinux3 ha.d]# ls -tlr
total 56
-rw-r–r– 1 root root 7184 Feb 6 2008 shellfuncs
-rw-r–r– 1 root root 692 Feb 6 2008 README.config
-rwxr-xr-x 1 root root 745 Feb 6 2008 harc
drwxr-xr-x 2 root root 4096 Jan 27 06:36 rc.d
drwxr-xr-x 2 root root 4096 Jan 27 06:47 resource.d
-rw-r–r– 1 root root 645 Jan 27 06:50 authkeys
-rw-r–r– 1 root root 10843 Jan 27 06:50 ha.cf
-rw-r–r– 1 root root 5905 Jan 27 06:50 haresources
-rw-r–r– 1 root root 7683 Jan 27 06:50 ldirectord.cf
[root@vmlinux3 ha.d]#

#****************************************************************************************************************/
Hearbeat service need to be add on level 3/5 and it should be come up automatically after system reboot.

[root@localhost LVS]# chkconfig –list | grep -i “heartbeat”

If not there, adding the service
[root@localhost LVS]# chkconfig –add heartbeat
[root@localhost LVS]# chkconfig –level 35 heartbeat on

Note: ldirectord service should be off.

[root@localhost LVS]# chkconfig –level 1235 ldirectord off
[root@localhost LVS]# chkconfig –list | egrep -i “heartbeat|ldirectord”
heartbeat 0:off 1:off 2:off 3:on 4:off 5:on 6:off
ldirectord 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@localhost LVS]#

# Setting up the tunnel IP and Kernel parameter to avoid ARP problem in /etc/rc.local file
[root@vmlinux4 ha.d]# cat /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don’t
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local
##############################################################################################
#Edited by Satyendra
#Setting up Tunnel IP (VIP)
/sbin/ifconfig tunl0 192.168.1.5 netmask 255.255.255.0 broadcast 192.168.1.255
#Setting up Kernel tunning parameter

sysctl -w net.ipv4.conf.ip_forward=1
sysctl -w net.ipv4.conf.all.rp_filter=0
sysctl -w net.ipv4.conf.all.arp_announce=2
sysctl -w net.ipv4.conf.all.arp_filter=1
sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.tunl0.rp_filter=0
sysctl -w net.ipv4.conf.tunl0.arp_announce=2
sysctl -w net.ipv4.conf.tunl0.arp_filter=0
sysctl -w net.ipv4.conf.tunl0.arp_ignore=3
############################################################################################

These are the following parameter which needs to be enabled for configuring heartbeat and ldirectord.

#/etc/ha.d/ha.cf file configuration

[root@vmlinux4 ha.d]# grep -v “#” ha.cf
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
warntime 10
initdead 60
udpport 694
bcast eth0 # Linux
auto_failback on
ping 192.168.1.2
respawn hacluster /usr/lib/heartbeat/ipfail
node vmlinux3.vmgraduate.com
node vmlinux4.vmgraduate.com
Note:
(1) auto_failback on has compatiblity with nice_failback on feature
(2) As auto_failback is on, the currently active linux director will now act as the master and when the failed linux director comes back online it will act as a standby.
(3) The ‘ipfail’ for heartbeat makes this possible by monitoring one or more external hosts
known as a ping node. Typically this would be a router or the switch itself. That is, if a host cannot access a ping node, it is not eligible
to hold any resources. Thus, if an interface fails on the active linux director, then one of the
ping nodes should become unavailable and fail-over will occur.
For ipfailover the following parameter needs to be configured on ha.cf file
ping 192.168.1.2
respawn hacluster /usr/lib/heartbeat/ipfail
192.168.1.2 can be gateway IP.

#/etc/ha.d/haresources file configuration

[root@vmlinux4 ha.d]# grep -v “#” haresources
vmlinux3.vmgraduate.com 192.168.1.5/32/eth0/192.168.1.5 ldirectord

Note:
(1)This file specifies the services for the cluster. Note that this file must be the
same at both nodes! You need the following line in your haresources file to heartbeat the
specified services, using the VIP (Virtual IP – Tunnel IP where DNS A record is configured) 192.168.1.5 and Master node name vmlinux3.vmgraduate.com as
an example,
vmlinux3.vmgraduate.com 192.168.1.5/32/eth0/192.168.1.5 ldirectord
This line tells heartbeat to start the Linux Director Daemon ldirectord, which in turn
will read the file /etc/ha.d/conf/ldirectord.cf to start and monitor the Linux Virtual
Server cluster.

#/etc/ha.d/authkey file configuration

[root@vmlinux4 ha.d]# grep -v “#” authkeys
auth 1
1 crc

Note:
(1) Here we are CRC based authentication
(2) There must be 600 file permission for this file.

#/etc/ha.d/ldirectord.cf file configuration

[root@vmlinux4 ha.d]# grep -v “#” ldirectord.cf
checktimeout=2
checkinterval=4
autoreload=yes
quiescent=yes
negotiatetimeout=3

virtual = 192.168.1.5:80
real = 192.168.1.3:80 ipip 40
real = 192.168.1.4:80 ipip 50
service = http
checktype = negotiate
request = “uptime.html”
receive = “LOAD”
protocol = tcp
scheduler = wlc
[root@vmlinux4 ha.d]#

Note:
(1) ipip is IP encapsulation method for LVS. you can use gate, masq methods as well
(2) This ldirector server (192.168.1.3) is also serving apache request that’s why we have decided weight 40 for this host and for other RIP (Real) server this is 50.
(3) uptime.html file should be placed on all the RIP servers Document root location i.e /var/www/html with LOAD word.

Services for Heartbeat/Ldirectord
/etc/init.d/heartbeat {start|stop|restart|try-restart|status|reload|force-reload}
/etc/init.d/ldirectord {start|stop|restart|try-restart|status|reload|force-reload}

Note: It should keep in mind that,Ldirectord service should NOT be automatically started. Heartbeat must be running on all the master/slave server to take care of ldirectord daemon.

Troubleshooting –
1) check out if ldirectord service is up
[root@vmlinux4 ha.d]# ps -aef | grep -i ldirectord | grep -v grep
root 6611 1 0 17:15 ? 00:00:01 /usr/bin/perl -w /etc/ha.d/resource.d/ldirectord start
Note: You can get the above output only on that server where ldirectord daemon would be running.

2) you can use the following command as well.
[root@vmlinux4 ha.d]# /etc/init.d/ldirectord status
ldirectord for /etc/ha.d/ldirectord.cf is running with pid: 6611
3) you can check the LVS status with /sbin/ipvsadm command. you can get the similer output like this.

[root@vmlinux4 ha.d]# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.1.5:http wlc
-> vmlinux4.vmgraduate.com:http Local 20 0 0
-> vmlinux3.vmgraduate.com:http Tunnel 0 0 0
[root@vmlinux4 ha.d]#

Note: You can follow the same practics for Slave LVS server except one change (Desireable)

1)Since this document is based on LVS server + Apache request handler so you can set ipip value less i.e 40 on ldirector.cf file
real = 192.168.1.4:80 ipip 40

You can start heartbeat in the slave node. Execute ipvsadm. You should not see that cluster.
Now shutdown or stop the heartbeat of the Master machine. Wait one minute. Check the
slave machine with ipvsadm. If things have been setup properly, the cluster should
have shifted to the backup machine

Note: If we need to setup public/private ethernet based LVS then we should setup Gateway seperately for Public ethernet and tunnel IP.

Advertisements

3 thoughts on “Linux – A LVS configuration on CentOS based on Tunnel IP Setup with Single Ethernet.

  1. Keiffer November 17, 2011 at 2:30 pm Reply

    I’m not quite sure how to say this; you made it exrtemely easy for me!

    • Marlee December 1, 2011 at 4:14 pm Reply

      Whoever edits and pblushies these articles really knows what they’re doing.

    • Jailyn December 3, 2011 at 6:07 am Reply

      I cannot tell a lie, that really heepld.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: