Friday, June 28, 2013

Compiling Latest Linux Kernel

Hi Friends,

As per requirement of our client, recently I compiled latest stable Linux kernel (linux-3.9.7) on my CentOS-6.3.
So as usual I wanna share steps for compiling the kernel :)

Below are the steps. Please follow if you also want to do the same.

Step 1) Download latest stable Linux Kernel from http://kernel.org/ and untar the package.

[root@localhost ~]# cd /usr/src/
[root@localhost src]# wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.9.7.tar.xz
[root@localhost src]#tar -xvf linux-3.9.7.tar.xz


step 2) Choose appropriate Module 

[root@localhost src]# cd linux-3.9.7
[root@localhost linux-3.9.7 ]# make menuconfig

Note: This will show kernel configuration screen . Navigate and deselect things (drivers) you do not need. In case of doubt just leave the initial option/selection. Save the configuration and exit the menu. 
After saving your configuration (it is stored in the ".config" file) exit from screen.

Step 3) compile the kernel

[root@localhost linux-3.9.7 ]#make

Note: This is time consuming task and will take more then hr's depending on your hardware.  

Step 4) copy all the modules to a new directory, "/lib/modules/3.9.7"

[root@localhost linux-3.9.7 ]#make modules_install

Step 5) copy compiled files into appropriate locations

[root@localhost linux-3.9.7 ]#make install

Step 6) Remove all generated files + config + various backup files 

[root@localhost linux-3.9.7 ]#make mrproper


That's all about Kernel compiling. And Please don't forget to edit grub.conf to change default boot option with new kernel.

Enjoy !!!!!!!

Thursday, June 27, 2013

SSH Port Forwarding

Hi Friends,

Let’s understand SSH Port forwarding concept with below examples.
I have two machine station-y (172.24.40.110) and station-z(172.24.40.120)

Example 1)

[root@station-z ~]# ssh -L 8080:yahoo.com:80 -N root@172.24.40.110
(I, SSH Client will listen for traffic on my host's port 8080 and send it to you, SSHD on 172.24.40.110. 

You will decrypt it and forward that traffic to port 80 on yahoo.com)
So after executing above command on station-z, when station-z tries to access http://localhost it will redirect to yahoo.com

Example 2)
[root@station-z ~]# ssh -R 8080:127.0.0.1:25 -N 172.24.40.110

(You the SSHD on 172.24.40.110 will listen for the traffic on your port 8080 and send it to me. I, ssh client will decrypt it and forward that traffic to port 25 on me as if it came from me.)


So after executing above command on station-z, when station-y tries to access their local port 8080 ([root@station-y ~]# telnet 0 8080)  it will redirect to port 25 of station-z 

Wednesday, June 26, 2013

Firewall Port Redirection through Iptables

Friends,

Recently I implemented firewall rule for incoming traffic on port 80 on firewall (10.10.0.100) will redirect to port 8080 of internal webserver(172.24.40.110).

Below is the command that I executed on my Linux based Firewall


[root@station-z ]#iptables -t nat -I PREROUTING-p tcp --dport 80 -d 10.10.0.100 -j DNAT --to 172.24.40.110:8080

IPTABLES NAT Rule Implementation

Friend,

Recently my network Admin has blocked Facebook access to all our Desktop-PC.

But fortunately I have access of one of Linux system where then haven’t implemented firewall rule.

So I MASQUERADE my outgoing traffic from my PC to that Linux System to get Facebook Access. J

It’s Very easy & if you want you could also test this on your Lab.

Example: - My PC (eth0: 172.24.40.110)

Linux PC [eth0: 172.24.40.120 (Lan Interface) eth1: 10.10.20.1 (Wan Interface)]

Step 1) On My PC (eth0: 172.24.40.110) set “Default Gatway to Linux machine(eth0: 172.24.40.120)

Step 2) On Linux machine set IP Forwarding

[root@station-y ]#vi /etc/sysctl.conf
net.ipv4.ip_forward = 1

Step 3) Apply NAT rule to MASQUERADE all traffic to eth1 (Wan Interface)


[root@station-y ]# iptables -t nat -I POSTROUTING -s 172.24.40.0/24 -o eth1 -j MASQUERADE

Friday, June 21, 2013

Installing Zabbix with postgresql in CentOS6

Friend , recently I configured Zabbix with postgresql in my office infrastructure.
We have one zabbix master server (192.168.0.1) and one pgsql database server (192.168.0.2)

Below are the step that I performs on configuring zabbix.

Step 1) Download and install RPM’s for postgresql in database server (192.168.0.2)

[root@localhost ~]#rpm -ivh pgdg-redhat91-9.1-5.noarch.rpm
[root@localhost ~]#yum install postgresql91-server postgresql91-contrib
[root@localhost ~]#/etc/init.d/postgresql-9.1 restart
[root@localhost ~]#service postgresql-9.1 initdb

Step 2) Configure Postgresql on (192.168.0.2)
[root@localhost ~]# su – postgres
-bash-4.1$ psql
postgres=# CREATE DATABASE zabbix WITH ENCODING='UTF-8';
postgres=# CREATE USER zabbix WITH PASSWORD 'zabbix';
postgres=# GRANT ALL ON DATABASE zabbix TO zabbix;
postgres=# GRANT ALL PRIVILEGES ON DATABASE zabbix to postgres;
postgres=# ALTER USER Postgres WITH PASSWORD 'redhat';
postgres=# \q
-bash-4.1$
-bash-4.1$ logout
[root@localhost ~]#vi /var/lib/pgsql/9.1/data/pg_hba.conf

****************
# "local" is for Unix domain socket connections only
local   all             all                                     md5
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5
host    all             all             10.10.2.0/16            md5
******************

[root@localhost ~]#vi /var/lib/pgsql/9.1/data/ postgresql.conf

**************
listen_addresses = '192.168.0.2'         # what IP address(es) to listen on;
                                        # comma-separated list of addresses;
                                        # defaults to 'localhost', '*' = all
                                        # (change requires restart)
port = 5432    
**************
[root@localhost ~]#/etc/init.d/postgresql-9.1 restart

Step 3) Configure php.ini in Zabbix server (192.168.0.1)

[root@localhost ~]#yum -y install gcc net-snmp-devel curl-devel gcc make curl-devel httpd php php-pgsql php-bcmath php-gd php-mbstring net-snmp-devel fping e2fsprogs-devel zlib-devel libgssapi-devel krb5-devel openssl-devel libidn-devel iksemel-devel libXpm openssh-clients  php-xmlrpc sharutils OpenIPMI-devel libssh2-devel php-xml

[root@localhost ~]#vi /etc/php.ini

*******
max_execution_time = 300
memory_limit = 128M
post_max_size = 16M
upload_max_filesize = 2M
max_input_time = 300
date.timezone = Asia/Kolkata
*******

Step 4) Download and install zabbix from source code for zabbix frontend (192.168.0.1)
[root@localhost ~]#groupadd zabbix
[root@localhost ~]#useradd -g zabbix –s /sbin/nologin zabbix
[root@localhost ~]#wget http://sourceforge.net/projects/zabbix/files/ZABBIX%20Latest%20Stable/2.0.6/zabbix-2.0.6.tar.gz                                                                                                                                                       
[root@localhost ~]#mv zabbix-2.0.6.tar.gz /opt/;cd /opt
[root@localhost ~]#tar -xvzf /opt/zabbix-2.0.6.tar.gz
[root@localhost ]#cd zabbix-2.0.0.6
[root@localhost ]#./configure --prefix=/opt/zabbix --enable-server --enable-agent --with-net-snmp --with-postgresql=/usr/pgsql-9.1/bin/pg_config --with-ssh2 --with-openipmi --with-libcurl
[root@localhost ]#make;make install

Step 5) Configure frontend
[root@localhost ~]#cp -R /opt/zabbix-2.0.6/frontends/php /var/www/html/zabbix
[root@localhost ~]#Service httpd restart

Step 5) Configure zabbix config files (192.168.0.1)
[root@localhost ]#vi /opt/zabbix/etc/zabbix_server.conf

*********
ListenPort=10051
LogFile=/opt/zabbix/log/zabbix_server.log
 PidFile=/opt/zabbix/log/zabbix_server.pid
 DBHost=192.168.0.2
DBName=zabbix
DBUser=zabbix
 DBPassword=zabbix
 StartPollers=10
 DisableHousekeeping=1
 CacheSize=128M
 HistoryCacheSize=128M
 TrendCacheSize=64M
 HistoryTextCacheSize=128M
 Timeout=30
***********

[root@localhost ]#vi /opt/zabbix/etc/zabbix_agentd.conf

************
LogFile=/tmp/zabbix_agentd.log
Server=127.0.0.1
ServerActive=127.0.0.1
Hostname=Zabbix server
************
Step 6) Import zabbix database to postgresql server

[root@localhost ]#cd /opt/zabbix-2.0.6/database/postgresql
 [root@localhost ]#   psql -U zabbix -h 192.168.0.2 -d zabbix -f schema.sql
  [root@localhost ]#psql -U zabbix -h 192.168.0.2 -d zabbix -f images.sql
  [root@localhost ]#psql -U zabbix -h 192.168.0.2 -d zabbix -f data.sql

Step 7) Installing Zabbix through web
Go to web browser and type below link




Download the config file and paste it to the location /var/www/html/zabbix/conf/zabbix.conf.php

Step 7) Copy init script for zabbix_server and zabbix_agentd

[root@localhost ]#cp /root/zabbix.2.0.6/misc/init.d/fedora/core/zabbix_server /etc/init.d/
[root@localhost ]#cp /root/zabbix.2.0.6/misc/init.d/fedora/core/zabbix_agentd /etc/init.d/
[root@localhost ]#vi /etc/init.d/ zabbix_server

**************
# Zabbix-Directory
        BASEDIR=/opt/zabbix

        # Binary File
        BINARY_NAME=zabbix_server
        # Full Binary File Call
        FULLPATH=$BASEDIR/sbin/$BINARY_NAME

        # PID file
        PIDFILE=/opt/zabbix/log/$BINARY_NAME.pid
******************

[root@localhost ]#vi /etc/init.d/ zabbix_agentd

***************** 
        # Zabbix-Directory
        BASEDIR=/opt/zabbix

        # Binary File
        BINARY_NAME=zabbix_agentd

        # Full Binary File Call
        FULLPATH=$BASEDIR/sbin/$BINARY_NAME

        # PID file
        PIDFILE=/tmp/$BINARY_NAME.pid
************
[root@localhost ]# service zabbix_server restart
[root@localhost ]# service zabbix_agentd restart

Now You have done your installation part. Try to login with credentials 

URL: http://192.168.0.1/zabbix
User: Admin
Password: zabbix

Wednesday, June 12, 2013

sshpass, A great tool for non-interactive ssh login

Friend,

Recently I needed to perform some maintenance activity on some of my Linux machines.

I got a list of 100 Linux machine.

Problem was that, We haven't configured password-less SSH but the good point is that all Linux machine have same root password. :)

So First I installed "sshpass" command and wrapped ssh over ssh pass on a script.

Below is the sample script to get Uptime of all hosts
================
#!/bin/bash
for i in `cat /tmp/data.txt`
do
sshpass -p 'password' ssh -o StrictHostKeyChecking=no root@$i "hostname;uptime|awk '{print $3,$4}'" >>/tmp/uptime_detail_txt
done
===========================

Where  StrictHostKeyChecking is for disable host key checking

Tuesday, June 4, 2013

Installation and configuration HAproxy on CentOS

Step1)  Linux server has a development environment set up
yum update
yum -y groupinstall 'Development Tools'
Step 2) Download and uncompress haproxy from source code.
tar xzf haproxy-1.4.23.tar.gz
 cd haproxy-1.4.23
 make TARGET=linux26
step3) copy the configuration files
mkdir /etc/haproxy
 cp haproxy /usr/sbin/haproxy
step 4) configure haproxy.cfg file according to your need.
vi /etc/haproxy.cfg
*********************
global
      maxconn 4096
      pidfile /var/run/haproxy.pid
      daemon
defaults
        log global
        option dontlognull
        balance leastconn
        clitimeout 60000
        srvtimeout 60000
        contimeout 5000
        retries 3
        option redispatch
listen exchange-http 10.10.3.156:80
      mode http
      stats enable
      stats uri /haproxy-stats
      balance source
      option forwardfor
      server cas1 10.10.3.145:80 check port 80 inter 20 rise 1 fall 1
      server cas2 10.10.3.146:80 check port 80 inter 20 rise 1 fall 1 backup
listen exchange-https-tr 10.10.3.156:444
        mode tcp
        balance source
        maxconn 10000
        server webserver1 10.10.3.145:444 check port 444 inter 20 rise 1 fall 1
        server webserver2 10.10.3.146:444 check port 444 inter 20 rise 1 fall 1 backup
listen exchange-https 10.10.3.156:443
        mode tcp
        balance source
        maxconn 10000
        server webserver1 10.10.3.145:443 check port 443 inter 20 rise 1 fall 1
        server webserver2 10.10.3.146:443 check port 443 inter 20 rise 1 fall 1 backup

listen exchange-https 10.10.3.156:25
        mode tcp
        balance source
        maxconn 10000
        server webserver1 10.10.3.145:25 check port 25 inter 20 rise 1 fall 1
        server webserver2 10.10.3.146:25 check port 25 inter 20 rise 1 fall 1 backup
Step 5) Start HAproxy as daemon
/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg –d
Step 6) If all goes well the create a init script for HAproxy
Vi /etc/init.d/haproxy
****************
#!/bin/sh
#
# chkconfig: - 85 15
# description: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited \
#              for high availability environments.
# processname: haproxy
# config: /etc/haproxy/haproxy.cfg
# pidfile: /var/run/haproxy.pid

# Script Author: Simon Matter <simon.matter@invoca.ch>
# Version: 2004060600

# Source function library.
if [ -f /etc/init.d/functions ]; then
  . /etc/init.d/functions
elif [ -f /etc/rc.d/init.d/functions ] ; then
  . /etc/rc.d/init.d/functions
else
  exit 0
fi

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ ${NETWORKING} = "no" ] && exit 0

# This is our service name
BASENAME=`basename $0`
if [ -L $0 ]; then
  BASENAME=`find $0 -name $BASENAME -printf %l`
  BASENAME=`basename $BASENAME`
fi

[ -f /etc/$BASENAME/$BASENAME.cfg ] || exit 1

RETVAL=0

start() {
  /usr/sbin/$BASENAME -c -q -f /etc/$BASENAME/$BASENAME.cfg
  if [ $? -ne 0 ]; then
    echo "Errors found in configuration file, check it with '$BASENAME check'."
    return 1
  fi

  echo -n "Starting $BASENAME: "
  daemon /usr/sbin/$BASENAME -D -f /etc/$BASENAME/$BASENAME.cfg -p /var/run/$BASENAME.pid
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$BASENAME
  return $RETVAL
}

stop() {
  echo -n "Shutting down $BASENAME: "
  killproc $BASENAME -USR1
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$BASENAME
  [ $RETVAL -eq 0 ] && rm -f /var/run/$BASENAME.pid
  return $RETVAL
}

restart() {
  /usr/sbin/$BASENAME -c -q -f /etc/$BASENAME/$BASENAME.cfg
  if [ $? -ne 0 ]; then
    echo "Errors found in configuration file, check it with '$BASENAME check'."
    return 1
  fi
  stop
  start
}

reload() {
  /usr/sbin/$BASENAME -c -q -f /etc/$BASENAME/$BASENAME.cfg
  if [ $? -ne 0 ]; then
    echo "Errors found in configuration file, check it with '$BASENAME check'."
    return 1
  fi
  /usr/sbin/$BASENAME -D -f /etc/$BASENAME/$BASENAME.cfg -p /var/run/$BASENAME.pid -sf $(cat /var/run/$BASENAME.pid)
}

check() {
  /usr/sbin/$BASENAME -c -q -V -f /etc/$BASENAME/$BASENAME.cfg
}

rhstatus() {
  status $BASENAME
}

condrestart() {
  [ -e /var/lock/subsys/$BASENAME ] && restart || :
}

# See how we were called.
case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart)
    restart
    ;;
  reload)
    reload
    ;;
  condrestart)
    condrestart
    ;;
  status)
    rhstatus
    ;;
  check)
    check
    ;;
  *)
    echo $"Usage: $BASENAME {start|stop|restart|reload|condrestart|status|check}"
    exit 1
esac

exit $?

chmod 755 /etc/init.d/haproxy
service haproxy restart
chkconfig haproxy on

Sunday, June 2, 2013

Installation and configuration Heartbeat on Linux

Hi All,
Recently I implement Heartbeat for high Availability and ha-proxy for load balancing for four exchange servers in my office.
So, I took two Linux nodes (ex- 192.168.0.2 and 192.168.0.3) as a front to balance four exchange nodes (ex- 192.168.0.4, 192.168.0.5, 192.168.0.6, 192.168.0.7). We need to balance loads for ports (25, 110,993,995,80,443,587). We got a floating ip 192.268.0.1.
First I configured Heartbeat for High Availability of Linux nodes.
Below are steps for configuration
A.Node details are as follows
Node1 :
IpAddress(eth0):192.168.0.2(To send heartbeat signals to other nodes)
Node2 :
IpAddress(eth0):192.168.0.3(To send heartbeat signals to other nodes)
Step1 : Install the following packages in the same order which is shown.
Step1(a) : Install the following packages on node1
#rpm -ivh heartbeat-2.1.2-2.i386.rpm
#rpm -ivh heartbeat-ldirectord-2.1.2-2.i386.rpm
#rpm -ivh heartbeat-pils-2.1.2-2.i386.rpm
#rpm -ivh heartbeat-stonith-2.1.2-2.i386.rpm

Step1(b) : Install the following packages on node2
#rpm -ivh heartbeat-2.1.2-2.i386.rpm
#rpm -ivh heartbeat-ldirectord-2.1.2-2.i386.rpm
#rpm -ivh heartbeat-pils-2.1.2-2.i386.rpm
#rpm -ivh heartbeat-stonith-2.1.2-2.i386.rpm
Step2 : By default the main configuration files (ha.cf, haresources and authkeys) are not present in /etc/ha.d/ folder we have to copy these three files from /usr/share/doc/heartbeat-2.1.2 to /etc/ha.d/
Step2(a) : Copy main configuration files from /usr/share/doc/heartbeat-2.1.2 to /etc/ha.d/ on node 1
#cp /usr/share/doc/heartbeat-2.1.2/ha.cf /etc/ha.d/
#cp /usr/share/doc/heartbeat-2.1.2/haresources /etc/ha.d/
#cp /usr/share/doc/heartbeat-2.1.2/authkeys /etc/ha.d/
Step2(b) : Copy main configuration files from /usr/share/doc/heartbeat-2.1.2 to /etc/ha.d/ on node 2
#cp /usr/share/doc/heartbeat-2.1.2/ha.cf /etc/ha.d/
#cp /usr/share/doc/heartbeat-2.1.2/haresources /etc/ha.d/
#cp /usr/share/doc/heartbeat-2.1.2/authkeys /etc/ha.d/
Step3 : Edit ha.cf file
Step3(a) : Edit ha.cf file as follows on node1
********************************************
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 25
warntime 10
initdead 50
udpport 694
bcast eth0
ucast eth0 192.168.0.2
auto_failback on
node station-x.example.com
node station-y.example.com
******************************
Step3(b) : Edit ha.cf file as follows on node2
*****************************
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 25
warntime 10
initdead 50
udpport 694
bcast eth0
ucast eth0 192.168.0.3
auto_failback on
node station-x.example.com
node station-y.example.com
********************************
Step4 : Edit haresources file
Step4(a) : Just specify below entry in last line of this file on node1
station-x.example.com 192.268.0.1
Step4(b) : Just specify below entry in last line of this file on node2
station-x.example.com 192.268.0.1
Step5 (a) : Edit authkeys file on node1
#vi /etc/ha.d/authkeys
auth 2
#1 crc
2 sha1 HI!
#3 md5 Hello!
Now save and exit the file
Step5 (b) : Edit authkeys file on node2
#vi /etc/ha.d/authkeys
auth 2
#1 crc
2 sha1 HI!
#3 md5 Hello!
Step6 : Edit /etc/hosts file to give entries of host-names for the nodes
Step6(a) : Edit /etc/hosts file on node1 as below
192.268.0.2 station-x.example.com station-x
192.168.0.3 station-y.example.com station-y
Step6(b) : Edit /etc/hosts file on node2 as below
192.268.0.2 station-x.example.com station-x
192.168.0.3 station-y.example.com station-y

Step7 : Start Heartbeat cluster
Step7(a) : Start heartbeat cluster on node1
#service heartbeat start
Step7(b) : Start heartbeat cluster on node2
#service heartbeat start

Note:
1)      If on starting Heartbeat you are getting hostname error, then please set proper hostname for both nodes
2)      Checking your Heartbeat cluster:
If your heartbeat cluster is running fine a Virtual Ethernet Interface is created on node1.
# ifconfig

eth0      Link encap:Ethernet  HWaddr 08:00:27:0B:31:59
          inet addr:192.168.0.2  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:25035 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13080 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:7108268 (6.7 MiB)  TX bytes:11011415 (10.5 MiB)

eth0:0    Link encap:Ethernet  HWaddr 08:00:27:0B:31:59
          inet addr:192.168.0.1  Bcast:192.168.0.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

MySQL master-master replication and auto-increment indexes

Hi Friends,
Recently I implemented mysql maste-master replication along with Heartbeat for high availability for our monitoring tool Zabbix.
So In this blog I am sharing master-master replication. I will discuss Hartbeat in my next blogs.
Please check below steps:

Step 1) Master 2 changes to /etc/my.cnf:
1.  [mysqld]
2.  datadir=/var/lib/mysql
3.  socket=/var/lib/mysql/mysql.sock
4.  old_passwords=1
5.   
6.  log-bin
7.  binlog-do-db=zabbix  # input the database which should be replicated
8.  binlog-ignore-db=mysql            # input the database that should be ignored for replication
9.  binlog-ignore-db=test
10.  
11. server-id=2
12. auto_increment_increment= 2 #For Auto-increment
13. auto_increment_offset   = 2
14.  
15. [mysql.server]
16. user=mysql
17. basedir=/var/lib
18.  
19.  
20. [mysqld_safe]
21. err-log=/var/log/mysqld.log
22. pid-file=/var/run/mysqld/mysqld.pid

Step 2) Master 1 changes to /etc/my.cnf:

1.  [mysqld]
2.  datadir=/var/lib/mysql
3.  socket=/var/lib/mysql/mysql.sock
4.  old_passwords=1
5.   
6.  log-bin
7.  binlog-do-db=zabbix  # input the database which should be replicated
8.  binlog-ignore-db=mysql            # input the database that should be ignored for replication
9.  binlog-ignore-db=test
10. server-id=1
11. auto_increment_increment= 2 #For auto-increments
12. auto_increment_offset   = 1
13.  
14. [mysql.server]
15. user=mysql
16. basedir=/var/lib
17.  
18. [mysqld_safe]
19. err-log=/var/log/mysqld.log
20. pid-file=/var/run/mysqld/mysqld.pid

Step 3) Restarting Mysql on both servers

#Service mysql restart

Step 4) Create a replication slave account on Master 2 for Master 1
mysql> grant replication slave on zabbix.* to 'zabbix'@’%’ identified by 'password';
And start master:
1.  mysql> start master;

Step 5) On master 2
1.  mysql> show master status;
2.  +------------------------+----------+--------------+------------------+
3.  | File                   | Position | Binlog_Do_DB | Binlog_Ignore_DB |
4.  +------------------------+----------+--------------+------------------+
5.  |mysqld-bin.000012       |      106 | zabbix            |     mysql,test   |
6.  +------------------------+----------+--------------+------------------+
7.  1 row in set (0.00 sec)

Step 6) On master 1:
1.  mysql> stop slave;
2.  mysql> CHANGE MASTER TO MASTER_HOST='192.168.0.5', MASTER_USER='zabbix', MASTER_PASSWORD='password', MASTER_PORT=3306,MASTER_LOG_FILE='mysqld-bin.000012', MASTER_LOG_POS=106, MASTER_CONNECT_RETRY=10;
3.  mysql> start slave;

Step 7) On master 1

8.  mysql> show master status;
9.  +------------------------+----------+--------------+------------------+
10. | File                   | Position | Binlog_Do_DB | Binlog_Ignore_DB |
11. +------------------------+----------+--------------+------------------+
12. |mysqld-bin.000015       |      106 | zabbix            |     mysql,test   |
13. +------------------------+----------+--------------+------------------+
14. 1 row in set (0.00 sec)

Step 8) On master 2:
4.  mysql> stop slave;
5.  mysql> CHANGE MASTER TO MASTER_HOST='192.168.0.4', MASTER_USER='zabbix', MASTER_PASSWORD='password', MASTER_PORT=3306,MASTER_LOG_FILE='mysqld-bin.000015', MASTER_LOG_POS=106, MASTER_CONNECT_RETRY=10;
6.  mysql> start slave;

Note: You could check slave status from masters by executing below command

mysql>show slave status\G;


Renew k8s certificates

Check If certificate expires: amikum@~:03:06:54(⎈ |local-cluster:default):sudo kubeadm certs check-expiration CERTIFICATE                EXP...