Linux Worms: July 2018

Monday, July 2, 2018

Keepalived configure for Failover

Friends,

Today I am showing Haproxy setup along with Keepalived. In this blog I will setup Keep alive. HaProxy setup will be done on my next blog.

Keepalived is used to achieve high availability by assigning a VIP to two or more servers.

Installation:-
I am doing this on two ubuntu servers having ip 172.16.1.209 and 172.16.1.211. The virtual IP will be 172.16.1.210.

Install packages

apt-get install keepalived

Configuring keepalived

Create the config file on the first server (172.16.1.209):


vim /etc/keepalived/keepalived.conf




global_defs {

  router_id NM-PRD-HAPROXY-209

}

vrrp_script haproxy {

script "killall -0 haproxy"

interval 1

weight 2

}

vrrp_instance 50 {

virtual_router_id 50

advert_int 1

priority 101

state MASTER

interface eth0

virtual_ipaddress {

172.16.1.210

  }

  track_script {

haproxy

  }

authentication {

        auth_type PASS

        auth_pass We$,J)Og&

    }

}

Create the config file on the second server (172.16.1.211)


vim /etc/keepalived/keepalived.conf




global_defs {

  router_id NM-PRD-HAPROXY-211

}

vrrp_script haproxy {

  script "killall -0 haproxy"

  interval 1

  weight 2

}

vrrp_instance 50 {

  virtual_router_id 50

  advert_int 1

  priority 102

  state MASTER

  interface eth0

  virtual_ipaddress {

    172.16.1.210

  }

  track_script {

    haproxy

  }

authentication {

        auth_type PASS

        auth_pass We$,J)Og&




    }

}

Note :-

We need to enable sysctl setting for no local ip binding allow on the Kernel.

Temporary:

echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind

Permanent:

Add this to /etc/sysctl.conf:

net.ipv4.ip_nonlocal_bind = 1

Enable with:

sysctl -p

Start service
When our Haproxy setup done, We need to start Keepalived on both servers:

service keepalived start

Run Rsync as daemon

Friends,

We all know rsync is a good tool to sync files/directories.
Every time your want to sync, you need to run rsync commands. But what if you want to run rsync as a daemon? Every change in the source will be synced to destination location manually?

Below is a small script to do this

On source location


archival@cme103:~$ mkdir rsync-as-daemon ;cd rsync-as-daemon

archival@cme103:~$touch /home/archival/rsync-as-daemon/rsync.lock;touch /home/archival/rsync-as-daemon/rsync.log

archival@cme103:~$ cat rsyncd.conf

pid file = /home/archival/rsync-as-daemon/rsyncd.pid

lock file = /home/archival/rsync-as-daemon/rsync.lock

log file = /home/archival/rsync-as-daemon/rsync.log

use chroot = false

port = 15000

reverse lookup = no

[app_logs]

path = /home/archival/log/

comment = Logs Rsync

read only = true

timeout = 300

Now run rsync as a daemon

archival@cme103:~$ rsync --daemon --config=/home/archival/rsync-as-daemon/rsyncd.conf

On Destination Please configure as below


logmanager@ip-10-101-2-145:~ mkdir rsync-client;cd rsync-client

logmanager@ip-10-101-2-145:~ cat rsync_client.sh

#!/bin/bash
SCRIPT_DIR=$(dirname $0)
cd $SCRIPT_DIR
TARGET_BASE_DIR="/opt/rsync_logs/"
RSYNC_INCLUDES="$SCRIPT_DIR/rsync_includes.cfg"
while true ;
do
    SERVER_URL=($(cat "$SCRIPT_DIR/rsync_url.list" | xargs))
    for URL in "${SERVER_URL[@]}"
    do
        IP=$(echo $URL | cut -d':' -f 1)
        TARGET_DIR="$TARGET_BASE_DIR/$IP"
        if [ ! -f $TARGET_DIR ]; then
            mkdir -p $TARGET_DIR
        fi
        rsync -rtm --append --include-from $RSYNC_INCLUDES --exclude '*' rsync://${URL} ${TARGET_DIR}
    done
done

logmanager@ip-10-101-2-145:~/rsync-client$ cat rsync_includes.cfg
### Directories and file that should be synced  
*/
platform*.log

logmanager@ip-10-101-2-145:~/ rsync-client$ cat rsync_url.list
## URL for Rsync
192.168.201.103:15000/app_logs/

Now run rsync client

logmanager@ip-10-101-2-145:~/rsync-client$/home/logmanager/rsync-client/rsync_client.sh

Bash to save history with time stamp

Some Useful bashrc stuff

Please put below in /etc/bash.bashrc

## No duplicates in the history

export HISTCONTROL=ignoredups

## for setting history length see HISTSIZE and HISTFILESIZE in bash


export HISTSIZE=100000

export HISTFILESIZE=100000

## Timestamp history

export HISTTIMEFORMAT="%b %d %a %T "

## Don't Overwrite. Append to bash history.

shopt -s histappend

##Adding costume $PATH

export PATH="/usr/local/zabbix/bin:$PATH"

##Adding Vim as default editor

export EDITOR=vim

postgress basic commands

1) Install Postgress in Ubuntu

airflow@ip-10-1-1-4:~/airflow$ sudo apt-get install postgresql postgresql-contrib

2) Login to postgress DataBase


ansible@ip-10-101-2-142:~$ sudo -u postgres psql

or

ansible@ip-10-101-2-142:~$ psql -h"localhost" -p 5432 -Uairflow airflow -W

Password for user airflow:

psql (9.5.12)

SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)

Type "help" for help.



airflow=#

3) create DataBase


airflow@ip-10-1-1-4:~/airflow$ sudo -u postgres createdb airflow

or

 airflow=# createdb airflow WITH ENCODING='UTF-8';

4) show database


airflow=# \d+

                                List of relations

 Schema |          Name           |   Type   |  Owner  |    Size    | Description

--------+-------------------------+----------+---------+------------+-------------

 public | sqpl_airflow_hosts      | table    | airflow | 16 kB      |

(1 rows)

5) Create user

postgres=# CREATE USER airflow WITH ENCRYPTED PASSWORD 'Ubuntu@123';

6) GRANT Permission to user


postgres=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public to airflow;



GRANT

7) connection info


airflow=# \conninfo

You are connected to database "airflow" as user "airflow" via socket in "/var/run/postgresql" at port "5432".

8) Show config file


airflow=# SHOW hba_file;

hba_file

--------------------------------------

/etc/postgresql/9.5/main/pg_hba.conf

(1 row)

9) Create Table


CREATE TABLE sqpl_airflow_hosts(

   ip character varying(15) NOT NULL,

   logical_group  character varying(200) NOT NULL,

   queue_name character varying(100) NOT NULL,

   param_kwargs character varying(1000) NOT NULL,

   host_state host_state_enum,

   primary key (logical_group,param_kwargs)

);

10) Describe table


airflow-# \d+ sqpl_airflow_hosts

 ip                     | character varying(15)     | not null  | extended |              |

 logical_group  | character varying(200)    | not null  | extended |              |

 queue_name     | character varying(100)   | not null  | extended |              |

 param_kwargs  | character varying(1000) | not null  | extended |              |

 host_state          | host_state_enum            |                | plain    |                 |

11) select query


airflow=# select * from sqpl_airflow_hosts;

 ip | logical_group | queue_name | param_kwargs | host_state

----+---------------+------------+--------------+------------

(0 rows)

12) Insert Query


airflow=# insert into sqpl_airflow_hosts values ('xxx','cme_engines','10.101.2.253_queue','--ip 192.168.201.105 --engine_id 12','RUNNING');

INSERT 0 1

airflow=# select * from sqpl_airflow_hosts;

 ip  | logical_group |     queue_name     |            param_kwargs             | host_state

-----+---------------+--------------------+-------------------------------------+------------

 xxx | cme_engines   | 10.101.2.253_queue | --ip 192.168.201.105 --engine_id 12 | RUNNING

(1 row)

13) Alter Table

airflow=# alter table sqpl_airflow_hosts add column params character varying[];


airflow=# \d+ sqpl_airflow_hosts

                              Table "public.sqpl_airflow_hosts"

    Column     |          Type           | Modifiers | Storage  | Stats target | Description

---------------+-------------------------+-----------+----------+--------------+-------------

 ip            | character varying(15)   | not null  | extended |              |

 logical_group | character varying(200)  | not null  | extended |              |

 queue_name    | character varying(100)  | not null  | extended |              |

 param_kwargs  | character varying(1000) | not null  | extended |              |

 host_state    | host_state_enum         |           | plain    |              |

 command       | character varying       |           | extended |              |

 params        | character varying[]     |           | extended |              |

Indexes:

    "sqpl_airflow_hosts_pkey" PRIMARY KEY, btree (logical_group, param_kwargs)



airflow=# alter table sqpl_airflow_hosts drop column param_kwargs;

airflow=# alter table sqpl_airflow_hosts add constraint pkey unique(logical_group, command, params);

ALTER TABLE

airflow=# \d+ sqpl_airflow_hosts

                             Table "public.sqpl_airflow_hosts"

    Column     |          Type          | Modifiers | Storage  | Stats target | Description

---------------+------------------------+-----------+----------+--------------+-------------

 logical_group | character varying(200) | not null  | extended |              |

 queue_name    | character varying(100) | not null  | extended |              |

 host_state    | host_state_enum        |           | plain    |              |

 command       | character varying      |           | extended |              |

 params        | character varying[]    |           | extended |              |

Indexes:

    "pkey" UNIQUE CONSTRAINT, btree (logical_group, command, params)

ALTER TABLE  sqpl_airflow_hosts ALTER COLUMN logical_group TYPE character varying(100);

14) Update table

airflow=# update sqpl_airflow_hosts set command = 'start.sh', params = {'--ip = 192.168.201.105', '--engine_id 12'} ;

15) Getting Size of DB



airflow=# select pg_relation_size('sqpl_airflow_hosts');
 pg_relation_size
------------------
             8192
(1 row)




airflow=# select pg_size_pretty(pg_database_size(current_database()));

 pg_size_pretty

----------------

 11 MB

(1 row)

16) Know who is connected


airflow=# SELECT datname,usename,client_addr,client_port FROM pg_stat_activity ;

 datname | usename | client_addr  | client_port

---------+---------+--------------+-------------

 airflow | airflow | 127.0.0.1    |       46896

 airflow | airflow | 127.0.0.1    |       37942

 airflow | airflow | 127.0.0.1    |       45694

 airflow | airflow | 10.101.2.82  |       52146

 airflow | airflow | 10.101.2.105 |       35482

 airflow | airflow | 10.101.2.253 |       38740

 airflow | airflow | 127.0.0.1    |       52116

 airflow | airflow | 127.0.0.1    |       40694

 airflow | airflow | 127.0.0.1    |       53752

 airflow | airflow | 127.0.0.1    |       53454

 airflow | airflow | 127.0.0.1    |       40546

 airflow | airflow | 127.0.0.1    |       35432

 airflow | airflow | 127.0.0.1    |       40392

 airflow | airflow | 127.0.0.1    |       40848

 airflow | airflow | 127.0.0.1    |       37928

(15 rows)

17 ) Reload postgress configuration file

select pg_reload_conf();

18 ) Show DataBase directory

airflow=# SHOW data_directory;
        data_directory
------------------------------
 /var/lib/postgresql/9.5/main
(1 row)

19) Insert into table

insert into sqpl_airflow_hosts values ('cme_engines','10.101.2.253_queue','RUNNING','cd /home/deployment && /home/deployment/airflow_scripts/start_engine.py', '{"--ip=192.168.201.105", "--engine_id=28"}');

20) Some Select examples


airflow=# select concat(command, ' ', array_to_string(params, ' ')) from sqpl_airflow_hosts ;
                                                   concat
-------------------------------------------------------------------------------------------------------------
 cd /home/deployment && /home/deployment/airflow_scripts/start_engine.py --ip=192.168.201.105 --engine_id=12
 cd /home/deployment && /home/deployment/airflow_scripts/start_engine.py --ip=192.168.201.105 --engine_id=13
 cd /home/deployment && /home/deployment/airflow_scripts/start_engine.py --ip=192.168.201.105 --engine_id=15
 cd /home/deployment && /home/deployment/airflow_scripts/start_engine.py --ip=192.168.201.105 --engine_id=17
 cd /home/deployment && /home/deployment/airflow_scripts/start_engine.py --ip=192.168.201.105 --engine_id=19
 cd /home/deployment && /home/deployment/airflow_scripts/start_engine.py --ip=192.168.201.105 --engine_id=21
 cd /home/deployment && /home/deployment/airflow_scripts/start_engine.py --ip=192.168.201.105 --engine_id=22
 cd /home/deployment && /home/deployment/airflow_scripts/start_engine.py --ip=192.168.201.105 --engine_id=23
 cd /home/deployment && /home/deployment/airflow_scripts/start_engine.py --ip=192.168.201.105 --engine_id=24
 cd /home/deployment && /home/deployment/airflow_scripts/start_engine.py --ip=192.168.201.105 --engine_id=28
(10 rows)

SUDO COMMANDS EXAMPLES

Syntax for sudoers file is as below

USER_ALIAS HOST_ALIAS=EFECTIVE_USER) COMMAND_ALIAS

Now Let's understand the components.

USER_ALIAS : User aliases are used to specify users, groups or netgroups.

examples:


User_Alias ADMINS = %admin

User_Alias WEBMASTERS = tom,rob

User_Alias LIMITED_USERS = !WEBMASTERS, !ADMINS

HOST_ALIAS: list of hostname, ip addresses, networks and netgroups

examples:


Host_Alias SERVERS = 192.168.0.1, 192.168.0.2, server1

Host_Alias WORKSTATIONS = 192.168.0.0/255.255.255.0, !SERVERS

EFECTIVE_USER : Who can run what as who

examples:

User_Alias EFECTIVE_USER = sam,rob

COMMAND_ALIAS: Lists of commands ( full path of commands) and directories

examples


Cmnd_Alias        SBIN_CMDS = /sbin

Cmnd_Alias        APP_CMDS =  /usr/bin/passwd, /sbin/service httpd *, /sbin/ifconfig : DB_CMDS = /bin/su – admin, /home/admin/zabbix_server : ADMIN_CMDS = /sbin/, /usr/sbin/

Now let's try some usecase

1) user rob to run /bin/su command with root privileges. Same time it restricts using any options/argument and login to root.

rob          ALL = (root)        /bin/su [!-]*[!root]*

2) This lets the admins run all the admin commands on the servers

ADMINS SERVERS= ADMIN_CMDS

3) This lets "rob" shutdown his own machine without a password

rob rob-machine= NOPASSWD: SHUTDOWN_CMDS

4) rob should run command hwclock only ( without any arguments)

rob        ALL = (ALL)          /sbin/hwclock “”

Linux Worms