本文目的是通過HAProxyKeepalived實現Nginx的高可用(High Availability)和負載均衡(Load Balance)。

HAProxy can run in two modes: TCP mode Layer 4 and HTTP Mode Layer 7. In Layer 4 TCP mode, HAProxy forwards the RAW TCP packets from the client to the application servers. In the Layer 7 HTTP mode, HAProxy is parsing the HTTP header before forwarding them to the application servers.

Preparation

準備兩臺lb主機,三臺nginx主機

操作系統均爲CentOS Linux release 7.2.1511 (Core)

Server Role IP
nginx1 nginx 192.168.0.131
nginx2 nginx 192.168.0.132
nginx3 nginx 192.168.0.133
loadbalancer1 lb 192.168.0.141
loadbalancer2 lb 192.168.0.142

通過HAProxy實現負載均衡,通過Keepalived實現虛擬地址漂移(vrrp)

虛擬主機IP爲 192.168.0.130,測試時直接通過該IP訪問Nginx主機。

Vagrantfile

測試主機使用Vagrant創建,如何使用Vagrant,可參考本人博文 Try to use Vagrant on CentOS7

以下是配置文件Vagrantfile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# -*- mode: ruby -*-
# vi: set ft=ruby :

# `Virtual IP`: 192.168.0.130

boxes = [
{ :name => :nginx1, :role => 'nginx', :ip => '192.168.0.131' },
{ :name => :nginx2, :role => 'nginx', :ip => '192.168.0.132' },
{ :name => :nginx3, :role => 'nginx', :ip => '192.168.0.133' },
{ :name => :loadbalance1, :role => 'lb', :ip => '192.168.0.141' },
{ :name => :loadbalance2, :role => 'lb', :ip => '192.168.0.142' },
]

Vagrant.configure(2) do |config|
config.vm.box = "CentOS7Minimal"

boxes.each do |opts|
config.vm.define opts[:name] do |config|
config.vm.network "public_network",ip: opts[:ip],bridge: "wlp3s0"
config.vm.host_name = "%s.vm" % opts[:name].to_s
config.vm.provision "shell", inline: <<-SHELL
sudo yum install -y chrony
sudo systemctl start chronyd
sudo systemctl enable chronyd
SHELL
if opts[:role] == 'nginx'
config.vm.provision "shell", inline: <<-SHELL
sudo yum install -y nginx
sudo systemctl start nginx
sudo systemctl enable nginx
SHELL
end
if opts[:role] == 'lb'
config.vm.provision "shell", inline: <<-SHELL
sudo yum install -y haproxy keepalived
SHELL
end
end
end

end

Configuations

Hosts

修改/etc/hosts文件

LoadBalancer Hosts

lb主機

/etc/hosts中寫入如下信息,分別是Nginx主機的IP和主機名

1
2
3
192.168.0.131 nginx1
192.168.0.132 nginx2
192.168.0.133 nginx3

Nginx Hosts

登錄各Nginx主機

/etc/hosts中寫入如下信息,LB主機的IP和主機名

1
2
192.168.0.141 loadbalance1
192.168.0.142 loadbalance2

LB

HAProxy

lb中的每臺主機上進行配置

  • haproxy.cfg
    1
    2
    3
    sudo cp -p /etc/haproxy/haproxy.cfg{,.bak}

    sudo vim /etc/haproxy/haproxy.cfg

寫入如下信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2

chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon

# turn on stats unix socket
stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000


#---------------------------------------------------------------------
#HAProxy Monitoring Config
#---------------------------------------------------------------------
listen haproxy3-monitoring *:8080 #Haproxy Monitoring run on port 8080
mode http
option forwardfor
option httpclose
stats enable
stats show-legends
stats refresh 5s
stats uri /stats #URL for HAProxy monitoring
stats realm Haproxy\ Statistics
#此處設置登錄haproxy stats頁面的認證用戶名、密碼
stats auth lemp:12345 #User and Password for login to the monitoring dashboard
stats admin if TRUE
default_backend nginxservers #This is optionally for monitoring backend


#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend main *:80
acl url_static path_beg -i /static /images /javascript /stylesheets
acl url_static path_end -i .jpg .gif .png .css .js

use_backend static if url_static
default_backend nginxservers

#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
backend static
balance roundrobin
server static 127.0.0.1:4331 check

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend nginxservers
balance roundrobin
#此處填寫Nginx主機IP
server nginx1 192.168.0.131:80 check
server nginx2 192.168.0.132:80 check
server nginx3 192.168.0.133:80 check

backend的名稱可自定義,此處爲nginxservers,該名稱決定了default_backend的名稱

backend填入Nginx主機IP

在瀏覽器中輸入http://HAProxyIP:8080/stats,輸入認證用戶名、密碼即可查看Nginx主機狀態信息。

  • /etc/rsyslog.conf
    1
    2
    3
    4
    5
    6
    sudo cp -p /etc/rsyslog.conf{,.bak}

    #取消以下兩行的註釋
    $ModLoad imudp
    $UDPServerRun 514
    #$UDPServerAddress 127.0.0.1

創建該文件

  • /etc/rsyslog.d/haproxy.conf
    寫入如下信息
    1
    2
    local2.=info     /var/log/haproxy-access.log    #For Access Log
    local2.notice /var/log/haproxy-info.log #For Service Info - Backend, loadbalancer

執行如下命令

1
2
3
sudo systemctl restart rsyslog
sudo systemctl start haproxy
sudo systemctl enable haproxy

Keepalived

  • /etc/sysctl.conf
    綁定非本機虛擬IP
    1
    2
    3
    #vim /etc/sysctl.conf

    net.ipv4.ip_nonlocal_bind = 1

執行sudo sysctl -p生效

  • /etc/keepalived/keepalived.conf
    1
    2
    3
    4
    5
    6
    7
    8
    9
    sudo cp -p /etc/keepalived/keepalived.conf{,.bak}

    sudo vim /etc/keepalived/keepalived.conf

    router_id #每一臺惟一
    priority #每一臺不一樣,優先級
    virtual_router_id 所有節點一致
    auth_pass 所有節點一致
    virtual_ipaddress 所有節點一致
Nodes loadbalance1 loadbalance2
router_id haproxy1 haproxy2
priority 101 100
virtual_router_id 10 10
weight 2 2
interface enp0s8 enp0s8

以下是loadbalance1中的配置,對應更改

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
global_defs {
notification_email {
[email protected]
[email protected]
[email protected]
}
notification_email_from [email protected]
smtp_server [email protected]
smtp_connect_timeout 30
#路由id,每臺主機惟一
router_id haproxy1
}

vrrp_script haproxy {
script "killall -0 haproxy"
interval 2
#權重
weight 2
}


vrrp_instance VI_1 {
state MASTER
#interface eth0
#網卡接口,根據實際情況設置
interface enp0s8
#虛擬路由id,保持一致
virtual_router_id 10
#優先級,每臺主機不一樣
priority 101
advert_int 1
authentication {
auth_type PASS
auth_pass 12345
}
#虛擬IP地址
virtual_ipaddress {
192.168.0.130
}
track_script {
haproxy
}
}

執行如下命令

1
2
sudo systemctl start keepalived
sudo systemctl enable keepalived

Nginx

在每一臺Nginx主機進行如下設置

1
sudo mv /usr/share/nginx/html/index.html{,.bak}

1
2
3
4
5
6
7
8
9
#在各自主機寫入如下信息
#nginx1
echo "<h1>nginx1.loadbalance.me</h1>" | sudo tee /usr/share/nginx/html/index.html

#nginx2
echo "<h1>nginx2.loadbalance.me</h1>" | sudo tee /usr/share/nginx/html/index.html

##nginx3
echo "<h1>nginx3.loadbalance.me</h1>" | sudo tee /usr/share/nginx/html/index.html

執行如下命令

1
2
sudo systemctl start nginx
sudo systemctl enable nginx

Testing

虛擬主機IP爲 192.168.0.130,測試時直接通過該IP訪問Nginx主機。

在Shell中直接執行如下命令

1
for ((i=0;i<9;i++)); do curl -s http://192.168.0.130 | sed '[email protected]<[^>]*>@@g'; done

可獲取Nginx服務器返回信息

Start All

所有服務啓動

  • loadbalance1

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    [email protected] ~]$ ip address show dev enp0s8
    3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:ff:35:fb brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.141/24 brd 192.168.0.255 scope global enp0s8
    valid_lft forever preferred_lft forever
    inet 192.168.0.130/32 scope global enp0s8
    valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:feff:35fb/64 scope link
    valid_lft forever preferred_lft forever
    [[email protected] ~]$
  • loadbalance2

    1
    2
    3
    4
    5
    6
    7
    8
    [[email protected] ~]$ ip address show dev enp0s8
    3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:0c:55:92 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.142/24 brd 192.168.0.255 scope global enp0s8
    valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe0c:5592/64 scope link
    valid_lft forever preferred_lft forever
    [[email protected] ~]$

可看到虛擬IP現在漂移到loadbalance1主機上

1
2
3
4
5
6
7
8
9
10
11
12
[[email protected] ~]$ for ((i=0;i<10;i++)); do curl -s http://192.168.0.130 | sed '[email protected]<[^>]*>@@g'; done
nginx1.loadbalance.me
nginx2.loadbalance.me
nginx3.loadbalance.me
nginx1.loadbalance.me
nginx2.loadbalance.me
nginx3.loadbalance.me
nginx1.loadbalance.me
nginx2.loadbalance.me
nginx3.loadbalance.me
nginx1.loadbalance.me
[[email protected] ~]$

可看到,所有Nginx主機都正常,說明HAProxy生效

登錄http://192.168.0.141:8080/stats查看HAProxy面板信息

Stop Nginx Host

停止一臺Nginx主機,此處停用nginx2,在該主機執行如下命令

1
sudo systemctl stop nginx

以模擬主機宕機

再次執行

1
2
3
4
5
6
7
8
9
10
11
12
[[email protected] ~]$ for ((i=0;i<10;i++)); do curl -s http://192.168.0.130 | sed '[email protected]<[^>]*>@@g'; done
nginx3.loadbalance.me
nginx1.loadbalance.me
nginx3.loadbalance.me
nginx1.loadbalance.me
nginx3.loadbalance.me
nginx1.loadbalance.me
nginx3.loadbalance.me
nginx1.loadbalance.me
nginx3.loadbalance.me
nginx1.loadbalance.me
[[email protected] ~]$

已經沒有nginx2.loadbalance.me

查看HAProxy面板信息

可看到Nginx2主機顯示紅色,說明該主機已經停止工作

Stop LB Host

停止一臺LoadBalance主機,此處停用loadbalance1,在該主機執行如下命令

1
2
sudo systemctl stop keepalived
sudo systemctl stop haproxy

以模擬主機宕機

  • loadbalance1

    1
    2
    3
    4
    5
    6
    7
    8
    [[email protected] ~]$ ip address show dev enp0s8
    3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:ff:35:fb brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.141/24 brd 192.168.0.255 scope global enp0s8
    valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:feff:35fb/64 scope link
    valid_lft forever preferred_lft forever
    [[email protected] ~]$
  • loadbalance2

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    [[email protected] ~]$ ip address show dev enp0s8
    3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:0c:55:92 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.142/24 brd 192.168.0.255 scope global enp0s8
    valid_lft forever preferred_lft forever
    inet 192.168.0.130/32 scope global enp0s8
    valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe0c:5592/64 scope link
    valid_lft forever preferred_lft forever
    [[email protected] ~]$

可看到虛擬IP現在漂移到loadbalance2主機上

1
2
3
4
5
6
7
8
9
10
11
12
[[email protected] ~]$ for ((i=0;i<10;i++)); do curl -s http://192.168.0.130 | sed '[email protected]<[^>]*>@@g'; done
nginx3.loadbalance.me
nginx1.loadbalance.me
nginx3.loadbalance.me
nginx1.loadbalance.me
nginx3.loadbalance.me
nginx1.loadbalance.me
nginx3.loadbalance.me
nginx1.loadbalance.me
nginx3.loadbalance.me
nginx1.loadbalance.me
[[email protected] ~]$

仍能正常訪問剩餘的Nginx主機

如果將loadbalance2也停用,則無法再正常訪問。

Strt Nginx Host

再次啓動nginx2以模擬主機上線

1
2
3
4
5
6
7
8
9
10
11
12
[[email protected] ~]$ for ((i=0;i<10;i++)); do curl -s http://192.168.0.130 | sed '[email protected]<[^>]*>@@g'; done
nginx3.loadbalance.me
nginx2.loadbalance.me
nginx1.loadbalance.me
nginx3.loadbalance.me
nginx2.loadbalance.me
nginx1.loadbalance.me
nginx3.loadbalance.me
nginx2.loadbalance.me
nginx1.loadbalance.me
nginx3.loadbalance.me
[[email protected] ~]$

說明HAProxy將請求正常負載到nginx2主機上。

關閉測試主機

1
2
3
4
5
6
7
[[email protected] HAProxyNginx]$ vagrant halt
==> loadbalance2: Attempting graceful shutdown of VM...
==> loadbalance1: Attempting graceful shutdown of VM...
==> nginx3: Attempting graceful shutdown of VM...
==> nginx2: Attempting graceful shutdown of VM...
==> nginx1: Attempting graceful shutdown of VM...
[[email protected] HAProxyNginx]$


References


Change Log

  • 2016.04.11 14:55 Mon Asia/Beijing
    • 初稿完成

  • Note Time: 2016.04.11 14:55 Mon
  • Note Location: Asia/Beijing
  • Writer: lempstacker