爲了能熟練Nginx,本人進行了相關測試操作,並將操作過程記錄下來。本文內容包括: LEMP環境安裝和簡單配置Nginx配置優化。而 Nginx使用實例 單獨寫一篇博客 。本文操作是在禁用SELinux,未啓用firewalldiptables的前提下進行。

配置優化部分參考 How to Configure Nginx for Optimized Performance

以下是操作系統環境

Info Details
OS Version CentOS Linux release 7.2.1511 (Core)
Kernel Version 3.10.0-327.10.1.el7.x86_64
PHP PHP 5.6.19
Nginx nginx/1.8.1
MariaDB 10.1.12-MariaDB

LEMP Installation

Repository

Yum Installation

repo目錄是/etc/yum.repos.d/

  • 如果安裝PHP5.6,須將remi.repo[remi-php56]下的enabled設置爲1;
  • 如果安裝PHP7.0,須將remi-php70.repo[remi-php70]下的enabled設置爲1;

PHP使用PHP-FPM來管理,故須安裝php-fpm,數據庫驅動使用php-mysqlnd

使用具體的倉庫,可用--enablerepo參數指定,如 –enablerepo=nginx

1
2
3
4
5
6
7
8
9
10
11
12
#安裝Apache httpd
yum -y install httpd
#安裝Nginx及相關依賴包
yum install openssl-devel pcre2-devel zlib-devel zip unzip
yum --enablerepo=nginx -y install nginx
#安裝PHP
yum --enablerepo=remi-php56 -y install php php-common php-fpm php-mysqlnd php-mbstring php-mcrypt php-xml php-pecl-apc php-cli php-pear php-pdo php-bcmath php-ctype php-devel php-opcache
#安裝MariaDB
yum install -y MariaDB-server MariaDB-client

Simple Configuration

各軟件配置文件默認安裝路徑

Software Configuration Files
PHP /etc/php.ini, /etc/php.d/
PHP-FPM /etc/php-fpm/php-fpm.conf, /etc/php-fpm.d/*.conf(www.conf)
Apache httpd /etc/httpd/conf/httpd.conf, /etc/httpd/conf.d
Nginx /etc/nginx/nginx.conf, /etc/nginx/conf.d/*.conf
MariaDB /etc/my.cnf, /etc/my.cnf.d

PHP Configuration

  • PHP配置文件: /etc/php.ini
1
2
cp -p /etc/php.ini{,.bak}
vim /etc/php.ini

修改 /etc/php.ini

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#關閉Nginx文件類型錯誤解析
;cgi.fix_pathinfo=1
cgi.fix_pathinfo=0
#設置時區
;date.timezone =
date.timezone = Asia/Shanghai
#禁止顯示PHP版本信息
;expose_php = On
expose_php = Off
#支持PHP短標籤
;short_open_tag = On
short_open_tag = Off
#禁用的功能函數
;disable_functions =
disable_functions = passthru,exec,system,chroot,scandir,chgrp,chown,shell_exec,proc_open,proc_get_status,ini_alter,ini_alter,ini_restore,dl,openlog,syslog,readlink,symlink,popepassthru,stream_socket_server,escapeshellcmd,dll,popen,disk_free_space,checkdnsrr,checkdnsrr,getservbyname,getservbyport,disk_total_space,posix_ctermid,posix_get_last_error,posix_getcwd, posix_getegid,posix_geteuid,posix_getgid, posix_getgrgid,posix_getgrnam,posix_getgroups,posix_getlogin,posix_getpgid,posix_getpgrp,posix_getpid, posix_getppid,posix_getpwnam,posix_getpwuid, posix_getrlimit, posix_getsid,posix_getuid,posix_isatty, posix_kill,posix_mkfifo,posix_setegid,posix_seteuid,posix_setgid, posix_setpgid,posix_setsid,posix_setuid,posix_strerror,posix_times,posix_ttyname,posix_uname

PHP-FPM Configuration

  • PHP-FPM配置文件: /etc/php-fpm/php-fpm.conf
1
2
cp /etc/php-fpm/php-fpm.conf{,bak}
vim /etc/php-fpm.d/www.conf

修改/etc/php-fpm.d/www.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
;listen = 127.0.0.1:9000
listen = /var/run/php-fpm/php-fpm.sock
#如果不指定爲nginx,無法正常監聽php-fpm.sock
;listen.owner = nobody
;listen.group = nobody
;listen.mode = 0660
listen.owner = nginx
listen.group = nginx
listen.mode = 0660
;user = apache
user = nginx
;group = apache
group = nginx

啓動PHP-FPM服務 systemctl start php-fpm

Nginx Configuration

修改/etc/nginx/conf.d/default.conf

1
2
cp /etc/nginx/conf.d/default.conf{,bak}
vim /etc/nginx/conf.d/default.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
server {
listen 80;
server_name localhost;
server_tokens off; #關閉版本信息顯示
#charset koi8-r;
#access_log /var/log/nginx/log/host.access.log main;
location / {
root /usr/share/nginx/html;
#添加上index.php
index index.php index.html index.htm;
try_files $uri $uri/ =404;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
#爲php文件設置fastcgi
location ~ \.php$ {
try_files $uri = 404;
root /usr/share/nginx/html;
fastcgi_pass unix:/var/run/php-fpm/php-fpm.sock;
fastcgi_index index.php;
#fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
#location ~ \.php$ {
# root html;
# fastcgi_pass 127.0.0.1:9000;
# fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
# include fastcgi_params;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}

注意: $document_root默認指向/etc/nginx/html/,需在location ~ \.php$指定root路徑

使用nginx -t檢測配置文件是否有語法錯誤

1
2
3
4
[[email protected] conf.d]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

啓動Nginx服務 systemctl start nginx

MariaDB Configuration

執行mysql_secure_installation進行安全配置嚮導操作

MariaDB配置文件 /etc/my.cnf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
[mysqld]
#port=3306
#datadir=/var/lib/mysql
#socket=/var/lib/mysql/mysql.sock
#user=mysql
character_set_server = utf8
collation_server = utf8_general_ci
default-time_zone = '+8:00'
# Security
skip_name_resolve = 1
symbolic_links = 0
interactive_timeout = 600 # time seconds
wait_timeout = 600 # time seconds
# InnoDB Variables Setting
default_storage_engine = InnoDB
innodb_file_per_table = 1
innodb_strict_mode = 1
innodb_file_format_check = 1
#innodb_file_format = Barracuda # For dynamic and compressed InnoDB tables
#innodb_file_format_max = Barracuda
innodb_buffer_pool_size = 1G # Go up to 80% of your available RAM
innodb_buffer_pool_instances = 8 # combine with innodb_buffer_pool_size,if its size < 1GB, the default value is 1
innodb_flush_method = O_DIRECT
# innodb_read_io_threads = 16 # If you have a strong I/O system or SSD
# innodb_write_io_threads = 32 # If you have a strong I/O system or SSD
# innodb_io_capacity = 10000 # If you have a strong I/O system or SSD
# innodb_io_capacity_max = 30000 # If you have a strong I/O system or SSD
innodb_flush_log_at_trx_commit = 2 # 1 for durability, 0 or 2 for performance
innodb_log_buffer_size = 16M
innodb_log_file_size = 256M # Bigger means more write throughput but longer recovery time
max_connections = 500 # Values < 1000 are typically good
max_user_connections = 300 # Limit one specific user/application
thread_cache_size = 500 # Up to max_connections makes sense
tmp_table_size = 32M
# Query
query_cache_type = 1
query_cache_size = 16M
max_allowed_packet = 64M
# Memory Table
#max_heap_table_size = 64M
# Slow Query Log
slow_query_log = 1
long_query_time = 1.5
slow_query_log_file = /var/log/mariadb_slow.log
#log_queries_not_using_indexes=1
#min_examined_row_limit = 100
log_timestamps = SYSTEM
# Record General Query Log
general_log_file = /var/log/mariadb_general.log
general_log = 1
# Error Log
log_error = /var/log/mariadb_error.log
log_warnings = 2
# log_error_verbosity = 3
innodb_print_all_deadlocks = 1
# Replication
server_id = 1
# auto_increment_increment = 2 # For Master/Master set-ups use 2 for both nodes
# auto_increment_offset = 1 # For Master/Master set-ups use 1 and 2
# Binary Logging
log_bin = mariadb-bin
log_bin_index = mariadb-bin.index #The index file for binary log file names.
#log_bin_basename = /var/lib/mysql/mariadb-bin #Holds the name and complete path to the binary log file. default format "datadir + '/' + hostname + '-bin'"
binlog_format=mixed
binlog_cache_size = 1M
binlog_stmt_cache_size = 1M
max_binlog_size = 64M
sync_binlog = 1
expire_logs_days = 15
binlog_row_image = full
# Relay Log
#relay_log = mariadb-relay-bin
#relay_log_index = mariadb-relay-bin.index # The name of the relay log index file for the default replication channel. default format "*host_name*-relay-bin.index"
#relay_log_basename = /var/lib/mysql/mariadb-relay-bin # Holds the name and complete path to the relay log file. default format "datadir + '/' + hostname + '-relay-bin"
#relay_log_info_repository = file
#relay_log_info_file = mariadb-relay-log.info
#relay_log_purge = 1 #Disables or enables automatic purging of relay log files as soon as they are not needed any more. The default value is 1 (ON).
#log_slave_updates = 1 # Use if Slave is used for Backup and PiTR
#read_only = 0 # Set to 1 to prevent writes on Slave
# skip_slave_start = 1 # To avoid start of Slave thread
[client]
#port=3306
#socket=/var/lib/mysql/mysql.sock
[mysqld_safe]
log-error=/var/log/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid

Create Remote Login User

數據庫通常不建議使用[email protected]%賬戶進行遠程登錄操作,此處以創建用戶lemp爲例,密碼lempstacker

1
2
3
4
5
#授權
MariaDB [(none)]> grant all on *.* to 'lemp'@'%' identified by 'lempstacker';
#刷新權限表
MariaDB [(none)]> flush privileges;

Create .my.cnf

也可在當前用戶家目錄下創建.my.cnf文件,可直接執行mysql登入數據庫。

1
2
3
4
[client]
user={USERNAME}
password={PASSWORD}
host={HOST}

其中的用戶名、密碼是數據庫賬戶的用戶名、密碼,該用戶名最好與當前系統用戶的名稱一致。

啓動MariaDB服務 systemctl start mariadb.service

Testing LEMP Suite

Test MariaDB Connection

測試MariaDB能否正常連接

Local Login

本地登錄

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
#查看文件中內容
[[email protected] ~]# cat ~/.my.cnf
[client]
user=root
password=lempstacker
host=localhost
#只用mysql命令直接登錄數據庫
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 6
Server version: 10.1.12-MariaDB MariaDB Server
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
#查看當前登錄用戶,系統時間
MariaDB [(none)]> select user(),now();
+----------------+---------------------+
| user() | now() |
+----------------+---------------------+
| [email protected] | 2016-03-11 09:34:18 |
+----------------+---------------------+
1 row in set (0.00 sec)
#查看用戶帳號[email protected]'%'的授權信息
MariaDB [(none)]> show grants for [email protected]'%'\G
*************************** 1. row ***************************
Grants for [email protected]%: GRANT ALL PRIVILEGES ON *.* TO 'lemp'@'%' IDENTIFIED BY PASSWORD '*52229031DF5348A387ACC38F11A5ED519DBFE6F6'
1 row in set (0.00 sec)
MariaDB [(none)]>

Romote Login

遠程登錄

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#遠程登錄,-u指定用戶, -p指定密碼, -h指定遠程主機地址
[[email protected] ~]$ mysql -ulemp -p -h 192.168.0.104
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 7
Server version: 10.1.12-MariaDB MariaDB Server
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
#查看當前登錄用戶,系統時間
MariaDB [(none)]> select user(),now();
+--------------------+---------------------+
| user() | now() |
+--------------------+---------------------+
| [email protected] | 2016-03-11 09:37:39 |
+--------------------+---------------------+
1 row in set (0.00 sec)
MariaDB [(none)]> \q
Bye

Test Web Server

在Nginx服務器目錄/usr/share/nginx/html/下創建info.php文件,測試LEMP能否正常工作,包括PHP和數據庫連接。

1
vim /usr/share/nginx/html/info.php
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
<?php
#測試數據庫連接
try {
$pdo = new PDO('mysql:host=localhost;port=3306','lemp','lempstacker');
$pdo->exec('set names utf8');
} catch (Exception $e) {
echo '數據庫連接失敗,報錯信息: '.$e->getMessage();
}
$sql = "select user,host from mysql.user";
$stmt = $pdo->prepare($sql);
$stmt->execute();
$rows = $stmt->fetchAll(PDO::FETCH_ASSOC);
echo '<pre>';
print_r($rows);
#顯示PHP INFO信息
#phpinfo();
?>

在瀏覽器地址欄中輸入 http://{IPADDR}(:PORT)/info.php,其中的{IPADDR}即主機IP地址

  • 如果是本機,則可用localhost127.0.0.1
  • 如果是遠程主機,則使用遠程主機的地址,如果不是80端口,則須手動指定端口

例如http://192.168.0.104/info.php

Nginx Optimization

Nginx配置優化,以達到最佳性能

Nginx官網有一篇 Pitfalls and Common Mistakes,對一些細節性的配置做了說明。

Worker Modifications

修改配置文件 /etc/nginx/nginx.conf,具體參數

爲Nginx配置合適的worker_processesworker_connections

  • worker_processes:分配主機CPU核心,根據實際情況可設置 1個多個,或 auto
  • worker_connections:每個worker_processes的最大連接數,受系統核心限制影響(默認1024);

  • 查看主機CPU核心:grep -c ^processor /proc/cpuinfo

  • 查看系統最大文件打開數: ulimit -nulimit -a | grep -Ei '^open files'

可設置epoll,可擴展的I/O事件通知機制(signal driven IO),確保I/O利用其最佳性能

也可設置multi_accept,使worker在同一時間允許所有新連接。

配置格式

1
2
3
4
5
6
7
8
9
10
# vim /etc/nginx/nginx.conf
#根據實際使用情況設置,N代表正整數,但不能大於CPU核心數
worker_processes {1|N|auto};
worker_rlimit_nofile 65536;
events {
worker_connections 65535;
use epoll;
multi_accept on;
}

HTTP and TCP Optimizations

Keep Alive

Keepalive connection 用於減少開啓/關閉連接所造成的CPU和網路消耗,支持clientsupstream servers

具體參數

可在http, server, location中設置,此處設置在http

配置格式

1
2
3
4
5
6
7
# vim /etc/nginx/nginx.conf
keepalive_timeout 50; #65
keepalive_requests 100000;
sendfile on;
tcp_nopush on;
tcp_nodelay on;

Buffer Size

設置緩衝區(Buffer)是爲了減輕磁盤I/O壓力

具體參數

配置格式

1
2
3
4
5
6
7
8
9
10
# vim /etc/nginx/nginx.conf
client_body_buffer_size 128k;
#超過大小會報413錯誤
client_max_body_size 10m;
client_header_buffer_size 1k;
large_client_header_buffers 4 4k;
output_buffers 1 32k;
postpone_output 1460;

Connection Queue

此處是內核參數調整,等待被Nginx接受的Connection Queue的大小值,謹慎操作。

具體參數

  • net.core.somaxconn: 具體路徑/proc/sys/net/core/somaxconn 默認會將其限制到128
    • 注意: 該值必須小於ulimit -n值,否則會報錯sysctl: setting key "net.core.somaxconn": Invalid argument
  • net.ipv4.tcp_max_tw_buckets:具體路徑/proc/sys/net/ipv4/tcp_max_tw_buckets
  • net.core.netdev_max_backlog: 具體路徑/proc/sys/net/core/netdev_max_backlog
    • 在CPU處理之前,packets可緩存在網卡中
1
2
3
4
5
6
7
[[email protected] ~]# cat /proc/sys/net/core/somaxconn
128
[[email protected] ~]# cat /proc/sys/net/ipv4/tcp_max_tw_buckets
4096
[[email protected] ~]# cat /proc/sys/net/core/netdev_max_backlog
1000

爲使更改後的參數值永久生效,可寫入文件/etc/sysctl.conf

1
2
3
4
5
# vim /etc/sysctl.conf
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65536
net.ipv4.tcp_max_tw_buckets = 1440000

使用命令sysctl -p載入文件/etc/sysctl.conf中的內核參數設置

1
2
3
4
5
6
7
8
9
10
11
[[email protected] ~]# sysctl -p
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65536
net.ipv4.tcp_max_tw_buckets = 1440000
[[email protected] ~]# cat /proc/sys/net/core/somaxconn
65535
[[email protected] ~]# cat /proc/sys/net/ipv4/tcp_max_tw_buckets
1440000
[[email protected] ~]# cat /proc/sys/net/core/netdev_max_backlog
65536

If there are error messages in the kernel log, increase the value until errors stop.

Timeouts

具體參數

配置格式

1
2
3
4
5
# vim /etc/nginx/nginx.conf
client_header_timeout 3m;
client_body_timeout 3m;
send_timeout 90s;

Static Asset Serving

靜態文件緩存

具體參數

配置格式

1
2
3
4
5
6
# vim /etc/nginx/nginx.conf
open_file_cache max=1000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 5;
open_file_cache_errors off;

參數解釋: 最多緩存1000個文件,有效期30s,有效期內至少訪問過5次,在20s內未被訪問的文件會取消緩存。

也可通過設置location實現,如

1
2
3
location ~* .(woff|eot|ttf|svg|mp4|webm|jpg|jpeg|png|gif|bmp|ico|css|js)$ {
expires 365d;
}

Gzipping Content

對於純文本文件,Nginx可使用gzip將其壓縮後再返回給client,可提高傳輸速度,減少網路帶寬消耗。相關參數在模塊ngx_http_gzip_module

具體參數

  • gzip 默認 off
  • gzip_comp_level 默認 level 1,可選範圍 [1,9]
  • gzip_min_length 默認 length 20,由響應header中Content-Length決定
  • gzip_buffers 默認 number size 是 32 4K | 16 8K

    By default, the buffer size is equal to one memory page. This is either 4K or 8K, depending on a platform.

  • gzip_proxied 默認 off,可選off | expired | no-cache | no-store | private | no_last_modified | no_etag | auth | any ...中的多個

  • gzip_types 默認 mime-type 是 text/html,可設置多個
  • gzip_disable 默認未設置 regex
  • gzip_vary 默認爲off,是否在響應header中添加Vary:Accept-Encoding
  • gzip_http_version ,默認是1.1

配置格式

1
2
3
4
5
6
7
8
9
# vim /etc/nginx/nginx.conf
gzip on;
gzip_vary on;
gzip_comp_level 5;
gzip_min_length 1000;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/html text/css application/javascript application/x-javascript text/javascript text/plain text/xml application/json application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype application/x-font-ttf application/xml font/eot font/opentype font/otf image/svg+xml image/vnd.microsoft.icon;
gzip_disable "MSIE [1-6]\.";

Filesystem Optimizations

修改內核參數,提升系統內存管理功能。

Connection Queue一樣,將相關參數寫入文件/etc/sysctl.conf,使用命令sysctl -p載入文件/etc/sysctl.conf中的內核參數設置。

Ephemeral Ports

Nginx作爲代理時,每個到達upstream server的連接使用的都是臨時端口。IPv4 local port range定義了端口範圍,TCP FIN timeout定義的端口被再次使用的間隔時間。

具體參數

  • net.ipv4.ip_local_port_range: 對應文件路徑 /proc/sys/net/ipv4/ip_local_port_range,默認範圍 [32768,61000]
  • net.ipv4.tcp_fin_timeout 15: 對應文件路徑 /proc/sys/net/ipv4/tcp_fin_timeout,默認時間 60s
1
2
3
4
# vim /etc/sysctl.conf
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_fin_timeout = 15

使用命令sysctl -p載入文件參數設置

1
2
3
4
5
6
7
8
9
[[email protected] ~]# vim /etc/sysctl.conf
[[email protected] ~]# sysctl -p
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_fin_timeout = 10
[[email protected] ~]# cat /proc/sys/net/ipv4/ip_local_port_range
1024 65000
[[email protected] ~]# cat /proc/sys/net/ipv4/tcp_fin_timeout
10

Scale TCP Window

TCP window scale可增加TCP會話接收窗口大小,最高值是65,535 bytes,參數值1爲啓用,0爲禁用

具體參數

  • net.ipv4.tcp_window_scaling: 對應文件路徑 /proc/sys/net/ipv4/tcp_window_scaling
1
2
3
# vim /etc/sysctl.conf
net.ipv4.tcp_window_scaling = 1

使用命令sysctl -p載入文件參數設置

1
2
3
4
5
6
[[email protected] ~]# vim /etc/sysctl.conf
[[email protected] ~]# sysctl -p
net.ipv4.tcp_window_scaling = 1
[[email protected] ~]# cat /proc/sys/net/ipv4/tcp_window_scaling
1

Backlog Packets Before Drop

數據包在內核刪除之前保存在backlog中的數量

具體參數

  • net.ipv4.tcp_max_syn_backlog: 對應文件路徑 /proc/sys/net/ipv4/tcp_max_syn_backlog 默認值128
1
2
3
# vim /etc/sysctl.conf
net.ipv4.tcp_max_syn_backlog = 3240000

使用命令sysctl -p載入文件參數設置

1
2
3
4
5
6
[[email protected] ~]# vim /etc/sysctl.conf
[[email protected] ~]# sysctl -p
net.ipv4.tcp_max_syn_backlog = 3240000
[[email protected] ~]# cat /proc/sys/net/ipv4/tcp_max_syn_backlog
3240000

Close connection on Missing Client Response

修改Nginx配置文件 /etc/nginx/nginx.conf

允許serverclient停止響應後關閉連接,釋放與socket相關的內存。

具體參數

配置格式

1
2
3
# vim /etc/nginx/nginx.conf
reset_timedout_connection on;

File Descriptors

File Descriptors是用於處理諸如連接和打開文件之類的操作系統資源。對於每個連接,Nginx可同時使用2個File Descriptor(文件操作符)

  • sys.fs.file_max設置了系統(所有進程)允許打開的File Descriptors數;
  • ulimit則設置了 Shell單個進程 允許打開的File Descriptors數;
    • nofile是最多能打開的File Descriptors數;

執行man proc,找到/proc/sys/fs/file-max,有如下敘述

This file defines a system-wide limit on the number of open files for all processes.

The kernel constant NR_OPEN imposes an upper limit on the value
that may be placed in file-max.

If you increase /proc/sys/fs/file-max, be sure to increase /proc/sys/fs/inode-max to 3-4 times the new value of /proc/sys/fs/file-max, or you will run out of inodes.

Privileged processes (CAP_SYS_ADMIN) can override the file-max
limit.

/proc/sys/fs/inode-max: This file contains the maximum number of in-memory inodes. On some (2.4) systems, it may not be present. This value should be 3-4 times larger than the value in file-max, since stdin, stdout and network sockets also need an inode to handle them. When you regularly run out of inodes, you need to increase this value.

執行man ulimit,搜索ulimit,有如下敘述

Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.

對於nofile,在文件/etc/security/limits.conf中有如下描述

nofile - max number of open file descriptors

具體參數

  • fs.file-max 文件對應路徑 /proc/sys/fs/file-max
  • nofile 在文件/etc/security/limits.conf中修改
1
2
3
# vim /etc/sysctl.conf
fs.file-max = 800000

使用命令sysctl -p載入文件參數設置

1
2
3
4
5
6
[[email protected] ~]# vim /etc/sysctl.conf
[[email protected] ~]# sysctl -p
fs.file-max = 800000
[[email protected] ~]# cat /proc/sys/fs/file-max
800000

修改文件/etc/security/limits.conf

1
2
3
4
5
6
# vim /etc/security/limits.conf
#星號*代表所有用戶,也可具體指定如nginx,數值看實際情況具體指定,此處暫設爲65536
# nofile - max number of open file descriptors
* soft nofile 65536
* hard nofile 65536

修改後須重啓系統才能生效

Error Logs

error_log logs/error.log warn; 定義了location和寫入錯誤日誌(error lgo)不同嚴重性級別(severity level)

具體參數

配置格式

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# vim /etc/nginx/nginx.conf
error_log /var/log/nginx/error.log warn;
http {
log_format compression '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" "$gzip_ratio"';
server {
gzip on;
access_log /var/log/nginx/nginx-access.log compression;
...
}
}


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# vim /etc/nginx/nginx.conf
error_log /var/log/nginx/error.log warn;
#當網站性能下降是可用於診斷問題
http {
log_format upstream_time '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"'
'rt=$request_time uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"';
server {
access_log /var/log/nginx/nginx-access.log upstream_time;
...
}
}

設置Conditional Logging

Conditional Logging

使用參數if啓用Conditional Logging,不記錄響應碼(response code)是2xx3xx的請求

1
2
3
4
5
6
7
8
# vim /etc/nginx/nginx.conf
map $status $loggable {
~^[23] 0;
default 1;
}
access_log /path/to/access.log combined if=$loggable;

Turning off Logging Completely

如果有其他記錄日誌方法或不需要記錄日誌,可關閉日誌記錄功能

access_logerror_log設置爲off

1
2
3
4
5
6
7
8
# vim /etc/nginx/nginx.conf
server {
listen 80;
server_name example.com;
access_log off;
error_log off;
}

Activity Monitoring

啓用Nginx狀態監控功能須有模塊with-http_stub_status_module支持,使用nginx -V 2>&1 | grep -o with-http_stub_status_module可查看是否加載該模塊。

:在Bash V4+中,可通過nginx -V |& grep -o with-http_stub_status_module查看。

通配命令

1
[[ $(bash --version | awk '{print gensub(/.* ([[:digit:]]{1}).*/,"\\1","g",$0);exit}') -ge 4 ]] && nginx -V 2>&1 | grep -o with-http_stub_status_module || nginx -V |& grep -o with-http_stub_status_module
1
2
3
[[email protected] ~]# nginx -V 2>&1 | grep -o with-http_stub_status_module
with-http_stub_status_module

開啓Nginx狀態監控功能

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
location /nginx_status {
stub_status on;
access_log off;
allow 123.123.123.123; # 允许访问的 IP
allow 127.0.0.1;
deny all;
}
server {
...
location /status {
stub_status on;
access_log off;
allow 123.123.123.123; # 允许访问的 IP
allow 127.0.0.1;
deny all;
}
}

注意location後的名稱就是在瀏覽器中需要輸入的名稱,如http://192.168.0.104/nginx_status

返回格式

1
2
3
4
Active connections: 2
server accepts handled requests
35 35 80
Reading: 0 Writing: 1 Waiting: 1

解釋

  • Active connections:Nginx當前打開的connection數;非用戶數,一個頁面可打開多個併發connection
  • server accepts handled requests
    • accepts: Nginx總共接收的connection數
    • handled: Nginx總共已處理的connection數,數值通常與accepts一致
    • requests:Nginx總共處理的request(請求)數,數值通常比handled
    • requests/handled 得到每個連接的請求數 80/35=2.2857 requests per connections
  • Reading:Nginx讀取 request header
  • Writing:Nginx讀取 request body, processes requeset 或 發響應給client
  • Waiting: keep-alive connections,其值等於 active – (reading + writing),取決於參數 keepalive_timeout

The first parameter sets a timeout during which a keep-alive client connection will stay open on the server side.

Mounting Partitions

掛載分區時通過noatimenodiratime減少磁盤I/O,掛載分區文件/etc/fstab

1
2
3
#/etc/fatab
UUID=662AD5322AD4FFCB /mnt/data ext4 defaults,noatime,nodiratime 0 0

執行mount -o remount /mnt/data重新掛載更改的分區。

Security Optimization

OCSP Stapling Configuration

OCSP stapling is a TLS/SSL extension which aims to improve the performance of SSL negotiation while maintaining visitor privacy.

OCSP (Online Certificate Status Protocol) is a protocol for checking if a SSL certificate has been revoked. It was created as an alternative to CRL to reduce the SSL negotiation time. With CRL (Certificate Revocation List) the browser downloads a list of revoked certificate serial numbers and verifies the current certificate, which increases the SSL negotiation time. In OCSP the browser sends a request to a OCSP URL and receives a response containing the validity status of the certificate.

操作過程參考How To Configure OCSP Stapling on Apache and Nginx。操作需要用到根證書(root CA)和中間證書(intermediate CA),此處選擇使用Let’s Encrypt,以下是相關頁面

Let’s Encrypt Root and Intermediate Certificates

1
2
3
4
5
6
#Active Root Certificates (ISRG Root X1)
https://letsencrypt.org/certs/isrgrootx1.pem
#Active Intermediate Certificates
#Let’s Encrypt Authority X3 (IdenTrust cross-signed)
https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem

執行如下命令生成所需的證書(/etc/nginx/ssl/letsencrypt-ca-cert.pem)

1
2
#此處存放在路徑/etc/nginx/ssl/下,名稱爲letsencrypt-ca-cert.pem
wget -q -O - https://letsencrypt.org/certs/isrgrootx1.pem https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem | tee -a /etc/nginx/ssl/letsencrypt-ca-cert.pem > /dev/null

在Nginx的server中添加如下內容

1
2
3
4
5
6
#OCSP Stapling Configuration
ssl_stapling on;
ssl_stapling_verify on; # Enables verification of OCSP responses by the server
#Let's Encrypt Root and Intermediate Certificates
ssl_trusted_certificate /etc/nginx/ssl/letsencrypt-ca-cert.pem;

執行nginx -t && nginx -s reload重新載入Nginx配置文件後,可執行如下命令檢測CSP Stapling是否工作正常

1
echo QUIT | openssl s_client -connect lemptest.tech:443 -status 2> /dev/null | sed -r -n '/^OCSP response/,/Next Update/p'

以下是測試過程

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[[email protected] ~]$ echo QUIT | openssl s_client -connect lemptest.tech:443 -status 2> /dev/null | sed -r -n '/^OCSP response/,/Next Update/p'
OCSP response:
======================================
OCSP Response Data:
OCSP Response Status: successful (0x0)
Response Type: Basic OCSP Response
Version: 1 (0x0)
Responder Id: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
Produced At: Dec 20 15:26:00 2016 GMT
Responses:
Certificate ID:
Hash Algorithm: sha1
Issuer Name Hash: 7EE66AE7729AB3FCF8A220646C16A12D6071085D
Issuer Key Hash: A84A6A63047DDDBAE6D139B7A64565EFF3A8ECA1
Serial Number: 0329DC63D27FB5891787B4B13377802B6F2F
Cert Status: good
This Update: Dec 20 15:00:00 2016 GMT
Next Update: Dec 27 15:00:00 2016 GMT

HTTP Public Key Pinning(HPKP) Configuration

HPKP is a HTTP response header that allows a host to deliver a set of fingerprints to cryptographic identities that the User Agent (UA) should accept for the site going forwards. Much like HSTS, HPKP is a Trust On First Use (TOFU) mechanism and the UA must establish at least one secure session with the host to receive the header intact. Once the header has been received, the UA will cache and apply the policy as per the directives included in the header. – https://scotthelme.co.uk/hpkp-http-public-key-pinning/

此處不做配置,相關閱讀

Configuration File Example

/etc/sysctl.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
# Avoid a smurf attack
net.ipv4.icmp_echo_ignore_broadcasts = 1
# Turn on protection for bad icmp error messages
net.ipv4.icmp_ignore_bogus_error_responses = 1
#Connection Queue
net.core.somaxconn = 65535 # max backlog
net.core.netdev_max_backlog = 65535 # max processor input queue
net.ipv4.tcp_max_tw_buckets = 1440000 # max timewait sockets held by system simultaneously
#Ephemeral Ports
net.ipv4.ip_local_port_range = 1024 65000 # outbound port range
net.ipv4.tcp_fin_timeout = 15 # short FIN timeout
#Scale TCP Window
net.ipv4.tcp_window_scaling = 1
#Backlog Packets Before Drop
net.ipv4.tcp_max_syn_backlog = 3240000
#File Descriptors
fs.file-max = 800000 # max open files
# Increase TCP buffer sizes
net.core.rmem_max = 16777216 # max read buffer
net.core.wmem_max = 16777216 # max write buffer
# Increase TCP max buffer size setable using setsockopt()
net.ipv4.tcp_rmem = 4096 87380 16777216 # TCP receive buffer
net.ipv4.tcp_wmem = 4096 65536 16777216 # TCP write buffer
# for high-latency network
net.ipv4.tcp_congestion_control = hybla
# for low-latency network, use cubic instead
# net.ipv4.tcp_congestion_control = cubic
net.core.rmem_default = 65536 # default read buffer
net.core.wmem_default = 65536 # default write buffer
net.ipv4.tcp_syncookies = 1 # Turn on syncookies for SYN flood attack protection
net.ipv4.tcp_tw_reuse = 1 # reuse timewait sockets when safe
net.ipv4.tcp_tw_recycle = 0 # turn off fast timewait sockets recycling
net.ipv4.tcp_keepalive_time = 1200 # short keepalive time
net.ipv4.tcp_mtu_probing = 1 # turn on path MTU discovery
# Turn on reverse path filtering
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
# Turn on and log spoofed, source routed, and redirect packets
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
# No source routed packets here
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
# Don't act as a router
# net.ipv4.ip_forward = 0
# net.ipv4.conf.all.send_redirects = 0
# net.ipv4.conf.default.send_redirects = 0
# Make sure no one can alter the routing tables
# net.ipv4.conf.all.accept_redirects = 0
# net.ipv4.conf.default.accept_redirects = 0
# net.ipv4.conf.all.secure_redirects = 0
# net.ipv4.conf.default.secure_redirects = 0
# Turn IPv6
# net.ipv6.conf.default.router_solicitations = 0
# net.ipv6.conf.default.accept_ra_rtr_pref = 0
# net.ipv6.conf.default.accept_ra_pinfo = 0
# net.ipv6.conf.default.accept_ra_defrtr = 0
# net.ipv6.conf.default.autoconf = 0
# net.ipv6.conf.default.dad_transmits = 0
# net.ipv6.conf.default.max_addresses = 1
# Turn on execshild
kernel.exec-shield = 1
kernel.randomize_va_space = 1

/etc/security/limits.conf

1
2
3
4
5
#星號*代表所有用戶,也可具體指定如nginx,數值看實際情況具體指定,此處暫設爲65536
# nofile - max number of open file descriptors
* soft nofile 655360
* hard nofile 655360

/etc/nginx/nginx.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
user nginx;
# worker_processes 1 or N or auto
worker_processes 2;
worker_rlimit_nofile 65536;
pid /var/run/nginx.pid;
events {
worker_connections 65536;
use epoll;
multi_accept on;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
charset utf-8;
server_tokens off; #關閉版本信息顯示
autoindex off; #禁止顯示目錄下文件,默認off
sendfile on;
tcp_nopush on;
tcp_nodelay on;
# [debug|info|notice|warn|error|crit|alert|emerg]
error_log /var/log/nginx/error.log warn;
access_log /var/log/nginx/access.log combined if=$loggable;
#Conditional Logging
map $status $loggable {
~^[23] 0;
default 1;
}
#log_format name string ...; default combined "...";
log_format cpmpression '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" "$gzip_ratio"';
# Concurrency Connections
# http://nginx.org/en/docs/http/ngx_http_limit_conn_module.html#limit_conn_zone
# One megabyte zone can keep about 32 thousand 32-byte states or about 16 thousand 64-byte states.
limit_conn_zone $binary_remote_addr zone=addr:10m;
#Keep Alive
keepalive_timeout 50;
keepalive_requests 100000;
#Timeouts
client_header_timeout 3m;
client_body_timeout 3m;
send_timeout 60s;
#Buffer Size
client_body_buffer_size 128k;
client_max_body_size 2m;
client_header_buffer_size 1k;
large_client_header_buffers 4 4k;
output_buffers 1 32k;
postpone_output 1460;
#Close connection on Missing Client Response
reset_timedout_connection on;
#Static Asset Serving
open_file_cache max=1000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 5;
open_file_cache_errors off;
# gzip compression
gzip on;
gzip_vary on;
gzip_comp_level 5;
gzip_buffers 16 8k;
gzip_min_length 1000;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/css application/javascript application/x-javascript text/javascript text/plain text/xml application/json application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype application/x-font-ttf application/xml font/eot font/opentype font/otf image/svg+xml image/vnd.microsoft.icon;
gzip_disable "MSIE [1-6]\.";
gzip_static on;
include /etc/nginx/conf.d/*.conf;
}

/etc/nginx/conf.d/default.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
server {
listen 80;
server_name localhost;
#access_log /var/log/nginx/log/host.access.log main;
location / {
# try_files $uri $uri/ =404;
root /usr/share/nginx/html;
index index.php index.html index.htm;
}
location ~ \.php$ {
#try_files $uri = 404;
root /usr/share/nginx/html;
fastcgi_pass unix:/var/run/php-fpm/php-fpm.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
location ~* .(woff|eot|ttf|svg|mp4|webm|jpg|jpeg|png|gif|bmp|ico|css|js)$ {
expires 365d;
log_not_found off;
access_log off;
}
location /status {
stub_status on;
access_log off;
allow 192.168.0.102; #以此爲例
deny all;
}
}

/etc/nginx/conf.d/ssl.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
# redirect http to https
server {
listen 80;
server_name lemptest.tech;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl;
server_name lemptest.tech;
#access_log /var/log/nginx/log/host.access.log main;
root /usr/share/nginx/html;
ssl_certificate /etc/letsencrypt/live/lemptest.tech/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/lemptest.tech/privkey.pem;
ssl_dhparam /etc/nginx/ssl/dhparam.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
#https://scotthelme.co.uk/doing-the-chacha-with-nginx/
ssl_ciphers "ECDHE-ECDSA-CHACHA20-POLY1305 ECDHE-RSA-CHACHA20-POLY1305 EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH DHE-RSA-CHACHA20-POLY1305 EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4 !SEED !CAMELLIA";
ssl_ecdh_curve secp384r1; # Specifies a curve for ECDHE ciphers.
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_session_tickets on;
ssl_session_ticket_key /etc/nginx/ssl/ticket.key; #the list of certificates will not be sent to clients
#OCSP Stapling Configuration
ssl_stapling on;
ssl_stapling_verify on; # Enables verification of OCSP responses by the server
#Let's Encrypt Root and Intermediate Certificates
ssl_trusted_certificate /etc/nginx/ssl/letsencrypt-ca-cert.pem;
# Google DNS, Open DNS, Dyn DNS
resolver 8.8.8.8 8.8.4.4 208.67.222.222 208.67.220.220 216.146.35.35 216.146.36.36 valid=300s;
resolver_timeout 5s;
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
add_header Content-Security-Policy 'default-src self';
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "DENY";
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Robots-Tag none;
add_header X-Download-Options noopen;
add_header X-Permitted-Cross-Domain-Policies none;
add_header Public-Key-Pins 'pin-sha256="iUUgoZuZgkGIbQ9x1lUQbvCJh+87iT1avjyzKKu7K3k=";pin-sha256="F2gQEUpXylr1jmAr6f9WNlFAMxORt597saJMqGCcoks="; max-age=2592000; includeSubDomains';
#if ($scheme = http) {
# return 301 https://$host$request_uri;
#}
# Disable unwanted HTTP methods
# 405 A request was made of a resource using a request method not supported by that resource;
if ($request_method !~ ^(GET|HEAD|POST)$ )
{
return 405;
}
# Deny Certain User-Agents or Bots:
if ($http_user_agent ~* LWP::Simple|wget|libwww-perl) {
return 403;
}
if ($http_user_agent ~ (msnbot|Purebot|Baiduspider|Lipperhey|Mail.Ru|scrapbot) ) {
return 403;
}
# Blocking Referral Spam
if ( $http_referer ~* (jewelry|viagra|nude|girl|nudit|casino|poker|porn|sex|teen|babes) ) {
return 403;
}
# Stop Hotlinking 防盜鏈
# location ~ .(gif|png|jpe?g)$ {
# valid_referers none blocked example.com *.example.com;
# if ($invalid_referer) {
# return 403;
# }
# }
# Deny execution of scripts
# deny scripts inside writable directories
# location ~* /(images|cache|media|logs|tmp)/.*.(php|pl|py|jsp|asp|sh|cgi)$ {
# return 403;
# error_page 403 /403_error.html;
# }
# file cache
# location ~* .(woff|eot|ttf|svg|mp4|webm|jpg|jpeg|png|gif|bmp|ico|css|js)$ {
# expires 365d;
# log_not_found off;
# access_log off;
# }
# location ~ ^/favicon\.ico$ {
# root /usr/share/nginx/html;
# }
}

SSL安全等級檢測 (SSL Server Test)

期間遇到一問題:訪問PHP頁面,不執行而是直接下載。檢查配置文件參數無誤,甚惑,後通過清空瀏覽器緩存解決。

References

Installation

Optimization

Security

Change Logs

  • 2016.03.11 18:18 Fri Asia/Beijing
    • 初稿完成
  • 2016.03.23 15:48 Wed Asia/Beijing
    • 內容更新,相關配置參數
  • 2016.03.30 21:56 Wed Asia/Beijing
    • 添加Mounting Partitions
  • 2016.08.19 10:41 Fri Asia/Shanghai
    • 添加參數worker_rlimit_nofile
  • 2016.12.08 14:18 Thu Asia/Shanghai
    • MariaDB配置文件/etc/my.cnf參數優化
  • 2016.12.20 23:18 Tue Asia/Shanghai
    • Nginx參數配置優化
  • 2016.12.21 11:33 Wed Asia/Shanghai
    • 添加Security Optimization安全相關

  • Note Time: 2016.03.11 18:18 Fri
  • Note Location: Asia/Beijing
  • Writer: lempstacker