Py学习  »  NGINX

从零学ELK系列(九):Nginx接入ELK(超详细图文教程)

当年的春天 • 4 年前 • 352 次点击  

【前言】

在前几篇博文中将ELK+Filebeat收集SpringBoot项目日志搭建完毕,本次我们将展示如何将Nginx接入我们搭建的日志系统,把步骤记录下来,一是方便自己以后安装,二是可以为大家做参考共享。

【一句总结一张架构图】

一、一句话总结学完本篇博文,你将学到什么?

Nginx接入ELK+Filebeat收集系统,Kibana设置展示日志

二、架构图

【SpringBoot接入ELK】

一、环境:

1、Windows系统(本人是win10环境)

2、VMware10.0.1

3、Centos 7.4

4、Xshell5

5、Docker 19.03

6、Elasticsearch 7.2.0

7、Kibana 7.2.0

8、Logstash 7.2.0

9、Filebeat 7.2.0

10、SpringBoot项目 (项目地址: https://github.com/dangnianchuntian/springboot 版本号1.7.0-Release)

11、Nginx 1.16.1

二、Nginx接入步骤:

1、CentOS安装Nginx

(1)安装Nginx组件epel

yum install epel-release

(2)yum安装nginx

yum -y install nginx

(3)启动nginx

nginx

(4)检查nginx是否启动成功

curl localhost

2、配置Nginx

(1)查找nginx的配置文件

find / -name nginx.conf

(2)编辑配置文件增加日志输出的格式

vim /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format json '{ "@timestamp": "$time_iso8601", '
                    '"time": "$time_iso8601", '
                    '"remote_addr": "$remote_addr", '
                    '"remote_user": "$remote_user", '
                    '"body_bytes_sent": "$body_bytes_sent", '
                    '"request_time": "$request_time", '
                    '"status": "$status", '
                    '"host": "$host", '
                    '"request": "$request", '
                    '"request_method": "$request_method", '
                    '"uri": "$uri", '
                    '"http_referrer": "$http_referer", '
                    '"body_bytes_sent":"$body_bytes_sent", '
                    '"http_x_forwarded_for": "$http_x_forwarded_for", '
                    '"http_user_agent": "$http_user_agent" '
                    '}';


    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    default_type        application/octet-stream;

    include /etc/nginx/conf.d/*.conf;

}

(3)编辑项目配置文件

vim /etc/nginx/conf.d/zh_boot.conf
server {
        listen       8081;
        server_name  192.168.37.129;        
        
        location / {
        
          proxy_pass http://192.168.37.129:8080;                            
                                 
        }
       access_log /elklogs/nginx-log/nginx_access.json json;
    }

(4)创建Nginx访问日志目录

mkdir /elklogs/nginx-log/ -p

(5)检查nginx配置文件是否正确

nginx -t

(6)重启nginx

nginx -s reload

3、启动项目(zh-boot)

//为防止影响将springadmin的client关闭
java -jar -Dspring.boot.admin.client.enabled=false zh-boot.jar

4、通过nginx访问项目swagger验证nginx配置正确

http://192.168.37.129:8081/swagger-ui.html

5、ELK+Filebeat增加nginx访问请求的收集

(1)在原来Logstash基础上增加收集nginx请求日志

a.编辑logstash配置文件

vim /data/elk/logstash/logstash.conf
input {
  beats {
     port => 5045
     codec => json
        }
}

filter {
    if [fields][service] == "zhboot" {
       date {
            match => [ "requestTime" , "yyyy-MM-dd HH:mm:ss" ]
            target => "@timestamp"


    

        }
       mutate {
           remove_field => "parent"
           remove_field => "meta"
           remove_field => "trace"
           remove_field => "tags"
           remove_field => "prospector"
           remove_field => "span"
           remove_field => "fields"
           remove_field => "severity"
           remove_field => "@version"
           remove_field => "exportable"
           remove_field => "input"
           remove_field => "pid"
           remove_field => "thread"
           remove_field => "beat"
           remove_field => "host"
           remove_field => "offset"
           remove_field => "log"
        }

    }
}

output {

   if [fields][service] == "nginx_zhboot" {
       elasticsearch {
            hosts => ["172.17.0.2:9200"]
            index => "nginx_zhboot_%{+YYYYMM}"
        }

   }

   elasticsearch {
      hosts => ["172.17.0.2:9200"]
      index => "%{[esindex]}_%{+YYYYMM}"
  }
}

b.停止logstash容器

docker stop lst

c.将配置文件copy到logstash容器内

docker cp logstash.conf lst:/usr/share/logstash/config/

d.启动logstash容器

docker start lst

(2)在原来Filebeat基础上增加收集nginx请求日志

a.编辑Filebeat配置文件

vim /data/elk/filebeat-7.2.0-linux-x86_64/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /elklogs/zh-boot-allrequest-log/*.json
  fields:
   service: zhboot
filebeat.config.modules:
  path: /data/elk/filebeat-7.2.0-linux-x86_64/modules.d/*.yml
  reload.enabled: false
output.logstash:
  hosts: ["172.17.0.4:5045"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /elklogs/nginx-log/*.json
  fields:
   service: nginx_zhboot
filebeat.config.modules:
  path: /data/elk/filebeat-7.2.0-linux-x86_64/modules.d/*.yml
  reload.enabled: false
output.logstash:
  hosts: ["172.17.0.4:5045"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

b.启动filebeat

sh /data/elk/filebeat-7.2.0-linux-x86_64/start.sh

三、进行验证:

1、正常请求的验证

(1)用Postman进行访问一下(用swagger直接访问的话不通过nginx)

(2)在Kibana中创建索引

a.create index pattern

b.Define index pattern

c.Configure settings

d.在Discover中查看刚才的访问

2、异常请求的演示

a.用postman模拟不存在的url

b.在Kibana中查看

【总结】

Nginx做为流量的入口,通过对入口的日志收集以及接下来配合相关错误的监控,可以有效的避免攻击,快速排错(比如和第三方交互,可以通过Nginx的日志确认第三方是否发起请求)。

Python社区是高质量的Python/Django开发社区
本文地址:http://www.python88.com/topic/52558
 
352 次点击