🧩 Overview & Objectives

In this guide, we’ll build a complete log collection and visualization pipeline using the ELK Stack — Elasticsearch, Logstash, Kibana, and Filebeat.

We’ll:

• Capture live Nginx logs in real time.

• Send and parse them through Logstash.

• Store them in Elasticsearch.

• Visualize them using Kibana dashboards.

By the end, you’ll have a fully functional monitoring system showing traffic trends, error rates, and response distributions.

⚙️ 1. Setup and Configure Nginx

We’ll create a simple frontend application served by Nginx that returns different HTTP response codes — 200, 404, and 500.

🧩 1.1 Install Nginx

sudo apt update
sudo apt install -y nginx
sudo systemctl enable --now nginx

🌐 1.2 Create a Basic HTML Page

sudo mkdir -p /var/www/frontend
sudo nano /var/www/frontend/index.html

Paste:

<!DOCTYPE html>
<html>
<head>
  <title>Frontend Test</title>
  <style>
    body {display: flex; align-items: center; justify-content: center; height: 100vh; flex-direction: column;}
  </style>
</head>
<body>
  <h1>Welcome to the Frontend!</h1>
  <p>Status: 200 OK</p>
</body>
</html>

💾 1.3 Configure Nginx with JSON Logs

sudo tee /etc/nginx/sites-available/frontend.conf > /dev/null <<'EOF'
log_format elk_logging escape=json '{'
    '"timestamp": "$time_iso8601",'
    '"remote_addr": "$http_x_forwarded_for",'
    '"proxy_addr": "$remote_addr",'
    '"request": "$request",'
    '"status": "$status",'
    '"body_bytes_sent": "$body_bytes_sent",'
    '"http_user_agent": "$http_user_agent"'
'}';

server {
    listen 80;
    server_name localhost;
    root /var/www/frontend;
    index index.html;
    access_log /var/log/nginx/frontend_access.log elk_logging;
    error_log /var/log/nginx/frontend_error.log;
    location / { try_files $uri $uri/ =404; }
    location /error500 { return 500; }
}
EOF

Enable and reload:

sudo ln -s /etc/nginx/sites-available/frontend.conf /etc/nginx/sites-enabled/
sudo rm /etc/nginx/sites-enabled/default
sudo nginx -t && sudo systemctl reload nginx

🔍 1.4 Test Endpoints

curl -I http://localhost/           # ✅ 200 OK
curl -I http://localhost/notfound   # ❌ 404 Not Found
curl -I http://localhost/error500   # 💥 500 Internal Server Error

1.5. Verify Access Logs

View the last few entries from the custom JSON log:

sudo tail -n 5 /var/log/nginx/frontend_access.log

Example output:

🚦 1.6 Simulate Load

Create a bash script:

sudo nano load_test.sh

Paste:

#!/bin/bash
URL="http://localhost"
DURATION=60
DELAY=0.2
START=$(date +%s)
while [ $(( $(date +%s) - START )) -lt $DURATION ]; do
  RAND=$((RANDOM % 10))
  case $RAND in
    [0-5]) ENDPOINT="/" ;;
    [6-8]) ENDPOINT="/notfound" ;;
    9) ENDPOINT="/error500" ;;
  esac
  curl -s -o /dev/null "$URL$ENDPOINT"
  sleep $DELAY
done

Run it:

chmod +x load_test.sh
./load_test.sh

🐳 2. ELK Stack with Docker Compose

Create docker-compose.yml in a folder (e.g., /root/elk-stack):

version: '3.8'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.3
    environment:
      - discovery.type=single-node
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
    ports:
      - "9200:9200"
    volumes:
      - esdata:/usr/share/elasticsearch/data

  logstash:
    image: docker.elastic.co/logstash/logstash:7.17.3
    depends_on: [elasticsearch]
    ports: ["5044:5044"]
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf

  kibana:
    image: docker.elastic.co/kibana/kibana:7.17.3
    depends_on: [elasticsearch]
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports: ["5601:5601"]

volumes:
  esdata:

⚙️ 3. Logstash Pipeline

logstash.conf:

input { beats { port => 5044 } }

filter {
  json { source => "message" skip_on_invalid_json => true }
  mutate { add_field => { "server" => "nginx" } }
}

output {
  stdout { codec => rubydebug }
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "nginx-logs-%{+YYYY.MM.dd}"
  }
}

🧾 4. Filebeat Configuration

Example /etc/filebeat/filebeat.yml for nginx:

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/nginx/frontend_access.log
    json.keys_under_root: true
    fields:
      server: nginx

output.logstash:
  hosts: ["<elk-server-ip>:5044"]

Run Filebeat container:

docker run -d --name=filebeat \
  -v /var/log/nginx/frontend_access.log:/var/log/nginx/frontend_access.log:ro \
  -v /etc/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro \
  docker.elastic.co/beats/filebeat:7.17.3

🚀 5. Start ELK Stack

cd /root/elk-stack
docker compose up -d

Access:

• Kibana → http://<server-ip>:5601

• Elasticsearch → http://<server-ip>:9200

📊 6. Visualize in Kibana

Once everything is running — Filebeat, Logstash, Elasticsearch, and Kibana — it’s time to visualize the logs collected from Nginx.

🌐 Access Kibana

Open your browser and go to:

http://<server-ip>:5601

If you see the Kibana dashboard, it means your Elasticsearch and Kibana containers are up and running successfully.

🧱 Step 1: Create Index Pattern

To view the Nginx logs in Kibana, you first need to create an Index Pattern.

1. From the left sidebar, go to : Stack Management → Kibana → Index Patterns

   

2. Click Create index pattern.

3. In the field for the pattern name, type: nginx* (We’re using nginx* because Logstash is sending logs with this naming pattern.)

4. Set the Timestamp field (usually @timestamp).

5. Then Click Create index pattern.

   

🔍 Step 2: View Logs in Discover

Now go to the Discover section from the sidebar.
You should start seeing live Nginx logs flowing in — each entry showing fields like status, request, and timestamp.

To get quick insights, you can create simple visualizations:

• A bar chart showing counts of each status code (200, 404, 500)

• A pie chart showing percentage distribution of responses

• A time-series graph showing traffic over time

These visualizations help you understand the overall health and error rate of your application traffic.

🧩 Summary

✅ Configured Nginx with JSON logs
✅ Shipped logs via Filebeat → Logstash → Elasticsearch
✅ Visualized in Kibana
✅ Built a full monitoring pipeline

💡 You can also extend this to Docker or microservice logs by changing Filebeat paths to /var/lib/docker/containers/*/*.log.