Documentation

Monitor web traffic with syslog input from Nginx

In this tutorial, we'll start analysing the web traffic on one or many Nginx web servers using Access Watch.

We'll use the syslog protocol that is applicable as of since Nginx version 1.7.1. If you have an older version of Nginx, you can try the file based monitoring.

Install Access Watch

On the same server where Nginx is running, or on a server that is reachable by it, install the Access Watch processor.

As a prerequirement, you'll need Node.js >= 7. Use nvm if you're in trouble.

nvm install node

During the beta phase, let's use Git and clone the public repository:

git clone https://github.com/access-watch/access-watch.git
cd access-watch
npm install

Configure Access Watch

In the default configuration, Access Watch will be listening for access logs in the combined format on port 1514 and in the access_watch format on port 1515.

We always recommand using the access_watch format, which is logging more detailed information and allows for a much better analysis than the regular combined format.

You can inspect default and example configurations in ./config/default.js and ./config/example.js file.

Now, you can create your own configuration in ./config/nginx.js:

const accessWatch = require('..')();

const { pipeline, input, format } = accessWatch;

/* Syslog input in Nginx 'combined' format */

const syslogNginxCombinedInput = input.syslog.create({
  name: 'Syslog (nginx combined format)',
  port: 1514,
  parse: format.nginx.parser({format: format.nginx.formats.combined})
})

pipeline.registerInput(syslogNginxCombinedInput)

/* Syslog input in Nginx 'access_watch' format */

const syslogNginxAccessWatchInput = input.syslog.create({
  name: 'Syslog (nginx access_watch format)',
  port: 1515,
  parse: format.nginx.parser({format: format.nginx.formats.accessWatch})
})

pipeline.registerInput(syslogNginxAccessWatchInput)

Configure Nginx

First, if you're following our recommendation and opted for the access_watch format, you need to define it in the Nginx configuration. This will not replace the standard log format, just create an additional one.

log_format access_watch '"$time_iso8601" "$remote_addr" "$http_host" "$request" $status "$http_user_agent" "$http_accept" "$http_accept_language" "$http_accept_charset" "$http_accept_encoding" "$http_from" "$http_dnt" "$http_connection" "$http_referer"';

Note that you're free to use whatever log_format, you just need to properly report it in the Access Watch configuration.

Second, you need to instruct Nginx where to send the access logs. If it's not the same, you need to make sure that Nginx can reach the server where Access Watch is running.

access_log syslog:server=localhost:1515,facility=local7,tag=nginx,severity=info access_watch;

In this example, there are 3 important things:

  1. If Access Watch is running on the same server, we can use localhost as IP address.
    If it's on a different server, replace localhost by the proper private or public IP address.
  2. We configured Access Watch to listen for syslog messages in the access_watch format on port 1515.
    We're properly passing that port in the configuration
  3. Finally, we're asking nginx to use the access_watch log format we previously configured.

Don't forget to reload nginx with the updated configuration. On Ubuntu, it would be:

service nginx reload

Start Access Watch

Ok, now go back to where Access Watch is installed and start it.

npm start config/nginx.js

Access Watch API and Interface will be served from port 3000 by default. You can change that using an environment variable:

PORT=3000 npm start config/nginx.js

Warning! There is no authentication mechanism in the Access Watch processor.

If the server is on the public internet, you will need to setup your firewall properly to restrict access to it.

Browse the interface

Now, you can point your browser to the IP/port where Access Watch is running. If you see data flowing, congrats you made it!