How to use OpenResty to perform health checks

Use OpenResty to perform health checks in your cloud environment as it incorporates nginx, Lua and comes with batteries included.

I really love it!

Installation

Install curl utility as it is a first dependency.

$ sudo apt install curl

Import OpenResty GPG public key using curl.

$ curl --silent https://openresty.org/package/pubkey.gpg | sudo apt-key add -

Define OpenResty repository.

$ echo "deb http://openresty.org/package/debian $(lsb_release -sc) openresty" | sudo tee /etc/apt/sources.list.d/openresty.list

Update package index.

$ sudo apt update

Install openresty package including recommended packages.

$ sudo apt install openresty
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  openresty-openssl openresty-opm openresty-pcre openresty-resty openresty-zlib
Suggested packages:
  openresty-restydoc
The following NEW packages will be installed:
  openresty openresty-openssl openresty-opm openresty-pcre openresty-resty openresty-zlib
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,938 kB of archives.
After this operation, 8,886 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://openresty.org/package/debian stretch/openresty amd64 openresty-zlib amd64 1.2.11-3~stretch1 [80.3 kB]
Get:2 http://openresty.org/package/debian stretch/openresty amd64 openresty-openssl amd64 1.1.0j-1~stretch1 [1,534 kB]
Get:3 http://openresty.org/package/debian stretch/openresty amd64 openresty-pcre amd64 8.42-1~stretch1 [258 kB]
Get:4 http://openresty.org/package/debian stretch/openresty amd64 openresty amd64 1.15.8.1-1~stretch1 [1,034 kB]
Get:5 http://openresty.org/package/debian stretch/openresty amd64 openresty-resty all 1.15.8.1-1~stretch1 [13.2 kB]
Get:6 http://openresty.org/package/debian stretch/openresty amd64 openresty-opm amd64 1.15.8.1-1~stretch1 [19.0 kB]
Fetched 2,938 kB in 1s (2,065 kB/s) 
[...]

Alternatively, install openresty package, but skip recommended packages.

$ sudo apt install --no-install-recommends openresty
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  openresty-openssl openresty-pcre openresty-zlib
Suggested packages:
  openresty-restydoc
Recommended packages:
  openresty-resty openresty-opm
The following NEW packages will be installed:
  openresty openresty-openssl openresty-pcre openresty-zlib
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,906 kB of archives.
After this operation, 8,712 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://openresty.org/package/debian stretch/openresty amd64 openresty-zlib amd64 1.2.11-3~stretch1 [80.3 kB]
Get:2 http://openresty.org/package/debian stretch/openresty amd64 openresty-openssl amd64 1.1.0j-1~stretch1 [1,534 kB]
Get:3 http://openresty.org/package/debian stretch/openresty amd64 openresty-pcre amd64 8.42-1~stretch1 [258 kB]
Get:4 http://openresty.org/package/debian stretch/openresty amd64 openresty amd64 1.15.8.1-1~stretch1 [1,034 kB]
Fetched 2,906 kB in 1s (1,538 kB/s)
[...]

I will recommend to install recommended packages on development server, but skip these on production where I like to keep things small.

Initial configuration

OpenResty nginx core configuration is stored inside /usr/local/openresty/nginx/ directory, basic settings in conf/nginx.conf configuration file.

You should configure SSL certificate as a basic precaution and change listening port, so it will not affect other services.

Application is controlled by the openresty service.

$ sudo systemctl status openresty
● openresty.service - full-fledged web platform
   Loaded: loaded (/lib/systemd/system/openresty.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-08-09 09:41:07 GMT; 2h 1min ago
 Main PID: 12674 (nginx)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/openresty.service
           ├─12674 nginx: master process /usr/local/openresty/nginx/sbin/nginx -g daemon on; master_process on;
           └─12675 nginx: worker process

Aug 09 09:41:07 stretch systemd[1]: Starting full-fledged web platform...
Aug 09 09:41:07 stretch systemd[1]: Started full-fledged web platform.

Reload this service after you modify configuration or Lua scripts.

HTTP health check

Install lua-resty-http package using OpenResty Package Manager.

Install openresty-opm package to use OpenResty Package Manager
$ sudo opm install ledgetech/lua-resty-http
* Fetching ledgetech/lua-resty-http  
  Downloading https://opm.openresty.org/api/pkg/tarball/ledgetech/lua-resty-http-0.14.opm.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20862  100 20862    0     0  62166      0 --:--:-- --:--:-- --:--:-- 62089
Package ledgetech/lua-resty-http 0.14 installed successfully under /usr/local/openresty/site/ .

Alter OpenResty configuration to include HTTP health check endpoint.

This configuration file location is /usr/local/openresty/nginx/conf/nginx.conf
  location /health/http {
    default_type "text/html";
    content_by_lua_file health/http.lua;
  }

Create Lua script to perform HTTP health check.

Lua script location is /usr/local/openresty/nginx/health/http.lua
local http = require("resty.http")
local httpc = http.new()

local url = "http://127.0.0.1"
local parameters = { keepalive = false }
local required_string = "Welcome to nginx!"

local res, err = httpc:request_uri(url, parameters)
if not res then
  ngx.status = 503
  ngx.header['Error-Message'] = err
  ngx.say("Error")
  return
end

if res.body and string.find(res.body, required_string) then
  ngx.status = 200
  ngx.say("OK")
  return
else
  ngx.status = 503
  ngx.header['Error-Message'] = "required string not found"
  ngx.say("Error")
  return
end

ngx.status = 501
ngx.header['Error-Message'] = "reached end of the script"
ngx.say("Error")

Reload OpenResty service.

$ sudo systemctl reload openresty

Green (success) health check.

$ curl http://192.168.50.191:81/health/http -v
*   Trying 192.168.50.191...
* TCP_NODELAY set
* Connected to 192.168.50.191 (192.168.50.191) port 81 (#0)
> GET /health/http HTTP/1.1
> Host: 192.168.50.191:81
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: openresty/1.15.8.1
< Date: Fri, 09 Aug 2019 14:39:00 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: keep-alive
< 
OK
* Connection #0 to host 192.168.50.191 left intact

Red (fail) health check.

$ curl http://192.168.50.191:81/health/http -v
*   Trying 192.168.50.191...
* TCP_NODELAY set
* Connected to 192.168.50.191 (192.168.50.191) port 81 (#0)
> GET /health/http HTTP/1.1
> Host: 192.168.50.191:81
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 503 Service Temporarily Unavailable
< Server: openresty/1.15.8.1
< Date: Fri, 09 Aug 2019 14:41:13 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: keep-alive
< Error-Message: connection refused
< 
Error
* Connection #0 to host 192.168.50.191 left intact
I will use the same procedure to create other health checks, so I can omit some obvious information later on.

nginx health check

This is a simplified version of the previous HTTP health check.

Configure nginx service (you can use /etc/nginx/sites-enabled/ping configuration file for example) to reply with pong to ping request.

server {
  listen 80 default_server;

  location /ping {
    access_log off;
    default_type "text/html";
    return 200 "pong\n";
  }
}

Alter OpenResty configuration to include nginx health check endpoint.

  location /health/nginx {
    default_type "text/html";
    content_by_lua_file health/nginx.lua;
  }

Create Lua script (health/nginx.lua) to perform this health check.

local http = require("resty.http")
local httpc = http.new()

local url = "http://127.0.0.1/ping"
local parameters = { keepalive = false }
local required_string = "pong"

local res, err = httpc:request_uri(url, parameters)
if not res then
  ngx.status = 503
  ngx.say("Error")
  return
end

if res.body and string.find(res.body, required_string) then
  ngx.status = 200
  ngx.say("OK")
  return
end

ngx.status = 501
ngx.say("Error")

Reload both services and verify this health check.

ProxySQL health check

Alter OpenResty configuration to include ProxySQL health check endpoint.

  location /health/proxysql {
    default_type "text/html";
    content_by_lua_file health/proxysql.lua;
  }

Create Lua script (health/proxysql.lua) to perform this health check.

local mysql = require("resty.mysql")
local db, err = mysql:new()
if not db then
  ngx.status = 503
  ngx.header['Error-Message'] = err
  ngx.say("Error")
  return
end

local proxysql_host = "127.0.0.1"
local proxysql_port = 6032
local proxysql_user = "admin"
local proxysql_pass = "admin"
local proxysql_timeout = 1000

db:set_timeout(proxysql_timeout)

local ok, err, errno, sqlstate = db:connect({ host = proxysql_host, port = proxysql_port, user = proxysql_user, password = proxysql_pass })
if not ok then
  ngx.status = 503
  ngx.header['Error-Message'] = err
  ngx.say("Error")
  return
end

local res, err, errno, sqlstate = db:query("select Variable_Value from stats.stats_mysql_global where Variable_Name='ProxySQL_Uptime'")
if res and res[1]["Variable_Value"] then
  ngx.status = 200
  ngx.header['ProxySQL-Uptime'] = res[1]["Variable_Value"]
  ngx.say("OK")
  return
end

ngx.status = 501
ngx.say("Error")
db:close()

MariaDB health check

Alter OpenResty configuration to include MariaDB health check endpoint.

  location /health/mariadb {
    default_type "text/html";
    content_by_lua_file health/mariadb.lua;
  }

Create Lua script (health/mariadb.lua) to perform this health check.

local mysql = require("resty.mysql")
local db, err = mysql:new()
if not db then
  ngx.status = 503
  ngx.header['Error-Message'] = err
  ngx.say("Error")
  return
end

local mariadb_socket = "/var/run/mysqld/mysqld.sock"
local mariadb_user = "admin"
local mariadb_pass = "admin"
local mariadb_timeout = 1000

db:set_timeout(proxysql_timeout)

local ok, err, errno, sqlstate = db:connect({ path = mariadb_socket, user = mariadb_user, password = mariadb_pass })
if not ok then
  ngx.status = 503
  ngx.header['Error-Message'] = err
  ngx.say("Error")
  return
end

local res, err, errno, sqlstate = db:query("select VARIABLE_VALUE from information_schema.GLOBAL_STATUS  where VARIABLE_NAME='Uptime'")
if res and res[1]["VARIABLE_VALUE"] then
  ngx.status = 200
  ngx.header['MariaDB-Uptime'] = res[1]["VARIABLE_VALUE"]
  ngx.say("OK")
  return
end

ngx.status = 501
ngx.say("Error")

PostgreSQL health check

Install pgmoon package using OpenResty Package Manager.

$ sudo opm install leafo/pgmoon
* Fetching leafo/pgmoon  
  Downloading https://opm.openresty.org/api/pkg/tarball/leafo/pgmoon-1.9.0.opm.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 17242  100 17242    0     0  22860      0 --:--:-- --:--:-- --:--:-- 22837
Package leafo/pgmoon 1.9.0 installed successfully under /usr/local/openresty/site/ .

Alter OpenResty configuration to include PostgreSQL health check endpoint.

  location /health/postgresql {
    default_type "text/html";
    content_by_lua_file health/postgresql.lua;
  }

Create Lua script (health/postgresql.lua) to perform this health check.

local pgmoon = require("pgmoon")

local pg_host = "127.0.0.1"
local pg_port = 5432
local pg_user = "admin"
local pg_pass = "admin"
local pg_database = "postgres"
local pg_timeout = 1000

local pg, err = pgmoon.new({ host = pg_host, port = pg_port, user = pg_user, password = pg_pass, database = pg_database })
if not pg then
  ngx.status = 503
  ngx.header['Error-Message'] = err
  ngx.say("Error")
  return
end

pg:settimeout(pg_timeout)

local ok, err = pg:connect()
if not ok then
  ngx.status = 503
  ngx.header['Error-Message'] = err
  ngx.say("Error")
  return
end

local res = pg:query("SELECT date_trunc('second', current_timestamp - pg_postmaster_start_time()) as pretty_uptime")
if res and res[1]["pretty_uptime"] then
  ngx.status = 200
  ngx.header['PostgreSQL-Uptime'] = res[1]["pretty_uptime"]
  ngx.say("OK")
  return
end

ngx.status = 501
ngx.say("Error")
pg:disconnect()

Redis health check

Alter OpenResty configuration to include Redis health check endpoint.

  location /health/redis {
    default_type "text/html";
    content_by_lua_file health/redis.lua;
  }

Create Lua script (health/redis.lua) to perform this health check.

local redis = require("resty.redis")
local red = redis:new()

local redis_socket = "unix:///var/run/redis/redis.sock"
local redis_password = "password"
local redis_timeout = 1000

red:settimeout(redis_timeout)

local ok, err = red:connect(redis_socket)
if not ok then
  ngx.status = 503
  ngx.header['Error-Message'] = err
  ngx.say("Error")
  return
end

local res, err = red:auth(redis_password)
if not res then
  ngx.status = 503
  ngx.header['Error-Message'] = err
  ngx.say("Error")
  return
end

local res, err = red:info("server")
if res then
  ngx.status = 200
  ngx.header['Redis-Uptime'] = string.match(res, "uptime_in_seconds:([0-9]+)")
  ngx.say("OK")
  return
end

ngx.status = 501
ngx.say("Error")
red:close()

Additional information