How to use HAProxy stats socket

Use HAProxy stats socket to determine current application status.

Initial information

Operating system.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 10 (buster)
Release:        10
Codename:       buster

HAProxy version.

$ sudo haproxy -v
HA-Proxy version 1.8.19-1+deb10u1 2019/11/27
Copyright 2000-2019 Willy Tarreau <[email protected]>

Default HAProxy configuration.

$ cat /etc/haproxy/haproxy.cfg 
global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        # An alternative list with additional directives can be obtained from
        #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

Socket permissions.

$ ls -l /run/haproxy/admin.sock 
srw-rw---- 1 root haproxy 0 Jan 11 21:30 /run/haproxy/admin.sock

Stats socket configuration

Access level for a stats socket

There are three distinct access levels: admin, operator and user.

Use the following table to quickly inspect available commands and required access level, but remember that these things are subject to change.
Socket command Permission level
Admin Operator User
help Yes Yes Yes
help message
prompt Yes Yes Yes
toggle interactive mode with prompt
quit Yes Yes Yes
disconnect
show tls-keys [id|*] Yes Yes Yes
show tls keys references or dump tls ticket keys when id specified
set ssl tls-key [id|keyfile] Yes Yes Yes
set the next TLS key for the or listener to
show errors Yes Yes
report last request and response errors for each proxy
disable agent Yes
disable agent checks (use 'set server' instead)
disable health Yes
disable health checks (use 'set server' instead)
disable server Yes
disable a server for maintenance (use 'set server' instead)
enable agent Yes
enable agent checks (use 'set server' instead)
enable health Yes
enable health checks (use 'set server' instead)
enable server Yes
enable a disabled server (use 'set server' instead)
set maxconn server Yes
change a server's maxconn setting
set server Yes
change a server's state, weight or address
get weight Yes Yes Yes
report a server's current weight
set weight Yes
change a server's weight (deprecated)
show sess [id] Yes Yes
report the list of current sessions or dump this session
shutdown session Yes
kill a specific session
shutdown sessions server Yes
kill sessions on a server
clear table Yes Yes Yes
remove an entry from a table
set table [id] Yes Yes Yes
update or create a table entry's data
show table [id] Yes Yes Yes
report table usage stats or dump this table's contents
clear counters Yes Yes
clear max statistics counters (add 'all' for all counters)
show info Yes Yes Yes
report information about the running process
show stat Yes Yes Yes
report counters for each proxy and server
show schema json Yes Yes Yes
report schema used for stats
show startup-logs Yes Yes Yes (but see commit 869efd)
report logs emitted during HAProxy startup
show resolvers [id] Yes Yes Yes
dumps counters from all resolvers section and associated name servers
set maxconn global Yes
change the per-process maxconn setting
set rate-limit Yes
change a rate limiting value
set severity-output [none|number|string] Yes Yes Yes
set presence of severity level in feedback information
set timeout Yes Yes Yes
change a timeout setting
show env [var] Yes Yes
dump environment variables known to the process
show cli sockets Yes Yes Yes
dump list of cli sockets
show fd [num] Yes Yes
dump list of file descriptors in use
show activity Yes Yes Yes
show per-thread activity stats (for support/developers)
disable frontend Yes
temporarily disable specific frontend
enable frontend Yes
re-enable specific frontend
set maxconn frontend Yes
change a frontend's maxconn setting
show servers state [id] Yes Yes Yes
dump volatile server information (for backend )
show backend Yes Yes Yes
list backends in the current running config
shutdown frontend Yes
stop a specific frontend
set dynamic-cookie-key backend Yes
change a backend secret key for dynamic cookies
enable dynamic-cookie backend Yes
enable dynamic cookies on a specific backend
disable dynamic-cookie backend Yes
disable dynamic cookies on a specific backend
show cache Yes
show cache status
add acl Yes Yes Yes
add acl entry
clear acl [id] Yes Yes Yes
clear the content of this acl
del acl Yes Yes Yes
delete acl entry
get acl Yes Yes Yes
report the patterns matching a sample for an ACL
show acl [id] Yes Yes Yes
report available acls or dump an acl's contents
add map Yes Yes Yes
add map entry
clear map [id] Yes Yes Yes
clear the content of this map
del map Yes Yes Yes
delete map entry
get map Yes Yes Yes
report the keys and values matching a sample for a map
set map Yes Yes Yes
modify map entry
show map [id] Yes Yes Yes
report available maps or dump a map's contents
show pools Yes Yes Yes
report information about the memory pools usage

Stats socket permissions

You can use a single socket or define multiple sockets using different permissions like mode and user/uid and group/gid ownership.

stats socket /run/haproxy/admin.sock mode 600 level admin 
stats socket /run/haproxy/operator.sock mode 660 level operator user haproxy group haproxy
stats socket /run/haproxy/user.sock mode 660 level user uid 106 gid 112
$ ls -l /var/run/haproxy
total 0
srw------- 1 root    haproxy 0 Jan 12 02:04 admin.sock
srw-rw---- 1 haproxy haproxy 0 Jan 12 02:04 operator.sock
srw-rw---- 1 haproxy haproxy 0 Jan 12 02:04 user.sock

Stats socket operations

Required utilities

Install socat to interact with stats socket and rlwrap, expect to use interactive mode.

$ sudo apt-get install expect socat rlwrap

One-shot mode

This is the simplest mode of operation, which is especially well suited for shell scripts.

$ echo "show pools" | sudo -u haproxy socat stdio unix-connect:/var/run/haproxy/admin.sock
Dumping pools usage. Use SIGQUIT to flush them.
  - Pool cache_st (16 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users [SHARED]
  - Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 0 failures, 2 users [SHARED]
  - Pool email_alert (48 bytes) : 59 allocated (2832 bytes), 6 used, 0 failures, 4 users [SHARED]
  - Pool tcpcheck_ru (64 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 5 users [SHARED]
  - Pool spoe_appctx (128 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 3 users [SHARED]
  - Pool spoe_ctx (144 bytes) : 47 allocated (6768 bytes), 5 used, 0 failures, 2 users [SHARED]
  - Pool h2s (160 bytes) : 56 allocated (8960 bytes), 14 used, 0 failures, 2 users [SHARED]
  - Pool h2c (240 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users [SHARED]
  - Pool http_txn (288 bytes) : 46 allocated (13248 bytes), 4 used, 0 failures, 1 users [SHARED]
  - Pool connection (384 bytes) : 59 allocated (22656 bytes), 7 used, 0 failures, 1 users [SHARED]
  - Pool hdr_idx (416 bytes) : 46 allocated (19136 bytes), 4 used, 0 failures, 1 users [SHARED]
  - Pool dns_resolut (480 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users [SHARED]
  - Pool dns_answer_ (576 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users [SHARED]
  - Pool stream (848 bytes) : 47 allocated (39856 bytes), 5 used, 0 failures, 1 users [SHARED]
  - Pool requri (1024 bytes) : 10 allocated (10240 bytes), 0 used, 0 failures, 1 users [SHARED]
  - Pool trash (16400 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users
  - Pool buffer (16408 bytes) : 7 allocated (114856 bytes), 2 used, 0 failures, 1 users [SHARED]
Total: 17 pools, 238712 bytes allocated, 45968 used.

It works everywhere and you can use it anytime.

Interactive mode of operation

This mode of operation requires more work as Debian does not provide socat with readline support due to licensing restrictions.

READLINE
    Uses GNU readline and history on stdio to allow editing and reusing input lines (example).
    Due to licensing restrictions the readline feature is disabled in Debian.  See BUGS.
    You can use STDIO instead.
socat manual page

This will not work out of the box on Debian.

$ sudo -u haproxy socat readline /run/haproxy/user.sock
2020/01/12 02:51:38 socat[23784] E unknown device/address "readline"

Use rlwrap and simple expect trick to access HAProxy stats socket interactively.

To be exact, you do not need to use expect, but this is the way I use it.
$ expect -c 'log_user 0;spawn sudo rlwrap -l haproxy_stats_socket.log -S "[email protected]$(hostname)> " -c socat /run/haproxy/admin.sock stdio; send "prompt\n"; interact' 

prompt
[email protected]> prompt

[email protected]> show cli sockets
# socket lvl processes
/run/haproxy/admin.sock admin all
/run/haproxy/operator.sock operator all
/run/haproxy/user.sock user all

[email protected]> quit
[email protected]>
(process exited)

Inspect

$ cat haproxy_stats_socket.log 


[rlwrap] Sun Jan 12 02:34:19 2020

prompt

> show cli sockets
# socket lvl processes
/run/haproxy/admin.sock admin all
/run/haproxy/operator.sock operator all
/run/haproxy/user.sock user all

> quit

Examples

Display basic information.

$ echo "show info" | sudo -u haproxy socat stdio /run/haproxy/user.sock   
Name: HAProxy
Version: 1.8.19-1+deb10u1
Release_date: 2019/11/27
Nbthread: 1
Nbproc: 1
Process_num: 1
Pid: 23505
Uptime: 0d 0h53m14s
Uptime_sec: 3194
Memmax_MB: 0
PoolAlloc_MB: 0
PoolUsed_MB: 0
PoolFailed: 0
Ulimit-n: 4051
Maxsock: 4051
Maxconn: 2000
Hard_maxconn: 2000
CurrConns: 0
CumConns: 13
CumReq: 13
MaxSslConns: 0
CurrSslConns: 0
CumSslConns: 0
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 0
ConnRateLimit: 0
MaxConnRate: 0
SessRate: 0
SessRateLimit: 0
MaxSessRate: 0
SslRate: 0
SslRateLimit: 0
MaxSslRate: 0
SslFrontendKeyRate: 0
SslFrontendMaxKeyRate: 0
SslFrontendSessionReuse_pct: 0
SslBackendKeyRate: 0
SslBackendMaxKeyRate: 0
SslCacheLookups: 0
SslCacheMisses: 0
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
ZlibMemUsage: 0
MaxZlibMemUsage: 0
Tasks: 4
Run_queue: 1
Idle_pct: 100
node: debian
Stopping: 0
Jobs: 5
Listeners: 4

Display defined backends.

$ echo "show backend" | sudo -u haproxy socat stdio unix-connect:/var/run/haproxy/user.sock
# name
backend-local-blog
backend-local-gitlab
backend-local-netdata
backend-gitlab-ssh
no-match

Display servers in specific backend.

$ echo "show servers state backend-local-gitlab" | sudo -u haproxy socat stdio /var/run/haproxy/user.sock
1
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port
5 backend-local-gitlab 1 blog 172.16.4.4 2 0 1 1 6763 1 0 2 0 0 0 0 - 80

Display statistics.

$ echo "show stat" | sudo -u haproxy socat stdio unix-connect:/var/run/haproxy/admin.sock
# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses,
ssh-gitlab-frontend,FRONTEND,,,2,2,2000,29,22380,44059,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,1,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,tcp,,0,1,29,,0,0,
web-frontend,FRONTEND,,,6,46,2000,733,834153,14638693,0,0,0,,,,,OPEN,,,,,,,,,1,3,0,,,,0,0,0,11,,,,0,914,227,9,0,0,,0,26,1150,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,0,11,733,0,0,0,
backend-local-blog,gitlab,0,0,0,3,,567,321344,11708170,,0,,0,0,0,0,no check,1,1,0,,,6584,,,1,4,1,,567,,2,0,,11,,,,0,343,220,4,0,0,,,,,1,0,,,,,12,,,0,1,1,92,,,,,,,,,,,,172.16.40.1:80,,http,,,,,,,,
backend-local-blog,BACKEND,0,0,0,3,200,567,321344,11708170,0,0,,0,0,0,0,UP,1,1,0,,0,6584,0,,1,4,0,,567,,1,0,,11,,,,0,343,220,4,0,0,,,,567,1,0,0,0,0,0,12,,,0,1,1,92,,,,,,,,,,,,,,http,roundrobin,,,,,,,
backend-local-gitlab,blog,0,0,0,5,,17,14498,1114860,,0,,0,0,0,0,no check,1,1,0,,,6584,,,1,5,1,,17,,2,0,,8,,,,0,10,7,0,0,0,,,,,0,0,,,,,591,,,0,0,6,8,,,,,,,,,,,,172.16.34.34:80,,http,,,,,,,,
backend-local-gitlab,BACKEND,0,0,0,5,200,17,14498,1114860,0,0,,0,0,0,0,UP,1,1,0,,0,6584,0,,1,5,0,,17,,1,0,,8,,,,0,10,7,0,0,0,,,,17,0,0,0,0,0,0,591,,,0,0,6,8,,,,,,,,,,,,,,http,roundrobin,,,,,,,
backend-local-netdata,netdata,0,0,0,10,,565,497615,1815412,,0,,0,0,0,0,no check,1,1,0,,,6584,,,1,6,1,,565,,2,0,,26,,,,0,561,0,4,0,0,,,,,0,0,,,,,3,,,0,0,2,5478,,,,,,,,,,,,127.0.0.1:19999,,http,,,,,,,,
backend-local-netdata,BACKEND,0,0,0,10,200,565,497615,1815412,0,0,,0,0,0,0,UP,1,1,0,,0,6584,0,,1,6,0,,565,,1,0,,26,,,,0,561,0,4,0,0,,,,565,0,0,0,0,0,0,3,,,0,0,2,5478,,,,,,,,,,,,,,http,roundrobin,,,,,,,
backend-gitlab-ssh,netdata,0,0,2,2,,29,22380,44059,,0,,0,0,0,0,no check,1,1,0,,,6584,,,1,7,1,,29,,2,0,,1,,,,,,,,,,,,,,1,0,,,,,31,,,0,0,0,309,,,,,,,,,,,,172.16.34.34:22,,tcp,,,,,,,,
backend-gitlab-ssh,BACKEND,0,0,2,2,200,29,22380,44059,0,0,,0,0,0,0,UP,1,1,0,,0,6584,0,,1,7,0,,29,,1,0,,1,,,,,,,,,,,,,,1,0,0,0,0,0,31,,,0,0,0,309,,,,,,,,,,,,,,tcp,roundrobin,,,,,,,
no-match,BACKEND,0,0,0,0,200,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,6584,,,1,8,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,0,0,0,0,0,0,0,-1,,,0,0,0,0,,,,,,,,,,,,,,http,roundrobin,,,,,,,

Use awk to parse HAProxy statistics and extract currrent sessions.

$ echo "show stat" | sudo -u haproxy socat stdio unix-connect:/var/run/haproxy/admin.sock | awk -F, '$2=="FRONTEND" {print $1 " " $5}' 
ssh-gitlab-frontend 0
web-frontend 11

Display startup logs.

$ echo "show startup-logs" | sudo -u haproxy socat unix-connect:/var/run/haproxy/admin.sock stdio
[WARNING] 011/010936 (30023) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.