Note to documentation contributors :
    This document is formatted with 80 columns per line, with even number of
    spaces for indentation and without tabs. Please follow these rules strictly
    so that it remains easily printable everywhere. If you add sections, please
    update the summary below for easier searching.

In this document it is assumed that the reader has sufficient administration
skills on a UNIX-like operating system, uses the shell on a daily basis and is
familiar with troubleshooting utilities such as strace and tcpdump.

HAProxy is a multi-threaded, event-driven, non-blocking daemon. This means it
uses event multiplexing to schedule all of its activities instead of relying on
the system to schedule between multiple activities. Most of the time it runs as
a single process, so the output of "ps aux" on a system will report only one
"haproxy" process, unless a soft reload is in progress and an older process is
finishing its job in parallel to the new one. It is thus always easy to trace
its activity using the strace utility. In order to scale with the number of
available processors, by default haproxy will start one worker thread per
processor it is allowed to run on. Unless explicitly configured differently,
the incoming traffic is spread over all these threads, all running the same
event loop. A great care is taken to limit inter-thread dependencies to the
strict minimum, so as to try to achieve near-linear scalability. This has some
impacts such as the fact that a given connection is served by a single thread.
Thus in order to use all available processing capacity, it is needed to have at
least as many connections as there are threads, which is almost always granted.

HAProxy is designed to isolate itself into a chroot jail during startup, where
it cannot perform any file-system access at all. This is also true for the
libraries it depends on (eg: libc, libssl, etc). The immediate effect is that
a running process will not be able to reload a configuration file to apply
changes, instead a new process will be started using the updated configuration
file. Some other less obvious effects are that some timezone files or resolver
files the libc might attempt to access at run time will not be found, though
this should generally not happen as they're not needed after startup. A nice
consequence of this principle is that the HAProxy process is totally stateless,
and no cleanup is needed after it's killed, so any killing method that works
will do the right thing.

HAProxy doesn't write log files, but it relies on the standard syslog protocol
to send logs to a remote server (which is often located on the same system).

HAProxy uses its internal clock to enforce timeouts, that is derived from the
system's time but where unexpected drift is corrected. This is done by limiting
the time spent waiting in poll() for an event, and measuring the time it really
took. In practice it never waits more than one second. This explains why, when
running strace over a completely idle process, periodic calls to poll() (or any
of its variants) surrounded by two gettimeofday() calls are noticed. They are
normal, completely harmless and so cheap that the load they imply is totally
undetectable at the system scale, so there's nothing abnormal there. Example :

  16:35:40.002320 gettimeofday({1442759740, 2605}, NULL) = 0
  16:35:40.002942 epoll_wait(0, {}, 200, 1000) = 0
  16:35:41.007542 gettimeofday({1442759741, 7641}, NULL) = 0
  16:35:41.007998 gettimeofday({1442759741, 8114}, NULL) = 0
  16:35:41.008391 epoll_wait(0, {}, 200, 1000) = 0
  16:35:42.011313 gettimeofday({1442759742, 11411}, NULL) = 0

HAProxy is a TCP proxy, not a router. It deals with established connections that
have been validated by the kernel, and not with packets of any form nor with
sockets in other states (eg: no SYN_RECV nor TIME_WAIT), though their existence
may prevent it from binding a port. It relies on the system to accept incoming
connections and to initiate outgoing connections. An immediate effect of this is
that there is no relation between packets observed on the two sides of a
forwarded connection, which can be of different size, numbers and even family.
Since a connection may only be accepted from a socket in LISTEN state, all the
sockets it is listening to are necessarily visible using the "netstat" utility
to show listening sockets. Example :

  # netstat -ltnp
  Active Internet connections (only servers)
  Proto Recv-Q Send-Q Local Address   Foreign Address   State    PID/Program name
  tcp        0      0 0.0.0.0:22      0.0.0.0:*         LISTEN   1629/sshd
  tcp        0      0 0.0.0.0:80      0.0.0.0:*         LISTEN   2847/haproxy
  tcp        0      0 0.0.0.0:443     0.0.0.0:*         LISTEN   2847/haproxy

HAProxy is started by invoking the "haproxy" program with a number of arguments
passed on the command line. The actual syntax is :

  $ haproxy [<options>]*

followed by one of more letters, and possibly followed by one or multiple extra
arguments. Without any option, HAProxy displays the help page with a reminder
about supported options. Available options may vary slightly based on the
operating system. A fair number of these options overlap with an equivalent one
in the "global" section. In this case, the command line always has precedence
over the configuration file, so that the command line can be used to quickly
enforce some settings without touching the configuration files. The current
list of options is :

  -- <cfgfile>* : all the arguments following "--" are paths to configuration
    file/directory to be loaded and processed in the declaration order. It is
    mostly useful when relying on the shell to load many files that are
    numerically ordered. See also "-f". The difference between "--" and "-f" is
    that one "-f" must be placed before each file name, while a single "--" is
    needed before all file names. Both options can be used together, the
    command line ordering still applies. When more than one file is specified,
    each file must start on a section boundary, so the first keyword of each
    file must be one of "global", "defaults", "peers", "listen", "frontend",
    "backend", and so on. A file cannot contain just a server list for example.

  -f <cfgfile|cfgdir> : adds <cfgfile> to the list of configuration files to be
    loaded. If <cfgdir> is a directory, all the files (and only files) it
    contains are added in lexical order (using LC_COLLATE=C) to the list of
    configuration files to be loaded ; only files with ".cfg" extension are
    added, only non hidden files (not prefixed with ".") are added.
    Configuration files are loaded and processed in their declaration order.
    This option may be specified multiple times to load multiple files. See
    also "--". The difference between "--" and "-f" is that one "-f" must be
    placed before each file name, while a single "--" is needed before all file
    names. Both options can be used together, the command line ordering still
    applies. When more than one file is specified, each file must start on a
    section boundary, so the first keyword of each file must be one of
    "global", "defaults", "peers", "listen", "frontend", "backend", and so on.
    A file cannot contain just a server list for example.

  -C <dir> : changes to directory <dir> before loading configuration
    files. This is useful when using relative paths. Warning when using
    wildcards after "--" which are in fact replaced by the shell before
    starting haproxy.

  -D : start as a daemon. The process detaches from the current terminal after
    forking, and errors are not reported anymore in the terminal. It is
    equivalent to the "daemon" keyword in the "global" section of the
    configuration. It is recommended to always force it in any init script so
    that a faulty configuration doesn't prevent the system from booting.

  -L <name> : change the local peer name to <name>, which defaults to the local
    hostname. This is used only with peers replication. You can use the
    variable $HAPROXY_LOCALPEER in the configuration file to reference the
    peer name.

  -N <limit> : sets the default per-proxy maxconn to <limit> instead of the
    builtin default value (usually 2000). Only useful for debugging.

  -V : enable verbose mode (disables quiet mode). Reverts the effect of "-q" or
    "quiet".

  -W : master-worker mode. It is equivalent to the "master-worker" keyword in
    the "global" section of the configuration. This mode will launch a "master"
    which will monitor the "workers". Using this mode, you can reload HAProxy
    directly by sending a SIGUSR2 signal to the master.  The master-worker mode
    is compatible either with the foreground or daemon mode.  It is
    recommended to use this mode with multiprocess and systemd.

  -Ws : master-worker mode with support of `notify` type of systemd service.

  -4 : force DNS resolvers to query and accept IPv4 addresses only ("A"
       records). This can be used when facing difficulties in certain
       environments lacking end-to-end dual-stack connectivity. It overrides
       the global "dns-accept-family" directive and forces it to "ipv4".

  -c : only performs a check of the configuration files and exits before trying
    to bind. The exit status is zero if everything is OK, or non-zero if an
    error is encountered. Presence of warnings will be reported if any.
    By default this option does not report a success message. Combined with
    "-V" this will print the message "Configuration file is valid" upon
    success.

    Scripts must use the exit status to determine the success of the
    command.

  -cc : evaluates a condition as used within a conditional block of the
    configuration. The exit status is zero if the condition is true, 1 if the
    condition is false or 2 if an error is encountered.

  -d : enable debug mode. This disables daemon mode, forces the process to stay
    in foreground and to show incoming and outgoing events. It must never be
    used in an init script.

  -dA[file] : dump an archive of all dependencies detected at boot time in the
    designated file in tar format, immediately after the configuration is done
    loading. This is equivalent to "set-dumpable libs", but instead of keeping
    the libs in memory, it dumps them into a file. This may be used after a
    core dump, in order to provide all necessary libraries to developers to
    permit them to exploit the core. This may not be available on all operating
    systems. It is highly recommended to use this with the regular
    configuration files, and optionally with "-c" when used manually, to make
    haproxy immediately exit after the dump, without starting. Example:

       $ haproxy -dA/tmp/libs.tar -c -f /etc/haproxy/haproxy.cfg

  -dC[key] : dump the configuration file. It is performed after the lines are
    tokenized, so comments are stripped and indenting is forced. If a non-zero
    key is specified, lines are truncated before sensitive/confidential fields,
    and identifiers and addresses are emitted hashed with this key using the
    same algorithm as the one used by the anonymized mode on the CLI. This
    means that the output may safely be shared with a developer who needs it
    to figure what's happening in a dump that was anonymized using the same
    key. Please also see the CLI's "set anon" command.

  -dD : enable diagnostic mode. This mode will output extra warnings about
    suspicious configuration statements. This will never prevent startup even in
    "zero-warning" mode nor change the exit status code.

  -dF : disable data fast-forward. It is a mechanism to optimize the data
    forwarding by passing data directly from a side to the other one without
    waking the stream up. Thanks to this directive, it is possible to disable
    this optimization. Note it also disable any kernel tcp splicing. This
    command is not meant for regular use, it will generally only be suggested by
    developers along complex debugging sessions.

  -dG : disable use of getaddrinfo() to resolve host names into addresses. It
    can be used when suspecting that getaddrinfo() doesn't work as expected.
    This option was made available because many bogus implementations of
    getaddrinfo() exist on various systems and cause anomalies that are
    difficult to troubleshoot.

  -dI : enable the insecure fork. This is the equivalent of the
    "insecure-fork-wanted" in the global section. It can be useful when running
    all the reg-tests with ASAN which need to fork addr2line to resolve the
    addresses.

  -dK<class[,class]*> : dumps the list of registered keywords in each class.
    The list of classes is available with "-dKhelp". All classes may be dumped
    using "-dKall", otherwise a selection of those shown in the help can be
    specified as a comma-delimited list. The output format will vary depending
    on what class of keywords is being dumped (e.g. "cfg" will show the known
    configuration keywords in a format resembling the config file format while
    "smp" will show sample fetch functions prefixed with a compatibility matrix
    with each rule set). These may rarely be used as-is by humans but can be of
    great help for external tools that try to detect the appearance of new
    keywords at certain places to automatically update some documentation,
    syntax highlighting files, configuration parsers, API etc. The output
    format may evolve a bit over time so it is really recommended to use this
    output mostly to detect differences with previous archives. Note that not
    all keywords are listed because many keywords have existed long before the
    different keyword registration subsystems were created, and they do not
    appear there. However since new keywords are only added via the modern
    mechanisms, it's reasonably safe to assume that this output may be used to
    detect language additions with a good accuracy. The keywords are only
    dumped after the configuration is fully parsed, so that even dynamically
    created keywords can be dumped. A good way to dump and exit is to run a
    silent config check on an existing configuration:

       ./haproxy -dKall -q -c -f foo.cfg

    If no configuration file is available, using "-f /dev/null" will work as
    well to dump all default keywords, but then the return status will not be
    zero since there will be no listener, and will have to be ignored.

  -dL : dumps the list of dynamic shared libraries that are loaded at the end
   of the config processing. This will generally also include deep dependencies
   such as anything loaded from Lua code for example, as well as the executable
   itself. The list is printed in a format that ought to be easy enough to
   sanitize to directly produce a tarball of all dependencies. Since it doesn't
   stop the program's startup, it is recommended to only use it in combination
   with "-c" and "-q" where only the list of loaded objects will be displayed
   (or nothing in case of error). In addition, keep in mind that when providing
   such a package to help with a core file analysis, most libraries are in fact
   symbolic links that need to be dereferenced when creating the archive:

       ./haproxy -W -q -c -dL -f foo.cfg | tar -T - -hzcf archive.tgz

   When started in verbose mode (-V) the shared libraries' address ranges are
   also enumerated, unless the quiet mode is in use (-q).

  -dM[<byte>[,]][help|options,...] : forces memory poisoning, and/or changes
    memory other debugging options. Memory poisonning means that each and every
    memory region allocated with malloc() or pool_alloc() will be filled with
    <byte> before being passed to the caller. When <byte> is not specified, it
    defaults to 0x50 ('P'). While this slightly slows down operations, it is
    useful to reliably trigger issues resulting from missing initializations in
    the code that cause random crashes. Note that -dM0 has the effect of
    turning any malloc() into a calloc(). In any case if a bug appears or
    disappears when using this option it means there is a bug in haproxy, so
    please report it. A number of other options are available either alone or
    after a comma following the byte. The special option "help" will list the
    currently supported options and their current value. Each debugging option
    may be forced on or off. The most optimal options are usually chosen at
    build time based on the operating system and do not need to be adjusted,
    unless suggested by a developer. Supported debugging options include
    (set/clear):
      - fail / no-fail:
        This enables randomly failing memory allocations, in conjunction with
        the global "tune.fail-alloc" setting. This is used to detect missing
        error checks in the code. Setting the option presets the ratio to 1%
        failure rate.

      - no-merge / merge:
        By default, pools of very similar sizes are merged, resulting in more
        efficiency, but this complicates the analysis of certain memory dumps.
        This option allows to disable this mechanism, and may slightly increase
        the memory usage.

      - cold-first / hot-first:
        In order to optimize the CPU cache hit ratio, by default the most
        recently released objects ("hot") are recycled for new allocations.
        But doing so also complicates analysis of memory dumps and may hide
        use-after-free bugs. This option allows to instead pick the coldest
        objects first, which may result in a slight increase of CPU usage.

      - integrity / no-integrity:
        When this option is enabled, memory integrity checks are enabled on
        the allocated area to verify that it hasn't been modified since it was
        last released. This works best with "no-merge", "cold-first" and "tag".
        Enabling this option will slightly increase the CPU usage.

      - backup / no-backup:
        This option performs a copy of each released object at release time,
        allowing developers to inspect them. It also performs a comparison at
        allocation time to detect if anything changed in between, indicating a
        use-after-free condition. This doubles the memory usage and slightly
        increases the CPU usage (similar to "integrity"). If combined with
        "integrity", it still duplicates the contents but doesn't perform the
        comparison (which is performed by "integrity"). Just like "integrity",
        it works best with "no-merge", "cold-first" and "tag".

      - no-global / global:
        Depending on the operating system, a process-wide global memory cache
        may be enabled if it is estimated that the standard allocator is too
        slow or inefficient with threads. This option allows to forcefully
        disable it or enable it. Disabling it may result in a CPU usage
        increase with inefficient allocators. Enabling it may result in a
        higher memory usage with efficient allocators.

      - no-cache / cache:
        Each thread uses a very fast local object cache for allocations, which
        is always enabled by default. This option allows to disable it. Since
        the global cache also passes via the local caches, this will
        effectively result in disabling all caches and allocating directly from
        the default allocator. This may result in a significant increase of CPU
        usage, but may also result in small memory savings on tiny systems.

      - caller / no-caller:
        Enabling this option reserves some extra space in each allocated object
        to store the address of the last caller that allocated or released it.
        This helps developers go back in time when analysing memory dumps and
        to guess how something unexpected happened.

      - tag / no-tag:
        Enabling this option reserves some extra space in each allocated object
        to store a tag that allows to detect bugs such as double-free, freeing
        an invalid object, and buffer overflows. It offers much stronger
        reliability guarantees at the expense of 4 or 8 extra bytes per
        allocation. It usually is the first step to detect memory corruption.

      - poison / no-poison:
        Enabling this option will fill allocated objects with a fixed pattern
        that will make sure that some accidental values such as 0 will not be
        present if a newly added field was mistakenly forgotten in an
        initialization routine. Such bugs tend to rarely reproduce, especially
        when pools are not merged. This is normally enabled by directly passing
        the byte's value to -dM but using this option allows to disable/enable
        use of a previously set value.

  -dR : disable SO_REUSEPORT socket option on listening ports. It is equivalent
    to the "global" section's "noreuseport" keyword. This may be applied in
    multi-threading scenarios, when load distribution issues observed among the
    haproxy threads (could be monitored with top).

  -dS : disable use of the splice() system call. It is equivalent to the
    "global" section's "nosplice" keyword. This may be used when splice() is
    suspected to behave improperly or to cause performance issues, or when
    using strace to see the forwarded data (which do not appear when using
    splice()).

  -dT : disable the use of ktls. It is equivalent to the "global" section's
   keyword "noktls". It is mostly useful when suspecting a bug related to
   ktls.

  -dV : disable SSL verify on the server side. It is equivalent to having
    "ssl-server-verify none" in the "global" section. This is useful when
    trying to reproduce production issues out of the production
    environment. Never use this in an init script as it degrades SSL security
    to the servers.

  -dW : if set, haproxy will refuse to start if any warning was emitted while
   processing the configuration. This helps detect subtle mistakes and keep the
   configuration clean and portable across versions. It is recommended to set
   this option in service scripts when configurations are managed by humans,
   but it is recommended not to use it with generated configurations, which
   tend to emit more warnings. It may be combined with "-c" to cause warnings
   in checked configurations to fail. This is equivalent to global option
   "zero-warning".

  -dZ : disable forwarding of data in "zero-copy" mode. It is equivalent to the
   "global" section's "tune.disable-zero-copy-forwarding" keyword. This may be
   helpful in case of issues with data loss or data integrity, or when using
   strace to see the forwarded data, as it also disables any kernel tcp
   splicing.

  -db : disable background mode and multi-process mode. The process remains in
    foreground. It is mainly used during development or during small tests, as
    Ctrl-C is enough to stop the process. Never use it in an init script.

  -dc : enable CPU affinity debugging. The list of selected and evicted CPUs as
    well as their topology will be reported before starting.

  -de : disable the use of the "epoll" poller. It is equivalent to the "global"
    section's keyword "noepoll". It is mostly useful when suspecting a bug
    related to this poller. On systems supporting epoll, the fallback will
    generally be the "poll" poller.

  -dk : disable the use of the "kqueue" poller. It is equivalent to the
    "global" section's keyword "nokqueue". It is mostly useful when suspecting
    a bug related to this poller. On systems supporting kqueue, the fallback
    will generally be the "poll" poller.

  -dp : disable the use of the "poll" poller. It is equivalent to the "global"
    section's keyword "nopoll". It is mostly useful when suspecting a bug
    related to this poller. On systems supporting poll, the fallback will
    generally be the "select" poller, which cannot be disabled and is limited
    to 1024 file descriptors.

  -dr : ignore server address resolution failures. It is very common when
    validating a configuration out of production not to have access to the same
    resolvers and to fail on server address resolution, making it difficult to
    test a configuration. This option simply appends the "none" method to the
    list of address resolution methods for all servers, ensuring that even if
    the libc fails to resolve an address, the startup sequence is not
    interrupted.

  -dt [<trace_desc>,...] : activates traces on stderr. Without argument, this
    enables all trace sources on error level. This can notably be useful to
    detect protocol violations from clients or servers. An optional argument
    can be used to specify a list of various trace configurations using ',' as
    separator. Each element activates one or all trace sources. Additionally,
    level and verbosity can be optionally specified on each element using ':'
    as inner separator with trace name. When entering an invalid verbosity or
    level name, the list of available keywords is presented. For example it can
    be convenient to pass 'help' for each field to consult the list first.

  -dv : disable the use of the "evports" poller. It is equivalent to the
    "global" section's keyword "noevports". It is mostly useful when suspecting
    a bug related to this poller. On systems supporting event ports (SunOS
    derived from Solaris 10 and later), the fallback will generally be the
    "poll" poller.

  -m <limit> : limit allocatable memory, which is used to keep process's data,
    to <limit> megabytes. This may cause some connection refusals or some
    slowdowns depending on the amount of memory needed for normal operations.
    This is mostly used to force haproxy process to work in a constrained
    resource consumption scenario. It is important to note that the memory is
    not shared between haproxy processes and a child process created via fork()
    system call inherits its parent's resource limits. So, in a master-worker
    mode this memory limit is separately applied to the master and its forked
    worker process.

  -n <limit> : limits the per-process connection limit to <limit>. This is
    equivalent to the global section's keyword "maxconn". It has precedence
    over this keyword. This may be used to quickly force lower limits to avoid
    a service outage on systems where resource limits are too low.

  -p <file> : write all processes' pids into <file> during startup. This is
    equivalent to the "global" section's keyword "pidfile". The file is opened
    before entering the chroot jail, and after doing the chdir() implied by
    "-C". Each pid appears on its own line.

  -q : set "quiet" mode. This disables the output messages. It can be used in
    combination with "-c" to just check if a configuration file is valid or not.

  -S <bind>[,bind_options...]: in master-worker mode, bind a master CLI, which
    allows the access to every processes, running or leaving ones.
    For security reasons, it is recommended to bind the master CLI to a local
    UNIX socket. The bind options are the same as the keyword "bind" in
    the configuration file with words separated by commas instead of spaces.

    Note that this socket can't be used to retrieve the listening sockets from
    an old process during a seamless reload.

  -sf <pid>* : send the "finish" signal (SIGUSR1) to older processes after boot
    completion to ask them to finish what they are doing and to leave. <pid>
    is a list of pids to signal (one per argument). The list ends on any
    option starting with a "-". It is not a problem if the list of pids is
    empty, so that it can be built on the fly based on the result of a command
    like "pidof" or "pgrep".

  -st <pid>* : send the "terminate" signal (SIGTERM) to older processes after
    boot completion to terminate them immediately without finishing what they
    were doing. <pid> is a list of pids to signal (one per argument). The list
    ends on any option starting with a "-". It is not a problem if the list
    of pids is empty, so that it can be built on the fly based on the result of
    a command like "pidof" or "pgrep".

  -v : report the version and build date.

  -vv : display the version, build options, libraries versions and usable
    pollers. This output is systematically requested when filing a bug report.

  -x <unix_socket> : connect to the specified socket and try to retrieve any
    listening sockets from the old process, and use them instead of trying to
    bind new ones. This is useful to avoid missing any new connection when
    reloading the configuration on Linux.

    Without master-worker mode, the capability must be enable on the stats
    socket using "expose-fd listeners" in your configuration.

    In master-worker mode, it does not need "expose-fd listeners", the master
    will use automatically this option upon a reload with the "sockpair@"
    syntax, which allows the master to connect directly to a worker without using
    any stats socket declared in the configuration. If you want to disable this,
    you can pass -x /dev/null.

A safe way to start HAProxy from an init file consists in forcing the daemon
mode, storing existing pids to a pid file and using this pid file to notify
older processes to finish before leaving :

   haproxy -f /etc/haproxy.cfg \
           -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

When the configuration is split into a few specific files (eg: tcp vs http),
it is recommended to use the "-f" option :

   haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
           -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
           -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
           -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

When an unknown number of files is expected, such as customer-specific files,
it is recommended to assign them a name starting with a fixed-size sequence
number and to use "--" to load them, possibly after loading some defaults :

   haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
           -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
           -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
           -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) \
           -f /etc/haproxy/default-customers.cfg -- /etc/haproxy/customers/*

Sometimes a failure to start may happen for whatever reason. Then it is
important to verify if the version of HAProxy you are invoking is the expected
version and if it supports the features you are expecting (eg: SSL, PCRE,
compression, Lua, etc). This can be verified using "haproxy -vv". Some
important information such as certain build options, the target system and
the versions of the libraries being used are reported there. It is also what
you will systematically be asked for when posting a bug report :

  $ haproxy -vv
  HAProxy version 1.6-dev7-a088d3-4 2015/10/08
  Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>

  Build options :
    TARGET  = linux2628
    CPU     = generic
    CC      = gcc
    CFLAGS  = -pg -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement \
              -DBUFSIZE=8030 -DMAXREWRITE=1030 -DSO_MARK=36 -DTCP_REPAIR=19
    OPTIONS = USE_ZLIB=1 USE_DLMALLOC=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

  Default settings :
    maxconn = 2000, bufsize = 8030, maxrewrite = 1030, maxpollevents = 200

  Encrypted password support via crypt(3): yes
  Built with zlib version : 1.2.6
  Compression algorithms supported : identity("identity"), deflate("deflate"), \
                                     raw-deflate("deflate"), gzip("gzip")
  Built with OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
  Running on OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
  OpenSSL library supports TLS extensions : yes
  OpenSSL library supports SNI : yes
  OpenSSL library supports prefer-server-ciphers : yes
  Built with PCRE version : 8.12 2011-01-15
  PCRE library supports JIT : no (USE_PCRE_JIT not set)
  Built with Lua version : Lua 5.3.1
  Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND

  Available polling systems :
        epoll : pref=300,  test result OK
         poll : pref=200,  test result OK
       select : pref=150,  test result OK
  Total: 3 (3 usable), will use epoll.

The relevant information that many non-developer users can verify here are :
  - the version : 1.6-dev7-a088d3-4 above means the code is currently at commit
    ID "a088d3" which is the 4th one after after official version "1.6-dev7".
    Version 1.6-dev7 would show as "1.6-dev7-8c1ad7". What matters here is in
    fact "1.6-dev7". This is the 7th development version of what will become
    version 1.6 in the future. A development version not suitable for use in
    production (unless you know exactly what you are doing). A stable version
    will show as a 3-numbers version, such as "1.5.14-16f863", indicating the
    14th level of fix on top of version 1.5. This is a production-ready version.

  - the release date : 2015/10/08. It is represented in the universal
    year/month/day format. Here this means August 8th, 2015. Given that stable
    releases are issued every few months (1-2 months at the beginning, sometimes
    6 months once the product becomes very stable), if you're seeing an old date
    here, it means you're probably affected by a number of bugs or security
    issues that have since been fixed and that it might be worth checking on the
    official site.

  - build options : they are relevant to people who build their packages
    themselves, they can explain why things are not behaving as expected. For
    example the development version above was built for Linux 2.6.28 or later,
    targeting a generic CPU (no CPU-specific optimizations), and lacks any
    code optimization (-O0) so it will perform poorly in terms of performance.

  - libraries versions : zlib version is reported as found in the library
    itself. In general zlib is considered a very stable product and upgrades
    are almost never needed. OpenSSL reports two versions, the version used at
    build time and the one being used, as found on the system. These ones may
    differ by the last letter but never by the numbers. The build date is also
    reported because most OpenSSL bugs are security issues and need to be taken
    seriously, so this library absolutely needs to be kept up to date. Seeing a
    4-months old version here is highly suspicious and indeed an update was
    missed. PCRE provides very fast regular expressions and is highly
    recommended. Certain of its extensions such as JIT are not present in all
    versions and still young so some people prefer not to build with them,
    which is why the build status is reported as well. Regarding the Lua
    scripting language, HAProxy expects version 5.3 which is very young since
    it was released a little time before HAProxy 1.6. It is important to check
    on the Lua web site if some fixes are proposed for this branch.

  - Available polling systems will affect the process's scalability when
    dealing with more than about one thousand of concurrent connections. These
    ones are only available when the correct system was indicated in the TARGET
    variable during the build. The "epoll" mechanism is highly recommended on
    Linux, and the kqueue mechanism is highly recommended on BSD. Lacking them
    will result in poll() or even select() being used, causing a high CPU usage
    when dealing with a lot of connections.

HAProxy supports a graceful and a hard stop. The hard stop is simple, when the
SIGTERM signal is sent to the haproxy process, it immediately quits and all
established connections are closed. The graceful stop is triggered when the
SIGUSR1 signal is sent to the haproxy process. It consists in only unbinding
from listening ports, but continue to process existing connections until they
close. Once the last connection is closed, the process leaves.

The hard stop method is used for the "stop" or "restart" actions of the service
management script. The graceful stop is used for the "reload" action which
tries to seamlessly reload a new configuration in a new process.

Both of these signals may be sent by the new haproxy process itself during a
reload or restart, so that they are sent at the latest possible moment and only
if absolutely required. This is what is performed by the "-st" (hard) and "-sf"
(graceful) options respectively.

In master-worker mode, it is not needed to start a new haproxy process in
order to reload the configuration. The master process reacts to the SIGUSR2
signal by reexecuting itself with the -sf parameter followed by the PIDs of
the workers. The master will then parse the configuration file and fork new
workers.

To understand better how these signals are used, it is important to understand
the whole restart mechanism.

First, an existing haproxy process is running. The administrator uses a system
specific command such as "/etc/init.d/haproxy reload" to indicate they want to
take the new configuration file into effect. What happens then is the following.
First, the service script (/etc/init.d/haproxy or equivalent) will verify that
the configuration file parses correctly using "haproxy -c". After that it will
try to start haproxy with this configuration file, using "-st" or "-sf".

Then HAProxy tries to bind to all listening ports. If some fatal errors happen
(eg: address not present on the system, permission denied), the process quits
with an error. If a socket binding fails because a port is already in use, then
the process will first send a SIGTTOU signal to all the pids specified in the
"-st" or "-sf" pid list. This is what is called the "pause" signal. It instructs
all existing haproxy processes to temporarily stop listening to their ports so
that the new process can try to bind again. During this time, the old process
continues to process existing connections. If the binding still fails (because
for example a port is shared with another daemon), then the new process sends a
SIGTTIN signal to the old processes to instruct them to resume operations just
as if nothing happened. The old processes will then restart listening to the
ports and continue to accept connections. Note that this mechanism is system
dependent and some operating systems may not support it in multi-process mode.

If the new process manages to bind correctly to all ports, then it sends either
the SIGTERM (hard stop in case of "-st") or the SIGUSR1 (graceful stop in case
of "-sf") to all processes to notify them that it is now in charge of operations
and that the old processes will have to leave, either immediately or once they
have finished their job.

It is important to note that during this timeframe, there are two small windows
of a few milliseconds each where it is possible that a few connection failures
will be noticed during high loads. Typically observed failure rates are around
1 failure during a reload operation every 10000 new connections per second,
which means that a heavily loaded site running at 30000 new connections per
second may see about 3 failed connection upon every reload. The two situations
where this happens are :

  - if the new process fails to bind due to the presence of the old process,
    it will first have to go through the SIGTTOU+SIGTTIN sequence, which
    typically lasts about one millisecond for a few tens of frontends, and
    during which some ports will not be bound to the old process and not yet
    bound to the new one. HAProxy works around this on systems that support the
    SO_REUSEPORT socket options, as it allows the new process to bind without
    first asking the old one to unbind. Most BSD systems have been supporting
    this almost forever. Linux has been supporting this in version 2.0 and
    dropped it around 2.2, but some patches were floating around by then. It
    was reintroduced in kernel 3.9, so if you are observing a connection
    failure rate above the one mentioned above, please ensure that your kernel
    is 3.9 or newer, or that relevant patches were backported to your kernel
    (less likely).

  - when the old processes close the listening ports, the kernel may not always
    redistribute any pending connection that was remaining in the socket's
    backlog. Under high loads, a SYN packet may happen just before the socket
    is closed, and will lead to an RST packet being sent to the client. In some
    critical environments where even one drop is not acceptable, these ones are
    sometimes dealt with using firewall rules to block SYN packets during the
    reload, forcing the client to retransmit. This is totally system-dependent,
    as some systems might be able to visit other listening queues and avoid
    this RST. A second case concerns the ACK from the client on a local socket
    that was in SYN_RECV state just before the close. This ACK will lead to an
    RST packet while the haproxy process is still not aware of it. This one is
    harder to get rid of, though the firewall filtering rules mentioned above
    will work well if applied one second or so before restarting the process.

For the vast majority of users, such drops will never ever happen since they
don't have enough load to trigger the race conditions. And for most high traffic
users, the failure rate is still fairly within the noise margin provided that at
least SO_REUSEPORT is properly supported on their systems.

In order to ensure that all incoming connections will successfully be served,
HAProxy computes at load time the total number of file descriptors that will be
needed during the process's life. A regular Unix process is generally granted
1024 file descriptors by default, and a privileged process can raise this limit
itself. This is one reason for starting HAProxy as root and letting it adjust
the limit. The default limit of 1024 file descriptors roughly allow about 500
concurrent connections to be processed. The computation is based on the global
maxconn parameter which limits the total number of connections per process, the
number of listeners, the number of servers which have a health check enabled,
the agent checks, the peers, the loggers and possibly a few other technical
requirements. A simple rough estimate of this number consists in simply
doubling the maxconn value and adding a few tens to get the approximate number
of file descriptors needed.

Originally HAProxy did not know how to compute this value, and it was necessary
to pass the value using the "ulimit-n" setting in the global section. This
explains why even today a lot of configurations are seen with this setting
present. Unfortunately it was often miscalculated resulting in connection
failures when approaching maxconn instead of throttling incoming connection
while waiting for the needed resources. For this reason it is important to
remove any vestigial "ulimit-n" setting that can remain from very old versions.

Raising the number of file descriptors to accept even moderate loads is
mandatory but comes with some OS-specific adjustments. First, the select()
polling system is limited to 1024 file descriptors. In fact on Linux it used
to be capable of handling more but since certain OS ship with excessively
restrictive SELinux policies forbidding the use of select() with more than
1024 file descriptors, HAProxy now refuses to start in this case in order to
avoid any issue at run time. On all supported operating systems, poll() is
available and will not suffer from this limitation. It is automatically picked
so there is nothing to do to get a working configuration. But poll's becomes
very slow when the number of file descriptors increases. While HAProxy does its
best to limit this performance impact (eg: via the use of the internal file
descriptor cache and batched processing), a good rule of thumb is that using
poll() with more than a thousand concurrent connections will use a lot of CPU.

For Linux systems base on kernels 2.6 and above, the epoll() system call will
be used. It's a much more scalable mechanism relying on callbacks in the kernel
that guarantee a constant wake up time regardless of the number of registered
monitored file descriptors. It is automatically used where detected, provided
that HAProxy had been built for one of the Linux flavors. Its presence and
support can be verified using "haproxy -vv".

For BSD systems which support it, kqueue() is available as an alternative. It
is much faster than poll() and even slightly faster than epoll() thanks to its
batched handling of changes. At least FreeBSD and OpenBSD support it. Just like
with Linux's epoll(), its support and availability are reported in the output
of "haproxy -vv".

Having a good poller is one thing, but it is mandatory that the process can
reach the limits. When HAProxy starts, it immediately sets the new process's
file descriptor limits and verifies if it succeeds. In case of failure, it
reports it before forking so that the administrator can see the problem. As
long as the process is started by as root, there should be no reason for this
setting to fail. However, it can fail if the process is started by an
unprivileged user. If there is a compelling reason for *not* starting haproxy
as root (eg: started by end users, or by a per-application account), then the
file descriptor limit can be raised by the system administrator for this
specific user. The effectiveness of the setting can be verified by issuing
"ulimit -n" from the user's command line. It should reflect the new limit.

Warning: when an unprivileged user's limits are changed in this user's account,
it is fairly common that these values are only considered when the user logs in
and not at all in some scripts run at system boot time nor in crontabs. This is
totally dependent on the operating system, keep in mind to check "ulimit -n"
before starting haproxy when running this way. The general advice is never to
start haproxy as an unprivileged user for production purposes. Another good
reason is that it prevents haproxy from enabling some security protections.

Once it is certain that the system will allow the haproxy process to use the
requested number of file descriptors, two new system-specific limits may be
encountered. The first one is the system-wide file descriptor limit, which is
the total number of file descriptors opened on the system, covering all
processes. When this limit is reached, accept() or socket() will typically
return ENFILE. The second one is the per-process hard limit on the number of
file descriptors, it prevents setrlimit() from being set higher. Both are very
dependent on the operating system. On Linux, the system limit is set at boot
based on the amount of memory. It can be changed with the "fs.file-max" sysctl.
And the per-process hard limit is set to 1048576 by default, but it can be
changed using the "fs.nr_open" sysctl.

File descriptor limitations may be observed on a running process when they are
set too low. The strace utility will report that accept() and socket() return
"-1 EMFILE" when the process's limits have been reached. In this case, simply
raising the "ulimit-n" value (or removing it) will solve the problem. If these
system calls return "-1 ENFILE" then it means that the kernel's limits have
been reached and that something must be done on a system-wide parameter. These
trouble must absolutely be addressed, as they result in high CPU usage (when
accept() fails) and failed connections that are generally visible to the user.
One solution also consists in lowering the global maxconn value to enforce
serialization, and possibly to disable HTTP keep-alive to force connections
to be released and reused faster.

HAProxy uses a simple and fast pool-based memory management. Since it relies on
a small number of different object types, it's much more efficient to pick new
objects from a pool which already contains objects of the appropriate size than
to call malloc() for each different size. The pools are organized as a stack or
LIFO, so that newly allocated objects are taken from recently released objects
still hot in the CPU caches. Pools of similar sizes are merged together, in
order to limit memory fragmentation.

By default, since the focus is set on performance, each released object is put
back into the pool it came from, and allocated objects are never freed since
they are expected to be reused very soon.

On the CLI, it is possible to check how memory is being used in pools thanks to
the "show pools" command :

  > show pools
  Dumping pools usage. Use SIGQUIT to flush them.
    - Pool cache_st (16 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccc40=03 [SHARED]
    - Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 0 failures, 2 users, @0x9ccac0=00 [SHARED]
    - Pool comp_state (48 bytes) : 3 allocated (144 bytes), 3 used, 0 failures, 5 users, @0x9cccc0=04 [SHARED]
    - Pool filter (64 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 3 users, @0x9ccbc0=02 [SHARED]
    - Pool vars (80 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccb40=01 [SHARED]
    - Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9cd240=15 [SHARED]
    - Pool task (144 bytes) : 55 allocated (7920 bytes), 55 used, 0 failures, 1 users, @0x9cd040=11 [SHARED]
    - Pool session (160 bytes) : 1 allocated (160 bytes), 1 used, 0 failures, 1 users, @0x9cd140=13 [SHARED]
    - Pool h2s (208 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccec0=08 [SHARED]
    - Pool h2c (288 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cce40=07 [SHARED]
    - Pool spoe_ctx (304 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccf40=09 [SHARED]
    - Pool connection (400 bytes) : 2 allocated (800 bytes), 2 used, 0 failures, 1 users, @0x9cd1c0=14 [SHARED]
    - Pool hdr_idx (416 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cd340=17 [SHARED]
    - Pool dns_resolut (480 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccdc0=06 [SHARED]
    - Pool dns_answer_ (576 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccd40=05 [SHARED]
    - Pool stream (960 bytes) : 1 allocated (960 bytes), 1 used, 0 failures, 1 users, @0x9cd0c0=12 [SHARED]
    - Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cd2c0=16 [SHARED]
    - Pool buffer (8030 bytes) : 3 allocated (24090 bytes), 2 used, 0 failures, 1 users, @0x9cd3c0=18 [SHARED]
    - Pool trash (8062 bytes) : 1 allocated (8062 bytes), 1 used, 0 failures, 1 users, @0x9cd440=19
  Total: 19 pools, 42296 bytes allocated, 34266 used.

The pool name is only indicative, it's the name of the first object type using
this pool. The size in parenthesis is the object size for objects in this pool.
Object sizes are always rounded up to the closest multiple of 16 bytes. The
number of objects currently allocated and the equivalent number of bytes is
reported so that it is easy to know which pool is responsible for the highest
memory usage. The number of objects currently in use is reported as well in the
"used" field. The difference between "allocated" and "used" corresponds to the
objects that have been freed and are available for immediate use. The address
at the end of the line is the pool's address, and the following number is the
pool index when it exists, or is reported as -1 if no index was assigned.

It is possible to limit the amount of memory allocated per process using the
"-m" command line option, followed by a number of megabytes. It covers all of
the process's addressable space, so that includes memory used by some libraries
as well as the stack, but it is a reliable limit when building a resource
constrained system. It works the same way as "ulimit -v" on systems which have
it, or "ulimit -d" for the other ones.

If a memory allocation fails due to the memory limit being reached or because
the system doesn't have any enough memory, then haproxy will first start to
free all available objects from all pools before attempting to allocate memory
again. This mechanism of releasing unused memory can be triggered by sending
the signal SIGQUIT to the haproxy process.

During a reload operation, the process switched to the graceful stop state also
automatically performs some flushes after releasing any connection so that all
possible memory is released to save it for the new process.

HAProxy normally spends most of its time in the system and a smaller part in
userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end
connection setups and closes per second at 100% CPU on a single core. When one
core is saturated, typical figures are :
  - 95% system, 5% user for long TCP connections or large HTTP objects
  - 85% system and 15% user for short TCP connections or small HTTP objects in
    close mode
  - 70% system and 30% user for small HTTP objects in keep-alive mode

The amount of rules processing and regular expressions will increase the user
land part. The presence of firewall rules, connection tracking, complex routing
tables in the system will instead increase the system part.

On most systems, the CPU time observed during network transfers can be cut in 4
parts :
  - the interrupt part, which concerns all the processing performed upon I/O
    receipt, before the target process is even known. Typically Rx packets are
    accounted for in interrupt. On some systems such as Linux where interrupt
    processing may be deferred to a dedicated thread, it can appear as softirq,
    and the thread is called ksoftirqd/0 (for CPU 0). The CPU taking care of
    this load is generally defined by the hardware settings, though in the case
    of softirq it is often possible to remap the processing to another CPU.
    This interrupt part will often be perceived as parasitic since it's not
    associated with any process, but it actually is some processing being done
    to prepare the work for the process.

  - the system part, which concerns all the processing done using kernel code
    called from userland. System calls are accounted as system for example. All
    synchronously delivered Tx packets will be accounted for as system time. If
    some packets have to be deferred due to queues filling up, they may then be
    processed in interrupt context later (eg: upon receipt of an ACK opening a
    TCP window).

  - the user part, which exclusively runs application code in userland. HAProxy
    runs exclusively in this part, though it makes heavy use of system calls.
    Rules processing, regular expressions, compression, encryption all add to
    the user portion of CPU consumption.

  - the idle part, which is what the CPU does when there is nothing to do. For
    example HAProxy waits for an incoming connection, or waits for some data to
    leave, meaning the system is waiting for an ACK from the client to push
    these data.

In practice regarding HAProxy's activity, it is in general reasonably accurate
(but totally inexact) to consider that interrupt/softirq are caused by Rx
processing in kernel drivers, that user-land is caused by layer 7 processing
in HAProxy, and that system time is caused by network processing on the Tx
path.

Since HAProxy runs around an event loop, it waits for new events using poll()
(or any alternative) and processes all these events as fast as possible before
going back to poll() waiting for new events. It measures the time spent waiting
in poll() compared to the time spent doing processing events. The ratio of
polling time vs total time is called the "idle" time, it's the amount of time
spent waiting for something to happen. This ratio is reported in the stats page
on the "idle" line, or "Idle_pct" on the CLI. When it's close to 100%, it means
the load is extremely low. When it's close to 0%, it means that there is
constantly some activity. While it cannot be very accurate on an overloaded
system due to other processes possibly preempting the CPU from the haproxy
process, it still provides a good estimate about how HAProxy considers it is
working : if the load is low and the idle ratio is low as well, it may indicate
that HAProxy has a lot of work to do, possibly due to very expensive rules that
have to be processed. Conversely, if HAProxy indicates the idle is close to
100% while things are slow, it means that it cannot do anything to speed things
up because it is already waiting for incoming data to process. In the example
below, haproxy is completely idle :

  $ echo "show info" | socat - /var/run/haproxy.sock | grep ^Idle
  Idle_pct: 100

When the idle ratio starts to become very low, it is important to tune the
system and place processes and interrupts correctly to save the most possible
CPU resources for all tasks. If a firewall is present, it may be worth trying
to disable it or to tune it to ensure it is not responsible for a large part
of the performance limitation. It's worth noting that unloading a stateful
firewall generally reduces both the amount of interrupt/softirq and of system
usage since such firewalls act both on the Rx and the Tx paths. On Linux,
unloading the nf_conntrack and ip_conntrack modules will show whether there is
anything to gain. If so, then the module runs with default settings and you'll
have to figure how to tune it for better performance. In general this consists
in considerably increasing the hash table size. On FreeBSD, "pfctl -d" will
disable the "pf" firewall and its stateful engine at the same time.

If it is observed that a lot of time is spent in interrupt/softirq, it is
important to ensure that they don't run on the same CPU. Most systems tend to
pin the tasks on the CPU where they receive the network traffic because for
certain workloads it improves things. But with heavily network-bound workloads
it is the opposite as the haproxy process will have to fight against its kernel
counterpart. Pinning haproxy to one CPU core and the interrupts to another one,
all sharing the same L3 cache tends to sensibly increase network performance
because in practice the amount of work for haproxy and the network stack are
quite close, so they can almost fill an entire CPU each. On Linux this is done
using taskset (for haproxy) or using cpu-map (from the haproxy config), and the
interrupts are assigned under /proc/irq. Many network interfaces support
multiple queues and multiple interrupts. In general it helps to spread them
across a small number of CPU cores provided they all share the same L3 cache.
Please always stop irq_balance which always does the worst possible thing on
such workloads.

For CPU-bound workloads consisting in a lot of SSL traffic or a lot of
compression, it may be worth using multiple processes dedicated to certain
tasks, though there is no universal rule here and experimentation will have to
be performed.

In order to increase the CPU capacity, it is possible to make HAProxy run as
several processes, using the "nbproc" directive in the global section. There
are some limitations though :
  - health checks are run per process, so the target servers will get as many
    checks as there are running processes ;
  - maxconn values and queues are per-process so the correct value must be set
    to avoid overloading the servers ;
  - outgoing connections should avoid using port ranges to avoid conflicts
  - stick-tables are per process and are not shared between processes ;
  - each peers section may only run on a single process at a time ;
  - the CLI operations will only act on a single process at a time.

With this in mind, it appears that the easiest setup often consists in having
one first layer running on multiple processes and in charge for the heavy
processing, passing the traffic to a second layer running in a single process.
This mechanism is suited to SSL and compression which are the two CPU-heavy
features. Instances can easily be chained over UNIX sockets (which are cheaper
than TCP sockets and which do not waste ports), and the proxy protocol which is
useful to pass client information to the next stage. When doing so, it is
generally a good idea to bind all the single-process tasks to process number 1
and extra tasks to next processes, as this will make it easier to generate
similar configurations for different machines.

On Linux versions 3.9 and above, running HAProxy in multi-process mode is much
more efficient when each process uses a distinct listening socket on the same
IP:port ; this will make the kernel evenly distribute the load across all
processes instead of waking them all up. Please check the "process" option of
the "bind" keyword lines in the configuration manual for more information.

For logging, HAProxy always relies on a syslog server since it does not perform
any file-system access. The standard way of using it is to send logs over UDP
to the log server (by default on port 514). Very commonly this is configured to
127.0.0.1 where the local syslog daemon is running, but it's also used over the
network to log to a central server. The central server provides additional
benefits especially in active-active scenarios where it is desirable to keep
the logs merged in arrival order. HAProxy may also make use of a UNIX socket to
send its logs to the local syslog daemon, but it is not recommended at all,
because if the syslog server is restarted while haproxy runs, the socket will
be replaced and new logs will be lost. Since HAProxy will be isolated inside a
chroot jail, it will not have the ability to reconnect to the new socket. It
has also been observed in field that the log buffers in use on UNIX sockets are
very small and lead to lost messages even at very light loads. But this can be
fine for testing however.

It is recommended to add the following directive to the "global" section to
make HAProxy log to the local daemon using facility "local0" :

      log 127.0.0.1:514 local0

and then to add the following one to each "defaults" section or to each frontend
and backend section :

      log global

This way, all logs will be centralized through the global definition of where
the log server is.

Some syslog daemons do not listen to UDP traffic by default, so depending on
the daemon being used, the syntax to enable this will vary :

  - on sysklogd, you need to pass argument "-r" on the daemon's command line
    so that it listens to a UDP socket for "remote" logs ; note that there is
    no way to limit it to address 127.0.0.1 so it will also receive logs from
    remote systems ;

  - on rsyslogd, the following lines must be added to the configuration file :

      $ModLoad imudp
      $UDPServerAddress *
      $UDPServerRun 514

  - on syslog-ng, a new source can be created the following way, it then needs
    to be added as a valid source in one of the "log" directives :

      source s_udp {
        udp(ip(127.0.0.1) port(514));
      };

Please consult your syslog daemon's manual for more information. If no logs are
seen in the system's log files, please consider the following tests :

  - restart haproxy. Each frontend and backend logs one line indicating it's
    starting. If these logs are received, it means logs are working.

  - run "strace -tt -s100 -etrace=sendmsg -p <haproxy's pid>" and perform some
    activity that you expect to be logged. You should see the log messages
    being sent using sendmsg() there. If they don't appear, restart using
    strace on top of haproxy. If you still see no logs, it definitely means
    that something is wrong in your configuration.

  - run tcpdump to watch for port 514, for example on the loopback interface if
    the traffic is being sent locally : "tcpdump -As0 -ni lo port 514". If the
    packets are seen there, it's the proof they're sent then the syslogd daemon
    needs to be troubleshooted.

While traffic logs are sent from the frontends (where the incoming connections
are accepted), backends also need to be able to send logs in order to report a
server state change consecutive to a health check. Please consult HAProxy's
configuration manual for more information regarding all possible log settings.

It is convenient to chose a facility that is not used by other daemons. HAProxy
examples often suggest "local0" for traffic logs and "local1" for admin logs
because they're never seen in field. A single facility would be enough as well.
Having separate logs is convenient for log analysis, but it's also important to
remember that logs may sometimes convey confidential information, and as such
they must not be mixed with other logs that may accidentally be handed out to
unauthorized people.

For in-field troubleshooting without impacting the server's capacity too much,
it is recommended to make use of the "halog" utility provided with HAProxy.
This is sort of a grep-like utility designed to process HAProxy log files at
a very fast data rate. Typical figures range between 1 and 2 GB of logs per
second. It is capable of extracting only certain logs (eg: search for some
classes of HTTP status codes, connection termination status, search by response
time ranges, look for errors only), count lines, limit the output to a number
of lines, and perform some more advanced statistics such as sorting servers
by response time or error counts, sorting URLs by time or count, sorting client
addresses by access count, and so on. It is pretty convenient to quickly spot
anomalies such as a bot looping on the site, and block them.

It is possible to query HAProxy about its status. The most commonly used
mechanism is the HTTP statistics page. This page also exposes an alternative
CSV output format for monitoring tools. The same format is provided on the
Unix socket.

Statistics are regroup in categories labelled as domains, corresponding to the
multiple components of HAProxy. There are two domains available: proxy and
resolvers. If not specified, the proxy domain is selected. Note that only the
proxy statistics are printed on the HTTP page.

The statistics may be consulted either from the unix socket or from the HTTP
page. Both means provide a CSV format whose fields follow. The first line
begins with a sharp ('#') and has one word per comma-delimited field which
represents the title of the column. All other lines starting at the second one
use a classical CSV format using a comma as the delimiter, and the double quote
('"') as an optional text delimiter, but only if the enclosed text is ambiguous
(if it contains a quote or a comma). The double-quote character ('"') in the
text is doubled ('""'), which is the format that most tools recognize. Please
do not insert any column before these ones in order not to break tools which
use hard-coded column positions.

For proxy statistics, after each field name, the types which may have a value
for that field are specified in brackets. The types are L (Listeners), F
(Frontends), B (Backends), and S (Servers). There is a fixed set of static
fields that are always available in the same order. A column containing the
character '-' delimits the end of the static fields, after which presence or
order of the fields are not guaranteed.

Here is the list of static fields using the proxy statistics domain:
  0. pxname [LFBS]: proxy name
  1. svname [LFBS]: service name (FRONTEND for frontend, BACKEND for backend,
     any name for server/listener)
  2. qcur [..BS]: current queued requests. For the backend this reports the
     number queued without a server assigned.
  3. qmax [..BS]: max value of qcur
  4. scur [LFBS]: current sessions
  5. smax [LFBS]: max sessions
  6. slim [LFBS]: configured session limit
  7. stot [LFBS]: cumulative number of sessions
  8. bin [LFBS]: bytes in
  9. bout [LFBS]: bytes out
 10. dreq [LFB.]: requests denied because of security concerns.
     - For tcp this is because of a matched tcp-request content rule.
     - For http this is because of a matched http-request or tarpit rule.
 11. dresp [LFBS]: responses denied because of security concerns.
     - For http this is because of a matched http-request rule, or
       "option checkcache".
 12. ereq [LF..]: request errors. Some of the possible causes are:
     - early termination from the client, before the request has been sent.
     - read error from the client
     - client timeout
     - client closed connection
     - various bad requests from the client.
     - request was tarpitted.
 13. econ [..BS]: number of requests that encountered an error trying to
     connect to a backend server. The backend stat is the sum of the stat
     for all servers of that backend, plus any connection errors not
     associated with a particular server (such as the backend having no
     active servers).
 14. eresp [..BS]: response errors. srv_abrt will be counted here also.
     Some other errors are:
     - write error on the client socket (won't be counted for the server stat)
     - failure applying filters to the response.
 15. wretr [..BS]: number of times a connection to a server was retried.
 16. wredis [..BS]: number of times a request was redispatched to another
     server. The server value counts the number of times that server was
     switched away from.
 17. status [LFBS]: status (UP/DOWN/NOLB/MAINT/MAINT(via)/MAINT(resolution)...)
 18. weight [..BS]: total effective weight (backend), effective weight (server)
 19. act [..BS]: number of active servers (backend), server is active (server)
 20. bck [..BS]: number of backup servers (backend), server is backup (server)
 21. chkfail [...S]: number of failed checks. (Only counts checks failed when
     the server is up.)
 22. chkdown [..BS]: number of UP->DOWN transitions. The backend counter counts
     transitions to the whole backend being down, rather than the sum of the
     counters for each server.
 23. lastchg [..BS]: number of seconds since the last UP<->DOWN transition
 24. downtime [..BS]: total downtime (in seconds). The value for the backend
     is the downtime for the whole backend, not the sum of the server downtime.
 25. qlimit [...S]: configured maxqueue for the server, or nothing in the
     value is 0 (default, meaning no limit)
 26. pid [LFBS]: process id (0 for first instance, 1 for second, ...)
 27. iid [LFBS]: unique proxy id
 28. sid [L..S]: server id (unique inside a proxy)
 29. throttle [...S]: current throttle percentage for the server, when
     slowstart is active, or no value if not in slowstart.
 30. lbtot [..BS]: total number of times a server was selected, either for new
     sessions, or when re-dispatching. The server counter is the number
     of times that server was selected.
 31. tracked [...S]: id of proxy/server if tracking is enabled.
 32. type [LFBS]: (0=frontend, 1=backend, 2=server, 3=socket/listener)
 33. rate [.FBS]: number of sessions per second over last elapsed second
 34. rate_lim [.F..]: configured limit on new sessions per second
 35. rate_max [.FBS]: max number of new sessions per second
 36. check_status [...S]: status of last health check, one of:
        UNK     -> unknown
        INI     -> initializing
        SOCKERR -> socket error
        L4OK    -> check passed on layer 4, no upper layers testing enabled
        L4TOUT  -> layer 1-4 timeout
        L4CON   -> layer 1-4 connection problem, for example
                   "Connection refused" (tcp rst) or "No route to host" (icmp)
        L6OK    -> check passed on layer 6
        L6TOUT  -> layer 6 (SSL) timeout
        L6RSP   -> layer 6 invalid response - protocol error
        L7OK    -> check passed on layer 7
        L7OKC   -> check conditionally passed on layer 7, for example 404 with
                   disable-on-404
        L7TOUT  -> layer 7 (HTTP/SMTP) timeout
        L7RSP   -> layer 7 invalid response - protocol error
        L7STS   -> layer 7 response error, for example HTTP 5xx
     Notice: If a check is currently running, the last known status will be
     reported, prefixed with "* ". e. g. "* L7OK".
 37. check_code [...S]: layer5-7 code, if available
 38. check_duration [...S]: time in ms took to finish last health check
 39. hrsp_1xx [.FBS]: http responses with 1xx code
 40. hrsp_2xx [.FBS]: http responses with 2xx code
 41. hrsp_3xx [.FBS]: http responses with 3xx code
 42. hrsp_4xx [.FBS]: http responses with 4xx code
 43. hrsp_5xx [.FBS]: http responses with 5xx code
 44. hrsp_other [.FBS]: http responses with other codes (protocol error)
 45. hanafail [...S]: failed health checks details
 46. req_rate [.F..]: HTTP requests per second over last elapsed second
 47. req_rate_max [.F..]: max number of HTTP requests per second observed
 48. req_tot [.FB.]: total number of HTTP requests received
 49. cli_abrt [..BS]: number of data transfers aborted by the client
 50. srv_abrt [..BS]: number of data transfers aborted by the server
     (inc. in eresp)
 51. comp_in [.FB.]: number of HTTP response bytes fed to the compressor
 52. comp_out [.FB.]: number of HTTP response bytes emitted by the compressor
 53. comp_byp [.FB.]: number of bytes that bypassed the HTTP compressor
     (CPU/BW limit)
 54. comp_rsp [.FB.]: number of HTTP responses that were compressed
 55. lastsess [..BS]: number of seconds since last session assigned to
     server/backend
 56. last_chk [...S]: last health check contents or textual error
 57. last_agt [...S]: last agent check contents or textual error
 58. qtime [..BS]: the average queue time in ms over the 1024 last requests
 59. ctime [..BS]: the average connect time in ms over the 1024 last requests
 60. rtime [..BS]: the average response time in ms over the 1024 last requests
     (0 for TCP)
 61. ttime [..BS]: the average total session time in ms over the 1024 last
     requests
 62. agent_status [...S]: status of last agent check, one of:
        UNK     -> unknown
        INI     -> initializing
        SOCKERR -> socket error
        L4OK    -> check passed on layer 4, no upper layers testing enabled
        L4TOUT  -> layer 1-4 timeout
        L4CON   -> layer 1-4 connection problem, for example
                   "Connection refused" (tcp rst) or "No route to host" (icmp)
        L7OK    -> agent reported "up"
        L7STS   -> agent reported "fail", "stop", or "down"
 63. agent_code [...S]: numeric code reported by agent if any (unused for now)
 64. agent_duration [...S]: time in ms taken to finish last check
 65. check_desc [...S]: short human-readable description of check_status
 66. agent_desc [...S]: short human-readable description of agent_status
 67. check_rise [...S]: server's "rise" parameter used by checks
 68. check_fall [...S]: server's "fall" parameter used by checks
 69. check_health [...S]: server's health check value between 0 and rise+fall-1
 70. agent_rise [...S]: agent's "rise" parameter, normally 1
 71. agent_fall [...S]: agent's "fall" parameter, normally 1
 72. agent_health [...S]: agent's health parameter, between 0 and rise+fall-1
 73. addr [L..S]: address:port or "unix". IPv6 has brackets around the address.
 74: cookie [..BS]: server's cookie value or backend's cookie name
 75: mode [LFBS]: proxy mode (tcp, http, health, unknown)
 76: algo [..B.]: load balancing algorithm
 77: conn_rate [.F..]: number of connections over the last elapsed second
 78: conn_rate_max [.F..]: highest known conn_rate
 79: conn_tot [.F..]: cumulative number of connections
 80: intercepted [.FB.]: cum. number of intercepted requests (monitor, stats)
 81: dcon [LF..]: requests denied by "tcp-request connection" rules
 82: dses [LF..]: requests denied by "tcp-request session" rules
 83: wrew [LFBS]: cumulative number of failed header rewriting warnings
 84: connect [..BS]: cumulative number of connection establishment attempts
 85: reuse [..BS]: cumulative number of connection reuses
 86: cache_lookups [.FB.]: cumulative number of cache lookups
 87: cache_hits [.FB.]: cumulative number of cache hits
 88: srv_icur [...S]: current number of idle connections available for reuse
 89: src_ilim [...S]: limit on the number of available idle connections
 90. qtime_max [..BS]: the maximum observed queue time in ms
 91. ctime_max [..BS]: the maximum observed connect time in ms
 92. rtime_max [..BS]: the maximum observed response time in ms (0 for TCP)
 93. ttime_max [..BS]: the maximum observed total session time in ms
 94. eint [LFBS]: cumulative number of internal errors
 95. idle_conn_cur [...S]: current number of unsafe idle connections
 96. safe_conn_cur [...S]: current number of safe idle connections
 97. used_conn_cur [...S]: current number of connections in use
 98. need_conn_est [...S]: estimated needed number of connections
 99. uweight [..BS]: total user weight (backend), server user weight (server)
 100. agg_server_status [..B.]: backend aggregated gauge of server's status
 101. agg_server_status_check [..B.]: (deprecated)
 102. agg_check_status [..B.]: backend aggregated gauge of server's state check
      status
 103. srid [...S]: server id revision
 104. sess_other [.F..]: total number of sessions other than HTTP since process
      started
 105. h1_sess [.F..]: total number of HTTP/1 sessions since process started
 106. h2_sess [.F..]: total number of HTTP/2 sessions since process started
 107. h3_sess [.F..]: total number of HTTP/3 sessions since process started
 108. req_other [.F..]: total number of sessions other than HTTP processed by
      this object since the worker process started
 109. h1req [.F..]: total number of HTTP/1 sessions processed by this object
      since the worker process started
 110. h2req [.F..]: total number of hTTP/2 sessions processed by this object
      since the worker process started
 111. h3req [.F..]: total number of HTTP/3 sessions processed by this object
      since the worker process started
 112. proto [L...]: protocol
 113. priv_idle_cur [...S]: current number of private idle connections
 114. reqbin [LFBS]: total number of request bytes received since the worker
      process started
 115. reqbout [LFBS]: total number of request bytes sent since the worker
      process started
 116. resbin [LFBS]: total number of response bytes received since the worker
      process started
 117. resbout [LFBS]: total number of response bytes sent since the worker
      process started
 118. cache_hint_hits [.FB.]: cumulative number of cache hits having served
      early hints

For all other statistics domains, the presence or the order of the fields are
not guaranteed. In this case, the header line should always be used to parse
the CSV data.

Both "show info" and "show stat" support a mode where each output value comes
with its type and sufficient information to know how the value is supposed to
be aggregated between processes and how it evolves.

In all cases, the output consists in having a single value per line with all
the information split into fields delimited by colons (':').

The first column designates the object or metric being dumped. Its format is
specific to the command producing this output and will not be described in this
section. Usually it will consist in a series of identifiers and field names.

The second column contains 4 characters respectively indicating the origin, the
nature, the scope and the persistence state of the value being reported. The
first character (the origin) indicates where the value was extracted from.
Possible characters are :

  M   The value is a metric. It is valid at one instant any may change depending
      on its nature .

  S   The value is a status. It represents a discrete value which by definition
      cannot be aggregated. It may be the status of a server ("UP" or "DOWN"),
      the PID of the process, etc.

  K   The value is a sorting key. It represents an identifier which may be used
      to group some values together because it is unique among its class. All
      internal identifiers are keys. Some names can be listed as keys if they
      are unique (eg: a frontend name is unique). In general keys come from the
      configuration, even though some of them may automatically be assigned. For
      most purposes keys may be considered as equivalent to configuration.

  C   The value comes from the configuration. Certain configuration values make
      sense on the output, for example a concurrent connection limit or a cookie
      name. By definition these values are the same in all processes started
      from the same configuration file.

  P   The value comes from the product itself. There are very few such values,
      most common use is to report the product name, version and release date.
      These elements are also the same between all processes.

The second character (the nature) indicates the nature of the information
carried by the field in order to let an aggregator decide on what operation to
use to aggregate multiple values. Possible characters are :

  A   The value represents an age since a last event. This is a bit different
      from the duration in that an age is automatically computed based on the
      current date. A typical example is how long ago did the last session
      happen on a server. Ages are generally aggregated by taking the minimum
      value and do not need to be stored.

  a   The value represents an already averaged value. The average response times
      and server weights are of this nature. Averages can typically be averaged
      between processes.

  C   The value represents a cumulative counter. Such measures perpetually
      increase until they wrap around. Some monitoring protocols need to tell
      the difference between a counter and a gauge to report a different type.
      In general counters may simply be summed since they represent events or
      volumes. Examples of metrics of this nature are connection counts or byte
      counts.

  D   The value represents a duration for a status. There are a few usages of
      this, most of them include the time taken by the last health check and
      the time a server has spent down. Durations are generally not summed,
      most of the time the maximum will be retained to compute an SLA.

  G   The value represents a gauge. It's a measure at one instant. The memory
      usage or the current number of active connections are of this nature.
      Metrics of this type are typically summed during aggregation.

  L   The value represents a limit (generally a configured one). By nature,
      limits are harder to aggregate since they are specific to the point where
      they were retrieved. In certain situations they may be summed or be kept
      separate.

  M   The value represents a maximum. In general it will apply to a gauge and
      keep the highest known value. An example of such a metric could be the
      maximum amount of concurrent connections that was encountered in the
      product's life time. To correctly aggregate maxima, you are supposed to
      output a range going from the maximum of all maxima and the sum of all
      of them. There is indeed no way to know if they were encountered
      simultaneously or not.

  m   The value represents a minimum. In general it will apply to a gauge and
      keep the lowest known value. An example of such a metric could be the
      minimum amount of free memory pools that was encountered in the product's
      life time. To correctly aggregate minima, you are supposed to output a
      range going from the minimum of all minima and the sum of all of them.
      There is indeed no way to know if they were encountered simultaneously
      or not.

  N   The value represents a name, so it is a string. It is used to report
      proxy names, server names and cookie names. Names have configuration or
      keys as their origin and are supposed to be the same among all processes.

  O   The value represents a free text output. Outputs from various commands,
      returns from health checks, node descriptions are of such nature.

  R   The value represents an event rate. It's a measure at one instant. It is
      quite similar to a gauge except that the recipient knows that this measure
      moves slowly and may decide not to keep all values. An example of such a
      metric is the measured amount of connections per second. Metrics of this
      type are typically summed during aggregation.

  T   The value represents a date or time. A field emitting the current date
      would be of this type. The method to aggregate such information is left
      as an implementation choice. For now no field uses this type.

The third character (the scope) indicates what extent the value reflects. Some
elements may be per process while others may be per configuration or per system.
The distinction is important to know whether or not a single value should be
kept during aggregation or if values have to be aggregated. The following
characters are currently supported :

  C   The value is valid for a whole cluster of nodes, which is the set of nodes
      communicating over the peers protocol. An example could be the amount of
      entries present in a stick table that is replicated with other peers. At
      the moment no metric use this scope.

  P   The value is valid only for the process reporting it. Most metrics use
      this scope.

  S   The value is valid for the whole service, which is the set of processes
      started together from the same configuration file. All metrics originating
      from the configuration use this scope. Some other metrics may use it as
      well for some shared resources (eg: shared SSL cache statistics).

  s   The value is valid for the whole system, such as the system's hostname,
      current date or resource usage. At the moment this scope is not used by
      any metric.

The fourth character (persistence state) indicates that the value (the metric)
is volatile or persistent across reloads. The following characters are expected :

  V   The metric is volatile because it is local to the current process so
      the value will be lost when reloading.

  P   The metric is persistent because it may be shared with other co-processes
      so that the value is preserved across reloads.

Consumers of these information will generally have enough of these 4 characters
to determine how to accurately report aggregated information across multiple
processes.

After this column, the third column indicates the type of the field, among "s32"
(signed 32-bit integer), "s64" (signed 64-bit integer), "u32" (unsigned 32-bit
integer), "u64" (unsigned 64-bit integer), "str" (string). It is important to
know the type before parsing the value in order to properly read it. For example
a string containing only digits is still a string an not an integer (eg: an
error code extracted by a check).

Then the fourth column is the value itself, encoded according to its type.
Strings are dumped as-is immediately after the colon without any leading space.
If a string contains a colon, it will appear normally. This means that the
output should not be exclusively split around colons or some check outputs
or server addresses might be truncated.

The stats socket is not enabled by default. In order to enable it, it is
necessary to add one line in the global section of the haproxy configuration.
A second line is recommended to set a larger timeout, always appreciated when
issuing commands by hand :

    global
        stats socket /var/run/haproxy.sock mode 600 level admin
        stats timeout 2m

It is also possible to add multiple instances of the stats socket by repeating
the line, and make them listen to a TCP port instead of a UNIX socket. This is
never done by default because this is dangerous, but can be handy in some
situations :

    global
        stats socket /var/run/haproxy.sock mode 600 level admin
        stats socket ipv4@192.168.0.1:9999 level admin
        stats timeout 2m

To access the socket, an external utility such as "socat" is required. Socat is
a swiss-army knife to connect anything to anything. We use it to connect
terminals to the socket, or a couple of stdin/stdout pipes to it for scripts.
The two main syntaxes we'll use are the following :

    # socat /var/run/haproxy.sock stdio
    # socat /var/run/haproxy.sock readline

The first one is used with scripts. It is possible to send the output of a
script to haproxy, and pass haproxy's output to another script. That's useful
for retrieving counters or attack traces for example.

The second one is only useful for issuing commands by hand. It has the benefit
that the terminal is handled by the readline library which supports line
editing and history, which is very convenient when issuing repeated commands
(eg: watch a counter).

The socket supports three operation modes :
  - non-interactive, silent
  - interactive, silent
  - interactive with prompt

The non-interactive mode is the default when socat connects to the socket. In
this mode, a single line may be sent. It is processed as a whole, responses are
sent back, and the connection closes after the end of the response. This is the
mode that scripts and monitoring tools use. It is possible to send multiple
commands in this mode, they need to be delimited by a semi-colon (';'). For
example :

    # echo "show info;show stat;show table" | socat /var/run/haproxy stdio

If a command needs to use a semi-colon or a backslash (eg: in a value), it
must be preceded by a backslash ('\').

The interactive mode allows new commands to be sent after the ones from the
previous lines finish. It exists in two variants, one silent, which works like
the non-interactive mode except that the socket waits for a new command instead
of closing, and one where a prompt is displayed ('>') at the beginning of the
line. The interactive mode is preferred for advanced tools while the prompt
mode is preferred for humans.

The mode can be changed using the "prompt
This keyword is available in sections :
Unix Socket commands
Master CLI commands" command. By default, it toggles the
interactive+prompt modes. Entering "prompt
This keyword is available in sections :
Unix Socket commands
Master CLI commands" in interactive mode will switch to
prompt mode. The command optionally takes a specific mode among which:

  - "n" : non-interactive mode (single command and quits)
  - "i" : interactive mode (multiple commands, no prompt)
  - "p" : prompt mode (multiple commands with a prompt)

Since the default mode is non-interactive, "prompt
This keyword is available in sections :
Unix Socket commands
Master CLI commands" must be used as the first
command in order to switch it, otherwise the previous command will cause the
connection to be closed. Switching to non-interactive mode will result in the
connection to be closed after all the commands of the same line complete.

For this reason, when debugging by hand, it's quite common to start with the
"prompt
This keyword is available in sections :
Unix Socket commands
Master CLI commands" command :

   # socat /var/run/haproxy readline
   prompt
   > show info
   ...
   >

Interactive tools might prefer starting with "prompt i" to switch to interactive
mode without the prompt.

Optionally the process' uptime may be displayed in the prompt. In order to
enable this, the "prompt timed" command will enable the prompt and toggle the
displaying of the time. The uptime is displayed in format "d:hh:mm:ss" where
"d" is the number of days, and "hh", "mm", "ss" are respectively the number
of hours, minutes and seconds on two digits each:

   # socat /var/run/haproxy readline
   prompt timed

   [23:03:34:39]> show version
   2.8-dev9-e5e622-18

   [23:03:34:41]> quit

When the timed prompt is set on the master CLI, the prompt will display the
currently selected process' uptime, so this will work for the master, current
worker or an older worker:

  master> prompt timed
  [0:00:00:50] master> show proc
  (...)
  [0:00:00:58] master> @!11955     <-- master, switch to current worker
  [0:00:01:03] 11955> @!11942      <-- current worker, switch to older worker
  [0:00:02:17] 11942> @            <-- older worker, switch back to master
  [0:00:01:10] master>

Since multiple commands may be issued at once, haproxy uses the empty line as a
delimiter to mark an end of output for each command, and takes care of ensuring
that no command can emit an empty line on output. A script can thus easily
parse the output even when multiple commands were pipelined on a single line.

Some commands may take an optional payload. To add one to a command, the first
line needs to end with the "<<\n" pattern. The next lines will be treated as
the payload and can contain as many lines as needed. To validate a command with
a payload, it needs to end with an empty line.

The payload pattern can be customized in order to change the way the payload
ends. In order to end a payload with something else than an empty line, a
customized pattern can be set between '<<' and '\n'. Up to 64 characters can be
used in addition to '<<', otherwise this won't be considered a payload. It
should be enough to use random payload patterns. For example, to use a PEM file
that contains empty lines and comments:

  # echo -e "set ssl cert common.pem <<%EOF%\n$(cat common.pem)\n%EOF%\n" | \
  socat /var/run/haproxy.stat -

Limitations do exist: The pattern "<<" must not be glued to the last word of the
line. The length of a command line must not be greater than tune.bufsize,
including the pattern starting the payload, but excluding the payload
itself. The payload size is limited to 128KB by default. This can be changed by
setting "tune.cli.max-payload-size" global parameter, with some cautions. Note
the pattern marking the end of the payload is part of this limit.

When entering a payload while in interactive mode, the prompt will change from
"> " to "+ ".

It is important to understand that when multiple haproxy processes are started
on the same sockets, any process may pick up the request and will output its
own stats.

The list of commands currently supported on the stats socket is provided below.
If an unknown command is sent, haproxy displays the usage message which reminds
all supported commands. Some commands support a more complex syntax, generally
it will explain what part of the command is invalid when this happens.

Some commands require a higher level of privilege to work. If you do not have
enough privilege, you will get an error "Permission denied". Please check
the "level" option of the "bind" keyword lines in the configuration manual
for more information.

Abort and destroy a temporary CA file update transaction.

See also "set ssl ca-file" and "commit ssl ca-file".

Abort and destroy a temporary SSL certificate update transaction.

See also "set ssl cert" and "commit ssl cert".

Abort and destroy a temporary CRL file update transaction.

See also "set ssl crl-file" and "commit ssl crl-file".

Starts an ACME certificate generation task with the given certificate name.
The certificate must be linked to an acme section, see section 12.8 "ACME"
of the configuration manual. See also "acme status".

Show the status of every certificates that were configured with ACME.

This command outputs, separated by a tab:
- The name of the certificate configured in haproxy
- The acme section used in the configuration
- The state of the acme task, either "Running", "Scheduled" or "Stopped"
- The UTC expiration date of the certificate in ISO8601 format
- The relative expiration time (0d if expired)
- The UTC scheduled date of the certificate in ISO8601 format
- The relative schedule time (0d if Running)

$ echo "@1; acme status" | socat /tmp/master.sock - | column -t -s $'\t'
# certificate   section  state      expiration date (UTC)  expires in        scheduled date (UTC)  scheduled in
ecdsa.pem       LE       Running    2020-01-18T09:31:12Z   0d 0h00m00s       2020-01-15T21:31:12Z  0d 0h00m00s
foobar.pem.rsa  LE       Scheduled  2025-08-04T11:50:54Z   89d 23h01m13s     2025-07-27T23:50:55Z  82d 11h01m14s

Add an entry into the acl <acl>. <acl> is the #<id> or the <name> returned by
"show acl". This command does not verify if the entry already exists. Entries
are added to the current version of the ACL, unless a specific version is
specified with "@<ver>". This version number must have preliminary been
allocated by "prepare acl", and it will be comprised between the versions
reported in "curr_ver" and "next_ver" on the output of "show acl". Entries
added with a specific version number will not match until a "commit acl"
operation is performed on them. They may however be consulted using the
"show acl @<ver>" command, and cleared using a "clear acl @<ver>" command.
This command cannot be used if the reference <acl> is a name also used with
a map. In this case, the "add map" command must be used instead.

Instantiate a new backend proxy with the name <name>.

Only TCP or HTTP proxies can be created. All of the settings are inherited
from <defproxy> default proxy instance. By default, it is mandatory to
specify the backend mode via the argument of the same name, unless <defproxy>
already defines it explicitly. It is also possible to use an optional GUID
argument if wanted.

Servers can be added via the command "add server". The backend is initialized
in the unpublished state. Once considered ready for traffic, use "publish
backend" to expose the newly created instance.

All named default proxies can be used, given that they validate the same
inheritance rules applied during configuration parsing. There is some
exceptions though, for example when the mode is neither TCP nor HTTP. The
available default proxy instances can be listed via the "show defaults"
additional command.

This command is restricted and can only be issued on sockets configured for
level "admin".

Add an entry into the map <map> to associate the value <value> to the key
<key>. This command does not verify if the entry already exists. It is
mainly used to fill a map after a "clear" or "prepare" operation. Entries
are added to the current version of the ACL, unless a specific version is
specified with "@<ver>". This version number must have preliminary been
allocated by "prepare acl", and it will be comprised between the versions
reported in "curr_ver" and "next_ver" on the output of "show acl". Entries
added with a specific version number will not match until a "commit map"
operation is performed on them. They may however be consulted using the
"show map @<ver>" command, and cleared using a "clear acl @<ver>" command.
If the designated map is also used as an ACL, the ACL will only match the
<key> part and will ignore the <value> part. Using the payload syntax it is
possible to add multiple key/value pairs by entering them on separate lines.
On each new line, the first word is the key and the rest of the line is
considered to be the value which can even contains spaces.

# socat /tmp/sock1 -
prompt

> add map #-1 <<
+ key1 value1
+ key2 value2 with spaces
+ key3 value3 also with spaces
+ key4 value4

>

Instantiate a new server attached to the backend <backend>.

The <server> name must not be already used in the backend. A special
restriction is put on the backend which must used a dynamic load-balancing
algorithm. A subset of keywords from the server config file statement can be
used to configure the server behavior (see "add server help" to list them).

By default, 'default-server' is ignored by a dynamically created server.
However, it is possible to use the 'from' server positional keyword to
explicitely preset the settings of the newly created server from another
server or a default-server instance, either named or unnamed. The additional
command "show default-server" lists the available entries.

Currently a dynamic server is statically initialized with the "none"
init-addr method. This means that no resolution will be undertaken if a FQDN
is specified as an address, even if the server creation will be validated.

To support the reload operations, it is expected that the server created via
the CLI is also manually inserted in the relevant haproxy configuration file.
A dynamic server not present in the configuration won't be restored after a
reload operation.

A dynamic server may use the "track" keyword to follow the check status of
another server from the configuration. However, it is not possible to track
another dynamic server. This is to ensure that the tracking chain is kept
consistent even in the case of dynamic servers deletion.

Use the "check" keyword to enable health-check support. Note that the
health-check is disabled by default and must be enabled independently from
the server using the "enable health" command. For agent checks, use the
"agent-check" keyword and the "enable agent" command. Note that in this case
the server may be activated via the agent depending on the status reported,
without an explicit "enable server" command. This also means that extra care
is required when removing a dynamic server with agent check. The agent should
be first deactivated via "disable agent" to be able to put the server in the
required maintenance mode before removal.

It may be possible to reach the fd limit when using a large number of dynamic
servers. Please refer to the "u-limit" global keyword documentation in this
case.

List the keywords supported for dynamic servers by the current haproxy
version. Keyword syntax is similar to the server line from the configuration
file, please refer to their individual documentation for details.

echo -e "set ssl ca-file cafile.pem <<\n$(cat rootCA.crt)\n" | \
socat /var/run/haproxy.stat -
echo -e "add ssl ca-file cafile.pem <<\n$(cat intermediate1.crt)\n" | \
socat /var/run/haproxy.stat -
echo -e "add ssl ca-file cafile.pem <<\n$(cat intermediate2.crt)\n" | \
socat /var/run/haproxy.stat -
echo "commit ssl ca-file cafile.pem" | socat /var/run/haproxy.stat -

Add an certificate in a crt-list. It can also be used for directories since
directories are now loaded the same way as the crt-lists. This command allow
you to use a certificate name in parameter, to use SSL options or filters a
crt-list line must sent as a payload instead. Only one crt-list line is
supported in the payload. This command will load the certificate for every
bind lines using the crt-list. To push a new certificate to HAProxy the
commands "new ssl cert" and "set ssl cert" must be used.

$ echo "new ssl cert foobar.pem" | socat /tmp/sock1 -
$ echo -e "set ssl cert foobar.pem <<\n$(cat foobar.pem)\n" | socat
/tmp/sock1 -
$ echo "commit ssl cert foobar.pem" | socat /tmp/sock1 -
$ echo "add ssl crt-list certlist1 foobar.pem" | socat /tmp/sock1 -

$ echo -e 'add ssl crt-list certlist1 <<\nfoobar.pem [allow-0rtt] foo.bar.com
!test1.com\n' | socat /tmp/sock1 -

Add an ECH key to a <bind> line. The payload must be in the PEM for ECH format.
(https://datatracker.ietf.org/doc/html/draft-farrell-tls-pemesni)

The bind line format is <frontend>/@<filename>:<linenum> (Example:
frontend1/@haproxy.conf:19) or <frontend>/<name> if the bind line was named
with the "name" keyword.

Necessitates an OpenSSL version that supports ECH, and HAProxy must be
compiled with USE_ECH=1. This command is only supported on a CLI connection
running in experimental mode (see "experimental-mode on"). Not available
when HAProxy is built with AWS-LC.

See also "show ssl ech" and "ech" in the Section 5.1 of the configuration
manual.

$ openssl ech -public_name foobar.com -out foobar3.com.ech
$ echo -e "experimental-mode on; add ssl ech frontend1/@haproxy.conf:19 <<%EOF%\n$(cat foobar3.com.ech)\n%EOF%\n" | \
  socat /tmp/haproxy.sock -
added a new ECH config to frontend1

Add an already loaded certificate to the list of certificates that can be
used for JWT validation (see "jwt_verify_cert" converter). This command does
not work on ongoing transactions.
See also "del ssl jwt" and "show ssl jwt" commands.
See "jwt" certificate option for more information.

Clear the max values of the statistics counters in each proxy (frontend &
backend) and in each server. The accumulated counters are not affected. The
internal activity counters reported by "show activity" are also reset. This
can be used to get clean counters after an incident, without having to
restart nor to clear traffic counters. This command is restricted and can
only be issued on sockets configured for levels "operator" or "admin".

Clear all statistics counters in each proxy (frontend & backend) and in each
server. This has the same effect as restarting. This command is restricted
and can only be issued on sockets configured for level "admin".