8. Logging

  1. For logging, HAProxy always relies on a syslog server since it does not perform
  2. any file-system access. The standard way of using it is to send logs over UDP
  3. to the log server (by default on port 514). Very commonly this is configured to
  4. 127.0.0.1 where the local syslog daemon is running, but it's also used over the
  5. network to log to a central server. The central server provides additional
  6. benefits especially in active-active scenarios where it is desirable to keep
  7. the logs merged in arrival order. HAProxy may also make use of a UNIX socket to
  8. send its logs to the local syslog daemon, but it is not recommended at all,
  9. because if the syslog server is restarted while haproxy runs, the socket will
  10. be replaced and new logs will be lost. Since HAProxy will be isolated inside a
  11. chroot jail, it will not have the ability to reconnect to the new socket. It
  12. has also been observed in field that the log buffers in use on UNIX sockets are
  13. very small and lead to lost messages even at very light loads. But this can be
  14. fine for testing however.
  15.  
  16. It is recommended to add the following directive to the "global" section to
  17. make HAProxy log to the local daemon using facility "local0" :
  18.  
  19.       log 127.0.0.1:514 local0
  20.  
  21. and then to add the following one to each "defaults" section or to each frontend
  22. and backend section :
  23.  
  24.       log global
  25.  
  26. This way, all logs will be centralized through the global definition of where
  27. the log server is.
  28.  
  29. Some syslog daemons do not listen to UDP traffic by default, so depending on
  30. the daemon being used, the syntax to enable this will vary :
  31.  
  32.   - on sysklogd, you need to pass argument "-r" on the daemon's command line
  33.     so that it listens to a UDP socket for "remote" logs ; note that there is
  34.     no way to limit it to address 127.0.0.1 so it will also receive logs from
  35.     remote systems ;
  36.  
  37.   - on rsyslogd, the following lines must be added to the configuration file :
  38.  
  39.       $ModLoad imudp
  40.       $UDPServerAddress *
  41.       $UDPServerRun 514
  42.  
  43.   - on syslog-ng, a new source can be created the following way, it then needs
  44.     to be added as a valid source in one of the "log" directives :
  45.  
  46.       source s_udp {
  47.         udp(ip(127.0.0.1) port(514));
  48.       };
  49.  
  50. Please consult your syslog daemon's manual for more information. If no logs are
  51. seen in the system's log files, please consider the following tests :
  52.  
  53.   - restart haproxy. Each frontend and backend logs one line indicating it's
  54.     starting. If these logs are received, it means logs are working.
  55.  
  56.   - run "strace -tt -s100 -etrace=sendmsg -p <haproxy's pid>" and perform some
  57.     activity that you expect to be logged. You should see the log messages
  58.     being sent using sendmsg() there. If they don't appear, restart using
  59.     strace on top of haproxy. If you still see no logs, it definitely means
  60.     that something is wrong in your configuration.
  61.  
  62.   - run tcpdump to watch for port 514, for example on the loopback interface if
  63.     the traffic is being sent locally : "tcpdump -As0 -ni lo port 514". If the
  64.     packets are seen there, it's the proof they're sent then the syslogd daemon
  65.     needs to be troubleshooted.
  66.  
  67. While traffic logs are sent from the frontends (where the incoming connections
  68. are accepted), backends also need to be able to send logs in order to report a
  69. server state change consecutive to a health check. Please consult HAProxy's
  70. configuration manual for more information regarding all possible log settings.
  71.  
  72. It is convenient to chose a facility that is not used by other daemons. HAProxy
  73. examples often suggest "local0" for traffic logs and "local1" for admin logs
  74. because they're never seen in field. A single facility would be enough as well.
  75. Having separate logs is convenient for log analysis, but it's also important to
  76. remember that logs may sometimes convey confidential information, and as such
  77. they must not be mixed with other logs that may accidentally be handed out to
  78. unauthorized people.
  79.  
  80. For in-field troubleshooting without impacting the server's capacity too much,
  81. it is recommended to make use of the "halog" utility provided with HAProxy.
  82. This is sort of a grep-like utility designed to process HAProxy log files at
  83. a very fast data rate. Typical figures range between 1 and 2 GB of logs per
  84. second. It is capable of extracting only certain logs (eg: search for some
  85. classes of HTTP status codes, connection termination status, search by response
  86. time ranges, look for errors only), count lines, limit the output to a number
  87. of lines, and perform some more advanced statistics such as sorting servers
  88. by response time or error counts, sorting URLs by time or count, sorting client
  89. addresses by access count, and so on. It is pretty convenient to quickly spot
  90. anomalies such as a bot looping on the site, and block them.