Apache Series — Apache HTTPD Server Status

In this post, I’d like to describe how Apache HTTP server’s mod_status can be used to debug some common problems like — “increased response time or connection failures” to requests.

Collecting Server Status

Adding the following snippet into the config, causes mod_status to produce a detailed server status report:

<Location /server-status>
SetHandler server-status
</Location>

The server status report can be obtained in either HTML format or plain-text format by using either of the following URLs:

http://<server-name>:<server-port>/server-status # for HTML format
http://<server-name>:<server-port>/server-status?auto #plain-txt format

It is preferable to obtain the server status in plain-text format as we could write tools that parse this output for analysis.

An example of server status page in plain-text format is shown below:

localhost
ServerVersion: Apache/2.4.54 (Unix)
ServerMPM: event
Server Built: Jul 27 2022 10:46:50
CurrentTime: Tuesday, 02-Aug-2022 11:09:36 IST
RestartTime: Tuesday, 02-Aug-2022 10:28:47 IST
ParentServerConfigGeneration: 1
ParentServerMPMGeneration: 0
ServerUptimeSeconds: 2448
ServerUptime: 40 minutes 48 seconds
Load1: 0.00
Load5: 0.01
Load15: 0.08
Total Accesses: 29092
Total kBytes: 1336
Total Duration: 475439
CPUUser: .95
CPUSystem: 1.35
CPUChildrenUser: .3
CPUChildrenSystem: .47
CPULoad: .125408
Uptime: 2448
ReqPerSec: 11.884
BytesPerSec: 558.85
BytesPerReq: 47.0254
DurationPerReq: 16.3426
BusyWorkers: 1
IdleWorkers: 249
Processes: 10
Stopping: 0
BusyWorkers: 1
IdleWorkers: 249
ConnsTotal: 0
ConnsAsyncWriting: 0
ConnsAsyncKeepAlive: 0
ConnsAsyncClosing: 0
Scoreboard: _________________________…………………….____________________W____________________________________________________________________________________________________________________________________________________________________________________________________________……………………………………………………………………………………………………………..
localhost
ServerVersion: Apache/2.4.54 (Unix)
ServerMPM: event
Server Built: Jul 27 2022 10:46:50
CurrentTime: Tuesday, 02-Aug-2022 11:09:41 IST

It should be possible to collect approximately 1 second samples of the server status by sending the /server-status?auto request periodically

while true; do curl http://localhost:8080/server-status?auto >> server_status.txt; sleep 1; done

Analyzing the status report

• Key metrics would be the number of Idleworkers vs BusyWorkers. In a heavily loaded server, there will be zero Idle workers. This results in the server/a specific httpd process not accepting any new connections due to non-availability of workers and queuing up incoming connections in the TCP Accept queue. Slow Accepts result in the client experiencing increased response times.
• I modified server_status.c to add a new metric “NotAcceptingProcess” to list the number of process that are not accepting any new connections due to the absence of worker threads. If the value of NotAcceptingProcess is > 0, it indicates that the load on the system is increasing and if this value reaches close to the ServerLimit value, it is further indicative of the load on system being too high than what the system can handle.
• Along with collecting server-status, it is also useful to collect the output of the following command to obtain the list of sockets that are waiting to be accepted:

ss -plnt sport = :8080  #In the output, look for the value under RECV-Q column.
STATE RECV-Q SEND-Q LocalAddress:port
LISTEN 510    511    *:8080

• A high value under RECV-Q column again points to the fact, that processes that are listening on 8080 are not accepting connections fast enough.
• Error messages in the error_log that indicate “Increase MaxRequestWorkers” in the same timestamp are further indicative of the load on the system not matching the Event MPM configurations of ServerLimit, MaxRequestWorkers