AH00016 Error in Apache Error Log

How the Issue Manifests

  • Apache does not start. The following errors are found in /var/log/error_log:

    [Sun Mar 27 01:05:00.675153 2022] [ssl:debug] [pid 20477:tid 140025818814592] ssl_engine_init.c(356): AH01886: SSL FIPS mode disabled
AH00016: Configuration Failed
[Sun Mar 27 01:05:02 2022] [suexec:notice] [pid 21230:tid 140717417961600] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
AH00016: Configuration Failed
[Sun Mar 27 01:07:22 2022] [suexec:notice] [pid 29343:tid 139922163656832] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
AH00016: Configuration Failed

  • Running httpd -t does not report any syntax errors

Cause of the Issue

This error is almost always caused by an SSL mismatch on a domain.

How to Resolve

  1. Log in to the server at the CLI as root, either via SSH or from the terminal

  2. At the CLI, run the following strace:

    strace -s500 -f /usr/sbin/httpd &> /root/straceoutput.txt

  3. Once the strace has completed, with a text editor, open /root/straceoutput.txt. The following example uses the Vim text editor:

    vim /root/straceoutput.txt

  4. Search the file for AH00016. Example results:

    [pid  8328] write(104, "[Sun Mar 27 02:00:53 2022] [ssl:emerg] [pid 8328:tid 140358718720128] AH02565: Certificate and private key domain.com:443:0 from /home/domainco/var/domain.com/ssl/domain.com.crt and /home/domainco/var/domain.com/ssl/domain.com.priv.key do not match\n", 254) = 254
[pid  8724] <... read resumed>"[Sun Mar 27 02:00:53 2022] [ssl:emerg] [pid 8328:tid 140358718720128] AH02565: Certificate and private key domain.com:443:0 from /home/domainco/var/domain.com/ssl/domain.com.crt and /home/domainco/var/domain.com/ssl/domain.com.priv.key do not match\n", 65536) = 254
[pid  8328] write(2, "AH00016: Configuration Failed\n", 30 <unfinished ...>
[pid  8724] write(3, "[Sun Mar 27 02:00:53 2022] [ssl:emerg] [pid 8328:tid 140358718720128] AH02565: Certificate and private key domain.com:443:0 from /home/domainco/var/domain.com/ssl/domain.com.crt and /home/domainco/var/domain.com/ssl/domain.com.priv.key do not match\n", 254) = 254
[pid  8724] read(0,  <unfinished ...>

  5. Naviagate to the ssl directory for the domain specified in the strace output. Domain-level SSL directories are located at /home/{unixuser}/var/{domain.com}/ssl, replacing {unixuser} and {domain.com} with the corresponding information

  6. Compare the certificate and private key files in that directory to the SSL found in SiteWorx

  7. Generating a new Let’s Encrypt sll will often resolve the issue

Things to Keep in Mind

If, after running the strace, there are no entries for “AH00016” listed in the output folder, it is possible that the strace timed out before it processed the afflicted domain. This commonly occurs on servers with several hundred to several thousand vhosts.

In this case, temporarily disabling individual domain logging should allow the strace to complete. This can be accomplished by running the following commands at the CLI, and then running the strace command, again:

sed -i 's/^  ErrorLog/  #ErrorLog/g' /etc/httpd/conf.d/vhost_*.conf
sed -i 's/^  CustomLog/  #CustomLog/g' /etc/httpd/conf.d/vhost_*.conf

After troubleshooting is completed, run the following to re-enable individual domain logging:

sed -i 's/^  #ErrorLog/  ErrorLog/g' /etc/httpd/conf.d/vhost_*.conf
sed -i 's/^  #CustomLog/  CustomLog/g' /etc/httpd/conf.d/vhost_*.conf