Postfix
Stress-Dependent ConfigurationPostfix
Stress-Dependent ConfigurationThis document describes the symptoms of Postfix SMTP server overload. It presents permanent main.cf changes to avoid overload during normal operation, and temporary main.cf changes to cope with an unexpected burst of mail. This document makes specific suggestions for Postfix 2.5 and later which support stress-adaptive behavior, and for earlier Postfix versions that don't.
Topics covered in this document:
Under normal conditions, the Postfix SMTP server responds immediately when an SMTP client connects to it; the time to deliver mail is noticeable only with large messages. Performance degrades dramatically when the number of SMTP clients exceeds the number of Postfix SMTP server processes. When an SMTP client connects while all Postfix SMTP server processes are busy, the client must wait until a server process becomes available.
SMTP server overload may be caused by a surge of legitimate mail (example: a DNS registrar opens a new zone for registrations), by mistake (mail explosion caused by a forwarding loop) or by malice (worm outbreak, botnet, or other illegitimate activity).
Symptoms of Postfix SMTP server overload are:
Remote SMTP clients experience a long delay before Postfix sends the "220 hostname.example.com ESMTP Postfix" greeting.
NOTE: Broken DNS configurations can also cause lengthy delays before Postfix sends "220 hostname.example.com ...". These delays also exist when Postfix is NOT overloaded.
NOTE: To avoid "overload" delays for end-user mail clients, enable the "submission" service entry in master.cf (present since Postfix 2.1), and tell users to connect to this instead of the public SMTP service.
The Postfix SMTP server logs an increased number of "lost connection after CONNECT" events. This happens because remote SMTP clients disconnect before Postfix answers the connection.
NOTE: A portscan for open SMTP ports can also result in "lost connection ..." logfile messages.
Postfix 2.3 and later logs a warning that all server ports are busy:
Oct 3 20:39:27 spike postfix/master[28905]: warning: service "smtp" (25) has reached its process limit "30": new clients may experience noticeable delays Oct 3 20:39:27 spike postfix/master[28905]: warning: to avoid this condition, increase the process count in master.cf or reduce the service time per client
Legitimate mail that doesn't get through during an episode of Postfix SMTP server overload is not necessarily lost. It should still arrive once the situation returns to normal, as long as the overload condition is temporary.
One measure to avoid the "all server processes busy" condition is to service more SMTP clients simultaneously. For this you need to increase the number of Postfix SMTP server processes. This will improve the responsiveness for remote SMTP clients, as long as the server machine has enough hardware and software resources to run the additional processes, and as long as the file system can keep up with the additional load.
You increase the number of SMTP server processes either by increasing the default_process_limit in main.cf (line 3 below), or by increasing the SMTP server's "maxproc" field in master.cf (line 10 below). Either way, you need to issue a "postfix reload" command to make the change effective.
Process limits above 1000 require Postfix version 2.4 or later, and an operating system that supports kernel-based event filters (BSD kqueue(2), Linux epoll(4), or Solaris /dev/poll).
More processes use more memory. You can reduce the Postfix memory footprint by using cdb: lookup tables instead of Berkeley DB's hash: or btree: tables.
1 /etc/postfix/main.cf: 2 # Raise the global process limit, 100 since Postfix 2.0. 3 default_process_limit = 200 4 5 /etc/postfix/master.cf: 6 # ============================================================= 7 # service type private unpriv chroot wakeup maxproc command 8 # ============================================================= 9 # Raise the SMTP service process limit only. 10 smtp inet n - n - 200 smtpd
NOTE: older versions of the SMTPD_POLICY_README document contain a mistake: they configure a fixed number of policy daemon processes. When you raise the SMTP server's "maxproc" field in master.cf, SMTP server processes will report problems when connecting to policy server processes, because there aren't enough of them. Examples of errors are "connection refused" or "operation timed out".
To fix, edit master.cf and specify a zero "maxproc" field in all policy server entries; see line 6 in the example below. Issue a "postfix reload" command to make the change effective.
1 /etc/postfix/master.cf: 2 # ============================================================= 3 # service type private unpriv chroot wakeup maxproc command 4 # ============================================================= 5 # Disable the policy service process limit. 6 policy unix - n n - 0 spawn 7 user=nobody argv=/some/where/policy-server
When increasing the number of SMTP server processes is not practical, you can improve Postfix server responsiveness by eliminating delays. When Postfix spends less time per SMTP session, the same number of SMTP server processes can service more clients in a given amount of time.
Eliminate non-functional RBL lookups (blocklists that are no longer in operation). These lookups can degrade performance. Postfix logs a warning when an RBL server does not respond.
Eliminate redundant RBL lookups (people often use multiple Spamhaus RBLs that include each other). To find out whether RBLs include other RBLs, look up the websites that document the RBL's policies.
Eliminate header_checks and body_checks, and keep just a few emergency patterns to block the latest worm explosion or backscatter mail. See BACKSCATTER_README for examples of the latter.
Group your header_checks and body_checks patterns to avoid unnecessary pattern matching operations:
1 /etc/postfix/header_checks: 2 if /^Subject:/ 3 /^Subject: virus found in mail from you/ reject 4 /^Subject: ..other../ reject 5 endif 6 7 if /^Received:/ 8 /^Received: from (postfix\.org) / reject forged client name in received header: $1 9 /^Received: from ..other../ reject .... 10 endif
Under conditions of overload you can improve Postfix SMTP server responsiveness by hanging up on suspicious clients, so that other clients get a chance to talk to Postfix.
Use "521" SMTP reply codes (Postfix 2.6 and later) or "421" (Postfix 2.3-2.5) to hang up on clients that that match botnet-related RBLs (see next bullet) or that match selected non-RBL restrictions such as SMTP access maps. The Postfix SMTP server will reject mail ) [Soma href="#hangup"> Disconnect suspicious SMTP clients
Under normal conditions, the Postfix SMTP server responds immediately when an SMTP client connects to it; the time to deliver mail is noticeable only with large messages. Performance degrades dramatically when the number of SMTP clients exceeds the number of Postfix SMTP server processes. When an SMTP client connects while all Postfix SMTP server processes are busy, the client must wait until a server process becomes available.
SMTP server overload may be caused by a surge of legitimate mail (example: a DNS registrar opens a new zone for registrations), by mistake (mail explosion caused by a forwarding loop) or by malice (worm outbreak, botnet, or other illegitimate activity).
Symptoms of Postfix SMTP server overload are:
Remote SMTP clients experience a long delay before Postfix sends the "220 hostname.example.com ESMTP Postfix" greeting.
NOTE: Broken DNS configurations can also cause lengthy delays before Postfix sends "220 hostname.example.com ...". These delays also exist when Postfix is NOT overloaded.
NOTE: To avoid "overload" delays for end-user mail clients, enable the "submission" service entry in master.cf (present since Postfix 2.1), and tell users to connect to this instead of the public SMTP service.
The Postfix SMTP server logs an increased number of "lost connection after CONNECT" events. This happens because remote SMTP clients disconnect before Postfix answers the connection.
NOTE: A portscan for open SMTP ports can also result in "lost connection ..." logfile messages.
Postfix 2.3 and later logs a warning that all server ports are busy:
Oct 3 20:39:27 spike postfix/master[28905]: warning: service "smtp" (25) has reached its process limit "30": new clients may experience noticeable delays Oct 3 20:39:27 spike postfix/master[28905]: warning: to avoid this condition, increase the process count in master.cf or reduce the service time per client
Legitimate mail that doesn't get through during an episode of Postfix SMTP server overload is not necessarily lost. It should still arrive once the situation returns to normal, as long as the overload condition is temporary.
One measure to avoid the "all server processes busy" condition is to service more SMTP clients simultaneously. For this you need to increase the number of Postfix SMTP server processes. This will improve the responsiveness for remote SMTP clients, as long as the server machine has enough hardware and software resources to run the additional processes, and as long as the file system can keep up with the additional load.
You increase the number of SMTP server processes either by increasing the default_process_limit in main.cf (line 3 below), or by increasing the SMTP server's "maxproc" field in master.cf (line 10 below). Either way, you need to issue a "postfix reload" command to make the change effective.
Process limits above 1000 require Postfix version 2.4 or later, and an operating system that supports kernel-based event filters (BSD kqueue(2), Linux epoll(4), or Solaris /dev/poll).
More processes use more memory. You can reduce the Postfix memory footprint by using cdb: lookup tables instead of Berkeley DB's hash: or btree: tables.
1 /etc/postfix/main.cf: 2 # Raise the global process limit, 100 since Postfix 2.0. 3 default_process_limit = 200 4 5 /etc/postfix/master.cf: 6 # ============================================================= 7 # service type private unpriv chroot wakeup maxproc command 8 # ============================================================= 9 # Raise the SMTP service process limit only. 10 smtp inet n - n - 200 smtpd
NOTE: older versions of the SMTPD_POLICY_README document contain a mistake: they configure a fixed number of policy daemon processes. When you raise the SMTP server's "maxproc" field in master.cf, SMTP server processes will report problems when connecting to policy server processes, because there aren't enough of them. Examples of errors are "connection refused" or "operation timed out".
To fix, edit master.cf and specify a zero "maxproc" field in all policy server entries; see line 6 in the example below. Issue a "postfix reload" command to make the change effective.
1 /etc/postfix/master.cf: 2 # ============================================================= 3 # service type private unpriv chroot wakeup maxproc command 4 # ============================================================= 5 # Disable the policy service process limit. 6 policy unix - n n - 0 spawn 7 user=nobody argv=/some/where/policy-server
When increasing the number of SMTP server processes is not practical, you can improve Postfix server responsiveness by eliminating delays. When Postfix spends less time per SMTP session, the same number of SMTP server processes can service more clients in a given amount of time.
Eliminate non-functional RBL lookups (blocklists that are no longer in operation). These lookups can degrade performance. Postfix logs a warning when an RBL server does not respond.
Eliminate redundant RBL lookups (people often use multiple Spamhaus RBLs that include each other). To find out whether RBLs include other RBLs, look up the websites that document the RBL's policies.
Eliminate header_checks and body_checks, and keep just a few emergency patterns to block the latest worm explosion or backscatter mail. See BACKSCATTER_README for examples of the latter.
Group your header_checks and body_checks patterns to avoid unnecessary pattern matching operations:
1 /etc/postfix/header_checks: 2 if /^Subject:/ 3 /^Subject: virus found in mail from you/ reject 4 /^Subject: ..other../ reject 5 endif 6 7 if /^Received:/ 8 /^Received: from (postfix\.org) / reject forged client name in received header: $1 9 /^Received: from ..other../ reject .... 10 endif
Under conditions of overload you can improve Postfix SMTP server responsiveness by hanging up on suspicious clients, so that other clients get a chance to talk to Postfix.
Use "521" SMTP reply codes (Postfix 2.6 and later) or "421" (Postfix 2.3-2.5) to hang up on clients that that match botnet-related RBLs (see next bullet) or that match selected non-RBL restrictions such as SMTP access maps. The Postfix SMTP server will reject mail ) [Soma href="#hangup"> Disconnect suspicious SMTP clients
Under normal conditions, the Postfix SMTP server responds immediately when an SMTP client connects to it; the time to deliver mail is noticeable only with large messages. Performance degrades dramatically when the number of SMTP clients exceeds the number of Postfix SMTP server processes. When an SMTP client connects while all Postfix SMTP server processes are busy, the client must wait until a server process becomes available.
SMTP server overload may be caused by a surge of legitimate mail (example: a DNS registrar opens a new zone for registrations), by mistake (mail explosion caused by a forwarding loop) or by malice (worm outbreak, botnet, or other illegitimate activity).
Symptoms of Postfix SMTP server overload are:
Remote SMTP clients experience a long delay before Postfix sends the "220 hostname.example.com ESMTP Postfix" greeting.
NOTE: Broken DNS configurations can also cause lengthy delays before Postfix sends "220 hostname.example.com ...". These delays also exist when Postfix is NOT overloaded.
NOTE: To avoid "overload" delays for end-user mail clients, enable the "submission" service entry in master.cf (present since Postfix 2.1), and tell users to connect to this instead of the public SMTP service.
The Postfix SMTP server logs an increased number of "lost connection after CONNECT" events. This happens because remote SMTP clients disconnect before Postfix answers the connection.
NOTE: A portscan for open SMTP ports can also result in "lost connection ..." logfile messages.
Postfix 2.3 and later logs a warning that all server ports are busy:
Oct 3 20:39:27 spike postfix/master[28905]: warning: service "smtp" (25) has reached its process limit "30": new clients may experience noticeable delays Oct 3 20:39:27 spike postfix/master[28905]: warning: to avoid this condition, increase the process count in master.cf or reduce the service time per client
Legitimate mail that doesn't get through during an episode of Postfix SMTP server overload is not necessarily lost. It should still arrive once the situation returns to normal, as long as the overload condition is temporary.
One measure to avoid the "all server processes busy" condition is to service more SMTP clients simultaneously. For this you need to increase the number of Postfix SMTP server processes. This will improve the responsiveness for remote SMTP clients, as long as the server machine has enough hardware and software resources to run the additional processes, and as long as the file system can keep up with the additional load.
You increase the number of SMTP server processes either by increasing the default_process_limit in main.cf (line 3 below), or by increasing the SMTP server's "maxproc" field in master.cf (line 10 below). Either way, you need to issue a "postfix reload" command to make the change effective.
Process limits above 1000 require Postfix version 2.4 or later, and an operating system that supports kernel-based event filters (BSD kqueue(2), Linux epoll(4), or Solaris /dev/poll).
More processes use more memory. You can reduce the Postfix memory footprint by using cdb: lookup tables instead of Berkeley DB's hash: or btree: tables.
1 /etc/postfix/main.cf: 2 # Raise the global process limit, 100 since Postfix 2.0. 3 default_process_limit = 200 4 5 /etc/postfix/master.cf: 6 # ============================================================= 7 # service type private unpriv chroot wakeup maxproc command 8 # ============================================================= 9 # Raise the SMTP service process limit only. 10 smtp inet n - n - 200 smtpd
NOTE: older versions of the SMTPD_POLICY_README document contain a mistake: they configure a fixed number of policy daemon processes. When you raise the SMTP server's "maxproc" field in master.cf, SMTP server processes will report problems when connecting to policy server processes, because there aren't enough of them. Examples of errors are "connection refused" or "operation timed out".
To fix, edit master.cf and specify a zero "maxproc" field in all policy server entries; see line 6 in the example below. Issue a "postfix reload" command to make the change effective.
1 /etc/postfix/master.cf: 2 # ============================================================= 3 # service type private unpriv chroot wakeup maxproc command 4 #