#ident "@(#)smail:RELEASE-3_2_0_121:CHANGES,v 1.147 2004/08/23 00:51:52 woods Exp" ---------------------------------------------------------------------------- SMAIL RELEASE RELEASE-3_2_0_121 This is Smail, release RELEASE-3_2_0_121. If you are new to Smail: Welcome! This file is essentially a compendium of release notes for the current and previous releases of Smail. If you have been running previous releases of Smail it is _critical_ that you read down through this file until you come to the notes regarding the version you are, or were, running. If you've been running an intermediate release since 3.2 then please do a 'diff -b' on the revision of this file from the version you were running and read all of the updates that have happened since your release was made. There are _very_ important differences between this release and previous releases that may affect the proper operation of Smail in your environment. I most strongly recommend that the default configuration be used wherever possible -- i.e. delete any old configuration files you don't need and delete all entries from 'config' that are un-important to your environment. The new default configurations include many improvements over the old examples from previous releases and indeed some old examples are not compatible with the current code. If you're running a simple "leaf" Internet node for one or a few local domains you likely only need a 'config' file with only a very few custom settings such as "domains", possibly "visible_name" and "more_hostnames", and maybe "smtp_accept_max" and "smtp_accept_queue". Some of the new security related settings may also be of interest. CHANGES IN THIS RELEASE (RELEASE-3_2_0_121) The following list is an overview of the changes since 3.2. See other lists further down in this file to discover what happened before that. Security: - The SMTP command parser has been fixed to implement a command line length limit (of the semi-arbitrary length of 1024 bytes) as specified in RFC-821 section 5.2.3 (page 42) and extended by RFC-1869 section 4.1.2 (page 3). This will prevent at least the major opportunity for remote Denial of Service attacks against smail. A warning will be logged for command lines sent that exceed this this limit. The remainder of an excessive line will be read at a rate of one character per second and flushed (which does mean that a sender can continue to send until the connection times out within smtp_receive_command_timeout). - use of setrlimit(), where available, has been made more robust. Data and stack segment sizes are always given "reasonable" limits. This should help prevent unexpected Denial of Service attacks, local or remote. - fopen_as_user() has again been re-written and stress tested in a number of environments -- it's safer than even 3.2, and it now works properly in more environments and should be more portable. - a new config variable "smtp_remote_allow" can be used to prevent third-parties from using your site for spoofing and/or as a (spam) relay. Note this doesn't stop your site from receiving spam directed to your users. It only prevents outside spammers from using your mailer as a relay to blast spam out at others. It can be a list of host and/or IP network numbers (in CIDR or inet_net_pton(3) notation) for clients you trust to relay through your server. The default setting is "localnet" which will match the "classical" IP network of the local address (or any source address of localhost) so if you have one full Class A, B, or C network, or even if your mail host is multi-homed on more than one full class-full network, the default configuration will prevent unauthorised third-party e-mail relay. WARNING: You will have to add a correct local definition for smtp_remote_allow if you have many local networks, or unset it if you wish to disable all relaying. If you have POP and/or IMAP users who want to relay through your mailer while they are remotely using their e-mail (i.e. while they are using someone else's network access and thus won't be matched in your local networks) then you might consider installing Eugene Crosser's WHOSON daemon and patching your POP/IMAP server(s) to record all POP/IMAP accesses in the WHOSON database and adding the magic keyword "whoson" to "smtp_remote_allow" so that smail will also permit remote relaying of e-mail from those users. NOTE: This feature blocks all third-party relay by resolving input addresses to determine if they are locally deliverable, deliverable to a host the local host MXs for, or if they're to be delivered remotely then the transport driver name is checked to ensure that it's not "tcpsmtp". This implies that remote SMTP clients can send to UUCP connected hosts, and if those UUCP neighbours permit incoming mail to be routed to some host that will forward it out via SMTP then 3rd party relay will still be possible. It is recommended that all UUCP hosts be configured as leaf nodes only and that they refuse to accept incoming mail that will again be delivered to a remote host, regardless of which transport is used. An upcoming Smail beta may offer a simple way of configuring a UUCP node such that it will only accept incoming mail from remote sites to be delivered locally. [This feature is based on the ideas contained in patches posted to smail3-users by Gray Watson ] - a related new variable, "smtp_local_sender_restrict", does basic sender address anti-spoofing protection by ensuring that any sender address which is locally deliverable come from an authorised client (i.e. the client's source address must match "smtp_remote_allow"). WARNING: If you host virtual domains for users who don't always use your mail server to relay their outgoing mail then you'll need to disable this feature so that they can send e-mail to themselves from some other relay server. Some future implementation may allow you to list which domains to do such checks on, or alternately optionally allow you to only do them for domains in $hostnames and $more_hostnames and not in domains handled by the "rewrite" router and similar. - RFC-822 header anti-spoofing can be simulated by simply forcing all "sender:" and "from:" headers to be re-written by using the transport attribute "remove_header" and either "insert_header" or "append_header" as desired. For example I think these might work [untested :-)] remove_header="${if:and{{origin:remote}{dest:remote}}:{from}}", remove_header="${if:and{{origin:remote}{dest:remote}}:{sender}}", insert_header="${if:and{{origin:remote}{dest:remote}}:{$from_field}}" - A new config variable "smtp_rbl_domains" can be used to prevent known spammers or other miscreants from sending what are almost certainly unwanted messages to your server via SMTP. The connecting client's IP address is converted into a "dotted quad" representation, the octets are reversed, and each listed domain name is appended to the result and looked up in the DNS (in much the same way PTR records are retrieved from the in-addr.arpa zone). If an A record is found the client will be rejected with a 550 status message and any corresponding TXT records will be retrieved and included in the text of that message. One such list is the MAPS RBL, available in DNS form suitable for use with Smail. The MAPS DUL is another similar list that's available for blocking direct connections from dynamic-IP ports. You can also create your own zones instead of, or in addition to, using the 'smtp_hello_reject_hosts' list. Unfortunately until all the mailers in the world are secure from theft-of-service attacks (i.e. safe from unauthorised relay (ab)use) this won't work 100%, but it sometimes helps. In the mean time you can also block all mail from known open relays using any or all of the ORBS derivatives and/or the MAPS RSS. A companion variable "smtp_rbl_except" allows you to list host and/or network IP numbers that you do not want to look up in any of the RBLs specified in smtp_rbl_domains, or reject because of such listings. - A new config variable "smtp_sender_rhsbl_domains" can be used to prevent connections from sites who try to use a sender domain with a target domain that can be found as a DNS A record in any of the listed domains. One set of such lists is at rfc-ignorant.com. You can also create your own zones to do sender-address domain rejection. A companion variable "smtp_sender_rhsbl_except" allows you to list domains that are explicitly permitted (and which automatically bypass the RHSBL lookup. - Known bad or undesirable sender addresses can be specified as a list of host domain name regular expressions in the new variable "smtp_sender_reject", or listed in an alias file style of database specified by the pair of new variables "smtp_sender_reject_db" and "smtp_sender_reject_db_proto". By default the latter pair of variables point to a list of sender addresses that is updated by the checkerr(8) script based on frozen double-bounce messages containing patterns matching known junk (viruses, etc.). Note that recipient addresses matching in "smtp_sender_reject" are also rejected since it's unlikely these addresses are deliverable and even if they are then their owners would not be able to send a reply back through the same gateway host. - A new config variable "smtp_max_recipients" with a default value of 100 limits the maximum number of recipients that can be specified in a single SMTP message at one time. If too many recipients are specified each over the maximum will be rejected with an SMTP message 452. I.e. if you don't trust your own users you can limit them with this. The default value is the minimum allowed by RFC 2821. A companion variable, "smtp_max_bounce_recipients", controls how many recipients can be specified when a null ("<>") sender address is given. This helps stop spammers from chunking junk at us as if it were a big bounce. By default this is of course just one (1) since normally only one recipient can ever be specified for a bounce message since there's only ever one SMTP envelope address to which a bounce can be returned to. WARNING: A common counter-example to this is bounces from mail-ing lists, or indeed bounces to any mailbox which may be aliased to more than one person at the same remote domain. Of course this only matters if this particular mailer instance is handling that remote domain (and if the remote mailer on the list host will attempt to send one message to multiple SMTP recipi-ents at a time). - a new pair of config variables "smtp_hello_verify" and "smtp_hello_verify_literal" are used to enable verification of the HELO/EHLO hostnames and domain literals against the client's peer address. NOTE: both are set on by default. You probably want to read the next paragraphs before losing your cool about this. A companion variable "smtp_hello_broken_allow" lets you specify a list of host and/or network IP numbers for selected hosts which send broken HELO/EHLO greetings and which are not likely to be fixed before too much mail bounces to their users (e.g. HOTMAIL.COM and MSN.com, etc. -- complaints should go to the "postmaster" mailbox at the appropriate domain). If you are an ISP you may also have to list your own dial-up nets if your users have broken mail clients or are incompetent at configuring their mailers. Note that connections with the same local socket and peer end-point addresses, i.e. those which originate on the local host, are always allowed to violate the HELO/EHLO rules. Note also that syntax errors in the HELO/EHLO command and its parameters are never tolerated. Further DNS configuration conformance of greeting names can be forced by turning on a new variable "smtp_hello_verify_ptr". See the smailconf(5) manual page for a discussion regarding RFC conformance vs. intent. HELO/EHLO errors are logged as "questionable" values even if they don't cause a rejected connection so that you can get an idea of just how good or how broken your neighbours mailer and DNS configurations are. WARNING: If you don't do anti-spoof IP filtering on your routers some of these verifications will be rendered useless! EVERYONE should do anti-spoof IP filtering at their borders. DNS spoofing is caught if the default setting of the new variable "smtp_hello_reject_dns_paranoid" is maintained. This setting causes smail to do DNS consistency checks in the same way as the TCP Wrappers "PARANOID" check. Note that Smail is also now very careful about what records it uses from a DNS reply -- it will ignore records for domains that it did not query for. Smail also now normally allows full maximum-sized DNS replies (64KB) in order to avoid dropping records from truncated replies. - As an alternative to strict HELO/EHLO verification, "smtp_hello_reject_hostnames" lets you specify a list of host domain name regular expressions which you want to reject even if you've disabled "smtp_hello_verify". The default list includes all the reserved names, as well as the expansion of "hostnames" so as to prevent forgers from trying to claim their clients are your local host. Client hosts matching the "smtp_hello_broken_allow", as well as of course those originating on the local host, are exempted from this check. The default list identifies known invalid and all reserved top level and second level domains. - There's a similar variable called "smtp_host_reject_hostnames" which specifies hostnames resolved from PTRs that should be rejected. By default this list is the same as that used in "smtp_hello_reject_hostnames". There are no exemptions from this list -- you have to fix your PTRs or change the list. The default list identifies known invalid and all reserved top level and second level domains. - The SMTP envelope sender address (i.e. MAIL FROM argument) is verified unless the client address is listed in the new variable smtp_sender_no_verify. In the case of mail received by TCP/SMTP the sender's domain name must have a valid DNS MX RR or A RR. If the target domain of the sender address is a locally handled one, or if the lowest precedence MX host for the target domain is locally handled, then the sender address is also verified to be deliverable too. A new config variable "smtp_sender_verify_mx_only" can be set to force only MX lookups (i.e. strict SMTP conformance, no RFC 1123 or RFC 974 leniency of allowing A RRs). - Since spammers are increasingly playing lower-level tricks with DNS Smail now checks the target address(es) of MX target hostnames against the list of IP/network numbers given in "smtp_bad_mx_targets". If you operate a gateway to/from a private network then you can list your internally handled MX domains and hosts in the associated "smtp_bad_mx_except" list. - Smail now fully honours the RFC requirements that all MX target names "MUST" be canonical host domain names, unless the new variable "allow_one_mx_target_cname_hack" is set in which case Smail will follow a single CNAME alias hop while looking for the address(es) of MX target hosts, but if and only if the desired A record does appear in the "additional" section of the DNS reply to the original MX query. Regardless of whether or not this flag is enabled each errant MX target will be logged and reported in the daily error check. - Smail can now control which domains are allowed to use it as a backup MX by specifying their host domain name regular expressions in "smtp_permit_mx_backup". By default smail will continue to relay to all higher-priority primary MX hosts for those domains which list it as a lower-priority MX. All backup MX support can be disabled by setting this list to "!.*". - A new set of config variables "smtp_error_delay", "smtp_expn_delay", and "smtp_vrfy_delay" are provided to control rate-limiting of EXPN and VRFY commands as well as the rate at which SMTP error responses will be delivered. "smtp_error_delay" defaults to 60 seconds (just a fraction of the maximum limit required by the RFCs). The latter two are set by default to ten seconds so as not to annoy humans doing debugging too much but to prevent automated dictionary-style address harvesting attacks and other kinds of stupid abuses of your mail server. The "smtp_error_delay" takes effect before the last line of any error response is given to the client so that some types of DoS attacks have far less impact. All of these delays help force attackers to waste more of their time. Some people call this kind of response rate limiting a "tar pit" -- it slows down a mis-behaved client because the client will then be forced to waiting for what it just thinks is a slow server. - a new pair of config variables, "header_checks" and "body_checks" allow you to specify a list of regular expressions which will be matched against the RFC 822 headers and the first "message_buf_size" bytes of the body, causing the message to be rejected if any match is made. This match is done just after the message has been successfully and completely read from the source it is arriving from, and just before the spool file is sync'ed to disk. However note that if against all recommendations you allow huge messages, and if you create some overly complex REs, then extra I/Os will be required to re-buffer the first chunk of the message and scanning may take longer than the sender may allow for while waiting for confirmation that the message has been queued. This feature is a very sharp and double-eged knife -- use it with extreme caution. If you use overly simplistic REs then you will reject mail that you probably don't want to reject. To avoid too much embarrassment these checks are ignored for message recipients listed in the new config variable "bypass_content_filtering_recipients", which by default matches any or mailbox. If you really want to block certain mail to your mailbox as well (such as Microsoft executable attachments, anti-virus notifications, etc.) then you can use a similar pair of config variables which are never bypassed: "header_checks_always" and "body_checks_always". - on systems which have a setusercontext() library function (e.g. most *BSDs), this call is used by the "pipe" driver to make sure that the child process obeys system specified limits for the user-ID it runs as. - Finally Smail can enforce max_message_size on all traffic. Even if the client ignores ESMTP SIZE we'll still stop them dead, though only after sucking up all the data they trhow at us. Too many common MUAs still don't use ESMTP and lusers with high-speed connections (and even some without) haven't got a clue about the resources large messages require to handle. Hopefully this'll help teach a few more to use FTP or HTTP or similar for their file transfers. WARNING: due to the way smail calculates the amount of free space available early in each SMTP session it must reserve enough space for (smtp_accept_max * max_message_size). If you get "not enough spool space" errors and don't know why then try setting more reasonable values for these two variables. Build configuration: - Smail now requires the PCRE library. A copy of the PCRE source is included in the "pcre" sub-directory. - conf/Make.local must now be created in addition to conf/EDITME - conf/lib/mkdepend.sh no longer modifies Makefiles in place but instead writes the additional dependency information to ".depend" files in each directory and these files are removed by "make clobber". Though some versions of "make" will read these files automatically by default, all the "make" invocations done when building from the root of the source tree will explicitly specify these files on the command-line, if they exist and whenever their use is important. - on older system you will need to install a recent BIND-8 and link with the resolver library it supplies. Smail now effectively (though not explicitly) requires the API provided only in the more recent BIND-8 and BIND-9 resolver libraries. Note that all older BIND resolvers have remote code-executuion vulnerabilities and should NEVER be used anywhere near any public network. - the insecure HAVE_SETEUID feature is no longer supported. - the NOBODY config variable was changed to SMAIL_NOBODY to avoid further conflict with Solaris-2.x (SunOS-5.x). This may also fix a related problem with SCO UNIX. - all of the Solaris-2.x (i.e. SunOS-5.x) configurations have been merged into one: solaris2. [fully tested on 5.4 and 5.5, compiled and minimally tested on 5.7 and 5.8, and fully tested on 5.9, including on sun4m and sun4u, though there are still problems with IOB_MAYBE_EMPTY_P() if you use 'gcc -m64'] - Smail is fully 64-bit ready! It has been fully tested on NetBSD/sparc64 (i.e. with a full 64-bit userland on a big-endian system with picky alignment restrictions), as well as on NetBSD/alpha (a little-endian system with picky alignment restrictions). - the sco3.2 (v4) configurations have been re-written and tested. They *should* work adequately for SCO UNIX 3.2v5 too. - hard-coded (re)setting of the MAKE macro in all Makefiles has been removed to facilitate using makes of other names. This will cause makes which don't set MAKE by default (or use environment variables) to lose, and the best option may be to obtain a newer make for your system, such as GNU Make (or 4.4BSD make). - an OSTYPE for LynxOS Real-Time un*x has been added: lynxos. - the rewrite router is included in the default configuration. - the default inet_addrs router (which uses the gethostbyaddr router driver) has the 'always' attribute set to reduce bogus DNS queries by the following bind_hosts router when gethostbyaddr finds a match. - several failings of the TEST_BASE configuration option have been fixed (but it's still far from perfect). - BSDI-2.x and IRIX-5.x have been updated and tested. - messy handling of "#!" vs. the pipe transport driver's command setting has been moved to config.h where it belonged all along. - A linux-glibc2 OS configuration has been added that uses DB style DBM libraries. Note that glibc2 seems to have 32-bit user-ids and may encounter problems if BOGUS_*ID values are not explicitly overridden by a local smail_nobody setting. Note also that glibc2 has a broken gethostbyaddr() that only returns one PTR, and the rest of its DNS resolver interface lies about its version. Always use BIND's resolver library! - there's a new "arpa-only" driver configuration which is like "arpa-network" but without pathalias, uuname, and queryprogram routers (i.e. without the basis for UUCP support. It can be used in conjunction with '-DNO_UUCP_TRANSPORTS'. Miscellaneous: - minor porting bug fixes for SysVr4, Linux, SunOS 4+5, etc. - lots of improvements to debug (-v N) output, better and more meaningful error reporting, esp. for DNS and database lookups. - memory leaks in the open_database() methods have been closed, making the NIS/YP code work more reliably again. - upgraded pathalias to the latest version 10 with ability to handle larger maps, internet connectivity (-I option), arpatxt de-commissioned. Added Mark Moreas' parse-only option (-p). There's rumoured to be a bug in this version of pathalias, but of course such issues should be taken up with the pathalias maintainer. - smart-host routing now happens for completely un-qualified host names if auth_domains contains an empty string. - smtp_accept_max and smtp_accept_queue should now work on most systems without ISO C compilers, even with optimization. - Note that the smtp_remote_allow option has been implemented by effectively permanently turning on what was once an un-documented compile time option: VERIFY_RCPTS which forced all RCPT TO address to be verified thus determining whether or not they are to be locally delivered. There doesn't seem to have be any adverse affect to this other than the additional internal processing required may be significant for large mailing lists -- it should only change the way bounces are generated. Note that recipient addresses must be verified even if queue_only is set as otherwise there's no way to see if they will be local or remote. Another associated new config variable, "smtp_recipient_no_verify", can be used to specify addresses of, or networks containing, local dumb MUA clients that deliver by SMTP but need a real bounce message, not a 400/500 error. [This will also fix problems for clients that use ESMTP pipelining but then don't multiply their timeout by the number of addresses they've submitted, such as Lotus Notes and problems with clients that ignore some errors, such as Novell Groupwise which blasts past 400 errors and sends the DATA anyway, though it notes the 400 and retries the message leading to possibly many duplicates if the defer condition is cleared.] WARNING: hosts permitted by smtp_recipient_no_verify implicitly bypass the smtp_remote_allow checks! DO NOT turn off verification for remote clients that you do not control network access for lest you make your mailer into a multi-stage open relay! NOTE: Although "large" lists should probably be handled by list processing software (LISTSERV or its many clones such as tulp, or Majordomo, or Smartlist, etc.) which would avoid the extra cost of processing IN RCPT TO: verification, it may still make sense to add a flag to the address structure that could be used to terminate processing as soon as the address is determined to be local (i.e. passed to a director). This would emulate the old "verify" functions without unnecessary code duplication and still ensure that remote addresses were passed all the way through the routers to determine the transport and thus still make it possible to block third-party relay on a per-transport basis as we now do with smtp_remote_allow. - The un-documented NO_VERIFY option has been renamed to the still un-documented NO_SMTP_EXPN option, which makes it possible to permanently disable EXPN at compile time. - the smtp_info variable has been renamed smtp_allow_expn and now only affects EXPN (VRFY, as required by RFC-1123, is always enabled). A compatibility alias remains for smtp_info. - the smtp_debug variable has been renamed smtp_allow_debug. A compatability alias remains for smtp_debug. - Duplicate addresses are now checked for in two stages -- first as literal duplicates without parsing right down to the mailbox, and then secondly after directing and routing is done by target address and transport name. This once and for all fixes the problem where virtual hosts using the directors domains attribute (for example) could not receive copies of messages that were also delivered to local users with the same mailbox name. - various security violations and other interesting events are which are logged to the normal logfile will match the regular expression ": remote [A-Z ]*:". These include such things as EXPN's, HELO/EHLO violations, MAIL FROM violations, RCPT TO and VRFY violations, etc., as well as DEBUG and EXPN events of interest. Note that if the log entry flags a warning, and not a violation, then the word "questionable" will be used instead of "invalid", and the operation will have been allowed. - the "forwardfile" director driver now always opens the address list file as the nobody user unless the target address is associated with a local user. This should only mean that regardless of how mail is addressed to the default "lists" director, the mailing list file will be opened as nobody, but all other directors using "forwardfile" (e.g. "dotforward") should continue as before and open the file as the user. This fixes the problem where sometimes list expansion would fail on operating systems which return an error for set{g,u}id(-1). - courtesy all the new SMTP verification features, the "true" sender hostname may be available from the PTR lookup on the sender address and if so is included in the new default Received: header if and only if it differs from the hostname offered by the HELO/EHLO command. - the "true" sender hostname is also included in the new-style "Received" log entries (in the HOST: field). It is enclosed in parenthesis following the sender hostname given by the remote host if and only if it is different. Note also that the space character that once separated the hostname from the sender's IP address literal is gone. This should make simple awk parsing of the Received log entries a bit easier, but if you once depended on this space you'll have to fix your log parser. Note that as before any portion of the entry could go missing if unavailable. - various dangling pointers have been tromped on and lots of generic lint has been picked off. - all the manual pages have been kept up to date and have also undergone significant editing and improvement. - Reply-To: (and of course Resent-Reply-To:) is now qualified. - normal daemon shutdown by SIGTERM is now logged. - slightly better and more correct (to RFC 821) handling of SMTP error replies, esp. w.r.t. messages with multiple recipients. - support for SunOS-4 malloc debugging and the Debugging Malloc Library by Mark Moraes has been added. See INSTALL and conf/EDITME-dist for more information. - fix some memory corruption that's been happening since the LSEARCH_REGEXP stuff was introduced. - SMTP source route addresses now have the appropriate element stripped from the address handed to the "next" host (RFC 821). - the gethostbyaddr router driver is now a bit smarter at how it handles the check_if_local flag by only doing gethostbyaddr() calls if the flag is set and checking all the host aliases too, and finally when it does find a local host match it creates a "parent" address so that the original is not lost in the case it is needed for error logs, etc. - the gethostbyaddr router driver no longer tries to return the name of the host in place of the literal IP address even if one was found in the check_if_local test -- this makes it more certain that the user's request to use a literal IP address won't get thwarted by inconsistent /etc/hosts files on those backwards systems that still don't use *only* the DNS for gethostbyaddr(). - the code for starting child processes has been significantly improved with far more extensive error checking and reporting. Note that this may cause failures in some existing configurations which appeared to work despite being incorrect. - the rather ancient, but also seemingly rarely used, director attribute "ignore_alias_match" has been documented. - the "owner" attribute for aliasfile director drivers has been fixed so that it is properly expanded. They now work as they should and as they do in forwardfile (e.g. used in the lists director) driver (they should now work for all director drivers). Note that the aliasfile code is still rather ugly and in-elegant -- the other drivers are much cleaner. - a numeric sign handling bug has been fixed in the internal "safe" str_printf() function. - mkaliases now tries to check for errors and use exit codes. mkline (used by mkaliases to parse alias files) now reports the key for entries with parsing errors making it easier to track down typos in alias files. - SMTP responses have been made more verbose and error responses in particular should now be more helpful and explanitory. - Two new variables have been added to assist with proper configuration on multi-homed machines: "listen_name" and "sending_name". - There is now better handling for multi-line SMTP replies, particularly of errors that need to be logged and reported in bounce messages. - The manual pages have been converted to use just the well-known V7 man(7) macros [which have always been more than adequate enough for documents such as these!] - The rewrite router now accepts multiple addresses in the right-hand field (making it much more compatible with normal alias files for better support of "virtual" domains, etc. - ESMTP ENHANCEDSTATUSCODES option is now supported (special thanks to Lyndon Nerenberg for doing this work!). - runq has been made more efficient -- it no longer delays when a host retry file is already locked and instead goes on to do real work right away. On busy sites that have to run lots of such processes deliveries should now be smoother. - handling of SMTP retry files has been "fixed" to use the string returned by inet_ntoa(3) for the file name (unless the destination is a smart-host, in which case the target domain name is used). This makes it possible for Smail to properly try all addresses listed for an MX target hostname, and to properly record the status of each of those individual machines. This should drastically improve Smail's ability to get mail through to domains which have many MX hosts and which often have "problems" with the most preferred hosts. Like Postfix though Smail will stop processing after finding any host which speaks SMTP, even if it just says "421 go away". - the checkerr script has been made more robust and now also cleans out stale retry files and reports statistics from yesterday's logfile. - most types of systems can now avoid over-filling spool_dirs space when receving messages via SMTP. SMTP clients which ignore ESMTP SIZE limits (or which don't use ESMTP) will not be able to use up disk space either (their messages will be rejected and deleted after they send the "." part of the DATA command). - recipient and sender addresses are now more strictly verified to honour RFC 821 syntax. - smail will now quote any sender and recipient addresses it sends as an SMTP client, as per the requirements of RFC 821. - a new 'mailq -E' option has been implemented to provide an easy way to examine messages that have been punted to the error queue, and a new script "unfreezemail" has been provided to make it easier to reprocess any failed deliveries for any messages in the error queue. - "mailq" now always prints a summary of both queues (but of course not if only explicit queue IDs are examined). - a new 'mailq -s' option has been implemented to avoid printing all the message details and instead print just the summary tallies. - mailq now shows the QueueSize for each message. - mailq shows the recipients of the message in a separate field. - mailq now also shows the recipients when a message uses '-t'. - a more-or-less standard PID file is now created when the daemon process is started. - the old "smtp" transport instance has been renamed to "inet_zone_smtp". - the distribution includes a "standard" portable rc/init.d style startup/shutdown script. - smail now accurately keeps track of smtpd and runq child processes on all supported systems except those derived from UNIX System V Release 3.x and which have not had a POSIX compatible sigaction(2) system call added to them (and currently smail will not work at all on those old systems). - runq children are killed when smail is signalled with SIGHUP and all children (smtpd and runq) are killed when smail is shut down with SIGTERM. - a new config variable "runq_max" can be used to limit the number of runq daemon processes that are running at any given time. - when signals are handled, or when child processes have died because of a signale, the event is logged with the signal name, not the number. - config file parsing has bene rationalised somewhat to allow unquoted strings to contain whitespace (only the leading whitespace on continued multi-line entries will be stripped) as well as to allow semicolon (;), equals (=), and comma (,) characters without requiring quoting. The C-style character escape mechanism has been extended to interpret "\s" as a space character and to interpret an escaped newline as license to strip the newline and any following whitespace (and a normal "\n" can of course still be used to include a newline in the string value). - variables which support host domain name and full-featured PCRE regular expressions allow comment/message sub-fields will honour backslash-escaped field separators, though of course since these must first be accepted by the input parser which does its own backslash escaping these escapes must be entered as double backslashes. Note that the comment/message sub-fields in these two list types extend to the end of the entry so the sub-field separator character does not have to be escaped. - variables which accept lists of IP numbers in CIDR notation, host domain name regular expressions, and full-featured PCRE regular expressions, now support negated items by prefixing them with and exclamation mark (!). The first matching item terminates the search, and if the matching item was negated then the search fails. - there's a new meta-expansion modifier called "foreach" which takes a variable name and a string as parameters. It iterates over the colon-separated list specified by the variable name and and appends the given string to each item thus allowing message/comment sub-fields to be expanded onto plain hostname lists when they're included in other lists. - there's a new meta-expansion modifier called "rxquote" which quotes all the special charaters in a string so that when used as a regular expression it will only match the exact original string. - the user director driver "prefix" attribute has been supplemented with a companion "suffix" attribute and both have been converted to be processed as regular expressions. The strings matching their values are stripped from the mailbox local part before it is matched against local usernames, but the stripped values are made available as variables for transport drivers, as well as being stored in new environment variables exported by the pipe transport driver. - there's an undocumented compile-time option for configuring the local transport drivers to support the Cyrus IMAP daemon which uses the new director driver "suffix" feature to parse out optional mailbox names from recipient addresses to support the Cyrus "deliver -m mailbox" feature. - the user director driver "ignore-case" attribute has been deprecated and a new global "ignore_user_case" config variable takes its place in order to finally fix the long standing bugs preventing caseless local-part matching of usernames. This new variable causes all directors to ignore the case of user names. Note that using this feature can have very adverse performance effects on busy mail servers, particularly if the user database is accessed via NIS/YP or some similar network protocol since the entire database may have to be scanned for every match attempt. - there's a new default "owner" director which is an instance of a "smartuser" director driver. It re-directs all unmatched "owner-*" addresses to the postmaster. - there's a new "error" director driver, and a new default "error" director entry using this driver. This driver matches any addresses with the following form and causes them to generate a permanent or temporary error as appropriate: :fail:"Error message text" :defer:"Defer message text" This makes it possible to have control over the error text for any address which is to be rejected. This can be used to give meaningful bounce messages for unknown addresses at virtual domains handled by the "rewrite" router, as well as other special tasks, such as providing a way to give immediate bounces for Cyrus IMAP mailboxes that are known to be over-quota. - many other minor fixes have been made to the default configuration attributes. - 'smail -bP' can now expand run-time variable, not just configuration attributes. - the internal string formatting subroutine has been enhanced to support printing of string values with a syntax compatible with the config file input parser so now in places where the output is expected to be a single token and/or to only be printable ASCII characters (e.g. especially in log files), backslash quoting will be used. [note that not all placed needing such encoding have been found and fixed yet] - 'smail -bP' now prints lists with nicely cut&paste-able newlines for easier readability and re-use. - the default "routers" configuration has been re-organized to to make various routing decisions in RFC mandated ways and to improve other common specialized uses. For example the default "smart_host" entry has been moved to the top of the list so that it can work properly. Now it really is possible to enable the smarthost driver just by defining the appropriate value for the "smart_path" variable. - a new "eval:" meta-expansion modifier is available. It can be useful in the "rewrite" file to allow expansion of variable substitutions found in database lookup results. - new "ltic{}" and "gtic{}" conditionals are available for use with "${if" - all the old, convoluted, unsupported, and now pretty much useless "UK Greybook" support has been completely removed. - there's a new "logsumm" script which generates summary reports from the main log file. It's run by the "checkerr" script and it's report is included in the daily report mailed to the local postmaster. - the "pipe" transport driver now has a great deal more control over how it responds to various `abnormal' conditions through the new "handle_write_errors", "defer_child_exitcodes", "bounce_child_exitcodes", and "ignore_child_exitcodes", which replace the old similar but far more confusing and less flexible attributes. - addresses given on the command line, as well as addresses input per line to "smail -bt", are now treated exactly the same as if they were given on a destination header in the message (e.g. for use with "smail -t"), and they are always fully qualified. This fixes many past confusions. - all delivery attempts are now logged in the per-message logfile, as is output from any "pipe" transport driver command, and "mailq -v" only shows the most recent deferral entries. This means the per-message logfile format has changed, though old files will be handled with the exception that details included in bounce messages of old messages might not look quite right, so you might want to flush your input queue and clean out your error queue before you upgrade. - a new variable "smtp_mua_only_hosts" has been added to allow you to block MUA-only clients from using a null return path. This eliminates fallout from ignorant users using poorly designed software to fake bounces of spam and the like. However it also blocks RFC 3798 MDNs, but that might be a "Good Thing(tm)" too. - the format of the timestamp used in log files and similar such places has been changed to conform more closely by default to ISO 8601 (i.e. it is now "YYYY/MM/DD HH:MM:SS"). The old format can be maintained by defining USE_OLD_TIMESTAMP. The only programs that should be affected would be those that sorted log records by timestamp, but no such programs are included with Smail. GNATS PRS Closed: - only PRs which resulted in changes to the code are listed here. There were other PRs closed which were classified as mistaken, duplicate, support requests, new feature change requests, etc. (Note that some of these PRs only apply to beta releases, but in cases where they had a major impact they are mentioned here.) PR# Category Class Synopsis ----- -------------- ------- --------------------------------------------------- 41 smail-security sw-bug Flawed security fixes 101 smail-security sw-bug Security problems related to pipe driver and smartuser 236 smail-bugs sw-bug Does not generate Resent-Date:. 241 smail-bugs sw-bug fopen_as_user(fn, "w", ...) will not create the file fn. 245 smail sw-bug without HAVE_SETEUID, fopen_as_user doesn't create files 244 smail-bugs sw-bug bug in parsing `debug' command in SMTP conversation 246 smail-bugs sw-bug "debug #" returns "bad number: {garbage}" 247 smail sw-bug wrong FORWARDTO_FILE if LOCAL_MAIL_FILE undefined 248 smail-docs doc-bug Obsolete cross-references 249 smail-docs doc-bug Typos 250 smail-docs doc-bug More typos 251 smail-bugs sw-bug bug in calling `mkaliases' program (with `-bi' switch) 253 smail-ports sw-bug fixes for smail-3.2 on FreeBSD 2.x 254 smail-bugs sw-bug smail-3.2 src/sysdep.c bug 255 smail-bugs sw-bug smail 3.2, remote from (null) problem 256 smail-bugs sw-bug appendfile fails to creat new files 257 smail-ports sw-bug Smail 3.2: support for SCO OSR5 262 smail-bugs sw-bug ERR_133, if file doesn't exist and HAVE_SETEUID is undefined 263 smail-ports sw-bug Linux-configuration is incorrect. 264 smail-bugs sw-bug smail cannot create a mailbox file if SETEUID not available 266 smail-bugs sw-bug Given an invalid queue_interval, smail may use it or crash 267 smail-enhance change Linux has many features that are not used by conf/os/linux 268 smail-docs doc-bug config manpage says max_message_size not implemented 269 smail-bugs sw-bug Seg Fault in receive_smtp() expanding unknown variable 270 smail-enhance change patch to 3.2 to deny mail spams 275 smail-bugs sw-bug unsharmap core dumps on Linux. (err should be static) 276 smail-bugs sw-bug compilation fails on domain.c. (missed vprintf()s) 277 smail sw-bug smail breaks sender addresses "" 278 smail-bugs sw-bug smail pd/pathalias fails compile on linux-2.0.29 and netbsd-1.2 280 smail sw-bug append_header claimed to be read-only 283 smail doc-bug Strange 'I' construction in man5/smail.an 284 smail-bugs sw-bug Unexpected closure of incoming connection after RCPT TO: 289 smail-enhance change anti-spam feature 292 smail-bugs sw-bug Extra Sender: is generated. 293 smail-docs doc-bug Default value of trusted_users is wrong in smailconf(5). 296 smail sw-bug dirp was being closed (thus freed) twice. 299 smail-port sw-bug INADDRSZ not defined here 300 smail sw-bug spelling error 303 smail-bugs sw-bug problems using hstrerror() and NEED_HSTRERROR 308 smail sw-bug COPY_STRING macro called with function arg 312 smail sw-bug checkerr PATH? 315 smail sw-bug Compilation failures when DRIVER_CONFIGURATION=bsd-network 317 smail sw-bug WIFSIGNALED(status) == TRUE for normal 3B2 SVr3 exit 323 smail sw-bug buffer overflow in smailconf.c 325 smail sw-bug Compile error in src/sysdep.c 327 smail sw-bug transports/tcpsmtp.c need hstrerror from smtprecv.c 330 smail sw-bug match_ip fails for pattern "*" in some cases 331 smail sw-bug class B addresses not matched with smtp_remote_allow 334 smail change strip prefix from $user in for the owner attribute in fwdfile.c 341 smail sw-bug smail generates spurious 'lock failed' entries in logfile 352 smail sw-bug Sender adress verification fails when sender is on local host 355 smail-security sw-bug Security problem with rmail 361 smail sw-bug gateway statemnet not functioning 366 smail sw-bug smail dumps core in dn_expand() for empty MX lookup responses 368 smail sw-bug rewrite router can lose email 370 smail sw-bug Bug in resolve.c with address like host!user@local.host.domain 374 smail change wrong address of FSF in COPYING file 378 smail sw-bug Needed to change hstrerror return type to const char * 408 smail sw-bug smart_host router with method attribute: host given by path is looked up 448 smail sw-bug compute_nobody() needs to be called in more places 460 smail doc-bug typo in src/silly.c - a list of open PRs can be obtained by sending e-mail to with a subject line consisting of the string "query-pr -qx". Other options to query-pr(1) can also be used to refine the query, or to obtain the full text of a PR. Please see the GNATS documentation for more information about the query-pr command. ---------------------------------------------------------------------------- CHANGES IN THE 3.2 RELEASE WARNING: you will have to re-generate your conf/EDITME file from scratch. The conf/EDITME files from previous versions are not at all supported!!! WARNING: Your old config files may need some minor fixes -- some superfluous variables have been removed, and there are new options. If you are a leaf Internet node, try running with *NO* config files. Security: - few os configs use HAVE=SETEUID any more -- it is very dangerous. - uses 4.4bsd's more secure /usr/libexec/mail.local where possible. - many minor bug fixes to prevent core dumps, etc. Configuration: - removed the check_path option from the appendfile transport driver, and from the include director (it wasn't implemented by fopen_as_user() anyway); also removed the (redundant -- it should have been check_path anyway) checkpath attribute from the forwardinclude driver. - version numbering and tracking support vastly simplified.... this implies some variables, such as patch_number, patch_date, and bat, have been eliminated Build Mechanisms: - support for 'make sources' and 'make nuke' (i.e. RCS/SCCS support) was deprecated and no longer working so has been removed from most Makefiles [we use CVS to manage sources] - Makefiles now expect to be run by either a SysVr2 or later augmented make, or a modern BSD make (4.4bsd), or by GNU Make; this is due to dependence on command-line argument passing via environment variables [old v7 derived/compatible makes will not work, though there may be some way to set MAKE on their command line which may mimic the new behaviour] Features: - "mailq -v" now accepts a numerical argument and as expected increases the debug level; the default of '1' mimics the previous behaviour. - It is now possible to use regular expressions for lsearch lookups in databases such as the aliases file. This feature is made use of by enclosing the expression (i.e. the left most field of the alias entry) in double quotes ("). See the MISC_DEFINES section in conf/EDITME-dist. - BERKNET and DECNET addressing (the silly old host:user and host::user) forms are no longer supported by default. If you are extremely sure you need these, they can be turned on. Read the source to find out how. DECNET should *never* have been supported, and BERKNET is so dead it's not funny. Be warned though that this code is no longer tested, and indeed may create parsing problems with other valid address forms which use ':' characters as syntax elements. - Multi-homed machines (i.e. machines with more than one routed interface), can now specify "listen_name" to force the SMTP daemon to listen only on a given interface. - new director "altuser": directs mail as per the user director, using an alternate passwd format file. This file may be accessed by any of the standard search methods, and so the interface looks just like that of aliasfile. - At long last RFC-1123 conformance for dates in headers (and logs), i.e. 4-digit years everywhere. Thanks to NUMATA, Toshinori for pointing this out in a patch submitted to comp.mail.smail on 7 June 1996. Portability: - more cleanup in the conf/os defaults to allow configuration with extremely simplified EDITME (defaults ala "vendor" config). - much lint removed from the code - beginnings of use of more portable C types, such as use of off_t, size_t, etc. See INSTALL for porting details. - support for 64bit architectures (Nigel Metheringham) Documentation: - the smail.5 manual page, split up during the 3.1.30 period, has been permanently split into separate manual pages, one to discuss each main configuration file. These new pages retain the "smail" prefix in their names to help keep them somewhat identifiable. ---------------------------------------------------------------------------- CHANGES IN THE 3.1.30.13 RELEASE NOTE: this list may not be complete.... Bug fixes: PR#46 - Xenix portability fixes - steve@nshore.org (Stephen J. Walick) PR#63 - Prefix string for appendfile transport was not expanded - rmwise@mcigate.apdev.cs.mci.com (Bob_Wise) PR#67 - Log message in fwdfile.c derefed a possibly NULL ptr - seveal people including root@tetrarch.mpd.co.za (J. Kean Johnston) PR#51 - sender_host_addr not derived for incoming SMTP via inetd PR#49 - wrong type used for optarg in pd/getopt/getopt.c PR#81 - statp never initialised by nialias routines (caused security failures). PR#45 - changed all gets() to fgets() for security PR#30 - Extra newline in log messages made selectable on LOG_EXTRA_NEWLINE PR#115 - Minor fix to strict header rewriting to prevent double qualification PR#93 - Buffers not flushed in no verify mode (smtp receive) PR#145 - Some SMTP temporary faults were marked as errors and not retried PR#95 - Sometimes a 5xx code in SMTP was not acted upon correctly PR#106 - Fixed comments in ltoival (string.c) PR#107 - Fixed comments in EDITME-dist on retry file format PR#108 - hash_predelivered_address could deref null pointer PR#132 - Corrected call in queryprog.c (could core dump) PR#88 - Fixes to field.c and enhancement to bindlib.c PR#158 - Make checks for type of file occur before locking in forwardfile director PR#165 - Fix to strcmpic to handle NULL strings PR#166 - Fix to route.c to fix core dump with address like ]user@domain PR#178 - Bind did not call res_init() - needed for bind 4.9.3 PR#182 - Transport running seteuid could try to open msglog files. PR#193 - Fast machines could issue duplicate message IDs. PR#181 - Unchecked fstats could lead to infinite loop PR#155 - cosmetic changes to locking debug messages PR#98 - broken security checks in fwdfile.c removed PR#205 - PR#203 broke some scripts by quoting values, now only does so if required. PR#120 - Finally got parse.c in pd/pathalias to build correctly! PR#225 - Smail -t on message with only bcc produced non rfc822 message - now fixed PR#229 - smail -bd (no -q int) never closed log files or checked configs for updates - now fixed TEMP - Backed out part of PR#45 since it prevented compilation!! Portability PR#49 - HP-UX 9.0 PR#125 - Added SETGROUPS to the sunos 4 configs PR#45 - BSD 4.4 (not quite complete). PR#90 - Added HDB UUCP to linux PR#164 - BSDI portability changes PR#153 - Contributed Solaris 2.3/2.3 ports PR#200 - Reordering of includes in appendfile.c to prevent problems on some systems. Enhancements PR#26 - Arguments to runq/mailq etc are interpreted as queue files to process. PR#86 - log attempts to SMTP to verify unknown addresses PR#87 - Make SMTP help reflect commands available PR#159 - Make locking in forwardfile director optional PR#141 - Rewrite router (Alan Barrett ) added PR#203 - Can now dump config files out using -bP PR#204 - Error handling changes - see smarthost and pipe PR#5 - changes to list error handing, addition of domains attribute to directors PR#101 - shell quote (shquote:) meta expansion put in for better security PR#210 - New meta-expansions - shquote, gt, lt. PR#211 - SMTP EXPN/VRFY only work if smtp_info config variable is set PR#10 - ESMTP support PR#62 - resolve_timeout times out things defered in directors/routers PR#219 - Smail generated date fields now include seconds PR#220 - Date fields now use numeric timezones with alphabetic as comment PR#226 - Smail now derives its domain on startup if possible. PR#227 - Man page changes [PART DONE AT PRESENT] Changes PR#5 - Removed GLOTZNET code - no longer needed PR#6 - Rmemoved Peter Honeyman code from direct.c TODO BEFORE RELEASE PR#141 - Rewrite router needs documenting. - partially done, please check PR#227 - Complete man page changes. ---------------------------------------------------------------------------- CHANGES IN THE 3.1.29.1 RELEASE This is a minor patch update to 3.1.29 fixing a set of bugs and providing support for one new OS. * The sense of the dns_search option was inverted in the bind router. This caused some sites behind wildcard MX records to have real problems. * Patch to IRIX 5.x config file, as sent with Announcements * A very minor security bug in the aliasfile director has been fixed. I haven't worked out a way of using this bug usefully so I think it was not a real security risk. * The use of #error broke some compilers. A declaration that was in error has also been fixed. * Support for AUX 3.x in Posix environment has been added. * A bit has been added to README about RFC1413 Thanks to those contributing - the major names I have are:- Olaf Kirch Jay Gaeta Jim Jagielski Lyndon Nerenberg ---------------------------------------------------------------------------- SPECIAL NOTE EDITME files from 3.1.28 *should* work to configure this release, however the EDITME-dist file has been expanded and updated significantly. It would be best if old configurations were applied to the new EDITME-dist file to produce a new EDITME file. Please read the notes in the EDITME-dist file. ---------------------------------------------------------------------------- CHANGES IN THE 3.1.29 RELEASE * Bug fixes for portability which co-incidently fix an unknown security hole! Thanks to Philip Hazel and Ian Kluft. * Bug fix to SMTP/retry code, also merged in earlier fixes to the SMTP/retry code which had not made the main code tree. * Various changes from tron including new log format (switchable at compile time - see SMAIL_LOG_STYLE), internet format from_ lines, fixed smtpd banners etc. [MAN PAGES NOT YET UPDATED] The 3.1.28 log format is the default for this release, but the log format default WILL change in the next release. * Added fixes for 2 major security bugs, one in forwardfile handling, one in DEBUG file handling, both allowing arbitary files to be created by any user on some or all platforms. * New RFC1413 code - replacing old TAP patches. See man pages for details. Also new structure allows other identification schemes to be added more easily. * Add in modified bindlib code (from Nigel Metheringham). Sigificant extra functionality and more modular design. * Integrate in patches to speed up queue reprocessing by hashing out addresses that have already failed/succeeded delivery before they are passed to the routers. This is a big win for mailing lists using the bind router. * Integrate in changes for queue grades. This makes it possible to determine which grades of mail are delivered immediately, and which are just queued for later processing. The grades processed by any one queue run can also be configured. This adds 2 config variables - delivery_grades & runq_grades, and one command line flag - -oG (documented in smail.5/8). * Add in quote metavariable function (previous documented, but unimplemented) * Add additional config files, update EDITME for new configs. CONTRIBUTORS TO RELEASE 3.1.29 Contributors include:- Philip Hazel, Ronald S Karr, Ian Kluft and others from the smail development team. The source distribution was co-ordinated by Nigel Metheringham, who is undoubtably responsible for any added errors. I apologise for any contributors I missed - some of the patches date back 2 years and have no listed contributor. ---------------------------------------------------------------------------- CHANGES IN THE 3.1.28 RELEASE Changes in this release * Add retry file lock timeouts, so that one process getting a lock when another process has a retry file lock will not wait forever. This is important for preventing blocking of queue-runs as a result of a single message transfer waiting for an extended time out. This was contributed by Chip Salzenberg, as an extension to his original host retry management support. * New router driver: rerouter, submitted by Uwe Doering. This new router can reroute UUCP-zone paths for greater efficiency, or to correct known routing defects. It can also be used, in a limited mode, for rerouting bounce messages. See the driver source, and the smail(5) man page for details. This code is experimental, and I have some reservations concerning some of its semantics, so use at your own risk. * Changed the format for log entries generated by smail. The new log entry formats are courtesy of Uwe Doering. I am not quite sure if I like them, but they are quite a lot more readable than the old ones. However, grepping is more difficult, given that the entries now take several lines. * Finally changed write_log and panic functions to use ANSI C-style variable-argument declarations, rather than traditional C. The particular usage of variable-argument functions in smail was causing problems with some C compilers. * Add INCLUDE_UTIME_H to the sun_os4 configuration file. This is needed to get struct utimbuf. * Add seteuid/setegid for HP-UX 8.0 as macros that use setresuid/setresgid. This allows use of shared NFS mounted mailbox directories. * Add SETEUID to HAVE list for sys5.4. * Add CPPFLAGS of "-systype bsd43" to mips-bsd43 configuration file, which is needed to get correct include files for BSD-universe compilation. * Fix spelling of tm_zone structure tag for use with MIPS and NeXT machines, or other machines that can get the timezone from struct tm. * Removed C++/G++ support from the sdbm.h include file. This was causing problems compiling on some systems, and is uneeded in the smail build environment, in any case. CONTRIBUTORS TO RELEASE 3.1.28 Contributors to this release include Knut-Hevard Aksnes, Neal Becker, Kevin Darcy, Uwe Doering, Hillel Markowitz, and Chip Salzenberg. ---------------------------------------------------------------------------- CHANGES IN THE 3.1.27 RELEASE Release 3.1.27 is a patch release to smail3.1.26. This release is primarily a bug fix, portability-enhancement release. The total size of the patch is larger than I expected it to be, based on the number of bugs fixed, but the changes aren't very intrusive, so users with custom enhancements should not have too many problems integrating them into this release. Also, at the time of the 3.1.27 release, I have a reasonably large backlog of changes that I intend to include in a near-term release. However, some of the fixes in this release are of sufficiently high priority that I don't want to delay the release further to include any more changes. Changes in this release: * A bug in the DNS lookup routines was fixed which was causing core dumps in dn_expand(). * The SHELL variable can now be set on the make command line. This allows the use of alternate, /bin/sh-compatible shells. Some systems (notably Ultrix and Xenix) have /bin/sh bugs which can be worked around by using a different shell. I have also tried to reduce the complexity of some shell scripts, in hopes that more native shells will work correctly without needing to introduce workarounds. * Add an AUTH_DOMAINS list to the EDITME file (alternately, auth_domains in the config file) for setting a list of domains for which your machine is authoritative. This prevents the smarthost router from matching non-existing hosts in domains for which your host has complete routing information. I included this in this release, despite the fact that it is an enhancement, because the change is small and because interactions with the smarthost router have caused significant confusion lately. paths files can be used for similar purposes, but smail has come to be used more extensively on pure Internet machines, which often lack paths files. Explaining why paths files must be used has become more trouble for me than it is worth. * The visible_domains and more_hostnames config file variables can now be set with the names "domains" and "gateway_names" for compatibility with the EDITME file. Again, explaining the differences here were becoming more trouble than the trouble of just making this change. * Some problems with uses of the tolower() function have (I hope) been fixed. Unfortunately (I guess), I don't have any non-POSIX systems any more, so I can't tell if my fixes are sufficient. * Some changes to the motorolla delta system configuration file were contributed by Francesco Potorti` . * Updates to the A/UX samples and conf/os files were added, as contributed by Bob Denny. * A typo in the next2.0 configuration file was fixed. Contributors working on NeXT systems have been driving me a bit nuts, given that their editor appears to add newlines and blank lines in the middle of patch files. I usually have to apply their patch files by hand and I haven't always managed to compensate. * A typo in the sun_os4 configuration file was fixed. This one was purely my fault; I can't blame it on anyone else. * Nigel Metheringham contributed conf/os/mips-bsd4.3. * Some problems with timezones were (hopefully) corrected for NeXT machines. * I completely rewrote conf/os/isc2.2.1. I ripped out all of the POSIX features. Attempts to use POSIX features ran into too many header file problems. Also, what worked with the version of ISC that I have access to didn't work with isc3.0. The new file should be sufficiently generic to work with most recent releases of Interactive. I have NOT tested this with gcc. If this configuration file doesn't work with gcc, then don't use gcc. * For systems that have the dbm.h or ndbm.h file in an awkward location, the MISC_DEFINES variables DBM_INCLUDE and NDBM_INCLUDE can be set to indicate the include paths to use. See conf/os/template for details. * A size_t reference in string.c, which was causing problems compiling on a number of systems, was changed to an "int". * The Date header now indicates (by default) the time that the spool file was created, rather than the time of message delivery. * Some typos in the smail(5) man page, with respect to UK-only bind attributes, were fixed. The attribute names should now appear when you format the man page. * I changed some of the scripts in samples/bsmtp to work better in the face of queue directories that exceed ARGMAX. Some alternate scripts were contributed which I may use instead, in a future release. * I changed the sender_proto (the name supplied after the "with" keyword in Received lines) for batched-SMTP input to "bsmtp". Compressed, batched input is now "cbsmtp". * Several bugs were fixed in the samples/generic/* file. In particular, for directors a missing ':' was added and a comment in the "lists" entry describing sender_okay was corrected. For transports, notify_comsat was added to the default "local" transport. * The program invocation caused by invoking smail as "newaliases" or with the "-bi" option, was changed to reset the effective user and group IDs to the real user and group IDs. The previous release did (correctly) reset the uid and gid if -oA was used to specify an alternate aliases file, but it probably allowed any user to rebuild the regular alias file. I say probably, because a bug was causing the uid and gid to be set to random stack garbage, which must have been 0 on most systems that tried the feature. * The accept() call on some systems (notably some SVR4 variants, though not the one I use) fail if descriptors 0 and 1 aren't in use. Smail now opens /dev/null on descriptors 0 and 1, when operating as a daemon. * The spool file format was changed (in an upward compatible, though not backward-compatible fashion) so that blank arguments do not screw up the argument list stored in spool files. In earlier releases, a blank argument, such as -F "", could corrupt the spool file, since it used a blank line to separate an argument list from the header list. I am rather surprised that this bug has managed to stay around so long. It is an original design flaw. * The HELO, RCPT TO, and MAIL FROM requests in SMTP now require a non-empty argument. However, the argument to HELO is still not checked for validity. * An extraneous utimbuf declaration was removed from src/sysdep.c that was causing difficulties with compilation on some systems. * The files contrib/micnet/*.dif were removed. These patch files were no longer relevent, since the files that they were relative to have changed substantially. * Split changes into a separate file from the README file. It is likely that the README file will split further in future releases. Some users have commented the the README file has become a bit unwieldy. A short list of changes not included in 3.1.27, which will likely be included in 3.1.28: * Some authentication of the sending hostname, based on IP addresses versus the name supplied in the HELO line to SMTP. * Some enhancements to the message-retry/message-timeout logic, including logic to remove stale files. For now, the following shell command can be used periodically to remove stale files: find /usr/spool/smail/retry -type -f -mtime +7 -exec rm {} \; Removing files older than the message-retry interval is reasonably safe, since message-timeout logic is based on the modify time of a message spool file, not on retry files. * A rerouter driver, submitted by Uwe Doering. To remain consistent with previous statements, I must restate that I dislike hosts that do auto-rerouting. However, this appears to be necessary on the European continent, given the rather arbitrary and beaurocratic nature of goings on there. * Significant changes to the logging code, to make log entries much more readable, if less grep-able. The changes were submitted by Uwe Doering. It will be relatively trivial to convert log files for use with grep. Perl scripts should have little problem with the new format, although be warned that all existing perl scripts will have to be rewritten for use with the next release. The smaillog program will be modified to account for the new format, since it is included in the release. CONTRIBUTORS TO RELEASE 3.1.27 Contributors to this release include Bob Denny, Uwe Doering, Greg Hackney, Philip Hazel, Ron Heiby, Nigel Metheringham, Jim O'Connor, Chip Salzenberg, Brian Taylor, Bill Trost, and Stephen J. Walick. ---------------------------------------------------------------------------- CHANGES IN THE 3.1.26 RELEASE Release 3.1.26 is a patch release to smail3.1.25. This release is primarily a functionality enhancement release. A number of bugs have been fixes as well. The total number of changes is quite large. As a result, people with customizations which were not included in 3.1.26 may have a fair amount of work to do. Important enhancements in this release: * Greatly enhanced expansion strings, with conditionals and file lookups. The original version of this code was written by Chip Salzenberg. * Per-transport-configurable header insertion and removal based on the new expansion code. In addition, the "From:" header can now be configured, including the ability to use a file to find a long form for local users (e.g, to get ronald.s.karr). The configurable header insertion/removal support can be used to support the Content-Length field, by adding the following generic attributes to the local transport: remove_header="Content-Length", append_header="Content-Length: $body_size" Of some value with SVR4, you can also add a Content-Type header field, if no Content-Type field already exists, with: append_header="${if !header:Content-Type Content-Type: text}" * The default Received: header now contains the "from" and "with" keywords, where such information is available. Information is determined from the HELO line in SMTP, the From_ line in regular uucp mail, or it can be set using the -oMr and -oMs options, for compatibility with sendmail. * Smail can now limit connections to the SMTP server, as long as a standalone smail SMTP daemon is used, rather than using inetd to start smtpd per connection attempt. Two levels are provided: a maximum number of connections that yield immediate message delivery, and a maximum number of allowed connections. This can greatly help the maximum load that smail places on a server, while allowing fast delivery if the server has light mail activity. This code was contributed by Chip Salzenberg. * Smail now tracks host accessibility for SMTP delivery. If a host is down, then mail to that host will be deferred immediately (rather than waiting for a connection timeout) until a definable time period expires. In addition, mail that is undeliverable for a definable time period will be bounced, rather than hanging around in the smail queues forever. The parameters for retry times and message timeouts are configurable on a per-host or per-domain basis. Several implementations of these capabilities were submitted, including implementations by Syd Weinstein and Dan Danz. I decided on a version by Chip Salzenberg, which included per-host configurability, and which (eventually) supported the host accessibility tracking. * Interactive SMTP receive processes can now timeout. By default, SMTP command receipt times out after 5 minutes, and receipt of a message after a DATA statement times out after 2 hours. This was added after an urgent, but gentle, request of Heiko Schlichting, who had 100 receiver processes hanging around waiting forever for SMTP input. The specific timeouts can be modified in the run-time config file by setting smtp_receive_command_timeout (default 5m) and smtp_receive_message_timeout (default 2h). * Support was added for JANET reverse-order domain lookups in the bind router. Contact Philip Hazel for complete support. The smail3 release contains only that code that needs to be within the smail binary itself. * Added support for NeXT netinfo databases (from Dan Danz). * The DNS can now be queried within the tcpsmtp transport, without involving the bind router driver. In previous releases, MX records could be used only if addresses were resolved by the bind router. Now, under configuration control, the tcpsmtp transport driver can locate MX records itself, such as for mail routed by a local paths file or by a queryprogram router. This support was added, at my request, by Chip Salzenberg. * Added a -oX flag to smail that sets the TCP port number used to listen for SMTP connections. This is helpful in testing, because this (along with the "service" attribute to the tcpsmtp driver) can be used to setup a test network that operates independently of an existing SMTP network. This can be used to make sure that smail works on your network before replacing an existing mailer. * The smail installation process now attempts to preserve replaced binaries from your original operating system in .SAV files. Thus, your existing /bin/rmail and /usr/lib/sendmail will not be removed, even if smail is reinstalled several times. Previous releases would (sometimes) keep one backup version (with a .O suffix), which would then be removed on the next install. The new process makes a .SAV backup file, if one does not exist already, allowing a one-time backup copy on the first installation of smail. .SAV backup files are made only for files that are considered likely to provided in the base operating system. * The uuwho utility now uses searches for domain information, if a full hostname match is not found. For example, the command: uuwho mitsu.veritas.com will find the entry for .veritas.com, which does exist while no entry for mitsu.veritas.com exists. This was added by Chip Salzenberg. * By "popular demand" :include: in mailing list and alias files can now be followed by white-space. This is for compatibility with sendmail. Comments are NOT allowed after :include:. For example, the following line: :include: /list-directory/mailing-list would previously have resulted in errors, but will now result in inclusion of addresses in the file /list-directory/mailing-list. * The -bi option from sendmail is now supported, and newaliases is now yet another link to smail. This support was added to allow existing YP makefiles (which call /usr/lib/sendmail to rebuild the YP mail.aliases dbm file) to use smail3 replacing sendmail. The original version of this enhancement was submitted by Bruce Jerrick. * The sdbm library is now integrated in the smail release. Systems that do not have a native ndbm library now use the sdbm functions to provide multiple DBM-like files. Native dbm (not ndbm) libraries can be used only if they support the dbmclose() function. This was added by Chip Salzenberg, who was planning to use this in association with his host status monitoring code, but later realized the error of his ways. But, the changes were left in because they seemed reasonable. * The comsat daemon is now supported in the appendfile driver, and is enabled, by default, in the "local" transport. * Mail which results in all recipients being eliminated (such as through duplicate elimination or sender elimination within a mailing list), will generate a bounce message. Some other changes: * The sender_okay flag (for configuring removal of the sender from aliasing and mailing list expansion) can now be configured from the EDITME file. * Pathalias was updated to match the most recent release available on uunet. * The compile process now echoes the results of variable substitition. The previous release would generate lines such as: . ./defs.sh; $CC $CFLAGS $INCLUDES -c foo.c now, compiles will generate something like: gcc -O -I/usr/include -c foo.c make -n will still yield annoyingly small amounts of real information. * A sufficiently high debugging level will now enable debugging in the resolver library. * Included some minor revisions to the bind router by Syd Weinstein, to fetch A records as a second pass after fetching MX records. Apparently, there are differences in what different servers return in a request for MX records (some return A records, too, some don't). * Sytems that require an alternate set of #include files to compile programs that use TCP/IP and sockets can define those #include files in the conf/os files or in the EDITME file. See conf/os/template for details. * Added some AUX support contributed by Robert B. Denny, which can be found in samples/AUX-support. Most of the changes relate to printing documentation. Some sample configuration files are also included. * Smail now uses SO_REUSEADDR and some appropriate descriptor closes, to allow the smtp/queue-run daemon to be restarted cleanly even if a receiver process has been forked. Previous releases could get errors in the bind() call indicating an address already in use. * A new attribute was added to the gethostbyname router (only_local_domain) that disables matching of hosts outside of the local domain. This was addedy by Dan Danz. * Man pages were previously installed with mode 555. They are now installed with mode 444. * Added a NOTES directory containing messages that I have sent or received that illustrate various points of interest. * Building of uuwho databases now differentiates between a #N line introducing a hostname, and a #NOTE, or #North America line introducing a random comment. Recent map entries caused extra, bogus, entries to be added to uuwho databases. * "A" records are now used in the order returned by DNS resolver lookups. Previously releases would reverse the A records, due to the way these records were being added to a list, internally. The RFCs require that A records be used in order, since they may be sorted by network preference. * The local-* transports for local-form delivery to remote hosts now does proper hop-count checking. These transports use the new local_xform attribute which does local-form transformations without indicating that delivery is actually local. Some bug fixes: * Mail to systems with no MX records, but an A record, will now be handled correctly by the bind router. I have regular posted a patch for this to various newsgroups and mailing lists, but did not include the fix in an earlier official patch release. * Fixed some minor problems with bounce-message generation. Bounce messages are generated by reinvoking smail and sending the bounce messages as a batched SMTP script. The sender was specified as <>, in accordance with RFC822, which smail considered a remote address. However, treating this as mail from a remote system was inappropriate, so an alternate sender token <+>, is now used to indicate a locally-generated bounce message. * Nul-terminated some generated strings that were not consistently nul terminated. * Fixed some problems resulting from routers that resolve to the local host, requiring further routing of the remainder address. * Fixed some problems in the trusted-code logic that was running into problems with the SMTP interaction generated by MH. * Errors in .forward files now generate bounces to real- rather than to the postmaster. * NEWS_SPOOL_DIR was not being stored in the defs.h file, yielding incorrect behavior from uuwho. * Fixed some problems with interactions between the -t and -f options. * Fixed some problems with varargs usage in log.c. * Include for Suns, to avoid a problem interacting with -O on Sun4's that could generate core dumps. * Fixed some segmentaion violations caused by bogus @foo addresses. * Fixed most (all?) places where smail was using a value immediately after it was freed. This allows smail to work, unmodified, with the SVR3 -lmalloc library, or other versions of malloc that don't leave data from the last free unmodified. * get_sender_addr(), which generates the sender address in various important contexts, was returning the same data between multiple calls, when it should have been returning different data. CONTRIBUTORS TO RELEASE 3.1.26 This patch was made possible primarily through the work of Chip Salzenberg, who contributed the first version of the enhanced string expansion code (though I extended it rather a lot), the SMTP connection limiting code, the reorganized bind code, plus the changes for the retry file. He also contributed a large number of minor enhancements and bug fixes. I don't have a good record of who else contributed code, fixes and suggestions for this patch, but here is a partial list: Tomas Ahl, Jack Bailey, Alan P Barrett, Mark Bixby, Randy Bush, Dan Danz, Bob Denny, Jon Diekema, Michael Faurot, David J. Fiander, Bill Hargen, Philip Hazel, Bill Heiser, Peter Honeyman, Bruce Jerrick, Patrick Lee, Simon Leinen, Hillel Markowitz, Bill Masek, Jan-Piet Mens, Les Mikesell, Lyndon Nerenberg, Jim O'Connor, Jim Pickering, Steve Piette, Francesco Potorti`, Heiko Schlichting, David Schmidt, Monty Solomon, Brian Taylor, Bill Trost, Gary S. Trujillo, Stephen J. Walick, Lauren Weinstein, Syd Weinstein, and Andreas Wettengel.