OpenBSD Amavisd
6.3. Amavisd-new
Edit /etc/amavis/conf.d/15-content_filter_mode and ucomment antivirus and spam checking
use strict;
- You can modify this file to re-enable SPAM checking through spamassassin
- and to re-enable antivirus checking.
- Default antivirus checking mode
- Uncomment the two lines below to enable it back
@bypass_virus_checks_maps = (
\%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);
- Default SPAM checking mode
- Uncomment the two lines below to enable it back
@bypass_spam_checks_maps = (
\%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);
1; # ensure a defined return
To integrate Amavis with Postfix we need to add a content_filter in /etc/postfix/main.cf.
content_filter = smtp-amavis:[127.0.0.1]:10024
Open /etc/postfix/master.cf and add this at the end of the file:
smtp-amavis unix - - - - 2 smtp
-o smtp_data_done_timeout=1200
-o smtp_send_xforward_command=yes
-o disable_dns_lookups=yes
-o max_use=20
127.0.0.1:10025 inet n - - - - smtpd
-o content_filter=
-o local_recipient_maps=
-o relay_recipient_maps=
-o smtpd_restriction_classes=
-o smtpd_delay_reject=no
-o smtpd_client_restrictions=permit_mynetworks,reject
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o smtpd_data_restrictions=reject_unauth_pipelining
-o smtpd_end_of_data_restrictions=
-o mynetworks=127.0.0.0/8
-o smtpd_error_sleep_time=0
-o smtpd_soft_error_limit=1001
-o smtpd_hard_error_limit=1000
-o smtpd_client_connection_count_limit=0
-o smtpd_client_connection_rate_limit=0
-o receive_override_options=no_header_body_checks,no_unknown_recipient_checks
and add this immediately below the “pickup” transport service:
-o content_filter=
-o receive_override_options=no_header_body_checks
This will prevent messages that are generated to report on spam from being classified as spam.
Amavisd-new listens on port 1024 where Postfix sends all of the mail coming into the server. After processing the email amavisd-new returns the mail to Postfix on port 1025 for final delivery.
Edit /etc/amavis/conf.d/50-user and paste this into the file:
use strict;
@local_domains_acl = ( ".$mydomain" );
$sa_spam_subject_tag = 'SPAM > '; $sa_tag_level_deflt = -999; # add spam info headers if at, or above that level $sa_tag2_level_deflt = 5; # add 'spam detected' headers at that level $sa_kill_level_deflt = 12; # triggers spam evasive actions
$final_virus_destiny = D_DISCARD; # (data not lost, see virus quarantine) $final_banned_destiny = D_REJECT; # D_REJECT when front-end MTA $final_spam_destiny = D_DISCARD; $final_bad_header_destiny = D_PASS; # False-positive prone (for spam)
- ------------ Do not modify anything below this line -------------
1; # ensure a defined return
Of course, you do not just want to paste stuff into configuration files without knowing what are you actually doing. So let us go through the file, line by line:
@local_domains_acl = ( ".$mydomain" );
This is a list of domains for which this server considers itself as a final destination. If you do not add all your domains here, they will not be processed by amavis, so if you have more domains on your server, except for the default line, this should look something like this:
@local_domains_acl = ( ".$mydomain", "domain2.com", "domain3.org" );
Later, we will see how we can automate this to read the information from the database.
$sa_spam_subject_tag = 'SPAM > '; $sa_tag_level_deflt = -999; # add spam info headers if at, or above that level $sa_tag2_level_deflt = 5; # add 'spam detected' headers at that level $sa_kill_level_deflt = 12; # triggers spam evasive actions
When SpamAssassin processes message, it gives them a score identifying what is the probability of message being a spam. $sa_tag_level_deflt tells amavis that ig the score is greater or equal to this level that amavis should append X-Spam headers to the message. The score of -999 means that we want to apply X-Spam header to all of our messages, so it is easier for us to see why is, or why is not, a message considered spam by examining the headers of the message.
$sa_tag2_level_deflt = 5; is the spam level the message needs to reach for amavis to apply the header X-Spam-Flag: YES so we (our mail client) knows that this message is considered to be spam. At this level the message subject is prefixed with the value of the variable $sa_spam_subject_tag. In this case the subject is prefixed with “SPAM > ”.
$sa_kill_level_deflt holds the value of the spam level that message needs to reach to do something with the message. What we are going to do with messages that are spam depends of the value of $final_spam_destiny variable. In this case we are discarding this messages.
This means that the message is not going to reach to recipients mailbox at all. But that does not mean that the message is lost. Spam messages that are discarded can still be fetched from the system. Default configuration is to store them in the /var/lib/amavis/virusmails folder. It is possible to store them in the database.
For releasing quarantined mail you need to add these lines:
$inet_socket_port = [10024,9998]; $interface_policy{'9998'} = 'AM.PDP-INET'; $policy_bank{'AM.PDP-INET'} = {
protocol => 'AM.PDP', # select Amavis policy delegation protocol inet_acl => [qw( 127.0.0.1 [::1] )], # restrict access to these IP addresses
- auth_required_release => 0, # don't require secret_id for amavisd-release
};
If you want to store everything in the database you have to add these lines as well:
@lookup_sql_dsn = ( ['DBI:mysql:database=DATABASE-NAME;host=127.0.0.1;port=3306', 'USERNAME', 'PASSWORD']); @storage_sql_dsn = @lookup_sql_dsn;
$virus_quarantine_method = 'sql:'; $spam_quarantine_method = 'sql:'; $banned_files_quarantine_method = 'sql:'; $bad_header_quarantine_method = 'sql:';
For this to work, you will have to create the database in MySQL and change the value of @lookup_sql_dsn variable to match your database, username & password. You will have to create the tables for this database by looking at the amavis documentation). When you set this all up, you can use Mailzu or Postvis Admin to retrieve the quarantined messages. For users of SquirrelMail webmail system, there as a plugin called AmavisNewSQL which enables access for users to quarantined messages that were addressed to them. [Note] Note
Quarantined messages (on the file system, or in the database) have to be deleted periodically!
If you do not want to quarantine any of the messages, change
$virus_quarantine_method = 'sql:'; $spam_quarantine_method = 'sql:'; $banned_files_quarantine_method = 'sql:'; $bad_header_quarantine_method = 'sql:';
to
$virus_quarantine_method = undef; $spam_quarantine_method = undef; $banned_files_quarantine_method = undef; $bad_header_quarantine_method = undef;
SQL support for amavis, except for easier access to quarantined messages also provides features to change spam setting on a per user basis. Once you create the database, and configure @lookup_sql_dsn this part is very easy achieved by reading through the amavis documentation.
Messages that have a spam value between $sa_tag2_level_deflt and $sa_kill_level_deflt are delivered into the recipients mailbox and marked with “SPAM >” in the subject and have the X-Spam-Flag: YES header added to them. Although this may be enough for you, since I use only IMAP access to the mailboxes I wanted to move this messages to the Junk folder inside the users mailbox automatically. This has some benefits: Inbox will not be filled with spam messages, but in case of a false positive, user could check his Junk folder to check for messages marked as spam. This folder can also be used for learning SpamAssassing what is spam, and if users move the messages that are not recognized as spam to this folder instead of deleting them SpamAssassin will get smarter about what is considered to be spam, and what is not.
It's time to restart Postfix and Amavis and commit changes using etckeeper.
/etc/init.d/postfix restart /etc/init.d/amavis restart etckeeper commit "Configured amavisd-new"
6.3.1. Store configuration in a database
Documents regarding the use of amavisd-new with an SQL server are located in /usr/share/doc/amavisd-new. Use gunzip to uncompress the files ending with .gz.
When amavisd-new works with an SQL server it can use two sets of tables. The first part is read only and has the following tables: users, mailaddr, wblist and policy. The second part is read/write and is used when you store quarantined emails into the SQL. The tables in the second part are: maddr, msgs, msgrcpt, quarantine.
We are going to setup both sets of tables in the same database named mail_amavis. Enter mysql and run:
CREATE DATABASE mail_amavis; CREATE USER 'amavis'@'localhost' IDENTIFIED BY 'new_password'; GRANT ALL PRIVILEGES ON `mail_amavis` . * TO 'amavis'@'localhost'; FLUSH PRIVILEGES;
Create an amavis.sql file and paste this into it:
-- local users CREATE TABLE users (
id int unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY, -- unique id priority integer NOT NULL DEFAULT '7', -- sort field, 0 is low prior. policy_id integer unsigned NOT NULL DEFAULT '1', -- JOINs with policy.id email varbinary(255) NOT NULL UNIQUE, fullname varchar(255) DEFAULT NULL, -- not used by amavisd-new local char(1) -- Y/N (optional field, see note further down)
);
-- any e-mail address (non- rfc2822-quoted), external or local, -- used as senders in wblist CREATE TABLE mailaddr (
id int unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY, priority integer NOT NULL DEFAULT '7', -- 0 is low priority email varbinary(255) NOT NULL UNIQUE
);
-- per-recipient whitelist and/or blacklist, -- puts sender and recipient in relation wb (white or blacklisted sender) CREATE TABLE wblist (
rid integer unsigned NOT NULL, -- recipient: users.id sid integer unsigned NOT NULL, -- sender: mailaddr.id wb varchar(10) NOT NULL, -- W or Y / B or N / space=neutral / score PRIMARY KEY (rid,sid)
);
CREATE TABLE policy (
id int unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY,
-- 'id' this is the _only_ required field
policy_name varchar(32), -- not used by amavisd-new, a comment
virus_lover char(1) default NULL, -- Y/N spam_lover char(1) default NULL, -- Y/N banned_files_lover char(1) default NULL, -- Y/N bad_header_lover char(1) default NULL, -- Y/N
bypass_virus_checks char(1) default NULL, -- Y/N bypass_spam_checks char(1) default NULL, -- Y/N bypass_banned_checks char(1) default NULL, -- Y/N bypass_header_checks char(1) default NULL, -- Y/N
spam_modifies_subj char(1) default NULL, -- Y/N
virus_quarantine_to varchar(64) default NULL, spam_quarantine_to varchar(64) default NULL, banned_quarantine_to varchar(64) default NULL, bad_header_quarantine_to varchar(64) default NULL, clean_quarantine_to varchar(64) default NULL, other_quarantine_to varchar(64) default NULL,
spam_tag_level float default NULL, -- higher score inserts spam info headers
spam_tag2_level float default NULL, -- inserts 'declared spam' header fields
spam_kill_level float default NULL, -- higher score triggers evasive actions
-- e.g. reject/drop, quarantine, ...
-- (subject to final_spam_destiny setting)
spam_dsn_cutoff_level float default NULL,
spam_quarantine_cutoff_level float default NULL,
addr_extension_virus varchar(64) default NULL, addr_extension_spam varchar(64) default NULL, addr_extension_banned varchar(64) default NULL, addr_extension_bad_header varchar(64) default NULL,
warnvirusrecip char(1) default NULL, -- Y/N
warnbannedrecip char(1) default NULL, -- Y/N
warnbadhrecip char(1) default NULL, -- Y/N
newvirus_admin varchar(64) default NULL,
virus_admin varchar(64) default NULL,
banned_admin varchar(64) default NULL,
bad_header_admin varchar(64) default NULL,
spam_admin varchar(64) default NULL,
spam_subject_tag varchar(64) default NULL,
spam_subject_tag2 varchar(64) default NULL,
message_size_limit integer default NULL, -- max size in bytes, 0 disable
banned_rulenames varchar(64) default NULL -- comma-separated list of ...
-- names mapped through %banned_rules to actual banned_filename tables
);
-- R/W part of the dataset (optional) -- May reside in the same or in a separate database as lookups database; -- REQUIRES SUPPORT FOR TRANSACTIONS; specified in @storage_sql_dsn -- -- Wayne Smith adds: When using MySQL with InnoDB one might want to -- increase buffer size for both pool and log, and might also want -- to change flush settings for a little better performance. Example: -- innodb_buffer_pool_size = 384M -- innodb_log_buffer_size = 8M -- innodb_flush_log_at_trx_commit = 0 -- The big performance increase is the first two, the third just helps with -- lowering disk activity. Consider also adjusting the key_buffer_size.
-- provide unique id for each e-mail address, avoids storing copies CREATE TABLE maddr (
partition_tag integer DEFAULT 0, -- see $sql_partition_tag
id bigint unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY,
email varbinary(255) NOT NULL, -- full mail address
domain varchar(255) NOT NULL, -- only domain part of the email address
-- with subdomain fields in reverse
CONSTRAINT part_email UNIQUE (partition_tag,email)
) ENGINE=InnoDB;
-- information pertaining to each processed message as a whole; -- NOTE: records with NULL msgs.content should be ignored by utilities, -- as such records correspond to messages just being processes, or were lost -- NOTE: instead of a character field time_iso, one might prefer: -- time_iso TIMESTAMP NOT NULL DEFAULT 0, -- but the following MUST then be set in amavisd.conf: $timestamp_fmt_mysql=1 CREATE TABLE msgs (
partition_tag integer DEFAULT 0, -- see $sql_partition_tag
mail_id varbinary(12) NOT NULL PRIMARY KEY, -- long-term unique mail id
secret_id varbinary(12) DEFAULT , -- authorizes release of mail_id
am_id varchar(20) NOT NULL, -- id used in the log
time_num integer unsigned NOT NULL, -- rx_time: seconds since Unix epoch
time_iso char(16) NOT NULL, -- rx_time: ISO8601 UTC ascii time
sid bigint unsigned NOT NULL, -- sender: maddr.id
policy varchar(255) DEFAULT , -- policy bank path (like macro %p)
client_addr varchar(255) DEFAULT , -- SMTP client IP address (IPv4 or v6)
size integer unsigned NOT NULL, -- message size in bytes
content binary(1), -- content type: V/B/S/s/M/H/O/C:
-- virus/banned/spam(kill)/spammy(tag2)/bad-mime/bad-header/oversized/clean
-- is NULL on partially processed mail
-- use binary instead of char for case sensitivity ('S' != 's')
quar_type binary(1), -- quarantined as: ' '/F/Z/B/Q/M/L
-- none/file/zipfile/bsmtp/sql/
-- /mailbox(smtp)/mailbox(lmtp)
quar_loc varbinary(255) DEFAULT , -- quarantine location (e.g. file)
dsn_sent char(1), -- was DSN sent? Y/N/q (q=quenched)
spam_level float, -- SA spam level (no boosts)
message_id varchar(255) DEFAULT , -- mail Message-ID header field
from_addr varchar(255) DEFAULT , -- mail From header field, UTF8
subject varchar(255) DEFAULT , -- mail Subject header field, UTF8
host varchar(255) NOT NULL, -- hostname where amavisd is running
FOREIGN KEY (sid) REFERENCES maddr(id) ON DELETE RESTRICT
) ENGINE=InnoDB; CREATE INDEX msgs_idx_sid ON msgs (sid); CREATE INDEX msgs_idx_mess_id ON msgs (message_id); -- useful with pen pals CREATE INDEX msgs_idx_time_num ON msgs (time_num); -- alternatively when purging based on time_iso (instead of msgs_idx_time_num): -- CREATE INDEX msgs_idx_time_iso ON msgs (time_iso);
-- per-recipient information related to each processed message; -- NOTE: records in msgrcpt without corresponding msgs.mail_id record are -- orphaned and should be ignored and eventually deleted by external utilities CREATE TABLE msgrcpt (
partition_tag integer DEFAULT 0, -- see $sql_partition_tag
mail_id varbinary(12) NOT NULL, -- (must allow duplicates)
rid bigint unsigned NOT NULL, -- recipient: maddr.id (dupl. allowed)
ds char(1) NOT NULL, -- delivery status: P/R/B/D/T
-- pass/reject/bounce/discard/tempfail
rs char(1) NOT NULL, -- release status: initialized to ' '
bl char(1) DEFAULT ' ', -- sender blacklisted by this recip
wl char(1) DEFAULT ' ', -- sender whitelisted by this recip
bspam_level float, -- spam level + per-recip boost
smtp_resp varchar(255) DEFAULT , -- SMTP response given to MTA
FOREIGN KEY (rid) REFERENCES maddr(id) ON DELETE RESTRICT,
FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE
) ENGINE=InnoDB; CREATE INDEX msgrcpt_idx_mail_id ON msgrcpt (mail_id); CREATE INDEX msgrcpt_idx_rid ON msgrcpt (rid);
-- mail quarantine in SQL, enabled by $*_quarantine_method='sql:' -- NOTE: records in quarantine without corresponding msgs.mail_id record are -- orphaned and should be ignored and eventually deleted by external utilities CREATE TABLE quarantine (
partition_tag integer DEFAULT 0, -- see $sql_partition_tag mail_id varbinary(12) NOT NULL, -- long-term unique mail id chunk_ind integer unsigned NOT NULL, -- chunk number, starting with 1 mail_text blob NOT NULL, -- store mail as chunks of octets PRIMARY KEY (mail_id,chunk_ind), FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE
) ENGINE=InnoDB;
INSERT INTO policy (id, policy_name,
virus_lover, spam_lover, banned_files_lover, bad_header_lover, bypass_virus_checks, bypass_spam_checks, bypass_banned_checks, bypass_header_checks, spam_modifies_subj, spam_tag_level, spam_tag2_level, spam_kill_level) VALUES (1, 'Non-paying', 'N','N','N','N', 'Y','Y','Y','N', 'Y', 3.0, 7, 10), (2, 'Uncensored', 'Y','Y','Y','Y', 'N','N','N','N', 'N', 3.0, 999, 999), (3, 'Wants all spam','N','Y','N','N', 'N','N','N','N', 'Y', 3.0, 999, 999), (4, 'Wants viruses', 'Y','N','Y','Y', 'N','N','N','N', 'Y', 3.0, 6.9, 6.9), (5, 'Normal', 'N','N','N','N', 'N','N','N','N', 'Y', -999, 6.9, 9.0), (6, 'Trigger happy', 'N','N','N','N', 'N','N','N','N', 'Y', 3.0, 5, 5), (7, 'Permissive', 'N','N','N','Y', 'N','N','N','N', 'Y', 3.0, 10, 20);
To run this SQL:
mysql -p mail_amavis < amavis-schema.sql
Now that we have the database set we need to reconfigure amavisd-new.
Open /etc/amavis/conf.d/50-user and add/set the the information that amavis needs to connect to the database (replace new_passwd with the password you chose for the amavis user):
@lookup_sql_dsn = ( ['DBI:mysql:database=mail_amavis;host=127.0.0.1;port=3306', 'amavis', 'new_passwd']); @storage_sql_dsn = @lookup_sql_dsn;
If you decided to split read and read/write parts of the amavis database you will need to set @storage_sql_dsn with the correct options.
As we mentiond before, we were using @local_domains_acl = ( ".$mydomain", "domain2.com", "domain3.org" ); to tell amavis to run spam and virus checks for mail addressed to one of this domains. This is a problem because every time we add another domain we have to edit the configuration file to add the new domain to the list of local domains.
To solve this problem we have two options. One is to just check all of the incoming mail to the system for spam and viruses regardless of the recipients domaint. We can do this by rewriting the query that amavis uses to check if the email address is local to the system:
$sql_select_policy = 'SELECT "Y" AS local, 1 AS id;
But, this is not the recommended way. I didn't even test this, so your milage may vary. The recommended way is to just insert every domain you host in the users table.
For example:
mysql> USE mail_amavis; Database changed INSERT INTO users VALUES ( NULL, 5, 5, '@example.com', NULL, 'Y');
To explain what this does, first lets take a look at the users table now:
mysql> select * from users; +----+----------+-----------+--------------+----------+-------+ | id | priority | policy_id | email | fullname | local | +----+----------+-----------+--------------+----------+-------+ | 1 | 5 | 5 | @example.com | NULL | Y | +----+----------+-----------+--------------+----------+-------+
This raw in the table will match all email directed to any email address @example.com domain (that is because the local is set to Y). If we add another raw to the table that has email set to someuser@example.com and priority is higher than 5 this users policy will have precedance over the domains policy.
And finaly the policy_id. In this example it says that all the email directed to @example.com domain will use a policy with an identifier 5. If you take a look at the amavis-schema.sql file we used to create the database for amavis, at the end we also inserted some policies, and the policy with an ID 5 was inserted like this:
INSERT INTO policy (id, policy_name,
virus_lover, spam_lover, banned_files_lover, bad_header_lover, bypass_virus_checks, bypass_spam_checks, bypass_banned_checks, bypass_header_checks, spam_modifies_subj, spam_tag_level, spam_tag2_level, spam_kill_level) VALUES (5, 'Normal', 'N','N','N','N', 'N','N','N','N', 'Y', -999, 6.9, 9);
We will skeep through most of the options (you have amavisd-new documentation if you want to go into detail) but we will look at the last three numbers inserted. spam_tag_level, spam_tag2_level and spam_kill_level are set to 3.0, 6.9 and 8.0 respectively. This means that emails that are flagged with over -999 (that should be all emails) by SpamAssassin will have X-Spam headers added to them. Emails tagged over 6.9 will have "SPAM" added to the subject and emails tagged over 9 will trigger spam evasive actions (depends on what you have $final_spam_destiny set to in the config file /etc/amavis/conf.d/50-user).
If you still want to set the same settings for all of the domains you can add a catchall user to the users table that will use the policy with ID 5.
INSERT INTO users VALUES (NULL, 0, 5, '@.', NULL, 'Y');
6.3.2. Maintenance
You will have to cleanup the records in the MySQL from time to time, so we are going to create a PHP script that is going to be run daily with CRON.
- !/usr/bin/php
<?php
/**
- Amavisd-new cleanup script
- Based on the Amavisd-new documentation shipped with amavisd-new package in
- Debian 6.0 (Squeeze). Version of Amavisd-new 2.6.4.
- @author Goran Juric
- /
$host = 'localhost'; $user = 'amavis'; $password = 'new_passwd'; $database = 'mail_amavis'; $keep_days = 14;
// Calculate the timestamp $prune_older_then = time() - ($keep_days * 24 * 60 * 60);
// Connect to the database $link = mysql_connect($host, $user, $password); if (!$link) {
die('Could not connect: ' . mysql_error());
} $db = mysql_select_db($database, $link); if (!$db) {
die ("Can't use $database : " . mysql_error());
}
// Delete old msgs records based on timestamps only (for time_iso see next), // and delete leftover msgs records from aborted mail checking operations mysql_query("DELETE FROM msgs WHERE time_num < $prune_older_then ;", $link)
or die(mysql_error());;
mysql_query("DELETE FROM msgs WHERE time_num < 60*60 AND content IS NULL;", $link)
or die(mysql_error());;
// Delete unreferenced e-mail addresses mysql_query("DELETE FROM maddr WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE sid=id) AND NOT EXISTS (SELECT 1 FROM msgrcpt WHERE rid=id);", $link)
or die(mysql_error());;
// When a FOREIGN KEY ... ON DELETE CASCADE is not used, tables msgrcpt // and quarantine need to be purged explicitly, e.g.: mysql_query("DELETE FROM quarantine WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE mail_id=quarantine.mail_id);", $link)
or die(mysql_error());;
mysql_query("DELETE FROM msgrcpt WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE mail_id=msgrcpt.mail_id);", $link)
or die(mysql_error());;
// Optimize tables mysql_query("OPTIMIZE TABLE msgs, msgrcpt, quarantine, maddr;", $link)
or die(mysql_error());;
// Close the database mysql_close($link);
Save this file to /etc/cron.daily/amavisd-new-cleanup (change the password at the top of the file) and make the script executable:
chmod +x /etc/cron.daily/amavisd-new-cleanup chmod 700 /etc/cron.daily/amavisd-new-cleanup
6.3.3. Utilities
If you want a GUI for managing your Amavisd-new configuration take a look at Postvis Admin.