SpamFiltering - Revision history

Medea61: /* Collect spam/ham for training with procmail */

2016-07-19T16:28:31Z

Collect spam/ham for training with procmail

← Older revision		Revision as of 12:28, 19 July 2016
Line 165:		Line 165:

	In /etc/postfix, I am maintaining a file named rt-aliases. It contains, for every queue:		In /etc/postfix, I am maintaining a file named rt-aliases. It contains, for every queue:
			<pre>
	rt: "\|/usr/bin/procmail -m /etc/procmailrcs/general"		rt: "\|/usr/bin/procmail -m /etc/procmailrcs/general"
	rt-comment: "\|/usr/bin/procmail -m /etc/procmailrcs/general-comment"		rt-comment: "\|/usr/bin/procmail -m /etc/procmailrcs/general-comment"
			</pre>

	in /etc/procmailrcs, the file "general" looks like:		in /etc/procmailrcs, the file "general" looks like:
			<pre>
	PATH=/usr/local/bin:/bin:/usr/bin		PATH=/usr/local/bin:/bin:/usr/bin
	LOGFILE=/var/log/procmail.log		LOGFILE=/var/log/procmail.log
	INCLUDERC=/etc/procmailrcs/_spamfilter		INCLUDERC=/etc/procmailrcs/_spamfilter
			</pre>

	The "_spamfilter" include does the actual collection:		The "_spamfilter" include does the actual collection:

	<~~nowiki~~># The lock file ensures that only 1 spamassassin invocation happens		<pre>
			# The lock file ensures that only 1 spamassassin invocation happens
	# at 1 time, to keep the load down.		# at 1 time, to keep the load down.
	#		#
Line 206:		Line 207:
	:0 c		:0 c
	/local/mailcopy/unconfirmed-ham		/local/mailcopy/unconfirmed-ham
	</~~nowiki~~>		</pre>

	== Notes & Other Resources ==		== Notes & Other Resources ==

Medea61: /* b. Unfiltered mail + ProcMail & SpamAssassin => Spam Queue */

2016-07-19T16:27:43Z

b. Unfiltered mail + ProcMail & SpamAssassin => Spam Queue

← Older revision		Revision as of 12:27, 19 July 2016
Line 93:		Line 93:

	Aliases:		Aliases:
			<pre>
	rt: "\|/usr/local/bin/procmail -m ACTION=correspond QUEUE=General /usr/local/etc/procmail/rt-deliver.rc"		rt: "\|/usr/local/bin/procmail -m ACTION=correspond QUEUE=General /usr/local/etc/procmail/rt-deliver.rc"
	rt-comment: "\|/usr/local/bin/procmail -m ACTION=comment QUEUE=General /usr/local/etc/procmail/rt-deliver.rc"		rt-comment: "\|/usr/local/bin/procmail -m ACTION=comment QUEUE=General /usr/local/etc/procmail/rt-deliver.rc"
			</pre>

	/usr/local/etc/procmail/rt-deliver.rc:		/usr/local/etc/procmail/rt-deliver.rc:

	<~~nowiki~~>#Messages >300000 characters proceed to recipient (unlikely to be spam)		<pre>
			#Messages >300000 characters proceed to recipient (unlikely to be spam)
	:0w		:0w
	* > 300000		* > 300000
Line 118:		Line 119:
	:0w		:0w
	\| /usr/local/rt3/bin/rt-mailgate --queue $QUEUE --action $ACTION --url http://localhost/rt/		\| /usr/local/rt3/bin/rt-mailgate --queue $QUEUE --action $ACTION --url http://localhost/rt/
	</~~nowiki~~>		</pre>

	=== c. ProcMail + SpamAssassin + Scrip => Spam Queue ===		=== c. ProcMail + SpamAssassin + Scrip => Spam Queue ===

Medea61: /* c. Pre-filtered mail + ProcMail => Spam Queue */

2016-07-19T16:26:55Z

c. Pre-filtered mail + ProcMail => Spam Queue

← Older revision		Revision as of 12:26, 19 July 2016
Line 57:		Line 57:
	* the absence of a second colon so there is no lockfile		* the absence of a second colon so there is no lockfile

	<~~nowiki~~>PATH=/usr/local/bin:/bin/:/usr/bin		<pre>
			PATH=/usr/local/bin:/bin/:/usr/bin
	LOGFILE=/var/log/procmail.log		LOGFILE=/var/log/procmail.log
	#rt-mailgate has already saved the message in the rt database		#rt-mailgate has already saved the message in the rt database
Line 79:		Line 80:
	:0wbh		:0wbh
	\| /usr/local/rt3/bin/rt-mailgate --queue default --action correspond --url http://localhost/rt/		\| /usr/local/rt3/bin/rt-mailgate --queue default --action correspond --url http://localhost/rt/
	</~~nowiki~~>		</pre>

	=== 2. Unfiltered messages ===		=== 2. Unfiltered messages ===

Medea61: /* b. Pre-filtered mail + Scrip => Queue */

2016-07-19T16:26:03Z

b. Pre-filtered mail + Scrip => Queue

← Older revision		Revision as of 12:26, 19 July 2016
Line 29:		Line 29:
	Create new Scrip using following information:		Create new Scrip using following information:

	Condition: On Create Action: User Defined *Custom action preparation code:		*Condition: On Create
			*Action: User Defined
	~~<nowiki>~~# Match ***** level spam. You might want to set this higher/lower if needed		*Custom action preparation code:
			<pre>
			# Match ***** level spam. You might want to set this higher/lower if needed
	my $match = '\\\\\*';		my $match = '\\\\\*';
	my $inMessage = $self->TransactionObj->Attachments->First;		my $inMessage = $self->TransactionObj->Attachments->First;
Line 39:		Line 41:
	my $spamLevel = $inMessage->GetHeader('X-Spam-Level');		my $spamLevel = $inMessage->GetHeader('X-Spam-Level');
	return ( $spamLevel !~ /$match/i ) ? 0 : 1;		return ( $spamLevel !~ /$match/i ) ? 0 : 1;
	</~~nowiki~~>		</pre>

	*Custom action cleanup code:		*Custom action cleanup code:
			<pre>
	my $newqueue = 'spam';		my $newqueue = 'spam';
	my ($status, $msg) = $self->TicketObj->SetQueue($newqueue);		my ($status, $msg) = $self->TicketObj->SetQueue($newqueue);
	return $status ? undef : 1;		return $status ? undef : 1;
			</pre>

	=== c. Pre-filtered mail + ProcMail => Spam Queue ===		=== c. Pre-filtered mail + ProcMail => Spam Queue ===

Medea61: Undo revision 26281 by Medea61 (talk)

2016-07-19T16:23:38Z

Undo revision 26281 by Medea61 (talk)

← Older revision		Revision as of 12:23, 19 July 2016
Line 29:		Line 29:
	Create new Scrip using following information:		Create new Scrip using following information:

	*Condition: On Create		Condition: On Create Action: User Defined *Custom action preparation code:
	*Action: User Defined
	*Custom action preparation code: <~~code~~>		<nowiki># Match ***** level spam. You might want to set this higher/lower if needed
	# Match ***** level spam. You might want to set this higher/lower if needed		my $match = '\\\\\*';
	my $match = '\\\\\*';		my $inMessage = $self->TransactionObj->Attachments->First;
	my $inMessage = $self->TransactionObj->Attachments->First;		return 0 if (!$inMessage); # if no message attachment - assume web UI
	return 0 if (!$inMessage); # if no message attachment - assume web UI		return 0 if (!$inMessage->GetHeader('Received')); # exit if not email message
	return 0 if (!$inMessage->GetHeader('Received')); # exit if not email message

	my $spamLevel = $inMessage->GetHeader('X-Spam-Level');		my $spamLevel = $inMessage->GetHeader('X-Spam-Level');
	return ( $spamLevel !~ /$match/i ) ? 0 : 1;		return ( $spamLevel !~ /$match/i ) ? 0 : 1;
	</code>		</nowiki>

			*Custom action cleanup code:

	*Custom action cleanup code: <code>		my $newqueue = 'spam';
	my $newqueue = 'spam';		my ($status, $msg) = $self->TicketObj->SetQueue($newqueue);
	my ($status, $msg) = $self->TicketObj->SetQueue($newqueue);		return $status ? undef : 1;
	return $status ? undef : 1;
	~~</code>~~

	=== c. Pre-filtered mail + ProcMail => Spam Queue ===		=== c. Pre-filtered mail + ProcMail => Spam Queue ===

Medea61 at 16:23, 19 July 2016

2016-07-19T16:23:21Z

← Older revision		Revision as of 12:23, 19 July 2016
Line 29:		Line 29:
	Create new Scrip using following information:		Create new Scrip using following information:

	Condition: On Create Action: User Defined *Custom action preparation code:		*Condition: On Create
			*Action: User Defined
	<~~nowiki~~># Match ***** level spam. You might want to set this higher/lower if needed		*Custom action preparation code: <code>
	my $match = '\\\\\*';		# Match ***** level spam. You might want to set this higher/lower if needed
	my $inMessage = $self->TransactionObj->Attachments->First;		my $match = '\\\\\*';
	return 0 if (!$inMessage); # if no message attachment - assume web UI		my $inMessage = $self->TransactionObj->Attachments->First;
	return 0 if (!$inMessage->GetHeader('Received')); # exit if not email message		return 0 if (!$inMessage); # if no message attachment - assume web UI
			return 0 if (!$inMessage->GetHeader('Received')); # exit if not email message

	my $spamLevel = $inMessage->GetHeader('X-Spam-Level');		my $spamLevel = $inMessage->GetHeader('X-Spam-Level');
	return ( $spamLevel !~ /$match/i ) ? 0 : 1;		return ( $spamLevel !~ /$match/i ) ? 0 : 1;
	</~~nowiki~~>		</code>

	*Custom action cleanup code:

	my $newqueue = 'spam';		*Custom action cleanup code: <code>
	my ($status, $msg) = $self->TicketObj->SetQueue($newqueue);		my $newqueue = 'spam';
	return $status ? undef : 1;		my ($status, $msg) = $self->TicketObj->SetQueue($newqueue);
			return $status ? undef : 1;
			</code>

	=== c. Pre-filtered mail + ProcMail => Spam Queue ===		=== c. Pre-filtered mail + ProcMail => Spam Queue ===

Admin: 2 revisions imported

2016-04-06T20:37:47Z

2 revisions imported

New page

= Spam Filtering =

By moving all SPAM to a single queue, you can easily purge (shred) it periodically. You might also take the opportunity to purge any RT users who do not have a ticket associated with them.

Please find documented below 6 setups for processing spam (1a-1c, 2a-2c). In general, you want to filter your messages at the MTA, before they get RT if you can.

The list archives also contain a number of [http://www.gossamer-threads.com/lists/rt/users/87056?search_string=Spam filtering in RT;#87056 useful tips]

== 0. Pre-filtering & scoring messages ==

There are a number of ways of doing so, such as procmail or [http://pthbb.org/manual/software/mailfilter mailfilter] which can do other somewhat-RT-specific pre-processing as well.

== 1. Pre-filtered mail ==

There are several options for processing messages once scored:

=== a. Pre-filtered mail + ::Filter::SpamAssassin (=> Spam Queue) ===

http://codepad.org/HQmnr0AG based on

http://documentroot.com/SpamAssassin.pm based on RT::Interface::Email::Filter::SpamAssassin from 3.8.1

This updated version of the filter provides several options for handling potential spam, such as rejecting all messages above a certain score, and filing others in a SPAM queue.

=== b. Pre-filtered mail + Scrip => Queue ===

In the example below a score of 5 or more is needed. Edit preparation code if you want to modify the level.

Create new Scrip using following information:

*Condition: On Create *Action: User Defined *Custom action preparation code:

<nowiki># Match ***** level spam. You might want to set this higher/lower if needed
my $match = '\*\*\*\*\*';
my $inMessage = $self->TransactionObj->Attachments->First;
return 0 if (!$inMessage); # if no message attachment - assume web UI
return 0 if (!$inMessage->GetHeader('Received')); # exit if not email message

my $spamLevel = $inMessage->GetHeader('X-Spam-Level');
return ( $spamLevel !~ /$match/i ) ? 0 : 1;
</nowiki>

*Custom action cleanup code:

my $newqueue = 'spam';
my ($status, $msg) = $self->TicketObj->SetQueue($newqueue);
return $status ? undef : 1;

=== c. Pre-filtered mail + ProcMail => Spam Queue ===

A procmail recipe (for each queue/(correspond|comment) pair) can be used to send spam to a spam queue in RT. Note that:

* the flags bh need to be set so rt-mailgate sees both the body and header of the message.
* the absence of a second colon so there is no lockfile

<nowiki>PATH=/usr/local/bin:/bin/:/usr/bin
LOGFILE=/var/log/procmail.log
#rt-mailgate has already saved the message in the rt database
#set DEFAULT to be somewhere real if you want procmail to save the message elsewhere
DEFAULT=/dev/null

#Messages >300000 characters proceed to recipient (unlikely to be spam)
#:0fwbh
:0wbh
* > 300000
| /usr/local/rt3/bin/rt-mailgate --queue default --action correspond --url http://localhost/

#if the spam trigger is fired send to spam queue
#:0fwbh
:0wbh
* ^X-Spam-Level: \*\*\*\*\*
| /usr/local/rt3/bin/rt-mailgate --queue spam --action correspond --url http://localhost/rt/

#if the spam trigger is not fired then send to expected destination
#:0fwbh
:0wbh
| /usr/local/rt3/bin/rt-mailgate --queue default --action correspond --url http://localhost/rt/
</nowiki>

=== 2. Unfiltered messages ===

=== a. Unfiltered mail + ::Filter::SpamAssassin (=> Spam Queue) ===

RT::Interface::Email::Filter::SpamAssassin above (1a) can also pass unscored messages off to SpamAssassin for processing.

=== b. Unfiltered mail + ProcMail & SpamAssassin => Spam Queue ===

This procmail recipe is similar to 1c. above, except that it handles previously unscored messages, and accepts many of the rt-mailgate parameters as environment variables so that it may be easily used with multiple aliases.

Aliases:

rt: "|/usr/local/bin/procmail -m ACTION=correspond QUEUE=General /usr/local/etc/procmail/rt-deliver.rc"
rt-comment: "|/usr/local/bin/procmail -m ACTION=comment QUEUE=General /usr/local/etc/procmail/rt-deliver.rc"

/usr/local/etc/procmail/rt-deliver.rc:

<nowiki>#Messages >300000 characters proceed to recipient (unlikely to be spam)
:0w
* > 300000
| /usr/local/rt3/bin/rt-mailgate --queue $QUEUE --action correspond --url http://localhost/rt/

#Is it spam?
:0fw: spamassassin.lock
* < 300000
| /usr/local/bin/spamc

#if the spam trigger is fired send to spam queue
:0w
* ^X-Spam-Status: Yes
| /usr/local/rt3/bin/rt-mailgate --queue Spam --action correspond --url http://localhost/rt/

#if the spam trigger is not fired then send to expected destination
:0w
| /usr/local/rt3/bin/rt-mailgate --queue $QUEUE --action $ACTION --url http://localhost/rt/
</nowiki>

=== c. ProcMail + SpamAssassin + Scrip => Spam Queue ===

http://www.soundwave.net/~wmono/rt/

== 3. Mis-handled messages ==

=== a. False positives: Greylist ===

2c above outlines a general (non-procmail specific) setup (steps 1 through 6 for RT) to automatically greylist suspect messages. Legitimate requestors may reply to an autoresponse and have their misfiled message requeued.

=== b. False negatives: QuickSpamhandler ===

Note: [[QuickSpamHandler]] seems to no longer be available. The [http://search.cpan.org/~ruz/RT-Extension-ReportSpam-0.02/lib/RT/Extension/ReportSpam.pm RT::Extension::ReportSpam] is fairly similar though.

This extension provides a few tools that can be used for manual triage of SPAM in low-traffic installation, or handling the odd message that leaks through your automated filters:

* Adds alternate forms of the "Unowned Tickets" and "My Tickets" global searches, which exclude messages in the spam queue.
* Adds a link to the "Search - Unowned Tickets" query which permits you to instantly vanquish a ticket if the subject is sufficient to convince you the message is SPAM.
* Adds a link to the Ticket tool bar that will instantly vanquish the displayed ticket.

Vanquished tickets are moved to a designated SPAM queue; the default is named "Possible SPAM".

==== Installation ====

* Download the [http://zephyrgeek.qatar.cmu.edu/RT/QuickSpamHandler.tgz archive]
* Unpack it
* ./configure
* make install
* Read and act on the output from the previous command.

Really, the only tricky bit is the database update. If I ever figure out a better way to do this I'll make sure to update the archive. Hopefully someone will find this useful. If you do, please drop me a line!

=== c. False negatives: Bookmarklet ===

An alternative to QuickSpamHandler above, this bookmarklet moves the displayed ticket to the spam queue (change 12 to the appropriate ID for your queue), and sets the status to rejected.

javascript:self.location=self.location+'&Status=resolved&Queue=12'

Thus you can distinguish messages which you have already decided are spam, from those which are automatically filed into that queue.

== Collect spam/ham for training with procmail ==

In /etc/postfix, I am maintaining a file named rt-aliases. It contains, for every queue:

rt: "|/usr/bin/procmail -m /etc/procmailrcs/general"
rt-comment: "|/usr/bin/procmail -m /etc/procmailrcs/general-comment"

in /etc/procmailrcs, the file "general" looks like:

PATH=/usr/local/bin:/bin:/usr/bin
LOGFILE=/var/log/procmail.log
INCLUDERC=/etc/procmailrcs/_spamfilter

The "_spamfilter" include does the actual collection:

<nowiki># The lock file ensures that only 1 spamassassin invocation happens
# at 1 time, to keep the load down.
#
:0fw: spamassassin.lock
* < 256000
| /usr/bin/spamassassin

# Mails with a score of 15 or higher are almost certainly spam (with 0.05%
# false positives according to rules/STATISTICS.txt). Let's put them in a
# different mbox. (This one is optional.)
:0:
* ^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
/local/mailcopy/almost-certainly-spam

# Work around procmail bug: any output on stderr will cause the "F" in "From"
# to be dropped. This will re-add it.
:0
* ^^rom[ ]
{
LOG="*** Dropped F off From_ header! Fixing up. "

:0 fhw
| sed -e '1s/^/F/'
}

:0 c
/local/mailcopy/unconfirmed-ham
</nowiki>

== Notes & Other Resources ==

Don't forget to run `newaliases` (or equivalent) after you alter your configuration if appropriate.

* [[SpamScore2Priority]]
* [http://pm-doc.sourceforge.net/pm-tips-body.html#procmail_flags Procmail flags documentation].