|
|
|
|
|
        |
|
|
|
|
|
  |
|
|
|
|
|
Main Home Page Tutorials Home Page Spam Filtering
Sub Sites
Our Top 10 Pages
|
Stopping SpamGetting rid of spam is probably the most talked about aspect of the Internet. This tutorial will take you through the steps to remove the vast majority of spam entering your mailbox provided you are an E-Government client.Filtering spam is a three-part process:
E-Governmentcom supplies the first part representing 98% of the work. The Nuts and Bolts of Spam FilteringSpam filtering is an elusive process that is essentially a fight between the spammers and the software programs trying to prevent spam. The system described in this tutorial is certainly not perfect; no system will ever achieve that lofty goal. However, the package we’re about to describe should eliminate over 95% of your spam.This tutorial is segmented into six sections:
If you understand the underlying principles of spam management you can skip directly to section five. You can also skip to section five if you’re impatient or if you only want results and have no desire to know how things work. Part One: What is Spam?For the sake of simplicity spam can be defined as mail entering our mailbox that we didn’t request or that we don’t want. If you have been involved with the Internet for a while you probably receive more spam than you do legitimate mail. If you have several email addresses and have been involved with the Internet for a long time, 95% of your mail may be spam.Spam is much like pornography in that it’s easy to recognize once you see it but almost impossible to define. One of the problems with any type of filtering is describing what constitutes spam. A rule saying any message containing the word "breast" should be considered spam would put a message about "breast cancer" into the trash basket. Part Two:. How Can You Use the Spam System?There are two categories of E-Government email accounts, POP3 and alias. An alias account simply forwards mail addressed to an account hosted on the E-Government server to a designated email account typically not hosted on E-Government. Since this mail is not processed by E-Government there is no mechanism for SPAM filtering on E-Government system.Mail sent to a POP3 account can be forwarded to an external account, processed by a web-based program, or transferred to your local PC and processed by a mail program such as Outlook or Eudora. Spam processing can be applied to POP3 mailboxes. Part Three: What are Rules?A rule is simply a statement with a conditional clause and an action clause. The action clause is executed if the condition is true. For example, if the body of an email messages contains the word Viagra, transfer the message to a folder named trash. You could also push it off the edge of the earth. That rule would certainly get rid of a lot of spam but it would also trash a joke containing the word Viagra. Rules can search a message body, subject, any headers, as well as any specific header such as the To, From, or Subject fields. Conditions include contains, does not contain, does not contain, case sensitivity, in additions to conjunctions such as "and" and "or."Let’s assume you get jokes from jokes@abc.com. You could specify a rule to trash any message where the body of subject contains "Viagra" unless the message is from jokes@abc.com. More on rules shortly. Part Four: 4. What Does E-Government do Behind the Scenes?E-Government has installed an extremely powerful spam management system. The system doesn’t actually delete any messages; it sets the stage for you to delete unwanted messages while preserving good messages.One of the problems with any spam management system is what is called a "false positive" which is a valid message identified as spam. E-Government will successfully tag the bulk of all messages as potential spam but you have to extend the process and specify rules that allow tagged messages, which are valid to you, to pass through the system. The part that E-Government provides is called a "black list" and the part you provide is called a "white list." In addition, you may want to define rules to add items to the black list. What E-Government does is place one or more header messages into an existing messages. You can then filter, or defeat the filtering, of any message so tagged. A variety of conditions can cause the E-Government filtering system to place one or more additional headers in a mail message, these are called X-Headers. The additional headers are what is used to filter, not the contents of a message or any of the conventional headers. The From: field is almost always forged in a spam message and as such is essentially useless for filtering purposes. The E-Government filtering system checks the domain of the actual sender and if the domain matches any number on one of the searched black list, a header looking like the following example is added to the message. X-IMAIL-SPAM-DNSBL: (SpamCop,154337584,127.0.0.2) The X-Header always begins with X-IMAIL-SPAM- followed by the specific class of rule and information about the specific message following the colon. In this example, the class of the headers is DNS (Domain Name Server) BL (black list) and the source of the black list was SpamCop. Since the external sources are constantly updating their black lists the filtering automatically remains current. X-Headers are also inserted when a message fails a validation test. The following is a summary listing of X-Header types; don’t be concerned if you don’t understand many of the terms. X-IMAIL-SPAM-ADDRBL: (name_of_service, message_ID, IP address/reason) The message matched an ADDR black list. X-IMAIL-SPAM-DNSBL: (name_of_service, message_ID, IP address/reason) The message matched a DNS black list. X-IMAIL-SPAM-HELOBL: (name_of_service, message_ID, IP address/reason) The message matched a HELO/EHLO black list. X-IMAIL-SPAM-HELODOMAIN: domain_name The message failed the HELO/EHLO domain validation. X-IMAIL-SPAM-INVALIDFROM: from_address The message contained an invalid "FROM" address. X-IMAIL-SPAM-IP4R: name_of_service The message matched an IP4R(PTR) black list. X-IMAIL-SPAM-STATISTIC: x% The message has been identified as spam by the statistical filter. X is the probability that an email is spam. X-IMAIL-SPAM-REVDNS: ip_address The message failed a DNS lookup based on the IP address. X-IMAIL-SPAM-RHSBL: (name_of_service, message_ ID, IP address/reason) The message matched a RHS black list. X-IMAIL-SPAM-PHRASE: %s A phrase in the message matched the phrase list. (%s) represents the matched phrase. X-IMAIL-SPAM-VALFROM: (message ID) The message failed the "MAIL FROM" address validation. X-IMAIL-SPAM-VALREVDNS: (message ID) The message failed the reverse DNS lookup validation. X-IMAIL-SPAM-VALHELO: The message failed the HELO/EHLO domain validation. X-IMAIL-SPAM-HTML-FEATURES:( The message contained the specified HTML tags. X-IMAIL-SPAM-URL-DBL:( The message contained HREF or IMG SRC tags with links to a domain name listed in the URL Domain Black List. Part Five. Customizing the System to Meet Your Needs.How you customize your system is a function of how much mail you receive on a daily basis. Let’s take a simple case for a user receiving fewer than 100 email messages a day.Read your mail as you normally would. If you use a mail reader such as Outlook or Eudora process all your mail and then close down the reader. Log into your email account at http://pobox.lhhosting.com First, create a mailbox to hold spam. Click on the link "Edit Mailboxes" on the bottom of the left side. On the next page is a section with the title, "Create a Mailbox." In the input box title "Name" enter the name of your choice, "spam" (without the quotes) has a nice ring. If you have never created a mailbox there are five mailboxes listed on the results page, Main, Deleted, Draft, Sent, and Spam. Now we’re ready for some serious spam hunting. There is a drop menu in the upper right of the page with the default setting of "Administrative Account Options." Drop the menu and click on the option "Change Processing Rules" it’s about three quarters of the way down. Next, click on the left side link "Add Rule" or click on the “Add” button at the bottom of the page. Next, click on the drop menu title "Field" and select "Header." Click on the radio button title "contains." In the big box title "Phrase" enter "IMAIL-SPAM" (without the quotes.) The phrase "IMAIL-SPAM" will catch all inserted header messages. Do not check either the "Case Sensitive Match" or the "Search String from File" boxes. Finally, click on the "Add Condition" button. You will then be returned to the same set of options with the rule you have just added inserted in the "Rule" area. Scroll to the bottom of the page. There is a drop menu with the default entry "Move the message to this mailbox." In the box title "Destination" enter "spam" (without the quotes) and, lastly, click on the "Finish" button. You have just created your first and most important filtering rule ever. If you make a mistake in entering your filter, see the section on editing below. After some period of time log back into your web-based email account before engaging your normal mail readers. After logging in you will see the number of messages in your Main and spam mailboxes. If you have no other mailboxes the sum of the two mailboxes represent the total count of mail messages received since your last logon. The messages in your spam mailbox may represent over 90% of your received messages. There is another important step, creating your white list. You now have to scan the spam mailbox to check for false positives, that is messages that have been flagged as spam that are actually good messages. When you first log on, or subsequently click on the top-of-the-page menu option, you will see the list of your available mailboxes, click on spam to open that mailbox and see a list of waiting messages. The initial defaulted number of messages is ten; you may want to increase the display count since you are only going to scan existing messages. Select "Edit My Preferences" from the top drop menu. About half way down select either 20 or 50 from the "Number of messages per page" option. Scroll down to the bottom and click on the "Save" button. From the main menu clicking on any mailbox opens that mailbox. Clicking on the spam mailbox link opens that mailbox displaying the first group of filtered messages. You can only open the mailbox if there are messages. Messages are listed in reverse chronological order unless you have changed the order in your preferences. Scroll down the list of messages observing the From and Subject columns looking for valid mail. If you think a particular message might be valid open the message. If it is indeed valid you have to add it to your white list. The most common type of valid message showing up as spam is mail that originates from your subscribed discussion lists or newsletters. Since their mission is sending mail to many people, discussion list and newsletters postings tend to be classified as spam. When you find a valid message in your spam mailbox you have to determine a unique identifying characteristic or a group of characteristics. There are six possible areas to scan for uniqueness: Subject, To, From, Sender, Header, and Body. The body is typically not a good choice since it changes from message to message. Good candidates are the Subject and From fields. Many list start the subject with the same code each time, as an example [DX-News]. When you find something that uniquely identifies the message make a filter. Follow the instructions above for making a filter but in this case make the destination mailbox "Main." There is one additional step after the filter is defined. New filters are always added at the bottom of the list. Filters are processed in the order in which they appear, so adding a filter to spare a message from being placed into the spam bucket is no good if it’s already been designated as spam. When the filters have been displayed observe the far right side of each line. There are up and down arrows, a pencil for editing a filter, and a trashcan for deleting a filter. The up and down arrows change the process order of a filter. Since filters are executed in the order in which they are listed, you have to move your white list filter above the header rule filters. Once the filter is higher in the processing order than the header rules the targeted message types are transferred to the Main mailbox before the spam filters get to them. Repeat this process as often as needed. When you have searched all the displayed messages for possible white list inclusion you can delete the spam residue. Scroll to the top of the page where all the messages are listed. Next, check the box to the left of the title From: This flags all displayed messages for deletion. Scroll to the bottom of the displayed messages and click on the Delete link on the left side. After you click on the Delete link, the next group of messages is displayed. Repeat this procedure until you have dealt with all of the messages in your spam mailbox. How long you continue with the process of looking for white list entries depends on your patience and how willing you are to have good messages occasionally deleted. I would suggest scanning all your spam messages for at least a week and preferably a month since some newsletters are only sent on a monthly basis. When you are satisfied with both your white and black lists you can permanently delete spam messages rather than having them go into your spam mailbox. Go to your filter listings and edit the rule that is used to transfer messages to the spam mailbox. Go to the bottom of the page and select "Delete" form the drop menu. Click on Finish and you’re done. Editing or Deleting a RuleOnce a rule has been created and saved it can be edited or deleted. Click on Change Processing Rules from the top of page drop menu. Scroll down to the desired rule and go to the right of Value field and click on the pencil.To modify an existing rule scroll down the displayed rule and click on it. All components of the displayed rule are then displayed in the area used to initially enter the rule. Change whatever needs changing and click on the Update Condition button. When you are finished with the modification scroll down to the bottom of the page and click on the Finish button. If you want to add a second condition, repeat the above process but click on either Insert AND or Insert OR depending upon your logic. As an example you could keep adding white list conditions to one rule and use the OR conjunction. Configuring With Higher Levels of SpamIf you receive much more than 100 pieces of mail at one time you may want to consider entering the spam filters individually instead of one catchall filter. TheIMAIL-SPAM-DNSBL category receives the most action so you might want to create a separate rule to filter on header containing IMAIL-SPAM-DNSBL. Continue adding filters for the other rules shown in What Does E-Government do Behind the Scenes section. ConclusionMany years ago there was a radio commercial saying "Try it you’ll like it." This certainly applies to spam filtering. It’s a bit of a job to set up initially but take it from one who had over 2,000 filters added one at a time, this system is magic.I would really appreciate any comments or suggestions on this tutorial. I would also appreciate your list of white pages, and how you filter them, for possible inclusion in the global white page filter set. Urb LeJeune urb@e-government.com |
|
|
Spam Filtering Other Major E-Government.com Sub Web Sites
Home Page - Media Central - FAQs E-Government.com 15 Hunter Drive Tuckerton, NJ 08087 Email: President Urban LeJeune Phone: 609-294-0320 - FAX: 609-294-0320
Content Copyright © 0000-2008 by E-Government.com |