SMTP Intrusion Detection

By its very nature, email servers have a public face. Companies offer email services to their employees and customers in order to facilitate collaboration with each other across the Internet. Every company that has a public domain allows SMTP traffic through their firewall in order to accept email messages from other users on the Internet. This high visibility makes an SMTP server potentially more susceptible to attack as apposed to other services. Therefore, you need to make sure that your email server is well protected from outside intruders and diffuse any harmful attacks before they occur. This white paper identifies weak links that are inherent to email servers and how to proactively prevent any future attacks.

SMTP Protocol - from security perspective

Many administrators spend a tremendous amount of time making sure that their network is secure and no outsiders with malicious intent can harm their system. It is said that your network security is as strong as your weakest link. This section describes a few potential security risks found in the SMTP protocol.

SMTP is designed to send and receive emails from other SMTP servers or clients. When a user within your company tries to send an email to a friend on the Internet, his/her email client will forward that message to your SMTP server, which in turn will then forward it to the recipient.

Prior to the actual email data being sent, a few messages are exchanged back and forth between the client and server. Both parties use these messages to identify each other and to provide a set of instructions that are mutually understandable. It is this conversation that can be exploited by a malicious user to find out potential vulnerabilities in your SMTP server.

The table below outlines the communication between a client and a server using SMTP. The sequence numbers in this table is referenced in the remainder of this document to identify each step.

Sequence	Source	Message	Description
1	Client	New Connection
2	Server	Header	This header may include the name and version of the SMTP server.
3	Client	Hello	Along with this hello the client usually sends the fully qualified domain name of the sender
4	Server	Hello	Server sends back a hello along with all of the extended verbs that it supports
5	Client	Authenticate	Client tries to authenticate. This is not a required step.
6	Server	Ok	Server sends an OK if the user is authenticated, otherwise returns an error code
7	Client	Mail from	Clients sends the email address of the sender
8	Server	Ok	The server typically accepts the sender, unless the email address violates some system policy
9	Client	Recipient	Client sends a list of recipients who should get this message
10	Server	Ok	Server may decide to reject or accept the recipient(s) based on local policy settings
11	Client	Data	This is the actual email. This message must conform to RFC 822 specifications
12	Server	Ok	If the message is accepted
13	Client	Quit	Client disconnects from the server

How to identify a malicious user

Many administrators find out about an intrusion after their system has been exploited. Before malicious users discover a weakness in your system, they must find out what is available for them to exploit. They discover these weaknesses by sending unexpected messages and/or inspecting the data returned by the server. In most cases an actual email is never sent. Consequently, it is very difficult to find out if such activity is taking place against your server.

You can proactively detect and diffuse any future attacks by inspecting and monitoring all clients connecting to your SMTP server who do not send any email.

The remainder of this paper explores possible intensions of a user if a connection is terminated without sending an email.

Termination after socket creation

This corresponds to sequence number 1 and 2 in the table above. Possible causes of this type of termination are:

• Network problems on either side.
• User wants to find out what server are you using. Many SMTP servers as well as some network firewalls provide a mechanism to return a modified string that does not reveal the servers true identity. Therefore, spammers usually do not rely on this information.
• Your server is experiencing a "Denial of Service" attack, which occurs when a user opens too many connections on your network but does not disconnect them. As a result, your server refuses a connection to a legitimate user trying to send an email.

Termination after Hello

This corresponds to sequence number 3 and 4 in the above table. Possible causes of such a termination are:

• Network problems on either side.
• User wants to find out about the extended features supported by the SMTP server. For instance if you are using MS Exchange, it will return XEXCH50 as one of its supported features. If a malicious user discovers a security hole in MS Exchange he/she can exploit this information in order to get into your system.
• Your server is experiencing a "Denial of Service" attack.

Termination after Authentication

This corresponds to sequence number 5 and 6 in the above table. Possible causes are:

• Network problems on either side.
• Someone is trying to guess user id/password on your system. In this case the user is actually only guessing for a password - they already know the user id, which is usually is the first part of an email address.

Termination after specifying a recipient list

This corresponds to sequence number 9 and 10 in the above table. Possible causes are:

• Network problems on either side.
• Someone is trying to find out if your SMTP server is an open relay.
• This can also be an indication of a dictionary attack, which happens when a malicious user sends an email to an invalid user names hoping that a small percentage will get through.

How can you protect your server?

Here are some suggestions on how to protect yourself from such attacks.

• Use a tool such as ITA Secure Messaging that can determine and report such activities. ITA Secure Messaging provides an intuitive way of capturing the IP addresses of connections where these clients originate from and suggests solutions for resolution to the problem.
• Frequently check your firewall logs to make sure no unwanted activity is taking place.
• Have your users change their password on a regular basis.

Summary

Many SMTP servers are capable of logging activity and traffic patterns. Although it is a very good practice to review these logs, they are not enough of a deterrent to set off an alarm before an attack is about to occur. Therefore, a more proactive and user-friendly approach is required to identify potential threats. ITA Secure Messaging's built-in Intrusion Detector provides a comprehensive solution to alert and identify possible intrusions into your network via an SMTP server before they wreck havoc.