Message Transfer

Submitting Messages Queue Limits and Foldering Routing Enqueueing Delays and Suspensions Dequeueing One of the main functions of the CommuniGate Pro Server is message transfer. Acting as an MTA (Message Transfer Agent), the server accepts messages from various sources (modules, internal kernel components, etc.), and delivers (transfers) them to remote or local destinations using the same or different modules. While all submitted messages are stored as individual files in the Queue directory inside the CommuniGate Pro "base directory", each message can be enqueued into several different queues (if it has several recipients). Each communication module can maintain one or several logical queues. For example, the SMTP module maintains one queue for each Internet domain.

The CommuniGate Pro Server has the following set of message sources:

• The SMTP module submits the messages received from mailer applications and from other mail servers via the Internet. The Submit Address tag used: SMTP.
• The RPOP module submits the messages retrieved from remote POP servers. The Submit Address tag used: RPOP.
• The Local Delivery module submits messages generated with the Account-level (Account and Domain) Automated Mail Processing Rules. The Submit Address tag used: RULE.
• The Local Delivery module re-submits messages directed to 'all' addresses and redirected to all forwarders in the specified domain. The Submit Address tag used: ALLF.
• The PIPE module submits the messages received from external applications via interprocess communication channels, and the messages generated and stored in the special Submitted directory. The Submit Address tag used: PIPE.
• The POP module submits the messages received from certain mailer applications employing the XTND XMIT protocol extension. The Submit Address tag used: POP.
• The IMAP module submits the messages received from MAPI client applications. The Submit Address tag used: IMAP.
• The Web User Interface module submits the messages composed within Web browsers. The Submit Address tag used: HTTP.
• The Enqueuer component submits messages generated with the Server-wide and Cluster-wide Automated Mail Processing Rules. The Submit Address tag used: SRULE.
• The Dequeuer component generates and submits delivery notification messages. The Submit Address tag used: DSN.
• The Web User Interface module generates and submits Message Disposition Notification ("Read Recipts") messages. The Submit Address tag used: MDN.
• The Calendaring component generates and submits Event Request Reply messages. The Submit Address tag used: ICAL.
• The LIST module submits messages to be distributed to mailing list subscribers. The Submit Address tag used: LIST.
• The LIST module submits reply messages generated when processing administration requests. The Submit Address tag used: LSRV.
• The LIST module submits messages directed to individual subscribers (Warnings, Hello, Bye, etc.) The Submit Address tag used: LSTM.
• The LIST module submits messages directed to the list owner The Submit Address tag used: LSTO.
• The LIST module re-submits messages directed a Group. The Submit Address tag used: GROUP.
• The Event Handler component generates and submits notification messages. The Submit Address tag used: EVENT.
• The Lawful Interception component generates and submits report messages. The Submit Address tag used: LWIT.

The CommuniGate Pro Server transfers messages to the following destinations:

• the SMTP module transfers messages to other SMTP mail servers via the Internet;
• the LIST module accepts and processes messages with mailing list postings, and with various list administration requests;
• the Local Delivery module transfers messages to local user mailboxes;
• the PIPE module transfers messages to external applications via interprocess communication channels;
• the kernel itself transfers (discards) messages routed to NULL or ERROR addresses;

The following diagram illustrates the message flow inside the CommuniGate Pro server.

Submitting Messages

All messages are created as temporary files. They are stored in the Queue directory as files with the .tmp extension. A module or a kernel component stores message envelope and the message itself in such a file and then submits it to the kernel for processing.

The message envelope is a set of text lines. Each line specifies either the message return-path, or one message recipient address, or message delivery options.

If a module fails to compose a message (for example, an SMTP connection breaks during message transfer), the module discards the temporary file, and the kernel deletes it.

When a message is completely composed and submitted to the kernel, the file extension is changed to .msg and the message is scheduled for processing.

When a systems restarts, a kernel module checks all files with the .msg extension stored in the Queue directory and resubmits them for processing.

You can use a Web browser to configure the Temporary Files manager. Open the Obscure page in the Settings section.

Temp File Options
TempFiler Log: Crashes OnlyFailuresMajor & FailuresProblemsLow LevelAll Info Recycle Temp Files: DisabledEnabled

TempFiler Log

This setting specifies what kind of information the Temporary Files manager should put in the Server Log. Usually you should use the Failures (file system error reports) level. But when you experience a problem with some module submitting messages, you may want to set this setting to Low-Level or All Info: in this case file i/o operations will be recorded in the System Log as well. When the problem is solved, set the TempFiler Log setting to its regular value, otherwise your System Log files will grow in size very quickly.

Recycle Temp Files

Enable this option to improve performance of your system under heavy load.

The Temporary Files manager Log records are marked with the TEMPFILE tag.

Queue Limits and Foldering

If a Server is heavily loaded, it can contain thousands of message files in its Queue directory. Many operating and file systems cannot efficiently process large directories. In this case the Queue directory can be split into several subdirectories, each containing a portion of Temporary and Queue files.

You may want to limit the total number of messages in the CommuniGate Pro Queue. If this number is reached, the modules start to reject attempts to submit new messages.

To specify the Queue processing options, open the Queue page in the Settings section of the WebAdmin Interface:

Message Queue
Log: Crashes OnlyFailuresMajor & FailuresProblemsLow LevelAll Info     Queue Limit: Unlimited010301003001000300010000300001000003000001000000 Queue Foldering: Auto010100

Queue Limit

If you specify a limit with this option, the CommuniGate Pro modules (SMTP, PIPE, RPOP, MAPI, WebUser) will stop accepting new messages when the number of messages in the Queue exceeds the specified limit.

Queue Foldering

This setting specifies where the CommuniGate Pro Server will create new Temporary files, and, as a result, how the message files will be stored in the Queue Directory.
If you select the 0 (zero) value, no subdirectory will be created in the Queue directory, and all Temporary files will be created directly in the Queue directory.
If you select the 10 or 100 value, subdirectory with NN names (where NN is a 2-digit number from 00 to 99) will be created in the Queue directory. The newly created Temporary files will be created inside these subdirectories. The server will use 10 or 100 subdirectories, depending on this setting value.
If you select the Auto value, Queue subdirectories will be used when the total number of messages in the Queue exceeds 5000.

Routing

When a message is submitted for processing, a kernel component examines its envelope information. Each recipient address is parsed and passed to the Router component. The Router component decides which module or kernel component should process each recipient address.

Enqueueing

When all recipient addresses are parsed and routed, the Enqueuer component applies the Server-Wide Rules to the message. Then it passes the message to the modules specified with the Router component.

Communication modules do not process messages immediately, but enqueue them into the module-specific queues. The SMTP module creates and maintains a queue for each Internet domain, the Local Delivery module creates and maintains a queue for each local account, etc.

You can use a Web browser to configure the Enqueuer component. Open the Queue page in the Settings section of the WebAdmin Interface.

Message Enqueuer
Log: Crashes OnlyFailuresMajor & FailuresProblemsLow LevelAll Info Processors: 1235102550 Hop Counter Limit: 1510203050other

Enqueuer Log

Use this setting to specify what kind of information the Enqueuer component should put in the Server Log. Usually you should use the Failures (file system error reports) level.

The Enqueuer component Log records are marked with the ENQUEUER tag.
The records created when applying the Server-Wide Rules are marked with the ENQUEUERRULES tag.

Processors

Use this setting to specify the number of Enqueuer processors (threads). Usually one Enqueuer thread is enough even for a heavy-loaded server. You should increase the number of Enqueuer processors if:

• you have specified many Server-Wide Rules;
• Server-Wide Rules use the Execute action to start external programs;
• an External Filter (anti-virus, content filtering, etc.) program is enabled.
• you have to handle heavy incoming load, with many messages being submitted every second via the SMTP or other modules.

The numEnqueuerMessages SNMP element shows the number of messages that have been recieved, but not enqueued yet. If this number is growing, you need to increase the number of Enqueuer processors.

Hop Counter Limit

When a message is being received by any host or module, it gets an additional Received: header field. The Hop Counter is the number of Received: header fields in the message. If a message contains too many Received: header fields, it may indicate that a message is in some sort of mail loop. This parameter specifies the Limit for the Hop Counter - any message that has more Received: header fields than specified in this setting is rejected by the ENQUEUER - without any attempt to deliver it to the recipients specified in the message envelope.

Delays and Suspensions

When a communication module fails to transfer a message, it uses the kernel queue management component to delay processing.

• a module can delay an entire queue: for example, the SMTP module can delay a queue created for an Internet domain, if it cannot connect to that domain or its relays;
• a module can delay an individual queued message: for example, the SMTP module can delay a message if the receiving host rejects this particular message (transition failure);
• a module can delay an individual recipient address in a queued message: for example, the SMTP module can delay an address if the receiving host rejects that particular address (transition failure).

Dequeueing

When a communication module transfers a message or when it rejects a message because of a fatal error, it removes the message from the module queue. The module composes a delivery report and passes it to the Dequeuer kernel component.

The Dequeuer component processes delivery information. If requested, it composes Delivery Status Notification (DSN) messages and submits them back to the system for delivery to the original message sender. When a message has several recipients, the Dequeuer module may choose to delay DSN generation, so each DSN message can contain reports about several recipients.

When all message recipients are processed and the message is dequeued from all queues, the Dequeuer component removes the message file from the Queue directory.

You can use a Web browser to configure the Dequeuer component. Open the Queue page in the WebAdmin Settings section.

Message Dequeuer

Log: Crashes OnlyFailuresMajor & FailuresProblemsLow LevelAll Info Processors: 1235102550 Reporting Delay: 0 seconds1 second5 seconds10 seconds15 seconds30 secondsminute2 minutes3 minutes5 minutes10 minutes15 minutes30 minuteshour2 hours3 hours6 hours12 hours Send Return-Receipts to: everybodyclientsauthenticatednobody On Failure, Return: headers by defaultbody by defaultalways headersalways body Copy Failure Reports to:

Dequeuer Log

Use this setting to specify what kind of information the Dequeuer component should put in the Server Log. Usually you should use the Major & Failures (delivery reports) level.
The Dequeuer component Log records are marked with the DEQUEUER tag.

Processors

Use this setting to specify the number of Dequeuer processors (threads). Usually one Dequeuer thread is enough even for a heavy-loaded server. Only if your Server performs some kind of special message processing and has to generate a lot of DSN messages, should you use several Dequeuer threads.

Reporting Delay

Use this setting to specify the maximum delay between the moment when a message was transferred or failed and the moment when a delivery report is generated. The more the delay, the more reports can be placed in one DSN message. A DSN message is generated immediately after the last message recipient is processed.

On Failure, Return

Use this setting to specify what portion of a failed message should be included into the DSN (error report) message.

• If the sender has not specified this option explicitly, and the headers by default option is selected, only the failed message headers will be returned;
• If the sender has not specified this option explicitly, and the body by default option is selected, the entire failed message will be returned;
• If the always headers option is selected, only the message headers are included into the DSN message, even if the message sender has specified that the entire message should be returned on failure.
• If the always body option is selected, the entire message is included into the DSN message, even if the message sender has specified that only the message headers should be returned on failure.

Copy Failure Reports

When this option is enabled, all error messages generated with the CommuniGate Pro Dequeuer are sent to both the failed message return-path and to the specified E-mail address.

Send Return-Receipts to

Positive reports (delivery reports and relay reports) are generated only if the message sender has requested them. Use this setting to specify who can request these reports:

everybody	reports are generated and sent whenever the message sender requests them.
clients	reports are generated and sent if the sender has submitted the message from a Client IP Address or if the message has been submitted by an authenticated user.
authenticated	reports are generated and sent if the message has been submitted by an authenticated user (via WebAdmin, SMTP AUTH, MAPI, etc.)
nobody	positive reports are not generated.