Skip to main content

IMAP Synchronization: Principles and Purpose

Purpose of Synchronization​

If your application only needs to fetch a list of emails, it does not require synchronization. The purpose of synchronization is to detect and respond to events, such as:

  • New email arrival
  • Email deletion
  • Label (flag) changes
  • Folder changes (e.g., move)
  • Bounce notifications

Synchronization Strategy in RustMailer​

RustMailer adopts a simple and robust synchronization model based on periodic full and incremental syncs, deliberately avoiding IMAP server-side push mechanisms like IDLE.

Although IDLE enables near real-time email updates, it comes with several engineering trade-offs:

  • Maintaining strict event ordering is complex, especially under concurrent state changes.
  • It relies on persistent TCP connections for each account, which can be fragile in unstable networks.
  • Managing many simultaneous long-lived connections is challenging in multi-account environments.
  • Not all IMAP servers support IDLE, and some implementations are unreliable or non-standard.

By avoiding IDLE, RustMailer ensures greater compatibility, stability, and control over synchronization behavior.

Since email is fundamentally a non-real-time, asynchronous communication medium, RustMailer prioritizes reliability and simplicity over instantaneous updates. If your application demands true real-time email event delivery (e.g., sub-5-second latency from arrival to notification), RustMailer might not be the best fit for that use case.

Core Sync Mechanism: UID and FLAGS​

RustMailer’s synchronization is based on:

  • UID (unique per message per folder)
  • FLAGS (e.g., \Seen, \Flagged, \Deleted)

The process works as follows:

  1. Retrieve UID and FLAGS for all emails in a folder.

  2. Sort FLAGS and compute a hash.

  3. Compare current state with the last known state:

    • New UIDs → New emails
    • Missing UIDs → Deleted or moved emails
    • Changed hash → Flag updates

  4. If needed, fetch email metadata by UID.

For minimal needs, you can use minimal sync mode, which only fetches UID and FLAGS without metadata. This is sufficient for event detection and is much faster.

Performance & Batch Size​

The sync performance depends on:

  • Network quality
  • IMAP server responsiveness

In tests, requesting 1000–2000 UIDs per batch yields the best results. This batch size is currently hardcoded but may become configurable in future versions.

Full vs Incremental Sync​

Full Sync​

  • Performed periodically with a longer interval than incremental sync.

  • The first full sync happens right after account registration.

  • Downloads and stores metadata for all emails in the selected folders.

  • Metadata is saved in: RUSTMAILER_ROOT_DIR/envelope.db

  • Full sync helps detect changes in:

    • Flags of very old emails
    • Emails moved across folders
    • Deleted messages not caught by incremental sync

  • Since historical emails rarely change, a longer interval (e.g., daily) is sufficient.

Incremental Sync​

  • Runs periodically (e.g., every 20–30 seconds).
  • Fetches new emails by scanning for UIDs greater than the local max UID.
  • Always checks the latest 200 emails for FLAG changes.
  • More frequent and lightweight than full sync.

If a very old email is updated, this change will be caught during the next full sync.

Selective Sync and Time Ranges​

Large Folders​

Some folders may contain hundreds of thousands of emails, but users typically only care about recent messages.

RustMailer supports:

  • Fixed date range (e.g., sync emails from 2024-10-01 onward)
  • Sliding window (e.g., only keep metadata for the past 3 months)
tip

When using sliding window mode, RustMailer will automatically purge local metadata for emails outside the defined window.
This affects only local cache — RustMailer will never delete or modify any data on the IMAP server,
unless explicitly instructed by the user via API calls (e.g., delete or move operations).

Folder Selection​

Not all folders need to be synced.

By default, RustMailer only syncs:

  • INBOX
  • Sent

After account registration, users can customize folder selection. RustMailer will:

  • List all folders in a tree structure
  • Handle folder names and encodings internally
  • Let users select folders without worrying about IMAP technicalities

Metadata-Only Synchronization​

RustMailer only synchronizes metadata of emails — not the full content or attachments. The synchronized metadata is defined in the EmailEnvelope structure, which includes:

  • attachments: metadata about each attachment (filename, MIME type, size, etc.)
  • body_meta: metadata for email body parts (e.g., plain text, HTML)

Both fields are extracted from the IMAP BODYSTRUCTURE, without downloading the full message content.

This design ensures:

  • Faster synchronization, even for large mailboxes
  • Minimal disk usage
  • Efficient web client rendering, since most users don’t open every message or download every attachment

When a user requests to read a message or download an attachment, RustMailer fetches the content from the IMAP server on demand and caches it locally under:

RUSTMAILER_ROOT_DIR/cache

Smart Disk Cache with LRU Eviction​

RustMailer’s content cache is:

  • Content-addressable, based on a hashed key
  • Automatically cleaned up when disk usage exceeds 85%
  • Cleaned using an LRU (Least Recently Used) policy to remove the least-accessed items first

This design provides a good balance between performance, resource efficiency, and privacy — cached content is not enumerable or exposed without an access key.