How to Deploy a Private Mail Server with Mailu in 1 Hour (Docker, DNS & SSL Guide)

Over the years working as a backend developer, I've dealt with numerous email-related requirements: registration verifications, password resets, business notifications... Initially, most teams jump straight to cloud providers like AWS SES or AliCloud/Tencent Cloud email push services, or third-party APIs. But if you, like me, occasionally need internal team email routing, want a professional @yourname.com inbox for a personal domain, or simply want full control over your communication infrastructure, commercial email hosting can be expensive and restrictive.

Today, I'll walk you through deploying a fully functional, production-ready mail server from scratch using Mailu—a comprehensive, Docker-native mail server solution. By the end of this guide, you'll have gone through environment setup, Docker Compose configuration, DNS record management (MX/SPF/DKIM), SSL certificate automation, and successfully sent your first email via Webmail.

Prerequisites

This tutorial assumes you have:

1. A Linux Server: Ubuntu 20.04/22.04 or Debian 11+ recommended. Minimum 2 vCPU and 2GB RAM. Mail services include multiple components (antispam, antivirus, etc.), and 1GB RAM will struggle.
2. A Custom Domain: e.g., mydomain.com. Email services must be bound to a domain to deliver properly.
3. DNS Management Access: You'll need to add TXT, MX, and other records at your domain registrar.
4. Basic Docker Knowledge: Just knowing what docker-compose up -d does is enough. I'll break down every configuration parameter.

Why Mailu? Alternatives like Modoboa or iRedMail exist, but Mailu stands out because it's pure Docker-native, meaning zero host environment intrusion. Its components are modular and replaceable, and it ships with built-in Let's Encrypt integration and an antispam engine out of the box.

Quick Start: Spin Up the Service in 3 Steps

Mailu provides an online config generator, but to truly understand how it works, we'll manually write a minimal, production-ready docker-compose.yml and .env file.

Step 1: Create Directories and Config Files

## Create project directory
mkdir -p ~/mailu && cd ~/mailu

## Create .env environment variables file
cat > .env << 'EOF'
DOMAIN=mydomain.com
HOSTNAME=mail
TLS_FLAVOR=letsencrypt

## Database settings (Mailu defaults to SQLite. PostgreSQL is recommended for production)
DB_FLAVOR=sqlite

## Initial admin account
INITIAL_ADMIN_ACCOUNT=admin
INITIAL_ADMIN_PW=YourStrongP@ssw0rd!

## Antispam score threshold
ANTISPAM_SPAM_THRESHOLD=5
ANTISPAM_GREYLISTING=on
EOF

Key parameters explained:

• TLS_FLAVOR=letsencrypt: Instructs Mailu to automatically request a Let's Encrypt certificate. If you already have one, change to cert and mount it manually.
• HOSTNAME=mail: The final hostname becomes mail.mydomain.com, which will be referenced in DNS records later.
• TLS_FLAVOR and DB_FLAVOR: Mailu-specific env vars. All service containers read the same .env, preventing hardcoded credentials.

Step 2: Write docker-compose.yml

Create docker-compose.yml in ~/mailu:

version: "3.8"

services:
  # Antispam filter
  antispam:
    image: ghcr.io/mailu/rspamd:master
    restart: always
    env_file: .env
    volumes:
      - "./data/rspamd:/data"

  # Webmail interface
  webmail:
    image: ghcr.io/mailu/webmail:master
    restart: always
    env_file: .env
    depends_on:
      - front
    volumes:
      - "./data/webmail:/data"

  # Admin dashboard
  admin:
    image: ghcr.io/mailu/admin:master
    restart: always
    env_file: .env
    depends_on:
      - front
      - db
    volumes:
      - "./data/admin:/data"

  # Database
  db:
    image: ghcr.io/mailu/postgresql:master
    restart: always
    env_file: .env
    volumes:
      - "./data/db:/data"

  # Frontend gateway (Nginx + TLS termination)
  front:
    image: ghcr.io/mailu/nginx:master
    restart: always
    env_file: .env
    ports:
      - "80:80"
      - "443:443"
      - "25:25"
      - "110:110"
      - "143:143"
      - "465:465"
      - "587:587"
      - "993:993"
      - "995:995"
    volumes:
      - "./data/certs:/certs"
      - "./dkim:/dkim"

dns:
  driver: default

Notice the front service exposes multiple ports because email protocols are separated:

• 25/465/587 → SMTP (sending)
• 110/995 → POP3 (legacy receiving, rarely used now)
• 143/993 → IMAP (recommended receiving, supports multi-device sync)

The front container bundles Nginx, handling: port exposure, TLS encryption/decryption, and routing protocol requests to backend services. This architecture means you don't need to write custom Nginx configs; Mailu abstracts it perfectly.

Step 3: Start the Services

docker-compose up -d

On first boot, Mailu initializes the database, generates DKIM keys, and creates the admin account. Check logs to confirm:

docker-compose logs -f front   # Check Nginx and certificate issuance logs
docker-compose logs -f admin   # Check admin dashboard initialization

Hands-On: Configuring DNS & Managing Your Domain

DNS is the most common pitfall for mail servers. Even with a perfectly running service, missing or incorrect DNS records will cause Gmail, Outlook, etc., to outright reject your emails or drop them in spam.

3.1 Required DNS Records

In your domain's DNS management console, add these records (using mydomain.com as example):

Retrieve the DKIM public key after container startup from the ./dkim directory:

cat ./dkim/mailu/mydomain.com.dns

Output looks like:

mail._domainkey IN TXT "v=DKIM1; k=rsa; p=MIGfMA...long string..."

Copy the TXT value exactly and paste it into your DNS console.

3.2 Add Domain to Mailu Admin

After services are running, visit https://mail.mydomain.com/admin and log in with the initial admin credentials from .env.

1. Go to Domains → Add Domain
2. Enter mydomain.com
3. Mailu will automatically verify DNS records (wait for global propagation, usually 10 mins to 2 hours)
4. Once verified, a green checkmark appears indicating success.

3.3 Create User Mailboxes

In the dashboard → Users → Add User, create your first email account:

• Email: you@mydomain.com
• Password: Set a strong password

Send & Receive Your First Email

Once configured, visit https://mail.mydomain.com/webmail and log in. You'll see the familiar Webmail interface (Mailu ships with Roundcube by default).

Test Sending: Send a test email to yourself. It should arrive instantly.
Test Receiving: Send an email from Gmail or QQ to you@mydomain.com. With correct DNS, it will land in your inbox.

Emails not sending? Check this:
Most cloud providers (AWS, Alibaba Cloud, Tencent Cloud) block port 25 by default. You must:

1. Request port 25 unblocking via the cloud console (usually requires use-case justification)
2. If unblocking fails, relay through a third-party SMTP (SendGrid, Amazon SES, etc.). Mailu's antispam and admin features will still work seamlessly.

Common Issues & Troubleshooting

Q: Certificate issuance fails?
A: Let's Encrypt requires port 80 to be accessible and the domain to resolve correctly. Verify with curl -I http://mail.mydomain.com returning a 200 OK, then restart the front container.

Q: Emails marked as spam?
A: 90% of the time, DNS is incomplete. Verify SPF, DKIM, and DMARC are all configured. Use mail-tester.com to send a test email; it scores deliverability and gives fix suggestions. A score of 9+ usually guarantees Gmail inbox placement.

Q: Can I switch databases?
A: Yes. Change DB_FLAVOR=sqlite to postgresql or mysql in .env and configure connection strings. SQLite is fine for personal/small teams, but production strongly recommends a relational database.

Q: How to backup?
A: The ./data directory holds all persistent data. Simplest method: tar -czf mailu-backup-$(date +%F).tar.gz ./data. Restore by extracting to the same path. Mailu doesn't depend on external DB states, making migration straightforward.

Summary

Today, we walked through:

1. Bootstrapping Mailu's full stack with .env + docker-compose.yml
2. Configuring MX/SPF/DKIM/DMARC for reliable delivery
3. Adding domains and users via the admin dashboard
4. Completing send/receive tests via Webmail
5. Troubleshooting common port and certificate issues

Email infrastructure seems daunting, but with the right tools, it's just a set of well-coordinated containers. Mailu neatly packages Nginx, Postfix, Dovecot, Roundcube, and Rspamd under Docker, so you focus on business configuration instead of low-level protocols.

Next Steps: Explore Mailu's API (Admin → API Docs) for automation like bulk user creation or alias management. Also, consider enabling Passkey login for stronger security than traditional passwords. It's highly recommended for production.

Leave a comment if you hit any snags, and I'll gladly help troubleshoot.