DKIM, MTA-STS, and TLS-RPT: Complete Email Authentication

DKIM Overview

DomainKeys Identified Mail (DKIM) adds a cryptographic signature to outgoing email messages. The sending mail server signs selected headers and the message body with a private key. Receivers retrieve the matching public key from DNS and verify the signature. If anything in the signed content was altered in transit, verification fails and downstream policies (like DMARC) can act on the result.

DKIM does not encrypt mail or prevent delivery of unsigned messages on its own. Its value comes from proving that a message genuinely originated from an authorized system and was not tampered with after signing. For a broader overview of how DKIM fits alongside SPF and DMARC, see our email authentication deep dive.

Key Generation

Generate a 2048-bit RSA key pair. Keys shorter than 2048 bits are considered insufficient -- 1024-bit keys are still technically functional but increasingly vulnerable, and anything below 1024 bits is trivially crackable. Before generating keys, make sure your SPF record is already in place, as both protocols are prerequisites for DMARC enforcement.

Generate a private key and extract the public key using OpenSSL:

openssl genrsa -out dkim-private.pem 2048
openssl rsa -in dkim-private.pem -pubout -outform PEM -out dkim-public.pem

The private key stays on your mail server and must be protected. The public key gets published in DNS.

Extract the base64-encoded public key (stripping the PEM headers) for the DNS record:

grep -v "^-" dkim-public.pem | tr -d '\n'

Publishing the DKIM DNS Record

DKIM records are published as TXT records under a selector subdomain following the format selector._domainkey.yourdomain.com. The selector is an arbitrary label that identifies which key pair signed the message.

selector1._domainkey.example.com. IN TXT "v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA..."

Key fields in the record:

• v=DKIM1 -- version identifier (required).
• k=rsa -- key type. RSA is the standard; Ed25519 (k=ed25519) is supported by some implementations but not universally.
• p= -- the base64-encoded public key. Setting p= to an empty value explicitly revokes the key.

If the public key is too long for a single TXT string (common with 2048-bit keys), split it into multiple quoted strings within the same record. DNS automatically concatenates them:

selector1._domainkey.example.com. IN TXT ("v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A"
"MIIBCgKCAQEA2vJUpn9flRgPEbQ07bnFG...")

Provider-Specific DKIM Setup

Google Workspace

1. Open the Google Admin console. Navigate to Apps > Google Workspace > Gmail > Authenticate email.
2. Select your domain and click Generate New Record. Choose 2048-bit key length and a selector prefix (default is google).
3. Google provides the TXT record value. Add it to your DNS as google._domainkey.yourdomain.com.
4. Wait for DNS propagation (up to 48 hours, typically under an hour), then return to the Admin console and click Start Authentication.

Google manages the private key. You never handle it directly.

Microsoft 365

1. In the Microsoft 365 Defender portal, go to Email & collaboration > Policies & rules > Threat policies > Email authentication settings > DKIM.
2. Select your domain. Microsoft auto-generates two CNAME records pointing to Microsoft-hosted DKIM keys:
   • selector1._domainkey.yourdomain.com CNAME selector1-yourdomain-com._domainkey.yourdomain.onmicrosoft.com
   • selector2._domainkey.yourdomain.com CNAME selector2-yourdomain-com._domainkey.yourdomain.onmicrosoft.com
3. Publish both CNAME records in your DNS.
4. Return to the portal and enable DKIM signing. Microsoft handles key rotation automatically using the two selectors.

Key Rotation Strategy

DKIM selectors make zero-downtime key rotation straightforward. The process:

1. Generate a new key pair and publish the public key under a new selector (e.g., selector2._domainkey.yourdomain.com).
2. Reconfigure your mail server to sign outbound messages using the new selector and private key.
3. Keep the old selector's DNS record active for at least 7 days. Messages already in transit or sitting in queues may still need the old key for verification.
4. Remove the old DNS record after the transition period, or set p= to empty to explicitly revoke it.

Rotate keys at least once per year. Rotate immediately if you suspect the private key has been compromised. Using date-based selector names (e.g., jan2026, q1-2026) makes it easy to identify stale keys during audits. With SPF and DKIM in place, you are ready to deploy DMARC policy enforcement to tie both protocols together under a single policy.

MTA-STS: Enforcing TLS for Inbound Mail

Mail Transfer Agent Strict Transport Security (MTA-STS) tells sending mail servers that your domain requires TLS-encrypted connections for inbound mail delivery. Without MTA-STS, an attacker performing a network-level attack can downgrade SMTP connections to plaintext or redirect mail to a rogue server, even if your server supports TLS.

Setting Up MTA-STS

MTA-STS requires two components: a policy file served over HTTPS and a DNS record.

1. Host the policy file. Create a file at https://mta-sts.yourdomain.com/.well-known/mta-sts.txt with the following content:

version: STSv1
mode: enforce
mx: mail.yourdomain.com
mx: mx-backup.yourdomain.com
max_age: 604800

• version -- must be STSv1.
• mode -- testing (report failures but do not block), enforce (reject delivery over non-TLS connections), or none (disable the policy).
• mx -- list each MX hostname that senders should accept. Wildcards are supported (e.g., mx: *.yourdomain.com).
• max_age -- policy cache lifetime in seconds. 604800 is one week. Use shorter values during initial deployment.

The file must be served over HTTPS with a valid certificate on the mta-sts subdomain. This is non-negotiable -- senders will reject the policy if the certificate is invalid.

2. Publish the DNS record. Add a TXT record at _mta-sts.yourdomain.com:

_mta-sts.yourdomain.com. IN TXT "v=STSv1; id=20260302001"

The id value is an opaque string. Change it whenever you update the policy file -- senders use it to detect policy changes and re-fetch the file. A timestamp or incrementing number works well.

Deployment recommendation: Start with mode: testing to collect failure reports via TLS-RPT (below) before switching to mode: enforce.

TLS-RPT: TLS Reporting

SMTP TLS Reporting (TLS-RPT, defined in RFC 8460) lets receiving domains request reports about TLS connection failures from sending mail servers. When a sender cannot establish a TLS connection to your MX servers -- due to certificate errors, MTA-STS policy failures, or other issues -- it sends a JSON report to the address you specify. This gives you visibility into delivery problems that would otherwise be invisible.

DNS Record Format

Publish a TXT record at _smtp._tls.yourdomain.com:

_smtp._tls.yourdomain.com. IN TXT "v=TLSRPTv1; rua=mailto:tls-reports@yourdomain.com"

• v=TLSRPTv1 -- version identifier.
• rua= -- reporting address. Use mailto: for email delivery or https: for a reporting endpoint that accepts JSON POSTs. Multiple destinations can be comma-separated.

Reports are sent daily by participating senders (major providers like Google and Microsoft send them reliably). Each report contains the sending domain, your MX hostnames, the type of failure encountered, and a count of affected sessions.

Set up TLS-RPT before enabling MTA-STS enforcement. The reports will reveal certificate mismatches, DNS misconfigurations, and other issues that would cause mail rejection under an enforced MTA-STS policy.

Verification Commands

Verify DKIM DNS Record

dig +short TXT selector1._domainkey.yourdomain.com

You should see the v=DKIM1; k=rsa; p=... record in the response.

Verify MTA-STS DNS Record

dig +short TXT _mta-sts.yourdomain.com

Verify MTA-STS Policy File

curl -s https://mta-sts.yourdomain.com/.well-known/mta-sts.txt

Verify TLS-RPT DNS Record

dig +short TXT _smtp._tls.yourdomain.com

Test TLS on Your Mail Server

openssl s_client -connect mail.yourdomain.com:25 -starttls smtp -servername mail.yourdomain.com

This initiates a STARTTLS handshake and displays the certificate chain. Check that the certificate is valid, not expired, and matches the MX hostname listed in your MTA-STS policy. A mismatch between the MX hostname and the certificate's Subject Alternative Names is the most common cause of MTA-STS enforcement failures.

Verify DKIM Signature on a Received Message

Examine the Authentication-Results header in any received email. Look for dkim=pass along with the selector and domain used. If you see dkim=fail, check that the DNS record is correctly published and that the signing configuration on your mail server matches the selector in the record.