flowchart TD
U[User changes status] --> S[ejabberd]
S --> C1[Contact 1]
S --> C2[Contact 2]
S --> C3[Contact ...]
S --> CN[Contact N]
CN -.- L[one stanza per contact, per change,<br/>all at once, all equal priority]
Revamping ejabberd for a Modern Chat App
Four places where the XMPP core design meets a modern mobile messenger and does not quite fit, and what we did about each of them.
The first time half of our users reconnected at once after a network blip, the server had to route close to a million presence stanzas in the space of a few seconds. Nothing in that moment was broken, exactly. Every one of those stanzas was the protocol doing precisely what it was specified to do, and that turned out to be the interesting part.
A few years back I wrote a short post here about how XMPP works and said that when it comes to performance I would recommend ejabberd. I still would. But I have since spent a long time running a chat platform on a private fork of it, with custom modules, MySQL and Redis behind it and mobile clients in front, and moments like that reconnection storm gave me a much longer list of caveats than I had back then.
XMPP (Extensible Messaging and Presence Protocol) is a federated and decentralised open standard for messaging, federated in roughly the same sense as email, so anyone can run their own server and those servers talk to each other. It started as Jabber, created by Jeremie Miller in 1998 and opened up to the community in 1999, and it was formalised at the IETF in 2004 as RFC 3920 and RFC 3921, which were later revised in 2011 into the documents we actually use today, RFC 6120 for the core and RFC 6121 for instant messaging and presence, with RFC 7622 covering the address format.
So the standard is more than 25 years old and we still build on it, and the reason being that the core really is mature. Connection management, TLS, addressing and stanza routing are all solved here in a way that a new protocol would take years to reach. The core itself only deals with messaging and presence though, and everything else lives in separate extension documents called XEPs (XMPP Extension Protocols). Group chat is a XEP, message archiving is a XEP, push notification is a XEP. That split matters more than it sounds and most of this post is about why.
Two terms worth having before we go further, since everything below is built out of them. All XMPP traffic is made of stanzas, small self-contained XML fragments, and there are only three kinds. A <message/> carries content from one address to another, a <presence/> announces availability, and an <iq/> (Info/Query) is a request and response pair used for everything that needs an answer, from fetching your contact list to enabling a server feature. When you see a stanza count in this post, that is the unit being counted, one XML fragment the server has to route.
There are several server implementations, including ejabberd, Openfire, MongooseIM, Prosody and Google Talk back when Google still federated. ejabberd is the one we care about here. It was started in 2002 by Alexey Shchepin and has been maintained by ProcessOne since, and the name is just Erlang plus jabber plus daemon. It became popular in large part because WhatsApp ran on a heavily customised version of it, although WhatsApp has diverged a long way from upstream since and moved much of its protocol off XMPP proper, so that lineage is more historical than current now.
But WhatsApp shaped what people expect. When somebody says modern chat app today they mean something that behaves like WhatsApp, and every messaging product gets measured against that whether you like it or not. The technology moved on and the product expectations moved on, and the open standard mostly did not move with them, which is why proprietary services keep adopting XMPP and then customising it heavily. That is exactly what we did.
ejabberd is worth the attention partly because of Erlang. Concurrency, process isolation, supervision trees and hot code upgrades are the things Erlang was built for in telecoms, and a chat server turns out to be very close to the same problem shape, so it remains one of the better demonstrations of what Erlang is actually for. It also has both private and public forks, MongooseIM by Erlang Solutions being the best known public one.
Here we will go through four areas where vanilla ejabberd stopped being enough for us, what the standards do and do not give you in each, and what we either shipped or drafted in response.
ImportantDisclaimer
All of this comes out of running one system, plus the case studies we wrote around it, so some of it is specific to our deployment, our schema, our client mix and our traffic.
The fork worked and served real traffic for years. It was also never properly stress tested. We fixed whatever hurt, in the order it hurt, and the fixes that shipped were the ones we could ship without redesigning the protocol underneath.
That means there are two different kinds of material below and you should read them differently. The push notification and message delivery work actually shipped and those numbers are production measured. The presence and group chat work exists as draft extension specifications and test environment measurements, so those numbers are projections and we never ran them at production scale. I have marked which is which as they come up.
The design problems generalise. The numbers do not. If you are picking an architecture for a production chat system, take this as a list of questions worth asking and then do your own research and your own load testing.
Where these four problems sit
Before going through them one by one it helps to see where each one lives inside the server, so here is the shape of the deployment limited to the parts we are discussing.
mobile and web clients
|
XMPP over TLS (c2s)
|
+---------------------------v----------------------------+
| ejabberd |
| |
| ejabberd_c2s ------------------ ejabberd_sm |
| connection, stream mgmt session registry |
| | | |
| | [1] presence routing |
| | mod_roster |
| | |
| +------------------------ [2] group chat |
| | mod_muc / mod_mix |
| | |
| +------------------------ mod_mam |
| | message archive |
| | | |
| | [4] offline queue |
| | + acknowledgement |
| | |
| +------------------------ [3] push hooks |
| | |
+---------------------------------------|-----------------+
|
RabbitMQ exchange
|
+---------------------+---------------------+
v v
Android push workers iOS push workers
| |
v v
FCM APNS
storage MySQL roster, archive, message status
Redis caching, deduplicationEverything above the broker is stock ejabberd structure. What sits inside the custom modules is our own chat logic and stays out of this post. Configuration for all of it lives in /opt/ejabberd/conf/ejabberd.yml and the modules attach through ejabberd_hooks.
1. Presence and roster
In XMPP core, presence is bound to the roster, which is just the contact list, so when you change your status the server broadcasts that to every contact holding a presence subscription and those subscriptions are usually bidirectional. The cost of one status change is therefore O(roster size), which is not the same thing as the number of conversations the user is actually having, and that gap is the whole problem.
Lets see what that looks like with numbers. Take a fairly modest server with 10,000 users, an average roster of 200 contacts, and one percent of them changing presence in a given minute:
10,000 users x 200 contacts x 0.01 = 20,000 presence stanzas per minuteThat is around 333 stanzas a second on a server that is doing nothing interesting. Now think about what happens after a network blip when half of your users reconnect at the same time:
10,000 users x 200 contacts x 0.5 = 1,000,000 presence stanzas in secondsThat second number is the one that matters, and the reason being that presence load has very little to do with how busy your users are. It tracks how many contacts they have accumulated over the years and how many of them happen to reconnect together. It also gets worse than the raw arithmetic suggests, because every recipient is treated equally so the contact you messaged thirty seconds ago and the one you last spoke to in 2019 get the same stanza at the same time, and because full stanzas go out every time so changing a status string re-sends the entire presence stanza including entity capabilities and priority when exactly one child element changed.
The whole problem in one picture:
The usual answer here is roster versioning and it tends to get oversold. It was originally XEP-0237 which is now Obsolete, with the mechanism folded into RFC 6121 itself, so you should be citing the RFC and not the old XEP. What it does is let the client present the roster version it has cached and the server sends interim pushes for each item modified since then, covering additions, modifications and removals, which does save real bandwidth at session establishment. What it does not give you is partial or lazy loading, so there is no way to ask for the twenty contacts you actually talk to and page in the rest later, and sync stays binary. More importantly it does nothing whatsoever about the broadcast fan-out, which is the actual scaling problem.
Note: Roster versioning is an optimisation on how a client learns who its contacts are. It has no opinion at all on how many stanzas go out when one of those contacts changes status. Two separate problems, and it only touches the first one.
If you step back, presence is really a publish-subscribe problem. One publisher, many subscribers, and a high tolerance for staleness, since nobody is harmed if an away status reaches a distant acquaintance eight seconds late. The XSF saw this too and the MIX family includes XEP-0403 (MIX-Presence), which shares presence through a channel presence node that participants subscribe to explicitly, so pubsub semantics decoupled from the roster. The right idea. Its status tells you how that went though, because XEP-0403 is Deferred, which in XSF process terms means an Experimental document that has gone twelve months without an update, and it carries a warning that implementation is not recommended for production. So if you want pubsub presence on XMPP today you build it yourself.
To be fair, the standards did ship one real mitigation, and any honest treatment of this problem has to address it. XEP-0352 (Client State Indication, or CSI, Stable) lets a client tell the server it has gone inactive, typically when the app moves to the background, and the server is then free to hold or drop presence updates until the client is active again. XEP-0286 (Mobile Considerations) recommends exactly this for mobile deployments, and if you run XMPP for mobile clients you should be using it. The reason it does not close the problem is direction. CSI is receiver-driven and binary, meaning each client can say I am not looking, hold my updates, and nothing more. It does nothing on the sending side, where one status change still fans out to every active contact in the roster at once, all with equal priority. What we wanted was sender-driven and graded, which is a different mechanism, and the two compose rather than compete. On the payload side there is prior art too, since XEP-0115 (Entity Capabilities, Stable) exists precisely to shrink what each presence stanza carries, and its intended successor XEP-0390 is Deferred, which by this point in the post should sound familiar.
What we proposed
We drafted an extension we call Graded Presence Delivery (GPD). Not to be confused with US patent 9,299,111, “Efficient presence distribution mechanism for a large enterprise”, which sits in the same space but addresses a different problem, a server-to-server ring topology for enterprise presence rather than the client delivery discipline described below.
The constraint we set ourselves was compatibility, so every subscriber still eventually receives presence and we are only changing the delivery discipline rather than the guarantee, with clients opting in and everyone else continuing to get standard broadcasts. The client enables it with an IQ set:
<iq type='set'
from='romeo@montague.example/orchard'
to='montague.example'
id='enable1'>
<enable xmlns='urn:xmpp:gpd:0'>
<delta-updates>true</delta-updates>
<priority-threshold>50</priority-threshold>
</enable>
</iq>Instead of iterating the roster on every presence change we keep a few bloom filters per user, a bloom filter being a space efficient probabilistic set that tells you either definitely not present or probably present while using a fraction of the memory a real set would need. We keep three of them, holding contacts with recent interactions, contacts explicitly marked as high priority, and contacts who recently viewed the user profile, so membership checks come out O(1) whether the roster holds 20 contacts or 2,000.
%% Classification becomes a filter lookup instead of a roster scan.
classify_recipient(SenderJID, RecipientJID) ->
Key = jid:to_binary(RecipientJID),
case bloom:member(active_filter(SenderJID), Key) of
true -> immediate;
false -> classify_by_interaction(SenderJID, RecipientJID)
end.Note: Bloom filters give you false positives but never false negatives, and for this design that direction is the safe one, since a false positive only means a contact gets promoted into an earlier delivery wave than they deserved. Somebody gets an update sooner rather than missing one. You will still need to tune filter size and hash count against your own roster distribution, and rotate the filters so that recent actually stays recent.
Recipients then get sorted into priority tiers by last interaction time and each tier goes out on its own schedule rather than all at once.
| Tier | Roughly | Delivered at |
|---|---|---|
| Immediate | ~25 highest value contacts | t+0s |
| High | ~75 recent contacts | t+2s |
| Normal | ~150 regular contacts | t+10s |
| Low | everyone else | t+30s |
The scoring behind that is deliberately boring:
-define(HOURS(N), (N) * 3600).
determine_priority(SenderJID, RecipientJID) ->
case last_interaction(SenderJID, RecipientJID) of
{ok, LastSeen} ->
Elapsed = erlang:system_time(second) - LastSeen,
if Elapsed < ?HOURS(24) -> high;
Elapsed < ?HOURS(168) -> medium;
true -> low
end;
not_found ->
medium
end.A client can also hint at priority directly on a roster item, which is useful when the user has explicitly pinned somebody:
<iq type='set'
from='romeo@montague.example/orchard'
id='priority1'>
<query xmlns='jabber:iq:roster'>
<item jid='mercutio@montague.example'>
<group>Friends</group>
<gpd:priority xmlns:gpd='urn:xmpp:gpd:0' value='80'/>
</item>
</query>
</iq>As a rule of thumb those thresholds should move with server load, so under pressure you tighten the high priority window and shrink the batch size, and the system sheds deferred work first while keeping active conversations sharp. The stanza itself carries wave metadata so the client knows where it sits in the sequence:
<presence from='romeo@montague.example/orchard'
to='juliet@capulet.example/balcony'>
<show>away</show>
<status>Writing sonnets</status>
<gpd:wave xmlns:gpd='urn:xmpp:gpd:0'
wave='1'
total-waves='4'
priority='immediate'/>
</presence>On the bandwidth side, a full presence stanza carries show, status, priority and entity capabilities, so changing one status string re-sends all of it. Delta encoding sends only the elements that actually moved and falls back to a full stanza when the delta would be larger:
<presence from='romeo@montague.example/orchard'
to='juliet@capulet.example/balcony'>
<gpd:delta xmlns:gpd='urn:xmpp:gpd:0' seq='42'>
<show>away</show>
</gpd:delta>
</presence>The seq attribute is doing real work there. Without it a client that misses a delta has no way of knowing its cached state has gone stale.
Waves do mean some contacts hold slightly stale presence for a few seconds, so a client that needs current state asks for it rather than waiting:
<iq type='get'
from='juliet@capulet.example/balcony'
to='montague.example'
id='fullstate1'>
<request-state xmlns='urn:xmpp:gpd:0'>
<contact jid='romeo@montague.example'/>
</request-state>
</iq>Note: This is the part that needs care. The obvious failure mode is a stampede where a thousand clients all request on demand presence in the same moment and you have rebuilt the load spike you were trying to remove. You want jitter on these, and you want them batchable so a client can ask about thirty contacts in one round trip instead of thirty.
Putting the four pieces together:
flowchart LR
P[Presence change] --> B[Bloom filters<br/>classify recipients]
B --> Q[Priority tiers,<br/>by last interaction]
Q --> W1[Wave 1<br/>t+0s]
Q --> W2[Wave 2<br/>t+2s]
Q --> W3[Wave 3<br/>t+10s]
Q --> W4[Wave 4<br/>t+30s]
W1 & W2 & W3 & W4 --> D[Delta or full stanza,<br/>per recipient]
There is a roster side to the same problem as well. You want lazy paginated roster loading ordered by last interaction, so a client with 2,000 contacts becomes usable after the first page instead of after a full sync, and presence broadcast then follows the same ordering so the contacts you actually talk to learn you are online first. Anybody outside the early pages falls back to the jittered on demand requests. None of that is exotic, it is just ordinary product engineering applied to a protocol that predates the constraint.
It is worth saying that of the three extensions we drafted, this roster side is the one with the cleanest gap in the standards. Presence has shipped and stalled answers nearby (CSI, XEP-0403), group chat has three competing ones, but I have not found any active proposal for lazy or paginated roster loading at all. If any of this work were ever taken to the XSF, this is the piece I would submit first, and the sensible opening move would be a mail to the standards list describing the problem rather than the solution, to find out whether anyone has attempted it before spending weeks on a formal specification.
I want to be honest about the size of this, because just add bloom filters reads much cheaper than it is. In ejabberd the work lands in prioritised routing and wave scheduling inside ejabberd_sm, contact priorities and interaction tracking inside mod_roster, a new delta engine, and a new module for the protocol itself with its opt-in handling and filter management. Call it 2,000 to 2,600 lines across four components, plus schema changes for interaction history and priority hints, plus caching for the filters and presence state, and all of that before the tests you would need to actually trust it.
The projected gains, and these are test environment measurements rather than production, sit around 70 to 80% less CPU for presence processing, 50 to 60% less bandwidth, and roughly 90% smoothing of the reconnection spike. Percentages should come with their arithmetic, so here is where each one comes from.
The 90% is the wave arithmetic. Take a 250 contact roster, which is what the wave table above adds up to. Only wave one, roughly 25 contacts, goes out during the reconnection window itself, and the remaining 225 are spread over the next 30 seconds, so 1 minus 25/250 puts 90% of the burst outside the spike.
The 50 to 60% is the delta arithmetic. A full presence stanza in our test environment averaged about 450 bytes once entity capabilities ride along, and a delta carrying one changed element about 125 bytes, which is a 72% cut per eligible update. Not every update is eligible though. First deliveries need full state, a delta that would exceed the full stanza falls back, and clients that never opted in keep receiving standard broadcasts, so blended across a realistic mix it lands between 50 and 60.
The 70 to 80% is a measured term plus a projected one. In a 1,000 update test run, presence CPU time dropped from 25.2 seconds to 8.8, a 65% cut, mostly because classification became an O(1) filter check instead of an O(roster) scan and stanza construction happened once per wave batch instead of once per recipient. The band stretches to 70 to 80 on the projection that delta encoding and larger rosters compound the saving, and that stretch is exactly the part to trust least. Treat the shape of the improvement as the claim and not the digits.
Note: The moment you introduce waves, everyone receives presence stops being obviously true and becomes something your implementation has to enforce. A node restarting mid wave, a scheduler falling behind under load, a client disconnecting between wave two and wave four, each of those is a way for an update to quietly disappear. Standard broadcast is wasteful but it is trivially correct, and you are giving that correctness up on purpose.
TODO: Predictive wave sizing based on observed reconnection patterns, and federated presence optimisation across server boundaries. Neither got beyond a sketch.
2. Group chat
XEP-0045 (Multi-User Chat) is Stable, widely implemented and still required by the XMPP Compliance Suites, and it also dates from the IRC era so the assumptions show. You join a room and you are in it only while connected, on joining you receive presence for every occupant, the full occupant list is part of how the room works, and you leave when you disconnect.
Now describe a modern group chat instead. Membership is persistent so you are in the group whether or not your phone has signal, groups run to thousands of members, nobody wants 5,000 presence stanzas when they open the app, and the member list is something you fetch when you tap the group info screen. Every one of those is the opposite of what MUC assumes, so the mismatch is structural rather than something you can tune your way out of. On the other hand MUC is what the Compliance Suites require, so you cannot simply walk away from it either.
Before comparing the protocols though, we need to be precise about what group chat costs you in storage, because this is the part that surprised us most. XEP-0313 (Message Archive Management, or MAM) is Stable and stores messages server side, which is fine for one to one chat, but for group chat the default behaviour stores the message once per recipient. So a single message to a 500 member group produces 500 archive writes and 500 stored copies of identical content, and a 10 KB message ends up costing 5 MB of storage.
The daily arithmetic is simple enough:
storage_operations_per_day = messages_per_day x average_group_sizeFor 10,000 users, 1,000 groups averaging 50 members and 50,000 messages a day, that is 2,500,000 write operations per day on a deployment that a spreadsheet would call small, and storage grows with member count times message count rather than just message count.
flowchart TD
M[One 10 KB message<br/>to a 500 member group] --> MAM[MAM, default behaviour]
MAM --> A1[Archive row,<br/>member 1]
MAM --> A2[Archive row,<br/>member 2]
MAM --> A3[Archive row,<br/>member ...]
MAM --> A500[Archive row,<br/>member 500]
A1 & A2 & A3 & A500 --> T[500 writes,<br/>5 MB stored]
Note: MAM behaving this way is perfectly reasonable for the case it was designed for. The trap is that it is the default, so you inherit the write amplification without ever having chosen it, and by the time your archives are large enough for query performance to degrade the schema is load bearing and the migration is expensive. Decide who owns message history, the server or the client, before you have users.
WhatsApp class systems made the opposite choice, where the client device is the archive and the server only stores undelivered messages with a TTL and acts mostly as a router. That has its own costs around multi device sync and history restore on a new phone, so it is not free either, but it is a deliberate decision and it determines your database architecture and your scaling ceiling.
There are three answers going around for the group chat gap and none of them is comfortable.
XEP-0369 (Mediated Information eXchange, or MIX) is the intended MUC successor and it separates channel membership from presence properly, so it is the standards track answer. It is also still Experimental, ejabberd’s own documentation labels mod_mix experimental, and the ProcessOne launch announcement warned that it would have trouble scaling, exposed denial of service opportunities and should not be relied on for large scale production, which is the vendor talking about their own module. The rest of the family is in a similar state, with XEP-0405 (MIX-PAM) Experimental and XEP-0406 (MIX-ADMIN) Deferred.
MucSub is ejabberd’s own answer, a subscription layer sitting on top of standard MUC that lets a user subscribe to room events without being an active occupant. It solves the persistent membership half of the problem and leaves MUC room semantics intact, and ProcessOne positioned it explicitly as a bridge to be used until the community is ready to stand behind MIX. It has no XEP number. Its practical appeal is real though, because it is the smallest change from a working MUC deployment and on ejabberd it is the only option here you can adopt without rewriting your group chat layer.
MUC Light is MongooseIM’s, a deliberately minimal room protocol under the namespace urn:xmpp:muclight:0. It was submitted to the XSF as a ProtoXEP and never advanced to an accepted number, so it lives in MongooseIM’s open extensions rather than in the XEP series. Its design choices are exactly the ones the storage problem above argues for, with owner and member affiliations instead of a complex permission model, member lists queried on demand rather than pushed, and no presence broadcast inside rooms at all. That last one is the single largest win available in group chat and MUC Light is the only one of the three that does it by design.
| MIX | MucSub | MUC Light | |
|---|---|---|---|
| Standing | XEP-0369, Experimental | ejabberd vendor extension | MongooseIM open extension |
| Persistent membership | Yes | Yes | Yes |
| In-room presence broadcast | Yes | Yes, MUC semantics | No |
| Member list | Full list distributed | MUC occupant model | On demand |
| Effort on ejabberd | Module exists, not production grade | Lowest, already there | Highest, not native |
| Standards alignment | Best | None | None |
| Scales to thousands | Not demonstrated | Inherits MUC costs | Yes, by design |
Read that honestly and MUC Light has the best architecture with the worst adoption story, while MucSub has it the other way round.
What we proposed for MIX
The problems with MIX at scale are not mysterious. Messages go out one participant at a time with no batching and no prioritisation so cost grows directly with group size, all presence changes reach all participants, complete participant lists get sent to everybody and membership changes push to everybody, and the storage fan-out above goes unaddressed. Our Scalable MIX draft went after each one. Presence changes become deltas rather than full state:
<presence-delta xmlns='urn:xmpp:mix:scale:0'
from='user@example.com'
channel='channel@mix.example.com'
seq='42'>
<changes>
<status>away</status>
</changes>
</presence-delta>Participant lists get paginated instead of arriving whole:
<iq type='get'
from='user@example.com/mobile'
to='channel@mix.example.com'
id='roster1'>
<participants-page xmlns='urn:xmpp:mix:scale:0'
index='0'
max='50'/>
</iq>Membership changes travel as deltas against a version:
<membership-delta xmlns='urn:xmpp:mix:scale:0'
channel='channel@mix.example.com'
version='123'
prev-version='122'>
<added>
<item jid='user5@example.com' nick='Charlie' role='member'/>
</added>
<removed>
<item jid='user3@example.com'/>
</removed>
</membership-delta>And the storage fan-out gets attacked directly by sending a content reference with a short preview instead of duplicating the whole body per recipient:
<message to='user@example.com'
from='channel@mix.example.com'
id='msg123'
type='groupchat'>
<content-ref xmlns='urn:xmpp:mix:scale:0'
hash='sha-256:a1b2c3...'
size='1024'
type='text/plain'>
<info>First 150 characters of the message, for preview.</info>
</content-ref>
</message>Projected effect, again projections and not production, is roughly 70 to 80% less server resource for large groups and 50 to 60% less bandwidth, with groups of a thousand or more becoming workable. The derivation is the fan out arithmetic from the storage section run in reverse, on the same 500 member group. Distribution batching turns 500 individual stanza constructions into one per server cluster. Presence tiering means only the high priority slice of members receives broadcasts at all, and if roughly 10% of a large group is active at any moment, that alone removes 90% of presence sends. Content references shrink the per member payload from the full 10 KB body to a reference of a couple of hundred bytes for every member who never opens the message. Blend those three across a realistic mix of small and large messages and active and idle members and you land around 70 to 80% for server work and 50 to 60 for bandwidth. Same caveat as before, since the blend ratios are assumptions, trust the direction more than the digits.
If you rebuild group chat storage around MUC Light semantics instead, the win is simple to state. The message body is stored once per group, not once per member. What stays per member is only the acknowledgements, a delivery row and a read row, small fixed-size records, while all 500 members share the single stored copy of the message itself. Run the earlier numbers through that design and the 10 KB message that cost 5 MB under default MAM now costs the 10 KB body plus 500 status rows of about a hundred bytes each, call it 60 KB, which is roughly 99% less stored and one body write instead of 500. That is the whole trick. Per message cost now scales with one body plus tiny rows rather than body times members, which is what makes group chat stay light as groups grow. We backed it with a wide column store for the bodies, since that workload is append heavy time series data, and kept the acknowledgement rows relational where transactions and indexes live. In test environment comparisons that shape held delivery latency roughly flat as group size grew toward 10,000 members, while a single relational database MIX deployment that still duplicated bodies per recipient degraded noticeably past about a thousand and struggled with hot groups during spikes.
So the design argument is not close. The adoption argument is a completely different matter though, because MUC Light is not native to ejabberd so you are porting a protocol across from another server, it is a single vendor extension so your client work is bespoke and there is exactly one implementation to learn from, the hybrid store means operating two database technologies plus a cache with cross database consistency to manage and hiring for both, and migrating an existing MUC or MIX deployment touches storage, protocol and every client you have shipped.
As a rule of thumb, pick the option whose failure mode you can actually afford. MucSub fails by inheriting MUC’s costs, which at least you can measure today. MIX fails at a scale you may never reach. MUC Light fails by being expensive to adopt and lonely to maintain.
TODO: A migration path from MucSub to MUC Light semantics that does not need a flag day. We sketched it and never finished it.
3. Push notifications
This section and the next describe work that actually shipped, so the numbers here are production measured.
XEP-0357 (Push Notifications) is Deferred, having expired out of Experimental through inactivity, and on its own that would only be a caveat. The real issue is what it covers. ejabberd’s mod_push documentation states plainly that it does not generate APNS (Apple Push Notification service) or FCM (Firebase Cloud Messaging) notifications directly, and that it is designed to work with app servers operated by third party vendors which then trigger delivery to the device.
So the standard handles the part XMPP already knew how to do, which is noticing that a user is offline and emitting an event, and everything after that is yours. Device token management including tokens that expire or rotate or belong to an app somebody uninstalled, platform specific payload construction for each vendor format, retries along with the judgement about what is even worth retrying, rate limiting so you do not get throttled, deduplication so a recovery scenario does not spam your users, and priority lanes because a call notification arriving forty seconds late is useless. That list is the entire operational difficulty of push and none of it appears in the specification.
Our original path was synchronous, so an offline message triggered a direct call toward the platform push services from inside the messaging core. That works until it does not. When a third party push endpoint goes slow, and they do, the back pressure lands on your XMPP server, and somebody else’s degraded API turns into your messaging latency.
Note: Any synchronous call from a hot path to a third party you do not control is this same bug wearing different clothes. The question to ask is not whether that API is reliable, it is what happens to your core loop when it takes thirty seconds to respond. If the answer involves your users noticing, the call belongs behind a queue.
The fix was to decouple delivery from the messaging core using RabbitMQ as a broker, so ejabberd publishes a notification event and moves on while dedicated push workers consume from the broker and deal with the vendors, one set for Android and one for iOS as in the diagram earlier. Three things came out of that. The messaging core stops waiting, since publishing to a local broker is fast and predictable and a vendor outage now fills a queue instead of stalling message delivery. Time critical notifications get their own lane, because call notifications and ordinary message notifications have very different deadlines and one queue for everything means your most urgent traffic waits behind your least urgent. And the workers become independently deployable, which matters more than it sounds because push logic changes far more often than messaging logic, vendors being what they are.
Before, the synchronous path, where a slow vendor becomes your latency:
flowchart LR
E1[ejabberd] -->|synchronous call, back pressure| V1[FCM / APNS]
After, decoupled behind the broker:
flowchart LR
E2[ejabberd] -->|publish, move on| R[RabbitMQ] --> W[Push workers] --> V2[FCM / APNS]
For reference the standard protocol side of this is just a client enabling push against its own app server:
<iq type='set' id='enable1'>
<enable xmlns='urn:xmpp:push:0'
jid='push-service.example.com'
node='device-node-id'/>
</iq>Everything past that IQ is the part you build. The module registers against ejabberd hooks to notice when a notification is warranted:
start(Host, Opts) ->
ets:new(?RABBITMQ_CONNECTION_TABLE,
[set, public, named_table, {read_concurrency, true}]),
ejabberd_hooks:add(store_mam_message, Host, ?MODULE, mam_message, 50),
Proc = gen_mod:get_module_proc(Host, ?PROCNAME),
ChildSpec = {Proc, {?MODULE, start_link, [Host, Opts]},
transient, 1000, worker, [?MODULE]},
supervisor:start_child(ejabberd_backend_sup, ChildSpec).You need to hook store_mam_message alongside offline delivery, the reason being that a message which gets archived and a message which goes to offline storage take different code paths, so a notification system watching only one of them will miss notifications in ways that are genuinely painful to debug later.
Recovery scenarios also generate duplicates, since a node restarts, a message gets reprocessed and your user gets the same notification twice. We used Redis with a short expiry as the guard:
notify_once(MessageId, Packet) ->
RedisKey = dedupe_key(MessageId),
case ejabberd_redis:get(RedisKey) of
{ok, undefined} ->
ejabberd_redis:set_async(RedisKey, MessageId, ?DEDUPE_EXPIRY),
notify(Packet);
_ ->
?WARNING_MSG("ignoring duplicate message: ~p", [MessageId]),
Packet
end.The expiry window is the tuning knob there. Too short and duplicates slip through during a slow recovery, too long and you are holding a large keyspace for no benefit. Ours ended up measured in minutes, sized against how long a worst case redelivery actually took.
Where we really got hurt was connection management. The first version leaked RabbitMQ connections under load and exhausted channels, so we ended up tracking broker connections in ETS (Erlang Term Storage) and checking liveness before publishing:
ensure_channel(ChannelPid, Host) ->
case is_pid(ChannelPid) andalso is_process_alive(ChannelPid) of
true -> ChannelPid;
false -> reconnect_channel(Host)
end.Worth pausing on why this module carries two caches rather than one. ETS lives inside the Erlang node, so a lookup is a memory read with no network hop, which is what you want on a path that runs for every single notification, and the only things kept there are a handful of connection handles that would be meaningless on any other node anyway. Dedupe keys have the opposite requirements. A duplicate can surface on a different node than the one that saw the message first, so that state has to be visible across the cluster, and it has to expire on its own or the keyspace grows without bound as messages flow through. Redis gives you both, cluster-wide visibility and TTL expiry, at the cost of a network round trip you can afford once per message but not once per lookup on hot connection state. As a rule of thumb, node-local and tiny goes in ETS, cluster-wide or self-expiring goes in Redis, and nothing goes into either without a bound on how large it can grow.
Note: Connection leaks are load dependent so you will not see them in development, where processes get recycled faster than they accumulate. Under production traffic it is a slow climb to exhaustion, and the symptom, which is that notifications stop, looks nothing at all like the cause, which is that channel handles were never released. If you are putting a broker on a hot path, monitor connection and channel counts as first class metrics from day one and not after the first incident.
What all of it bought us, measured in production, was peak throughput above 50,000 notifications per minute which was roughly four times the direct approach, a 99.8% delivery success rate with the broker buffering absorbing third party outages, and average delivery latency down by around 65%. The delivery success number is the one I would point at, because the broker did not make the vendors any more reliable. It turned a vendor outage into a queued notification instead of a lost one.
4. Message delivery and acknowledgements
ejabberd’s offline message store holds messages for users who are not connected and applies a storage quota, which is a fixed number of messages and by default the same number for everybody. That sounds reasonable enough, and as a defence against one user’s queue growing without bound it is.
Now think about who actually hits that limit though. It is never the idle account, because an idle user receives few messages and their queue stays small however long they stay away. The user who hits the quota is the one in several active conversations and a busy group or two who happens to have their phone off for a few hours, which is to say your most engaged user. And when the quota fills, messages are dropped. Not delayed, dropped, silently, from the point of view of both sender and recipient.
Note: Check what your server actually does at the quota boundary, because the behaviour at the limit matters more than the limit itself. Some configurations discard the oldest message, some reject the newest, and some bounce an error that the sender’s client may never surface. All three are message loss as far as the user is concerned. We have a quota and we lose messages turn out to be the same sentence.
Raising the quota buys you time and fixes nothing, the reason being that a fixed number applied uniformly does not match a usage distribution that is nowhere near uniform, and a quota generous enough for your heaviest user has stopped protecting you from anything.
What we did instead was stop treating the offline store as the mechanism of record for undelivered messages. MAM already stores the conversation, so if a message is in the archive then it exists whether or not it also sits in an offline queue, which makes the archive the source of truth and takes the queue off the critical path. Around that you want to cache the recent window, since most reconnects need the last stretch of each active conversation rather than the full history and keeping that in a cache means the common case never touches the archive tables, which is the same query pattern that degrades as archives grow. Then you use push as the delivery signal, so rather than hoping the user reconnects before their queue overflows, a notification tells the device something is waiting and the client syncs from the archive.
Delivery then stops being a question about queue capacity and becomes a question about archive writes and sync, both of which you can monitor. Note that this only works because of the push work in the previous section. Without a reliable push path nothing prompts the client to sync in the first place.
The other half of this is acknowledgements. XEP-0184 (Message Delivery Receipts) is Stable and does exactly what it says, so when you receive a message you send back a receipt, and for a live conversation that is fine. The problem is catching up. A user opens the app after a few hours with 50 unread messages in a conversation, which under the standard model means 50 stanzas from client to server and another 50 from server to the original sender forwarding each receipt. That is 100 stanzas to communicate a single piece of information, namely that this person has now read up to here. Multiply it by everybody reconnecting after an outage, which is the same mass reconnection scenario from the presence section, and receipts turn into their own load spike.
There is no bulk acknowledgement in the standard, so we built one. The client batches the message IDs into a single stanza:
<message from='user@example.com' to='contact@example.com' id='ack-1'>
<seen_messages chat_type='chat'>
<seen_message id='msg-1' jid='contact@example.com' time='...'/>
<seen_message id='msg-2' jid='contact@example.com' time='...'/>
<seen_message id='msg-3' jid='contact@example.com' time='...'/>
</seen_messages>
</message>The server updates all of them in one transaction and sends a single batched response back to the original sender:
<message from='contact@example.com' to='user@example.com' type='chat'>
<acknowledgements chat_type='chat'>
<acknowledge id='msg-1' type='seen'/>
<acknowledge id='msg-2' type='seen'/>
<acknowledge id='msg-3' type='seen'/>
</acknowledgements>
</message>The whole exchange, standard and batched:
sequenceDiagram
participant R as Recipient
participant S as ejabberd
participant O as Original sender
Note over R,O: Standard XEP-0184, 50 unread messages
loop 50 times
R->>S: receipt for one message
S->>O: forwarded receipt
end
Note over R,O: Bulk acknowledgement
R->>S: seen_messages, 50 ids in one stanza
S->>S: one UPDATE, one transaction
S->>O: acknowledgements, 50 ids in one stanza
The arithmetic is the whole argument here. For N messages the standard approach costs 2N stanzas while the batched approach costs 2 whatever N happens to be, so acknowledging 50 messages goes from 100 stanzas down to 2. On the server side the update wants to be one statement rather than a loop:
UPDATE message_status
SET status = 2, seen_at = NOW()
WHERE status != 2
AND to_user = ?
AND message_id IN (...);You need to wrap that in a transaction, because a partial update leaves some messages marked seen and others not and the client has no way to detect the gap, so it will simply believe it acknowledged everything.
Note: Batching moves the failure rather than removing it. One stanza carrying 50 acknowledgements is one stanza that can fail and take 50 acknowledgements with it, so where a lost receipt used to be one wrong read indicator it is now fifty. Make the batch operation idempotent so a retry is harmless, and put a bound on the batch size, because acknowledge everything since I was last online is unbounded by definition and will eventually meet a user who was away for a month.
The protocol change is the easy part of that. Rolling it out is not, because you do not control which version of your app your users are running, and an older client will keep sending individual receipts and will not understand a batched response. So the server has to detect client capability and fall back, which we keyed on the client’s reported app version per platform against a configured minimum, generating individual receipts for anybody below it. That fallback path is not temporary scaffolding either. It lives in your codebase for as long as you support old app versions, which in practice means years, and any protocol extension you add on the client side comes with that cost attached whether you counted it or not.
So, vanilla or fork
Four areas and the same pattern underneath all of them. XMPP was specified for a world of always on desktop clients, small rosters, ephemeral room membership and servers acting as archives, and every one of those assumptions is inverted on mobile. The protocol is not wrong, it is just answering a question that is no longer the one being asked.
Every serious Erlang XMPP deployment has responded the same way, by extending past the standards. MongooseIM shipped MUC Light as its own open extension, ProcessOne shipped MucSub as a bridge over MUC while its MIX implementation stayed experimental, and we built a custom push pipeline and drafted presence, roster and group chat extensions. Three teams, working separately, all reaching the same conclusion, which is that the standards give you a reliable core and an incomplete edge, and the edge is where modern chat actually lives.
That leaves a real decision. You can stay on vanilla ejabberd, stick to stock modules and stable XEPs, apply the kind of fixes in the push section which is broker decoupled delivery, disciplined caching and managed infrastructure, and accept the ceiling that comes with it. Roster coupled presence, chat room era groups, server heavy storage. Below roughly the hundreds of thousands of users mark, or anywhere federation and protocol conservatism matter more than product polish, this is the right choice and everything above is a distraction. Or you fork and redesign, keeping the mature core which is connection management, TLS, stanza routing and the Erlang/OTP supervision model, and replacing the parts that fight you at scale. MongooseIM is the existence proof that this works. It is also a large ongoing commitment to protocol work that nobody else is going to maintain for you.
There is no universally right answer there, but there is a dishonest one, which is assuming vanilla ejabberd plus configuration will carry a modern messenger to scale. It will not, and the place you find that out is production.
Our own fork sat somewhere in the middle, which is the honest and slightly uncomfortable answer. We fixed whatever hurt in the order it hurt and never stress tested the result properly. The presence and group chat work is what we would have built with more time, the push and delivery work is what we actually shipped, and I have tried to keep that line visible throughout because it is probably the most useful thing here.
Personally I have never run MongooseIM or Prosody in production, so I cannot tell you how MUC Light behaves at scale from my own experience, only from what we measured in a test environment. What I can say is that ejabberd held up well for a long time and the places it stopped holding up were all places where the standard, and not the server, had run out of road. I would still recommend it. I would just go in knowing which four things you are eventually going to have to build yourself.
Appendix: standards status
Checked against the current documents on xmpp.org in August 2026. Worth noting that the XSF renamed the Draft status to Stable back in 2021, so older documents citing Draft mean what is now called Stable. Statuses are shown both for the period the work happened and as they stand today.
| Standard | Title | During the work (2022 to 24) | As of Aug 2026 | Version |
|---|---|---|---|---|
| RFC 6120 | XMPP Core | Proposed Standard | Proposed Standard | 2011 |
| RFC 6121 | XMPP IM and Presence | Proposed Standard | Proposed Standard | 2011 |
| XEP-0045 | Multi-User Chat | Stable | Stable | 1.35.5 |
| XEP-0085 | Chat State Notifications | Final | Final | 2.1 |
| XEP-0115 | Entity Capabilities | Stable | Stable | 1.6.0 |
| XEP-0184 | Message Delivery Receipts | Stable | Stable | 1.4.0 |
| XEP-0198 | Stream Management | Stable | Stable | 1.6.3 |
| XEP-0237 | Roster Versioning | Obsolete, folded into RFC 6121 | Obsolete | 1.3 |
| XEP-0286 | Mobile Considerations on LTE Networks | Active | Active | 1.0.0 |
| XEP-0313 | Message Archive Management | Stable (since late 2021) | Stable | 1.1.3 |
| XEP-0352 | Client State Indication | Stable | Stable | 1.0.0 |
| XEP-0357 | Push Notifications | Deferred | Deferred | 0.4.1 |
| XEP-0369 | MIX Core | Experimental | Experimental | 0.14.6 |
| XEP-0390 | Entity Capabilities 2.0 | Deferred | Deferred | 0.3.2 |
| XEP-0403 | MIX-Presence | Deferred | Deferred | 0.3.2 |
| XEP-0405 | MIX-PAM | Experimental | Experimental | 0.5.3 |
| XEP-0406 | MIX-ADMIN | Experimental, then Deferred in 2023 | Deferred | 0.3.1 |
| MUC Light | MongooseIM open extension, urn:xmpp:muclight:0 |
No XEP number | No XEP number | n/a |
| MucSub | ejabberd vendor extension | No XEP number | No XEP number | n/a |
P.S:
If you are starting a new deployment rather than fixing an existing one, the single decision that constrains everything else is who owns message history, the server or the client. Make that one deliberately and early. Almost every problem in this post gets easier or harder depending on which way you went.
I hope this post saves you a few of the lessons we paid for in production. Happy building 🙂