We're trying something new. Well, trying two new things.
We're starting to blog about Wattpad's journey into the world of microservices. And we're blogging on our own platform.
We're proud of the really hard problems we have to solve every day and w...
Orchestration: In a distributed system, "a local view from the perspective of one participant". A manager coordinates the actions of many supporting actors to complete a task.
Choreography: "Coordination from a global multi-participant perspective, without a central controller." Each actor both publishes events of potential interest to other actors and subscribes to events of interest from other actors. A task is completed when all supporting actors have finished reacting to the seed event and all consequently published events.
Late last year we completed the migration of our comments system from our PHP monolith to a new Go microservice. Each comment is a small piece of text attached to a story chapter or to a paragraph of a chapter, with an author. You can reply to comments, creating discussion threads, or comment on the story itself. Sounds like a great little standalone microservice, doesn't it?
When you read comments, you don't just want to know what they say — you want to know who wrote them. In order to render comments properly, we need the commenter's username and avatar, the timestamp when the comment was written, and the message. Here's what comments look like today:
Oops! This image does not follow our content guidelines. To continue publishing, please remove it or upload a different image.
Where orchestration hurts
Some services are the authoritative source of their data, some have eventually consistent private datastores that mirror a subset of content from a collaborator, and some are completely stateless functions with an HTTP API. When the result of a method call is necessary to handle a client request, service orchestration is working well for us.
For example, when you post a comment (please do!) your browser sends an HTTP request to our API gateway service which makes an HTTP request to our authentication service to make sure you're an active user and logged in, and then makes a request to the comment service to post the comment. The gateway service includes your user ID and the comment when posting to the comment service, so one thing we could do is pass on some info about you, including your username and avatar, and the comment service could save those along with your comment.
Unfortunately, if we did that and then you changed your avatar, comments you'd left before would be showing the avatar you had when you posted them. Maybe you change your username too — then the old comments look like they're from someone else! Clearly not what we want.
Another possibility would be that we only keep your user ID with your comment and whenever anyone wants to read your message, the comment service could make a request to the user service to retrieve your username and avatar and use those to fill in the response. This would work, but at almost 2,000 pages of comments served per second, often showing 10 comments per page, we'd be hitting the user service up to 20,000 times per second just to look up usernames and avatars that each change very rarely.
Instead, in addition to the authoritative comment datastore which contains your user ID and message, the comment service also has an eventually consistent store for user info.
