Why it mattered
One of the vendor’s clients needed an additional service: take in a large stream of messages, store them, send mass messages on demand and enrich them with data from the client’s directory. It had to work fast, and the client needed to see it working soon.
The proof of concept arrived in a very short time and met the performance requirements with an order of magnitude to spare. The cooperation with the client continued.
How it works
- Requirements and architecture worked out with the client, together with colleagues.
- A stateless, multi-threaded application. Python 3 services whose workers keep nothing between two steps, so the load spreads over parallel threads.
- PostgreSQL at the centre. What the service knows lives in one database: the messages that came in and the ones to send. The workers meet there and nowhere else.
- Queries in batches of 10,000. For speed, the database is read and written ten thousand messages at a time instead of row by row.
- Integration with the client’s REST API and its LDAP directory.