Each range maintains an array of incoming message queues, referred to here as inboxes. Additionally, each range maintains and processes an array of outgoing message queues, referred to here as outboxes. Both inboxes and outboxes are assigned to keys; messages can be sent or received on behalf of any key. Inboxes and outboxes can contain any number of pending messages.
Messages can be deliverable, or executable.
Deliverable messages are defined by Value objects - simple byte arrays - that are delivered to a key’s inbox, awaiting collection by a client invoking the ReapQueue operation. These are typically used by client applications wishing to be notified of changes to an entry for further processing, such as expensive offline operations like sending emails, SMSs, etc.
Executable messages are outgoing-only, and are instances of PutRequest,IncrementRequest, DeleteRequest, DeleteRangeRequest orAccountingRequest. Rather than being delivered to a key’s inbox, are executed when encountered. These are primarily useful when updates that are nominally part of a transaction can tolerate asynchronous execution (e.g. eventual consistency), and are otherwise too busy or numerous to make including them in the original [distributed] transaction efficient. Examples may include updates to the accounting for successive key prefixes (potentially busy) or updates to a full-text index (potentially numerous).
These two types of messages are enqueued in different outboxes too - see key formats below.
At commit time, the range processing the transaction places messages into a shared outbox located at the start of the range metadata. This is effectively free as it’s part of the same consensus write for the range as the COMMIT record. Outgoing messages are processed asynchronously by the range. To make processing easy, all outboxes are co-located at the start of the range. To make lookup easy, all inboxes are located immediately after the recipient key. The leader replica of a range is responsible for processing message queues.
A dispatcher polls a given range’s deliverable message outbox periodically (configurable), and delivers those messages to the target key’s inbox. The dispatcher is also woken up whenever a new message is added to the outbox. A separate executor also polls the range’s executable message outbox periodically as well (again, configurable), and executes those commands. The executor, too, is woken up whenever a new message is added to the outbox.
Formats follow in the table below. Notice that inbox messages for a given key sort by the
Messages are processed and then deleted as part of a single distributed transaction. The message will be executed or delivered exactly once, regardless of failures at either sender or receiver.
Delivered messages may be read by clients via the ReapQueue operation. This operation may only be used as part of a transaction. Clients should commit only after having processed the message. If the transaction is committed, scanned messages are automatically deleted. The operation name was chosen to reflect its mutating side effect. Deletion of read messages is mandatory because senders deliver messages asynchronously and a delay could cause insertion of messages at arbitrary points in the inbox queue. If clients require persistence, they should re-save read messages manually; the ReapQueue operation can be incorporated into normal transactional updates.