ShoalStack is now

platform201
Background Job Processing

Asynchronous Message Queues
for Backend Services

A message queue keeps slow work from delaying an API or another service, but it is one more cloud resource to build. platform201 builds it for you: add the queue to the visual graph, and then deploy.

Add a Message Queue
Without the Cloud Setup

A producer service sends messages, and a consumer service reads them. Place a queue between them on the graph, and platform201 creates it when you deploy, so you do not build it by hand.

Add the Queue as a Node

Add a queue node, and set how long it keeps messages. The node stays on the graph next to your services, so you can see what connects to it.

my-appAPIQueueWorkerQueue settingsSelect a nodeTopicordersRetention1 day7 daysSubscriptionorders-worker

Connect Producers and Consumers

Connect the queue to each service. The connection passes the queue values to the variable names your code reads, so you do not copy them yourself.

ConnectionQueueAPIOutputQueue to APIQueue valueVariable nameTopic nameQUEUE_TOPICProject IDPROJECT_IDThe API code reads these variable names.

Reach Databases Through a Consumer

A queue has no direct connection to a database. Connect the database to the consumer, and the consumer saves the data your application keeps.

my-app3 nodes, 1 connection3 nodes, 2 connectionsQueueDatabaseWorker

Deploy All the Parts Together

Deploy the queue, your producer, your consumer, and your database as one environment. Then follow each deployment in its status and its log.

my-appAPIQueueWorkerDatabaseDeployDeployment historyVersion 1SucceededVersion 2SucceededVersion 3PendingSucceeded

Give Slow or Uneven Work a Place to Wait

A queue is most useful when one service should not wait for another service. The producer sends a message and continues, and the consumer reads the message when it is ready.

Slow Work Outside the Request

An API sends a message and responds at once. A container or a function does the slow job as the worker.

Spikes of Incoming Requests

A spike of requests waits in the queue. A worker reads messages at its own pace and saves the results.

Work Across Languages

A Node.js API and a Python or Go worker share one queue. The services agree on the message format.

Example Setup:
A Queue for Image Jobs

If your API accepts image jobs, a worker can save each result to Cloud SQL. A queue connects the two on the graph, and the API keeps responding while the worker is paused.

1

Add the Nodes to the Graph

A new environment has a gateway connected to a container. Use that container as the API. Then add a worker, a queue, and a Cloud SQL database.

2

Connect the API to the Queue

Connect the queue to the API. Map the queue values to the variable names your API code reads. The API code sends a message for each image job.

3

Give the Worker Its Messages

Add a subscription to the queue. The worker reads its messages there. Connect the queue and the database to the worker, and map their values to variables.

4

Deploy and Pause the Worker

Deploy the environment. Pause or redeploy the worker. Keep sending image jobs. The API keeps responding, and the worker saves the results when it returns.

Send Your Next Slow Job to Its Own Queue

Add a queue node with one subscription, and connect a producer to it. Then connect a consumer to read the messages. Connect a database to the consumer when you need one.

FAQ

A background worker is a service that does slow jobs outside a web request. An API accepts the request, responds at once, and leaves the job for the worker. Common jobs include resizing an image or creating a report, because a user should not wait for them. In platform201, a worker is a container or a function that you connect to a queue on the graph.