Workload Shapes: When (and How) to Use staRburst

Lever 1 — Task length: is each task worth shipping?

Every task pays a fixed tax: it’s serialized, uploaded to S3, pulled by a worker, and its result uploaded back. Plus the pool pays a one-time ~75–90 s cold start. If the task’s actual compute is smaller than that tax, the cloud only makes things slower.

Two measured runs, same 20–30 workers, opposite verdicts — the only thing that changed is per-task work:

# Heavy tasks (minutes each) — the cloud wins
results <- starburst_map(scenarios, run_expensive_model, workers = 30)

# Trivially cheap tasks — just do it locally
results <- lapply(inputs, quick_function)

Rule: if each task is sub-second, stay local. Seconds-to-minutes per task, with many of them, is the sweet spot.

Lever 2 — Task count: batch, or fall off a cliff

This is the one that surprises people. staRburst creates one task per element of .x, and each task is a separate client-side S3 submit and collect — in a serial loop. It does not batch for you. So a “big parameter walk” submitted naively as one task per point doesn’t just get slow — it hits a wall on the client, before any compute happens.

Same 10,000-point sweep, two ways — both measured on live AWS, identical work:

Read that again: the same 10,000 evaluations took 2.9 minutes for $0.13 batched, versus 77 minutes for $3.46 naive — a ~65× slowdown and ~27× cost blowup, purely from task count. The naive run spent over an hour with the client submitting 10,000 individual tasks to S3 one at a time (~1.5/sec) while 50 workers sat mostly idle — you pay for the workers the whole time. Nothing about the computation changed; only how it was packaged.

This is the trap behind “just fan out my 10M-point sweep”: at 1M or 10M tasks the naive path doesn’t take minutes longer, it takes hours-to-days and never gets cheaper. Batching isn’t an optimization here — it’s the difference between working and not.

# ❌ DON'T: 10,000 tasks — ~10,000 serial S3 submits from your laptop
results <- starburst_map(1:10000, evaluate_point, workers = 50)

# ✅ DO: batch into ~100 chunks — 100 tasks, same 10,000 evaluations
batches <- split(1:10000, ceiling(seq_along(1:10000) / 100))
results <- starburst_map(batches, function(ids) lapply(ids, evaluate_point),
                         workers = 50)
results <- unlist(results, recursive = FALSE)  # flatten

Pick a batch size so each task runs for a couple of minutes:

per_item_seconds <- 0.5              # measure this on a small local sample
target_task_seconds <- 120           # aim for ~2 min per task
batch_size <- ceiling(target_task_seconds / per_item_seconds)  # ~240 here

Rule: thousands of tiny tasks is an anti-pattern. Batch fine-grained work so you have ~dozens-to-hundreds of tasks, each doing real work. A 1M- or 10M-point sweep must be batched — never submitted one point at a time.

Lever 3 — Fan-out width: more workers is not always better

Once you’ve sized your tasks, how many workers should you request? More workers cut wall-clock only up to the number of tasks, and with diminishing returns:

• Workers > tasks: the extra workers sit idle — you pay for them, they do nothing. 30 tasks on 100 workers still only uses 30.
• Stragglers: the batch isn’t done until the slowest task is. Past a point, adding workers doesn’t help because you’re waiting on one long task, not queue depth.
• Quota & startup: more workers = more instances to launch (longer, more likely to hit limits) and, on Fargate, more vCPU quota consumed.
• Cost scales with workers, speed doesn’t (past #tasks). Doubling workers beyond your task count doubles cost for zero speedup.

# 200 tasks: ~50 workers is a reasonable ratio (each worker handles ~4 tasks)
results <- starburst_map(param_sets, fit_model, workers = 50)

# 200 tasks on 500 workers: 300 workers idle, you pay for all 500 — don't

Rule of thumb: workers ≈ number of tasks (capped at what your quota and budget allow), so each worker gets roughly one to a few tasks. Beyond workers == tasks there is no speedup left to buy.

Lever 4 — Cost vs. time: what are you optimizing?

staRburst reports an estimated cost per run. The key intuition: wall-clock and cost trade off differently.

• Adding workers (up to #tasks) buys wall-clock at ~constant total cost — the same compute-seconds are just spread over more machines for less time. That’s the good trade: finish sooner for roughly the same dollars.
• Adding workers beyond #tasks buys nothing and raises cost (idle instances).
• Spot instances (staRburst’s default, use_spot = TRUE) are ~70% cheaper than on-demand for interruption-tolerant batch work — almost always the right default here.
• Warm pools amortize the ~90 s startup across many runs, so repeated jobs (parameter sweeps, daily pipelines) get dramatically better cost/time than a single cold run implies.

# Cheapest for a one-off: enough workers to cover the tasks, spot on (default)
starburst_map(tasks, fn, workers = length(tasks), use_spot = TRUE)

# Repeated runs: a warm pool pays startup once — see the detached-sessions guide

Rule: optimize wall-clock by matching workers to tasks; optimize cost by using spot (default) and warm pools for anything you run more than once.

Pro tip — shard your S3 writes across prefixes

If your tasks write their own output to S3 (large results you don’t want to serialize back through the result channel), have each task write to a distinct prefix, not all to the same one. S3 scales throughput per prefix, so hundreds of tasks hammering a single prefix serialize against each other, while spreading them across prefixes lets S3 parallelize:

# ❌ Hundreds of tasks writing under ONE prefix — throughput bottleneck
key <- sprintf("s3://bucket/results/%d.parquet", i)

# ✅ Shard across prefixes so S3 parallelizes the writes
shard <- i %% 100
key <- sprintf("s3://bucket/results/shard=%02d/%d.parquet", shard, i)

A cheap, high-leverage change for write-heavy fan-outs: same data, far more aggregate write throughput, because you’re no longer funneling every worker through one hot prefix.

Putting it together: a diagnosis checklist

Before you burst a workload, ask:

1. Is each task more than a second of work? If no → stay local (Lever 1).
2. Do you have thousands of tiny tasks? If yes → batch to ~dozens–hundreds (Lever 2).
3. Are you requesting more workers than tasks? If yes → cap at #tasks (Lever 3).
4. Will you run it more than once? If yes → warm pool + spot (Lever 4).
5. Do tasks write big outputs to S3? If yes → shard across prefixes (pro tip).

Get these right and staRburst turns a multi-hour local grind into minutes for pennies. Get them wrong and you’ll pay more to go slower — the mechanics don’t care which field you’re in, only about the shape of the work.

See also: vignette("performance") for the underlying overhead model, and vignette("detached-sessions") for warm pools and long-running jobs.