# Large-data linking with myIO

## When to use this

myIO’s default SVG path handles up to about 20,000 rendered marks comfortably. Beyond that, the big-data tier uses a coordinator, an engine adapter, and optional Canvas or WebGL rendering so charts can respond to brush and zoom interactions over millions of rows. Opt in per chart by calling `setBigData(widget, source)`.

## Installation

Big-data features require two optional components.

First, install the Suggested R packages used for Arrow encoding, DuckDB queries, downloads, checksums, and status output:

    install.packages(c("arrow", "duckdb", "DBI", "base64enc", "cli", "curl", "openssl"))

Second, install the DuckDB-WASM runtime when you plan to use the browser engine. The runtime is downloaded on demand by `myIO::install_duckdb_wasm()` and is not bundled in the CRAN tarball. It is cached under `tools::R_user_dir("myIO", "cache")`. For airgapped machines, place `duckdb-mvp.wasm` and `duckdb-browser-mvp.worker.js` in a local directory and call `install_duckdb_wasm(from = "/path/to/dir")`.

    install.packages(c("arrow", "duckdb", "DBI", "base64enc", "cli", "curl", "openssl"))
    myIO::install_duckdb_wasm()
    myIO::duckdb_wasm_status()

## Attaching big data

`setBigData()` accepts several source types. A `data.frame` is encoded as inline Arrow IPC. This is convenient for portable HTML, but it warns above 50 MB and hard-errors above 200 MB.

    \dontrun{
    library(myIO)

    big <- data.frame(
      id = seq_len(1e6),
      x = rnorm(1e6),
      y = rnorm(1e6)
    )

    myIO(engine = "wasm") |>
      addIoLayer(type = "point", label = "points",
                 mapping = list(x_var = "x", y_var = "y")) |>
      setBigData(big, rowkey_col = "id")
    }

An `arrow::Table` uses the same inline IPC path.

    \dontrun{
    library(arrow)
    library(myIO)

    tab <- arrow_table(big)

    myIO(engine = "wasm") |>
      addIoLayer(type = "point", label = "points",
                 mapping = list(x_var = "x", y_var = "y")) |>
      setBigData(tab, rowkey_col = "id")
    }

For larger static assets, pass a local path or URL ending in `.parquet`, `.csv`, `.arrow`, or `.feather`.

    \dontrun{
    myIO(engine = "wasm") |>
      addIoLayer(type = "histogram", label = "x",
                 mapping = list(x_var = "x")) |>
      setBigData("data/observations.parquet", rowkey_col = "id")

    myIO(engine = "wasm") |>
      addIoLayer(type = "point", label = "remote",
                 mapping = list(x_var = "x", y_var = "y")) |>
      setBigData("https://example.org/observations.csv", rowkey_col = "id")
    }

A `DBI` connection is server-engine-only. Provide `table = "..."` so myIO can read the schema.

    \dontrun{
    library(DBI)
    library(duckdb)
    library(myIO)

    con <- dbConnect(duckdb())
    dbWriteTable(con, "observations", big)

    myIO(engine = "server") |>
      addIoLayer(type = "point", label = "points",
                 mapping = list(x_var = "x", y_var = "y")) |>
      setBigData(con, table = "observations", rowkey_col = "id")
    }

## The engine argument

Use `engine = "auto"`, `"server"`, `"wasm"`, or `"svg"` on `myIO()`. `"auto"` is the default: a Shiny session resolves to `"server"`; otherwise it resolves to `"wasm"`. `"server"` runs queries in R with `duckdb` and streams Arrow batches to the browser, which is a good fit when a Shiny server already exists. `"wasm"` runs DuckDB in the browser from the cached WASM runtime, which fits static Quarto or R Markdown HTML. `"svg"` forces the legacy SVG path without the coordinator and is mainly useful for testing.

## Crosstalk threshold

By default, myIO broadcasts row keys to `crosstalk::SharedData` only when the selected row count is at or below 100,000. Below the threshold, sibling htmlwidgets such as plotly, leaflet, and reactable can react to myIO brushes. Above it, upward broadcast is suppressed; myIO-to-myIO linking still works through predicates, a one-shot console info is emitted, and the footer badge reads `linked: predicate-only`.

Tune the limit with:

    options(myIO.crosstalk_threshold = 50000L)

The threshold is per selection, not per chart. A narrow brush on a million-row source can still broadcast if it matches few rows.

## File-protocol limitation

When a Quarto or R Markdown HTML file is opened directly from the file manager with the `file://` protocol, Chromium blocks dynamic module imports. myIO detects this and falls back to the SVG path with a one-shot console info. To use the WASM engine on a local static HTML, serve it with `servr::httd()` or `quarto preview`.

## Performance expectations

Input rows | Engine | Renderer | Interaction
--- | --- | --- | ---
<= 20k | `svg` (default) | D3 SVG | Full brush/zoom, publication-quality
20k-100k | `svg` + aggregation | D3 SVG | Smooth; tooltips on pre-aggregated data
100k-1M | `wasm` or `server` | Canvas or WebGL | Sub-200ms brush re-aggregation (WASM), sub-500ms (server Shiny)
1M-10M | `wasm` or `server` | WebGL | Target: 60fps pan/zoom; brush re-agg < 300ms

## Limits and gotchas

Inline IPC above 200 MB hard-errors; use file paths or a `DBI` connection. The Crosstalk threshold depends on which rows match the current selection. The WASM binary is about 22 MB, downloads once per user per version, and is cached indefinitely; clear it with `clear_duckdb_wasm_cache()`. On Posit Connect or shinyapps.io, use the `"server"` engine; `install_duckdb_wasm()` is not needed on the server.

## Minimal complete example

    \dontrun{
    library(myIO)

    install.packages(c("arrow", "duckdb", "DBI", "base64enc", "cli", "curl", "openssl"))
    myIO::install_duckdb_wasm()

    set.seed(1)
    events <- data.frame(
      id = seq_len(250000),
      time = as.POSIXct("2026-01-01", tz = "UTC") + seq_len(250000),
      x = rnorm(250000),
      y = rnorm(250000),
      group = sample(LETTERS[1:4], 250000, replace = TRUE)
    )

    myIO(engine = "wasm") |>
      addIoLayer(type = "point", label = "events",
                 mapping = list(x_var = "x", y_var = "y", color = "group")) |>
      setBrush(direction = "xy") |>
      setBigData(events, rowkey_col = "id")
    }