# Preparing your data for FINN

This vignette is a standalone reference for preparing forest-inventory data into FINN. It documents the inputs of FINN: columns names and units, and the helper functions for data preparation `makeObsData()` → `resolveSiteIDs()` → `makeInitCohorts()`.

    library(FINN)
    library(data.table)

## What FINN needs

FINN is calibrated against clearly defined inputs:

- `obs_dt`: demographic responses per site × year × species: `siteID, year, ba, dbh, trees, growth, mort, n_at_risk, n_died, reg, species, species_name`. Year starts at 1; the initial state (year 0) is not included. NAs are allowed in any response. Units: `ba` m², `dbh` cm, `trees` count, `growth` relative growth rate, `reg` trees ha⁻¹. Mortality comes back as a pair of counts, `n_at_risk` (trees alive at the start of the interval) and `n_died` (how many of them were dead at the end), with the rate `mort = n_died / n_at_risk` derived for convenience. That pair is the `cbind(died, survived)` response of a binomial GLM, and it is what the default `mortality = "binomial"` loss consumes: the counts carry the sample size, so a 60-tree observation outweighs a 1-tree one.
- `env_dt`: `siteID, year, <env vars…>` in natural units; must cover every site × year in `obs_dt`. No manual scaling is needed: FINN z-standardizes the predictors internally when fitting (`env_autoscale = TRUE`, the default) and reuses the same constants at prediction time.
- `init_trees`: the initial state used to build the starting cohorts: `siteID, patchID, year, species, dbh, trees, living`.
- `dist_dt` (optional): `siteID, year, intensity` (fraction of patches disturbed that year).

You assemble these from two raw ingredients: a tree list (one row per tree × inventory) and a site environment table. The rest of this vignette builds the FINN tables from those two.

## Start from a raw tree list

    # An example from raw data: Oregon FIA sites, 4 patches each, 3 inventories.
    # example_tree_dt.csv and example_env_dt.csv (the climate, loaded later) are both
    # built by dev/make_extdata.R from the data-raw/ source; see data-raw/README.md.
    tree_dt <- fread(system.file("extdata", "example_tree_dt.csv", package = "FINN"))
    str(tree_dt)
    #> Classes 'data.table' and 'data.frame':   500 obs. of  13 variables:
    #>  $ siteName     : chr  "11_66410" "11_66410" "11_66410" "11_66410" ...
    #>  $ patchName    : chr  "11_66410_1" "11_66410_1" "11_66410_1" "11_66410_2" ...
    #>  $ treeName     : chr  "11_66410_1_100" "11_66410_1_100" "11_66410_1_100" "11_66410_2_103" ...
    #>  $ year         : int  2001 2011 2021 2001 2011 2021 2001 2011 2021 2001 ...
    #>  $ species_name : chr  "Pseudotsuga menziesii" "Pseudotsuga menziesii" "Pseudotsuga menziesii" "Pseudotsuga menziesii" ...
    #>  $ dbh          : num  47.2 57.4 65 26.9 30.7 ...
    #>  $ status       : chr  "new" "alive" "alive" "new" ...
    #>  $ status_before: chr  "" "new" "alive" "" ...
    #>  $ mort_cause   : chr  "" "" "" "" ...
    #>  $ reg          : logi  TRUE NA NA TRUE NA NA ...
    #>  $ mort         : logi  NA NA NA NA NA NA ...
    #>  $ living       : logi  TRUE TRUE TRUE TRUE TRUE TRUE ...
    #>  $ complete     : logi  TRUE TRUE TRUE TRUE TRUE TRUE ...
    #>  - attr(*, ".internal.selfref")=<externalptr>

A raw tree record must include: `siteName, patchName, treeName, year` (calendar year of the inventory), `species_name`, `dbh` (cm), and a `status` of `new` / `alive` / `dead`. That is enough: `makeObsData()` reconstructs each tree’s previous state itself, by sorting on `treeName` and `year`. Our example also contains columns for `status_before` and a `complete` flag marking trees from fully re-measured plots.

## Acquiring raw data (example: US FIA)

Turning a public inventory into the raw tree list above needs external downloads, so it is shown but not run. For the US FIA program:

    library(rFIA)
    fia <- readFIA(dir = "data/FIA", common = TRUE, states = "OR")
    # build tree_dt: dbh = DIA * 2.54 (inch -> cm); status from STATUSCD / PREV_STATUS_CD;
    # mort_cause from AGENTCD. See playground/calibration/FIA/FIA_dataprep_vignette.R

## Define status flags

FINN’s builders key off two logical flags derived from `status`:

    # reg    = newly recruited this inventory
    # living = currently alive
    tree_dt[, reg    := status == "new"]
    tree_dt[, living := status %chin% c("alive", "new")]
    tree_dt[, .(n = .N), by = .(status, reg, living)]
    #>    status    reg living     n
    #>    <char> <lgcl> <lgcl> <int>
    #> 1:    new   TRUE   TRUE   193
    #> 2:  alive  FALSE   TRUE   239
    #> 3:   dead  FALSE  FALSE    68

There is deliberately no tree-level `mort` flag here. `makeObsData()` derives mortality itself, from `status` and `living`, as a closed cohort: for each interval it takes the trees that were alive at the start, and counts how many of them were dead at the end. Both columns are pinned to the tree’s state at the start of the interval, which matters because FIA re-identifies a tree’s species between visits; a hand-rolled rate can otherwise book a death against the species a tree ended as while the denominator counted the species it started as, and report more deaths than trees.

## Build `obs_dt` with `makeObsData()`

`makeObsData()` aggregates the tree list to per-patch demographic responses and lumps rare species into a single `"other"` class. Each argument controls a data-quality filter:

    obs_list <- makeObsData(
      tree_dt           = tree_dt[complete == TRUE],  # only fully re-measured plots
      plotsize          = 0.06,            # ha per patch (scales recruitment to per-ha)
      aggregate_by_site = FALSE,           # keep patch-level resolution
      fix_period_length = 10,              # drop sites whose remeasure gap != 10 yr
      dbh_growth_thresh = c(-10, 50),      # drop sites with implausible dbh change
      NspeciesQuantile  = 0.98,            # keep species covering 98% of stems; rest -> "other"
      minNyears         = 2                # need >= 2 inventories per patch
    )
    obs_dt <- obs_list$obs_dt
    head(obs_dt)
    #>    siteName  patchName  year   species_name    ba   dbh trees growth growth_n
    #>      <char>     <char> <int>         <char> <num> <num> <int>  <num>    <int>
    #> 1: 11_66410 11_66410_1  2001 Abies concolor     0    NA     0     NA        0
    #> 2: 11_66410 11_66410_1  2011 Abies concolor     0    NA     0     NA        0
    #> 3: 11_66410 11_66410_1  2021 Abies concolor     0    NA     0     NA        0
    #> 4: 11_66410 11_66410_1  2001  Abies grandis     0    NA     0     NA        0
    #> 5: 11_66410 11_66410_1  2011  Abies grandis     0    NA     0     NA        0
    #> 6: 11_66410 11_66410_1  2021  Abies grandis     0    NA     0     NA        0
    #>     mort n_at_risk n_died   reg
    #>    <num>     <int>  <int> <num>
    #> 1:    NA         0      0     0
    #> 2:    NA         0      0     0
    #> 3:    NA         0      0     0
    #> 4:    NA         0      0     0
    #> 5:    NA         0      0     0
    #> 6:    NA         0      0     0

Use `Nspecies` (a hard cap) or `NspeciesQuantile` (smallest set covering a stem fraction) to control how many species you model; everything else becomes `"other"`, which FINN still tracks as a competitor.

## Environmental data

Acquisition from rasters (e.g. WorldClim) needs external files, so it is shown but not run. The result is a table of environmental predictors in natural units, one row per site × year:

    library(terra)
    pts  <- vect(site_dt, geom = c("x", "y"), crs = "EPSG:4326")
    bio1 <- rast("wc2.1_30s_bio_1.tif")
    env_dt[, temp := terra::extract(bio1, project(pts, crs(bio1)))[, 2]]

    env_dt <- fread(system.file("extdata", "example_env_dt.csv", package = "FINN"))
    head(env_dt)
    #>    siteName  year     temp tempmax tempmin  prec precseas precwarmq
    #>      <char> <int>    <num>   <num>   <num> <int>    <num>     <int>
    #> 1: 11_66410  2001 10.69167    22.8     2.7  1880 70.15234       103
    #> 2: 11_66410  2011 10.69167    22.8     2.7  1880 70.15234       103
    #> 3: 11_66410  2021 10.69167    22.8     2.7  1880 70.15234       103
    #> 4: 17_65658  2002  6.48750    26.1    -6.6   417 55.10012        56
    #> 5: 17_65658  2012  6.48750    26.1    -6.6   417 55.10012        56
    #> 6: 17_65658  2022  6.48750    26.1    -6.6   417 55.10012        56

There is no manual standardization step. FINN z-scales the predictors internally when you fit (`env_autoscale = TRUE`, the default), stores the per-variable mean/sd on the model, and re-applies them at prediction time. So you supply (and keep) `env_dt` in natural units throughout. `env_dt` must cover every `siteName × year` present in `obs_dt`.

## Resolve IDs across tables

Raw tables use names (`siteName`, `patchName`, `species_name`); FINN works with integer indices aligned across all tables. `resolveSiteIDs()` does that alignment and can build the initial cohorts in one pass. Pass the processed tree list `obs_list$tree_dt` (not the raw one): `makeObsData()` returns it with the same species lumping as `obs_dt`, so the two stay consistent.

    inputs <- resolveSiteIDs(
      tree_dt = obs_list$tree_dt,   # processed tree list (species lumped like obs_dt)
      obs_dt  = obs_dt,
      env_dt  = env_dt,
      createInitCohorts = TRUE
    )
    # obs_dt / env_dt now carry integer siteID/patchID/species, aligned across tables
    head(inputs$obs_dt)
    #>    siteID  year    ba   dbh trees growth growth_n  mort n_at_risk n_died   reg
    #>     <int> <num> <num> <num> <num>  <num>    <int> <num>     <int>  <int> <num>
    #> 1:      1     0     0   NaN     0    NaN        0    NA         0      0     0
    #> 2:      1     1     0   NaN     0    NaN        0    NA         0      0     0
    #> 3:      1     2     0   NaN     0    NaN        0    NA         0      0     0
    #> 4:      2     0     0   NaN     0    NaN        0    NA         0      0     0
    #> 5:      2     1     0   NaN     0    NaN        0    NA         0      0     0
    #> 6:      2     2     0   NaN     0    NaN        0    NA         0      0     0
    #>    species   species_name
    #>      <int>         <char>
    #> 1:       1 Abies concolor
    #> 2:       1 Abies concolor
    #> 3:       1 Abies concolor
    #> 4:       1 Abies concolor
    #> 5:       1 Abies concolor
    #> 6:       1 Abies concolor
    inputs$species_dt
    #>     species           species_name
    #>       <int>                 <char>
    #>  1:       9  Pseudotsuga menziesii
    #>  2:      11                  other
    #>  3:       8        Pinus ponderosa
    #>  4:       5     Larix occidentalis
    #>  5:       2          Abies grandis
    #>  6:       4 Juniperus occidentalis
    #>  7:       7        Pinus monticola
    #>  8:       6         Pinus contorta
    #>  9:       1         Abies concolor
    #> 10:      10      Tsuga mertensiana
    #> 11:       3       Abies shastensis

## Initial cohorts

With `createInitCohorts = TRUE`, `resolveSiteIDs()` returns a ready `CohortMat` describing the starting state; its `dims` are `c(sites, patches, cohorts)`:

    init_cohorts <- inputs$initCohorts
    init_cohorts
    #> An `nn_module` containing 0 parameters.
    #>
    #> -- Buffers ---------------------------------------------------------------------
    #> * dbh: Float [1:8, 1:4, 1:14]
    #> * trees: Float [1:8, 1:4, 1:14]
    #> * species: Long [1:8, 1:4, 1:14]

To build cohorts yourself from a year-0 tree slice, call `makeInitCohorts()` directly; it expects `siteID, patchID, species, dbh, treeName` (+ optional `trees`):

    init_trees <- inputs$tree_dt[living == TRUE & year == 1]
    makeInitCohorts(init_trees, Nspecies = uniqueN(inputs$obs_dt$species))

## Checklist before fitting

- Period length is consistent across inventories (`fix_period_length`); FINN assumes a fixed remeasure interval.
- The initial state (the first inventory) is used for `init_trees`, and the responses in `obs_dt` start after it.
- `env_dt` covers every `siteID × year` in `obs_dt`; missing rows break the fit.
- Units: `dbh` in cm, `ba` in m², `reg` in trees ha⁻¹; env predictors in natural units (FINN standardizes them internally).
- Species IDs are contiguous integers `1..N`, aligned across tables; always go through `resolveSiteIDs()` rather than hand-coding them.

You now have `obs_dt`, `env_dt`, and `init_cohorts`, the inputs the Fitting FINN to forest inventory data vignette consumes.

    sessionInfo()
    #> R version 4.5.0 (2025-04-11)
    #> Platform: aarch64-apple-darwin20
    #> Running under: macOS 26.5.1
    #>
    #> Matrix products: default
    #> BLAS:   /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/lib/libRblas.0.dylib
    #> LAPACK: /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.1
    #>
    #> locale:
    #> [1] C
    #>
    #> time zone: Europe/Berlin
    #> tzcode source: internal
    #>
    #> attached base packages:
    #> [1] stats     graphics  grDevices utils     datasets  methods   base
    #>
    #> other attached packages:
    #> [1] data.table_1.17.8 FINN_0.1.0
    #>
    #> loaded via a namespace (and not attached):
    #>  [1] vctrs_0.6.5        cli_3.6.6          knitr_1.50         rlang_1.2.0
    #>  [5] xfun_0.57          processx_3.8.6     generics_0.1.4     torch_0.15.1
    #>  [9] coro_1.1.0         glue_1.8.0         bit_4.6.0          ps_1.9.1
    #> [13] scales_1.4.0       grid_4.5.0         abind_1.4-8        evaluate_1.0.5
    #> [17] tibble_3.3.0       lifecycle_1.0.5    compiler_4.5.0     dplyr_1.1.4
    #> [21] RColorBrewer_1.1-3 Rcpp_1.1.0         pkgconfig_2.0.3    farver_2.1.2
    #> [25] R6_2.6.1           tidyselect_1.2.1   pillar_1.11.0      callr_3.7.6
    #> [29] magrittr_2.0.3     tools_4.5.0        bit64_4.6.0-1      gtable_0.3.6
    #> [33] ggplot2_3.5.2