---
title: "Conditional Correlations"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Conditional Correlations}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

```{r, include = FALSE}
options(tibble.width = Inf)
```


# Introduction

**Conditional correlations** identify conditions under which two numeric
variables are strongly related. In `nuggets`, the basic scheme is:

> *xvar* ~ *yvar* | *condition*

This reads as: the variables `xvar` and `yvar` are highly correlated in the
sub-data satisfying the given condition.

For example:

> `study_time ~ test_score | hard_exam`

This means that for difficult exams, the amount of study time is strongly
related to the obtained test score.

The `dig_correlations()` function searches over generated conditions and
computes correlations for all selected combinations of numeric variables. It
supports the standard correlation methods implemented by
[`stats::cor.test()`](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/cor.test.html),
namely Pearson, Kendall, and Spearman correlations.

Before going further, load the packages used in this vignette:

```{r, message = FALSE}
library(nuggets)
library(dplyr)     # for data manipulation
```

For a broader overview of the package workflow, see `vignette("nuggets")`.


# A Small Working Dataset

`dig_correlations()` expects **logical condition columns** and **numeric
variables** to correlate. The built-in `iris` dataset is convenient for this,
because it already contains several numeric measurements and one factor column
that can be transformed into condition predicates.

In the next example, we:

- convert `Species` into dummy logical predicates with `partition()`,
- add two logical helper conditions,
- add two derived numeric variables that will be useful in examples.

```{r}
iris_corr <- iris |>
    mutate(long_sepal = Sepal.Length >= median(Sepal.Length),
           wide_petal = Petal.Width >= median(Petal.Width),
           sepal_ratio = Sepal.Length / Sepal.Width,
           petal_ratio = Petal.Length / Petal.Width) |>
    partition(Species)

head(iris_corr, n = 3)
```

The dummy columns created from `Species` and the logical helper columns can be
used to generate conditions, while the original and derived numeric columns can
be used in the correlation tests.

For more information on preparing data for pattern extraction, see
`vignette("data-preparation")`.


# Basic Conditional Correlation Search

The simplest search chooses condition predicates and numeric variables, then
limits the condition length and support:

```{r}
corr_basic <- dig_correlations(iris_corr,
                               condition = where(is.logical),
                               xvars = c(Sepal.Length, Sepal.Width, sepal_ratio),
                               yvars = c(Petal.Length, Petal.Width, petal_ratio),
                               min_length = 0,
                               max_length = 2,
                               min_support = 0.2)

corr_basic |>
    arrange(desc(abs(estimate))) |>
    head(n = 6)
```

The result is a tibble where each row represents one discovered pattern. The
main columns are:

- `condition` - the generated condition,
- `support` - relative frequency of the condition,
- `xvar`, `yvar` - the correlated variable pair,
- `estimate` - the correlation coefficient,
- `p_value` - significance of the test,
- `n` - number of rows in the corresponding sub-data,
- `alternative`, `method` - additional information about the test.


# Controlling the Search Space

The `condition`, `xvars`, and `yvars` arguments accept
[tidyselect expressions](https://tidyselect.r-lib.org/reference/language.html).
This makes it easy to restrict the search to specific groups of columns.

As with other `dig_*()` functions, the search can be controlled with
`min_length`, `max_length`, `min_support`, `max_support`, and `max_results`.

For example, the following search uses only species predicates as conditions
and correlates all sepal variables against all petal variables:

```{r}
corr_species <- dig_correlations(iris_corr,
                                 condition = starts_with("Species"),
                                 xvars = starts_with("Sepal"),
                                 yvars = starts_with("Petal"),
                                 min_length = 1,
                                 max_length = 1,
                                 min_support = 0.3)

head(corr_species, n = 6)
```




# Choosing the Correlation Method

The `method` argument selects which correlation coefficient is used:

- `"pearson"` for linear relationships,
- `"spearman"` for monotone relationships based on ranks,
- `"kendall"` for rank-based association with Kendall's tau.

Under the hood, every combination uses
[`stats::cor.test()`](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/cor.test.html)
with the corresponding `method` argument. The `estimate`, `statistic`,
`p_value`, and confidence-interval columns in the result map directly to the
values returned by `cor.test()`.

Here is the same basic search with Spearman correlation. As we are O.K. with
non-exact p-values, we can set `exact = FALSE` (otherwise a warning is issued 
for small sub-data sizes):

```{r}
corr_spearman <- dig_correlations(iris_corr,
                                  condition = where(is.logical),
                                  xvars = c(Sepal.Length, Sepal.Width),
                                  yvars = c(Petal.Length, Petal.Width),
                                  method = "spearman",
                                  exact = FALSE,
                                  min_length = 1,
                                  max_length = 1,
                                  min_support = 0.2)

head(corr_spearman, n = 6)
```

If you are interested specifically in positive or negative relationships, use
the `alternative` argument with values `"greater"` or `"less"`.


# Correlations on the Whole Dataset

If `condition = NULL`, `dig_correlations()` computes correlations on the whole
dataset only. This is useful when you want the same output structure without
searching over conditions.

```{r}
corr_whole <- dig_correlations(
    iris_corr,
    condition = NULL,
    xvars = starts_with("Sepal"),
    yvars = starts_with("Petal")
)

corr_whole
```

This is effectively a convenient way to compute a structured set of pairwise
correlation tests between selected columns.


# Notes on Interpretation

When reading discovered correlations, it is useful to keep the following points
in mind:

- a large absolute `estimate` indicates a stronger relationship,
- `p_value` reflects statistical evidence for the chosen alternative, but 
  should be interpreted with caution due to multiple comparisons (see below),
- `support` and `n` describe how much data contributed to the pattern,
- longer conditions describe more specific contexts, but usually with smaller
  support.

Conditional correlations are therefore most useful when the relationship
between two variables changes across subgroups and would be hidden in a
single global correlation.


## Multiple Comparisons

When `dig_correlations()` is applied to a dataset, it simultaneously tests
correlations for a potentially large number of condition–variable-pair
combinations. Each test produces a `p_value`, but interpreting any individual
`p_value` as if it were the result of a single pre-planned test is misleading:
if hundreds of tests are performed at level 0.05, several false discoveries are
expected by chance alone.

The patterns returned by `dig_correlations()` are therefore best understood as
**generated hypotheses** - promising associations that deserve further
scrutiny - rather than as confirmed findings. This is known as the problem of
*simultaneous statistical inference* or *multiple comparisons*.

A standard remedy is to **adjust the p-values** to control either the
family-wise error rate (FWER) or the false discovery rate (FDR):

- **FWER-controlling methods** (e.g. Bonferroni, Holm) ensure that the
  probability of making *any* false discovery across all tests stays below a
  chosen threshold. They are conservative when the number of tests is large.
- **FDR-controlling methods** (e.g. Benjamini–Hochberg, abbreviated BH) allow
  a small *proportion* of discoveries to be false positives. This is less
  stringent than FWER control and retains more patterns in exploratory analyses.

R's built-in `p.adjust()` function supports both families. The example below
applies Holm correction (FWER) and Benjamini–Hochberg correction (FDR) to the
result of a search:

```{r}
corr_basic$p_holm <- p.adjust(corr_basic$p_value, method = "holm")
corr_basic$p_bh   <- p.adjust(corr_basic$p_value, method = "BH")

corr_basic[, c("condition", "xvar", "yvar", "p_value", "p_holm", "p_bh")]
```

After adjustment, you can filter by the corrected p-values:

```{r}
corr_basic[corr_basic$p_bh < 0.05, ]
```

In a large exploratory search you may prefer the FDR approach (BH) because it
keeps more patterns visible while still limiting the expected fraction of false
discoveries. Use FWER control (Holm) when you need stronger guarantees.


# Related Tools

`dig_correlations()` is a specialized wrapper around `dig_grid()`. If you need
custom statistics on pairs of variables under generated conditions, see
`vignette("custom-patterns")` for the more general workflow.

For interactive inspection of discovered patterns, you can also use:

```{r, eval = FALSE}
explore(corr_basic, iris_corr)
```


# Summary

This vignette showed how to search for conditional correlations with
`nuggets`:

1. prepare logical condition predicates and numeric variables in one dataset,
2. use `dig_correlations()` to search over generated conditions,
3. select condition and variable columns with tidyselect expressions,
4. control the search with support, condition length, and result limits,
5. choose a correlation method appropriate for the data and interpretation.

For related material, see:

- `vignette("data-preparation")` for creating condition predicates,
- `vignette("contrast-patterns")` for condition-dependent statistical
  differences,
- `vignette("custom-patterns")` for custom grid-based analyses,
- `vignette("nuggets")` for the package overview.
