# Usmile

<!-- badges: start -->

[![R-CMD-check](https://github.com/bbwieckowska/Usmile/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/bbwieckowska/Usmile/actions/workflows/R-CMD-check.yaml)
[![Lifecycle: maturing](https://img.shields.io/badge/lifecycle-maturing-blue.svg)](https://lifecycle.r-lib.org/articles/stages.html)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![GitHub release](https://img.shields.io/github/v/release/bbwieckowska/Usmile)](https://github.com/bbwieckowska/Usmile/releases)

<!-- Uncomment after publication on CRAN. -->
<!--
[![CRAN status](https://www.r-pkg.org/badges/version/Usmile)](https://CRAN.R-project.org/package=Usmile)
[![CRAN downloads](https://cranlogs.r-pkg.org/badges/grand-total/Usmile)](https://cran.r-project.org/package=Usmile)
-->

<!-- badges: end -->

## Overview

**Universal Smile Layout for Explanation (Usmile)**

`Usmile` is an R package for threshold-free, class-specific evaluation
and comparison of probabilistic binary classifiers.

The package implements the U-smile methodology, which evaluates changes
in predicted probabilities separately for:

- non-events with improved prediction;
- non-events with worsened prediction;
- events with worsened prediction;
- events with improved prediction.

The U-smile plot visualizes these four subclasses in a fixed order. The
resulting shape provides an immediate graphical assessment of whether a
new model improves or worsens prediction for the non-event and event
classes.

The package supports the following U-smile coefficients:

- `BA`: absolute average change in prediction;
- `RB`: relative change in prediction error;
- `I`: proportion of individuals with changed prediction;
- `rLR`: relative likelihood-ratio measure of prediction improvement.

The methodology is intended for probabilistic binary classifiers and
does not require selecting a classification threshold.

## Main functionality

`Usmile` provides functions for:

- comparing two fitted probabilistic binary classification models;
- comparing vectors of predicted probabilities;
- calculating class- and subclass-specific U-smile coefficients;
- visualizing prediction improvement and worsening;
- generating U-smile, PIW, ROC, precision-recall, and calibration plots;
- applying externally fitted probability calibration models;
- analysing training and external test datasets.

The core functions include:

| Function | Purpose |
|---|---|
| `UScalc_mdl()` | Compare two fitted models |
| `USprep_mdl()` | Extract outcomes and predicted probabilities from a fitted model |
| `USbind_out()` | Combine prepared outputs from reference and new models |
| `UScalc_raw()` | Calculate U-smile coefficients from predicted probabilities |
| `USplot()` | Draw a U-smile plot |
| `PIWplot()` | Draw a prediction improvement-worsening plot |
| `ROCplot()` | Compare ROC curves |
| `PRCplot()` | Compare precision-recall curves |
| `CLBplot()` | Assess probability calibration |
| `USfit_calibrator()` | Fit a probability calibration model |
| `USapply_calibrator()` | Apply a fitted probability calibration model |

## Installation

### CRAN

After the package is published on CRAN, it can be installed with:

```r
install.packages("Usmile")
```

### Development version

The current development version can be installed from GitHub:

```r
# install.packages("remotes")
remotes::install_github("bbwieckowska/Usmile")
```

## Quick start

The following example compares a reference logistic regression model
with a larger model containing additional predictors.

```r
library(Usmile)

data("heart_disease_train")
data("heart_disease_test")

heart_disease_train$disease <- as.factor(
  heart_disease_train$disease
)

heart_disease_test$disease <- as.factor(
  heart_disease_test$disease
)

reference_model <- stats::glm(
  disease ~ 1,
  data = heart_disease_train,
  family = stats::binomial()
)

new_model <- stats::glm(
  disease ~ ill_high_asym + age + cp,
  data = heart_disease_train,
  family = stats::binomial()
)
```

### Evaluation on the training dataset

```r
results_train <- UScalc_mdl(
  ref_model = reference_model,
  new_model = new_model,
  y_coef = "rLR",
  testing = FALSE
)

results_train$results
```

```r
USplot(
  plot_data = results_train$plot_data,
  y_coef = "rLR",
  net = TRUE,
  crit = 2
)
```

### Evaluation on an external test dataset

```r
results_test <- UScalc_mdl(
  ref_model = reference_model,
  new_model = new_model,
  y_coef = "rLR",
  dataset = heart_disease_test,
  testing = TRUE
)

results_test$results
```

```r
USplot(
  plot_data = results_test$plot_data,
  y_coef = "rLR",
  net = TRUE,
  crit = 2
)
```

## Comparison from predicted probabilities

The U-smile coefficients can also be calculated without passing fitted
model objects.

The input data frame must contain:

- `y`: observed binary outcome;
- `p_ref`: probability predicted by the reference model;
- `p`: probability predicted by the new model.

```r
prediction_data <- data.frame(
  y = c(0, 0, 1, 1),
  p_ref = c(0.10, 0.35, 0.60, 0.85),
  p = c(0.05, 0.40, 0.55, 0.92)
)

results_raw <- UScalc_raw(
  raw_data = prediction_data,
  y_coef = "rLR",
  n_vars_diff = 1
)

USplot(
  plot_data = results_raw,
  y_coef = "rLR",
  net = TRUE,
  crit = 2
)
```

This workflow is model-agnostic because it requires only observed
outcomes and predicted probabilities.

## Probability calibration

Calibration must be fitted independently of the dataset used for final
model evaluation.

```r
calibration_outcomes <- c(0, 0, 0, 1, 1, 1)

calibration_predictions <- c(
  0.10,
  0.25,
  0.40,
  0.55,
  0.75,
  0.90
)

# Fit the calibrator on an independent calibration dataset
calibrator <- USfit_calibrator(
  y = calibration_outcomes,
  p = calibration_predictions,
  method = "logistic"
)

# Example predictions from a separate test dataset
test_predictions <- c(
  0.15,
  0.35,
  0.65,
  0.85
)

calibrated_probabilities <- USapply_calibrator(
  object = calibrator,
  p = test_predictions
)

calibrated_probabilities
```

Available calibration methods are:

- `"logistic"`;
- `"intercept"`;
- `"isotonic"`.

The calibration model should not be estimated using the final test
dataset.

## Shiny application

A point-and-click implementation of the U-smile workflow is available
as a companion Shiny application:

<https://barbarawieckowska.shinyapps.io/ShinyApp/>

The Shiny application is intended for interactive analyses,
demonstrations, and educational use.

## Documentation

Function-level documentation is available directly in R:

```r
help(package = "Usmile")
?UScalc_mdl
?UScalc_raw
?USplot
```

The package citation can be displayed with:

```r
citation("Usmile")
```

A PDF reference manual can be generated from the package source with:

```r
devtools::build_manual()
```

## Methodological background

The original U-smile framework, including the BA, RB, and I
coefficients and the prediction improvement-worsening matrix, was
introduced in:

Kubiak KB, Więckowska B, Jodłowska-Siewert E, Guzik P (2024).
*Visualising and quantifying the usefulness of new predictors
stratified by outcome class: The U-smile method.*
PLOS ONE, 19(5), e0303276.  
<https://doi.org/10.1371/journal.pone.0303276>

The three-level U-smile approach and its evaluation under class
imbalance were described in:

Więckowska B, Kubiak KB, Guzik P (2025).
*Evaluating the three-level approach of the U-smile method for
imbalanced binary classification.*
PLOS ONE, 20(4), e0321661.  
<https://doi.org/10.1371/journal.pone.0321661>

The likelihood-based extension of the methodology and the `rLR`
coefficient were described in:

Więckowska B, Guzik P (2026).
*Usmile likelihood evaluation provides robust threshold free
assessment of binary classification models for balanced and imbalanced
datasets.*
Scientific Reports, 16, 10000.  
<https://doi.org/10.1038/s41598-026-40545-z>

## Citation

When using the package, cite both the software version and the
publication describing the methodological component used in the
analysis.

The recommended citations can be obtained with:

```r
citation("Usmile")
```

## Reproducibility

For reproducible analyses, report at least:

- the `Usmile` package version;
- the R version;
- the coefficient used: `BA`, `RB`, `I`, or `rLR`;
- whether results were calculated on training or test data;
- the definitions of the reference and new models;
- any probability calibration procedure;
- the random seed for stochastic modelling algorithms.

Package and R versions can be recorded with:

```r
packageVersion("Usmile")
sessionInfo()
```

## Contributing

Bug reports and feature requests can be submitted through the GitHub
issue tracker:

<https://github.com/bbwieckowska/Usmile/issues>

Contributions should preserve compatibility with the documented
statistical definitions of the U-smile coefficients. Changes affecting
numerical results should include appropriate tests and documentation.

## License

`Usmile` is released under the MIT License. See the `LICENSE` file for
details.