---
title: "Data Cleaning: From KOSIS Raw Data to Tidy Format"
output: rmarkdown::html_vignette
date: "`r format(Sys.Date(), '%B %d, %Y')`"
vignette: >
  %\VignetteIndexEntry{Data Cleaning: From KOSIS Raw Data to Tidy Format} 
  %\VignetteEngine{knitr::rmarkdown}
  %\usepackage[utf8]{inputenc}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  warning = FALSE,
  message = FALSE,
  eval = FALSE,
  # Set to FALSE since this requires API keys and large data downloads
  out.width = "100%"
)
```

```r
library(tidycensuskr)
library(dplyr)
library(tidyr)
library(plyr)
library(kosis)

```

## Introduction

This vignette demonstrates how to clean and transform raw Korean census data from the Korean Statistical Information Service (KOSIS) API into a tidy, analysis-ready format. While there are existing R packages for accessing Korean statistical data (such as the `kosis` package for general KOSIS data access), **there are currently no readily available software solutions specifically designed for retrieving and processing Korean census data in a tidy, analysis-friendly format** for demographic and socioeconomic research.

### The Challenge

Korean census and administrative data presents several challenges for researchers:

1. **Complex API URLs**: KOSIS APIs require detailed parameter specifications with cryptic codes
2. **Multiple data sources**: Different datasets (population, tax, mortality) come from different API endpoints
3. **Inconsistent formatting**: Raw data often comes in wide format with Korean-language column names
4. **Administrative code mapping**: Different datasets use different administrative code systems
5. **Data integration**: Combining multiple datasets requires careful matching of administrative units

This vignette shows how the `tidycensuskr` package addresses these challenges by providing pre-processed, tidy datasets and demonstrating the underlying data cleaning workflow.

## Raw Data Sources

The data cleaning process involves retrieving data from multiple KOSIS API endpoints:

You need to know following information to retrieve data from KOSIS API:

- **API Key**: You need to register for an API key from KOSIS.
- **API Parameters**: Each API endpoint requires specific parameters such as `itmId`, `objL1`, `objL2`, etc. These parameters define the data you want to retrieve.
- **Output Format**: The API supports various output formats, but we will use JSON for easier processing in R.


### API call parameters

| Parameter | Type | Description | Required? |
|-----------|------|-------------|-----|
| `apiKey` | String | Your KOSIS API key | Yes |
| `orgId` | String | Organization identifier | Yes |
| `tblId` | String | Table identifier | Yes |
| `objL1` | String | First class code | Yes |
| `objL2`-`objL8` | String | Second to eighth class code | Optional |
| `itmId` | String | Item identifier(s) | Yes |
| `prdSe` | String | Update cycle | Yes |
| `format` | String | Output format (e.g., JSON) | Yes |


### Data variables

To note, an output can include the following variables:

| Field name (all caps) | Description | Data type | Remarks |
|-----|--------------|-------|------|
| ORG_ID | Organization code | VARCHAR2(40) | Yes |
| TBL_ID | Table identifier | VARCHAR2(40) |  |
| TBL_NM | Table name | VARCHAR2(300) |  |
| C1 - C8 | Class identifier (1-8) | VARCHAR2(40) | 2-8 may be omitted if nonexisting |
| C1_OBJ_NM - C8_OBJ_NM | Class code (1-8) | VARCHAR2(3000) |  |
| C1_OBJ_NM_ENG - C8_OBJ_NM_ENG | Class code in English (1-8) | VARCHAR2(3000) |  |
| C1_NM - C8_NM | Class name (1-8) | VARCHAR2(3000) |  |
| C1_NM_ENG - C8_NM_ENG | Class name in English (1-8) | VARCHAR2(3000) |  |
| ITM_ID | Item identifier | VARCHAR2(40) |  |
| ITM_NM | Item name | VARCHAR2(3000) |  |
| ITM_NM_ENG | Item name in English | VARCHAR2(3000) |  |
| UNIT_ID | Unit identifier | VARCHAR2(40) |  |
| UNIT_NM | Unit name | VARCHAR2(1000) |  |
| UNIT_NM_ENG | Unit name in English | VARCHAR2(1000) |  |
| PRD_SE | Data update cycle | VARCHAR2(20) |  |
| PRD_DE | Data period | VARCHAR2(8) |  |
| DT | Data value | VARCHAR2(100) |  |
| LST_CHN_DE | Date of change | VARCHAR2(8) |  |

It is worth noting that all fields are character, and numeric values are stored as strings. This means that you will need to convert them to numeric types after loading the data.


### API Key
To use the KOSIS API, you need to register for an API key. You can obtain one by visiting the [KOSIS API registration page](https://kosis.kr/openapi/serviceUse/serviceUseUnityReg_01Detail.do) and following the instructions.

The webpage is only available in Korean and may not be available outside of Korea.


### Example URL

The URL below is an example of how to retrieve general tax data from KOSIS. You will need to replace `인증키없음` with your actual API key. Dissecting this URL can help you understand how to construct your own API calls.

```{r tax-url}
url_tax_general <- 
  "https://kosis.kr/openapi/Param/statisticsParameterData.do?method=getList&apiKey=인증키없음&itmId=T001+& &objL2=15133SGH0M+&objL3=&objL4=&objL5=&objL6=&objL7=&objL8=&format=json&jsonVD=Y&prdSe=Y&newEstPrdCnt=1&outputFields=TBL_ID+TBL_NM+OBJ_ID+OBJ_NM+OBJ_NM_ENG+NM+NM_ENG+ITM_ID+ITM_NM+ITM_NM_ENG+UNIT_NM+UNIT_NM_ENG+&orgId=133&tblId=DT_133N_A3212"
```


| Parameter | Value |
|-----------|-------------------|
| `apiKey` | "인증키없음" (meaning "no authentication") as it is for demonstration. |
| `orgId` | "133" |
| `tblId` | "DT_133N_A3212" |
| `objL1` | "A0201", "A0202", ... , "A1801" |
| `objL2` | "15133SGH0M" |
| `itmId` | "T001" |
| `prdSe` | "Y" |
| `newEstPrdCnt` | "1" |
| `outputFields` | "TBL_ID", "TBL_NM", ... , "UNIT_NM_ENG" |


Adding more context, multiple values for `objL1` and `itmId` can be specified by separating them with a plus sign (`+`). For example, `itmId=T001+T002+T003` would retrieve data for multiple items.

Try interpreting the following URL to understand how to construct your own API calls:

```{r pop-url}
url_pop <-
  "https://kosis.kr/openapi/Param/statisticsParameterData.do?method=getList&apiKey=인증키없음&itmId=T00+T60+& &objL2=ALL&objL3=000+&objL4=&objL5=&objL6=&objL7=&objL8=&format=json&jsonVD=Y&prdSe=Y&newEstPrdCnt=1&outputFields=TBL_ID+TBL_NM+OBJ_ID+OBJ_NM+OBJ_NM_ENG+NM+NM_ENG+ITM_ID+ITM_NM+ITM_NM_ENG+UNIT_NM+UNIT_NM_ENG+&orgId=101&tblId=DT_1IN1509"
```


## Data Retrieval

### Setting up API Access
```r
library(tidycensuskr)
library(dplyr)
library(tidyr)
library(kosis)

# Set KOSIS API key (stored in a secure file)
kosiskey <- readLines("~/.kosiskey")[1]
tidycensuskr::set_kosis_key("~/.kosiskey")
```

### Downloading Raw Data
```r
# Download raw datasets from KOSIS API
df_tax <- kosis::getStatDataFromURL(url_tax_general)

# Load administrative district lookup table
sgg_lookup <-
  read.csv(
    system.file(
      file.path("extdata", "lookup_district_code.csv"),
      package = "tidycensuskr"
    ),
    fileEncoding = "EUC-KR"
  )
```

## Data Cleaning Workflow

### 1. Administrative Code Mapping

The first challenge is creating a consistent mapping between different administrative code systems used across datasets:

```r
# Create administrative code mapping for provinces (sido)
sidocd_range <- tibble::tribble(
  ~sido_kr, ~sido_cd, ~sido_txcd,
  "서울특별시", "11", "02",
  "부산광역시", "21", "15", 
  "대구광역시", "22", "13",
  "인천광역시", "23", "03",
  "광주광역시", "24", "10",
  "대전광역시", "25", "06",
  "울산광역시", "26", "16",
  "세종특별자치시", "29", "09",
  "경기도", "31", "04",
  "강원특별자치도", "32", "05",
  "충청북도", "33", "07",
  "충청남도", "34", "08",
  "전라북도", "35", "11",
  "전라남도", "36", "12",
  "경상북도", "37", "14",
  "경상남도", "38", "17",
  "제주특별자치도", "39", "18"
)
```

### 2. Tax Data Processing

Transform raw tax data into a standardized format:

```r
df_tax_compact <- df_tax |>
  dplyr::transmute(
    adm2_code = C1,           # Administrative code
    value = DT   # Tax value in million KRW
  ) |>
  dplyr::inner_join(
    sgg_lookup[, c("sgg_tax_global", "sido_en", "sigungu_1_en", "adm2_code")],
    multiple = "first"
  )

```

### 3. Population Data Processing

Clean and reshape population data with gender disaggregation:

```r
df_pop2 <- df_pop |>
  dplyr::mutate(
    sex = plyr::mapvalues(C2, c(0, 1, 2), c("total", "male", "female")),
    type = plyr::mapvalues(ITM_ID, c("T00", "T60"), c("population_total", "population_nonrelative"))
  ) |>
  dplyr::select(C1, C1_NM, sex, type, DT) |>
  tidyr::pivot_wider(
    names_from = c(type, sex),
    values_from = DT
  ) |>
  dplyr::rename(
    sigungu_cd = C1,
    sigungu_kr = C1_NM
  ) |>
  dplyr::mutate(
    adm2_code = as.integer(adm2_code)
  ) |>
  dplyr::inner_join(
    sgg_lookup[, c("adm2_code", "sido_en", "sigungu_1_en")],
    by = "adm2_code",
    multiple = "first"
  )
```

## Transforming to Tidy Format

The key feature of `tidycensuskr` is converting wide-format, multiple-dataset structure into a single, long-format tidy dataset:

### 1. Convert Each Dataset to Long Format

```r
# Tax data to long format
df_tax_long <- df_tax_compact |>
  dplyr::select(2:5) |>
  tidyr::pivot_longer(
    cols = "value"
  ) |>
  dplyr::mutate(
    type = "tax",
    class1 = "global",
    class2 = "total",
    unit = "million KRW"
  ) |>
  dplyr::select(-name)

# Population data to long format
df_pop_long <- df_pop2 |>
  dplyr::select(-2) |>
  tidyr::pivot_longer(
    cols = 2:7
  ) |>
  tidyr::separate(col = "name", into = c("type", "class1", "class2"), sep = "_")

```

### 2. Combine into Single Tidy Dataset

```r
# Bind all datasets into one comprehensive long-format dataset
censuskor <- dplyr::bind_rows(
    df_tax_long,
    df_pop_long
  ) |>
  dplyr::rename(
    adm1 = sido_en,
    adm2 = sigungu_1_en
  ) |>
  dplyr::mutate(
    year = 2020
  ) |>
  dplyr::select(
    year, adm1, adm2, adm2_other, adm2_code,
    type, class1, class2, unit, value
  )
```

## The Result: A Tidy Dataset

The final `censuskor` dataset has a consistent structure:

- **year**: Census year (2020)
- **adm1**: Province-level administrative name (in English)
- **adm2**: District-level administrative name (in English)  
- **adm2_code**: Numeric administrative code
- **type**: Data type ("population", "tax", "mortality")
- **class1**: First-level classification (e.g., "global", "income", "All causes")
- **class2**: Second-level classification (e.g., "total", "male", "female")
- **unit**: Unit of measurement
- **value**: Numeric value

### Example Usage

```r
library(tidycensuskr)
library(dplyr)

# Load the cleaned dataset
data(censuskor)

# View the structure
head(censuskor)

# Filter for Seoul population data
seoul_pop <- censuskor |>
  filter(adm1 == "Seoul", type == "population", class1 == "population") |>
  select(adm2, class2, value) |>
  pivot_wider(names_from = class2, values_from = value)

head(seoul_pop)
```

These data can be joined with `sf` objects bundled in the package to make maps and perform spatial analysis

## Notes on `adm2_code` changes
Districts (_si-gun-gu_) are the second-level administrative units in South Korea. Each district is assigned a unique code called `adm2_code`. However, these codes can change over time due to administrative boundary changes, mergers, or reclassifications.

Notable changes were made in all rural districts (_gun_) in 2022; the third digit of `adm2_code` got addition of 2. For example, _Ulleung-gun_ changed from `37430` to `37630`. This change may affect longitudinal analyses that span multiple years and pose challenges in updating `censuskor` dataset by developers. Some datasets in KOSIS are **retrospectively** updated to reflect the new codes, while most KOSIS datasets retain the old codes. Therefore, developers should be advised to check the `adm2_code` for each dataset and year with this code:

```r
# imported data
some_census_table

# check adm2_code
any(substr(some_census_table$adm2_code, 3, 3) %in% c("5", "6"))
```

If `TRUE`, it indicates that the dataset uses the **new** `adm2_code` system (post-2022) regardless of the raw data year. If `FALSE`, it uses the **old** system (pre-2022).


**Note**: This vignette shows the data cleaning process for developers. The actual `censuskor` dataset in the package is the result of this workflow and is ready for immediate use.
