library(charlatan)Charlatan creates realistic looking data. By now you have seen some examples of the high level and low level APIs. But maybe you would like to see some examples of generating data for specific use-cases? Then this vignette is for you. It will show you how to create fake transactional data, fake health care (PII) data and finally an example of a plumber API that returns fake data.
But first a slight detour into different ways of faking data.
Simulation of data (creating fake data that looks realistic) is not always the best option, there are other options that we will discuss first.
Using real actual data is great for live systems. You want actual data for actual predictions or actual inference. It is, however smart to reduce your data to only the columns you need.
Using real actual data is a bad idea for demonstration use-cases. Personal Identifyable Information (PII) are sensitive data and need to be handled with care. Even if you strip away the full names it is relatively easy to pinpoint specific people on the basis of several data points.
Pseudonymous or anonymous data is a safer way to deal with personal identifyable data. By stripping away PII or replacing values with other random values you make it harder to link the data back to actual people. The main pro in using this kind of data is that you keep the data structure intact. The relationship between variables remain, and you can do predictions/inference on that.
It is quite a lot of work to make your data truly anonymous. You need surprisingly little data to de-anonomize data and link it back to actual people.
Creating a fake dataset from scratch is relatively
easy when the data is small. Using the build-in functions
likesample, runif, and rnorm etc.
is very doable. It is slightly faster than using this package.
Using completely random values does make it difficult to talk to stakeholders, they would not understand the random values. It is also really a lot of work to generate more complex data or many different columns.
This package creates realistic looking values, but does not help you in creating relations in a data set. For instance you can create streets, cities and postal codes, but those components are unrelated, the postal code is not related to the street nor to the city.
This package creates realistic names, jobs, streets, business names, phone numbers and much more. It also can create those things for different countries and languages (locales). For example:
Next are a few examples of faked data with the help of the {charlatan} package. The examples are with the en_US locale, but work for many locales.
This example shows transactional data, from an ecommerce website for example, and will also show you how to add some logical structure to your data. This example comes from an issue opened on our github.
Let’s imagine we sell clothes on the internet. What would that data look like?
I would need:
# setup
fraudster_cl <- fraudster("en_US")
n <- 5
set.seed(1235)We first create a few categories and subcategories. This is too specific for the {charlatan} package (and different for every store). I imagine you have a better idea for this data then I have.
In this example I have categories Shoes, Jeans and Dresses. All Shoes have a prefix that starts with 1, all Jeans have prefix starting with 2, and Dresses start with 5. We combine the prefix with a random number to have consistent product ids ( I have no idea if clothing stores actually do this, but it looked neat).
# create product data
products <- data.frame(
prefix = c(rep(1, 5), rep(2, 2), rep(5, 2)),
product_id = fraudster_cl$integer(n = 9, min = 1000, max = 9999),
main_category = c(rep("Shoes", 5), rep("Jeans", 2), rep("Dresses", 2)),
sub_category = c(
"Dress shoes", "Tennis shoes", "Boots", "Hiking boots", "Country & Western style boots",
"Regular fit", "Straight fit",
"Summer dress", "Evening gown"
)
)
## when you have {dplyr} installed there are way cleaner ways to do this
products$product_id <- as.integer(sprintf("%s%s", as.character(products$prefix), products$product_id))
products
#> prefix product_id main_category sub_category
#> 1 1 12006 Shoes Dress shoes
#> 2 1 17515 Shoes Tennis shoes
#> 3 1 16826 Shoes Boots
#> 4 1 13916 Shoes Hiking boots
#> 5 1 13979 Shoes Country & Western style boots
#> 6 2 23811 Jeans Regular fit
#> 7 2 22250 Jeans Straight fit
#> 8 5 59459 Dresses Summer dress
#> 9 5 51427 Dresses Evening gownThen we create the orders with a price, a product id, and location. Orders also have an email address and are shipped to an address.
# create orders
orders <- data.frame(
order_id = fraudster_cl$integer(n = n, min = 10000, max = 90000),
location_id = fraudster_cl$integer(n = n, min = 1, max = 5),
price_paid = fraudster_cl$integer(n = n, min = 1, max = 9900) / 100,
product_id = sample(products$product_id, size = n, replace = TRUE),
order_email = fraudster_cl$email(n = n),
customer_name = fraudster_cl$name(n = n),
shipping_address = fraudster_cl$address(n = n)
)Finally we combine everything:
# combine orders and transactions
example_transactions <- merge(orders, products)
## reorder the columns to let it make more sense.
example_transactions[, c("order_id", "location_id", "product_id", "main_category", "sub_category", "price_paid", "customer_name", "order_email", "shipping_address")]
#> order_id location_id product_id main_category sub_category
#> 1 38900 5 13979 Shoes Country & Western style boots
#> 2 82163 5 13979 Shoes Country & Western style boots
#> 3 26387 2 16826 Shoes Boots
#> 4 49425 1 17515 Shoes Tennis shoes
#> 5 32062 4 17515 Shoes Tennis shoes
#> price_paid customer_name order_email
#> 1 68.89 Sal Klein meghan99@group.com
#> 2 21.13 Darnell Stiedemann dillon.bartoletti@gmail.com
#> 3 74.89 Santana Langosh wyman.brody@gmail.com
#> 4 74.75 Earley Hickle howell.samantha@hotmail.com
#> 5 31.19 Dr. Andra White MD helmer.hayes@corwin.net
#> shipping_address
#> 1 05992 Dona Squares\nMooreview, MH 32273
#> 2 2481 Plummer Drive\nMicaylatown, VA 45528
#> 3 86826 Zeke Lodge Apt. 861\nEast Saigeland, TN 70514-0551
#> 4 407 Herzog Pass\nEast Enochfurt, NJ 23062
#> 5 516 Marks Forest Apt. 007\nTowneton, MT 48453Notice that customer_name and email are completely unrelated. You could create a customer ‘table’ like the product table above to create a bit more structure.
Here is how you simulate protected health information with {charlatan}. Here we use the low level api.
This example also comes from an issue in our github.
First a setup:
# setup the providers
ap <- AddressProvider_en_US$new()
pp <- PersonProvider_en_US$new()
ip <- InternetProvider_en_US$new()
lp <- LoremProvider_en_US$new()
SSNP <- SSNProvider_en_US$new()
dtp <- DateTimeProvider$new()
np <- NumericsProvider$new()
pnp <- PhoneNumberProvider_en_US$new()
set.seed(1235)We don’t have a list of counties in the US (there are 3007 of them).
So we will use a random word from the LoremProvider with
county. It is probably nicer if you wrap this into a function.
Generate a single ‘record’ for a person:
prot_health <- list(
first_name = pp$first_name(),
last_name = pp$last_name(),
phone_number = pnp$render(),
fax_number = pnp$render(),
street = ap$street_address(),
zipcode = ap$postcode(),
email = ip$email(),
county = paste0(lp$word(), " county"),
SSN = SSNP$render(),
dob = as.Date(dtp$date_time_between("1930-01-01", "1990-12-31")),
# I've decided record number is an integer between 10000 - 99999
medical_record_number = np$integer(min = 10000, max = 99999),
ip_address = ip$ipv4()
)
prot_health
#> $first_name
#> [1] "Efrem"
#>
#> $last_name
#> [1] "Schaden"
#>
#> $phone_number
#> [1] "243.246.1773"
#>
#> $fax_number
#> [1] "+68(4)7385942080"
#>
#> $street
#> [1] "2414 Howell Stravenue"
#>
#> $zipcode
#> [1] "54142"
#>
#> $email
#> [1] "alta70@gmail.com"
#>
#> $county
#> [1] "benefit county"
#>
#> $SSN
#> [1] "275-05-1676"
#>
#> $dob
#> [1] "1954-06-03"
#>
#> $medical_record_number
#> [1] 76973
#>
#> $ip_address
#> [1] "133.80.147.164"We can also create medical records in sequence with a custom function. In the following example I create a sequence of events based on a date.
#' Generate a bunch of dates in sequence
gen_med_record <- function(date_value, events = 4, event_types = c("admission", "x-ray", "blood-test", "general exam")) {
days <- sort(np$integer(events, 1, 365))
result <- data.frame(
date = date_value + days
)
result$event <- sample(event_types, size = nrow(result), replace = TRUE)
result
}
result <- gen_med_record(date_value = as.Date("2022-03-01"), events = 5)
## And add the generated data to this dataframe too
result$medical_record_number <- prot_health$medical_record_number
result$last_name <- prot_health$last_name
result$date_of_birth <- prot_health$dob
result
#> date event medical_record_number last_name date_of_birth
#> 1 2022-05-12 blood-test 76973 Schaden 1954-06-03
#> 2 2022-05-14 general exam 76973 Schaden 1954-06-03
#> 3 2022-10-03 x-ray 76973 Schaden 1954-06-03
#> 4 2022-11-21 x-ray 76973 Schaden 1954-06-03
#> 5 2023-02-09 general exam 76973 Schaden 1954-06-03You can also create a plumber API that returns realistic data. For this example to work you need the {plumber} package installed.
# plumber.R
fraudster_cl <- fraudster('en_US')
#* Create a random address
#* @param n how many do you want
#* @get /adress
function(n=1){
list(address=fraudster_cl$address())
}
#* Create a random company
#* @param n how many do you want
#* @get /company
function(n=1){
list(address=fraudster_cl$company())
}Then run the file like this to start an API:
plumber::plumb("R/plumber.R") %>% plumber::pr_run()
As you can see this package helps you in generating plausible data, but to add structure (relationships between variables) to your data you need to do work yourself.