# microdatasus 3.0.0

## Downloads

* Reworks `fetch_datasus()` around an internal registry of supported systems and
  manifests built from the files actually published by DataSUS. Each relevant
  FTP directory is listed only once per call, avoiding redundant connections
  and URLs for files that do not exist.
* Selects a single file for each system, period, state, and file part.
  Definitive/current data take precedence over preliminary data, and current
  data take precedence over historical copies. Results retain deterministic
  period, state, and file-part order.
* Uses `curl` directly for directory listings and downloads, with per-operation
  timeouts and up to two retries for transient network failures. Downloads
  remain sequential, display transfer progress by default, and no longer modify
  the global `options("timeout")`.
* Downloads through temporary partial files, validates file size and DBC
  contents, and reliably removes temporary files. With `stop_on_error = FALSE`,
  successfully read files are returned after a consolidated warning; with
  `stop_on_error = TRUE`, any failure aborts the operation.
* Applies `vars` before accumulating results and combines files only once,
  reducing repeated copies. When `track_source = TRUE`, the `source` column is
  retained even when `vars` is supplied, and an existing `source` column now
  produces a clear error.
* Strengthens validation of years, months, states, logical arguments, `vars`,
  and fractional `timeout` values. Monthly systems require months, annual
  systems ignore them with one warning, and `"all"` can no longer be combined
  with individual states.
* Replaces the former global 1996 lower-year restriction with historical limits
  specific to each information system. Existing public arguments, their order,
  and their defaults remain unchanged.
* Adds `quiet` at the end of the `fetch_datasus()` signature. Transfer progress
  and status messages, including each file name, are displayed by default; set
  `quiet = TRUE` to hide them. Warnings and errors remain visible.
* Uses immediate `cli` warning alerts when month arguments or state selections
  are ignored, including when `quiet = TRUE`.
* Standardizes download messages across microdata and auxiliary tables, with
  consistent status verbs, semantic `cli` formatting, indexed file progress,
  and one diagnostic item per failed directory or file.
* Hardens `fetch_cadger()` and `fetch_sigtab()` with shared timeout, retry, and
  temporary-file cleanup logic, and adds ZIP integrity and schema validation.

## DBC support

* Removes the dependency on `read.dbc` and reads DBC files with vendored C code
  adapted from `healthbR`. Thanks to Sidney Bissoli for the open-source code.
* Adds `read_dbc()` for local DBC files, with optional preservation of the
  column types inferred from DBF metadata.
* Hardens `read_dbc()` by rejecting directories and empty or unreadable input
  files before decompression, validating missing, unknown-size, and empty
  decompression output, and wrapping DBF read failures in contextual errors
  with stable `microdatasus_dbc_error` subclasses.
* Expands local, CRAN-safe DBC tests for argument validation, type preservation,
  leading zeros, paths with spaces, repeat reads, temporary-file cleanup, and
  specific malformed header and compressed-stream errors.

## Processing, documentation, and testing

* Adopts `dplyr::recode_values()` because `dplyr::case_match()` is deprecated.
* Expands and corrects the documentation for all public functions, including
  network behavior, actual return types, compatibility arguments, and links to
  the corresponding chapters of the book *Sistemas de Informação em Saúde no
  Brasil*.
* Replaces skipped download tests with deterministic simulated listings and
  transfers. The suite now covers supported systems, historical, current, and
  preliminary files, multipart data, complete argument validation, retries,
  timeouts, empty listings and manifests, cleanup, corrupt and empty files,
  partial success, `vars`, `track_source`, multiple states, and auxiliary-table
  schema validation. Local `file://` fixtures exercise `curl` without network
  access; live DataSUS smoke tests remain opt-in and are always skipped on CRAN.
* Runs the active GitHub Actions R CMD check workflow for pushes and pull
  requests involving the `dev` branch as well as `main` and `master`.
* Requires R 4.1.0 or later.

# microdatasus 2.5.0
* Function `process_sinan_malaria`, fix variable type when pre-processing the variable `NU_IDADE_N` (#132)
* Function `process_sinasc`, fix typo when pre-processing the variable `PESO` (#141).
* Function `process_sia`, fix typo when pre-processing the variable `PA_MUNPCN` (#137).
* Function `process_cnes` for CNES-ST, fix `MICR_REG` and `DISTRSAN` conversion.
* Function `process_sih`, fix `NACIONAL` and `HOMONIMO` codes, add more `FAEC_TP` codes (#131).
* Remove `dtplyr` package dependence.

# microdatasus 2.4.3
* Remove `dtplyr` usage due incompatibility on recent version.

# microdatasus 2.4.2
* Fix error when processing data with `process_sinan_chikungunya()`.

# microdatasus 2.4.1
* Fix typo in process functions that was changing variable names (`process_sim()`, `process_sinasc()`,`process_sinan_zika()`, `process_sinan_malaria()`, `process_sinan_leishmaniose_visceral()`, `process_sinan_leishmaniose_tegumentar()`, `process_sinan_dengue()`, `process_sinan_chikungunya()`, `process_sinan_chagas()`).

# microdatasus 2.4.0
* Update documentation on `fetch_datasus()`.
* Fetch SINAN-CHAGAS data.
* Process SINAN-CHAGAS data.
* Fetch SINAN-LEISHMANIOSE-VISCERAL and SINAN-LEISHMANIOSE-TEGUMENTAR data.
* Process SINAN-LEISHMANIOSE-VISCERAL and SINAN-LEISHMANIOSE-TEGUMENTAR data.
* Fetch SINAN-LEPTOSPIROSE data.
* Fix message when downloading data for all UFs.

# microdatasus 2.3.5
* Treat age field `NU_IDADE` on pre-processing SINAN on old files.

# microdatasus 2.3.4
* Fix error on tabNaturalidade, issue #123. Thanks @SophiaDamianoRovere and @hafermoraes for the suggestions.
* Fix FANTASIA names on CNES, issue #125.

# microdatasus 2.3.3
* Add leading zeros to CNES TIPO_UNID processing.

# microdatasus 2.3.2
* Fix processing of `ESPEC` variable on `process_sih` function (#113).
* Use read.dbc package from Github repo.
* Update TP_UNID processing on `process_cnes` function (PR #112).

# microdatasus 2.3.1
* Import read.dbc package from CRAN.
* Remove CADGER data.
* General package adjustments for CRAN submission.
* Examples of process functions use data samples.
* Update package citation.

# microdatasus 2.3.0
* All process functions uses dplyr verbs and dtplyr for fast processing.
* All process functions returns a tibble.
* Add `track_source` argument to `fetch_datasus` function to create a variable called `source` with the file name that originated the row.
* Updated CBO (CNES), CADGER (CNES), EQUIPE (SIA), SIGTAB (SIA) reference tables.
* Sample data for health information systems.
* Data files with XZ compression.
* Internal tests restructured.
* Progress bar removed from `process_sim` due increased processing time.

# microdatasus 2.2.7
* Enhance messages of `fetch_datasus`.
* Timeout the connection check when DataSUS FTP is down or not reachable.
* If there is no Internet or the DataSUS FTP is down or not reachable, the `fetch_datasus` will return `NULL`.
* `process_sim` shows a progress bar.

# microdatasus 2.2.6
* Age correction for SINAN data.

# microdatasus 2.2.5
* Patch to correct Invalid multibyte string error on chikungunya processing. 

# microdatasus 2.2.4
* process_sia and process_sih internally uses now data.table and dtplyr

# microdatasus 2.2.3
* As the package {read.dbc} is not available on CRAN, this patch uses the Github version.
* Issue #89 points out that files from SIM-EXT older than 2006 present 7-digits variable lengths for CODMUNRES and CODMUNOCOR. Those codes are now truncated to 6-digits standard by process_sim function.
* Related to #66, #84 and #86. Some files are very big to download, especially those from SIA and SIH. A timeout argument was added to the fetch_datasus with a default of 240 seconds.
* process_sinasc pull request #91 fixes issue #90, related to CODOCUPMAE variable
* Related to #79, now the function process_sia downloads an updated version from the SIGTAB table from DataSUS when nome_proced is TRUE. 

# microdatasus 2.2.2
* process_sinasc correct old code for missing on ESCMAE

# microdatasus 2.2.1
* process_sinasc with new codes to process sex variable, avoiding missing results.
* process_cnes corrections to avoid NA introduction by coercion.
* process_sih corrects MUNIC_RES type

# microdatasus 2.2.0
* fetch_datasus function uses read.dbc function with as.is set to TRUE for better performance.
* Minor bugs corrections.

# microdatasus 2.1.3
* process_cnes uses {data.table} backend for performance

# microdatasus 2.1.2
* read.dbc is back on CRAN

# microdatasus 2.1.1
* Patch to use development version of read.dbc package.

# microdatasus 2.1.0
* Bug correction that impacted download of all UFs from monthly data health systems.

# microdatasus 2.0.6
* Updated codes of tabNaturalidade
* SIA-PA fetch files bug corrected.

# microdatasus 2.0.5
* Bug correction to download CNES-LT files.

# microdatasus 2.0.4
* Bug correction to download all UFs.

# microdatasus 2.0.3
* SINAN-Dengue bug correction.

# microdatasus 2.0.2
* SINASC bug correction (ESCMAE2010).

# microdatasus 2.0.1
* Ages in minute unit.

# microdatasus 2.0.0
* Integrated download of old, current, and preliminar data.

# microdatasus 1.4.8
* Tests correction
* Typo correcntion

# microdatasus 1.4.7
* Bug correction for download with newer R versions.

# microdatasus 1.4.6
* Bug correction at process_sia function.

# microdatasus 1.4.5
* Bug correction at process_sim function.

# microdatasus 1.4.4
* Updates SINAN functions for DataSUS changes in file structures, from per UF basis to all data (BR).

# microdatasus 1.4.3
* Correct SIH COD_IDADE value 5 for ages above 100 years.

# microdatasus 1.4.2
* Correct codmun handling for CNEST-ST data. Issue #38

# microdatasus 1.4.1
* Check local Internet connection and DataSUS FTP server availability before download.
* Argument to fetch_datasus to stop download if there is an error.
* Now is possible to download preliminar data from SIM-DO and SIM-DOFET with "SIM-DO-PRELIM" as information system at fetch_datasus.

# microdatasus 1.4.0
* Download and preprocess SINAN Malaria files.

# microdatasus 1.3.1
* Update functions documentation.

# microdatasus 1.3.0
* Download and preprocess SINAN Zika files.
* Minor error message corrections.

# microdatasus 1.2.0
* Download and preprocess SINAN Chikungunya files.

# microdatasus 1.1.4
* Minor correctiont at process_sinan_dengue function.

# microdatasus 1.1.3
* Documentation correction process_sinan_dengue function.

# microdatasus 1.1.2
* Document and export process_sinan_dengue function.

# microdatasus 1.1.1
* Fix NAT_JUR field at CNES files.

# microdatasus 1.1.0
* SINAN DENGUE files download and processing.

# microdatasus 1.0.0

* Complete overhaul of the package, meeting current R package standards.
* All functions revised.
* SIH code tables updated for COVID-19.
* Process CNES ST and PF.
* Added a `NEWS.md` file to track changes to the package.
