rduckhts_depth_state_solutions() and
rduckhts_depth_state_segments() fit the bundled
ichorCNA-style copy-number state model to corrected bins from
rduckhts_depth_bins(): tumour fraction, ploidy and rank of
every start, and each solution’s copy-number segments. Arguments are
checked in R before any query; table_name publishes the
result as a table.duckhts_somalier_bam_counts reads its panel
when the query is bound, on the calling thread, instead of on a DuckDB
worker thread. On Windows, closing the private panel instance on a
worker thread left a DuckDB thread_local that the
Rtools-built duckdb package destroys on freed memory when
the worker exits, which ended the R session with status
0xC0000005 at dbDisconnect(shutdown = TRUE),
most often on Windows Server 2022.
rduckhts_somalier_bam_counts() and
rduckhts_ancestry_bam() are unchanged in behaviour.Bundled regional depth fractions handle large-coordinate bin midpoints without overflow or rounding. Sex-depth reference counts exclude samples with no usable calibration level, correctly leave out the tested sample, and reject NULL contigs instead of including them in the autosomal baseline.
Bundled insert-size metrics accept large finite deviation settings without an integer overflow when computing the histogram cutoff.
The depth wrapper and bundled SQL reject gc_step
below 1/2147483647, keeping GC stratum numbers within the
integer output range.
The bundled depth correction centers on eligible autosomal bins with positive reads. Samples with at least half their bins empty keep valid normalization; empty bins retain zero ratios when their bin and stratum rules pass. Training quantiles stay within the observed rate range, preserving constant-rate samples when continuous-quantile interpolation rounds beyond that range.
rduckhts_bam_insert_size_counts() exposes native
BAM/CRAM insert-size counts;
rduckhts_bam_insert_size_counts_sql() exposes the SQL
composition, temporarily setting and restoring DuckDB threads. Both arms
remain available; native takes the unsuffixed name by measurement in
benchmarks/benchmark_insert_size_counts.md.
duckhts_insert_size_metrics() summarizes either counts
relation in SQL. Bundled wrapper fixtures retain BAM/CRAM and riker
parity in 17 files (15.6 KB). Semantics credit Fulcrum Genomics riker
0.4.1 (MIT), the executable oracle: reject first-segment and QC-fail
records, use POS + TLEN for the forward read’s mate end, and label equal
strands TANDEM. These upstream rules supersede the design
note’s draft rules. Picard is context only.
The bundled function catalog documents chr20 validation limits of the low-pass mixture estimator, including weak small-share detection and unvalidated genome-wide precision.
The bundled function catalog documents autosomal low-pass validation across 90 strata with 200 replicates each. At 5% share, positive interval limits occur in 199–200/200 unrelated and 160–165/200 IBD1 replicates at 0.1x, and 200/200 across tested relationships at 0.4x. Only 30/75 primary strata meet every declared criterion, so a universal detection limit is not established. The catalog also describes chr20 small-share validation limits.
The bundled low-pass mixture estimator uses exact binary sums with fixed-size per-block accumulators and reproducible parallel results. SQL validation and join memory is controlled by DuckDB, separately from the native byte budget.
rduckhts_depth_bins(),
rduckhts_depth_region_fraction() and
rduckhts_depth_sex_fraction() wrap the new depth macros:
GC-corrected bin ratios from bin counts, the share of cells with a
one-copy change of each named region, and the share of a second genome
of the other sex from X and Y depth, each estimator with an optional
reference set. Empty bins do not enter the outlier quantiles that select
the training bins. The tests compare every value with an independent
base R implementation of the same formulas and recover a planted gain
and an XX/XY mixture.
rduckhts_roh_counts(fit_table = ...) reads the
counts relation once. A counts relation that held a join was kept in
full in memory by DuckDB, about three times the memory of the call
without fit_table for 100 samples. The runs and the errors
are unchanged.
rduckhts_count_mixture_lowpass() exposes the bundled
extension’s separate conditional second-genome share estimator for
low-pass counts. Error, artefact and homozygosity calibration are
required. Results include bounded block-based limits, QC read/pair
totals and an unresolved diagnostic for matched-frequency relationship
hypotheses. The default interval floor is 30 informative blocks;
min_blocks supports small-fixture checks. Existing
count-error fit results are unchanged.
rduckhts_count_error_fit() returns
artefact_weight_sd after artefact_weight: the
block interval of the artefact weight, formed as
contamination_sd is, and NA with fewer than
three blocks. No other value changes.
rduckhts_roh_counts(fit_table = ...) needs an
artefact_weight_sd column in the fit relation
(NA counts as 0) and returns stable_fraction
for each run: the share of the run’s bases that are also called when the
artefact weight moves down and up by artefact_weight_sd,
and gained_bases: the bases that the higher weight calls
next to the run but outside every run. The other columns are unchanged,
and the decode takes about three times as long. Without
fit_table the columns are unchanged.
rduckhts_roh(site_rule = "bcftools", af_order = ...)
names a column of af_table that gives the order of its
lines, for example a line number. Several rows may then share a
position, and the row with the least value is used, as
bcftools roh --AF-file reads the first line of a position.
A NULL value, a value repeated within a position and a name
that is not a column are errors. af_order must be one
nonempty string and needs site_rule = "bcftools".
rduckhts_roh_counts() gains
allele_balance = 0.5 and artefact_weight = 0,
the numbers that rduckhts_count_error_fit() reports under
these names: the share of a heterozygote’s reads that show the counted
allele, and the chance that a site’s counts are uniform on 0 to depth
whatever its genotype. With a positive artefact_weight, one
site can change the odds between the two states only by a bounded
factor, as gt_error does in rduckhts_roh().
The defaults give the previous runs and qualities. Both are validated in
R (allele_balance in [0.35, 0.65],
artefact_weight in [0, 0.05]); with fit_table,
the two are read from the fit relation, which must now hold
allele_balance and artefact_weight
columns.
The README describes two known DuckDB 1.5.x issues that affect
rduckhts_roh(), rduckhts_roh_counts(),
rduckhts_count_error_fit() and
rduckhts_somalier_matched_contamination() with
duckdb 1.5.x: an out-of-memory error can end the R session
(https://github.com/duckdb/duckdb/issues/26441), and a failed query with
5 or more threads can keep its native memory until the session ends
(https://github.com/duckdb/duckdb/issues/26871). The bundled function
catalog notes them for the same functions.
The bundled read_bam() decodes from a CRAM only the
fields that the selected columns need, with the same values. With
decompression threads, a query without SEQ, QUAL, CIGAR and tag columns
no longer grows in memory with the alignments read, and one without SEQ
and QUAL loads no reference for a slice that covers one reference. Tag
columns with threads still grow toward a bound, and SEQ or QUAL with
threads still load the whole contig reference;
decompression_threads := 0 avoids both. A table made by
rduckhts_bam() reads every column and is unchanged; queries
on its bam_data view gain the change.
The bundled readers leave out a MAP pair whose key is not valid
UTF-8 (attributes_map of rduckhts_gff(),
rduckhts_gtf() and rduckhts_genbank(),
AUXILIARY_TAGS of rduckhts_bam(),
key_values of rduckhts_hts_header()); such a
key used to become a NULL MAP key. A value that is not valid UTF-8 is
still NA.
rduckhts_roh(af_table = ..., site_rule = "bcftools")
uses the bundled duckhts_roh_af_table_bcftools macro, which
chooses the record of a repeated listed position as
bcftools roh --AF-file does: file order, indel class, the
next record’s POS on any contig, the first claiming record, and its
alleles. It needs one af_table row per position and a
sorted, BGZF-compressed VCF or BCF. The default "matching"
is unchanged. New bundled fixtures roh_repeated.vcf.gz and
roh_repeated_af.tsv.gz back the tests.
rduckhts_bcf(include_file_offset = TRUE) and the
bundled read_bcf(..., include_file_offset := true) add a
FILE_OFFSET column after FILTER: the BGZF
virtual position just after each record. Every scan mode gives a record
the same value, and ordering by it restores file order, also among
records at one position. Input that is not BGZF-compressed gives
NA. The default schema is unchanged. New bundled fixtures
(bcf_offset.bcf, bcf_offset.vcf.gz,
bcf_offset_plain.vcf and the expected positions in
bcf_record_ends.tsv) back the tests.
Headers stored by the Parquet convert functions keep their line
breaks; the bundled duckhts_raw_header_text() joined lines
with a literal backslash-n.
Added rduckhts_bam_mate_overlap_counts() and the
bundled duckhts_bam_mate_overlap_counts(): read 1 and read
2 of one fragment compared where both mates align, counted by base
quality and base. The disagreement rate is about twice the instrument
error rate, with no reference, mask or contamination model. Alignments
must be in coordinate order, and max_pending bounds the
mates held while they wait for their mate.
rduckhts_somalier_relatedness() has a
max_pairs argument (default 50,000,000): when it compares
all pairs, a sketch relation with more pairs is an error before any pair
is compared. The bundled
duckhts_somalier_relatedness_all_pairs() now compares the
pairs on DuckDB’s threads; the rows do not change with the thread
count.
rduckhts_count_error_fit() evaluates the
beta-binomial without the rounding error that inflated the
log-likelihood near the binomial limit, and its optimizer checks the
width of the whole simplex before it stops.
The bundled duckhts_bam_mismatch_counts() (and so
rduckhts_bam_mismatch_counts()) counts a read base written
as = as a match instead of leaving it out, and a VCF mask
leaves out the last base of an SVLEN deletion when a reference window
starts there, as a BCF mask did.
Bundled ROH decoding uses two traceback bits per site, reducing per-thread site workspace from 38 to approximately 36.25 bytes per site without changing output values. Emissions and forward/posterior probabilities remain doubles.
The bundled GenBank reader reuses named qualifier values across a feature’s joined-location segments. The function reference documents all-NULL columns for requested keys absent from every GenBank, GFF or GTF feature, with SQL examples for asserting key presence.
rduckhts_roh_counts(fit_table = ...) uses each
sample’s fitted seq_error and contamination
through the bundled duckhts_roh_counts_fitted macro. A
missing fit, a NULL parameter or a duplicate fit row is an error; fit
status filtering remains caller SQL, and spreads and artefact weight do
not enter the emission model.
Added rduckhts_count_error_fit() and the bundled
duckhts_count_error_fit(): read error, the share of reads
of a second genome, allele balance, spreads and an artefact weight
fitted from allele read counts at sites of known frequency, one row per
sample, with an interval from 10 Mb blocks and a second fit for a
related second genome.
Added rduckhts_bam_mismatch_counts() and the bundled
duckhts_bam_mismatch_counts(): aligned read bases counted
against the reference by mate, cycle, base quality and substitution,
with an optional mask of known variants. With such a mask, the
mismatches that remain are mostly errors of the read, the library or the
alignment, with the variants that the mask does not hold, so their rate
is an upper bound on the error rate. Small fixtures
(bam_mismatch.*) are bundled for the example and the
tests.
rduckhts_ancestry_proportions() compares chromosome
names with duckhts_contig_key(), the contig key of the
extension, in place of a regular expression and integer casts in SQL.
chrX now matches X, chrM matches
MT, and the letter case of chr does not
matter. A leading zero is now another name for a text reference:
chr01 no longer matches 1. A reference with an
integer chromosome column still compares numbers, for every DuckDB
integer type and for long references as for wide ones, and duplicates
are counted by the same key as the join, so chr01 and
chr1 at one position are duplicates for an integer
reference instead of two variants. Names that give no number, such as
X and Y, join nothing for an integer reference
and are not duplicates of each other.
rduckhts_bam_mismatch_counts() stops with an error
when a kept alignment ends past the length of its contig in the
reference, instead of leaving the bases past the end out of the counts;
the reference does not match the alignment header in that case. For CRAM
input, reference_index_path now reaches the CRAM decoder
too, so a FASTA index that is not next to the FASTA is read and none is
built beside it; the CRAM fixture (bam_mismatch.cram) is
bundled for that test.
The function reference states, under “Contig names”, how each function compares contig names across two sources, including the cases that return no rows without an error.
SQL that joins read_bam(),
read_tabix(), read_gff(),
read_gtf() or read_bed() plans better: the
bundled extension now gives DuckDB a row estimate for a whole-file
indexed scan.
rduckhts_bam() and read_bam() return
the right rows for the region items * (reads without
coordinates) and . (the whole file) when a BAM index has no
per-reference statistics, and for a BAM in which no read is placed.
These cases failed or returned no rows.
rduckhts_bam() and read_bam() with a
region no longer walk every reference of the index when the
query is prepared; only a full-file scan needs that work.
rduckhts_bam() and read_bam() no longer
repeat rows when a BAI or CSI index has alignments of a reference and no
statistics for it (the statistics are optional in both formats). A
full-file scan with more than one thread could return some records
twice.
Reading an indexed CRAM with rduckhts_bam(),
rduckhts_hts_index() or
rduckhts_hts_index_spans() no longer reads past the end of
the CRAM index object. Results are unchanged.
The documentation of rduckhts_roh_counts() states
how its per-read seq_error relates to the per-site
gt_error of rduckhts_roh(): the read-count
decode ends a run at a heterozygous site that the genotype decode at
gt_error = 30 passes.
rduckhts_somalier_relatedness() is faster for all
pairs, with the same rows: it checks the contents of each sketch once
instead of twice per pair. The bundled extension adds
duckhts_somalier_relatedness_all_pairs() for this.
The max_site_bytes argument of
rduckhts_roh() and rduckhts_roh_counts() is
documented exactly: each call applies its own limit to the bytes held by
all ROH decodes in the process.
rduckhts_roh() accepts long ancestry reference and
sample-proportion relations to calculate ancestry-weighted allele
frequencies, with optional clamping.
Added rduckhts_roh_counts(): runs of homozygosity
from a relation of allele read counts (for example Somalier site counts
from a BAM or CRAM), with a per-read sequencing error and an optional
contamination fraction (#318).
rduckhts_roh() and
rduckhts_roh_counts() gain max_sites and
max_site_bytes. The bundled extension now holds the sites
of each sample and chromosome in bounded native buffers (16 bytes per
site, 24 for read counts), and exceeding a limit is an error (#329).
Decoding many samples at once needs far less memory, and the runs are
unchanged.
An out-of-memory error during an ROH decode could abort the R session; the decode no longer keeps its site lists in DuckDB aggregate state.
rduckhts_roh_counts() rejects fractional read
counts, which were rounded, and accepts a counts relation in any order.
ROH arguments are checked even when the input has no record.
Direct SQL calls supplying genetic_map now use
duckhts_roh_map, duckhts_roh_af_table_map,
duckhts_roh_af_table_bcftools_map,
duckhts_roh_ancestry_map,
duckhts_roh_counts_map, or
duckhts_roh_counts_fitted_map; calls supplying
reference_table use
duckhts_depth_region_fraction_ref or
duckhts_depth_sex_fraction_ref; labelled table-index builds
use duckhts_cgranges_from_table_labelled. These replace the
corresponding extra-positional-argument calls under the base names. R
wrappers retain their signatures and select the SQL name
internally.
Bundled private SQL helpers are renamed:
duckdb_munge_preset_map ->
duckhts__munge_preset_map,
duckdb_munge_resolved_map ->
duckhts__munge_resolved_map,
duckhts_htslib_is_indel ->
duckhts__htslib_is_indel,
duckhts_json_path_key ->
duckhts__json_path_key,
duckhts_parquet_copy_sql ->
duckhts__parquet_copy_sql,
duckhts_parquet_ident_list ->
duckhts__parquet_ident_list,
duckhts_parquet_metadata_args ->
duckhts__parquet_metadata_args,
duckhts_quote_ident ->
duckhts__quote_ident, duckhts_quote_string
-> duckhts__quote_string, and
duckhts_raw_header_text ->
duckhts__raw_header_text. R wrapper names and arguments are
unchanged.
Bundled SQL macros duckdb_liftover,
duckdb_munge, and duckdb_munge_metal become
duckhts_liftover, duckhts_munge, and
duckhts_munge_metal, respectively, without aliases; R
wrapper names and arguments are unchanged.
Requires duckdb 1.5.0 or newer; loading the bundled extension on an older DuckDB now fails. Its stable C API target (v1.2.0) is an ABI floor and does not imply support for older DuckDB releases.
Consequence prediction has moved to DuckVEP and its R package,
Rduckvep (https://github.com/RGenomicsETL/DuckVEP).
rduckhts_haplotypes() and the bundled consequence fixtures
are removed; the bundled extension no longer carries the DuckVEP
functions. Load DuckVEP with Rduckvep::rduckvep_load(con)
on an Rduckhts connection for annotation. CSQ, ANN and BCSQ parsing in
rduckhts_bcf() is unchanged.
The bundled extension installs SQL macros into the database
catalog only for a writable in-memory default database.
rduckhts_connect() and rduckhts_load() instead
install connection-local macros automatically for database files,
including read-only files, without modifying the file catalog. Use
rduckhts_install_macros(con) for other DBI or pool
connections with the extension loaded; it is idempotent and follows a
caller transaction’s rollback.
duckhts_cgranges_from_query() and
duckhts_cgranges_overlaps_bulk() are removed from the
bundled extension. Use the table macro
SELECT * FROM duckhts_cgranges_from_table(name, table_name, chrom_col, start_col, end_col[, label_col])
over any table, view or TEMP table on your connection, and
SELECT ..., unnest(duckhts_cgranges_overlaps_list(...)) for
bulk probing. The README examples use them.
rduckhts_somalier_bam_counts() and
rduckhts_ancestry_bam() keep their arguments, but the
native reader now takes a local Parquet panel, so
panel_table is written to a scratch Parquet file for the
call; temporary tables and uncommitted rows are therefore accepted.
Remote panel_parquet paths are no longer read.
The Somalier panel now carries X/Y sites and its identity
changes. rduckhts_somalier_import_sites() keeps records on
the Somalier v0.3.4 X/Y aliases, numbered after every autosomal site,
and the panel, frequency and sketch digests differ from earlier versions
even for autosomal-only panels, so recompute persisted digests and
sketches. Counts from rduckhts_somalier_vcf_counts() and
rduckhts_somalier_bam_counts() include X/Y sites (which
ignore filter_policy), while sketches, relatedness, CHARR
and matched contamination use only the autosomal sites, with unchanged
numeric results.
New rduckhts_roh() finds runs of homozygosity in a
VCF/BCF with the bcftools roh model, returning
sample, chrom, start,
end, length, n_markers and
quality. Allele frequencies come from an INFO tag
(af_tag) or a table (af_table);
genetic_map (a chrom, pos,
cm table), rec_rate, hw_to_az,
az_to_hw, gt_error (GT-only emissions, no
FORMAT/PL needed) and samples are optional. Segments equal
bcftools roh on the bundled
roh_fixture.vcf.gz; the native
duckhts_roh_segments() kernel is available from SQL. Start
and end positions are identical; quality matches the one decimal
bcftools prints.
New regions_var argument on
rduckhts_geno() and rduckhts_bcf() names a
session variable holding typed 0-based half-open intervals for
read_geno(regions := ...) /
read_bcf(regions := ...). The variable must exist (an
unknown name would otherwise mean an unrestricted scan).
New rduckhts_geno_sites() fetches full-site
genotypes for requested (chrom, pos, ref, alt) alleles from
an indexed VCF/BCF on the caller connection: it validates the requests,
plans distinct positions with SET VARIABLE, runs one
read_geno() scan and joins exact alleles, returning every
request with record_index, full_alt,
alt_index, calls (including GP,
DS or HS via format_fields) and a
position-level match_status (matched,
allele_not_at_site, ref_mismatch,
absent). Nothing is trimmed, split, flipped or lifted over.
Its session variable and temporary request table are removed on exit,
including after an error.
New rduckhts_somalier_sex() reports X/Y dosage
evidence per sample: depth relative to autosomal depth, usable X site
and het/hom-alt counts, a Y signal, and an XX, XY or ambiguous call with
a status, following Somalier v0.3.4. It is review evidence, not a
diagnosis. y_gate = "sample" (default) avoids the
dependence of Somalier’s Y check on the rest of the batch;
"cohort" reproduces it.
rduckhts_somalier_find_sites() selects canonical,
provenance-bearing Somalier panels from typed population relations,
Parquet, or VCF/BCF with caller-visible interval and allele exclusions.
Interval relations use chrom, start,
end, so a view over read_bed() can be passed
directly. Equal AF scores retain input scan order and X/Y sites remain
unspaced by default, matching Somalier v0.3.4. The
tie_order = "lexical" and
sex_spacing = "enforced" options provide deterministic
lexical ties and X/Y minimum-distance spacing.
rduckhts_ancestry_proportions() estimates ancestry
proportions from allele-frequency or diploid-dosage relations against
keyed wide or long reference products, with matching audits and
correlation gates on unrounded coefficients.
reversed_variants counts every site read as
1 - f, including strand-flipped reversals;
flipped_variants counts every strand-flipped site.
Reference loci touching any input locus must be unique; matched loci
need finite loadings and frequencies in [0, 1]. Every sample returns a
row for every reference group, with NULL proportions and
no_matched_variants when nothing matches. Group identifiers
round-trip exactly; wide references can map columns to identifiers with
group_ids. Missing genotypes are dropped.
rduckhts_ancestry_geno() takes dense
read_geno call lists and requires the declaration
non_reference_only = FALSE; sparse call lists cannot supply
zero dosages.
rduckhts_ancestry_bam() counts reference-panel sites
in indexed BAM/CRAM, using allele-fraction or balance-rule genotype
frequencies, and applies the same proportion engine.
rduckhts_ancestry_panel() selects biallelic unambiguous
SNVs with complete, valid reference values and forward-strand alleles,
and publishes the panel only after it validates. When more sites are
eligible than max_sites, every contig keeps at least one
site (only the largest max_sites contigs when there are
more contigs than that) and the rest are shared in proportion to each
contig’s eligible sites minus that one, spread evenly along it.
Eligibility checks keep constant state per locus, so building a panel
from a full genome-wide reference needs a fraction of the memory it
did.
rduckhts_genbank() accepts attributes
to expose qualifier keys as VARCHAR columns that equal
attributes_map[key], computed only when selected. Tests
compare every named column with the map lookup on the bundled phiX174
record and cover the rejected names.
rduckhts_gff() and rduckhts_gtf()
accept attributes to create named attribute columns. The
bundled read_gff and read_gtf SQL readers also
support attributes := ['key', ...] with values matching
attributes_map.
rduckhts_bed() accepts error_policy to
skip short BED data lines or include diagnostic rows with their physical
line numbers and raw text in the bundled extension. The default still
raises an error.
The bundled CIGAR SQL functions accept an optional final
strict BOOLEAN. FALSE is the default; TRUE raises an error
for invalid input under the checked grammar. Errors name the function
and the 1-based packed-op index or text operation-start byte where
available. SQL NULL arguments and no-CIGAR sentinels retain their
per-function outcomes.
Add cigar_aligned_blocks(cigar, pos): extracts
contiguous aligned segments (M, =, X) from a CIGAR string or binary
array into a STRUCT of parallel lists (ref_start,
query_start, width). Use it to convert
alignments into genomic intervals for coverage, junction, and range
overlap analysis without custom SQL loops.
The bundled Emscripten extension accepts read-only
blob: URLs in webR. A browser File can be exposed as an
object URL, with a separate index_path URL for region
queries; callers must retain both URLs until queries finish. Chromium
148.0.7778.96 worker XHR tests found HEAD fails with status 0, ranged
GET returns 206 with size in Content-Range, ranges crossing
EOF are truncated, and out-of-range or revoked URLs fail with status 0.
Native builds are unchanged. In webR, blob: URLs are exempt
from Module.duckhtsWasmHttpConfig’s
enforceHostAllowlist, since they have no hostname and make
no network request.
A closed database file is released and can be reopened in the
same R process after cgranges or
rduckhts_somalier_bam_counts() have been used; the bundled
extension no longer keeps connections into the loading database. This
fixes reopening on Windows. A tinytest closes and reopens a file
database after using both.
rduckhts_somalier_vcf_counts() streams the VCF into
its panel join and keeps call payloads only for panel coordinates, so
peak memory follows the panel rather than the VCF length. Counts,
statuses and validation are unchanged.
The bundled DuckHTS extension loads on DuckDB 2.0; SQL lambdas
use the lambda x: syntax (DuckDB >= 1.3). It registers
native functions before dependent SQL macros and includes the underlying
diagnostic in initialization errors.
The bundled extension validates complete text and packed CIGARs
with checked lengths and consumed spans. CIGAR metrics and presence
checks return NULL for malformed suffixes or arithmetic outside BIGINT,
including a bad suffix after a matching operator. Unsupported requested
operators return NULL in both forms. Requested-operator case folding is
ASCII and locale-independent. Full-input checking adds validation work
to operator-presence calls. cigar_aligned_blocks grows
text-output child lists geometrically, including for CIGARs with long
runs of leading zeroes, while decoding each text CIGAR once.
The bundled extension scans tabix, GFF3 and GTF fields once up to the last projected field, including attributes when requested. Strict GFF3 validation still counts the entire line; SQL results and diagnostics are unchanged.
The bundled read_hts_header reports a scan error if
it cannot reserve or size its parsed key_values MAP
list.
Bundled bgzip, bgunzip, idxstats, and mosdepth outputs (including
CSI indexes) use exclusive creation when overwrite = FALSE;
a failed operation does not remove an output that the invocation did not
open. On POSIX, overwrite = TRUE replaces a symlink output
entry with a new file without writing through to its referent. On
Windows, overwrite preserves the target of a file symlink.
Table-reading wrappers preserve an existing table when
overwrite = TRUE fails validation or cannot read an input
file, and table-creating wrappers reject an existing table before
resolving input files when overwrite = FALSE.
The bundled extension reports consistent Somalier CHARR count errors whether public SQL validation or the native aggregate rejects the input first.
In webR, an explicit blob: index_path
is no longer copied into the in-memory filesystem and kept for the rest
of the session, and an empty blob: File reads as an empty
input (zero rows), matching a zero-byte file natively, instead of
failing to open.
The package is licensed GPL (>= 2) (previously declared GPL-3
while shipping an MIT licence file); the bundled DuckHTS extension
sources remain MIT, as listed in inst/COPYRIGHT.
The bundled function catalog documents
bam_bin_counts(include_unmapped := FALSE) and its synthetic
no-coordinate row, and names the type-probe macro’s parameter as
registered (candidate_type_name). Package tests compare
every catalog entry with a registered overload of the loaded extension:
kinds, scalar and aggregate argument counts, native table options, and
macro parameter names.
The package README footer shows contributor avatars and credits Ryan Ward / Nurture Bio for GenBank support, with links to upstream acknowledgements.
This package bundles DuckHTS 1.5.2.
FORMAT/AD extraction through packed sketches, relatedness,
CHARR, and directional matched contamination.rduckhts_somalier_import_sites() for converting an
already selected sites VCF/BCF into the typed panel and
population-frequency relation, with allele and frequency orientation
kept together.rduckhts_somalier_bam_counts() for panel-aligned
BAM/CRAM A/B/other extraction with explicit reference/index inputs,
read/base filters, overlap policy, resource limits and transport
settings. A committed panel relation or Parquet file is prepared once;
worker_count controls native DuckDB scan jobs with isolated
reader/reference state and bounded hash-indexed overlap names, while
decompression_threads separately controls htslib workers
per handle. Indexed requests use resolved reference IDs, preserving
valid contig names that contain region-expression punctuation. Returned
data frames are ordered by site index; caller TEMP panels and
uncommitted changes are not visible during retained-connection panel
preparation.@SQ lengths and accepts a non-colocated explicit FASTA
index without creating a default reference sidecar.rduckhts_somalier_vcf_counts() for complete
panel-aligned VCF/BCF FORMAT/AD evidence with exact
allele-slot mapping, explicit FILTER policy, unavailable-site rows and
source record/sample provenance.rduckhts_somalier_sketches() and
rduckhts_somalier_relatedness() for panel-checked count
evidence, reusable packed sketches and selected-pair statistics. Tables
or ordinary Parquet files can supply inputs and retain typed results;
unavailable count tuples remain distinct from measured zero counts.duckhts_somalier_verify_relatedness()
checks persisted pair results against their sealed sketches, including
identity, denominators, metrics and nullable numerical status.duckhts_somalier_verify_sketches() rebuilds
sketches from retained evidence and detects raw-count or availability
changes even when the classified genotype is unchanged. Invalid retained
classification settings return false instead of becoming
rebuild-aggregate settings.rduckhts_somalier_charr() and
rduckhts_somalier_matched_contamination() over measured
count channels and aligned population-B allele frequencies. They retain
numerical status and usable-site denominators, and reject changed
panel/evidence identity. The matched method preserves receiver/anchor
direction and evaluates only requested ordered pairs without a
pair-sized native workspace. CHARR results are stable across parallel
aggregate reduction order.max_threshold_work limit shared
across distinct measured depths for both contamination wrappers;
exhaustion errors without partial results. Sketches constructed with
IEEE negative-zero balance settings round-trip.rduckhts_genbank() for GenBank flat-file features
in rduckhts_gff()’s column shape, with an optional parsed
qualifier MAP. join()/order()
locations give one row per segment in biological order with the CDS
phase carried across segments, complement(...) sets strand
-, Parent links to the gene sharing a
/locus_tag in any file order, repeated qualifiers
comma-join, and the record-level source feature is dropped.
Truncated or malformed records, including a FEATURES table not followed
by a sequence section, are errors naming the feature and line.rduckhts_genbank_to_fasta() to write each record’s
ORIGIN sequence as FASTA under the same name reported as
seqname. Bgzipped records are read in place through
htslib’s hFILE layer. The output is renamed into place only after a
clean read, so an existing file is never lost. output_path,
line_width and overwrite are validated before
any SQL runs.rduckhts_geno() for typed arbitrary-ploidy GT/PS
calls and rduckhts_bcf_samples() for original-header sample
identities. Sample selection retains original sample indices; sparse
scans preserve variant records while omitting calls without a called
ALT.format_fields = c("AD", "DP", "GQ"), including values on
records without GT. Header types/cardinalities and missing elements
remain intact. Single-file and multi-file BCF wrappers also support
per-call sample selection.include_filter = TRUE to retain the same physical
VCF/BCF record’s FILTER beside typed calls; PASS, unapplied dot and
named failures remain distinguishable without a separate reader
join.read_bcf() rather than reporting it as
[PASS].raw_gt = TRUE to retain exact original VCF genotype
text, including leading phase markers and literal missing alleles. BCF
rejects this option because it does not store the original text.decode_error_policy rather than being silently
truncated.rduckhts_haplotypes() for shared CDS/protein paths
with complete carrier and contributor evidence, coding blocks, local
coding status and aligned CDS/protein differences. Per-call capacities
bound native replay, alignment and HGVS storage; exhaustion raises
errors. DuckDB input/output memory is outside the native workspace
limit.input_mode = "source_records" for original GT spelling and
complete ALT lists. Preserve missing/unphased, conditional, overwritten
and noncoding contributors. Known REF slots remain distinct from
selected skipped ALTs. Model/REF failures retain unavailable sequence
rather than fabricated output.hgvs = TRUE for supported protein HGVS with
explicit unsupported states and max_hgvs_reference_bytes.
Frame restoration does not erase physical edits or downstream
contributors; stop_in_displaced_frame is not a protein
rescue prediction. nominal_length_diff differs from rebuilt
CDS length when source replacement clipping changes the applied
span.duckvep_transcript_projection(),
duckvep_breakend_geometry() and
duckvep_repeat_alleles() through SQL/DBI. These provide
typed transcript display, raw BND preparation and bounded paired
exact-repeat preparation. Correct the paired-BND truncation/intergenic
union. BND replacement text is not inserted-only sequence; repeat
summaries do not establish an exact allele. Transcript display shares
decoded reference sequences and extracts local codon spans to reduce
per-event memory use.Haplotype replay remains alpha: local coding facts and supported protein HGVS are not complete whole-haplotype SO/IMPACT/NMD, compound DNA HGVS, structural composition or altered-splicing prediction. Strict phase policy does not select a different HGVS nomenclature standard. Pinned Ensembl VEP 116 is the compatibility target, with unresolved cases and policy differences preserved. Circular annotation and mapped MANE-on-GRCh37 enrichment are not provided.
FILE_OFFSET only for BGZF BAM, as the
virtual position after each record. SAM, CRAM and uncompressed BAM
return missing values; use explicit SQL ordering when required.rduckhts_fasta_nuc()
reports reference-fetch errors instead of omitting intervals.Built field against the bundled extension metadata.read_bcf_v2 and
read_bcf_appender. Use read_bcf() or supported
R wrappers with projection, explicit sequential mode where needed, and
DuckDB CTAS/INSERT/COPY for materialization. Experimental-only arguments
are not transferred.duckhts_build() manual compiler.
Package configure scripts own compilation; maintainer dependency
installation is explicit.post_cds_bases are removed. Alpha coding blocks use
event_indices for physical-edit provenance, replacing the
count-only field.NeedsCompilation: yes because package
configuration compiles target-specific DuckHTS and htslib artifacts
outside the conventional R src/ directory. Binary
repositories can no longer treat an x86_64 package as
architecture-independent and reuse it on Linux ARM64. Native Linux ARM64
coverage now builds and audits a clean source tarball, installs it,
loads the linux_arm64 extension, and validates the bundled
htslib receiptwindows_arm64_mingw extension, and validates the bundled
htslib receiptrduckhts_connect() as the single package-owned
connection path. It explicitly permits the locally compiled bundled
extension, uses temporary DuckDB extension/secret storage when
supported, and disables implicit known- extension installation and
loading. Package examples and extension-dependent tinytests now use this
path, including on CRAN’s Linux libc++ builds where the
duckdb package otherwise disables extension loading.
duckhts_load(NULL) delegates to the same connection
contract, while rduckhts_load() remains available for
caller-owned connections. File-backed connections reject a reused DuckDB
driver, whose creation policy cannot be changed, and release a newly
owned driver if connection or extension loading failsbcftools_norm_row out-of-memory cleanup
correction so a failed internal allocation cannot use an indeterminate
allele-buffer countbwCleanup
definition<NON_REF>, bare *, and .
ALT rows produce no annotations; literal ALTs from mixed records remain
small variants; and <*> produces the generic coding
or retained start/stop consequence with empty HGVS. Record-level
INFO/END remains provenance for literal ALTs and
<*>. The package docs and tests also warn that VEP
116’s literal allele-length handling can emit an ablation term when a
long-REF <*> row completely contains a featureRtinycc::tcc_call_symbol() APIs, and bundle bounds-checked
BAQ tag renaming so GCC 11 package builds do not emit the former
zero-size-region warningRtinycc downstream-linking test dependency before strict
package checksduckvep_allele_geometry(...), which returns the
raw VCF span, VEP feature span, minimized edit span, and insertion point
used by the consequence engine. The bundled cumulative consequence/HGVS
path now reuses one projected transcript edit and one coding fact object
per event/transcript pair, with release-specific VEP-116 HGVS behavior
selected through a named compatibility policy rather than
formatter-local constants. Add DBI coverage for the geometry contract
and regenerate the package function catalogduckhts_bcftools_norm(...)
derived-query macro accept the VARCHAR[] ALT column
returned by read_bcf(). The scalar and list forms of
duckhts_alt_to_list(...) and
bcftools_norm_row(...) are now registered as DuckDB
overload sets, matching the package’s documented normalization
workflowBUFFER_TOO_SMALL
before changing the caller buffer when that intermediate sequence
exceeds its capacityrduckhts_htslib_config() as the versioned
installed-package contract for downstream C/C++ packages linking to the
bundled htslib. It resolves exact headers, shared/static libraries,
static dependencies, loader flags, plugins and compiled features from
the installed package, and rejects receipt/header/runtime version
disagreement through the extension-owned htslib diagnostics. Add thin
rduckhts_htslib_info() and
rduckhts_htslib_version() helpers plus an in-memory
downstream compile/link test, built through Rtinycc with active-SDK
system headers on macOS, that opens bundled BAM, CRAM, BCF, and VCF
files. When link is omitted, select the shared or static
contract chosen during package configuration. The compatibility
htslib_rpath() helper follows that configured contract, so
static-only installations return an empty loader path instead of
validating a missing shared libraryread_bigwig(...) table
function and rduckhts_bigwig() materialization/view helper.
Region vectors use htslib’s one-based inclusive syntax while returned
intervals retain stored zero-based, half-open coordinates. Add the
pinned upstream libBigWig fixture, DBI tests, package examples, Devon
Ryan’s copyright-holder attribution, and the vendored MIT license.
Oversized chromosome tables, malformed chromosome-tree IDs, truncated
zoom headers, unreadable R-tree indexes, oversized block allocations,
and truncated data blocks return reader errors before indexed storage or
interval decodingincomplete_terminal_codon_variant&inframe_insertion&stop_gained
together with c.280_281insAGT and the apparently
contradictory p.Ter94= protein rendering. A bundled DBI
regression exercises both rare states through the public
duckvep_annotate(..., rich = TRUE, hgvs = TRUE) relation;
its terminal-codon case uses an exact remapped GRCh38
transcript/reference fixture with enough downstream sequence to pin the
one-base VEP 3-prime shiftTer?, while
later frameshifts can repeat VEP’s late original-coordinate,
original-allele stop search; and terminal complete-feature clamping can
render a duplicated transcript base even when minimization moved the
differing base outside the transcript. Bundled SQL behavior is pinned by
exact HGVSc/HGVSp witnessesduckvep_annotate(..., rich := TRUE) so DBI
workflows can request VEP-facing consequence, IMPACT, region,
amino-acid, NMD, overlap-object, and explicit DuckHTS audit text without
losing the compact numeric fields.
rich := TRUE, hgvs := TRUE returns those fields and
independent-event HGVS from one fused native pass. Add DBI coverage for
the public rich/HGVS schema and regenerate the package function catalog
from the public SQL authorityduckvep_annotate(...) relation reject
conflicting supported symbolic ALT and explicit structural-type values,
and treat explicit hgvs := NULL as the documented
FALSE default instead of filtering small variants from both
annotation lanesduckvep_annotate(events_table, model_name, hgvs := false, ...)
relation and generated duckvep_so_terms() lookup. DBI
callers can keep one narrow mixed small/SV/BND event table, request
independent-event HGVS without changing schemas, decode consequence
masks after filtering, and join selected rows back by event identity.
Add a bundled public-surface tinytest and a deterministic
model-to-annotation example in the rendered package READMEduckhts_contig_key(...) for DBI workflows that
need a conservative join key across chr-prefixed VCF/dbSNP
contigs and Ensembl-style region names. The helper normalizes
mitochondrial M/MT to MT and
uppercase X/Y but deliberately does not guess
numeric sex chromosomes, accessions, patches, or alternate lociannotation_index
values across DuckDB vector and disjoint ordered partition starts. The
resident model remains immutable and shared. Tests also cover zero-,
10,000-, and 50,000-base transcript distances so the bundled behavior is
not specialized to the default distanceduckvep_annotate_hgvs(...) for DBI workflows
that need compact independent-event consequence rows together with
transcript c./n. and default-VEP protein
p. HGVS suffixes, the applied 3-prime shift, and explicit
supported/unresolved/not-applicable states. Bundled
duckvep_model_load(...) can bind an existing indexed
reference FASTA through an exact ordinal/name/length relation; it does
not create an index and retains open read descriptors for the validated
FASTA, .fai, and optional .gzi. Linux workers
reopen those descriptors, Windows keeps a resolved source under
deny-write sharing, and other POSIX workers use independent
resolved-source handles with identity checks rather than sharing
/dev/fd seek state. Annotation workers own separate faidx
handles, reuse contained sorted reference windows, and reject detectable
in-place source mutation around a fetch. Explicit NULL optional model
queries and reference parameters behave like omission. Bundled tests
cover reference-backed substitution and insertion rendering through the
public DBI surface. Transcript rows admitted only by an
upstream/downstream distance report HGVS not_applicable,
matching VEP’s absent HGVSc, rather than an unresolved projection.
Literal exonic SNP HGVSc also retains VEP 116’s phase-aware CDS-start
fast path without shifting intronic SNP, indel, or multi-base feature
coordinates. Protein HGVS reports not_applicable when the
VEP feature has a leading or trailing genomic-to-peptide mapper Gap.
Reference failures use VEP’s cached complete-feature coding predicate,
so a 5-prime-UTR insertion that could shift into CDS remains
protein-unresolved rather than falsely not applicable. A bundled model
without FASTA now reports missing_reference when retained
uploaded REF padding or an anchor cannot be checked by the prepared CDS,
instead of validating only the minimized differing REF. The bundled
reference path keeps VEP’s exact +/-1000 shift slice separate from
complete uploaded-REF validation and adjacent duplication-source lookup,
so retained padding does not change the shift and copied sources longer
than 1000 bases still render as dup. Endpoint-overlapping
transcript edits retain VEP’s clipped transcript-slice coordinates, and
protein replay reproduces VEP’s one/two-base alternate-CDS trimming
assignment bug before appending the 3-prime UTR. Bundled HGVS execution
now reuses the kernel-prepared model, derives each reference first stop
once, defers CDS projection until protein HGVS needs it, scans a virtual
single-edit CDS instead of rebuilding the complete alternate sequence,
and renders through a reusable worker buffer. The bundled consequence
sidecar shortcut remains fail-closed for length-changing splice
overlaps: without a positive frameshift fact, those edits run the
complete peptide delta so their VEP-compatible frameshift HGVSp is
retainedtranscript_ablation semantics as
symbolic deletions. Equal-length containing alleles also preserve VEP
116’s otherwise-empty endpoint UTR terms without leaking an internal
unknown-coding fact into coding_sequence_variant; bundled
DBI tests pin both public resultsintergenic_variant beside
mate-derived feature_truncation. Add bundled DBI
regressions at zero and wider caller distancesSTR/TANDEM_REPEAT/CNV:TR spans
and preserves structural tandem-repeat identity while matching VEP 116’s
tandem-duplication consequence predicates. Bundled paired-BND annotation
now returns RegulatoryFeature and MotifFeature rows found at either
endpoint, once per feature, in addition to transcript consequences;
local exact hits keep the base object term, mate-only exact hits use
VEP’s generic HIGH-impact feature_truncation, and a shifted
local point on the same contig but outside and within 5000 bases adds
intergenic_variant to that mate-discovered object. The
local base term wins when both points hit one object exactly, and mixed
transcript/regulation batches preserve one result list per input event.
A caller-selected transcript distance above 5000 does not widen VEP’s
separate fixed structural-breakend allele-admission cap. Add DBI
tinytests for these surfaces and update generated function documentation
Structural adapters consume nominal POS/END
geometry, matching VEP 116’s registered consequence predicates; callers
retain CIPOS/CIEND, inserted sequence, and raw
repeat metadata beside the result for provenance, later HGVS, and
round-trip renderingread_fastq(...) scans so FASTQ
headers longer than the BAM query-name limit remain usable when
NAME / PAIR_ID are not requested and no
paired-file comparison needs them; name-producing scans retain the
htslib-compatible limit. An explicit bundled
duckhts_fastq_qc(..., max_cycles) value below 128 now also
caps the initial per-group cycle allocationduckhts_fastq_qc(sequence, quality [, max_cycles]) for DBI
workflows that need exact global and per-cycle FASTQ quality statistics
without expanding one row per base; the bundled scalar, AVX2, ARM NEON,
and wasm SIMD128 implementations share one dispatch contract, and
tinytests compare forced scalar with automatic selection over grouped
and malformed inputsread_fastq(...) scans by parsing
FASTQ directly over htslib transport into projected DBI columns instead
of round-tripping through temporary BAM records; multiline records,
htslib-style query names, paired/interleaved checks, string and packed
nt16/Phred output, quality conversion, and count behavior are retained,
while truncated quality blocks now surface as R/DBI errors. Add bundled
multiline, packed-output, and truncation tinytestsduckhts_build(...) to obtain the DuckVEP kernel
source list from the package’s shared source-list helper; the fallback
no longer references a variable local to
duckhts_bootstrap(...)read_bcf(...),
read_bcf_v2(...), read_gff(...),
read_gtf(...), read_tabix(...), and
read_bcf_appender(...) calls now return each matching
record once. The appender partitions region_threads work by
primary contig while retaining one native iterator for that contig’s
complete interval set, so thread count no longer changes its target
schema or row multiset; remove the thread-only
duckhts_region_idx column and retain opt-in
FILE_OFFSET for restoring file order. Bundled
read_bam(...) already uses the equivalent deduplicating
alignment iterator. Keep BCF work claiming iterative and bounded to 16
workers, with bundled regressions over 256 leading header-only contigs
and empty requested contigs. Build htslib as GNU C17 so GCC 16 does not
apply its GNU C23 default to this pre-C23 dependency. Add bundled
tinytests and a Fedora GCC 16 R-devel package check matching CRAN’s
Fedora compiler environmentduckvep_model_load(...) accepts the compact feature
projection, while duckvep_annotate(...), its compact form,
and exact structural adapters return typed transcript or
regulation/motif rows from one call; cold funcgen metadata remains
joinable by feature ordinal. The existing eight-argument model-receipt
call remains valid and accepts regulation_features_table as an optional
named relation. Bundled model receipts validate, count, and hash feature
geometry, exclude the same EMAR rows as VEP 116, and the offline fixture
includes exact core/funcgen provenance plus three real MotifFeature rows
on KI270395.1duckvep_annotate_breakend(...) and
duckvep_annotate_breakend_compact(...) functions. DBI
callers provide both raw VCF loci in one row and retain raw ALT, bracket
orientation, event identity, and provenance as ordinary columns; the
bundled engine queries transcripts around both endpoints and returns the
VEP-116 consequence-set union once per transcript. Mate-only
consequences have NULL rich region / zero compact region maskduckvep_annotate_sv(...) and
duckvep_annotate_sv_compact(...); DBI callers provide
one-based inclusive spans for DEL/DUP/TDUP/INV/CNV/UNKNOWN, or
start = end = P for an INS after reference base P, plus
explicit copy direction. Contradictory metadata fails as an error. The
bundled consequence mask retains stable assignments for all 41 terms in
VEP 116’s registry, including regulatory-region and
transcription-factor-binding-site terms1,0 range immediately before CDS base 1; the cached
first-101-CDS-base result is reused only when both spans are
identicalduckvep_annotate(...) and
duckvep_annotate_compact(...) now accept VEP-style separate
upstream and downstream distances in addition to the existing shared
distance; omitted values retain the 5,000-base default and zero disables
that directionnon_coding_transcript_exon_variant instead of a false
mature_miRNA_variant; the regression covers both transcript
strandsduckvep_model_load(...) and are applied to the reference
peptide during annotation; unsupported edit shapes remain fail-closed.
Numeric-only attribute columns inferred as integers by dump staging are
accepted at the importer boundary. Model receipts hash and count
mature-miRNA segments and peptide edits, and the bundled GRCh38 and
GRCh37 fixtures exercise real mitochondrial codon-table and peptide-edit
behaviorduckvep_ensembl_transcripts(...) now applies VEP’s
core-transcript selection and projects mature-miRNA cDNA attributes into
genomic exon segments; duckvep_model_load(...) accepts
those segments through optional mature_mirna_query, and
annotation returns mature_miRNA_variant with VEP’s
precedence over generic non-coding exon terms. The bundled engine also
fixes insertion placement at transcript/exon edges, VEP’s empty-UTR
endpoint behavior, and coding edits on CDS_END_NF
transcripts with a one- or two-base partial terminal codon; DBI tests
cover the complete Ensembl-import-to-resident-model pathduckvep_annotate_compact(...), returning the
same DuckVEP transcript consequences as numeric SO/region masks and
stable status, reason, amino-acid, and NMD codes so DBI workflows can
filter and join before rendering strings. The bundled rich annotation
path also caches DuckDB vector validity and string lengths per chunk;
SQL and tinytests check numeric-code semantics and parity with
duckvep_annotate(...)intergenic_variant;
duckvep_model_load(...) accepts the complete 13-column
projection while retaining its 11/12-column compatibility, and older
models return missing_transcript_flank rather than
guessingduckvep_annotate(...) now matches VEP 116 for
equal-length features crossing from 5-prime UTR into the start codon:
changing the start returns start_lost without a
co-occurring stop term, while preserving ATG returns
start_retained_variant even when later coding bases in the
same uploaded feature change; DBI regressions pin each stateduckvep_annotate(...) now matches VEP 116 when
unchanged bases retained in an equal-length uploaded feature widen the
peptide-predicate window beyond the trimmed sequence edit, including
representation-dependent
start_lost&start_retained_variant,
stop_retained_variant, missense_variant, and
stop_gained results pinned by paired DBI regressions;
retained REF mismatches and ambiguous widened windows now return
explicit unresolved reasons instead of retrying the smaller editcoding_sequence_variant rather than a stop or missense
consequence derived from a smaller trimmed editduckvep_annotate(...) reproduce VEP 116
when ALT-only mismatch bases extend from an exonic REF-shaped feature
into an intron, including VEP’s three-base interval-tree cache boundary
and co-emission with coding and splice termsduckvep_annotate(...) classify
length-changing coding edits from VEP 116’s codon-local predicate inputs
rather than whole-protein or net-length shortcuts. This fixes
stop-gained insertions whose preserved peptide flank lies after the new
stop, and delins that change length by a complete codon but are
protein_altering_variant rather than in-frame
insertion/deletionduckvep_annotate(...) match VEP 116’s
terminal-stop insertion predicate order, including non-modulo-three
insertions that VEP reports as inframe_insertion together
with coding_sequence_variant, stop_lost, or
stop_retained_variant; unsupported contexts remain
explicitduckvep_annotate(...) now keeps uploaded VCF
geometry, VEP parser feature geometry, and the semantic sequence edit
distinct, fixing candidate, splice, start-codon, and terminal-stop
consequences for padded and multi-base allelesduckvep_annotate(...) now predicts whether
eligible stop-gained, frameshift, splice-donor, and splice-acceptor
consequences trigger or escape nonsense-mediated decay under the pinned
VEP Plugins release/116 policy, with separate boolean escape reasons and
explicit unresolved state when coding placement is unavailable. This
remains distinct from NMD_transcript_variant, which
identifies an already curated nonsense-mediated-decay transcript
biotyperead_bcf() and read_bcf_v2() now
recognize the Format=... CSQ schema spelling in Ensembl
variation release VCFs and expose typed VEP_* columns
through DBI queriesduckvep_model_receipt(...) provenance column names
and add roughly 116 KiB of offline Ensembl-116 GRCh38 and
GRCh37/GENCODE-19 fixtures to package tests, covering real MANE,
ordinary coding, and mitochondrial missing-sequence behavior without
network access during CRAN builds or checksduckvep_ensembl_regions(...),
duckvep_ensembl_transcripts(...), and
duckvep_model_receipt(...), allowing DBI workflows to
prepare a validated, provenance-hashed resident DuckVEP model directly
from Ensembl core relations and matching tiled FASTA sequence.
Unsupported Ensembl RNA/peptide edits keep their model flags but return
explicit missing-sequence state instead of ordinary coding predictions,
including _rna_edit records carried by either transcript or
translation attributes; bundled SQL and tinytests cover both strands,
nested exon projection, receipt generation, and resident loading-1 in bundled sequence-backed
models when translation begins after 5-prime UTR within that exon,
matching VEP’s zero-prefix interpretation while retaining exact
prepared-CDS validationstop_lost
or stop_retained_variant instead of a false frameshift. The
resident transcript query accepts an optional twelfth
post_cds_bases BLOB with up to three transcript-oriented
bases; when a terminal deletion needs absent tail sequence, the result
is explicitly unresolved as missing_transcript_tailduckvep_model_load(...) calls
report no_feature_in_loaded_model as unresolved; the named
transcript_coverage_complete := TRUE parameter requires
contig lengths before returning supported
intergenic_variant, and rejects coordinates beyond those
lengthsduckvep_annotate(...) handling of padded
small variants: whole-allele padding no longer turns a one-base
substitution into an MNV, and VCF position-1 insertions/deletions with a
following padding base are acceptedNMD_transcript_variant for variants inside transcripts
imported with the nonsense_mediated_decay biotypeduckvep_model_load(...)
loads and validates one of several named resident transcript models from
committed DuckDB relations, duckvep_annotate(...) returns
explicit per-transcript consequence rows for biallelic small variants,
and duckvep_model_drop(...) releases a model. Add an
end-to-end DBI tinytest using the bundled extension and a
sequence-backed coding transcriptbam_nt16_counts and
nt16_gc_counts logical kernels on AArch64 with NEON, so
Apple Silicon and other arm64 package builds no longer fall back to
scalar for BAM-bin GC counting or nt16 seq_gc_content(...);
add conditional tinytests that require every reported complete backend
to own all three sequence-kernel slots and match scalar results.
Emscripten package builds now keep bam_bin_counts(...) on
its synchronous no-index path by disabling unavailable htslib worker
threadscigar_* helpers
(cigar_has_soft_clip, cigar_has_hard_clip,
cigar_left_soft_clip, cigar_right_soft_clip,
cigar_query_length,
cigar_aligned_query_length,
cigar_reference_length, cigar_has_op) are
overloaded to accept a UINTEGER[] binary CIGAR from
read_bam(cigar_representation = 'binary'), and bundled
seq_hash_2bit(...) is overloaded to accept a
UTINYINT[] of htslib nt16 codes from
read_bam(sequence_encoding = 'nt16'). Both are
bit-identical to the text path, so DBI pipelines analyze the
projection-pushed binary columns without decoding to text; add bundled
tinytest coveragerduckhts_simd_kernel_info() now reports an
additional nt16_gc_counts logical kernel, and the bundled
nt16 seq_gc_content(...) overload is routed through the
SIMD dispatch framework; GC results are unchanged (a scalar reference
backend is the correctness oracle), and forced scalar vs
auto agree (bundled tinytest coverage added)seq_revcomp(...) and
seq_canonical(...) are overloaded to accept BAM nt16
sequences: in addition to text they take the UTINYINT[] of
htslib nt16 codes from read_bam(sequence_encoding = 'nt16')
and return UTINYINT[], so DBI pipelines can
reverse-complement and canonicalize directly on the nt16 column.
Non-ACGTN codes yield NULL as in the text path; results are
bit-identical to the text path after decoding; add bundled tinytest
coverageseq_gc_content(...) is overloaded to accept BAM
nt16 sequences: in addition to a text sequence it now takes the
UTINYINT[] of htslib nt16 codes returned by
read_bam(sequence_encoding = 'nt16'), so DBI pipelines can
compute GC directly on the projection-pushed BAM column without decoding
to text. The nt16 result is bit-identical to the text path; add bundled
tinytest coverageread_bcf(...), read_bcf_v2(...),
and read_bcf_appender(...) gain
decode_error_policy := 'null'|'warn'|'error' for corrupt
BCF FORMAT/INFO header-vs-payload type clashes; the default
null policy materializes NULLs, warn emits a
DuckHTS warning and materializes NULLs, and error raises a
DuckDB/R error. FORMAT and INFO decode now preflight under every policy
(including the default null), so corrupt inputs neither
trigger htslib exit(1) termination nor leak raw bytes; an
INFO field whose header claims Type=String over a numeric
payload previously returned truncated garbage under the default policy
and now materializes NULL. Add bundled corrupt-BCF fixtures (including a
reverse String-header/numeric-payload clash) plus tinytest coverage
while keeping valid mixed-ploidy Number=G FORMAT records
acceptedrduckhts_simd_kernel_info() now reports an
additional bam_nt16_counts logical kernel, and bundled
bam_bin_counts(...) GC base counting is routed through the
SIMD dispatch framework; GC results are unchanged (a scalar reference
backend is the correctness oracle)read_bcf(...), projected read_bcf_v2(...)
scans, and non-parallel read_bcf_appender(...) as DuckDB/R
errors instead of treating htslib parse/read failures as EOF; run
bundled read_bcf_appender(...) writes in an internal
transaction so malformed input rolls back target-table side effects; add
a malformed-POS fixture plus tinytest coverageread_bcf_v2(...) table
function for DBI users, preserving read_bcf(...) schema
compatibility while adding sample pushdown, INFO/FORMAT/VEP field
filters, projection-aware VCF unpacking, persistent decode caches, and
count-only shortcuts for benchmarkingread_bcf_appender(...)
benchmark helper for DBI users; at introduction, its single-stream path
was transactional while the region_threads > 1 path was
explicitly best-effort and lacked rollback (the current development
entry above records its later transactional, thread-invariant
contract)hts_region_union_query(...)
scalar macro for DBI users, which generates a
UNION ALL BY NAME query string reading one HTS file through
separate per-region scans; the generated query does not deduplicate
boundary-spanning recordsread_bcf_appender(...) so a region query
matching no records still creates and commits the empty target table
instead of rolling back its DROP/CREATE TABLE,
which previously could leave a stale prior table under
overwrite := true or never create the targetadditional_csq_column_types so invalid CSQ/ANN/BCSQ type
override rules fail as DuckDB/R errors instead of proceeding with
ambiguous parsed-annotation typingrduckhts_load() now expose DuckDB runtime
type-support probes for VARIANT and
GEOMETRYrduckhts_bcftools_norm() /
duckhts_bcftools_norm(...) site-preserving table-macro
query shape by removing the extra correlated scalar LATERAL
subquery around bcftools_norm_row(...), eliminating the
site-preserving LEFT_DELIM_JOIN plan overhead while
preserving split-mode ALT row semantics and caller columns whose names
collide with DuckHTS helper-column names used internally by earlier
macro forms; add tinytest coverage for DuckDB’s suffixed behavior when
callers already have normalized-output column namesscan_mode = "auto"|"sequential"
controls through the R wrappers and multi-file helpers for
read_bcf, read_bam, read_fasta,
read_fastq, read_bed, read_gff,
read_gtf, and read_tabix, so callers can force
full-file streaming/counting instead of index-backed count or parallel
scan paths where applicable; sequential mode is rejected for region
queriesbcftools_norm_row(...) /
rduckhts_bcftools_norm() for already-normalized plain ACGTN
allele rows by skipping kstring left-realignment setup when trim
predicates prove the row is unchanged; avoid per-row FASTA path
duplication after the vector-local cache is established, reuse larger
bounded per-thread reference windows, and document/defensively serialize
htslib FASTA fetches while keeping normalization reference caches
thread-local to avoid the faidx cache race class fixed in
https://github.com/RGenomicsETL/duckhts/issues/17 /
https://github.com/RGenomicsETL/duckhts/pull/18rduckhts_bcftools_norm() /
duckhts_bcftools_norm(...) gVCF-aware for vt/vcfnorm-style
row normalization: <NON_REF> and
<*> reference-block alleles now pass through with
GVCFReferenceBlock, and mixed real-plus-gVCF-symbolic
alleles normalize the real alleles while preserving symbolic alleles and
caller-supplied reference-block END in site-preserving
output; mixed * plus real alleles now follows the same
ignored-symbolic path, while *-only rows remain
SpanningDeletion; bundled phased GT/PL/GP/DS/PS FORMAT
fixtures, including haploid/triploid/tetraploid Number=G
cardinality cases, and tinytests pin phase-separator preservation
through read_bcf(...)rduckhts_bcf_convert_parquet(),
rduckhts_bam_convert_parquet(),
rduckhts_gff_convert_parquet(), and
rduckhts_tabix_convert_parquet() around the bundled
extension SQL builders duckhts_*_convert_parquet_sql(...);
these convert DuckHTS scans to Parquet with DuckHTS write-format
metadata, preserved raw headers, optional corrected header text,
SQL-filter provenance, selected-column/partition metadata, arbitrary
user metadata via R named lists/extension
metadata := map(...), optional caller-managed JSON-file
metadata when DuckDB’s json extension is available, and
partitioned-output support for DuckLake-style registration of premade
Parquet files#CHROM/sample header line in
bundled read_hts_header(..., mode := 'raw'), so Parquet
metadata written from VCF/BCF inputs has the complete header needed for
future VCF/BCF regenerationrduckhts_simd_kernel_info(), keeping R wrappers thin while
reporting one row per logical kernel and preserving backend-agnostic
SQL/R conformance tests for seq_gc_content(...)selectable versus
available diagnostics in generated docs, and preserve ASCII
SQL quotes for the rendered
duckhts_simd_set_backend('auto'|'scalar'|backend) catalog
callHAVE_* macro guards from all
bundled SIMD backend translation units; compile-time gate is now
defined(__x86_64__) && (defined(__GNUC__) || defined(__clang__))
for x86 backends, available without autoconf; runtime dispatch and
scalar fallback behavior are unchanged; add scalar-vs-auto backend R
correctness tests for rduckhts_simd_set_backend() /
seq_gc_content(...) covering GC=0/0.5/1.0, embedded-N
calling, and soft-masked lowercase bases.validate_simd_backend() R helper from
SIMD wrapper functions; backend-name normalization and validation now
belong to the extension, while the R wrappers only enforce that
backend is a single non-missing character stringrduckhts_simd_backend(),
rduckhts_simd_requested_backend(),
rduckhts_simd_backend_available(), and
rduckhts_simd_set_backend(), route bundled
seq_gc_content(...) through the new eager
scalar/optional-AVX2 runtime dispatch scaffold while preserving scalar
fallback behavior on ARM, wasm, and scalar-only builds, add
runtime-gated AVX-512, ARM NEON, and wasm SIMD128 backend translation
units where compiler-supported, keep the manual
duckhts_build() rebuild path wired to the SIMD sources, and
add README examples for the scalar/auto SIMD flowrduckhts_liftover() /
bcftools_liftover(...) FASTA contig alias handling during
source/destination reference validation and sequence fetches, and align
bundled spanning-deletion * allele handling with upstream
bcftools +liftover: inputs such as 23,
24, 26, X, Y,
MT, and chr* aliases now resolve through the
same canonical path, avoiding spurious SourceRefMismatch
rejects for X/Y/MT indels when the bundled chain names and FASTA names
differ only by canonical aliasing; bundled *-allele rows
now follow upstream swap/ref-add semantics instead of taking the
symbolic short-circuit path, full-file GIAB conformance against
installed bcftools +liftover is now exact, and bundled
SQL/tinytest coverage now pins the 23 -> chrX,
SWAP=2, and SWAP=-1 regressionsrduckhts_load() sessions for both the
bcftools-style and raw upstream numeric surfaces:
variantkey(...) now matches bcftools %VKX /
+add-variantkey on 1-based VCF rows,
large/ambiguous/symbolic alleles keep the official hashed nonreversible
mode, regionkey(...) adds 0-based half-open span keys plus
overlap helpers, bundled tinytests pin reversible and hashed cases, and
the package README now includes concrete DBI examples for VariantKey /
RegionKey usagerduckhts_bcftools_norm(..., split_multiallelic = TRUE) row
preservation for ref-only and empty-ALT inputs: rows with
ALT='.', NULL ALT values, empty ALT lists, or NULL ALT list
elements no longer disappear from split-mode DBI results, bundled
tinytest coverage now pins the expected RefOnly /
NullInput statuses and alt_index behavior,
bundled rduckhts_bcf() / rduckhts_bcf_multi()
now expose decompression_threads = 0 for explicit htslib
worker-thread control on bgzipped VCF/BCF reads, and the package README
now includes a concrete normalization examplerduckhts_fasta_index() now returns the generated
.fai path when index_path = NULL instead of an
empty string, bundled regression coverage now also pins default-path
returns for BGZF compression/decompression and BAM/BCF/tabix index
builders, and the rduckhts_bgzip() /
rduckhts_bgunzip() wrappers now correctly propagate
keep = FALSE instead of silently falling back to the
extension default keep := TRUErduckhts_bcftools_norm() and bundle
bcftools_norm_row(...) /
duckhts_bcftools_norm(...) for bcftools/vt-style
FASTA-backed variant normalization from DBI queries: ALT inputs may be
either comma-delimited VARCHAR or VARCHAR[],
the bundled result appends pos_normed,
end_pos_normed, ref_normed,
alt_normed, normed, and
norm_status, split mode emits one row per ALT with
alt_index, and bundled SQL/tinytest coverage now exercises
sequence, multiallelic, symbolic
<DEL>/<DUP>, and missing-contig
rowsrduckhts_liftover() /
bcftools_liftover(...) indel parity in two exact upstream
rewrite points: repeat-run source extension now keeps extending across
the cached source-reference window boundary when needed, and the bundled
clip-pad Needleman-Wunsch path now keeps the best shift
even when candidate alignment scores are negative instead of leaving
padded intervals unshifted; bundled SQL/tinytest coverage now includes
dedicated repeat-run and clip-pad regression fixtures, and the real-data
conformance workflow reaches exact parity with installed
bcftools +liftover on GIAB HG001 chr20 plus the full HG006
GRCh37 benchmark VCFrduckhts_liftover() /
bcftools_liftover(...) row rejection for invalid
source-reference indel and difficult-SNP inputs: rows that fail the
source-FASTA validation path now stay in the result with
mapped = FALSE and
reject_reason = 'SourceRefMismatch' instead of fabricating
padded lifted alleles or aborting the query; bundled tests and README
examples now reflect the reject-row behaviorrduckhts_pileup() and bundle native
read_pileup(...) for region-scoped BAM pileups with
per-position chrom, pos, depth,
bases, and quals; expose bundled
read_bam(..., cigar_representation := 'binary') through
rduckhts_bam(..., cigar_representation = "binary") and
multi-file BAM wrappers, returning packed BAM CIGAR ops as
UINTEGER[]; and expose explicit gzi_path
arguments in rduckhts_fasta(),
rduckhts_fasta_multi(), and
rduckhts_fasta_nuc() so packaged bgzipped FASTA workflows
can use relocated .gzi sidecarsrduckhts_fasta_nuc() /
fasta_nuc(...) nucleotide counting on capable x86_64 hosts
with an AVX2+popcnt fast path selected via htslib-style runtime
dispatch, while preserving the scalar fallback everywhere elserduckhts_bam_index(): native remote BAM/BCF/tabix/FASTA/BED
reads now apply htslib block/cache tuning by access pattern, while
wasm/browser builds use the same policy with smaller budgets appropriate
for the XHR-backed worker runtime; the bundled vendored htslib also now
exposes a pre-opened sam_index_build4(...) entry point so
bam_index(...) can be tuned before remote index
construction beginsrduckhts_bcf() / read_bcf(...)
scanning stability for records where FILTER lists were
emitted without reserving list-vector capacity, which could crash with
allocator corruption
(double free/invalid pointer) during
full-table reads; FILTER entries now reserve child-list
space before writes and scans are stable on files previously triggering
crashes-Wpedantic during Unix and Windows package builds while
leaving vendored htslib on its upstream warning flagswasm_http_hfile.c
translation unit so native package builds do not warn about an empty
source file under pedantic C diagnosticsconfigure.win libcurl detection: the
package now requires a successful curl_easy_init link using
the detected pkg-config libcurl dependency closure before
enabling htslib remote URL support, and otherwise disables
libcurl/S3/GCS cleanlyrduckhts_gff() / rduckhts_gtf() and multi-file
wrappers: attributes_list = TRUE returns
MAP(VARCHAR, VARCHAR[]) with grouped multi-values and GFF3
percent-decoding, while attributes_pairs = TRUE returns
LIST<STRUCT(key VARCHAR, value VARCHAR, idx INTEGER)>
for exact key/value/index records; attributes_map = TRUE
remains the backward-compatible raw scalar mapread_gff(..., strict := true) through
rduckhts_gff(strict = TRUE) and
rduckhts_gff_multi(strict = TRUE), enabling GFF3 structural
validation from R/DBI workflows, including wrong field counts and
malformed attribute segments, while keeping the default GFF reader
permissive for existing ingestion pipelinesrduckhts_score() /
bcftools_score(...) so summary_path can be a
character vector or callers can use summaries_list_file;
multiple TSV/SSF summaries are scored in one genotype scan,
log_path can write per-PRS matching/audit counts for
loaded, matched, allele-mismatch, and duplicate markers,
summaries_list_file directory scans are deterministic and
ignore index sidecars, generated score/count column names are validated
for uniqueness, and score accumulation now follows upstream
bcftools +score float32 summation more closelyhtslib 1.23.1/system-requirements wording and redact
transient temp-file paths in rendered example output so regenerated
README diffs stay deterministicduckhts_cgranges_overlaps_list(...), a
vectorized scalar overlap expander that returns LIST-of-STRUCT hit
records so DBI queries can expand provider rows with
UNNEST(...) without generated bulk-probe SQL; package tests
cover one-row-per-hit expansion over regular tables and bundled BED
data, and the existing duckhts_cgranges_overlaps_bulk(...)
probe path now also handles DuckDB string vector lengths safelyduckhts_cgranges_from_query(...) ingestion
of DuckDB string vectors by respecting string lengths instead of
assuming NUL-terminated buffers; this fixes cgranges construction from
providers such as read_bed(...) with longer chromosome
names and adds package regression coverageduckhts_cgranges_has_overlap(...) and
duckhts_cgranges_count_overlaps(...), enabling DBI queries
to stream provider rows through an already-finalized session cgranges
index for filtering/count annotations without the materializing
overlaps_bulk query-string path; add package-level coverage
for overlap, contain, and NULL probe semanticsduckhts_cgranges_overlaps_bulk(...) for
SQL-first bulk cgranges probing from R/DBI sessions: one table-function
call now streams a query of probe intervals through a finalized cgranges
index, supports mode = 'overlap'|'contain', accepts an
optional query_row_id_col, and otherwise emits 1-based
probe ordinals as query_row_id; add package-level
regression coverage for the new bulk pathduckhts_cgranges_* entry points in the
generated function catalog and package README, add bundled DBI smoke
coverage for the session-scoped cgranges registry API, and include a
packaged overlap-conformance script reference for
bedtk-style parity checksrduckhts_fasta_nuc() /
fasta_nuc(...) GC and AT percentages for intervals
containing N: pct_gc and pct_at
now use only informative A/C/G/T bases in the denominator,
so ambiguous bases no longer depress reported bin/interval composition
percentages; add bundled regression coverageduckhts_cgranges_from_query(...), which runs the source
query on an extension-owned DuckDB connection and builds the cgranges
index in C before publishing it to the session registry;
duckhts_cgranges_from_table(...) remains deferred for
nowRduckhtsrduckhts_bam_bed_coverage(), bundling native
duckhts_bam_bed_coverage(...) for samtools coverage-like
regional summaries over BED targets with DuckHTS-specific
pre/post-filter columns and read-mode strand-specific post summaries;
bundled SQL/tinytest coverage now checks expected outputs on the
packaged mixed BAM fixture, and fragment_mode /
processing_threads are exposed but currently reserved for
later phasesrduckhts_bam_bed_coverage() /
duckhts_bam_bed_coverage(...) peak memory by allocating and
freeing per-region working depth buffers during scan processing instead
of retaining them for the whole BED, tile large target intervals
internally when computing covered-base breadth, keep the tiled
implementation single-pass, align min_depth > 1
mean-depth behavior with samtools coverage, and expose
decompression_threads so package callers can set htslib
BAM/CRAM decode worker counts explicitlyrduckhts_samtools_idxstats(), bundling native
duckhts_samtools_idxstats(...) for samtools
idxstats-compatible BAM/CRAM/SAM summaries with indexed BAM fast-paths
and scan fallback; package SQL/tinytest coverage now checks BAM
fast-path output, CRAM fallback output, explicit
index_path, and overwrite errorsREADME.html, .Rcheck, staged
duckhts_extension/htslib build outputs, wasm/webR harness
byproducts, and stray root-level index files under
r/Rduckhts/; add top-level make clean_local to
purge the reproducible package-side artifactsprocessing_threads parameter to
rduckhts_mosdepth() and bundled
duckhts_mosdepth(...) for parallel contig processing:
workers claim contigs atomically and write output in header order; on
the NA12878 WGS benchmark with 2 processing threads, fast mode is 1.38x
faster, default mode 1.40x faster, and fragment mode 1.61x faster than
mosdepth v0.3.13, all byte-identical; new default is
processing_threads = 2rduckhts_mosdepth() defaults to
threads = 2 (decompression) and
processing_threads = 2 (parallel contigs) for better
out-of-the-box WGS performanceduckhts_extension/htslib/{include,lib}/; add
inst/htslib_config.R (generated from
htslib_config.R.in at configure time) providing
htslib_cflags(), htslib_libs(),
htslib_rpath(), and htslib_version() for
downstream R packages that link against the bundled htslibconfigure.win to stage htslib headers into
include/htslib/ alongside lib/, matching Unix
configurebam_bin_counts(...) /
rduckhts_bam_bin_counts() to return a dense fixed-bin
layout across each selected contig span, including zero-count bins up to
the contig end instead of only observed bins; this gives downstream
CNV/sample serializers stable per-contig bin shapes, and the package
docs/tests now describe and validate the dense contractrduckhts_bam_bin_counts() and bundle native
bam_bin_counts(...) fixed-width BAM/CRAM binning in the
package. The new wrapper exposes mapq,
require_flags, exclude_flags, and
rmdup = "none"|"flag"|"streaming" duplicate handling,
always returns per-bin forward/reverse totals, and can add per-bin
GC/MAPQ summaries via stats = "gc", "mq", or
"gc,mq"; bundled extdata now includes the tiny WisecondorX
BAM/CRAM fixtures used by the new SQL/R tests, and the package README
now includes a native bin-count examplerduckhts_mosdepth() examples to the package README,
including windowed fragment coverage output and preview of the generated
BED.gz regions file, and refresh the generated function-catalog text so
the packaged mosdepth description matches the current v0.3.13 parity
surfacerduckhts_mosdepth() and bundled
duckhts_mosdepth(...) to cover the pinned local
mosdepth 0.3.13 option surface for indexed BAM/CRAM input:
fragment_mode = TRUE now matches upstream
--fragment-mode full-fragment insert coverage for proper
pairs, default mode is supported with CIGAR-aware coverage plus
mate-overlap correction, read_groups = "..." filters RG
tags, min_frag_len / max_frag_len filter
absolute template length, and use_median = TRUE switches
by = "<window|bed>" outputs from mean to median; add
bundled SQL/R/conformance coverage for BAM and CRAM
fast/fragment/default/median cases.rduckhts_mosdepth() and bundled
duckhts_mosdepth(...) fast-mode parity with
quantize = "...", writing mosdepth-style
.quantized.bed.gz + CSI output, and add bundled tests for
quantized output plus explicit by = "<bed>"
validation.rduckhts_mosdepth() and bundled
duckhts_mosdepth(...) fast-mode parity with
thresholds = "..." for
by = "<window|bed>", writing mosdepth-style
.thresholds.bed.gz + CSI outputs; also align window/BED
mean accumulation and window-region distribution bucketing with upstream
mosdepth’s current implementation behavior, and add bundled
SQL/R/native-conformance coverage for the new outputs.big,
empty-tids, overlapping-pairs,
ovl, nanopore, and related BED files) in
inst/extdata/ for stronger mosdepth parity testing, and
record Brent Pedersen as the original mosdepth author in the package
metadata/copyright bundle.rduckhts_mosdepth() and bundled
duckhts_mosdepth(...): the native mosdepth-compatible
fast-mode rewrite now accepts indexed CRAM input via
fasta = ... when required by htslib, and exposes
precision_digits = 2 as an explicit wrapper argument
instead of relying on the MOSDEPTH_PRECISION environment
variable; add bundled BAM/CRAM tests plus explicit precision
validation.README.Rmd with runnable compression/indexing
examples covering rduckhts_bgzip(),
rduckhts_bgunzip(), rduckhts_bam_index(),
rduckhts_bcf_index(), and
rduckhts_tabix_index(), then regenerate the rendered
package README outputs.decompression_threads to
rduckhts_bam() and rduckhts_bam_multi(),
matching the bundled
read_bam(..., decompression_threads := 2) SQL parameter.
The previous hardcoded htslib worker-thread count is now the documented
default, and 0 disables per-file worker threads.COUNT(*) queries across
the HTS readers: read_bam(...), read_bcf(...),
read_tabix(...), read_gff(...),
read_gtf(...), and indexed read_bed(...) now
use index metadata for full-file count-only scans when DuckDB projects
no output columns; read_fasta(...) uses faidx
sequence counts when an index is available and otherwise counts FASTA
headers directly; read_fastq(...) continues to count raw
FASTQ records directly when no projected columns are needed, while
preserving paired/interleaved validation errors.rduckhts_bam_multi,
rduckhts_bcf_multi, rduckhts_fastq_multi,
rduckhts_fasta_multi, rduckhts_bed_multi,
rduckhts_tabix_multi, rduckhts_gff_multi,
rduckhts_gtf_multi. Each follows the standard
(con, table_name, files, ..., overwrite) convention,
creates a DuckDB table with a filename column, and accepts
an optional .params data.frame for per-file parameter
overrides (e.g. per-sample regions or index paths). File expansion uses
DuckDB’s glob() so S3 URLs work transparently.hts_union_query(reader, pattern, params)
SQL scalar macro for pure-SQL multi-file reading via
SELECT * FROM query(hts_union_query('read_bam', '*.bam')).README.Rmd now covers the full
Module.duckhtsWasmHttpConfig parameter set
(headers, allowHosts,
enforceHostAllowlist, withCredentials,
allowInsecureAuth), explicitly notes that webR consumers
can set that config from R via webr::eval_js() without
editing the host page, and covers practical wasm/browser behaviors such
as same-origin setup, CORS requirements, .csi to
.tbi fallback, and non-fatal Range warnings
under the local http.server harness.configure now includes the shared
header from src/include/ via the bootstrapped
inst/duckhts_extension/include/wasm_socket_compat.h copy,
keeping the bundled browser build aligned with the extension sources
without changing native package builds.htslib: the Emscripten/webR configure path now
builds only libhts.a, links
duckhts.duckdb_extension directly against that static
archive, and no longer relies on runtime loading of bundled
libhts.so* files in webR/browser environments.http / https
backend in the bundled extension: src/wasm_http_hfile.c now
registers a synchronous XHR-backed htslib scheme handler
from the DuckDB extension entry point, so browser wasm builds can read
same-origin and CORS-enabled remote HTS URLs without going through
libcurl sockets.libcurl disabled in configure:
r-wasm/webr ships /opt/webr/wasm/lib/libcurl.a
and the emcc link test against it passes, but libcurl’s
connect() calls from a SIDE_MODULE still trigger a webR
Emscripten message-bus error (resolved is not a function)
on first network use, so the package-owned XHR backend is the supported
wasm HTTP path.wasm_http_hfile.c now caches object sizes from
Content-Range/Content-Length, clamps range
requests when size is known, short-circuits reads at/after EOF, and uses
a GET Range: bytes=0-0 fallback for SEEK_END
size discovery when HEAD metadata is unavailable; this
avoids cross-origin 416 failures on .tbi index EOF probes
(including GTEx tabix in webR/browser).200 OK,
wasm_http_hfile.c now caches the full object per open
handle and serves later reads from that in-memory cache to avoid
repeated full downloads, while still emitting one-time warnings when
Range is ignored and when large fallback payloads (>=64 MiB) are
used.Module.duckhtsWasmHttpConfig:
supports custom headers (including bearer auth), host allowlisting,
optional withCredentials, and a default HTTPS-only guard
that blocks Authorization on non-HTTPS URLs unless
allowInsecureAuth is explicitly enabled.Module.duckhtsWasmHttpConfig with
enforceHostAllowlist in the bundled wasm backend: when
enabled, requests to hosts outside allowHosts are blocked
instead of merely omitting configured headers.configure: preserve webR/Emscripten ${LDFLAGS}
on the final duckhts.duckdb_extension link so the
SIDE_MODULE settings reach the extension itself, and export
duckhts_init_c_api explicitly for DuckDB’s loader. This
fixes webR/browser rduckhts_load() failures where DuckDB
could not find a usable init export in
duckhts.duckdb_extension.linux_i686_musl for the Emscripten/webR path in
configure, matching the platform value you are using for
browser-side loading tests.rwasm / r-universe: the
package configure script now preserves injected
NAME=VALUE cache overrides, forwards explicit
--build / --host triplets into the vendored
htslib ./configure, forwards webR’s Emscripten
port flags for zlib/bzip2, seeds wasm-safe
Autoconf cache results for zlib/bzip2/socket
probes, injects a tiny Emscripten-only socket compatibility shim for
recv/send/closesocket, and
disables the optional htslib features that are not
available in the stock webR/r-universe wasm toolchain
(libcurl, S3, GCS,
lzma, plugins); this fixes the original
ac_cv_func_getrandom=no: command not found failure and the
subsequent nested htslib cross-compile probe failures
without changing native configure behavior.htslib in the linked archive,
avoiding unresolved symbols such as bcf_readrec at
LOAD.test_bam_file_offset: cast COUNT(*)
results to INTEGER in SQL so the DuckDB driver returns R
integer rather than numeric (BIGINT maps to
double in the duckdb R driver), restoring expect_identical
assertions.read_hts_index_spans(...) /
rduckhts_hts_index_spans(): the span view now returns real
chunk rows from CSI/TBI/BAI indexes, including populated
bin, chunk_beg_vo, chunk_end_vo,
chunk_bytes, seq_start, and
seq_end values instead of placeholder NAs;
BCF-backed calls also avoid the previous noisy tbx probe
warning on .csi indexes.FILE_OFFSET column to rduckhts_bam() /
read_bam(...): exposes the BGZF virtual file offset after
each record. Zero runtime overhead (macro over already-open struct
fields). Enables ORDER BY FILE_OFFSET in SQL
LAG() / LAST_VALUE() window functions to
reproduce exact BAM file order for streaming deduplication algorithms.
Together with the // integer-division operator and
LAST_VALUE(... IGNORE NULLS), this permits exact
replication of WisecondorX’s larp/larp2 state machine in pure SQL,
confirmed at 0 mismatches across 25,115 non-zero bins on a real NIPT
BAM.bcftools_liftover(...) /
rduckhts_liftover() cache and realignment hardening:
per-thread chain/FASTA contexts are now bounded instead of accumulating
for the lifetime of worker threads, and scalar left-alignment no longer
reuses stale traceback state after failed/empty alignments.read_bam(...) / rduckhts_bam()
and read_bcf(...) / rduckhts_bcf() indexed
parallel full scans when headers contain leading empty contigs: contig
claiming now retries iteratively instead of recursively, and the BAM
reader no longer returns an empty chunk after successfully handing off
to the next contig.htslib configuration now distinguishes
windows_amd64_mingw from windows_amd64_rtools,
keeping the smaller configure.win-style library set on
MinGW while restoring the fuller static libcurl dependency
closure needed on Rtools. CURL_STATICLIB remains on built
objects rather than ./configure probes.windows_amd64_rtools builds: the
package build now pins
CC/AR/RANLIB from
R CMD config, avoiding mixed compiler/library selection
when vendored htslib is configured, and keeps the MinGW
static-libcurl configuration aligned with Rtools
libcurl.a.read_bcf(...) / rduckhts_bcf()
mapping of fixed-count INFO/FORMAT arrays: exact-cardinality fields such
as Number=2 and Number=4 now materialize as
DuckDB array/list columns instead of silently dropping all but the first
value.read_bcf(...) / rduckhts_bcf()
handling of string FORMAT lists such as DRAGEN FORMAT/LAA:
Number != 1 string FORMAT fields now materialize as
VARCHAR[] instead of triggering DuckDB internal assertion
failures.duckdb_munge(...) /
rduckhts_munge() multithreaded FASTA lookups: FASTA index
handles are now thread-local and FASTA fetches are synchronized in
munge, avoiding intermittent fai_retrieve
failures and aborts when fasta_ref is used with
PRAGMA threads > 1.rduckhts_score(): polygenic risk score computation
backed by the bcftools +score plugin, supporting
GT/DS/HDS/AP/GP/AS dosage modes, all major GWAS summary presets (PLINK,
PLINK2, REGENIE, SAIGE, BOLT, METAL, PGS, SSF/GWAS-SSF), GWAS-VCF
multi-PRS scoring, p-value thresholding, sample subsetting, and
region/filter controls.rduckhts_munge(): GWAS summary statistics
normalization backed by bcftools +munge, with FASTA
reference allele resolution, swap-aware effect/frequency transforms, and
METAL meta-analysis column support.rduckhts_liftover(): variant coordinate liftover
backed by bcftools +liftover using UCSC chain files, with
full indel normalization, INFO/END lifting, and MT passthrough.rduckhts_bed() for BED3–BED12 interval files and
rduckhts_fasta_nuc() for nucleotide composition over BED
intervals or fixed-width bins.rduckhts_bgzip(),
rduckhts_bgunzip(), rduckhts_bam_index(),
rduckhts_bcf_index(), and
rduckhts_tabix_index().rduckhts_hts_header(),
rduckhts_hts_index(),
rduckhts_hts_index_spans(), and
rduckhts_hts_index_raw().rduckhts_bam() and
rduckhts_fastq() (quality_representation,
input_quality_encoding) and
rduckhts_detect_quality_encoding() for heuristic FASTQ
encoding detection.sequence_encoding := 'nt16' parameter to
rduckhts_bam(), rduckhts_fasta(), and
rduckhts_fastq() for raw htslib nt16 sequence output as
UTINYINT[].sam_flag_bits() and
sam_flag_has(), CIGAR utility functions, and
is_forward_aligned().Conditionaly enable plugins in windows
Updates the configure script to avoid check faillure on CRAN MacOS
Update the extension version to 0.1.3
duckhts
0.1.2.CRAN Submission
Bump bundled duckhts extension version to 0.1.1.
Initial development release.
Bundles the DuckHTS DuckDB extension and htslib for HTS file readers.
Adds table-creation helpers for VCF/BCF, BAM/CRAM, FASTA/FASTQ, GFF/GTF, and tabix.