Preparing your data

Most real-world data files import cleanly because Diafunc auto-detects the relevant dialect bits — delimiter, quote, encoding, header row, decimal separator. The sections below cover the dialect choices Diafunc handles automatically, what to do when the heuristic guesses wrong, and the small set of formats / variants that need a pre-process step.

For the full per-format matrix and known limitations, see supported file formats.

The platform sniffs file encoding via BOM detection (UTF-8, UTF-16 LE / BE) plus juniversalchardet for the rest (ISO-8859-1, Windows-1252, common East-Asian encodings). UTF-8 is the safest default; if you control the export, prefer it.

Symptoms of a mis-sniffed encoding: garbled non-ASCII characters in the preview ("é" instead of "é"). The preview's dialect card shows the detected encoding; if it's wrong, re-save the file as UTF-8 from the source application and re-upload.

Diafunc auto-detects ,, ;, and \t (tab) delimiters by scanning the first kilobyte of bytes. The ; delimiter is common in German, Austrian, and French exports because those locales use , as the decimal separator (see below).

Quote handling follows RFC 4180: double-quoted fields, embedded "" escapes the quote, newlines inside quoted fields are allowed. Non-standard escaping schemes (e.g. backslash-escapes) are not parsed — re-export the file with standard quoting.

The first row is treated as the header by default; if it looks like data (all-numeric, all-blank, etc.) Diafunc inserts synthetic V1, V2, … names. The preview's dialect card lets you toggle "First row is data" if the heuristic guessed wrong.

Numeric strings are parsed with the dot (.) as the decimal separator. If your source uses a comma (European locale), the import either:

• Treats the column as text when commas appear in obviously-numeric values. You then need to re-export with . separators, or run a function to swap , for . before downstream analysis.
• Reads correctly when the comma is also the delimiter in a ;-delimited CSV — the parser binds ; as field separator and treats numbers natively.

When you control the source, prefer . for decimals. When you don't, the cleanest fix is a one-line transform inside Diafunc (table.column("price").apply(s -> s.replace(',', '.')).asDouble()) after import.

Each format has its own convention for "this cell has no value":

Format	Missing token(s)	Renders in Diafunc as
CSV / TSV	Empty field; NA, N/A, null, NaN, -, ?	empty string
Parquet / Arrow / ORC / Avro	native null	empty string
JSON	null literal	empty string
SPSS	SYSMIS	empty string
Stata	byte > 100; int > 32740; long > 2147483620; NaN floats	empty string
R RDS	NA_integer_, NA_real_, NA_character_	empty string

The rendering is consistent: empty string everywhere downstream. You can re-encode ("" → 0, "" → "unknown") inside a function after import — the platform never silently substitutes for you.

Several formats can hold multiple logical tables in one file:

• Excel — multiple worksheets (Sheet1, Sheet2, …)
• SQLite — multiple tables in sqlite_master
• Zip / Tar archives — one importer per entry; each entry can be any other supported format
• HDF5 — every dataset in the group tree
• NetCDF — every variable
• HTML — every <table> element

When you upload one of these, the preview shows an inner-table picker. Pick which table you want to import — only one per upload. To import additional tables from the same file, re-upload and pick a different selection. The CLI and API accept --selection <id> / ?selection=<id>.

The platform auto-flattens nested objects with dotted column names: {"user": {"name": "Alice", "email": "alice@x"}} becomes columns user.name and user.email. Arrays-of-objects flatten the same way per element.

The heuristic struggles in two cases:

1. Single root object containing one array under a key. The platform surfaces a warning suggesting you pre-flatten — strip the wrapper or re-export as NDJSON.
2. Arrays-of-arrays / heterogeneous arrays. No safe column projection; rejected with a clear error. Convert to objects (with field names) first.

For YAML, the same shape rules apply. For XML, the row pattern is the highest-cardinality child of root — if your data uses a custom XPath, register a user-defined importer.

A short list of variants that need a pre-process step:

• Stata pre-117 (Stata 12 and earlier): re-export from Stata as v13+ (release 117) or as CSV.
• Stata strL long-string columns: re-encode as fixed-width str# if you need the column.
• SPSS .zsav ($FL3, zlib-framed): re-save as plain .sav from SPSS or PSPP.
• R ASCII-serialized RDS: re-save with saveRDS(x, file, ascii=FALSE). The default in modern R is already binary XDR.
• R .rda / .RData multi-object archives: loaded with load() in R, not readRDS(), and have a different on-disk format. Re-save the relevant object alone with saveRDS(x, "x.rds").
• HDF5 / NetCDF rank > 2: project to ≤ 2 dimensions in the source tool before exporting (e.g. fix one axis to a single index).
• Parquet partitioned datasets (directories of part-files): concatenate to a single file with pyarrow or duckdb before uploading.