Preparing your data

Most real-world data files import cleanly because Diafunc auto-detects the relevant dialect bits: delimiter, quote, encoding, header row, decimal separator. The sections below cover the dialect choices Diafunc handles automatically, what to do when the heuristic guesses wrong, and the small set of formats / variants that need a pre-process step.

For the full per-format matrix and known limitations, see supported file formats.

The platform sniffs file encoding via BOM detection (UTF-8, UTF-16 LE / BE) plus juniversalchardet for the rest (ISO-8859-1, Windows-1252, common East-Asian encodings). UTF-8 is the safest default; if you control the export, prefer it.

Symptoms of a mis-sniffed encoding: garbled non-ASCII characters in the preview ("é" instead of "é"). The preview's dialect card shows the detected encoding; if it's wrong, re-save the file as UTF-8 from the source application and re-upload.

Diafunc auto-detects ,, ;, and \t (tab) delimiters by scanning the first kilobyte of bytes. The ; delimiter is common in German, Austrian, and French exports because those locales use , as the decimal separator (see below).

Quote handling follows RFC 4180: double-quoted fields, embedded "" escapes the quote, newlines inside quoted fields are allowed. Non-standard escaping schemes (e.g. backslash-escapes) are not parsed. Re-export the file with standard quoting.

The first row is treated as the header by default; if it looks like data (all-numeric, all-blank, etc.) Diafunc inserts synthetic V1, V2, … names. The preview's dialect card lets you toggle "First row is data" if the heuristic guessed wrong.

Numeric strings are parsed with the dot (.) as the decimal separator. If your source uses a comma (European locale), the import either:

• Treats the column as text when commas appear in obviously-numeric values. You then need to re-export with . separators, or run a function to swap , for . before downstream analysis.
• Reads correctly when the comma is also the delimiter in a ;-delimited CSV: the parser binds ; as field separator and treats numbers natively.

When you control the source, prefer . for decimals. When you don't, the cleanest fix is a one-line transform inside Diafunc (table.column("price").apply(s -> s.replace(',', '.')).asDouble()) after import.

Each format has its own convention for "this cell has no value":

Format	Missing token(s)	Renders in Diafunc as
CSV / TSV	Empty field; NA, N/A, null, NaN, -, ?	empty string
Parquet / Arrow / ORC / Avro	native null	empty string
JSON	null literal	empty string
SPSS	SYSMIS	empty string
Stata	byte > 100; int > 32740; long > 2147483620; NaN floats	empty string
R (RDS and workspaces)	NA_integer_, NA_real_, NA_character_	empty string
SAS / SAS Transport	., ._, .A to .Z	empty string
ARFF	?	empty string
Fixed-width text	blank field	empty string
LAS well logs	the file's NULL value (often -999.25)	empty string
NetCDF / HDF5	_FillValue, missing_value, NaN	empty string
FITS / NASA CDF	TNULL and NaN; FILLVAL	empty string
SDMX / JSON-stat / PC-Axis	no observation; a symbol in place of a number (., .., -)	empty string
GeoJSON / Shapefile / GeoPackage / GeoParquet	a feature without a geometry	empty position columns

The rendering is consistent: empty string everywhere downstream. You can re-encode ("" → 0, "" → "unknown") inside a function after import. The platform never silently substitutes for you.

Some exports mark a missing measurement with a number instead, such as 99, 999 or -999.25. The table keeps those values as uploaded, the import preview names each column that uses one, and an analysis treats those cells as missing and says so in its Data chapter.

Several formats can hold multiple logical tables in one file:

• Excel and OpenDocument: multiple worksheets (Sheet1, Sheet2, …)
• SQLite, DuckDB, Access and SQL dumps: every table
• Zip / Tar archives: one importer per entry; each entry can be any other supported format
• SAS Transport: every data set in the library
• R workspaces: every data frame and matrix
• HDF5 and NetCDF: every dataset or variable, with variables along a shared dimension also offered together
• NumPy .npz and MATLAB: every array or variable
• TDMS and MDF: every group or channel group, and for MDF all groups on one time axis
• LAS 3.0: every data section
• FITS, NASA CDF and ROOT: every table extension, every time base, every tree
• SDMX, JSON-stat and PC-Axis: one table per choice of the dimension that becomes the columns, plus the long form with one row per value
• Shapefile zips and GeoPackage: every layer
• HTML: every <table> element

When you upload one of these, the preview shows an inner-table picker. Pick which table you want to import: only one per upload. To import additional tables from the same file, re-upload and pick a different selection. The CLI and API accept --selection <id> / ?selection=<id>.

The platform auto-flattens nested objects with dotted column names: {"user": {"name": "Alice", "email": "alice@x"}} becomes columns user.name and user.email. Arrays-of-objects flatten the same way per element.

Two wrapped shapes are read as well: a root object that wraps arrays of records (such as {"data": [...]}, typical of APIs) imports its largest array and names the others in the warning, and an array of arrays imports as rows, using a leading row of names as the header. JSON written by pandas to_json in any orient imports as a table. Heterogeneous arrays with no consistent columns are rejected with a clear error: convert them to objects with field names first.

For YAML, the same shape rules apply. For XML, the row pattern is the highest-cardinality child of root. If your data uses a custom XPath, register a user-defined importer.

A short list of variants that need a pre-process step:

• Stata value labels are not applied: labelled variables import as their codes.
• MATLAB table / timetable variables cannot be read, because MATLAB does not publish their layout: export them with writetable(T, "T.csv").
• Python pickles of anything other than pandas, NumPy and plain Python data are rejected, because Diafunc never runs a pickle's code: save the frame with df.to_parquet(...) instead.
• Excel 95 and older, and password-protected workbooks: save as .xlsx without a password.
• pg_dump custom-format archives (-Fc, -Fd): dump again with --format=plain.
• MDF raw bus frames (CAN, LIN) are not decoded: decode them with the bus database (DBC) first.
• Minitab worksheets: save them from Minitab as CSV or Excel.
• Shapefiles come in parts: zip the .shp, .shx and .dbf together (with the .prj and .cpg when present). A .shp alone cannot be read, and the outline of a feature is not imported, only its position.
• Statistical cubes (SDMX, JSON-stat, PC-Axis) open with the indicators as columns. If the table you need has another dimension across the top, pick that pivot in the table picker, or pick the long form.
• ROOT branches with a variable number of values per entry (the jets or leptons of an event) and object branches are named and left out: write a flat tree in ROOT first, for example with RDataFrame Define and Snapshot. RNTuple files are not read.
• NASA CDF constants that do not change from record to record are not part of a table; only the variables that vary along a time base are.
• Tile-compressed FITS tables (.fits.fz): decompress them with funpack first.
• Parquet partitioned datasets (directories of part-files): zip the directory and upload the archive, or choose the part-files together in the Analyze wizard. The parts are stacked into one table and key=value folder names become columns.