Data Sources

A data source is a long-lived binding between remote data and a Diafunc Table. Instead of downloading a spreadsheet or exporting a database dump by hand every time the numbers change, you point a data source at the origin once and Diafunc keeps a Table in sync with it. Each refresh fetches the current contents from the source and overwrites the target Table, so the rest of your project (analyses, functions, notebooks) always reads the latest snapshot without any manual re-import.

Data sources live inside a project and are managed from the Data Sources section of the Lab. You can refresh them on demand, or attach a schedule so they wake up and re-import on their own.

A data source can also feed an analysis directly. What it fetches is then one of the files the analysis's data is built from, and every refresh builds that data again. See "A data source that feeds an analysis" below.

Each data source has a connector type that decides where the data is fetched from and which configuration fields it needs. The connectors registered on this deployment are listed by diafunc data-sources types (or the list_data_source_types tool); the current set is:

Type	Wire name	Description
Google Sheets	google_sheets	Pulls a Google Sheet as CSV. Paste any share, edit, or export URL; the sheet must be readable by anyone with the link.
S3 URL	s3_url	Fetches any HTTPS URL: a public S3 / GCS object, a pre-signed URL, or a generic endpoint. An optional Authorization header lets you reach protected URLs.
AWS S3 (signed)	s3_sigv4	Reads a private S3 object with AWS access keys, signing the request with SigV4. An endpoint override and path-style addressing make it work with S3-compatible services such as MinIO, Cloudflare R2, and Scaleway.
Database (JDBC)	jdbc	Opens a read-only connection to PostgreSQL, MySQL, or SQLite and streams the result of a SELECT into the Table.

Whatever the origin, the fetched content is parsed the same way an uploaded file would be. See Supported file formats for how CSV and other formats are interpreted into columns and rows.

Open the Data Sources section of a project and choose New data source. Creation is a short four-step wizard:

1. Type: pick one of the connectors above.
2. Config: give the source a display name and fill in the connector-specific fields (a sheet URL, the S3 credentials, or the database dialect, host and SELECT). Required fields are validated before you can continue, and Test connection checks the configuration on the spot (see below).
3. Target Table: pick the Table that each refresh will overwrite from a dropdown of the project's Tables, or choose Create a new table to have one created and linked for you.
4. Schedule: leave it manual-only or pick a refresh cadence (see below).

Finishing the wizard with Create and run initial fetch saves the data source and runs the first import synchronously, so you immediately see whether the connection works and land on the detail page with the result of that first run.

The same thing can be done from the CLI. data-sources create runs the initial fetch synchronously and prints the resulting status:

Create a data source

# Create a Google Sheets data source that overwrites a target Table,
# refreshed automatically every day at 02:00 UTC.
diafunc data-sources create \
  --project-id my-project \
  --project-branch main \
  --target-entity-id sales-table \
  --target-entity-branch main \
  --name "Weekly sales sheet" \
  --type google_sheets \
  --config url=https://docs.google.com/spreadsheets/d/<id>/edit \
  --schedule-cron "0 0 2 * * ?"

Additional commands manage the source over its lifetime:

Manage a data source

# List the connector types this deployment supports
diafunc data-sources types

# Trigger an out-of-schedule refresh right now
diafunc data-sources refresh <id>

# Inspect the recent run history (newest first)
diafunc data-sources runs <id>

# Switch a data source back to manual-only
diafunc data-sources update <id> --clear-schedule

The data of an analysis is built from one or more files: files with the same columns are stacked, a file with other columns that shares a key adds columns, and a file that is not one row per observation is read through a reshape (see Several files). A data source can be one of those files. What it fetches is treated exactly like an uploaded file, so an analysis can be built from an uploaded file of batches plus a database query of lab results that is matched to them by batch number and fetched again every night.

In the upload step of an analysis, choose Add from a data source. Pick the connector, fill in its fields, give it a name and, if you want, a schedule. Test connection works as for any data source. Fetch and add fetches the data once, and the fetched file appears in the list of files marked Data source. The plan below the list states how all the files fit together, and the data is built when you go on, exactly as with uploaded files.

What a refresh does:

1. It fetches the current content.
2. It checks the new file against the way the data was last built: the same files stacked, the same keys, the same reshapes. Nothing is asked again and nothing is decided anew.
3. When the file still fits, it replaces the old file and the analysis's data is built again.
4. When it no longer fits, for example because a key column is gone, a key that had to be unique is not unique any more, or a stacked file now has other columns, the refresh fails and the analysis keeps the file and the data it had. The reason is on the data source's detail page and in its run history. Fix the origin and refresh again, or open the upload step of the analysis and confirm a new plan.
5. While a run of the analysis or an import of its data is under way, the refresh is skipped and tried again at the next scheduled time.

A refresh never starts a run by itself. For a new report from fresh data, give the analysis a trigger with Refresh the data first switched on (see Analysis triggers).

The report says where its data comes from: the section on the raw data names each file a data source keeps up to date, the data source, and when the file was last fetched. In the Lab, the data source's detail page shows Feeds the analysis with a link to it instead of a target Table. Deleting the data source leaves the analysis its file and its data, which nothing refreshes any more; removing the file from the analysis deletes its data source with it.

A Table that holds the data of an analysis cannot be the target of a Table data source: creating one is refused and names the analysis to add the data source to.

From the CLI, --analysis replaces the four flags that name a Table. The command fetches, builds the data from the plan the platform proposes, and prints what was built:

A data source that feeds an analysis

# Fetch a database query every night and make it one of the files
# the analysis is built from.
diafunc data-sources create \
  --analysis a1b2c3d4-.... \
  --name "Lab results" \
  --type jdbc \
  --config dialect=postgresql --config host=db.example.com \
  --config database=lab --config user=reader --config password=... \
  --config "query=SELECT batch, purity, moisture FROM results" \
  --schedule-cron "0 0 2 * * ?"

# Run the analysis every Monday at 06:00 UTC on freshly fetched data.
diafunc analyze triggers create a1b2c3d4-.... \
  --kind CRON --schedule "0 0 6 ? * MON" --refresh-first

Test connection verifies a configuration without running an import. It is available in the wizard's Config step (as soon as the connector's required fields are filled in) and on the detail page of an existing source, where the test uses the stored credentials, so you can re-check a source without re-entering secrets.

What the test does depends on the connector. Database (JDBC) sources open a short-lived connection to the database and close it again without running a query; AWS S3 (signed) sources look up the object's metadata with the configured keys. A successful probe reports "Connection verified." Google Sheets and S3 URL sources cannot be probed without downloading the full content, so for those the test validates the configuration shape and reports "Configuration looks valid. This source type can't be probed without a full fetch." A failed test shows the reason inline (an unreachable host, a missing bucket or object, or denied access) so you can correct the configuration before creating the source or starting a refresh. Test results never include your credentials, and editing any config field clears the previous verdict.

Every data-source operation is also available to the conversational assistant and to external agents through the Diafunc MCP tools: create_data_source, list_data_sources, get_data_source, update_data_source, delete_data_source, refresh_data_source, list_data_source_runs, get_data_source_run, get_data_sources_dashboard and list_data_source_types. The tool arguments mirror the CLI flags (a project, a target Table entity, a connector type, and a flat string map of connector config), so you can ask the assistant to wire up an import and schedule it in plain language.

A schedule is a Quartz cron expression. When it is set, Diafunc wakes the data source up on that cadence and re-imports without any interaction; when it is blank, the source is manual-only. The wizard offers a few ready-made cadences (hourly, every six hours, daily at 02:00 UTC, weekly on Monday, or a custom cron), and you can change the schedule at any time on the detail page or with data-sources update. Passing an empty schedule (--clear-schedule) switches a source back to manual-only.

Regardless of the schedule, you can always force an import immediately with Refresh now on the detail page, the data-sources refresh command, or the refresh_data_source tool. Because every refresh fetches and rewrites the target Table, refreshes consume compute and transfer credits like any other operation. See Credits for how that is metered.

Every import, scheduled or manual, is recorded as a run. The detail page lists the recent runs with their start and finish time, status, the number of rows imported, the amount of data transferred, the credits charged, what triggered the run, and any error message. If the most recent run failed, a banner at the top of the page surfaces the error so you can fix the configuration and refresh again. The same records are available through data-sources runs / data-sources run on the CLI and the list_data_source_runs / get_data_source_run tools.

When a project has several data sources, the Dashboard gives a single overview of all of them. Roll-up cards summarise the total number of sources, how many are scheduled, how many have failed recently, and when the last activity happened; a table below lists each source with its type, schedule, current status, last run, next scheduled time, recent failure count, and last transfer size. Rows that have failed recently are highlighted so problems stand out at a glance, and any row opens straight into its detail page. The same summary is available programmatically through get_data_sources_dashboard.

Secret configuration fields (database passwords, S3 secret keys, Authorization headers, session tokens) are stored encrypted and never returned in clear text. When you view or edit a data source, those fields show as [redacted]. Leave a redacted field untouched to keep the existing credential, or type a new value to rotate it. Database sources always open a read-only connection, and query results are capped, so an import cannot modify the source system.