Data Sources

A data source is a long-lived binding between remote data and a Diafunc Table. Instead of downloading a spreadsheet or exporting a database dump by hand every time the numbers change, you point a data source at the origin once and Diafunc keeps a Table in sync with it. Each refresh fetches the current contents from the source and overwrites the target Table, so the rest of your project (analyses, functions, notebooks) always reads the latest snapshot without any manual re-import.

Data sources live inside a project and are managed from the Data Sources section of the Lab. You can refresh them on demand, or attach a schedule so they wake up and re-import on their own.

Each data source has a connector type that decides where the data is fetched from and which configuration fields it needs. The connectors registered on this deployment are listed by diafunc data-sources types (or the list_data_source_types tool); the current set is:

Type	Wire name	Description
Google Sheets	google_sheets	Pulls a Google Sheet as CSV. Paste any share, edit, or export URL; the sheet must be readable by anyone with the link.
S3 URL	s3_url	Fetches any HTTPS URL — a public S3 / GCS object, a pre-signed URL, or a generic endpoint. An optional Authorization header lets you reach protected URLs.
AWS S3 (signed)	s3_sigv4	Reads a private S3 object with AWS access keys, signing the request with SigV4. An endpoint override and path-style addressing make it work with S3-compatible services such as MinIO, Cloudflare R2, and Scaleway.
Database (JDBC)	jdbc	Opens a read-only connection to PostgreSQL, MySQL, or SQLite and streams the result of a SELECT into the Table.

Whatever the origin, the fetched content is parsed the same way an uploaded file would be — see Supported file formats for how CSV and other formats are interpreted into columns and rows.

Open the Data Sources section of a project and choose New data source. Creation is a short four-step wizard:

1. Type — pick one of the connectors above.
2. Config — give the source a display name and fill in the connector-specific fields (a sheet URL, the S3 credentials, or the database dialect, host and SELECT). Required fields are validated before you can continue, and Test connection checks the configuration on the spot (see below).
3. Target Table — pick the Table that each refresh will overwrite from a dropdown of the project's Tables, or choose Create a new table to have one created and linked for you.
4. Schedule — leave it manual-only or pick a refresh cadence (see below).

Finishing the wizard with Create and run initial fetch saves the data source and runs the first import synchronously, so you immediately see whether the connection works and land on the detail page with the result of that first run.

The same thing can be done from the CLI. data-sources create runs the initial fetch synchronously and prints the resulting status:

Create a data source

Additional commands manage the source over its lifetime:

Manage a data source

Test connection verifies a configuration without running an import. It is available in the wizard's Config step (as soon as the connector's required fields are filled in) and on the detail page of an existing source, where the test uses the stored credentials, so you can re-check a source without re-entering secrets.

What the test does depends on the connector. Database (JDBC) sources open a short-lived connection to the database and close it again without running a query; AWS S3 (signed) sources look up the object's metadata with the configured keys. A successful probe reports "Connection verified." Google Sheets and S3 URL sources cannot be probed without downloading the full content, so for those the test validates the configuration shape and reports "Configuration looks valid. This source type can't be probed without a full fetch." A failed test shows the reason inline — an unreachable host, a missing bucket or object, or denied access — so you can correct the configuration before creating the source or starting a refresh. Test results never include your credentials, and editing any config field clears the previous verdict.

Every data-source operation is also available to the conversational assistant and to external agents through the Diafunc MCP tools: create_data_source, list_data_sources, get_data_source, update_data_source, delete_data_source, refresh_data_source, list_data_source_runs, get_data_source_run, get_data_sources_dashboard and list_data_source_types. The tool arguments mirror the CLI flags — a project, a target Table entity, a connector type, and a flat string map of connector config — so you can ask the assistant to wire up an import and schedule it in plain language.

A schedule is a Quartz cron expression. When it is set, Diafunc wakes the data source up on that cadence and re-imports without any interaction; when it is blank, the source is manual-only. The wizard offers a few ready-made cadences (hourly, every six hours, daily at 02:00 UTC, weekly on Monday, or a custom cron), and you can change the schedule at any time on the detail page or with data-sources update. Passing an empty schedule (--clear-schedule) switches a source back to manual-only.

Regardless of the schedule, you can always force an import immediately with Refresh now on the detail page, the data-sources refresh command, or the refresh_data_source tool. Because every refresh fetches and rewrites the target Table, refreshes consume compute and transfer credits like any other operation — see Credits for how that is metered.

Every import, scheduled or manual, is recorded as a run. The detail page lists the recent runs with their start and finish time, status, the number of rows imported, the amount of data transferred, the credits charged, what triggered the run, and any error message. If the most recent run failed, a banner at the top of the page surfaces the error so you can fix the configuration and refresh again. The same records are available through data-sources runs / data-sources run on the CLI and the list_data_source_runs / get_data_source_run tools.

When a project has several data sources, the Dashboard gives a single overview of all of them. Roll-up cards summarise the total number of sources, how many are scheduled, how many have failed recently, and when the last activity happened; a table below lists each source with its type, schedule, current status, last run, next scheduled time, recent failure count, and last transfer size. Rows that have failed recently are highlighted so problems stand out at a glance, and any row opens straight into its detail page. The same summary is available programmatically through get_data_sources_dashboard.

Secret configuration fields (database passwords, S3 secret keys, Authorization headers, session tokens) are stored encrypted and never returned in clear text. When you view or edit a data source, those fields show as [redacted]. Leave a redacted field untouched to keep the existing credential, or type a new value to rotate it. Database sources always open a read-only connection, and query results are capped, so an import cannot modify the source system.