How Diafunc measures quality

Quality says how well a model predicts on rows the search never saw, marked down for the size of its formula. 0% is no better than always predicting the average, and 100% is exact. This page describes how that figure comes about, what stands beside it in a report, and which checks a report runs on its own model.

Nothing here needs a setting. Every analysis is measured this way.

Before the search for a model starts, a part of your table is set aside. The search works with the rest. When it has finished, the model it returned is scored on the rows that were set aside, the held-out rows. A model cannot score well there by memorizing the table.

How many. One fifth of the rows, but at least 10 and at most 2,000, and never more than half of the table. A table of 308 rows holds out 62 and searches on 246. A table of 50,000 rows holds out 2,000.

Which ones. It depends on the table.

The draw depends only on your data and your settings, so the same analysis run again holds out the same rows. There is no setting for the share or the method.

On a table too small to hold rows back, the report marks its figures as not held out.

These are the definitions of the Glossary, which holds the other terms of a report as well.

For a quantity target, quality sits at or just below the R² of the same rows. The gap between the two is what the size of the formula costs. A short formula found on many rows reads almost exactly its R². A long formula found on few rows reads visibly lower, because few rows cannot justify a long formula. With more rows the two figures come together.

The discount depends on how many rows the search had, not on how many rows the model is checked against. A small held-out part does not lower the quality by itself; it makes the interval around the figure wider.

Quality is never rounded up, so only an exact result reads 100%. A model that does worse than the average reads 0%. During the search the same measure, on the rows the search works with, is what candidates are ranked by. Those figures describe the search and tend to flatter the model it chose, which is why the figures to quote are the held-out ones.

Every held-out figure comes with a 95% confidence interval. It is computed by drawing the held-out rows again and again, a thousand times, and scoring the same formula on each draw: the formula never changes, only the rows. A wide interval means the figure rests on few rows.

In the public Yacht Hydrodynamics report the quality is 99.27% on 62 held-out rows, with an interval from 98.69% to 99.66%.

A quality figure alone does not say whether the model was worth discovering. Every report therefore scores simple reference predictors, the baselines, on the same held-out rows.

A baseline is a measuring stick. The report shows its scores and nothing else; an analysis has one model per target.

The verdict under the gauges names the strongest baseline and both figures, for example "Well ahead of the Linear regression baseline" or "Level with the Linear regression baseline". When the model is level with a straight line, the relation is linear and the extra structure adds nothing, and the report says so. A model that barely beats its baseline is listed as a concern, and one that loses to it as a critical one.

The Concerns chapter lists what the analysis found against its own data and its own model, the critical ones first. Some of the checks, with the thresholds they use:

A column that tracks the target that closely is one of two things: a strong predictor, or a data leak, a column derived from the target after the fact. The data cannot tell which. The report asks you whether the column is known before the outcome, and does not pass a verdict it cannot support.

The Yacht report shows the checks at work. Its model has a quality of 99.27%, and the report still lists a critical concern: the misses in the top third of the predictions are 12.9 times as spread as in the bottom third. Its Deployment Readiness chapter gives the verdict Not ready for that reason. Read the report.

Three further guards are pass or fail: every prediction is a finite number, predictions stay within the range of the target widened by two standard deviations, and no prediction runs away more than ten standard deviations from the mean. They say nothing about quality. A failure means the formula misbehaves somewhere in the data it was shown.

The figures above describe the model as a whole. For a single prediction the report gives a range, taken from the misses on the held-out rows: beside a prediction it reads, for example, "9 in 10 within" a stated distance. A deployed model returns the same kind of range with every prediction, as a 90% interval, from the 5th to the 95th percentile of the held-out misses.

Whether such ranges hold is checked as well. For a quantity target the report compares, at several levels, how many held-out rows should fall inside the range with how many do.

Quality and Error apply to every kind of target. The comparison is always with the predictor that knows nothing: the average for a quantity, the average count for a count, the most common class for a category. A model that only repeats the most common outcome therefore reads a quality of 0%, also when that outcome covers 95% of the rows.

Each kind has measures of its own beside them, such as accuracy and balanced accuracy for a yes or no target. They are described under Target kinds.

Every figure and every formula in a report is computed. A language model writes the narrative parts: the headline, the findings, the summary, the answers to your questions. It writes them from the results of the analysis.

For that it receives column names, summary statistics, the discovered models with their scores, the questions you asked, and group labels such as region = NW. The rows of your table are never sent.

Cause. A model describes how the columns move together. Whether one causes the other is a question the data alone cannot answer.

Outside the range. Predictions for inputs beyond what the table covered are less reliable than inside it. The formula shows you what it would do there, and the figures do not vouch for it.

A small table. There is no fixed minimum, but a quality measured on few rows is uncertain. The check before launch warns below about a hundred rows per target.

A different future. Held-out rows come from the same table. If what you measure changes, the figures describe the past.

The model is discovered by Ufinq, Diafunc's Symbolic AI technology. It is benchmark-tested: evaluated on SRBench, the standard Symbolic Regression benchmark. A benchmark sweep runs every week on public datasets, and the results are published at ufinq.com/benchmarks, with no cherry-picked workloads. We make no claim to beat anyone.