Shape · Data quality and flow control
Expectations
Runs a set of quality rules and outputs a scorecard with one row per rule, its failed count and pass rate. Unlike Assert, it never stops the run.
Also known as: Data quality report
How it works
Each rule has a kind: Row count (min, max or exact), No nulls (one or more columns, with an optional tolerance as a share of rows, 0 by default), Columns present, Unique, Accepted values, Value range, Regex match (a partial match unless you anchor the pattern), Freshness (the column's newest value is within N minutes, hours or days of now), Geometry valid and Custom predicate (a boolean SQL expression every row must satisfy; a NULL result counts as a violation). A rule missing its column or value is ignored. Instead of stopping, the node outputs a scorecard with one row per rule: rule (its label, such as No nulls in email), column (NULL for Row count and Custom predicate), total, failed, pass_rate (NULL when total is 0) and passed, the same verdict Assert would give, so a No nulls rule with a tolerance can pass with a failed count above 0.
Inputs and outputs
Inputs
data— Table
Outputs
scorecard— Table
Settings
The Attributes panel of Expectations, as in the app: the controls work, but nothing runs — there is no data behind it.
The same settings, as the Assistant writes them
{
rules: Rule[]
}— non-negative = value_range with min 0. Output: rule, column, total, failed, pass_rate, passed. where Rule = { kind: "row_count", min?: number, max?: number, exact?: number } | { kind: "not_null", columns: string[], maxNullRatio?: number } | { kind: "columns_present", columns: string[] } | { kind: "unique", columns: string[] } | { kind: "accepted_values", column: string, values: string[] } | { kind: "value_range", column: string, min?: number, max?: number } | { kind: "regex_match", column: string, pattern: string } | { kind: "freshness", column: string, maxAge: number, unit: "minutes"|"hours"|"days" } | { kind: "geometry_valid", column: string } | { kind: "custom_sql", expression: string } // custom_sql expression = a boolean DuckDB SQL predicate every row must satisfy (a NULL result counts as a violation), e.g. "end_date >= start_date"; regex_match pattern = an RE2 regex tested with regexp_matches, a partial match unless anchored, e.g. "^[A-Z]{2}[0-9]{4}$"
Example
Track the quality of a weekly extract: No nulls in email, Value range on amount with min 0, Regex match on postcode. Write the scorecard to a table each week to see the trend.
| rule | column | failed | pass_rate |
|---|---|---|---|
| No nulls in email | 12 | 0.994 | |
| amount ≥ 0 | amount | 0 | 1.0 |
| postcode matches ^[0-9]{2}-[0-9]{3}$ | postcode | 41 | 0.979 |