Shape · Data quality and flow control

Gate

Passes data on only when a condition holds. The condition can test the data itself, a separate control table or a parameter. Otherwise everything after it is skipped and the run still succeeds. For example, publish only if the extract has rows.

Also known as: Only if, skip downstream

How it works

Open when picks the test. The data has matching rows checks the data input; the control input has matching rows checks the optional control input instead; both use the Conditions list (column, operator, and a Value, another Column or a boolean SQL Expression), combined with AND or OR, and with no conditions they simply ask whether there is any row at all. A parameter rule holds compares one column of the first row of the table wired into the params port with a value. When the gate is closed nothing is output: every node after it is marked skipped, with the Title in the badge, and the run still succeeds. A closed gate is not an empty table, so a sink behind it writes nothing rather than an empty file.

Inputs and outputs

Settings

{
  name?: string
  mode: "data"|"control"|"param"
  conditions: [{ column, operator: "="|"!="|">"|"<"|">="|"<="|"contains"|"is null"|"is not null"|"is empty"|"is not empty", value, rhsKind?: "value"|"column"|"expression", rhsColumn?, rhsExpression? }]
  combinator: "AND"|"OR"
  paramColumn?: string
  paramOperator?: string
  paramValue?: string
}

Example

Publish only when the extract has rows: Open when set to the data has matching rows, no conditions, and a Write to DB after the gate. To send an alert only on big days, add a condition amount > 10000 instead.

More in Data quality and flow control

• Assert — Checks the data against rules, such as no blanks, unique keys, allowed values or freshness, and stops the run if one fails. Data that passes flows through unchanged.
• Expectations — Runs a set of quality rules and outputs a scorecard with one row per rule, its failed count and pass rate. Unlike Assert, it never stops the run.
• Wait For — Holds a branch back until other steps have finished, then passes its data on. For example, send a Slack message only once the database write is done.
• Compare Tables — Lines up two versions of a table by key columns and puts each value side by side, flagging every cell that changed. A row on only one side shows blanks for the other.