Shape · Text
Tokenize
Finds every match of a regular expression in a text column, such as each number in 'a12 b34'. It spreads the matches across columns, gives each its own row, or turns each match's capture groups into columns.
Also known as: Extract all matches, text to columns
How it works
Pick the Column to tokenize and a Pattern (RE2, like Regex Extract), optionally Case-insensitive. Output chooses the shape. To columns replaces the column with <column> 1 to <column> N, where N is the most matches any row has or the Max columns you set; a row with more matches than that loses the rest and the run warns. To rows gives one row per match in the column's place; a row with no match disappears unless Keep rows with no match (with a NULL token) is ticked. Groups gives one row per match and one column per capture group, named groups keeping their names and the others numbered; a row with no match keeps one row of nulls.
Inputs and outputs
Inputs
data— Table
Outputs
data— Table
Settings
The Attributes panel of Tokenize, as in the app: the controls work, but nothing runs — there is no data behind it.
The same settings, as the Assistant writes them
{
column: string
pattern: string
caseInsensitive: boolean
mode: "columns"|"rows"|"groups"
maxColumns?: number
atLeastOneRow: boolean
}pattern = an RE2 regex (DuckDB regexp_extract_all: no lookaround, no backreferences), e.g. "\d+" for every number. columns: every match into "<column> 1..N" in the column's place; rows: one row per match (atLeastOneRow keeps unmatched rows with a NULL); groups: one row per match, one column per capture group (named groups keep their names)
Example
A codes column holds a12 b34. Pattern \d+ with To columns gives two columns; with To rows it gives two rows.
| codes 1 | codes 2 |
|---|---|
| 12 | 34 |