Shape · Columns

Audit Stamp

Adds columns that record when the row was loaded, from which source, and in which run. Use it before writing to a warehouse so you can trace every row.

Also known as: Lineage, batch ID

How it works

Under Columns to add all four columns are ticked by default: _loaded_at, a UTC timestamp of when the node ran; _loaded_date, its date; _source, the text in _source value or, when that is blank, the name of the drone the rows came from (null if none can be found); and _batch_id, an id of the form batch-<UTC time>-<hex> minted once per execution. Every row of one execution gets the same timestamp and batch id, and a later execution mints new ones. Existing columns are never changed; a stamp column whose name the input already has is added with a _1 suffix.

Inputs and outputs

Settings

{
  columns?: ("_loaded_at"|"_loaded_date"|"_source"|"_batch_id")[]
  source?: string
}

Example

You load a daily orders file into a warehouse table and want to trace each row later. Put Audit Stamp just before the writer, set _source value to orders_daily, and each run appends rows that share one _batch_id and _loaded_at, so a bad load can be found and removed by its batch.

More in Columns

• Select Columns — Keeps only the columns you pick and drops the rest.
• Rename — Gives columns new names. Type old-to-new pairs, or let the data decide at run time: take the names from the first row (which is then dropped), from a names table on a second input, or from an expression over each old name.
• Cast Types — Changes a column's type, such as text to number or text to date. You choose what happens to a value that won't convert: stop the run, or make it blank with or without a warning.
• Parse — Turns messy text into real values: dates and times, numbers with separators or currency signs, true/false, durations, measurements with units, and coordinates.
• Add Column — Adds a column worked out by a formula, a fixed value, today's date or the current time, or a parameter. For example, margin = Price − Cost. Excel's finance functions such as PMT work here by name; see Formulas and finance functions.
• Multi-field Formula — Applies one formula to many columns at once, such as rounding every price column. Overwrite the columns or write new ones beside them.
• Multi-row Formula — Works out a column from the current row and its neighbours, including the column's own value on the previous row. Use it for fill-down, running totals or carrying the last value forward.
• Fill Missing — Replaces blanks with a fixed value, an empty string, or the previous or next value, or drops those rows. It can also drop rows or columns that are entirely empty.
• Replace Values — Swaps text in a column for other text, such as 'N/A' to blank or 'NY' to 'New York'. Each rule replaces its text wherever it appears in a value, as plain text or a pattern.
• Replace via Mapping — Swaps values in one or more columns using a second table of from → to pairs, such as a code list turning 'NY' into 'New York'. Values with no match stay as they are, or fail the run if you choose.
• Interpolate NULLs — Fills blank numbers by estimating from the known values around them, in a straight line or by taking the nearest one. For example, missing sensor readings.
• Clip Values — Caps numbers at a lower and/or upper bound. Anything below becomes the minimum, anything above becomes the maximum.
• Bin — Turns a number into a labelled band, using your own break points or equal-width buckets. For example, ages into 0–17, 18–64 and 65+.
• Format Table — Sets how columns print in previews and reports: currency, percent, decimals, thousands separators, date masks, alignment and hidden columns. The values themselves don't change.
• Format to Text — Turns a number, date or time column into formatted text: thousands separators, fixed decimals, currency or a date pattern. The pattern can come from another column, row by row. Unlike Format Table, this changes the data.