Input · Folders and loops

Directory

Lists the files in a folder, one row per file, instead of reading their rows. Wire it into a loop to process every file in turn.

Also known as: Batch of files, dynamic input

How it works

Add a folder from Add data source → Folder and drop it on the canvas; the node lists the files under it rather than reading them. File glob selects which files become rows; the default **/*.{csv,tsv,parquet,...} matches every format a loop can read, recursively. .json and .txt are left out on purpose, since folders often carry sidecar manifests and notes; add json to the glob when the folder holds JSON data. Each row has path, name, ext, size and mtime. Wire the node into a Loop's items port, and the Current item inside the loop reads each file.

Inputs and outputs

Settings

Example

A folder of monthly exports with a README.txt beside them. File glob **/*.csv lists only the exports:

More in Folders and loops

• S3 Directory — Lists the objects in an S3 location that match a pattern, one row per object, so a loop can process each one.
• DB Directory — Lists the tables in a database connection, one row per table, so a loop can run once per table. You can filter the list by name.
• Sheets — Lists the sheets of an Excel workbook, one row per sheet, so a loop can read each sheet in turn.
• Current item — Sits inside every loop and gives you the item the loop is on right now, such as the current file read as a table. It is added for you and can't be removed.
• File — Reads the file whose path sits in a column of its input: Parquet, CSV, JSON, Excel or a map file, on disk or in S3. Wire a loop's Current item into it to read a different file each time.