Input · Web and APIs

Web Page

Reads a web page or a saved .html file. Add it with Add data source, then choose what to read: a table, repeating items such as product cards or list entries, contact details, or the whole page as text. It can follow "next" links or a URL pattern to read several pages, and render a page built with JavaScript before reading it.

Also known as: Web scraping, scrape a table, Wikipedia

How it works

Add it from Add data source → Web Page with the page's address, or a saved .html file. Read chooses what the page gives you: A table (Find tables lists the page's tables and you pick one, followed afterwards by its caption, heading or headers), Repeating items (Pick items… on the page itself, one row per card or list entry), Contact details (addresses, phones and emails, with a column ready for Geocode) or Whole page (one row with page_text; Keep the HTML too adds page_html and Keep failed pages as rows adds fetch_error). Pages reads one page, follows "next" links or a URL pattern; when a read stops at Max pages with more pages to follow, the row offers Read N more. A copy of the page is saved; Refresh from web fetches it again. For a page that builds its content with JavaScript, tick Render with JavaScript: in Add data source, in the panel, or from Find tables and Pick items… when they find nothing. Daedalus then loads the page in a hidden, sandboxed window, waits until it goes quiet (or, under More, until the CSS selector in Wait for appears), for up to 20 seconds, and saves that rendered copy for every Read mode. Page can be bound to a column of the Current item inside a loop, or, outside a loop, to a column of any table wired into the reader, whose first row gives the address.

Inputs and outputs

Settings

{ url: string, table?: string (words from the table's caption, heading or column names, e.g. "cities by population"), read?: "table" | "contacts" | "page" }  // contacts = addresses / phones / emails on the page, with a Geocode-ready address column; page = one row with the page's visible text (page_text), for text that is not in a table

Example

A Wikipedia list of cities. Read: A table, Find tables, pick "cities by population". The panel then shows the table it follows and Change table… if the page changes.

More in Web and APIs

• REST Fetch — Pulls data from a web address into a table. Enter the URL and press Generate fetch, and the code that fetches and parses the response is written for you.
• Extract from HTML — Reads pages you already have as HTML in a column, such as a Web Page source's page HTML or stored email bodies, the way the Web Page source reads a page: a table, repeating items, contact details, the text or the links.