Web scraping for Summer Teams

100%

Web scraping for Summer Teams

✓ Published0🌍 Public

Last edited Aug 6, 2020

Created on Aug 6, 2020

This example shows how to extract participant names from Vanderbilt University Library’s summer project listings and reformat them into CSV rows of project–participant pairs. It scrapes the HTML page using `fetch:text` and `html:parse`, then navigates the DOM with XPath to select project headings and member text. The code uses `fn:tokenize` and `fn:replace` to split and normalize names into a “last,first” format, with special handling for suffixes like Jr. or III. Output is generated via the XQuery `output:method` and `output:csv` declarations, producing a comma-separated table.

AI-generated description

Similar vizzes

Loading thumbnail…

XQuery to clean up data from an Excel-generated webpage & generate an EAD document

This example shows how XQuery 3.1 can clean data scraped from an Excel-generated HTML table and use it to generate an EAD (Encoded Archival Description) finding aid. The `fetch:text()` and `html:parse` functions extract rows from a legacy webpage, normalize the text content, and repackage the fields into JSON maps. A second function transforms those entries into EAD `<c02>` components, inserting metadata like titles, abstracts, and thumbnail image links from the Lantern Slide collection at Vanderbilt. Finally, the script rebuilds a complete EAD document, injecting the new `<c02>` elements into the existing archival description hierarchy, preserving the original structure while enriching it with parsed data.

CCliffordAnderson

61% match

Loading thumbnail…

Quick alphabetic sorting of Vanderbilt email addresses

This example demonstrates a straightforward method for alphabetically sorting a list of Vanderbilt email addresses extracted from a Markdown file. The code loads the email addresses using `fn:unparsed-text-lines`, then performs a case-sensitive sort based on a specific portion of the address. It applies a regular expression via `fn:replace` to isolate the username segment after the first period, and orders the results with an `order by` clause in XQuery 3.1. The output is the sorted list of emails, showing the sorting logic directly in the query.

CCliffordAnderson

60% match

Loading thumbnail…

Gist f72a3553521c158f0eb0dccbe774455e

This example demonstrates a web scraper that extracts detailed trail information from a hiking website. It shows how structured data can be programmatically retrieved from HTML pages, specifically pulling out features, overview, and description sections. The code uses the Nokogiri library to parse the document and XPath queries to locate the relevant heading elements. After extracting each section, it cleans the text by removing newlines and excess whitespace. The scraper returns a hash containing the three descriptive fields, making the data readily accessible for further processing or visualization.

WWill Carter

59% match

Loading thumbnail…

Code snippets for XQuery Working Group (Text Mining at Scale)

This example demonstrates a text-mining workflow for querying the Victorian Women Writers Project corpus stored in a BaseX database, using XQuery and XPath Full Text 1.0. It walks through creating structured records from TEI XML, saving them to a new database, and running full-text searches with scoring, stemming, and thesaurus-based matching. The code exercises functions like `db:replace`, `ft:mark`, and `ft:extract` to highlight matching text, while also showing stop-word filtering and boolean operators for queries.

CCliffordAnderson

58% match

Loading thumbnail…

Gist 162dbae87c5cab9f59fd433e235bafb2

This example demonstrates how to extract and collate poems from the Victorian Women Writers Project, a corpus of TEI P5-encoded texts. It shows the process of filtering documents by genre, retrieving individual poems, and normalizing their text. The code uses XQuery 3.1 with the BaseX database and its `fn:collection`, `fn:doc`, and `fn:normalize-space` functions. It outputs the results as CSV using BaseX's serialization options, specifying comma separators and headers, while converting poem text to lowercase.

CCliffordAnderson

58% match

Loading thumbnail…

XPath Examples - Mellon Institute

This example demonstrates how XPath queries navigate and extract specific content from a TEI-encoded document of the Mellon Institute at Vanderbilt. It shows the progression from simple absolute paths to descendant-or-self axes, then to predicate-based filtering, and finally to more complex relational paths. The code uses the XPath API to select titles, names, sex attributes, and speaker-related paragraphs, referencing the updated TEI Guidelines for the sex element. The example is presented as a gist authored by CliffordAnderson, with the source data being the structured XML markup of the institute’s records, illustrating query patterns for text extraction and attribute matching.

CCliffordAnderson

57% match

Loading thumbnail…

Gist 5ea363dfe112d08281a8c89a3f31bcff

This example scrapes trail data from a website, extracting features, overview, and description sections into a structured hash. It uses Nokogiri to parse the HTML and XPath queries to locate the relevant headings and their adjacent content, cleaning the text by removing line breaks. The code demonstrates a straightforward scraping pattern for capturing semantic sections, though it does not include visualization logic or rendering libraries. The data source is a single trail page URL, and the output is a formatted description ready for further processing or display.

WWill Carter

57% match

Loading thumbnail…

XQuery Working Group Challenge

This example transforms the PROIEL treebank’s Latin New Testament XML into a sequence of verse elements, each tagged with its citation and concatenated token forms. It fetches the remote XML via `fn:doc`, extracts distinct citation parts, and filters tokens by matching `@citation-part` to build structured verses. The code uses XQuery 3.1’s element constructors and `fn:data` to output text, demonstrating a direct pipeline from hierarchical treebank data to flat verse-like text.

CCliffordAnderson

57% match