Sample code for extracting, transforming, and loading IA metadata into BaseX

100%

Sample code for extracting, transforming, and loading IA metadata into BaseX

✓ Published0🌍 Public

Last edited Nov 4, 2016

Created on Oct 20, 2016

This example demonstrates an XQuery 3.1 pipeline for harvesting Open Library search results and persisting them as structured XML in a BaseX database. It first fetches JSON via `fetch:text` and parses it with `json:parse`, then converts each record into a canonical `book` element with attributes for key and full-text availability, and child elements for title, authors, dates, publishers, and subjects. Subsequent scripts extend the process to paginate through large result sets, and a final query retrieves full book texts from the Internet Archive, storing them alongside the metadata.

AI-generated description

Similar vizzes

Loading thumbnail…

RESTXQ example using map:find with BaseX 6.0 beta

This example demonstrates a RESTXQ service that queries the Open Library API for book titles and returns matching results as XML. The code defines an XQuery function annotated with `%rest:path` and `%rest:GET`, which receives a book title and a key parameter. It constructs a URL to the Open Library search endpoint, retrieves the JSON response using `fn:json-doc`, then flattens and searches the resulting structure with `array:flatten` and `map:find` from BaseX 6.0. The matched items are wrapped in a custom `results` element.

CCliffordAnderson

69% match

Loading thumbnail…

Code snippets for XQuery Working Group (Text Mining at Scale)

This example demonstrates a text-mining workflow for querying the Victorian Women Writers Project corpus stored in a BaseX database, using XQuery and XPath Full Text 1.0. It walks through creating structured records from TEI XML, saving them to a new database, and running full-text searches with scoring, stemming, and thesaurus-based matching. The code exercises functions like `db:replace`, `ft:mark`, and `ft:extract` to highlight matching text, while also showing stop-word filtering and boolean operators for queries.

CCliffordAnderson

67% match

Loading thumbnail…

XQuery Samples for Blue Mountain Project

This example shows a sequence of XQuery scripts that analyze the language metadata of TEI-encoded journals from the Blue Mountain Project, a digital archive of avant-garde publications. The scripts query a collection called `bluemountain-transcriptions` using the `fn:collection` function, counting documents, extracting distinct languages from `tei:langUsage`, and identifying journals with multiple `tei:textLang/@mainLang` attributes. Later scripts partition documents by language, using `db:create` to store French texts in a new database, and verify the results by listing the URIs of the partitioned files. The code relies on the XQuery 3.1 standard and the eXist-db database API for collection access and database creation.

CCliffordAnderson

66% match

Loading thumbnail…

BaseX Server

This example shows a minimal RESTful web application built with BaseX, a native XML database, that serves XHTML pages for a poetry corpus. The code defines four GET endpoints—root, `/about`, `/credits`, and `/poem/{name}`—each returning a simple HTML page via the `view:display-contents` function. The implementation uses BaseX’s RESTXQ annotations (`%rest:path`, `%rest:GET`) and XQuery modules (`api.xqm`, `model.xqm`, `view.xqm`). The `model:credits` function supplies static HTML content, while the view module constructs the page skeleton. Data is hardcoded in XQuery, with no external datasets or JavaScript libraries involved.

CCliffordAnderson

64% match

Loading thumbnail…

Convert JSON to Dublin Core in XML

This example shows a live conversion of movie search results from JSON into Dublin Core XML metadata. It retrieves data from the OMDB API using `fetch:text` and `fn:parse-json` to parse the response, then extracts the `Search` array. For each movie, it creates a Dublin Core `title` and `relation` element, mapping the movie title and poster URL respectively. The code uses XQuery 3.1 with the `dc` namespace, and the final output is an XML `<movies>` wrapper. The example demonstrates a straightforward data transformation pipeline, not a visual chart, but it is included in the gallery for its metadata-handling technique.

CCliffordAnderson

64% match

Loading thumbnail…

XQuery Blue Mountain Project - Part III

This example shows a pipeline for transforming a single TEI XML document from the Blue Mountain Project—a digitized 1920 issue of the French journal *Action*—into clean, article-level TEI files. The first script removes facsimile elements, end-hyphenation markers, and line breaks, replacing `<lb>` elements with spaces. A second script extracts metadata and article content into a fresh TEI template, while a third builds individual TEI files for each article. The code uses XQuery 3.1 with the `fn:doc` API for document access and node operations like `copy`, `modify`, and `replace node`.

CCliffordAnderson

64% match

Loading thumbnail…

Full Text IA

This example visualizes the full-text metadata of a book collection, showing how documents are distributed across subject categories. It loads book records from an XML database and filters out any documents tagged with the "Protected DAISY" subject. The visualization uses D3.js’s force-directed layout to position nodes, with each book represented as a point linked to its subject keywords. The code retrieves data via an XQuery query, and the rendering relies on D3’s force simulation to cluster related items, making thematic groupings visible at a glance.

CCliffordAnderson

63% match

Loading thumbnail…

XQuery for Partitioning the Blue Mountain Project

This example demonstrates a pipeline for cleaning, partitioning, and exporting articles from the Blue Mountain Project’s TEI transcriptions into language-specific databases. It shows how XQuery 3.1 scripts transform raw periodical XML by removing facsimile and hyphenation artifacts, splitting documents into individual articles, and grouping them by `mainLang` attributes into new databases. The workflow then exports French-language texts as plain .txt files, and R code using the `tm` and `SnowballC` libraries performs text mining, including term-document matrices and stemming.

CCliffordAnderson

63% match