Code snippets for XQuery Working Group (Text Mining at Scale)

100%

Code snippets for XQuery Working Group (Text Mining at Scale)

✓ Published0🌍 Public

Last edited Oct 4, 2019

Created on Oct 2, 2019

This example demonstrates a text-mining workflow for querying the Victorian Women Writers Project corpus stored in a BaseX database, using XQuery and XPath Full Text 1.0. It walks through creating structured records from TEI XML, saving them to a new database, and running full-text searches with scoring, stemming, and thesaurus-based matching. The code exercises functions like `db:replace`, `ft:mark`, and `ft:extract` to highlight matching text, while also showing stop-word filtering and boolean operators for queries.

AI-generated description

Similar vizzes

Loading thumbnail…

Gist ccc08dae693660d7aa1b8cf2708fd13d

This example demonstrates how to query and highlight text passages in the Victorian Women Writers Project TEI corpus using XQuery 3.1. It shows a progression from simple full-text searches for repeated terms to fuzzy matching via `ft:search` and `ft:extract`, and finally visualizes results by marking hits with `ft:mark` and segmenting them into tumbling windows. The code relies on BaseX’s full-text and windowing capabilities to extract and structure paragraph-level matches containing “garden” or “house.”

CCliffordAnderson

81% match

Loading thumbnail…

XQuery Full Text

This example demonstrates the evolution of text search in XQuery, from basic string matching to full-text querying with linguistic processing. It shows how `fn:contains` and `fn:matches` perform literal pattern matching, while the XQuery Full Text 3.1 features—`contains text`, `using stemming`, `ft:mark`, and the `score` clause—enable case-insensitive, stem-aware, ranked search results. The code queries a collection of TEI-encoded texts (`vwwp_tei`) and returns matched paragraphs, with scoring wrapped in XML output and string formatting via `format-number`.

CCliffordAnderson

75% match

Loading thumbnail…

Gist 162dbae87c5cab9f59fd433e235bafb2

This example demonstrates how to extract and collate poems from the Victorian Women Writers Project, a corpus of TEI P5-encoded texts. It shows the process of filtering documents by genre, retrieving individual poems, and normalizing their text. The code uses XQuery 3.1 with the BaseX database and its `fn:collection`, `fn:doc`, and `fn:normalize-space` functions. It outputs the results as CSV using BaseX's serialization options, specifying comma separators and headers, while converting poem text to lowercase.

CCliffordAnderson

72% match

Loading thumbnail…

XQuery Full-Text Examples

This example demonstrates XQuery Full-Text search and scoring over the TEI-encoded poems of *Les Fleurs du Mal*. It shows basic text matching with the `contains text` operator, then extends to counting results via `ft:count()`, highlighting matches with `ft:mark()`, extracting snippets with `ft:extract()`, and ranking hits by relevance using `ft:score()` with descending order. The final `search.xqy` combines scoring, marking, and base-URI display, filtering out a specific document. The code uses the XQuery 3.1 `=>` arrow operator for function chaining, and relies on the `fn:collection()` function to access the poem corpus.

CCliffordAnderson

71% match

Loading thumbnail…

XQuery Full Text

This example demonstrates how XQuery 3.1 can perform full-text search and fuzzy string matching directly within a query. It shows four distinct text-processing capabilities: proximity-based word matching using the `contains text` operator with distance constraints, stop-word filtering, Levenshtein distance calculation via the `strings:levenshtein` function, and phonetic comparison using `strings:soundex`. The code relies on the XQuery standard library functions for string operations rather than external visualization libraries, highlighting the language's built-in text analytics power.

CCliffordAnderson

70% match

Loading thumbnail…

XQuery for Partitioning the Blue Mountain Project

This example demonstrates a pipeline for cleaning, partitioning, and exporting articles from the Blue Mountain Project’s TEI transcriptions into language-specific databases. It shows how XQuery 3.1 scripts transform raw periodical XML by removing facsimile and hyphenation artifacts, splitting documents into individual articles, and grouping them by `mainLang` attributes into new databases. The workflow then exports French-language texts as plain .txt files, and R code using the `tm` and `SnowballC` libraries performs text mining, including term-document matrices and stemming.

CCliffordAnderson

69% match

Loading thumbnail…

Exploring your Collection

This visualization shows a collection of TEI-encoded texts by querying their titles and full-text content, highlighting matches for terms like “vin” or co-occurring words like “mal” and “je” within the same sentence. It first demonstrates how to list all titles in the collection, then filters those titles using XQuery Full-Text expressions via the `contains text` operator. The code uses the BaseX or eXist-db XQuery 3.0 engine, leveraging the `fn:collection()` function to access documents and the TEI namespace for element traversal, with no external visualization libraries—the output is plain text.

CCliffordAnderson

69% match

Loading thumbnail…

XQuery Samples for Blue Mountain Project

This example shows a sequence of XQuery scripts that analyze the language metadata of TEI-encoded journals from the Blue Mountain Project, a digital archive of avant-garde publications. The scripts query a collection called `bluemountain-transcriptions` using the `fn:collection` function, counting documents, extracting distinct languages from `tei:langUsage`, and identifying journals with multiple `tei:textLang/@mainLang` attributes. Later scripts partition documents by language, using `db:create` to store French texts in a new database, and verify the results by listing the URIs of the partitioned files. The code relies on the XQuery 3.1 standard and the eXist-db database API for collection access and database creation.

CCliffordAnderson

68% match