Skip to main content
SimilarExport ZIP
100%

Text Mining at Scale (XQuery Working Group) Natural Language Processing

✓ Published0🌍 Public
CCliffordAnderson
Last edited Oct 11, 2019
Created on Oct 11, 2019

This example demonstrates how to analyze a corpus of genuine and fake news articles using XQuery and the Aylien Natural Language Processing API. It shows how to create a database of cleaned article records, list and rank news sources by veracity, and perform text analysis services such as sentiment classification, entity extraction, and automatic summarization. The code relies on XQuery 3.1 functions for data processing and HTTP requests to communicate with Aylien’s TextAPI.

AI-generated description

Similar vizzes

Loading thumbnail…

NLP with Aylien

This example demonstrates sentiment analysis by sending a sample text to the Aylien Text API, which returns hashtags summarizing the text’s tone and topics. The code constructs an HTTP request using XQuery’s `http:send-request` and the `fn:encode-for-uri` function, targeting the API’s `/hashtags` endpoint with a fixed string about a skeptical reaction to a plan. The result, formatted as XML, would display the generated hashtags but does not visualize them; instead, it focuses on the API interaction itself, relying solely on the Aylien service without any graphical rendering or charting library.

CCliffordAnderson
72% match
Loading thumbnail…

Code snippets for XQuery Working Group (Text Mining at Scale)

This example demonstrates a text-mining workflow for querying the Victorian Women Writers Project corpus stored in a BaseX database, using XQuery and XPath Full Text 1.0. It walks through creating structured records from TEI XML, saving them to a new database, and running full-text searches with scoring, stemming, and thesaurus-based matching. The code exercises functions like `db:replace`, `ft:mark`, and `ft:extract` to highlight matching text, while also showing stop-word filtering and boolean operators for queries.

CCliffordAnderson
66% match
Loading thumbnail…

XQuery Full Text

This example demonstrates the evolution of text search in XQuery, from basic string matching to full-text querying with linguistic processing. It shows how `fn:contains` and `fn:matches` perform literal pattern matching, while the XQuery Full Text 3.1 features—`contains text`, `using stemming`, `ft:mark`, and the `score` clause—enable case-insensitive, stem-aware, ranked search results. The code queries a collection of TEI-encoded texts (`vwwp_tei`) and returns matched paragraphs, with scoring wrapped in XML output and string formatting via `format-number`.

CCliffordAnderson
61% match
Loading thumbnail…

XQuery Full Text

This example demonstrates how XQuery 3.1 can perform full-text search and fuzzy string matching directly within a query. It shows four distinct text-processing capabilities: proximity-based word matching using the `contains text` operator with distance constraints, stop-word filtering, Levenshtein distance calculation via the `strings:levenshtein` function, and phonetic comparison using `strings:soundex`. The code relies on the XQuery standard library functions for string operations rather than external visualization libraries, highlighting the language's built-in text analytics power.

CCliffordAnderson
60% match
Loading thumbnail…

XQuery for Partitioning the Blue Mountain Project

This example demonstrates a pipeline for cleaning, partitioning, and exporting articles from the Blue Mountain Project’s TEI transcriptions into language-specific databases. It shows how XQuery 3.1 scripts transform raw periodical XML by removing facsimile and hyphenation artifacts, splitting documents into individual articles, and grouping them by `mainLang` attributes into new databases. The workflow then exports French-language texts as plain .txt files, and R code using the `tm` and `SnowballC` libraries performs text mining, including term-document matrices and stemming.

CCliffordAnderson
59% match
Loading thumbnail…

XQuery Blue Mountain Project - Part III

This example shows a pipeline for transforming a single TEI XML document from the Blue Mountain Project—a digitized 1920 issue of the French journal *Action*—into clean, article-level TEI files. The first script removes facsimile elements, end-hyphenation markers, and line breaks, replacing `<lb>` elements with spaces. A second script extracts metadata and article content into a fresh TEI template, while a third builds individual TEI files for each article. The code uses XQuery 3.1 with the `fn:doc` API for document access and node operations like `copy`, `modify`, and `replace node`.

CCliffordAnderson
57% match
Loading thumbnail…

For XQuery4Humanists

This example visualizes the frequency and distribution of words extracted from a TEI-encoded poem, processing the text to remove punctuation, normalize case, and tokenize it into a list. The visualization shows the resulting word list as a simple textual display, highlighting the raw output of the XQuery processing pipeline. The code uses XQuery 3.0 functions like `fn:tokenize`, `fn:lower-case`, and `fn:replace` to clean the data, pulling the source directly from a remote XML file hosted on GitHub. The rendering relies solely on the XQuery return, with no additional JavaScript or charting libraries involved.

CCliffordAnderson
56% match
Loading thumbnail…

Gist e71cb443307994e24e63465bac636d3e

This example demonstrates how to extract and summarize the content of a New York Times opinion article about regulating artificial intelligence. The notebook uses the `newspaper` library to download and parse the article, returning its title, publication date, and initial text. It then calls the `nlp()` method to generate a list of keywords and a multi-sentence summary, showing a straightforward pipeline from a URL to condensed article insights. The code relies on `Article` from `newspaper` and standard pandas imports.

GGeorgeMcIntire
56% match