Skip to main content
SimilarExport ZIP
100%

[百度/维基百科爬虫]给定一个词条，去百度和维基百科中直接查找 #爬虫

✓ Published0🌍 Public
SShawn91
Last edited Jan 25, 2019
Created on Jan 9, 2019

This example demonstrates a Python web crawler that queries Baidu Baike and Chinese Wikipedia for a given search term, falling back to Google Search and the English Wikipedia to locate a corresponding Chinese article. The script uses `requests` with custom user-agent headers and proxy settings, and `BeautifulSoup` (`bs`) to parse HTML. It fetches Baidu pages directly when available, and for Wikipedia, it locates the English entry via Google, extracts the Chinese interlanguage link, and retrieves that page. The visualization shows the logic and branching of these HTTP requests.

AI-generated description

Similar vizzes

Loading thumbnail…

[百度百科简介内容解析] #爬虫

This example parses the summary table from Baidu Baike encyclopedia pages to extract short names, abbreviations, and foreign-company division names. It uses the BeautifulSoup library to locate the HTML element with class `basic-info cmn-clearfix`, then a helper function `get_baike_shorten_names` finds a definition term (like `dt` for “英文名”) and retrieves its sibling `dd` value, cleaning out bracketed reference numbers with a regular expression. The code demonstrates a reusable scraping pattern for structured infobox data from Baidu Baike, returning the cleaned text or an empty string when no match is found.

SShawn91
75% match
Loading thumbnail…

use chrome to crawl

This example demonstrates automated web crawling and interaction using Chrome's remote interface. The code navigates through URLs from a seed list, captures page HTML and window environment variables, and programmatically clicks all elements with click event listeners. It manages JavaScript dialogs and navigation timeouts, while logging errors. It relies on Chrome's Remote Interface (CDP), along with Bluebird for promise handling, and custom detectors for JS library identification.

NN0taN3rd
50% match
Loading thumbnail…

Gist 58c79c81e9a950a0cdafc7d2d0275742

This example shows a dynamic, browser-automated web-scraping workflow that captures rendered page content. It uses Selenium’s `webdriver` to launch a Chrome browser, wait for elements via `WebDriverWait` and `expected_conditions`, and then extracts page HTML with BeautifulSoup. The script programmatically sends keyboard inputs via `Keys`, and also employs the `requests` library for direct HTTP calls. It parses the returned data with `json`, timestamps actions with `datetime`, and structures results into a pandas DataFrame. The process includes randomized delays (`random` and `time`) to mimic human browsing behavior, though no explicit visualization or rendering library is used beyond the raw captured markup.

AArizonaTay
48% match
Loading thumbnail…

Gist 2559ad6fd17831c9259a8af6f38ec027

This example filters a Common Crawl index file to identify archived web pages. It reads a CDXJ file using the `read_cdxj.py` script, which parses each line with the `CDXObject` class from `pywb.warcserver.index.cdxobject`. For each record, it checks whether the MIME type contains "html" and the HTTP status code equals 200, then prints the corresponding URL. The visualization shows which URLs meet these criteria, highlighting successful HTML captures from the crawl data.

NN0taN3rd
48% match
Loading thumbnail…

Requesting data from the Github API

This example demonstrates a simple user-driven data request to the Wikipedia API. It shows how a text input and submit button trigger a `fetch` call to the opensearch endpoint, retrieving and displaying search results based on the submitted subject. The code uses the browser’s built-in `fetch` API to request JSON data, then logs the response and result titles to the console via chained `.then()` promises. The interface includes a loader placeholder and a result article, though the visible rendering of the results is minimal, focusing on the request-response flow.

LLaurens
46% match
Loading thumbnail…

HTTP GET request in Python

Three examples demonstrate HTTP GET requests in Python using the `requests` library. The first queries the OMDb API for movie data, decoding JSON responses into dictionaries and printing key-value pairs. The second fetches an HTML page from python.org and parses it with BeautifulSoup to prettify the markup. The third retrieves documentation HTML from DataCamp and prints the raw text. Together, they show JSON and HTML handling, with sources from OMDb, python.org, and DataCamp.

FFrieseWoudloper
46% match
Loading thumbnail…

Research patterns for finding information online

This example maps a decision tree for accessing paywalled or restricted research information. It presents a flowchart with a starting question about plain-text access, branching to external services like the Wayback Machine, archive.is, Library Genesis, Bing’s cached pages, and Sci-Hub, the last requiring a DOI via Crossref search. The visualization is built using Mermaid’s graph syntax with embedded HTML hyperlinks in node labels. The graph clearly shows alternative retrieval routes while the accompanying text warns about legal and ethical considerations for reuse.

JJastor11
44% match
Loading thumbnail…

WGET example

The visualization shows a network of hyperlinks and page relationships from a website crawl, using a force-directed graph layout. It employs D3.js’s `d3.forceSimulation` to position nodes (representing pages and resources) and `d3.link` to draw edges, with node size reflecting in-degree. The example is generated from a Wget command (`wget -mkEpnp http://example.org`) that mirrors a site, and the code parses the resulting HTML to extract links and build the graph. Data source is the locally mirrored site structure, rendered via SVG with transparent node fills and gray links, emphasizing connectivity patterns without animation or interaction.

HHugoberry
44% match