Skip to main content
Log in
S

Shawn91

@Shawn91·33 public vizzes

Loading thumbnail…

[从 json 中提取 value] #dict

This example demonstrates a recursive Python function that safely extracts values from nested JSON structures using a dot-notation path like "basics.name" or "basics.1.name". It shows how the function handles missing keys, empty lists, and numeric indexes by converting string digits to integers when traversing arrays or objects. The implementation relies solely on standard Python types (dict and list indexing) and recursion, with no external libraries. The code defines `extract_field(j, field)` as the core API, making it a reusable utility for parsing JSON data from a gist source.

Jul 19, 2019
Loading thumbnail…

[提取 json 的 keys 并拼接] #dict

This example shows how to recursively extract every key path from a nested JSON structure, producing a flat list of dot-separated strings. The code defines a function `extract_keys` that traverses dictionaries and lists, appending each key or index to a running path string. It demonstrates the logic for handling string leaf nodes, which trigger the final append, and uses Python’s built-in `isinstance` checks to distinguish between dict, list, and string types. The function returns the cleaned list of full key paths, as shown in the docstring example.

Jul 19, 2019
Loading thumbnail…

[print_unicode] python2 下在命令行中正常打印嵌套在 dict/list 内的 unicode 字符 #python #string

This example shows a Python 2 utility function that makes nested Unicode characters inside dictionaries and lists print correctly to the command line. The function `print_uni` takes any string `s`, applies `repr` to capture its escaped form, and then decodes that representation using `"unicode-escape"` to reveal the actual Unicode glyphs. It relies on the built-in `repr` function and the string `decode` method, demonstrating a workaround for Python 2’s default ASCII-based printing of container structures. The code is a standalone snippet from a gist by Shawn91, focusing on a common debugging pain point.

Jul 8, 2019
Loading thumbnail…

[文件夹过滤特定文件] 从文件夹中过滤出符合特定条件的文件的路径 #python

This example demonstrates how to gather full file paths for files in a directory that match user-defined criteria. It defines a `filter_files_in_dir` function that takes a base directory, an optional custom filter function, and an optional file extension. Using Python’s `os.listdir` and `itertools.filterfalse`, it iterates over directory entries, applying the filter and a suffix check via `str.endswith`. Matching filenames are joined with the directory using `os.path.join` and returned as a list. The code shows a reusable pattern for filtering file collections without external dependencies.

May 16, 2019
Loading thumbnail…

[mongodb_safe_update] 更新 MongoDB 数据库前提取所有会进行修改的内容，并返回。确认无误后再批量修改。建议在 Jupyter 中使用。#mongodb #pymongo

This example demonstrates a safe workflow for updating MongoDB documents by previewing changes before applying them. It defines a class that copies affected documents into a temporary collection, applies the update query there, and samples a random subset of diffs for visual inspection using `pprint` and `jsondiff`. After manual review, the user confirms the batch write with PyMongo’s `bulk_write` and `ReplaceOne`, optionally cleaning up the temporary collection. The code uses `MongoClient`, `InsertOne`, and `count_documents` to manage the staged updates.

May 13, 2019
Loading thumbnail…

[create_mapping_for_normalization]将大量同义词转换为映射表，如 ibm,国际商业机器公司,ibm中国 转换为 {'国际商业及其公司':'ibm', 'ibm中国':'ibm'} #dict

This example shows a Python utility that converts a plain-text list of synonym groups into a normalization mapping dictionary, e.g., turning "ibm,国际商业机器公司,ibm中国" into `{'国际商业及其公司':'ibm', 'ibm中国':'ibm'}`. The function `create_mapping_for_normalization` reads either a file path or a list, splits each line by a configurable separator, and uses a `key` function to designate the standard term among each group. It optionally skips duplicate words to avoid overwriting existing mappings. The included test demonstrates the process with sample data containing company names in multiple languages. The code relies solely on Python’s standard library, using built-in file I/O and data structures.

May 13, 2019
Loading thumbnail…

[多个lists 元素各种组合] #array

This example demonstrates how to generate all possible combinations of elements from multiple lists, such as job attributes like `c=[1,2]`, `edu=['高中','专科']`, and `exp=['不限','1-3年']`. It uses Python’s `itertools.combinations` and `itertools.product` to produce every combination of one or more items, optionally constrained to full-length tuples when `full_length=True`. The output is a list of tuples, as shown in the commented result, which can be used for scenario enumeration or filtering in data processing pipelines.

Apr 11, 2019
Loading thumbnail…

[Print Jupyter/IPython All Cell’s Outputs] 让 Jupyter 的 cell 中所有表达式结果都显示出来 #python #jupyter

This example demonstrates how to configure a Jupyter notebook so that every expression in a cell displays its output, rather than only the final one. It sets the IPython option `InteractiveShell.ast_node_interactivity` to `"all"` within a Python script, causing the kernel to print results for each statement. The code uses the `IPython.core.interactiveshell` module and assigns the string value directly to the shell’s attribute. The behavior is shown immediately after execution, with no additional libraries or data sources involved.

Apr 8, 2019
Loading thumbnail…

[check_string_contain_relation]给定 a list of strings，检查哪些字符串元素是其他字符串元素的 substring #string

This example demonstrates a string containment check: given a list of strings, it identifies which elements are substrings of other elements in the list. The code first joins all strings using a separator and then uses recursive string searching to count occurrences. It leverages Python’s built-in `str.find` method within a custom recursive function, `_appear_at_least_n_times`, to efficiently stop searching after finding the required number of matches. The final output is a filtered list containing only those strings that appear at least twice in the combined text, as shown by the example returning `['产科']` for the input `['妇产科', '产科', '儿科']`.

Mar 4, 2019
Loading thumbnail…

[json和python对象相互转换] #python

This example demonstrates two-way conversion between JSON data and Python objects using the standard `json` module. It first shows how `json.loads` with an `object_hook` parameter transforms JSON input into a custom `JSONObject` class, allowing attribute-style access to fields like `data.name` instead of dictionary indexing. The second part illustrates the reverse process with a `serialize_instance` function that converts a `Point` class instance into a dictionary, adding a `__classname__` key to preserve type information. Both conversions rely solely on Python's built-in `json` library and basic class definitions, with no external dependencies.

Feb 19, 2019
Loading thumbnail…

[创建临时文件和文件夹] #python

This example demonstrates four Python recipes for creating temporary files and directories using the `tempfile` module. It shows anonymous temporary files via `TemporaryFile`, named temporary files with `NamedTemporaryFile` and its configurable `prefix`, `suffix`, and `dir` parameters, and temporary directories through `TemporaryDirectory`. The code also queries the system’s temporary file location with `gettempdir()`. Each use case is illustrated with a `with` block, emphasizing automatic cleanup, and the example writes and reads text to confirm functionality.

Feb 19, 2019
Loading thumbnail…

[遍历直到遇到特定值]不断遍历，直到遇到一个特定值为止 #python

This example demonstrates how to iterate over data until a specific sentinel value is encountered, using Python’s built-in `iter` function with a callable and a sentinel argument. It shows reading a text file in fixed-size chunks—10 characters at a time—via `f.read(10)` inside a lambda, stopping when an empty chunk is returned. The visualization highlights the mechanism of `iter` with a sentinel: the first parameter must be callable when a sentinel is present, unlike the standard iterable-only form. This pattern is useful for terminating loops based on data content rather than a fixed count.

Feb 18, 2019
Loading thumbnail…

[合并多个已经排序的 lists] 多个已经排序过的 lists，将这些 lists 合并并排序 #array

This example demonstrates how to merge multiple pre-sorted lists into a single sorted sequence. It uses Python's `heapq.merge` function, which accepts two sorted lists—here, `a = [1, 4, 7, 10]` and `b = [2, 5, 6, 11]`—and returns a generator yielding items in ascending order. The code then iterates through the generator with a `for` loop, printing each merged value. The key constraint is that `heapq.merge` only works correctly when all input lists are already sorted, as shown in this minimal, console-based illustration.

Feb 18, 2019
Loading thumbnail…

[flatten list]铺平 list of lists #array

This example demonstrates two ways to flatten nested lists of arrays in Python. The first method uses a lambda function with a list comprehension to produce a new flattened list. The second method defines a generator function, `flatten`, which recursively yields items from nested iterables while ignoring strings and bytes, using the `Iterable` class from the `collections` module. The example data `[1, 2, [3, 4, [5, 6], 7], 8]` is processed by the generator, printing each number in order.

Feb 18, 2019
Loading thumbnail…

[多个字典做相同操作] #dict

This example demonstrates how Python’s `collections.ChainMap` groups multiple dictionaries into a single, view-like mapping that supports standard dict operations. It shows that lookups, length checks, key enumeration, and value retrieval behave as if the dicts were merged, with the first dict taking precedence for duplicate keys. The code also illustrates that assignment and deletion operate only on the first dict, while attempting to delete a key absent from that first dict raises a `KeyError`. The example uses ChainMap to combine `a` and `b` into `c`, then prints results from both.

Feb 15, 2019
Loading thumbnail…

[对生成器进行过滤] 生成器由于无法事先知道长度，因此无法直接使用切片操作，但是可以使用 itertools.islice() 来自动遍历；也可以使用 dropwhile 来过滤开头一部分元素 #python

This example demonstrates two techniques for working with Python generators when standard slicing and filtering are not directly available. It shows how `itertools.islice()` can apply slice-like operations to a generator by traversing it, converting the result to a list to extract values. It also illustrates `itertools.dropwhile()` to skip leading lines from a file that start with `#`, reading from `/etc/passwd` until the first non-comment line. Both approaches highlight the use of iterator utilities to handle streams without knowing their length beforehand.

Feb 15, 2019
Loading thumbnail…

[批量转换 ui 至 py 文件] 将本文件同文件夹下所有 .ui 文件转换为 .py 文件 #pyqt5

This example shows a batch conversion workflow that transforms all .ui files in the current directory into .py files for PyQt5 development. It first lists every file with a .ui extension, then derives a corresponding output filename by replacing the extension with .py. The process invokes the pyuic5 command-line tool through Python's os.system call, executing the conversion for each file. The script uses the os and os.path modules for file system operations and path manipulation.

Feb 3, 2019
Loading thumbnail…

[复杂的四舍五入] 例如对大数在十位、百位上四舍五入 #number

This example demonstrates a visual note on Python’s built-in `round` function, showing both standard decimal rounding and rounding to larger place values (tens, hundreds, thousands). The code first illustrates that `round` follows banker’s rounding at the .5 boundary, where 1.5 and 2.5 both yield 2, and then showcases negative `ndigits` arguments to round an integer like 1627731 down to 1627730, 1627700, or up to 1628000. The source is a Python script (likely run in an interactive shell) that uses only the standard library’s `round` API, with no external data or plotting library involved.

Jan 31, 2019
Loading thumbnail…

[split pandas cell into rows]将一个包含字符串或 list 的 cell，使用特定字符，将 cell 分成多行， 同行其他 cells 复制值 #pandas

This example shows a pandas data-wrangling technique that transforms a DataFrame by splitting a single cell containing a delimited string or list into multiple rows while duplicating the other column values across the new rows. The function `splitDataFrameList` accepts a DataFrame, the target column name, and a separator string, then uses `df.apply` along the rows and a helper function to parse each cell. It relies on the `collections.Iterable` check and the `str.split` method to separate values, ultimately constructing a new DataFrame from the accumulated row dictionaries via `pd.DataFrame`.

Jan 25, 2019
Loading thumbnail…

[find matching parenthesis]找到字符串中每个对称括号的位置。对于不对称的 () 会忽略 #python #string

This example visualizes the positions of matching parentheses in a string by mapping each opening parenthesis to its corresponding closing parenthesis. It shows how a stack-based algorithm can track nesting, returning a dictionary where keys are indices of `(` and values are indices of the matching `)`. Asymmetric parentheses are ignored. The code uses Python’s built-in `enumerate` to iterate through characters, a list `pstack` as a stack, and a dictionary `toret` to store results. No external libraries or rendering are involved; the output is a plain data structure.

Jan 25, 2019
Loading thumbnail…

[提取字符串中英文部分] #string

This example demonstrates a Python utility that splits a mixed Chinese-English string into its Chinese and English components, returning them as a tuple. It shows how the function identifies contiguous Chinese characters (including punctuation and spaces) from either the start or end of the input, then isolates the remaining text as the English portion. The code relies on the `re` module’s compiled patterns to match Chinese ranges (`\u4e00-\u9fa5`) and an English punctuation set, with fallback logic to handle cases where the Chinese segment is too short. The implementation includes configurable thresholds for minimum lengths of both extracted parts.

Jan 25, 2019
Loading thumbnail…

[统计mongodb 中某个 key 对应的 value 的出现频数] 统计一个 mongodb 数据库的 collection 中，所有 documents 内，统计某个特定字段的不同值各自出现次数。 #mongodb #pymongo

This example shows how to count the frequency of distinct values for a specified field across all documents in a MongoDB collection. The code defines a Python function using PyMongo’s `aggregate` method with MongoDB aggregation pipelines. It handles both simple fields and fields containing arrays of subdocuments, where a subfield is specified. The function filters documents with `$match`, unwinds arrays with `$unwind`, groups by value with `$group`, and sorts results by count using `$sort`.

Jan 25, 2019
Loading thumbnail…

[百度/维基百科爬虫]给定一个词条，去百度和维基百科中直接查找 #爬虫

This example demonstrates a Python web crawler that queries Baidu Baike and Chinese Wikipedia for a given search term, falling back to Google Search and the English Wikipedia to locate a corresponding Chinese article. The script uses `requests` with custom user-agent headers and proxy settings, and `BeautifulSoup` (`bs`) to parse HTML. It fetches Baidu pages directly when available, and for Wikipedia, it locates the English entry via Google, extracts the Chinese interlanguage link, and retrieves that page. The visualization shows the logic and branching of these HTTP requests.

Jan 25, 2019
Loading thumbnail…

[百度百科简介内容解析] #爬虫

This example parses the summary table from Baidu Baike encyclopedia pages to extract short names, abbreviations, and foreign-company division names. It uses the BeautifulSoup library to locate the HTML element with class `basic-info cmn-clearfix`, then a helper function `get_baike_shorten_names` finds a definition term (like `dt` for “英文名”) and retrieves its sibling `dd` value, cleaning out bracketed reference numbers with a regular expression. The code demonstrates a reusable scraping pattern for structured infobox data from Baidu Baike, returning the cleaned text or an empty string when no match is found.

Jan 25, 2019
Loading thumbnail…

[remove duplicates while preserving order] #array

This example demonstrates a Python function that removes duplicate elements from a sequence while preserving the original order of the remaining items. The function `remove_duplicates_while_preserving_order(seq)` uses a set (`seen`) to track encountered values, filtering the input via a list comprehension. The logic relies on the fact that `not seen.add(x)` evaluates to `True` only for unseen items, allowing the conditional expression to add new elements to the set while keeping the output in the same relative order as the input. The code shows a common Python idiom for deduplication without relying on external libraries.

Jan 25, 2019
Loading thumbnail…

[返回数组中 k 个最大值和最小值] #array

This example demonstrates how to extract the k largest and smallest values from a collection using Python’s `heapq` module. It first applies `heapq.nlargest` and `heapq.nsmallest` directly to a list of integers, showing the top and bottom three elements. The code then illustrates the use of a `key` parameter to sort a list of dictionaries by a specific field, such as `price`, to identify the cheapest and most expensive items in a portfolio dataset.

Jan 25, 2019
Loading thumbnail…

[中文标点转英文] #string

This example demonstrates a Python utility that converts Chinese punctuation marks into their English equivalents in a given string. The function `punc_chinese2english` builds translation tables using the built-in `str.maketrans` method, mapping a set of characters such as full-width commas, periods, and quotation marks to standard ASCII punctuation. It applies this mapping via the string `translate` method, returning the converted text. The code explicitly lists the Chinese and English punctuation pairs, making the transformation rule clear and easy to modify.

Jan 25, 2019
Loading thumbnail…

[匹配大小写替换]使用正则，在无视大小写的情况下，匹配字符串 A 并替换为 B，同时又希望 B 的大小写与 A 保持一致 #string

This example demonstrates how to use Python’s `re.sub` with a callback function to replace a string while preserving the original capitalization style. The `matchcase` function returns a replacement function that checks the matched text’s case—all uppercase, all lowercase, or capitalized—and applies the corresponding case to the replacement word. The code shows two calls with `flags=re.IGNORECASE`, where 'Python' becomes 'Snake' and 'PYTHON' becomes 'SNAKE', using the `re` module’s substitution API.

Jan 25, 2019
Loading thumbnail…

[注音符号字母转换] #string

This example demonstrates how to normalize Unicode text that contains accented or composite characters into plain ASCII, showing the transformation before and after processing. It uses Python’s `unicodedata` module, specifically the `normalize` function with `NFKD` and `NFD` forms, followed by encoding to ASCII with the `'ignore'` error handler, or by using a dictionary of combining characters with `translate` to strip diacritical marks. The code includes two methods: one that removes non-ASCII characters entirely, and another that preserves them while eliminating accent marks, with comments explaining the behavior for both Python 2 and 3.

Jan 25, 2019
Loading thumbnail…

[获得所有变量/属性的名称和值] #python

This example demonstrates how to retrieve all variable names and their values in Python, both at the module level and for object attributes, using the built-in `vars()` function. It shows that calling `vars()` without arguments prints the current local symbol table, including special module attributes like `__name__` and `__package__` in addition to user-defined variables such as a dictionary `a`. The code then applies `vars()` to an instance of a custom `Info` class, outputting its attribute dictionary `{'name': 'John'}`. The example relies solely on Python’s standard library, explicitly using `vars()` to inspect namespaces.

Jan 25, 2019
Loading thumbnail…

[循环过程中记录最后/开头 N 次循环结果]在一篇文档中搜索包含特定关键词的行，保存文档最开头包含特定关键词的几行，或文档结尾包含特定关键词的几行 #python #array

This example demonstrates how to preserve only the last N occurrences of lines matching a keyword while searching through a document, yielding each matching line along with the saved history. The code uses Python’s `collections.deque` with a `maxlen` parameter to automatically drop the oldest entries once the buffer exceeds the specified size. A generator function `search` iterates over the input lines, checks for the keyword pattern, and yields both the current match and the deque containing previous matches. The example input `['abc', 'bcd', 'hahaha', 'acd', 'cef']` is processed with a history limit of 3, producing a final deque showing only the last three matched lines.

Jan 20, 2019
Loading thumbnail…

[远程数据库导出 Mongodb] #mongodb

This example demonstrates a command-line workflow for exporting a remote MongoDB collection to a local JSON file. The script uses the `mongoexport` utility with explicit `--username`, `--password`, `--host`, `--port`, and `--authenticationDatabase` parameters to authenticate against the `admin` database, then extracts documents from the `spider_resume_test` collection in the `spider` database. It also applies `--skip` and `--limit` options to paginate the export, writing the results to `cv.json` on the desktop. The command shows a direct CRUD operation without any visualization library; the output is a raw data file rather than a rendered chart.

Oct 29, 2018
Loading thumbnail…

[一次性导入多个 json 到 mongodb] #mongodb

This example demonstrates how to batch-import multiple JSON files into a MongoDB database using shell scripting and the `mongoimport` command-line tool. It shows a loop that iterates over all `.json` files in the current directory, executing `mongoimport` for each one with a specified database name and collection. The script relies on `ls -1` to list files and a `while read` loop to process them sequentially, passing each filename to `mongoimport`. The visualization highlights the simplicity of automating bulk data ingestion without additional libraries or APIs.

Oct 25, 2018