Skip to main content
100%

download-hf-dataset-as-jsonl.ipynb

✓ Published0🌍 Public
CCliffordAnderson
Last edited Jun 30, 2025
Created on Jun 30, 2025

This example demonstrates an ETL pipeline that downloads a HuggingFace dataset of Wikipedia stubs about women in religion and converts it into JSONL files formatted for OpenAI fine-tuning. It shows how to transform raw dataset splits into the chat-based structure with system, user, and assistant roles. The notebook uses the `datasets` library’s `load_dataset` and `to_json` methods to handle training and validation splits separately, saving the output to `/content/`. A custom `create_openai_format` function then wraps each example in the conversational structure required by OpenAI’s API.

AI-generated description

Similar vizzes