CartoDB CSV Bulkloader for Python

100%

CartoDB CSV Bulkloader for Python

✓ Published0🌍 Public

Last edited Jun 19, 2013

Created on Jun 19, 2013

This example demonstrates a Python script that bulk-loads CSV data into a CartoDB table via the CartoDB SQL API, converting latitude and longitude columns into geometry using the CDB_LatLng function. It reads rows from a local CSV file, batches inserts of up to 25,000 rows per request, and skips records already present by checking a unique identifier against the table’s latest entry. The code uses Python’s csv, urllib2, and json modules to parse data and send HTTP POST requests, with error handling that writes failed SQL statements to a log file.

AI-generated description

Similar vizzes

Loading thumbnail…

command-line python interface for manipulating data on CartoDB

This example demonstrates a command-line Python tool for managing and manipulating data stored on CartoDB, allowing users to upload files, overwrite or append rows, add columns, drop tables, and export datasets. It first shows the workflow of interacting with CartoDB’s SQL and Import APIs, then explains the underlying implementation. The script uses the `requests` library for HTTP calls, `urllib2` for polling import status, and `argparse` for command-line options, translating between CartoDB’s data types and PostgreSQL types during column alterations and insertions.

Aandrewxhill

77% match

Loading thumbnail…

Merging an entire directory of data and loading it to CartoDB

This example shows a shell script that uploads every GPX file in a directory to an existing CartoDB table. It demonstrates how to merge an entire folder of GPS track data into a single cloud-hosted dataset. The script relies on a Python utility called `cartodb-utils.py` to handle the upload process. For each `.gpx` file, the script calls the `append` command, passing the file path, table name, API key, and CartoDB account name as arguments. This approach automates the bulk ingestion of geospatial data without manual intervention.

Aandrewxhill

70% match

Loading thumbnail…

Batch API uploads to CARTO

This example shows how to upload large datasets to CARTO by splitting them into smaller chunks and using the SQL Batch API to merge them. It demonstrates three chained batch queries: creating a combined table with `CREATE TABLE ... UNION ALL`, dropping the `cartodb_id` column with `ALTER TABLE`, and finalizing with `cdb_cartodbfytable`. The code references CARTO’s SQL API and batch query chaining, alongside a Python script from `carto-etl` and GDAL/ogr2ogr alternatives for record-by-record inserts.

Aandrewbt

69% match

Loading thumbnail…

quick example of using the batch api over python

This example shows how to submit asynchronous SQL jobs to CartoDB’s Batch API from Python, using a query that updates geometries by replacing them with centroids. It demonstrates the workflow of starting a job with a POST request, then polling its status via a GET request, and includes a helper to register new tables with CartoDB’s `CDB_Cartodbfytable` function. The code uses the `requests` library to handle HTTP calls, along with `urllib.quote_plus` for query encoding, and relies on the `cartodb.com` API endpoints for job creation and status retrieval.

Aandrewxhill

68% match

Loading thumbnail…

Import CartoDB file with cURL

This example demonstrates how to upload a file into a CartoDB account using the command-line tool cURL, showing the complete import workflow from submission to completion. The shell script sends the file to CartoDB’s `/api/v1/imports/` endpoint, extracts the job ID from the JSON response, and then polls the import status until it reports “complete.” It uses cURL for HTTP requests and `sed` for parsing the JSON, with platform-specific regex variations for macOS (Darwin) and Linux. The script takes three arguments: the CartoDB username, API key, and the filename to import.

Aandrewxhill

67% match

Loading thumbnail…

Notes

A set of tutorial notes for CartoDB demonstrates a progression through mapping workflows, from importing CSV data with latitude/longitude coordinates to styling visualizations with CartoCSS and querying tables via SQL. The notes cover data editing, georeferencing, privacy settings, and integrating CartoDB.js for web maps, referencing external resources like colorbrewer2.org and PostGIS documentation.

Aandrewxhill

65% match

Loading thumbnail…

Fun SQL statements for CartoDB

This example shows how to explore and analyze tornado and county data in CartoDB using SQL statements. It demonstrates basic queries for filtering, sorting, and aggregating records, followed by spatial functions that inspect geometry types, compute areas, distances, and intersections. The code uses PostGIS functions such as `ST_Area`, `ST_Distance`, `ST_Intersects`, and `ST_DWithin`, alongside CartoDB-specific helpers like `CDB_LatLng`. It also includes updates to add tornado counts and density per county, highlighting how SQL can drive interactive map visualizations.

Aandrewxhill

64% match

Loading thumbnail…

Efficient bulk update and insert

This example demonstrates two efficient SQL patterns for bulk loading election results into a CartoDB table named `counties_results`. The `insert.sql` file shows how to add new rows using a single `INSERT INTO ... VALUES` statement with multiple tuples, while `update.sql` performs a bulk update by joining the table against a `FROM (VALUES ...)` clause, matching records on `st_usps` and `fips`. Both files contain identical county-level presidential vote percentage data, including Democratic, Republican, and other party shares, ready to be applied to a spatial database.

Aandrewxhill

63% match