Skip to main content
100%

Merging an entire directory of data and loading it to CartoDB

✓ Published0🌍 Public
Aandrewxhill
Last edited Jan 20, 2015
Created on Jan 15, 2015

This example shows a shell script that uploads every GPX file in a directory to an existing CartoDB table. It demonstrates how to merge an entire folder of GPS track data into a single cloud-hosted dataset. The script relies on a Python utility called `cartodb-utils.py` to handle the upload process. For each `.gpx` file, the script calls the `append` command, passing the file path, table name, API key, and CartoDB account name as arguments. This approach automates the bulk ingestion of geospatial data without manual intervention.

AI-generated description

Similar vizzes

Loading thumbnail…

command-line python interface for manipulating data on CartoDB

This example demonstrates a command-line Python tool for managing and manipulating data stored on CartoDB, allowing users to upload files, overwrite or append rows, add columns, drop tables, and export datasets. It first shows the workflow of interacting with CartoDB’s SQL and Import APIs, then explains the underlying implementation. The script uses the `requests` library for HTTP calls, `urllib2` for polling import status, and `argparse` for command-line options, translating between CartoDB’s data types and PostgreSQL types during column alterations and insertions.

Aandrewxhill
73% match
Loading thumbnail…

Batch API uploads to CARTO

This example shows how to upload large datasets to CARTO by splitting them into smaller chunks and using the SQL Batch API to merge them. It demonstrates three chained batch queries: creating a combined table with `CREATE TABLE ... UNION ALL`, dropping the `cartodb_id` column with `ALTER TABLE`, and finalizing with `cdb_cartodbfytable`. The code references CARTO’s SQL API and batch query chaining, alongside a Python script from `carto-etl` and GDAL/ogr2ogr alternatives for record-by-record inserts.

Aandrewbt
71% match
Loading thumbnail…

CartoDB CSV Bulkloader for Python

This example demonstrates a Python script that bulk-loads CSV data into a CartoDB table via the CartoDB SQL API, converting latitude and longitude columns into geometry using the CDB_LatLng function. It reads rows from a local CSV file, batches inserts of up to 25,000 rows per request, and skips records already present by checking a unique identifier against the table’s latest entry. The code uses Python’s csv, urllib2, and json modules to parse data and send HTTP POST requests, with error handling that writes failed SQL statements to a log file.

Aandrewxhill
70% match
Loading thumbnail…

Import CartoDB file with cURL

This example demonstrates how to upload a file into a CartoDB account using the command-line tool cURL, showing the complete import workflow from submission to completion. The shell script sends the file to CartoDB’s `/api/v1/imports/` endpoint, extracts the job ID from the JSON response, and then polls the import status until it reports “complete.” It uses cURL for HTTP requests and `sed` for parsing the JSON, with platform-specific regex variations for macOS (Darwin) and Linux. The script takes three arguments: the CartoDB username, API key, and the filename to import.

Aandrewxhill
65% match
Loading thumbnail…

Import CartoDB file with cURL (Mac version)

A shell script demonstrates how to import a file into a CartoDB account using cURL. It sends the file to CartoDB's import API endpoint, parses the returned job ID with sed, then polls the import status endpoint every two seconds until the state reports "complete." The script requires the CartoDB username, API key, and filename as command-line arguments.

Aandrewxhill
61% match
Loading thumbnail…

Gist 5830532

This example displays a map with a set of geographic features loaded from a CartoDB SQL query, which returns data in GeoJSON format. The visualization uses the Leaflet mapping library to render the map and a `L.geoJson` layer to draw the features. Each feature is processed with an `onEachFeature` callback that assigns a `cartodb_id` property to the corresponding Leaflet layer and adds it to a `drawnItems` layer group, which is then added to the map. The data source is the `leaflet_data` table from the osm2 CartoDB account.

Aandrewxhill
61% match
Loading thumbnail…

Downloads a double-zipped shapefile from NYC Socrata, separates it, and uploads to CartoDB

This shell script downloads a double-zipped shapefile of New York City building data from the Socrata open-data API, unpacks it, and uploads the two resulting shapefiles to CartoDB as separate map layers. It uses `curl` for all HTTP requests, `unzip` and `zip` for archive manipulation, and the CartoDB SQL and Import APIs to drop existing tables, submit each zipped shapefile, and poll job status until import completion. The script handles platform-specific `sed` syntax for Darwin and Linux.

Aandrewbt
60% match
Loading thumbnail…

quick example of using the batch api over python

This example shows how to submit asynchronous SQL jobs to CartoDB’s Batch API from Python, using a query that updates geometries by replacing them with centroids. It demonstrates the workflow of starting a job with a POST request, then polling its status via a GET request, and includes a helper to register new tables with CartoDB’s `CDB_Cartodbfytable` function. The code uses the `requests` library to handle HTTP calls, along with `urllib.quote_plus` for query encoding, and relies on the `cartodb.com` API endpoints for job creation and status retrieval.

Aandrewxhill
59% match