Skip to main content
SimilarExport ZIP
100%

7z vs zip avec données IGN

✓ Published0🌍 Public
TThomasG77
Last edited Apr 5, 2024
Created on Apr 5, 2024

This example compares the storage size and remote read performance of 7z versus zip compression for French geographic data from the IGN ADMIN-EXPRESS-COG dataset. It shows that while 7z produces a smaller file (562M vs 693M), zip is vastly faster for remote consumption, with `ogrinfo` reading a shapefile in 0.504 seconds versus 36.717 seconds via the `vsicurl` virtual filesystem. The benchmark also experiments with GeoParquet, demonstrating even faster access at 0.338 seconds. Tests use GDAL's OGR tools for shapefile and Parquet reading.

AI-generated description

7z vs zip avec données IGN

Nous avons compressé le fichier 7z

wget https://data.geopf.fr/telechargement/download/ADMIN-EXPRESS-COG/ADMIN-EXPRESS-COG_3-1__SHP_WGS84G_FRA_2023-05-03/ADMIN-EXPRESS-COG_3-1__SHP_WGS84G_FRA_2023-05-03.7z
7z x ADMIN-EXPRESS-COG_3-1__SHP_WGS84G_FRA_2023-05-03.7z
zip -r ADMIN-EXPRESS-COG_3-1__SHP_WGS84G_FRA_2023-05-03.zip ADMIN-EXPRESS-COG_3-1__SHP_WGS84G_FRA_2023-05-03

Comparaison des tailles compressées

562M	ADMIN-EXPRESS-COG_3-1__SHP_WGS84G_FRA_2023-05-03.7z
693M	ADMIN-EXPRESS-COG_3-1__SHP_WGS84G_FRA_2023-05-03.zip

Test 7z

time ogrinfo /vsi7z/vsicurl/https://labs.webgeodatavore.com/partage/ADMIN-EXPRESS-COG_3-1__SHP_WGS84G_FRA_2023-05-03.7z/ADMIN-EXPRESS-COG_3-1__SHP_WGS84G_FRA_2023-05-03/ADMIN-EXPRESS-COG/1_DONNEES_LIVRAISON_2023-05-03/ADECOG_3-1_SHP_WGS84G_FRA/EPCI.shp

INFO: Open of `/vsi7z/vsicurl/https://labs.webgeodatavore.com/partage/ADMIN-EXPRESS-COG_3-1__SHP_WGS84G_FRA_2023-05-03.7z/ADMIN-EXPRESS-COG_3-1__SHP_WGS84G_FRA_2023-05-03/ADMIN-EXPRESS-COG/1_DONNEES_LIVRAISON_2023-05-03/ADECOG_3-1_SHP_WGS84G_FRA/EPCI.shp'
      using driver `ESRI Shapefile' successful.
1: EPCI (Polygon)

real	0m36,717s
user	0m30,227s
sys	0m0,529s

Test zip

Pour lister un zip à distance

# https://github.com/marcograss/partialzip
partialzip list https://labs.webgeodatavore.com/partage/ADMIN-EXPRESS-COG_3-1__SHP_WGS84G_FRA_2023-05-03.zip

time ogrinfo /vsizip/vsicurl/https://labs.webgeodatavore.com/partage/ADMIN-EXPRESS-COG_3-1__SHP_WGS84G_FRA_2023-05-03.zip/ADMIN-EXPRESS-COG_3-1__SHP_WGS84G_FRA_2023-05-03/ADMIN-EXPRESS-COG/1_DONNEES_LIVRAISON_2023-05-03/ADECOG_3-1_SHP_WGS84G_FRA/EPCI.shp

INFO: Open of `/vsizip/vsicurl/https://labs.webgeodatavore.com/partage/ADMIN-EXPRESS-COG_3-1__SHP_WGS84G_FRA_2023-05-03.zip/ADMIN-EXPRESS-COG_3-1__SHP_WGS84G_FRA_2023-05-03/ADMIN-EXPRESS-COG/1_DONNEES_LIVRAISON_2023-05-03/ADECOG_3-1_SHP_WGS84G_FRA/EPCI.shp'
      using driver `ESRI Shapefile' successful.
1: EPCI (Polygon)

real	0m0,504s
user	0m0,327s
sys	0m0,017s

Bilan:

  • Taille de fichiers plus grosse avec le zip que le 7zip en stockage
  • Vitesse en consommation distante nettement plus perfomante avec le ZIP

Bonus: expérimentation GéoParquet

for i in $(ls ADMIN-EXPRESS-COG_*/ADMIN-EXPRESS-COG/1_*/ADECOG_*/*.shp);
  do filename=$(basename $i);
     filenamenoext=${filename%.*};
     ogr2ogr -f Parquet demo_$(echo $filenamenoext | tr '[:upper:]' '[:lower:]').parquet $i
done;
# scp *.parquet alias_ssh:/mypath/partage/

time ogrinfo /vsizip/vsicurl/https://labs.webgeodatavore.com/partage/epci.parquet

INFO: Open of `/vsicurl/https://labs.webgeodatavore.com/partage/epci.parquet'
      using driver `Parquet' successful.
1: epci (Multi Polygon)

real	0m0,338s
user	0m0,176s
sys	0m0,030s

Similar vizzes

Loading thumbnail…

Gist 0bdc45cce93d15fce0d1991d2b4b9e9d

This example demonstrates how to convert a compressed LiDAR point-cloud dataset from IGN’s French national archive into a ZIP archive for easier distribution. The shell script downloads a 7z file via `wget`, decompresses it with `unp`, recompresses the contents into a ZIP using `zip`, and then reports the final disk usage with `du -cksh`. It relies on standard command-line utilities rather than JavaScript libraries, showing a practical data-preparation workflow for large geospatial files.

TThomasG77
68% match
Loading thumbnail…

GDAL VSICURL tests to remote BDTOPO Geopackage France

This example tests GDAL’s VSICURL virtual file system by streaming a remote GeoPackage of French administrative data, specifically focusing on mairie (town hall) geometries. It demonstrates two ogr2ogr commands that query the BDTOPO dataset using SQL joins to extract mairie footprints as multipolygons and their centroids as points, converting them to GeoJSON with EPSG:4326. The visualization shows how both polygon boundaries and point locations are derived from the same remote source, highlighting the ability to perform spatial SQL operations directly on cloud-hosted vector data without local downloads.

TThomasG77
63% match
Loading thumbnail…

Generation GeoJSON des mairies avec code INSEE

This example generates a GeoJSON file of French town halls (mairies) with their INSEE commune codes, distinguishing principal towns from associated communes. It uses `ogr2ogr` with an SQLite dialect to join commune boundaries with their chief-town points from the IGN ADMIN-EXPRESS dataset, applying `PointOnSurface` as a fallback when no official point exists. The resulting output is written with RFC 7946 compliance for modern web mapping. The script downloads a 7z archive and extracts the necessary shapefiles before performing the spatial join.

TThomasG77
57% match
Loading thumbnail…

Export DVF geolocalisées en Geoparquet

This example converts French land-registry transaction data (DVF) from compressed CSV files into a single spatially-indexed GeoParquet file. It downloads yearly datasets from data.gouv.fr, loads them into a DuckDB database, and creates a point geometry for each transaction before exporting with a Hilbert-curve spatial ordering. The script leverages DuckDB’s spatial extension, the `ST_Point` and `ST_Hilbert` functions, and parquet compression via zstd.

TThomasG77
57% match
Loading thumbnail…

Gist 4ab7840c00aa352f278636ea3f3f0d3e

This example demonstrates streaming merge of two French cadastral parcel GeoJSON files into a single unified feature collection, output as a continuous JSON stream to standard output. It shows how to combine multiple geospatial datasets without loading them entirely into memory. The code uses the `geojson-stream` library for parsing and stringifying GeoJSON features, Node.js `fs.createReadStream` for file input, and the `zlib` module to decompress gzipped source files. Data originates from the Etalab cadastre dataset, with separate scripts handling zipped and unzipped input files.

TThomasG77
55% match
Loading thumbnail…

Lien Adresse et parcelles

This example shows the relationship between French addresses and land parcels, producing flat CSV files per department and for the whole country from the IGN BAN PLUS dataset. The workflow uses a bash pipeline combining Python with `xml.etree.ElementTree` to fix VRT files, GDAL’s `ogr2ogr` and `ogrinfo` for data conversion and indexing, `jq` to parse layer names, and `xsv` to concatenate departmental rows into a nationwide file. Results are gzip-compressed and organized into `france` and `departements` directories.

TThomasG77
55% match
Loading thumbnail…

Exemples de consommation de DVF via parquet et Duckdb

This example demonstrates how to extract and filter French property transaction data (DVF) from a cloud-hosted Parquet file using DuckDB's SQL engine, then export the filtered results as GeoJSON for specific geographic areas. It shows three distinct query scopes: a single commune (Nantes), a multi-commune agglomeration, and a department-level export. The workflow relies on DuckDB's `INSTALL spatial` and `LOAD spatial` extensions, which enable reading the Parquet file directly from the `data.gouv.fr` URL. The `COPY` command with `FORMAT gdal` and `DRIVER 'GeoJSON'` handles the conversion, while explicit type casting (e.g., `::DOUBLE`) ensures correct numeric fields like `valeur_fonciere` and coordinates (longitude/latitude), and preserves the geometry column for spatial output.

TThomasG77
54% match
Loading thumbnail…

Gist 349fc7fda8932252cbcb4ee8a0b605c8

This example tracks the availability and licensing of the French IGN’s BD TOPO® departmental datasets by organizing URLs for both December 2021 and March 2021 releases. It shows a structured index of geographic data files, with each entry linking to a specific departmental Shapefile resource. The visualization uses a data-driven list format, likely rendered with JavaScript’s `fetch` to parse the provided CSV or JSON and display the sorted URLs. It highlights the variation in resource IDs across departments and time periods, emphasizing the dataset’s scope without mapping or spatial rendering.

TThomasG77
54% match