Web Feature Service (WFS)

Get maximum record count per request from GetCapabilities document:

library(httr)
library(xml2)
library(magrittr)

request <- "https://geodata.nationaalgeoregister.nl/bag/wfs?service=WFS&version=2.0.0&request=GetCapabilities"
doc <- GET(request) %>% content(as = "text", encoding="UTF-8") %>% read_xml() 
xpath <- "//ows:Constraint[@name='CountDefault']/ows:DefaultValue"
max_record_count <- doc %>% xml_find_first(xpath) %>% xml_integer()
max_record_count
## [1] 1000

Get number of hits for query:

request <- paste0("https://geodata.nationaalgeoregister.nl/bag/wfs?service=WFS&version=2.0.0",
                  "&request=GetFeature&typename=bag:verblijfsobject&cql_filter=bag:woonplaats='Zuidhorn'&resultType=hits")
doc <- GET(request) %>% content(as = "text", encoding="UTF-8") %>% read_xml()
xpath <- "//wfs:FeatureCollection/@numberMatched"
hits <- doc %>% xml_find_first(xpath) %>% xml_integer()
hits
## [1] 3560

Number of records in request response:

library(sf)
## Linking to GEOS 3.6.1, GDAL 2.2.3, proj.4 4.9.3
request <- paste0("https://geodata.nationaalgeoregister.nl/bag/wfs?service=WFS&version=2.0.0&request=GetFeature",
                  "&typename=bag:verblijfsobject&cql_filter=bag:woonplaats='Zuidhorn'&startIndex=0&resultType=results")
number_of_records <-  nrow(st_read(request))
## Reading layer `verblijfsobject' from data source `https://geodata.nationaalgeoregister.nl/bag/wfs?service=WFS&version=2.0.0&request=GetFeature&typename=bag:verblijfsobject&cql_filter=bag:woonplaats='Zuidhorn'&startIndex=0&resultType=results' using driver `GML'
## Simple feature collection with 1000 features and 15 fields
## Active geometry column: geometrie
## geometry type:  POINT
## dimension:      XYZ
## bbox:           xmin: 219973.5 ymin: 582728 xmax: 225038 ymax: 585853
## epsg (SRID):    NA
## proj4string:    NA
number_of_records
## [1] 1000

The response contains an URL to retrieve the next 1000 records:

request <- paste0("https://geodata.nationaalgeoregister.nl/bag/wfs?service=WFS&version=2.0.0&request=GetFeature",
                  "&typename=bag:verblijfsobject&cql_filter=bag:woonplaats='Zuidhorn'&startIndex=0&resultType=results")
doc <- GET(request) %>% content(as = "text", encoding="UTF-8") %>% read_xml()
url <-  xml_attr(doc, "next")
url
## [1] "https://geodata.nationaalgeoregister.nl/bag/wfs?TYPENAME=bag%3Averblijfsobject&REQUEST=GetFeature&RESULTTYPE=results&VERSION=2.0.0&SERVICE=WFS&COUNT=1000&CQL_FILTER=bag%3Awoonplaats%3D%27Zuidhorn%27&STARTINDEX=1000"

However the URL in the next attribute is not used for request paging.

ESRI hosted feature service

Get maximum number of records per request:

library(jsonlite)

request <- paste0("https://services.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services",
                  "/USA_Railroads_1/FeatureServer?f=json")
response <- fromJSON(request) 
max_record_count <- response$maxRecordCount
max_record_count
## [1] 1000

Get number of hits for query:

request <- paste0("https://services.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services",
                  "/USA_Railroads_1/FeatureServer/0/query?where=1=1&returnCountOnly=true&f=geojson")
response <- fromJSON(request) 
hits <- response$properties$count
hits
## [1] 173114

Number of records in request response:

request <- paste0("https://services.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services",
                  "/USA_Railroads_1/FeatureServer/0/query?where=1=1&outFIelds=OBJECTID&returnGeometry=true&f=geojson")
number_of_records <-  nrow(st_read(request))
## Reading layer `OGRGeoJSON' from data source `https://services.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services/USA_Railroads_1/FeatureServer/0/query?where=1=1&outFIelds=OBJECTID&returnGeometry=true&f=geojson' using driver `GeoJSON'
## Simple feature collection with 173114 features and 1 field
## geometry type:  LINESTRING
## dimension:      XY
## bbox:           xmin: -158.1522 ymin: 20.88018 xmax: -66.98391 ymax: 64.92615
## epsg (SRID):    4326
## proj4string:    +proj=longlat +datum=WGS84 +no_defs
number_of_records
## [1] 173114

All 173114 are fetched! So sf apparently supports request paging for ESRI hosted feature services.