Extracting Original Data from ArcGIS Server REST Dynamic Map Services Using Quantum GIS

Para pengguna ArcGIS Server tentu sudah sangat paham dengan yang namanya REST Map Services. REST (representation state transfer) Map services merupakan format distribusi data spasial via Web yang digunakan ESRI untuk mendukung diseminasi data spasial via web. Map services menggunakan REST protocol dapat digenerate dan dihosting menggunakan perangkat lunak ArcGIS server. Bentuk umum REST mirip dengan OGC WMS, dimana client akan me-request data spasial kepada server yang akan di-return oleh server dalam bentuk gambar peta dan beberapa atribut yang diperlukan. Layanan REST dapat diakses dengan memasukkan URL tertentu disertai serangkaian tag query untuk memanggil data. Katalog REST URL dapat diakses dari REST Service directory. Contoh REST Service directory yang dapat diakses publik milik Kementerian/Lembaga/Instansi antara lain :

1. Kementerian PU
2. BIG
3. INA-SDI
4. BNPB
5. Kementerian Pertanian
6. Kementerian Kehutanan
7. Kementerian Perhubungan,,
8. dan masih banyak lagi instansi pemerintah yang sudah membangun layanan data spasialnya (link cari sendiri yah).

Satu hal yang membedakan REST dan WMS adalah, REST mendukung beberapa jenis output, selain output image, REST juga mendukung output KMZ, JSON dan AMF. Sebelum postingan ini dibuat, saya termasuk orang yang meyakini bahwa output REST secara umum hanya images, sehingga kita tidak dapat mengakses data asli dari peta tersebut (kecuali ada fasilitas feature access dan geodata services dari si penyedia layanan). Tapi begitu saya menyadari ada format KMZ dan JSON sebagai salah satu output format default, suddenly the world turned upside down. Yeah in those two formats, we can get the original hosted geospatial data. Format JSON dalam hal ini lebih baik dari KMZ karena JSON mampu menyimpan tabel atribut sesuai data asli (tidak seperti KML/KMZ yang data atributnya dikonversi menjadi HTML page).

Untuk contoh saya menggunakan Quantum GIS 2.6 Brighton yang sudah mendukung pemanggilan protokol Javascript Object Notation (JSON) agar hasil request bisa diparsing langsung sebagai data spasial. Selain itu, query dari REST directory langsung juga bisa. Here are the steps :

1. Get REST data URL from REST Services Directory, misal saya ambil dari layanan berikut : http://geodata.epa.gov/arcgis/rest/services/OAR/USEPA_NEI_2005/MapServer/

2. Open QGIS 2.6

3. Add Vector Layer, pilih Protocol Mode, kemudian masukkan URL di step 1 dan ditambah baris kode berikut :

http://geodata.epa.gov/arcgis/rest/services/OAR/USEPA_NEI_2005/MapServer/1/query?where=objectid+%3D+objectid&outfields=*&f=json 

4. QGIS will load and extract the desired data (with attributes too)

5. Save as Shapefile or any format you desire

KNOWN LIMITATION

Query dari REST services umumnya dibatasi sampai 999 records per query Sehingga kalau data yang diekstrak recordsnya lebih dari 1000 maka anda harus melakukan iterasi atau query berulang menggunakan OBJECTID sebagai filter, sampai seluruh data selesai di-download. agak rempong sih memang, tapi bersyukur sajalah wahai fakir data. 

Semoga bermanfaat :D

UPDATE!!!!!

untuk kelanjutan diskusi tentang known limitation diatas, silahkan lihat link INI. 

UPDATE 6 NOVEMBER 2014

1. Untuk cara mudah mengekstrak dan mengeksport data dari REST Map Services, A good guy has created a set of phyton code to do the task, grab it HERE .
Cara penggunaannya cukup mudah, buka file exportshp.py menggunakan text editor, kemudian ganti parameter URL ke parameter Map Services yang anda inginkan, kemudian RUN script di Phyton IDLE, PhytonWin atau phyton console yang lain. Unfortunately this method only working for ArcGIS 10.1 and later


2. Collaborating with my friend Marhensa Aditya Hadi, we have created two script to do the downloading and conversion semi automatically (working for ArcGIS server 9 sampai 10.2).

a. The first file is a microsoft excel file (written by Marhensa) that generate download links from map services by 1000 records. Yang harus anda lakukan cukup mengisikan parameter URL dari Map Services di kolom Map Services URL, Layer ID dari layer data yang akan diambil, dan nama Layer output. Links will be generated automatically once you drag down the cell. The File Is HERE. Save the Generated Links into TXT file, and import it into Download Manager Software, let the download manager handle the downloading job.

b. The second file is a phyton script that I write to do the batch conversion from downloaded JSON's to Shapefile. Yang harus anda seting cukup lokasi folder dari file JSON hasil download. Run the script from ArcGIS Phyton Window, IDLE or other Phyton console that support import ArcPy. Sayang konversi JSON ke shapefile hanya bisa dilakukan di ArcGIS versi 10.1 ke atas. The file is HERE. (Script ini akan menampilkan peta hasil konversi secara otomatis di ArcMap jika kode dijalankan di ArcGIS Phyton Window, terkadang proses ini akan memakan memory, untuk mencegahnya, tambahkan baris kode ini "arcpy.env.addOutputsToMap = False" di kode phyton). 

UPDATE 4 JULI 2015

In case JSON Format Messed up or didnt converted correctly, you could use KML/KMZ format instead. But as we all know, KMZ format using html/xml to store data attributes so it could displayed correctly in Web. This is something that we (GIS Users) sometimes doesnt want because we needs it kept stored as attribute table. Conversion between KML to other GIS format is straighforwardly easy. there are numerous softwares and tools that could do that, but most of them couldnt parse the html popup data to attribute table in correct manner. As far I know, only Global Mapper and Safe FME that could do it almost precisely, so just use them and KML if JSON format didnt works

UPDATE 27 Oktober 2015

Just found another similar tool in GITHUB. Just track this link https://github.com/tannerjt/AGStoShapefile and follow the instructions there.

 UPDATE 27 Desember 2015

Found two more tools


1. https://github.com/Schwanksta/python-arcgis-rest-query
2. http://neagis.com/2013/08/19/using-arcgis-com-to-extract-data-from-map-services/

 UPDATE 25 Februari 2016

Found another tool, this tim in a form of geoprocessing toolbox which can you run from ArcGIS Desktop

http://epro.maps.arcgis.com/home/item.html?id=16e5cc64178941839eca62837f168ec9

UPDATE 5 Agustus 2017

Additional Query Syntax for ArcGIS Server 10.4 or later

geometryType=esriGeometryEnvelope&spatialRel=esriSpatialRelIntersects&relationParam=&outFields=*&returnGeometry=true&geometryPrecision=&outSR=&returnIdsOnly=false&returnCountOnly=false&orderByFields=&groupByFieldsForStatistics=&returnZ=false&returnM=false&returnDistinctValues=false&returnTrueCurves=false&f=pjson

UPDATE 5 Februari 2018

Found another tool, check this GITHUB Repo :
https://gist.github.com/TheBryanMac/da76aea0b736aa3da99a80408f6db42f#file-mapservice2shapefileiterate-py (save it to py file, and run from arcmap phyton windows or your preferred phyton shell/CLI)


another robust tool

https://github.com/openaddresses/pyesridump

install using python 3, pip, and just run the esri2gejson command from CLI

UPDATE SEPTEMBER 2018 
New python code to works for more than 1000 records (add "&token=token" parameters for every baseURL and f=pjson syntax for token based Map Services)

import arcpy
import urllib2
import json

# Setup
arcpy.env.overwriteOutput = True
baseURL = "http://services.gis.ca.gov/arcgis/rest/services/Environment/Wildfires/MapServer/0"
fields = "*"
outdata = "H:/cal_data/data.gdb/testdata"

# Get record extract limit
urlstring = baseURL + "?f=json"
j = urllib2.urlopen(urlstring)
js = json.load(j)
maxrc = int(js["maxRecordCount"])
print "Record extract limit: %s" % maxrc

# Get object ids of features
where = "1=1"
urlstring = baseURL + "/query?where={}&returnIdsOnly=true&f=json".format(where)
j = urllib2.urlopen(urlstring)
js = json.load(j)
idfield = js["objectIdFieldName"]
idlist = js["objectIds"]
idlist.sort()
numrec = len(idlist)
print "Number of target records: %s" % numrec

# Gather features
print "Gathering records..."
fs = dict()
for i in range(0, numrec, maxrc):
  torec = i + (maxrc - 1)
  if torec > numrec:
    torec = numrec - 1
  fromid = idlist[i]
  toid = idlist[torec]
  where = "{} >= {} and {} <= {}".format(idfield, fromid, idfield, toid)
  print "  {}".format(where)
  urlstring = baseURL + "/query?where={}&returnGeometry=true&outFields={}&f=json".format(where,fields)
  fs[i] = arcpy.FeatureSet()
  fs[i].load(urlstring)

# Save features
print "Saving features..."
fslist = []
for key,value in fs.items():
  fslist.append(value)
arcpy.Merge_management(fslist, outdata)
print "Done!"

 Latest toolbox for ArcGIS Pro : https://github.com/nicogis/DownloadService , or here is the CODE.

Another tool from geonet : https://community.esri.com/docs/DOC-6496-download-arcgis-online-feature-service-or-arcgis-server-featuremap-service?commentID=60020 , and here is the CODE.


UPDATE 1 May 2019

Good Python Tool Which Support Features iteration.

https://github.com/geolibrerian/mapservicereader

Run run.py after you download the packages , and perform pip install pyshp before using it. Here is the mirror CODE