Vishful thinking…

Can Yahoo Pipes or Microsoft Popfly replace ArcGIS Model Builder

Posted in ArcGIS, ESRI, GIS by viswaug on September 15, 2008

Yahoo Pipes and Microsoft Popfly are powerful online tools that allow us to aggregate, mashup and process data from various sources on the Internet to produce the desired output. These tools source their data from various URLs(resources) that can be added into the pipe (or model) and they can be manipulated using “Operators” that Yahoo Pipes make available.

Capture2 Capture3

GIS data can be fetched from ArcGIS Server through its RESTful endpoints in the format desired like JSON, KML etc. The “Operators” really doesn’t do too much for us here but they can be used as desired for limited purposes. The “Sub-element” operator especially can come in quite handy to get just at the desired part of the source data.

The operators that will interest the GIS folks are the geo-processing tasks that can be again accessed at RESTful URL endpoints in ArcGIS Server. The “UrlBuilder” shown below will come in real handy to direct from the source into the geo-processing task.


The only thing that is missing right now is that I haven’t figured out how I can get a point geometry from a geo-coding task to be passed as the value for a query parameter for another geo-processing task.

But taking this line of thinking further, using Yahoo Pipes or Microsoft Popfly instead of ArcGIS model builder may not be that far fetched. I am not saying that it is going to happen, it does have a long way to go, but it could be a poor-man’s model builder.

Tagged with: , ,

KML support in ArcGIS Server 9.3

Posted in ArcGIS, ESRI, GIS by viswaug on September 14, 2008

I was looking into the KML support available in ArcGIS Sever 9.3 and realized that is only available for the results of queries on the MapServer, geoprocessing results and geocoding results. The “identify” results of the MapServer and the results of the geometry services like Project, Simplify, Buffer etc do not support output in KML format.

Tagged with: , ,

A better way to read ArcGIS Server logs…and some weird errors being logged

Posted in Uncategorized by viswaug on September 14, 2008

The logs generated by ArcGIS Server come in real handy sometimes when trying to troubleshoot some problems with the server. Also, if you are developer creating custom “Sever Object Extensions (SOE)“, the “ILogSupport” interface provides the facility to add your own custom messages to the ArcGIS Server logs. Since, there is no way to debug the Server Object Extensions (at least I don’t know of any), logging messages from inside the SOE is crucial to be able to debug and troubleshoot the Server Object Extensions both during its development and also after deployment. Viewing the ArcGIS Server log file always used to be a pain in the ‘you know where’ since it was just a huge and constantly growing text file. I used to be using my favorite text editor Notepad++ to read the log file which notified me when the log file changes and updates it automatically. But still that only went so far to take the pain out of viewing the logs. Here is some information that will enable you understand the ArcGIS Server logs better.

Log Codes Overview

Core Server Log Codes

Map Service Log Codes

The ArcGIS Server log file can be found under the “C:\program files\arcgis\server\user\log” directory. A new log file is created whenever the ArcSOM service is restarted.

Recently, I came across an utility called “BareTail” (yeah, I know what you are thinking…snap out of it) that made reading the ArcGIS Server log files a lot less painful. They do have a free version of the software that can be downloaded from here. The utility can open large text files (> 2GB) and tracks the end of the file constantly thus eliminating the need to constantly scroll all the time. The utility also has a really cool feature where it highlights lines containing specific words with a specified color. This is a great feature for the ArcGIS Server log files since every log file entry contains a string representing the TYPE of the log message. The different types of ArcGIS Server log messages are:


This TYPE string in the log message, allows us to highlight the lines in the log file with different colors are shown below. This lets the ArcGIS Server logs to be read so much easier.



I have saved the color scheme shown above into a configuration file. The configuration file can be downloaded from here. You can import the color scheme directly into your BareTail install and set it to be the default.

Being able to read the log files this clearly allowed me to notice errors being logged that I would have never noticed before. For instance, I noticed that every time I previewed a non-cached MapService in ArcCatalog, a “Method Failed.” ERROR (See Below) was being logged when the “MapServer.GetTileCacheInfo” was called. I don’t think calling the “MapServer.GetTileCacheInfo”method for a non-cached MapService should fail and be logged as an ERROR. Not quite sure why this happens on my server but it doesn’t make me feel good about it.

<Msg time=’2008-09-13T17:38:37′ type=’ERROR’ code=’100005′ target=’TalonMap.MapServer’ methodName=’MapServer.GetTileCacheInfo’ machine=’Talon’ process=’2832′ thread=’2908′ elapsed=’0.00019′>Method failed.HRESULT = 0x80004005 : Unspecified errorĀ  .</Msg>

BareTail is also a great tool for viewing other log files. If you are using Log4NET to create logs in your applications, BareTail will come in real handy for viewing those log files also. I will actually be using Log4NET to log messages from my Server Object Extensions instead of using the ArcGIS Server log files which will allow me to concentrate only on the messages logged by my component.

Tagged with: , ,

Why is it important to follow RESTful principles when developing a REST API

Posted in Uncategorized by viswaug on September 8, 2008

When developing a RESTful API, it is important to follow the REST principles as much as possible. Sometimes it may not be apparent as to why, hopefully the following will illustrate the importance of following the REST principles. The web by itself is RESTful, REST treats everything as a resource and it has set of verbs or HTTP methods mapped to the operations that can be performed on resources.

Fetch the resource – HTTP GET
Insert a new resource – HTTP POST
Update a resource – HTTP PUT
Delete a resource – HTTP DELETE

And when you are writing your own REST API to perform one of the above operations on a resource, it is good practice to always to use the corresponding HTTP method to do so. Why is this important and when will you run into problems following the above principle? Well, let’s say that you have a HTTP GET URL endpoint that takes an identifier for a resources and actually deletes the resource instead of just fetching it. I can’t remember which one but one of the wiki implementations out there had beed doing the same where they were deleting web pages in the wiki when a HTTP GET request was made to an URL. And one fine night, the “GoogleBot” (Google’s web spider or crawler) came along and started making HTTP GET requests to that URL hoping to index the resource at that URL. This resulted in most of the web pages in the wiki being deleted and in a huge loss of information. So, your REST API will also be susceptible to such disasters if the REST principles are not followed strictly.

Following the REST principle, might be a problem when you need to send in a lot of information in the query string of the HTTP GET URL. For example, if you want to be able to fetch all the parcels inside a polygon, a representation of the polygon needs to be passed into the HTTP URL GET endpoint to be able to perform the query. In these cases, the length of the URL could become a problem. Normally, it is not a good idea to let the length of your URL exceed 1024. A lot of organizations might restrict the length of the URLs by using tools like UrlScan (this feature is built right into IIS 7.0). In these cases, you might have to end up POSTing the information to the URL, or in other words using a HTTP POST instead of HTTP GET. Actually, this is exactly what the ESRI REST API does. The JS API actually examines the length of the URL and if the length is larger than a certain number (the default is 2000), they POST the information to REST server instead of using GET(via a proxy, which i will write about soon). The ArcGIS REST server actually accepts both GETs and POSTs at the same URL endpoint since they are implemented as HTTPHandlers. Currently, the REST support in WCF does not allow the developer to accept both GETs and POSTs at the same URL endpoint.

When a different HTTP method is being used instead of the correct method, it is a good practice to do so while also adding the “X-HTTP-Method-Override" header to the HTTP request. Here is how Google uses this technique to get past some restrictive firewalls.

Why is a HTTP GET better than a HTTP POST? Well, your browser only caches the outputs from HTTP GET requests and never from a HTTP POST request. And the developer can gain control over how long the browser cache will be valid by setting the “Cache-Control” and the “Expires” headers in the HTTP response.

Updated post with the correct HTTP method operation after Sean Gillies pointed it out.

Tagged with: ,