Exposing ArcGIS Server Object Extensions as a Web Service – Will save you some licenses

What is good about building your application’s features as a Server Object Extension? Here are some

1. Server Objects Extensions run inside the ArcGIS Server Context and lets you avoid those calls that go across processes and sometimes across machines. This can deliver a significant boost to your application’s performance and improve its reliability and scalability.
2. Provides a way to incur some expensive initialization logic just once when the Server Object spins up. Server Object Extensions live and die with the Server Objects.
3. Some caching logic can also be implemented on the Server Object Extensions to eliminate avoidable loads on the ArcGIS Server. Caching here can go a long way in improving the performance of the application.
4. Allows developers to expose coarse-grained functionality by writing straight ArcObjects code. Not all developers fully recognize when to use ‘new’ or ‘CreateObject’ to create ArcObjects classes.

After the Server Object Extension (SOE) has been written, you will want to expose the service to your web applications. If you are writing Web Services to expose the SOE functionalities to your clients, you will still need to use the ‘AGSServerConnection’ or the ‘GISServerConnection’ classes to access the SOE’s methods. This means that the web server machine where you web service is hosted will need an ESRI license. If you want to farm out your web server, you will need an ESRI license for every one of your web servers.

But there is a way around this, your SOEs can be exposed as SOAP web services directly. This means that your clients can access the SOE as web services on the ArcGIS Server machine directly. Or you can build proxy web services or REST services that in turn expose the service to the clients. In order to expose your SOE’s methods as web services, you will need to do two things

1. Create a new WSDL file with the definitions for the methods on the SOE. The name of the WSDL file should be the same as the name with which the SOE was registered with ArcGIS Server referred to as the Extension Type. The file should be saved in the ‘C:\Program Files\ArcGIS\XmlSchema’ folder.
2. Implement the ‘IRequestHandler2’ interface on the Server Object Extension.

Once the above two steps have been accomplished, ArcGIS Server will now publish your SOE’s method as a web service. When the clients request a WSDL from the MapServer, it will serve the updated WSDL with the SOE’s method definitions in it. And when the clients make a request to the SOE’s web service, ArcGIS Server will route the request to the ‘HandleStringRequest’ method on the ‘IRequestHandler2’ interface implemented by the SOE.

The implementation of the ‘HandleStringRequest’ method on the SOE should parse through the SOAP request sent into it and retrieve the input parameters. The input parameters can then be used to execute the request and return the SOAP formatted response. This could be a major pain and I always had a major hang-up with writing code to parse SOAP request messages and form SOAP responses. But I just recently came to know that ESRI has recently exposed some internal classes that did exactly that. This wasn’t available in the 9.2 version and I don’t believe in the 9.3 betas. The ESRI class that would help in doing this is the ‘MessageClass’ class.

Yes, it is a little more work. But once you get the plumbing right once, it is easily repeatable. I believe that the extra effort on the developers part is well worth it in terms of $$$ saved in licensing.

Advertisement

Inserting spatial data in SQL Server 2008

I have been working on inserting data from FeatureClasses into SQL Server 2008 tables. There are a lot of examples online showing how spatial data can be inserted into SQL Server 2008 tables. Here is one from Hannes.

I took from cue from there and proceeded to do what every good programmer should do…parameterize the INSERT query. Here is a simplified version of the scenario I was trying to tackle…

sqlCommand.CommandText = INSERT INTO foo (id, geom) VALUES(1,@g);

sqlCommand.InsertParameter(“@g”, ‘POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0))’);

The above INSERT command worked fine. But I also wanted to insert the geometry or geography with the WKID value from the FeatureClass. So I changed the above INSERT statement to use the geometry’s static method and provided the WKID as below

sqlCommand.InsertParameter(“@g”, ‘geometry::STGeomFromText(POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0)), 0)’);

But that didn’t go quite as expected. The INSERT statement was throwing an error saying that it did not recognize ‘geometry::STGeomFromText’ and was expecting ‘POINT, POLYGON, LINESTRING…’ etc.

System.FormatException: 24114: The label geometry::STGeomFrom in the input well-known text (WKT) is not valid. Valid labels are POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, MULTIPOLYGON, or GEOMETRYCOLLECTION.

This led me to use the Udt SqlParamter type to get around the problem like below. This way I could use the SqlGeometry type in the ‘Microsoft.SqlServer.Types’ namespace and set the SRID directly on its ‘STSRID’ property. The ‘SqlGeometry’ and ‘SqlGeography’ types can be found in the Feature Pack for SQL Server 2008 that can be found here. The SqlGeometry reference can then be used as the value for the SqlParameter.

SqlParameter parameter = new SqlParameter( parameterName, value );

parameter.Direction = ParameterDirection.Input;
parameter.ParameterName = parameterName;
parameter.SourceColumn = sourceColumn;
parameter.SqlDbType = SqlDbType.Udt;
parameter.UdtTypeName = “GEOMETRY”;
parameter.SourceVersion = DataRowVersion.Current;

command.Parameters.Add( parameter );

The ‘UdtTypeName’ should be set to ‘GEOGRAPHY’ for inserting geography fields.

And it worked like a charm.

Update: Heard from Morten saying that using the SqlGeography and SqlGeometry types are almost twice as fast as using WKB (Well Known Binary). Using WKB is by itself faster than using WKT (Well Known Text).

A handy tool for OpenLayers enthusiasts – OpenLayers Architect

I stumbled upon this cool tool called “OpenLayers Architect” that lets you build your map page by letting you add and manage the layers that will be displayed in the map. A real handy tool for ‘visual’ people like me. Lets you manage the layer settings through a dialog like below and lets you do a whole bunch other stuff. I don’t think it is robust yet but I think it is a cool concept. A step closer towards ArcMAP for the web…

Hopefully, ESRI can build such a tool for their JS APIs in the future…

Can Yahoo Pipes or Microsoft Popfly replace ArcGIS Model Builder

Yahoo Pipes and Microsoft Popfly are powerful online tools that allow us to aggregate, mashup and process data from various sources on the Internet to produce the desired output. These tools source their data from various URLs(resources) that can be added into the pipe (or model) and they can be manipulated using “Operators” that Yahoo Pipes make available.

GIS data can be fetched from ArcGIS Server through its RESTful endpoints in the format desired like JSON, KML etc. The “Operators” really doesn’t do too much for us here but they can be used as desired for limited purposes. The “Sub-element” operator especially can come in quite handy to get just at the desired part of the source data.

The operators that will interest the GIS folks are the geo-processing tasks that can be again accessed at RESTful URL endpoints in ArcGIS Server. The “UrlBuilder” shown below will come in real handy to direct from the source into the geo-processing task.

The only thing that is missing right now is that I haven’t figured out how I can get a point geometry from a geo-coding task to be passed as the value for a query parameter for another geo-processing task.

But taking this line of thinking further, using Yahoo Pipes or Microsoft Popfly instead of ArcGIS model builder may not be that far fetched. I am not saying that it is going to happen, it does have a long way to go, but it could be a poor-man’s model builder.

KML support in ArcGIS Server 9.3

I was looking into the KML support available in ArcGIS Sever 9.3 and realized that is only available for the results of queries on the MapServer, geoprocessing results and geocoding results. The “identify” results of the MapServer and the results of the geometry services like Project, Simplify, Buffer etc do not support output in KML format.

A better way to read ArcGIS Server logs…and some weird errors being logged

The logs generated by ArcGIS Server come in real handy sometimes when trying to troubleshoot some problems with the server. Also, if you are developer creating custom “Sever Object Extensions (SOE)“, the “ILogSupport” interface provides the facility to add your own custom messages to the ArcGIS Server logs. Since, there is no way to debug the Server Object Extensions (at least I don’t know of any), logging messages from inside the SOE is crucial to be able to debug and troubleshoot the Server Object Extensions both during its development and also after deployment. Viewing the ArcGIS Server log file always used to be a pain in the ‘you know where’ since it was just a huge and constantly growing text file. I used to be using my favorite text editor Notepad++ to read the log file which notified me when the log file changes and updates it automatically. But still that only went so far to take the pain out of viewing the logs. Here is some information that will enable you understand the ArcGIS Server logs better.

Log Codes Overview

Core Server Log Codes

Map Service Log Codes

The ArcGIS Server log file can be found under the “C:\program files\arcgis\server\user\log” directory. A new log file is created whenever the ArcSOM service is restarted.

Recently, I came across an utility called “BareTail” (yeah, I know what you are thinking…snap out of it) that made reading the ArcGIS Server log files a lot less painful. They do have a free version of the software that can be downloaded from here. The utility can open large text files (> 2GB) and tracks the end of the file constantly thus eliminating the need to constantly scroll all the time. The utility also has a really cool feature where it highlights lines containing specific words with a specified color. This is a great feature for the ArcGIS Server log files since every log file entry contains a string representing the TYPE of the log message. The different types of ArcGIS Server log messages are:

ERROR

WARNING

INFO

DETAILED

DEBUG

This TYPE string in the log message, allows us to highlight the lines in the log file with different colors are shown below. This lets the ArcGIS Server logs to be read so much easier.

I have saved the color scheme shown above into a configuration file. The configuration file can be downloaded from here. You can import the color scheme directly into your BareTail install and set it to be the default.

Being able to read the log files this clearly allowed me to notice errors being logged that I would have never noticed before. For instance, I noticed that every time I previewed a non-cached MapService in ArcCatalog, a “Method Failed.” ERROR (See Below) was being logged when the “MapServer.GetTileCacheInfo” was called. I don’t think calling the “MapServer.GetTileCacheInfo”method for a non-cached MapService should fail and be logged as an ERROR. Not quite sure why this happens on my server but it doesn’t make me feel good about it.

<Msg time=’2008-09-13T17:38:37′ type=’ERROR’ code=’100005′ target=’TalonMap.MapServer’ methodName=’MapServer.GetTileCacheInfo’ machine=’Talon’ process=’2832′ thread=’2908′ elapsed=’0.00019′>Method failed.HRESULT = 0x80004005 : Unspecified error  .</Msg>

BareTail is also a great tool for viewing other log files. If you are using Log4NET to create logs in your applications, BareTail will come in real handy for viewing those log files also. I will actually be using Log4NET to log messages from my Server Object Extensions instead of using the ArcGIS Server log files which will allow me to concentrate only on the messages logged by my component.

System.Drawing.Imaging Namespace

I was really taken aback today when I read that Microsoft does not support the use of the “System.Drawing.Imaging” Namespace within a windows or ASP.NET service. I have never had a problem with it when I have used it before and don’t expect to have a problem with it. But it is surprising that Microsoft doesn’t support it. Just thought some others might have the same reaction to it as I did…

Why is it important to follow RESTful principles when developing a REST API

When developing a RESTful API, it is important to follow the REST principles as much as possible. Sometimes it may not be apparent as to why, hopefully the following will illustrate the importance of following the REST principles. The web by itself is RESTful, REST treats everything as a resource and it has set of verbs or HTTP methods mapped to the operations that can be performed on resources.

Fetch the resource – HTTP GET
Insert a new resource – HTTP POST
Update a resource – HTTP PUT
Delete a resource – HTTP DELETE

And when you are writing your own REST API to perform one of the above operations on a resource, it is good practice to always to use the corresponding HTTP method to do so. Why is this important and when will you run into problems following the above principle? Well, let’s say that you have a HTTP GET URL endpoint that takes an identifier for a resources and actually deletes the resource instead of just fetching it. I can’t remember which one but one of the wiki implementations out there had beed doing the same where they were deleting web pages in the wiki when a HTTP GET request was made to an URL. And one fine night, the “GoogleBot” (Google’s web spider or crawler) came along and started making HTTP GET requests to that URL hoping to index the resource at that URL. This resulted in most of the web pages in the wiki being deleted and in a huge loss of information. So, your REST API will also be susceptible to such disasters if the REST principles are not followed strictly.

Following the REST principle, might be a problem when you need to send in a lot of information in the query string of the HTTP GET URL. For example, if you want to be able to fetch all the parcels inside a polygon, a representation of the polygon needs to be passed into the HTTP URL GET endpoint to be able to perform the query. In these cases, the length of the URL could become a problem. Normally, it is not a good idea to let the length of your URL exceed 1024. A lot of organizations might restrict the length of the URLs by using tools like UrlScan (this feature is built right into IIS 7.0). In these cases, you might have to end up POSTing the information to the URL, or in other words using a HTTP POST instead of HTTP GET. Actually, this is exactly what the ESRI REST API does. The JS API actually examines the length of the URL and if the length is larger than a certain number (the default is 2000), they POST the information to REST server instead of using GET(via a proxy, which i will write about soon). The ArcGIS REST server actually accepts both GETs and POSTs at the same URL endpoint since they are implemented as HTTPHandlers. Currently, the REST support in WCF does not allow the developer to accept both GETs and POSTs at the same URL endpoint.

When a different HTTP method is being used instead of the correct method, it is a good practice to do so while also adding the “X-HTTP-Method-Override" header to the HTTP request. Here is how Google uses this technique to get past some restrictive firewalls.

Why is a HTTP GET better than a HTTP POST? Well, your browser only caches the outputs from HTTP GET requests and never from a HTTP POST request. And the developer can gain control over how long the browser cache will be valid by setting the “Cache-Control” and the “Expires” headers in the HTTP response.

Updated post with the correct HTTP method operation after Sean Gillies pointed it out.

Missing features in the ESRI JS API v1.1

As I am in the process of building my current project using the ESRI JS API v1.1, I am beginning to come across some of the currently missing features in JS API and the REST API that in my opinion are required to build any enterprise GIS web mapping application. We are currently in the process of filling in those holes by building the missing functionality ourselves. So, if you are just considering the JS API for your projects, take the following into consideration.

1) The REST API currently does not provide the MIN and MAX extent information for scale dependent layer visibility. This will be needed to build a Table of Contents for the map where the layers that are not visible at the current scale can be greyed out.

2) The REST API currently also does not provide the swatch image information that will be required to build the Table of Contents. Both the MIN & MAX extent information and the swatch images are available in the SOAP API, so building a REST service to provide these missing information shouldn’t be difficult. The fact that the missing information in available in the SOAP API is important because it will allow the Table of Contents to be populated even when the user adds a map service from an external URL dynamically.

3) The JS API currently does not allow the user to zoom or pan when the user is in the process of drawing a point, polyline or polygon feature.

4) The JS API currently does not allow the user to add or edit vertices in an existing feature. This feature might be available in the future versions of the JS API.

5) The JS API currently does not document how to build custom layer types to add to the map like ArcIMS or WMS layers etc. But I think the documentation and samples to do this is coming in the future versions. Hopefully v1.2…

6) I am not quite sure about the client-side graphics layers performance yet. If you are using the client-side graphics layer for displaying the selection set, then having 10 polygons each with a 1000 vertices selected could significantly affect the drawing performance of the graphics layer. Leveraging the server-side graphics layer for this circumstance would be ideal, but this is currently not possible with the JS API. But with some work, leveraging the server-side graphics layer in the JS API is possible.