Vishful thinking…

Bing map issues with IE6 & Silverlight 2 / ESRI Silverlight API 1.0

Posted in ArcGIS, ESRI, Silverlight by viswaug on August 17, 2009

I have been having troubles for a while now getting the Bing map layers to show in IE6 with the Silverlight 2 installed. Unfortunately, when I got down to the bottom of it, it turned out to be a bug in Silverlight 2 that actually prevents the Bing map layers to show in the ESRI Silverlight API 1.0 map control. Initially, I wasn’t sure why the bing map layers where not showing up on just the IE6 browsers but displayed and worked just fine on IE7+ and Firefox. To check the issue for yourself, if you are using an IE6 browser with the Silverlight 2 plugin, just browse over the Bing maps demo site setup by ESRI to check it out for yourself. you will see the error message shown below.

Part of the error message displayed above in the alert window (“Make sure you are generating a VE token from the same token service as the VE Layer ServerType: Staging or Production”) is actually not accurate. The demo site simply handles the ‘InitializationFailed‘ event on the Bing TileLayer and displays the message above not matter what the error actually was. This had mislead me initially and had sent me hunting for clues in the wrong direction. One other weird I noted here in the ESRI Silverlight API 1.0 is that the event arguments for the ‘InitializationFailed‘ actually doesn’t provide you with the error information. But the error information is actually available in the ‘InitializationFailure‘ property of the layer which I thought was a little confusing. Hoping this will change in the next versions of the API.

After spending a lot of time on fiddler, snooping around the requests made to the Bing servers & searching the web, I finally figured out the reason why the Bing layers do not show up on the map. The reason turned out to be a bug in the Silverlight 2 plug-in in IE6. The error had to do with Silverlight 2 plug-in in IE6 having troubles handling compressed content returned from the server. More details about the bug can be found here. Since the compressed content in this case comes from the Bing servers, I had not control over it at all and thus it eventually turned out that the problem couldn’t be solved in a good way. But there was an advanced setting in IE6 that I could turn off to get the bing layers to show up in the map. Under the ‘Tools’ menu in IE6, select the ‘Internet Options’ item. In the ‘Internet Options’ dialog, click on the ‘Advanced’ tab and in the settings disable the ‘Use HTTP 1.1’ option.

But this workaround does have its own drawbacks of not being able to use some of the advanced HTTP 1.1 capabilities like compression etc. So, please be mindful of it before adopting the solution.

Some salvaging facts,

  • This bug has been fixed in Silverlight 3. So, users who are still using IE6 but have upgraded to the Silverlight 3 plug-in will not have this problem.
  • IE6 is losing market share – (though a good chunk of the users on the web still use IE6)
  • The Silverlight 2 plug-in is no longer available for download from Microsoft.
  • Windows update upgrades user with Silverlight 2 to Silverlight 3.

But, if you are writing a public application using the ESRI Silverlight API 1.0 that needs to support Silverlight 2 plug-in on IE6, you definitely might want to consider the above issue and display a message to you users asking them to upgrade or turn off the ‘Use HTTP 1.1’ option in IE6.

Advertisements

Deployment hassles with the ‘black box’ MXD file format

Posted in ArcGIS, ESRI by viswaug on July 18, 2009

The MXD file format has always remained a ‘black box’ with its proprietary binary file format which cannot be opened or modified without ArcMap / ArcCatalog / ArcObjects. It’s even got a voodoo doctor that magically treats all its ailments, the ‘MXD Doctor’ software utility. This is fine in the most of the desktop world where the user most probably has access to ArcMap / ArcCatalog. But in the world of server applications that use ArcGIS Server, it does become quite a hassle. The problem occurs during the deployment phase. During the development and the demo phase, we might have prepared the MXD file with all the layers referencing FeatureClasses in the development / demo  geodatabase. But when you move the MXD to the production environment, the layers should now be modified to reference the FeatureClasses from the production geodatabase. Well, to switch the FeatureClass data source reference for the layers in the MXD, you will need access to ArcMap / ArcCatalog. IMHO, I don’t believe that we should be installing ArcMap / ArcCatalog in the production server environment, especially for a reason like the one mentioned above. Installing a full-blown desktop GIS software on a server environment might not make the IT folks very happy and might open up / expose security vulnerabilities. Given the binary nature of the MXD file, I can’t open up the file in a text editor and switch the data sources. And AFAIK, I don’t believe that there is a tool / script that will help make the switch of the data sources without the need for ArcMAP / ArcCatalog. Also, it is not just the MXD file format, the other ESRI file format like ‘.sde’ and ‘.gds’ etc all are in a binary format which makes deployment more painful that it should really be.

Exploring/Exploiting the query operation in the ArcGIS REST API

Posted in ArcGIS, ESRI, GIS by viswaug on July 9, 2009

I was starting to look harder at the query operation in the ArcGIS Server REST API today to figure if I can leverage it to get specific results I needed instead of building my own REST services. The scenario I was looking at was to be able to get the records corresponding to the minimum and the maximum values in a given field. To provide an example, let’s say you have a ‘States’ layer with a field called ‘POP2008’ that contains population numbers for the year 2008. In this case, I want to obtain the state records with the minimum and the maximum population for the year 2008. At the outset, there didn’t seem to be a way to do it. After spending some time investigating the possibilities with the REST API query operation, I found that the above was in fact possible. Assuming that the FeatureClass name of the ‘States’ layer is ‘States_DTL’, the query below will produce the desired results.

POP2008 = (SELECT MIN(POP2008) from States_DTL) OR POP2008 = (SELECT MAX(POP2008) from States_DTL)

I was surprised to find out that I was able to use the name of a FeatureClass in the query. Agreed, the user of the REST API will/might not know the name of the FeatureClass. But the user could get lucky and be able to guess the name of the FetaureClass after ‘n’ tries. The FeatureClass / Table being used doesn’t even need to be a part of the MapService. I am not yet sure about how far this behavior can be exploited. Nevertheless, I found that this behavior was a little more than interesting.

Reversed co-ordinate axis order for EPSG:4326 vs CRS:84 when requesting WMS 1.3.0 images

Posted in ArcGIS, ESRI, GIS by viswaug on March 15, 2009

I spent quite a bit of time last week over this and thought it might help others who might encounter the same thing as I did. I was requesting images from the WMS service version 1.3.0 in ArcGIS Server and everything was fine as long I was requesting images with the “CRS:84” value for the co-ordinate reference system. According to the GetCapabilities document returned from the WMS Server, both the “CRS:84” and the “EPSG:4326” values are suported by the WMS server.

http://sampleserver1.arcgisonline.com/ArcGIS/services/Specialty/ESRI_StatesCitiesRivers_USA/MapServer/WMSServer?VERSION=1.3.0&REQUEST=GetCapabilities

But when I started requesting images with “EPSG:4326”, I was seeing some really weird behaviour. I spent quite a bit of time trying to figure out what was going on without luck. The OGC team at ESRI helped me out and pointed out that when using WMS 1.3.0, the co-ordinate axis order in reversed for CRS:84 and EPSG:4326. So, when requesting images in CRS:84 use the coordiate axis order of (long, lat) and when using EPSG:4326 use the (lat, long) order.

http://sampleserver1.arcgisonline.com/ArcGIS/services/Specialty/ESRI_StatesCitiesRivers_USA/MapServer/WMSServer?VERSION=1.3.0&REQUEST=GetMap&CRS=CRS:84&BBOX=-178.217598,18.924782,-66.969271,71.406235&WIDTH=765&HEIGHT=360&LAYERS=0,1,2&STYLES=,,pointSymbolizer&EXCEPTIONS=application/vnd.ogc.se_xml&FORMAT=image/png&BGCOLOR=0xFFFFFF&TRANSPARENT=TRUE&SLD=http://sampleserver1.arcgisonline.com/arcgis/wms/slds/point_pointSymbolizer.xml

http://sampleserver1.arcgisonline.com/ArcGIS/services/Specialty/ESRI_StatesCitiesRivers_USA/MapServer/WMSServer?VERSION=1.3.0&REQUEST=GetMap&CRS=EPSG:4326&BBOX=18.924782,-178.217598,71.406235,-66.969271&WIDTH=765&HEIGHT=360&LAYERS=0,1,2&STYLES=,,pointSymbolizer&EXCEPTIONS=application/vnd.ogc.se_xml&FORMAT=image/png&BGCOLOR=0xFFFFFF&TRANSPARENT=TRUE&SLD=http://sampleserver1.arcgisonline.com/arcgis/wms/slds/point_pointSymbolizer.xml

But when requesting images from WMS Server version 1.1.1 with EPSG:4326, we should still use the coordinate axis order of (long, lat)

http://sampleserver1.arcgisonline.com/ArcGIS/services/Specialty/ESRI_StatesCitiesRivers_USA/MapServer/WMSServer?VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:4326&BBOX=-178.217598,18.924782,-66.969271,71.406235&WIDTH=765&HEIGHT=360&LAYERS=0,1,2&STYLES=,,pointSymbolizer&EXCEPTIONS=application/vnd.ogc.se_xml&FORMAT=image/png&BGCOLOR=0xFFFFFF&TRANSPARENT=TRUE&SLD=http://sampleserver1.arcgisonline.com/arcgis/wms/slds/point_pointSymbolizer.xml

Do these standards really help when the standards are not really STANDARD?

Then comes the issue of versioning of these standards, especially the OGC standards. The OGC standards are so reliant on one another that the different versions that exist in each standard actually adds a lot more complexity to systems that actually use them. For example, the SLD 1.1.0 standard (Styled Layer Descriptor) depends on the FE 1.1.0 standard (Filter Encoding), the SE 1.1.0 standard (Symbology Encoding) and the GML 3.1.1 standard (Geography Markup Language). As and when the standards evolve and higher versions are created, the versions of the standards they in-turn depend on change making a big mess for the systems that actually have to implement these standards.

Options for solving the problem of a graphics layer for the web

Posted in ArcGIS, ESRI, GIS by viswaug on March 5, 2009

I realize that most people developing web mapping applications have already solved this problem one way or the other. But some solutions are better than others for certain scenarios and needs to be paid attention to. I also want to mention that there are numerous other ways that this can be accomplished, but I am going to limit this post to the ones that I think are the good ones. The graphics layer can really be maintained at three different tiers the client (browser), web server, ArcGIS Server. Here are some notes and observations on each

Client(Browser) – When the graphics layer is maintained on the client-tier, it is drawn by the browser as VML/SVG depending on if you are using Internet Explorer / FireFox etc. In the case of the Flex and the upcoming Silverlight mapping controls, the graphics would be drawn as the corresponding framework’s graphic elements. The geometries of all the graphics are pulled from the ArcGIS Server to the browser (REST API).

Pros:

  • To identify any feature on the graphics layer, a round trip to the Web or the ArcGIS Server is not required.
  • The feature symbology/styles can be changed without a round trip to the server.
  • The graphics layer can be more interactive, like highlight feature on mouse over etc.

Cons:

  • The drawing performance can be slow when trying to draw large number of features in the graphics laye. The performance in the case of Flex and Silverlight controls will be better than the drawing performance of SVG/VML DOM elements.
  • You could be drawing only a few features (ploygons/polylines), but each of those features might have a large number of vertices. Imagine drawing county boundaries with thousands of vertices.
  • When drawing large number of features, large amounts of data (relatively) is pulled in by the client (browser).

Web Server – When the graphics layer is maintained on the Web server, it is drawn by the ElementGraphicsLayer or the FeatureGraphicsLayer classes in the Web ADF. These classes use GDI internally to create the graphic layer map images. These classes create images that are then sent down to the browser to be overlaid on top of the map element. Non-ESRI solutions can use the SharpMap project to create the graphic layer map images.

Pros:

  • Can draw large number of features relatively fast. Again drawing exorbitantly large number of feature can still make it slower.
  • Features can be organized into layers and easily styled using the available renderers like UniqueValueRenderer etc.

Cons:

  • Less interactive since highlighting features on mouse over will need a trip back to the web server.
  • Identify operations will need a trip back to the web server.
  • Style changes need trip back to web server.
  • The web server’s session storage and thus the memory could get overloaded.

ArcGIS server – There are many ways by which you could have the ArcGIS Server handle the drawing of the graphic features for you. But I am only going to discuss the cleanest of them all. In this case, when the graphics layer is drawn on/by the ArcGIS Server, the features and their symbology are passed onto the ArcGIS Server to be drawn using the ExportMapImage method in the SOAP API. The MapDescription argument of this method contains an array of GraphicElements that will be drawn by the ArcGIS Server.

Pros:

  • The graphic layer is embedded on the map image itself and doesn’t need to be pulled in as a separate image thus saving a little bit (very little) of bandwidth.
  • A good solution when it comes to printing/generating print map images with the graphics layer.

Cons:

  • Potentially large amounts of data is pushed between the web server and the ArcGIS Server.
  • The graphic layer’s features will need to be maintained on the web server’s session.

One thing to keep in mind is that none of the above solutions is a great solution (even thought they can be workable) for labeling features on the graphics layer.

ArcToSQL2008 – An ArcCatalog Command to export FeatureClasses to MS SQL SERVER 2008

Posted in ArcGIS, ESRI, GIS, SQL Server 2008, Utilities by viswaug on February 12, 2009

I had written this tool 3-4 months ago and it had been sitting on my machine waiting to be wrapped with an installer project. I finally got around to building a setup and deployment project for it and it is ready to distribute. The tool can be downloaded below

ArcToSQL2008

Once installed, you should be able to open the “Tools -> Customize” dialog in ArcCatalog

arctosql2008_1

Under the “GeoSpatial.NET” Categories, the “ArcToSQL2008” command should be available. Drag the command to any of the toolbars.

arctosql2008_2

Select the FeatureClass you want to export to the SQL Server 2008 database and click the “ArcToSQL2008” button on the toolbar. The dialog below should pop-up. Enter the name of the SQL Server 2008 database server and click connect. The Database drop-down will get populated with all available databases on the server. Select a database from the list and enter the name of the table that you want the FeatureClass exported to. Click OK. The FeatureClass will get exported and a message box should pop-up when the import is completed.

arctosql2008_3

Some highlights of the tool:

  • Exports FeatureClasses in a Projected Co-ordinate System as SQLGeometry in SQL 2008.
  • Exports FeatureClasses in a Geographic Co-ordinate System as SQLGeography in SQL 2008.
  • Any errors that occur during the export are logged in a log file found under in the install directory.
  • The tool is built using a broader GIS framework that should enable developers to customize the export process to their liking. I still have a little bit of house cleaning on the code for the broader GIS framework. I will post when it is all done. Hopefully that will be real soon. Contact me if you want to look at it sooner…

Things left to do:

  • Better UI showing the progress of the export operation.
  • More flexibility to the export operation by allowing the user to select the fields to be exported.
  • The export operation can be made faster by using the geometry builder classes to construct the SQLGeometry and SQLGeography objects. I took the shortcut for now and just parsed out the geometries from WKT strings.

Please let me know if you find any bugs in the tool using the “contact me” link on the right pane of this blog.

Options for implementing Spatial Validation rules

Posted in ArcGIS, ESRI by viswaug on February 10, 2009

I had a pretty engaging dialog with one of my colleagues today about how and where to implement spatial validation rules for one of the enterprise GIS web application that we are working on. I thought I would discuss some of those options here to possibly invite more suggestions or criticisms of some of the approaches

Implement and resolve the spatial validations on the client-side web interface only. Since this application is the only one updating the SDE, lets not spend a lot of effort performing spatial validations on the server-side also. Well, to put it in simpler terms, if the user is sketching a “Land Owner Boundary” that crosses state boundaries, then use the AGS REST API to make sure that the sketched shape does not cross state boundaries and if it does use a custom REST service to clip to the required boundaries. So, the spatial validation and the resolution is performed by the client before the shape is sent back to the server to be added as a new feature. But not performing all the validations on the server also just doesn’t feel right.

Perform all the spatial validation involved on the ArcGIS Server machine in Feature Class Extensions by implementing the various interfaces exposed in the ArcObjects SDK. This way all the data getting into the SDE has been passed through the same pipeline of validation methods before being inserted/updated into the SDE. If you are interested in going this route, check out the following interfaces that are available and will help get the job done. IClassExtension, IObjectClassExtension, IObjectClassEvents, IObjectClassValidation, IFeatureClassExtension. This method seems like a very good way too do things but there are some drawbacks to it. The first of which is implementing validations can get a little hairy fast and all the validation logic needs to be written in ArcObjects which is not a fast way of doing things. It would be easier, fast and more robust to construct Geo-processing models to perform these spatial validations. The bigger one I think is that when handling the OnCreate event or implementing the ValidateRow method, there is no way for us to pass in more context information into these methods which are necessary to perform the spatial validations required.

I personally think that the decision to put the validations in these feature class extensions as mentioned above is more similar to the argument of putting domain logic in Stored Procedures and Triggers in relational databases. It is very powerful, but also very constraining and rigid. Also, not all validations can be implemented this way.

Perform the spatial validation in the Domain Object model sitting on the web server( or ArcGIS server) machine. In our web application, we have a data access layer that performs the CRUD operations on an Oracle DB and the SDE. We also have the domain objects that models the entities in our domain/project and in turn use the data access layer to perform the operations required. This is a very flexible place to position the spatial validation logic since in this layer of the application, we have access to not only the entire domain object of the feature(all feature attributes) and other involved domain objects (related features etc) instead of just the data access objects which by themselves(individually) don’t tell us everything about that feature. This methods also lends itself better to unit testing than the previous two methods.

I am personally leaning towards the third option of having the spatial validations performed in the domain objects. Well, that is not to say that the spatial validation/resolutions need not be done on the client-side. I don’t think there is no escaping that nor should we attempt to. In the past, for Windows Forms based database applications, we could have represented all the validation logic in one place (on our POCO’s for example) using some library like the Validation Application Block in the Enterprise Library. This validation logic can be used on the UI (windows forms using IDataErrorInfo) and also on the server-side validations. But since we are swimming in the world wide web, the same validation logic needs to be represented in JavaScript using something like the jQuery Validation plug-in rules for example. Now, stack on top of that spatial validation, and it gets even better…<grin>

Automating Start/Stop AGS and AGS services

Posted in Agile, ArcGIS, ESRI by viswaug on February 10, 2009

I was working on setting up our automated build process for one of the projects I am working on and needed a way to start/stop the ArcGIS server running on a remote machine and also start/stop specific ArcGIS services running on remote machines. I was looking to see if ArcGIS Server provided any command line tools to perform these tasks so that I can just call these commands from my build file. But there is no out-of-the-box command line tools that come with ArcGIS Server which I could use for that purpose. So, I had to improvise and use some utilities and cook up a little bit of code.

To start/stop ArcGIS Services running on a remote machine, I was able to use the “PsService” from the Sysnternals Process utilities. Here is the crux of what PsService does.

PsService displays the status, configuration, and dependencies of a service, and allows you to start, stop, pause, resume and restart them. Unlike the SC utility, PsService enables you to logon to a remote system using a different account, for cases when the account from which you run it doesn’t have required permissions on the remote system

Usage: psservice [\\computer [-u username] [-p password]] <command> <options>

So, we can use the “stop” command and provide the service name for ArcGIS Server process “ArcServerObjectManager” as the option. Here is an example showing how it can be used.

psservice \\MachineName -u username -p password stop ArcServerObjectManager

Since, I had to call it from my NAnt build file, here is a sample of how it can be done

Setting up NAnt build file properties

<property name =“AGSMachineName” value=“myMachine”/>

<property name =“AGSAdminUser” value=“myUserName”/>

<property name =“AGSAdminPassword” value=“myPassword”/>

<!–comma seperated list of MapServices that needs to started/stopped–>

<property name =“AGSMapServices” value=“BaseMap, Boundaries, DynamicMap”/>

Targets to Start and Stop ArcGIS Server

<target name =“StopAGS”>

<exec program=“psservice” basedir=“.\tools\pstools\” append=“true”>

<arg line=“\\${AGSMachineName} -u ${AGSAdminUser} -p ${AGSAdminPassword} stop ArcServerObjectManager”/>

</exec>

</target>

<target name =“StartAGS”>

<exec program=“psservice” basedir=“.\tools\pstools\” append=“true”>

<arg line=“\\${AGSMachineName} -u ${AGSAdminUser} -p ${AGSAdminPassword} start ArcServerObjectManager”/>

</exec>

</target>

Now that’s good. The first hurdle has been crossed. But I still needed to start and stop specific MapServices that I desired. I couldn’t shortcut my way through this one and had to write some code to do this. The code to do it was pretty simple and didn’t turn out to be a pain.

The tool to start/stop ArcGIS services can be downloaded here

The source code for this tool can be downloaded here

The syntax to use the tool from the command line is

Usage: arcgisservice [serverName] [serviceName] [serviceType] [start | stop | restart] <username> <password>

The “username” & “password” for a user belonging to the “agsadmin” group can be specified here. If they are not specified, your current windows credentials will be used. And here is how it can be used in the NAnt build file.

Targets to Start and Stop ArcGIS MapServices specified as a comma seperated list

<target name=“StopMaps”>

<foreach item=“String” in=“${AGSMapServices}” delim=“,” property=“mapService”>

<exec program=“agsUtil” basedir=“.\tools\AGSTools\” append=“true”>

<arg line=” ${AGSMachineName} ${mapService} MapServer stop ${AGSAdminUser} ${AGSAdminPassword}”/>

</exec>

</foreach>

</target>

<target name=“StartMaps”>

<foreach item=“String” in=“${AGSMapServices}” delim=“,” property=“mapService”>

<exec program=“agsUtil” basedir=“.\tools\AGSTools\” append=“true”>

<arg line=“${AGSMachineName} ${mapService} MapServer start ${AGSAdminUser} ${AGSAdminPassword}”/>

</exec>

</foreach>

</target>

The target above will loop through and stop/start all the MapServices specified as a comma separated list in the “AGSMapServices” NAnt property.

And if you want to do the above and much more through another GUI didn’t want ArcCatalog installed all over the place, then the SOEXplorer would be the way to go.

A possible solution to getting your Geodatabase under source control

Posted in ArcGIS, ESRI by viswaug on January 21, 2009

A lot has already been said on the importance of getting your database under source control. If you haven’t given thought to including the database into your SVN, now is a good time to start and these tools can help. Various command line tools currently exist to script out the schema and optionally even the data to SQL script files on disk that can be checked into the SVN along with your source code. SQL Compare from redgate has the most complete solution with some features to die for. It will let you diff your current database with a SQL script that was generated earlier… yeah that’s nice. But it is gonna cost but well worth the money in my opinion. If you don’t want to dish out the $$$ for SQL Compare then the “Database Publishing Wizard” on CodePlex can step in to help your cause. It even has a command line version so that it can be easily incorporated into your automated build process.

Well, why did I say all that? I had been thinking about how I could get my Geodatabase under source control. Well, for starters we can export the Geodatabase to the XML Workspace Document format manually from ArcCatalog and check-in the XML document into source control.

export_xml_workspace

export_x603734039

If you are interested in automating that process of exporting your Geodatabase to XML then the ‘IGdbXmlExport‘ interface is your buddy and can be used to export and import schema and data. A NAnt or MSBuild task can written to include it as a part of your automated build process.

Filter Encoding Standard 1.1 and the curious case of the missing ‘IN’ operator

Posted in ArcGIS, ESRI, GIS by viswaug on January 20, 2009

I was working with the Filter Encoding (FE) 1.1 standard recently and was surprised to find that it did not have an equivalent to the ‘IN’ operator. The FE 1.1 standard currently supports the following comparison operator.

  •  
    • PropertyIsEqualTo
    • PropertyIsNotEqualTo
    • PropertyIsLessThan
    • PropertyIsGreaterThan
    • PropertyIsLessThanOrEqualTo
    • PropertyIsGreaterThanOrEqualTo
    • PropertyIsLike
    • PropertyIsBetween
    • PropertyIsNull

That was a big bummer for me since when I tried to use multiple “PropertyIsEqualTo” comparison operator coupled with the logical “OR” operator the performance was VERY bad. I am using ArcGIS server 9.3 like always.

The OGC support in ArcGIS Server is really great in case your are wondering. ArcGIS Server 9.3 supports the Filter Encoding 1.1, Styled Layer Descriptor 1.0 and other OGC standards as listed here http://www.esri.com/software/standards/standards_tables.html. Support for Styled Layer Descriptor 1.1 is not there yet but may happen in the future versions.

The FE 1.1 standard does include support for the ‘GmlObjectId’ object identifiers way of querying data. This can serve as the ‘IN’ operator if you are querying by the object identifiers. Unfortunately, I wasn’t. The Filter Encoding does include support for ‘Functions’ where custom filters can be supported by the GIS Server. And since, the ‘IN’ comparison operator is missing, it can be implemented as a custom ‘Function’ by GIS Servers to serve clients who desperately that functionality…