Vishful thinking…

Enhance your development experience with the ESRI Silverlight API using the Microsoft Reactive Extensions

Posted in .NET, ArcGIS, C#, ESRI, Uncategorized by viswaug on January 31, 2010

In my humble opinion, Reactive Extensions is one of the most exciting libraries to come out of Microsoft lately. And that is saying a lot since there have been quite a bit of good libraries coming out of Redmond recently. The Rx for .NET takes a lot of pain out of asynchronous programming and has drastically reduced the amount of code required to handle and orchestrate responses from multiple web services. Given that Silverlight is a client-side technology, most of the Silverlight applications make calls to web services and display results to the user as needed. A common application scenario is where the Silverlight client needs to make multiple calls to multiple web services and orchestrate calls to other web services based on responses from previous web service calls. Handling this scenario can get quite a bit complicated when you have to maintain contexts to store response states from the various web services. This is where Rx for .NET comes in real handy. The following scenario and the solutions should illustrate how Rx for .NET can help when programming with the ESRI Silverlight API for example.

When adding ArcGIS Map service layers to the Map control in the ESRI Silverlight API, the map control makes a HTTP request for every layer added to request the metadata for the map service. Depending on whether the request to the service metadata succeeds or fails the layer object would raise a ‘Initialized’ event or a ‘InitializationFailed’. So, a layer ‘load’ can be considered to be complete when either the ‘Initialized’ or the ‘InitializationFailed’ has fired. If we want to initiate another sequence of operations when all the ‘load’ of all the layers on the map have completed, like say populate a legend/ Table of Contents, then we will have to wait for all the layer ‘load’ to be complete (Success/Failure). Making a request to the legend web service in the above scenario can get quite cumbersome. But Rx for .NET can simplify things quite a bit. Follow the code sample below to accomplish the task.

The two extension methods below will help us watch for the ‘load’ complete for a layer and to watch for all the layers in a LayerCollection to complete. As you can see, we use the ‘Merge’, ‘Take’ and the ‘ForkJoin’ observable extension methods on the IObservable to watch for the right sequence of events.

public static class ExtensionMethods

{

    public static IObservable<IEvent<EventArgs>> GetLayerLoadCompleted( this Layer layer )

    {

        if( layer == null )

            throw new ArgumentNullException( "layer" );

 

        if( layer.IsInitialized == true )

        {

            IEvent<EventArgs>[] events = new IEvent<EventArgs>[ 1 ] { null };

            events.ToObservable<IEvent<EventArgs>>();

        }

 

        IObservable<IEvent<EventArgs>> oSuccess = Observable.FromEvent<EventArgs>

            (

                ev => layer.Initialized += ev,

                ev => layer.Initialized += ev

            );

 

        IObservable<IEvent<EventArgs>> oFailed = Observable.FromEvent<EventArgs>

            (

                ev => layer.InitializationFailed += ev,

                ev => layer.InitializationFailed += ev

            );

 

        IObservable<IEvent<EventArgs>> obsCompleted = oSuccess.Merge<IEvent<EventArgs>>( oFailed );

 

        IObservable<IEvent<EventArgs>> oFirst = obsCompleted.Take<IEvent<EventArgs>>(1);

 

        return oFirst;

    }

 

    public static IObservable<IEvent<EventArgs>[]> GetLayersLoadCompleted( this IEnumerable<Layer> layers )

    {

        if( layers == null )

            throw new ArgumentNullException( "layers" );

 

        return Observable.ForkJoin<IEvent<EventArgs>>

            (

                from item in layers

                select item.GetLayerLoadCompleted()

            );

    }

}

And now, we can add a handler to be executed when all the layers have loaded like below.

public MainPage()

{

    InitializeComponent();

 

    IDisposable subscription = null;

    subscription = myMap.Layers.GetLayersLoadCompleted().Subscribe

        (

            ( args ) =>

            {

                HtmlPage.Window.Alert( "All layers have loaded." );

                subscription.Dispose();

            }

        );

}

Advertisements

Black magic behind the MSD Map Services?

Posted in ArcGIS, ESRI by viswaug on September 29, 2009

Much has been said about the performance boost that can be achieved by running MSD map services over MXD map services. Here I will talk a little bit about the other side of the MSD map services which trips up developers like me. The only information that I was able to find about the capabilities/short-comings of the MSD map services from ESRI was on their blog post here. Since, I develop and use a few custom Server Object Extensions (SOEs), I noticed that some of the SOEs that I had written earlier had stopped working when registered with MSD map services. And the following piece of information from the blog post mentioned earlier seemed to indicate the cause of the problem.

What capabilities (server object extensions) are supported for MSD-based map services?

The WMS and KML capabilities are available with MSD-based services, as well as the default Mapping capability. Custom server object extensions can also work with MSD-based services but they cannot access fine-grained ArcObjects; they must use the coarse-grained methods on MapServer and its related classes.

The important part of the above statement is tha ‘custom SOEs cannot access fine-grained ArcObjects’. Well, what does that really mean? It kinda means that you cannot access the fine-grained ArcObjects features that are exposed through the IMapServerObjects interface on the MapServer server object. But being from the .NET world, I had trouble wrapping my head around how the same MapServer server object class can serve both the MXD map services and also the MSD map services and behave differently for each service type. How can the same MapServer class offer the IMapServerObjects features for MXD map services but not for MSD map services? I tried casting(QI) the MapServer server object obtained from a MSD map service to an IMapServerObjects interface which actually resulted in an exception being thrown. But the same cast worked on the MapServer object obtained from a MSD map service. My first thought when I noticed this behavior was that maybe the MSD map services are actually served by a different new server object class and not the MapServer class that serves the MXD map services. The ESRI  9.3.1 documentation doesn’t mention anything about a new server object. So, either ESRI didn’t document it or the same MapServer has developed multiple-personalities in AGS 9.3.1. To test it out, I obtained the CLSID of the server object that served MSD map services and the MXD map services. The were the exact SAME. That probably means that the same MapServer class is just behaving differently in different usage scenarios by leveraging some COM black magic which I don’t have too much knowledge about.

Since, we can’t get at the rich features in the IMapServerObjects interface, the custom SOEs on MSD map services really become only a little more useful than custom COM utility objects. So, some of the custom server object extensions that developers wrote earlier for MXD map services will not work with MSD map services. And that is a real bummer for me. One way (that I really really hate) to overcome this limitation is to run another MXD map service for the MSD map service. The MSD map service makes the maps and the MXD map service is only used to obtain and use the features of the  custom server object extension. I do really hate the idea of running a another expensive map service process on the AGS machine only for the purpose of using the features of the custom server object extensions. Although, I do have some ideas on a better ways of working around these limitations. The most frustrating part about this whole thing is that I think internally ESRI does have access to fine grained ArcObjects for the MSD map services and they just don’t let us know how to gain access to the fine grained ArcObjects. Why/How do I say that? The reason for my suspicions that the above is the case is because, the WMS and KML server object extensions in-fact DO work with MSD map services also. Which means that those server object extensions are able to access fine-grained ArcObjects to do what they need to do.

Update : Another change that relates to SOEs in AGS 9.3.1 is regarding it registration process. As many might have noticed, their SOEs shows (or doesn’t showup :)) up without any label in the ArcCatalog->Service Properties->Capabilities tab. It took me a while to realize what was happening. I had initially assumed that the SOE wasn’t showing up in the list. But later, luckily I discovered that the SOE was in-fact displaying but there was just no label for it in the Capabilities list box. In 9.3, the IServerObjectExtensionType interface has been superseded by the IServerObjectExtensionType2 interface ArcObjects style. So, when you register the SOE with ArcGIS Server by calling the CreateExtensionType, the returned object should be casted to the type IServerExtensionType2 and the ‘DisplayName‘ property should be set on it before calling AddExtensionType. This ‘DisplayName‘ property is what shows up in the capabilities tab and if it is not set, the entry for the SOE in the capabilities tab will be empty. Earlier, the extension’s Name property used to show up in the capabilities tab. But no longer, it is the DisplayName property that is used to label the entry in the capabilities tab list box. I hope ESRI finds some time to update their documentation about this over here.

Tagged with: ,

Thoughts on making the ESRI REST API a little more scalable

Posted in ESRI by viswaug on September 22, 2009

These are just a couple of my thoughts on how the ESRI REST API can be made a little more scalable. The ESRI REST API has been a boon to me and probably many other GIS developers out there who had to get knee deep in the ADF and build usable apps with it. The ADF has gotten a lot better in 9.3 and the ADF guys at ESRI have done a good job of improving the framework by leaps and bounds. But still, I stay away from the ADF partly because it is still not easy to use and debug. Definitely not as easy to use as the REST API with JS/Flex/Silverlight clients. The REST API inherently leverages the power of ‘HTTP goodness’ and makes the scalability of the apps that are built with the light-weight clients much more scalable. The scalability is achieved because of the fact that the data is cached as necessary both the REST server and the client browser.

One of the concerns I (and others) have with the current architecture of the REST server in AGS 9.3 is with the way how the cached map services are served out by the server. Even though, all the tiles have already been cooked and saved on disk. The default way the tiles are served out by the server is through the AGS REST server. This means that the tiles/images on the disk are not directly served out by IIS itself but the AGS REST handlers (implemented as IHTTPHandler in the library) need to get involved. Every tile request coming into the handler is served out really fast but IIS does have to call into the ASP.NET module that spins up a new thread for this request and this thread is blocked until the image request has been completed. Again, let me re-iterate. This image request-response by the REST handler is very fast. But, why do we need to be satisfied with fast when we can be FASTER. If the tile requests are made directly to just the tile images on a web server (doesn’t need to have AGS) by default, the request doesn’t even need to get to the AGS (which now doesn’t need to be externally accessible and can be behind your firewall if need be) and this tile request can be satisfied by just the web server without having AGS involved. This way cached layers can be served to the light-weight clients by just the web server which could in the cloud or the images can be stored and served directly from Amazon storage services. These things are currently possible with the light-weight clients but ESRI can definitely do their part to make that process much easier. All that’s fine but why is the performance going to be any better? Well, if the web server (IIS) is just serving out tile images from the disk, we can enable kernel-mode caching in IIS to server out these tiles which is magnitudes faster than serving it through the handler. When kernel-mode caching is enabled in IIS, the request is satisfied purely by IIS without the request having to enter the ASP.NET and be served by the IIS worker process. So, yeah it is MUCH MUCH faster with kernel-mode caching.

Also, in my snooping around the REST API, I found that the handlers in the REST API are implemented as IHTTPHandlers. The scalability of the API can be improved a little if these handled are written as IHTTPAsyncHandlers. When using IHTTPAsyncHandlers the threads serving these requests are not blocked until the request is served. Since, the serialization to JSON etc currently happen on the REST web server in AGS 9.3, the time to serve the requests might be non-trivial. So, this makes a higher number of threads available in the thread pool for the IIS worker process to use. This will definitely increase the scalability of the ESRI REST API and your web application. Given, there are things that we need to pay attention to when using this async programming model since, the number of threads could explode when serving very large number of requests and the process might just spend a lot time setting up the threads and cleaning up after them.

Those are just two thoughts I had on the subject today and I managed to put in words. I am also all ears to other ideas/criticisms on the subject.

I am also hoping that in the near future, ESRI’s SOAP and REST API have the exact same service contracts that are just exposed at different endpoints. Currently, the SOAP API seems to more feature-filled and the REST API seems to be playing catch up. Aren’t SOAP and REST just different ways of accessing the same exact GIS feature set in ArcGIS Server? Alright, I will end the rant right here. 🙂

GeoJSON & GPX support added to the ESRI Silverlight API Contrib

Posted in ArcGIS, ESRI, GIS, Silverlight by viswaug on August 18, 2009

I’ve added support for the GeoJSON and GPX layer types to the ESRI Silverlight API Contrib project on CodePlex. Adding GeoJSON or GPX layer to the ESRI Silverlight API map is real easy and you would add it just like adding a FeatureLayer to the map.

Adding a GeoJSON layer to the map

<esri:Map x:Name="MyMap" Grid.Row="1">
            <esri:Map.Layers>
                <esri:ArcGISTiledMapServiceLayer ID="StreetMapLayer"
Url="http://server.arcgisonline.com/ArcGIS/rest/services/ESRI_Imagery_World_2D/MapServer"/>
                <slContrib:GeoJSONLayer ID="geoJSONLayer"
                     URL="http://localhost/SLMaps.Web/Data/JSON.txt" />
            </esri:Map.Layers>
</esri:Map>

Adding a GPX layer to the map

<esri:Map x:Name="MyMap" Grid.Row="1">
            <esri:Map.Layers>
                <esri:ArcGISTiledMapServiceLayer ID="StreetMapLayer"
Url="http://server.arcgisonline.com/ArcGIS/rest/services/ESRI_Imagery_World_2D/MapServer"/>
                <slContrib:GPXLayer ID="gpxLayer"
                 URL="http://localhost/SLMaps.Web/Data/fells_loop.txt" />
            </esri:Map.Layers>
</esri:Map>

This adds to the GeoRSS layer support already present in the library.

Other than just adding these data sources as layers to the map, The GeoJSONReader & GPXReader classes can also be used to just parse the data source and obtain the geometry and attribute values of the features in the data source if/as required. This should allow for the users to upload GeoJSON & GPX data from their GPS units directly from their computers to the Silverlight map control without the need to upload files to the server. Allowing the workflow to be smoother, faster and simple both on the server-side & client-side.

Bing map issues with IE6 & Silverlight 2 / ESRI Silverlight API 1.0

Posted in ArcGIS, ESRI, Silverlight by viswaug on August 17, 2009

I have been having troubles for a while now getting the Bing map layers to show in IE6 with the Silverlight 2 installed. Unfortunately, when I got down to the bottom of it, it turned out to be a bug in Silverlight 2 that actually prevents the Bing map layers to show in the ESRI Silverlight API 1.0 map control. Initially, I wasn’t sure why the bing map layers where not showing up on just the IE6 browsers but displayed and worked just fine on IE7+ and Firefox. To check the issue for yourself, if you are using an IE6 browser with the Silverlight 2 plugin, just browse over the Bing maps demo site setup by ESRI to check it out for yourself. you will see the error message shown below.

Part of the error message displayed above in the alert window (“Make sure you are generating a VE token from the same token service as the VE Layer ServerType: Staging or Production”) is actually not accurate. The demo site simply handles the ‘InitializationFailed‘ event on the Bing TileLayer and displays the message above not matter what the error actually was. This had mislead me initially and had sent me hunting for clues in the wrong direction. One other weird I noted here in the ESRI Silverlight API 1.0 is that the event arguments for the ‘InitializationFailed‘ actually doesn’t provide you with the error information. But the error information is actually available in the ‘InitializationFailure‘ property of the layer which I thought was a little confusing. Hoping this will change in the next versions of the API.

After spending a lot of time on fiddler, snooping around the requests made to the Bing servers & searching the web, I finally figured out the reason why the Bing layers do not show up on the map. The reason turned out to be a bug in the Silverlight 2 plug-in in IE6. The error had to do with Silverlight 2 plug-in in IE6 having troubles handling compressed content returned from the server. More details about the bug can be found here. Since the compressed content in this case comes from the Bing servers, I had not control over it at all and thus it eventually turned out that the problem couldn’t be solved in a good way. But there was an advanced setting in IE6 that I could turn off to get the bing layers to show up in the map. Under the ‘Tools’ menu in IE6, select the ‘Internet Options’ item. In the ‘Internet Options’ dialog, click on the ‘Advanced’ tab and in the settings disable the ‘Use HTTP 1.1’ option.

But this workaround does have its own drawbacks of not being able to use some of the advanced HTTP 1.1 capabilities like compression etc. So, please be mindful of it before adopting the solution.

Some salvaging facts,

  • This bug has been fixed in Silverlight 3. So, users who are still using IE6 but have upgraded to the Silverlight 3 plug-in will not have this problem.
  • IE6 is losing market share – (though a good chunk of the users on the web still use IE6)
  • The Silverlight 2 plug-in is no longer available for download from Microsoft.
  • Windows update upgrades user with Silverlight 2 to Silverlight 3.

But, if you are writing a public application using the ESRI Silverlight API 1.0 that needs to support Silverlight 2 plug-in on IE6, you definitely might want to consider the above issue and display a message to you users asking them to upgrade or turn off the ‘Use HTTP 1.1’ option in IE6.

Deployment hassles with the ‘black box’ MXD file format

Posted in ArcGIS, ESRI by viswaug on July 18, 2009

The MXD file format has always remained a ‘black box’ with its proprietary binary file format which cannot be opened or modified without ArcMap / ArcCatalog / ArcObjects. It’s even got a voodoo doctor that magically treats all its ailments, the ‘MXD Doctor’ software utility. This is fine in the most of the desktop world where the user most probably has access to ArcMap / ArcCatalog. But in the world of server applications that use ArcGIS Server, it does become quite a hassle. The problem occurs during the deployment phase. During the development and the demo phase, we might have prepared the MXD file with all the layers referencing FeatureClasses in the development / demo  geodatabase. But when you move the MXD to the production environment, the layers should now be modified to reference the FeatureClasses from the production geodatabase. Well, to switch the FeatureClass data source reference for the layers in the MXD, you will need access to ArcMap / ArcCatalog. IMHO, I don’t believe that we should be installing ArcMap / ArcCatalog in the production server environment, especially for a reason like the one mentioned above. Installing a full-blown desktop GIS software on a server environment might not make the IT folks very happy and might open up / expose security vulnerabilities. Given the binary nature of the MXD file, I can’t open up the file in a text editor and switch the data sources. And AFAIK, I don’t believe that there is a tool / script that will help make the switch of the data sources without the need for ArcMAP / ArcCatalog. Also, it is not just the MXD file format, the other ESRI file format like ‘.sde’ and ‘.gds’ etc all are in a binary format which makes deployment more painful that it should really be.

Exploring/Exploiting the query operation in the ArcGIS REST API

Posted in ArcGIS, ESRI, GIS by viswaug on July 9, 2009

I was starting to look harder at the query operation in the ArcGIS Server REST API today to figure if I can leverage it to get specific results I needed instead of building my own REST services. The scenario I was looking at was to be able to get the records corresponding to the minimum and the maximum values in a given field. To provide an example, let’s say you have a ‘States’ layer with a field called ‘POP2008’ that contains population numbers for the year 2008. In this case, I want to obtain the state records with the minimum and the maximum population for the year 2008. At the outset, there didn’t seem to be a way to do it. After spending some time investigating the possibilities with the REST API query operation, I found that the above was in fact possible. Assuming that the FeatureClass name of the ‘States’ layer is ‘States_DTL’, the query below will produce the desired results.

POP2008 = (SELECT MIN(POP2008) from States_DTL) OR POP2008 = (SELECT MAX(POP2008) from States_DTL)

I was surprised to find out that I was able to use the name of a FeatureClass in the query. Agreed, the user of the REST API will/might not know the name of the FeatureClass. But the user could get lucky and be able to guess the name of the FetaureClass after ‘n’ tries. The FeatureClass / Table being used doesn’t even need to be a part of the MapService. I am not yet sure about how far this behavior can be exploited. Nevertheless, I found that this behavior was a little more than interesting.

Flashing a graphic in the ESRI Silverlight API

Posted in ESRI, Silverlight by viswaug on May 28, 2009

Being able to flash a shape on the map is a nice usability feature in GIS applications to relate shapes on the map to attributes in a data table or data grid. Also, most GIS users have gotten used to the feature in ArcMAP. I had written up an extension method to the Graphic class in the ESRI Silverlight API to be able to do this easily with a simple function call. Here is the code for the extension method.

        public static void Flash( this Graphic graphic )

        {

            Flash( graphic, 200, 10 );

        }

 

        public static void Flash( this Graphic graphic, double milliseconds, int repeat )

        {

            int count = 1;

            repeat = repeat * 2;

            Symbol tempSymbol = graphic.Symbol;

            Storyboard storyboard = new Storyboard();

            storyboard.Duration = TimeSpan.FromMilliseconds( milliseconds );

            graphic.Symbol = null;

            storyboard.Completed += ( sender, e ) =>

            {

                if( count % 2 == 1 )

                    graphic.Symbol = tempSymbol;

                else

                    graphic.Symbol = null;

 

                if( count <= repeat )

                    storyboard.Begin();

 

                count++;

            };

            storyboard.Begin();

        }

 

After including the above extension method in your project just call the ‘Flash’ method on the graphic object

graphic.Flash();

Note that I am using the Storyboard as the timer which is preferred over of the DispatcherTimer.

This has also been added to the ESRI Silverlight API Contrib project here.

Silverlight databinding limitations

Posted in ESRI, GIS, Silverlight by viswaug on April 14, 2009

Target of databinding in Silverlight “HAS” to be a “FrameworkElement” and not just a “DependencyObject” like in WPF. I am not sure that there is a reason why databinding to “DependencyObject”s in Silverlight is explicitly omitted by Microsoft. Seems like it might be an oversight. But even after many people have complained about it in the forums, the databinding to “DependencyObject”s feature doesn’t seem to be included in the beta version of the latest Silverlight 3 library.

So, what does that mean"? Where does it really become a limitation? Well, it really does limit our options when trying to databind values to the “Transform” classes like the “RotateTransform”, “ScaleTransform”, “SkewTransform”, “TranslateTransform” etc. Let’s say that you are symbolizing your points as an Ellipse or a Rectangle, and you want to be able to size your symbols based on some value in the Attributes collection of your Graphic. The best way to do this would be to apply a ScaleTransform on the Ellipse or Rectangle (on the RenderTransform property) and databind the attribute values to the ScaleX and the ScaleY dependency properties of the ScaleTransform. But since the ScaleTransform is only a DependencyObject and not a FrameworkElement, we cannot bind to the ScaleX and ScaleY properties of the ScaleTransform. This scenario can easily be solved by binding to the Width and Height property of the Rectangle or Ellipse. But there are a lot of other scenarios that cannot be overcome as easily. Like for example, you will not be able to label along a line by databinding to a RotateTransform on a TextBlock for example. Also in the ESRI Silverlight API, we cannot apply a TextSymbol to a polyline or polygon geometry.

Reason why JSON output is going to be much faster in ArcGIS Server 9.4

Posted in ESRI by viswaug on April 2, 2009

At this year’s ESRI dev summit, ESRI was showing off how the performance of the JSON output generation by the ArcGIS Server REST API is going to be much faster. I was curious as to how this could be since it is the same JSON that is being generated. The answer turned out to be pretty simple. Here is how JSON is generated today

93JSONmessaging

As you can see from the image above, at 9.3 the ArcGIS Server REST API does the work of reading (deserialize) the JSON sent in from the client (browser) over HTTP and converts (serialize) that appropriately into a SOAP message. This SOAP message is then sent over DCOM to the ArcGIS Server 9.3. The ArcGIS Server then converts the SOAP messages to ArcObjects types and does the work it needs to do. When the work is done, the ArcObjects types are converted back to SOAP and sent back to the REST API. The REST API then converts the SOAP types back to JSON and then sends the JSON results back to the client(browser).

In the 9.4, the JSON is going to be generated at the ArcGIS Server itself and thus avoiding a lot of serialization & deserialization to and from SOAP / JSON. Avoiding these intermediary data conversion steps are the primary reason for the increased performance of JSON output in 9.4.