December | 2008 | Vishful thinking...

ArcObject, PostGIS and spatial operations performance

Posted in GIS by viswaug on December 19, 2008

I have caught myself cursing at ArcObjects numerous times for not being performant for some of the spatial operations I had done using it in the past. I have also heard the same about it from other developers I have come across in the past. But what I was always curious about was, how does ArcObjects compare in performance to the other open-source spatial libraries like PostGIS? Since I have always been swimming in the ESRI and .NET ocean, I have little or no knowledge on how the PostGIS and Java worlds performed. So, it was interesting to listen to John Coryat speak this thoughts on how using the PostGIS library (that sits on top of the PostgreSQL database) makes spatial operations up to 10 -100 times slower than if those operations where performed using the spatial operations built into Postgresql.

Both ArcObjects and PostGIS suffer from the same drawback that make them significantly slower, they both are spatial libraries that run outside of the process space of the database where the spatial data is stored. Please see comments from Paul Ramsey below explaining that PostGIS actually runs within the process space of the database and how recents changes have made it considerably faster. ArcObjects runs outside of the process space of the database where the spatial data is stored.And that takes a major toll on the performance. Most databases these days including MS SQL Server 2008 pack a powerful set of spatial operations built right now that can satisfy the requirements for some cases but definitely not all of them. The spatial operations that are built right into the databases should always be prefered over using external spatial libraries since they run magnitudes faster. But the problem is that spatial operations included in those databases are not comprehensive and they all don’t pack the same geometry operations either. Most of them do support the SQL functions outlined by the OGC like SQL Server 2008 but some don’t. The most notable memberwould be MySQL.

To overcome or to add more capability to the spatial operations in these databases, an external spatial library become a necessity. Expecting the database vendors to develop spatial operations and to have to them keep up with the developments in the industry would not be an attractive option. This puts us, the spatial developers, in a catch-22 situation wherein we need to give up on performance to gain more powerful spatial capabilities. So, we may not be able to FIX the performance issues with spatial operations but just patch it as needed.

Even though John Coryat talks about his experience with PostGIS and PostgreSQL performance comparisions, that still doesn’t give a comparision between ArcObjects and PostGIS performance. It would be interesting to see some performance benchmarks between the two though.

2 comments

ArcGIS Server – Now faster and prettier in 9.3.1?

Posted in ArcGIS, ESRI by viswaug on December 19, 2008

I have to say that I am intrigued by this announcement about improvements in 9.3.1

What’s Coming in ArcGIS 9.3.1?

Especially the coming improvements in the dynamic map publishing below and also the Map Server Document (MSD) format.

High-Performance Dynamic Map Publishing

New faster rendering engine

Outperforms equivalent ArcIMS services.

Produces significantly better-looking maps.

Shortens map caching time.

Quicker, smoother zoom and pan.

New Map Optimization toolbar in ArcMap helps you tune-up your map documents before publishing to ArcGIS Server.

Review and respond to errors, unsupported content, and warnings about items that will slow down your dynamic map services.

Preview your optimized map document and estimated rendering time.

Save your optimized map document to the new Map Server Document (MSD) format.

2 comments

Creating a LEAN and FAST custom marker overlay for Google Maps

Posted in Uncategorized by viswaug on December 18, 2008

Before I start, I need to say that the link to the code below was provided to me by Pamela Fox. The sample below illustrates how to create a light weight custom marker GOverlay for Google maps that is REALLY FAST. And it allows for a multitude of customizations as you see fit. It was just too good not to share. Please find the the tweak I made to the code to allow for it to do a little bit of thematic mapping in just minutes. The demo below also allows you to check out the performance of the custom overlay with large number of markers loaded.

Here are the reasons provided by Pamela Fox for the default marker overlay in Google Maps to be slower than the custom marker overlay below.

The default marker class (GMarker) is designed so that it works well across-browsers, has hot spot shape, can be draggable, has a shadow, prints well in any browser, etc. It is not optimized for performance.

Here is the custom marker overlay code sample.

/* A Bar is a simple overlay that outlines a lat/lng bounds on the
 * map. It has a border of the given weight and color and can optionally
 * have a semi-transparent background color.
 * @param latlng {GLatLng} Point to place bar at.
 * @param opts {Object Literal} Passes configuration options - 
 *   weight, color, height, width, text, and offset.
 */
function MarkerLight(latlng, opts) {
  this.latlng = latlng;
  
  if (!opts) opts = {};
  
  this.height_ = opts.height || 32;
  this.width_ = opts.width || 32;
  this.image_ = opts.image;
  this.imageOver_ = opts.imageOver;
  this.clicked_ = 0;
}
  
/* MarkerLight extends GOverlay class from the Google Maps API
 */
MarkerLight.prototype = new GOverlay();
  
/* Creates the DIV representing this MarkerLight.
 * @param map {GMap2} Map that bar overlay is added to.
 */
MarkerLight.prototype.initialize = function(map) {
  var me = this;
  
  // Create the DIV representing our MarkerLight
  var div = document.createElement("div");
  div.style.border = "1px solid white";
  div.style.position = "absolute";
  div.style.paddingLeft = "0px";
  div.style.cursor = 'pointer';
  
  var img = document.createElement("img");
  img.src = me.image_;
  img.style.width = me.width_ + "px";
  img.style.height = me.height_ + "px";
  div.appendChild(img);  
  
  GEvent.addDomListener(div, "click", function(event) {
    me.clicked_ = 1;
    GEvent.trigger(me, "click");
  });
  
  map.getPane(G_MAP_MARKER_PANE).appendChild(div);
  
  this.map_ = map;
  this.div_ = div;
};
  
/* Remove the main DIV from the map pane
 */
MarkerLight.prototype.remove = function() {
  this.div_.parentNode.removeChild(this.div_);
};
  
/* Copy our data to a new MarkerLight
 * @return {MarkerLight} Copy of bar
 */
MarkerLight.prototype.copy = function() {
  var opts = {};
  opts.color = this.color_;
  opts.height = this.height_;
  opts.width = this.width_;
  opts.image = this.image_;
  opts.imageOver = this.image_;
  return new MarkerLight(this.latlng, opts);
};
  
/* Redraw the MarkerLight based on the current projection and zoom level
 * @param force {boolean} Helps decide whether to redraw overlay
 */
MarkerLight.prototype.redraw = function(force) {
  
  // We only need to redraw if the coordinate system has changed
  if (!force) return;
  
  // Calculate the DIV coordinates of two opposite corners 
  // of our bounds to get the size and position of our MarkerLight
  var divPixel = this.map_.fromLatLngToDivPixel(this.latlng);
  
  // Now position our DIV based on the DIV coordinates of our bounds
  this.div_.style.width = this.width_ + "px";
  this.div_.style.left = (divPixel.x) + "px"
  this.div_.style.height = (this.height_) + "px";
  this.div_.style.top = (divPixel.y) - this.height_ + "px";
};
  
MarkerLight.prototype.getZIndex = function(m) {
  return GOverlay.getZIndex(marker.getPoint().lat())-m.clicked*10000;
}
  
MarkerLight.prototype.getPoint = function() {
  return this.latlng;
};
  
MarkerLight.prototype.setStyle = function(style) {
  for (s in style) {
    this.div_.style[s] = style[s];
  }
};
  
MarkerLight.prototype.setImage = function(image) {
  this.div_.style.background = 'url("' + image + '")';
}
  

7 comments

Google Maps API or the VirtualEarth API? From a developer’s viewpoint

Posted in Uncategorized by viswaug on December 16, 2008

I came across an interesting blog post by the Redfin developers about their experience with moving from the VirtualEarth API to the Google Maps API. That got me thinking about my experiences with both the APIs. I am not going to be talking about whose imagery looks better or has a higher resolution here. But, as a developer, I should say that I actually prefer the Google Maps API over the VirtualEarth API. For most cases, both the APIs are really to use and are comparable in their feature sets. But here are something in the Google Maps API that I miss in the VirtualEarth API.

The TileOverlay feature in Google Maps is present in the form of a custom tile layer in VirtualEarth. The GroundOverlay feature in the Google Maps API is absent in VirtualEarth. Wouldn’t this feature come in handy?

The Google Maps API exposes their shape drawing and editing API but VirtualEarth doesn’t. This is a BIG one for me. Almost a MUST have. I don’t understand why the shape drawing and editing functionality available in live maps is not exposed in the API. I also love the vertex editing functionality and its usability especially how easy it is to insert a new vertex in a polygon.

The latitude/longitude encoding feature in the Google Maps API is a nice to have feature and will help save bandwidth. The GPolyline factory method helps create geometry overlays straight from the encoded string. Try out the encoding here.

The getLength and getArea functions on GPolyline and GPolygon are good enough for most cases where accuracy is not real important.

Google maps API provides static methods to handle XML documents and fragments (GXml) and methods to apply Xslt to Xml (GXslt). The VirtualEarth API doesn’t contain any such equivalents.

The geo-coding functionality in Google Maps also has added functionality like

A client side geocode cache.

Ability to set an extent within which to geocode using the setViewport function on the GClientGeocoder.

Geocoding via HTTP requests.

A useful set of utility libraries for the Google maps API.

The StreetView is currently only available in google maps

Did I miss anything else either with the Google Maps API or the VirtualEarth API?

7 comments

Conceptualizing AtomPub for Map services

Posted in Uncategorized by viswaug on December 8, 2008

I have written about some of the advantages that a REST API can bring to exposing GIS data. This is not just for delivering map images but also for delivering and managing vector data. We have been building a RESTful API to not only expose vector GIS data, but to perform all of the CRUD operations on the data as required. Since most of our mapping applications are built using the ESRI JS API, the REST API makes it a breeze to access those functionalities through URL endpoints.

Not to revisit the topic again but the idea behind REST is to use HTTP as an ‘application protocol’ rather than as a transport protocol like SOAP web services do. One protocol that uses HTTP the way it is meant to be used is the AtomPub protocol. I am not going to go into details about the AtomPub protocol here since there a lot of documentation available online that do a good job of explaining the AtomPub protocol. But in brief terms,

The Atom Publishing Protocol is an application-level protocol for publishing and editing Web Resources using HTTP [RFC2616] and XML 1.0 [REC-xml].

AtomPub is both a format and a protocol. And I think that the AtomPub is a natural way to expose data especially also GIS data. I am going to go over how in my opinion GIS data exposed as map services seamlessly falls into the AtomPub data container structure. The figure below illustrates a simplified data container structure in AtomPub.

As you can see from the above figure, an AtomPub service document contains a list of ‘Workspaces’ which in turn contains a list of ‘Collections’. The ‘Collections’ are represented by Atom Feed documents. The Atom Feed document contain a list of ‘Entrys’. In my opinion, here is how the ‘Service Document’ in the AtomPub protocol relates to a map service. (I am only referring to ESRI Map services below).

Service Document – The Map Service itself – The entry point that describes the location and capabilities of collections

Workspace – Data Frames or Group Layers – A logical grouping of collections

Collection/Atom Feeds – Feature Layers – A Resource that contains a set of Member Resources. Collections are represented as Atom Feeds.

Entry – Feature – Represents an individual piece of content/Data

Categories Document/Categories Element – Feature Type – Provides metadata to describe an Atom entry. Categories Document describe Categories that are allowed in collections.

The AtomPub ‘Category’ element can also be used to represent feature types like ‘LinearRing’, ‘Polygon’, ‘LineString’, ‘MultiLineString’, ‘MultiPoint’ and be attached to Atom entries to provide metadata. They can also be used to represent the domain entity represented by the Feature Layer like National Forest etc…

Service Document Sample:

<?xml version=“1.0” encoding=‘utf-8’?>

<atom:title>Transportation</atom:title>

<atom:title>National Highways</atom:title>

</collection>

<atom:title>Rivers</atom:title>

<accept>image/json</accept>

<accept>image/xml</accept>

</collection>

</workspace>

<atom:title>Water Bodies</atom:title>

<atom:title>Lakes</atom:title>

<accept>application/atom+xml;type=entry</accept>

<atom:category scheme=“http://vishcio.us/geometry/polygon” term=“polygon” />

<atom:category scheme=“http://vishcio.us/Lakes” term=“Lakes” />

</categories>

</collection>

</workspace>

</service>

Atom Feed Document:

<?xml version=“1.0” encoding=“utf-8”?>

<title>Example Feed</title>

<subtitle>A subtitle.</subtitle>

<email>johndoe@example.com</email>

</author>

<entry>

<title>Atom-Powered Robots Run Amok</title>

</entry>

</feed>

Categories Document:

<?xml version=“1.0” ?>

<app:categories xmlns:app=“http://www.w3.org/2007/app” xmlns:atom=“http://www.w3.org/2005/Atom” fixed=“yes” scheme=“http://example.com/cats/big3”>

<atom:category term=“animal” />

<atom:category term=“vegetable” />

<atom:category term=“mineral” />

</app:categories>

The above is just my take on how the AtomPub elements can relate to map services. I think this is the next logical step in making map services more resource oriented and will allow for dynamic discoverability and ease in application development. ESRI map services already have the concept of a service document that can be accessed at the links below.

http://sampleserver1.arcgisonline.com/arcgis/rest/services

http://sampleserver1.arcgisonline.com/arcgis/rest/services?f=pjson

Supporting the AtomPub output format for that service document should be pretty simple.

1 comment

URL & URI What’s the difference?

Posted in Uncategorized by viswaug on December 3, 2008

I am going to attempt to highlight the difference between URLs and URIs the way I see it. I am sure the purist might disagree with my take on it. So here goes nothing…

URI – Uniform Resource Identifier – This identifies a resource uniquely. The identifier used could optionally contain the location and the name or not. Ok, you have read that before…what does it really mean? Here is how I see it.

Take ISBN number for instance. It helps you identify the book uniquely. Let’s assume you are interested in the book with the ISBN number ‘ISBN-10: 0596519206′. Using an ISBN number you can confirm as to whether any book you are looking at is the book you are interested in. But using the number, you cannot get access to the book, you can only confirm whether you have found the book or not. This doesn’t mean that an URI cannot tell you how to access the resource also or not, but it SHOULD at least let you differentiate that resource from all others.

URL – Uniform Resource Locator – This lets you know how to get access to that particular resource. In most cases, URI is contained as a sub-part within the URL. URL contains not just the resource identifying information (URI) but also the resource locating information. Well, take for example, the above book resource, a URL for the book could be ‘http://www.amazon.com/dp/0596519206/‘. Here you can see that the ISBN number (URI) is contained as a part of the URL. But the URL also provides you more information that lets you gain access to it. Like, it lets you know that the resource can be accessed on the ‘World Wide Web’ via the HTTP protocol at the Amazon address using the ISBN number.

That said, I would also like to add that the two terms URL and URI are used so interchangeably these days that the subtle difference really doesn’t have a lot of significance.

1 comment