Vishful thinking…

Little known bottlenecks for ArcGIS Server Map cache generation performance

Posted in ArcGIS, ESRI, GIS, Uncategorized by viswaug on March 22, 2008

I had a chance to ask some of the ESRI developers the reason why the time required for our map cache generation didn’t reduce by very much when we started running SOCs on multiple machines (more powerful machines with Dual Quad core processors) with increased SOC instances. The bottleneck in these cases happen to be the following

  • ArcSDE
  • Disk speed (RPM) of the cache destination

Ok, now for the reasons as to why they are bottlenecks and some suggestions on how they can be worked around. The map service for which we were generating caches for, had almost all of its layers coming in from a single ArcSDE instance. When a large number of SOC instances (or processes) start to generate the map cache, all of them are accessing the same ArcSDE instance concurrently to generate the map cache. This multiple simultaneous access of the ArcSDE could slow down access to the map data and thus slow down map cache generation. One way this bottleneck could be worked around is by copying the ArcSDE data to multiple file geodatabases on different machines and referencing different layers in the map service from different file geodatabases. This way access to the map data is not throttled at just a single machine and is split amongst different machines.

All the SOC instances are also outputting their cache files to the same output destination. This means that this could also be a bottleneck if the disk speed of the output destination is not fast enough. One way that I can think of to work around this issue is by running different map cache generation processes for different scale levels and have their outputs going to different machines. Once done, the caches for the different scale levels can be copied to a single destination(ESRI uses SecureCopy to copy map caches between disks for their ArcGIS Online services). But, I don’t believe that this technique would save any time at all and would be too much trouble for not enough returns. So, just spend some money on a really fast hard disk.

Larger tile sizes for map caches take lesser disk space and generating them take lesser time. ArcGIS Online uses a 512X512 tile size whereas VirtualEarth and Google Maps use a 256X256 tile size.

6 Responses

Subscribe to comments with RSS.

  1. James Fee said, on March 22, 2008 at 4:08 pm

    Moving forward I’m going to start having to pay more attention to disk speed where the map cache is located. There were quite a few people who were “surprised” to realize that this was so important. Processor speed has so little to do with these map tiles and that has/had been the focus of AGS deployments.

  2. max said, on May 1, 2008 at 4:16 pm

    in reference to the tile size used in google earth and virtual earth, does the smaller tile size yield faster data retrieval/query/return over the web? If so is it a significant difference?
    What is the smallest tile size that you could create and have the efficiency of the client side data maximized?
    As I recall, the hard drive block size can also be a factor. I have not had any performance increases with the limited testing I have done when formatting the hard drive block size to match the tile size of the cache. I would like to hear of any correlated success in this. I believe it is also dependent on the image type that you are outputting in the cache directory. Anyone had luck using SVG in the output of the cache?

  3. Darren said, on March 6, 2009 at 4:01 pm

    It seems like a lot of work just to get map caches created. The amoount of time to copy data to file geodatabases and to copy caches back to their needed location would be too time consuming. I believe that ArcGIS Server needs a better method of using the spatial indexing in SDE where dynamic data that changes hourly can be displayed. Caches are fine for Aerial Photography, Contours, DTM Grids, and other static data but not for non-static layers. It appears the scale dependent rendering is the only thing that can be done. Caching on the fly is useless. Hopefully, ESRI will come up with a more usable solution. We’re staying with ArcIMS until ArcGIS Server can handle dynamic data better. There needs to be a mechanism that can detect changes in the data on version post and rebuild the cache just in the changed areas in the background.

  4. Shaun said, on June 27, 2009 at 11:56 am

    I’ve been creating and re-creating cache on servers with q-core and up to 8+8+8 config for all of SoCal. Creating cache down to ~18K is no problem (whole thing finishes in about 1 hour). Anything after this scale, and I have to start partial caching because the process never completes or shows 100% and never releases dll’s from memory. If I kill it after watching nothing happen on the server for several days, what I see is roughly 700K tiles w/ no data. Running the “recreate empties” task, I still run into the same issue where the thing won’t finish. Since 9.3 sp1 can’t be driven programmatically, I’m going to try the workaround you suggested as the ideas for resolving this issue aren’t going to be found in better hardware.

  5. David said, on December 22, 2009 at 1:04 pm

    Dear All,

    I would like to generate map cache from shape file, I was wondering if some one can help me how I can do this?
    Thanks in advance.
    My email: dshojaee@gmail.com

  6. David said, on December 23, 2009 at 5:15 am

    I have to add that I need this for mobile applications.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: