Tuesday, March 18, 2014

Vector Maps - Introduction

Here's the short version:  Vector Maps!  Yes we can!

Edit: Go check out Mapbox vector tiles and Thunderforest Lighting vector tiles too.

Somewhere in Brazil

WhirlyGlobe-Maply has had vector support for years.  As the versions went by that got much better, but vector maps were still a leap.  There were three missing pieces:
  • Paging: Efficient loading of individual tiles.
  • Styling: Specifying how a map looks.
  • Workflow: Getting from raw data to pretty map (open source, 'natch)

I'd done a few experiments and I was pretty clear on how to solve this.  All I needed was clients with the money to get it done.  I found them, we did it, and it works!

There's a lot to cover, so let's look at the big picture.

TileMill


It all starts with TileMill, MapBox's excellent free map making tool.  With TileMill you set up your data and then style your map with CartoCSS.  It's fairly intuitive and, best of all, well documented.

Braaaaaziiiiiil!

Under the hood Mapnik is the open source workhorse for TileMill.  Mapnik takes the data for the map, the styling information and makes pretty map tiles.

Mapnik is tres cool, but we're not actually using it to do the work.

Mapnik XML


Okay, so we're not using Mapnik, but we are using its configuration file.  See, Mapnik has this great file format that specifies absolutely everything you could want in one place.


Yup it's XML.  XML is like reading a headache, but it's fantastic for data processing.  It's got references to data sources, like Shapefiles or PostGIS queries.  It's got styles laid out in glorious detail.  It's got everything we need.  And did you know you could get Mapnik XML out of TileMill? 

Bet you always pick MBTiles.

We style the pretty map in TileMill, we export the Mapnik XML and that goes directly into the next stage.

Carto Vector Dice


Here's where we go off the rails.  I've got a new command line program that does the rest of the work.

The vectors going into TileMill are bad for rendering.  We need things diced, sorted, and organized.  That's what this program does.  It sifts, chops, and organizes the data by style and level.

That's where it gets tricky.  CartoCSS and TileMill aren't really designed for vector maps and they cheat really well with image tiles.  Rather than carefully select the right data to go in, you can throw it all in and let priority sort it out.  Now, that's not entirely fair, it does slow down your tile construction, but it's a much bigger problem with vectors.

Well.... how about just the one 40MB tile?

carto_vector_dice can sort out much of that.  Referencing the same data multiple times is really common in CartoCSS.  We can catch this and use multiple styles for the same data.  Even so, picking the right detail for the right level is something of an art.

Once all the dicing, sorting, and grouping is done, carto_vector_dice writes all this glorious data out to a database.

Maply Vector Tiles


The vector tiles are stored in a sqlite database.  There's one tile per (x,y,level) and a table of style information.  The data for each tile is compressed and stored in a blob, encoding all the traditional vector types and attribution.

Water Table.  I'm hilarious.
Putting everything into one database makes it easy to move around, but we've also planned for the future.  The style table uses UUIDs so we can reference styles globally.  The tiles are self contained so fetching over the network will be a cinch.

Vector Map Rendering


All of this work has one goal:  Get the vectors to the renderer.  Maply can now render a vector map with the following code.

Seriously, that's it.  There's a lot going on under the hood, but it's all surprisingly rational.  And it works great.

Insira espirituoso legenda aqui

One thing I gotta point out here.  We're not rendering map tiles and drawing the images.  We're Drawing.  The Actual.  Vectors.

Look, smartphones can handle this.  They've been able to for years now.  There's no reason not to render the vectors in a lot of cases.

I Want It!


So can you use it?  Well, in theory you can.  If you're a developer and you're familiar with GIS and a bunch of other stuff, probably.  If you're not, I would wait a bit.

I use this stuff with clients and I'm very happy with it, but it's got some rough edges.  Every new CartoCSS style is an adventure.  If you try it yourself, be patient and maybe drop me an email.

Future Stuff


The big, risky part is done and works.  There's plenty more to do, though.  I've got a long, long list of things clients may want.  If you want one of them, you could be a client too!

So that's it.  Vector maps work in Maply.  The workflow is in place and it's open source.  Look for improvements in the future and, of course, I'll have more to say on the blog.

4 comments:

  1. Hi,
    Very interesting post. I wondered if you had looked the GeoPackage OGC format?

    I think it may have the potential to be the solution to tile caches and vectors for use in desktop and mobile applications.

    The dev version of GDAL supports writing to GeoPackage.

    Cheers

    Tim

    ReplyDelete
    Replies
    1. GeoPackage is cool, but I don't want the spatialite overhead in my vector data blobs. I also want the freedom to do terrible, horrible things to the data to squeeze the size down even further. Then there's the network package angle...

      So while I'm glad to see them working on that, I'm not interested in using it myself *in this particular case*. If I need to deal with routing or if I have clients who want to run queries on the data, then I could see the utility.

      Of course, if I have a client who's using GeoPackage then I'll be delighted to add support. The internals of this part of Maply are well structured to support multiple approaches.

      Delete
  2. Hello,
    I run into this post from a answer you left at stackoverflow. I am using Tilemill and MBTiles for offline mapping in a App for Android and iOS. I am looking for a possibility to use (and create) Vector Maps (for offer more detailed maps) and I found your project, that looks like what I need.
    Last point I am not sure about is, if the way you describe above is a possibility to create Vector Maps, that a can use in the WhirlyGlobe-Maply toolkit. Or if you recommend any other toolchain for creating offline Vector Maps, that I could use with your toolkit.
    Thanks a lot in advance.
    Greetings

    ReplyDelete
    Replies
    1. I'm using vector tiles on a few client projects and they work, with a lot of caveats. At this point it's not a plug and play thing.

      Unfortunately, it sounds like others who have tried to do this without my help have run into problems. So it may work, or it may not depending on what version of vector tiles you're using (3 vs 4) and possibly a few other factors, like compression.

      I'm planning to do a little testing on this myself and work up something comprehensive, but I haven't had a chance yet.

      Delete