A more technical summary of how I built my tileserver – part 1

I thought I should put a discussion of how I did this, with much more detail, as I am sure there are other people out there in the world who might want to do something similar.

This is part 1. I’ll post part 2 later.

Background

I wanted to be able to serve Openstreetmap-style map tiles of my own fictional planet, in the same way that the site OpenGeofiction does, but using my own data set.

This process of building a tileserver is separate from the job of setting up an Openstreetmap-style apidb database to be able to edit the data set using tools such as iD, Potlatch, or JOSM. I’m still working on that.

Platform and Preliminaries

I deliberately set up my server on Ubuntu 16.04 (a flavor of Debian Linux) because I knew that OpenGeofiction runs in this environment. I’m not actually sure, but I assume Openstreetmap does too, though, given its scale, that may not be exactly the case, anymore – more likely it’s got a kind of customized, clustered Linux fork that has some genetic relationship to Ubuntu.

I thought it would therefore be easier to replicate the OpenGeofiction application stack.

Before starting this work, I had already installed MySQL and Apache and Mediawiki – except for Apache, however, these are not relevant to setting up a tileserver.

I had also already set up PostgreSQL (the preferred Openstreetmap database server), so the preliminary mentions of setting up this application were skipped.

Finally, using Apache’s sites-available config files and DNS, I had set up a subdomain on my server, tile.geofictician.net, to be the “outside address” for my tileserver. This will hopefully mean that if I ever decide to separate my tileserver from other things running on the same server, it will be somewhat easier to do.

S2OMBaTS with Deviations

Starting out, I mostly followed the steps and documentation at switch2osm.org’s detailed tutorial, here. Below, I refer to this page as S2OMBaTS (“switch2osm manually building a tile server”).

So I don’t see any need to repeat everything it says there. I just followed the steps given on that webpage exactly and religiously. What I’ll document are only the spots where I had to do something differently. These are my “deviations.”

  1. Where S2OMBaTS suggests creating a ‘renderaccount’ on the server to own all the tileserver-related directories and tools, I used my non-root regular username. I’m not sure this is good practice, and if I were setting something up as a “production” environment, I’d be more careful to segregate ownership of this collection of files, applications and services.
  2. There are some problems with authenticating a non-root user for PostgresSQL (‘root’ being the infamous ‘postgres’ superuser). I had to edit the /etc/postgresql/9.5/main/pg_hba.conf file so that the authentication method was “trust”

    [css]
    # Database administrative login by Unix domain socket
    local all postgres trust

    # TYPE DATABASE USER ADDRESS METHOD

    # "local" is for Unix domain socket connections only
    local all all trust
    [/css]

    I think this might be a bad solution from a security standpoint, but it’s the only one I could find that I understood and could get to work. PostgreSQL security is weird, to me. My DBA experience was entirely with SQLServer and Oracle, back in the day, and those databases’ security are integrated to OS security more tightly, I think. Similarly, MySQL seems to assume linkages between system users and database users, so security for the matched pairs of users are linked. But it seems like PostgreSQL doesn’t work that way.

  3. Where S2OMBaTS suggests using the URI=/hot/ in the /usr/local/etc/renderd.conf file (which seems intended to hijack other applications’ support for the already-existing “HOT” – Humanitarian Openstreetmap Team – layer). I used URI=/h/ instead, which was entirely arbitrary and I could just as easily have used something more meaningful, as at OpenGeofiction, with e.g. URI=/osmcarto/.
  4. To test my installation, of course, I had to load some test data. S2OMBaTS uses a geofabrik snapshot of Azerbaijan. I decided just for the sake of familiarity, to use a snapshot of South Korea. I had to spend quite a bit of time researching and tweaking the individual osm2pgsql options (parameters) to get it to run on my itty-bitty server, even for a fairly small dataset like South Korea’s OSM snapshot, so here’s the osm2pgsql invokation I used to load the data (YMMV).
    osm2pgsql --database gis --create --slim  --multi-geometry --keep-coastlines --hstore --verbose --tag-transform-script ~/src/openstreetmap-carto/openstreetmap-carto.lua --cache 2500 --cache-strategy optimized --number-processes 1 --style ~/src/openstreetmap-carto/openstreetmap-carto.style ~/data/south-korea-latest.osm.pbf
    

At this point, I reached the end of the S2OMBaTS tutorial.

Loading my own planet

I then had to customize things to load my own planet instead of a largely “naked earth” with South Korea well-mapped. The first step was easy enough. I just replaced the South Korea pbf extract with a pbf of my own planet, and re-ran the osm2pgsql step. I got the pbf extract of my planet by working with some kludges and with JOSM on my desktop machine. It was a “simplified” planet – just the continent outlines, a few cities, two countries with their admin_level=2 boundaries, and one tiny outlying island with lots of detail, which I borrowed from my city-state Tárrases at OpenGeofiction. It was composed as a kind of “test-planet” to keep things simple but hopefully test most of what I wanted to achieve in my tileserver.

Here’s the load script for that (essentially the same as used for South Korea, above).

osm2pgsql --database gis --create --slim  --multi-geometry --keep-coastlines --hstore --verbose --tag-transform-script ~/src/openstreetmap-carto/openstreetmap-carto.lua --cache 2500 --cache-strategy optimized --number-processes 1 --style ~/src/openstreetmap-carto/openstreetmap-carto.style ~/data/gf-planet.osm.pbf

The problem, of course, is that if you run the render at this point, you get all the features of the new planet, but the continent outlines and the land-water distinction is “inherited” from earth. That’s because the mapnik style being used is referencing the shapefiles produced by and downloaded from Openstreetmap. The creators of the Openstreetmap software, including the OSM carto style, didn’t take into account the possibility that someone would try to use their software to show a map of somewhere that wasn’t planet Earth, and consequently, these need for these shapefiles is “hardcoded.” So the “Earth” shapefiles have to be substituted by alternate shapefiles extracted from the alternate planet dataset.

Customizing Coastlines and Shapefile Hell

This was the hardest part for me. It took me more than a week to figure it all out. I’m not experienced with shapefiles, and don’t really understand them, and the process by which shapefiles get extracted from the OSM global dataset in a format that can be used by the openstreetmap-carto mapnik style is very poorly documented, online. So it was a lot of google-fu and experimentation, and downloading QGIS and teaching myself a bit about shapefiles, before I could get things working. It’s not clear to me that I really did it the right way. All I can say is that it seems to work.

The first steps I took were to try to simplify my task. I did this by chasing down the shapefile dependencies in the mapnik style sheet, and manually removing the ones that seemed less important. I did this mostly through trial and error.

The only file that needs to be edited to accomplish this simplification is the main mapnik xml file: <YOUR-PATH>/openstreetmap-carto/mapnik.xml. Bear in mind, though, that this file is the output of the carto engine (or whatever it’s called). By editing it, I have “broken” it – I won’t be able to upgrade my OSM carto style easily. But this is just a test run, right? I just wanted to get it to work.

So I edited the <YOUR-PATH>/openstreetmap-carto/mapnik.xml file and deleted some stuff. You have to be comfortable just going in and hacking around the giant xml file – just remember to only delete things at the same branch level of the tree structure so you don’t end up breaking the tree.

I removed the <Style></Style> sections that mentioned Antarctic icesheets – there were two. As things stand, my planet has no Antarctic icesheets, so why try to incorporate shapefiles supporting them?

Then, I eliminated the<Style></Style> section mentioning the <YOUR-PATH>/openstreetmap-carto/data/ne_110m_admin_0_boundary_lines_land directory, since these are land-boundaries for Earthly nations. I figured if I couldn’t see land-boundaries for my planet’s nations at low zooms, it would be no big deal. It’s not clear to me that this has been implemented on OpenGeofiction, either.

I also discovered that in fact, this file doesn’t even point to the <YOUR-PATH>/openstreetmap-carto/data/world_boundaries directory. So there was no need to worry about that one.

So that left me with two shapefiles I had to recreate for my own planet’s data:
<YOUR-PATH>/openstreetmap-carto/data/land-polygons-split-3857/land_polygons.shp and <YOUR-PATH>/openstreetmap-carto/data/simplified-land-polygons-complete-3857/simplified_land_polygons.shp.

Let’s just summarize by saying that this is what took so long. I had to figure out how to create shapefiles that the mapnik style would know what to do with, so that my continents would appear on the render. It took a lot of trial and error, but I’ll document what’s working for me, so far.

*** To be continued ***

[Update 20180923: Continues here]

Music to map by: Héctor Acosta, “Tu Veneno.”

Published by

Luciano Jones

that guy who likes to make maps.

2 thoughts on “A more technical summary of how I built my tileserver – part 1”

  1. Thilo Stapf, the creator of OpenGeofiction, took the time to comment on this write-up, and sent it to me in email (because comments on this blog aren’t yet configured correctly, apparently – I sort of knew that, but wasn’t expecting comments so soon!). I’ll post his message here in comment form, along with my reply.

    a few remarks about your blog entry (actually I wanted to comment on it directly, but it asks for a login).

    The coastline part was indeed always what caused the most headache, so I can legitimately say that I feel your pain. IMO the right way to do it is to use the “OSMCoastline” tool:

    https://wiki.openstreetmap.org/wiki/OSMCoastline

    Nevertheless I too started with creating shapefiles myself (wrote a Perl script), because OSMCoastline did not yet exist then.

    I worked around the problem with the other shapefiles not by removing anything from mapnik.xml, but just generated an empty shapefile which I then copied multiple times. OGF does in fact not show country boundaries at zoom level 3 and below:

    http://tile.opengeofiction.net/util/map_scale.html?map=C/3.005786878064508/43.26330/50.22701&map2=OSM/x/49.13363/42.83179&pxpt=0.277&scale=51200000

    (Maybe I can generate a suitable shapefile with some tweaked version of the “simplifiedAdminBoundaries” script that I use for the OGF overview map. I guess I’ll give it a try.)

    BTW, for quick coastline extraction, I seriously recommend to consider installing the Overpass API. This is useful for a lot of things, and OGF as it is today would have a hard time to live without it. Besides coastline generation, I use it for exporting the backup files, and it has a role in my tile expiry algorithm as well as in generating the coastline error map. Moreover, it can be used to build some quick and dirty search functionality, like here:

    http://tile.opengeofiction.net/util/search.htm

    (Now that martinum4 has gone lost for a while, OGF might even need to fall back to using that method.)

    1. Wow, feedback from the expert! I very much appreciate it.

      I think setting up commenting so that it works well on my blog might be more work than it’s worth – the blog is only a few days old and it’s already been swamped by weird wordpress blog spammers/scammers/phishers of various kinds. That’s the hazard of “hosting your own” as opposed to using a hosting service (as I do on my “main” blog), which has high powered and well configured spam prevention tools.

      I have somewhat deliberately held off from requesting your advice or help… partly because I have always found that the BEST way to teach myself new skills is to “figure it out by myself” rather than being told, if that makes sense. So I kind of move from puzzle to puzzle, solving each one in turn as best I can, and it has been very educational.

      Indeed, I found the OSMCoastline tool. That’s “Part 2” of my write up, where I will explain that in great detail. Except for Mr Topf’s INSTALL.md write up on github, which is pretty good but limited, documentation on the OSMCoastline tool is essentially non-existent, so it took a while for me to figure out how to use it, and I intend to document that so that I can contribute to the “googleable documentation” pool.

      It didn’t occur to me that an “empty” shapefile might be a way to bypass the requirement. It makes sense – that way your OSM Carto install remains “clean” and thus upgradeable. Obviously that’s the way to go, but I may go in the direction of forking my own map style, instead, with the dependencies permanently removed – I don’t feel compelled to stick to the OSM standard, because Rahet and Mahhal are a different sort of project than OGF. They’re science fiction planets, and their technology level is “different” – not necessarily higher or lower, just different. I’ll never need motorways and all the classes of highways in the same way OSM or OGF need them. I may play around with some alternate mapnik styles.

      Once I figure out the Rails Port and get that up and running, I’ll probably take a break from setup stuff and see about just doing some editing againt the two maps. I have a problem to solve which you didn’t, too: I intend to make some kind of “planet switching” script – so I can have one planet loaded where I can work on it, and render it, but then switch to the other. I don’t think it will be that hard (just a matter of backing one up, then deleting it from the database and loading the other), but it will mean each planet’s existence is discontinuous on the server. Rahet and Mahhal are “twin planets” and I want to develop both of them in parallel. I suppose at some point, if it’s all going well and I’m feeling rich, I could build a second server for the second planet, but I’d like to get comfortable with what I’m doing first and see how it goes.

      Having said that, I suppose I need to figure out the Overpass API at some point. I didn’t realize you were relying on it directly for the coastline updates.

      I’m actually much more interested in using the server to play around with the topo stuff. I started to try to do that on my desktop and realized I just didn’t have the space or power to do it. That’s probably what pushed me to getting a server, more than any other thing … otherwise I was perfectly happy to edit Rahet and Mahhal in JOSM and not worry about the rendering aspect too much. So my thinking has been, “FIRST, replicate the basic OGF stack… THEN, I can start messing with topo.”

      Anyway, I am very grateful for your comments and thoughts. I definitely intend to remain with OGF (with my typical lack of focus made worse by this new side project). I’ve always know that if I got completely “stuck” on something, I could contact you for advice, but as I said, I learn best by trying and doing, so I’m mostly forcing myself to work through things on my own.

      Incidentally, the last link in your message (http://tile.opengeofiction.net/util/search.htm) gives me a 404.

      Happy mapping!

Comments are closed.