Geofictician Blog – Page 5

On Castine and its many tribulations

Castine is an imaginary country that once existed on the imaginary planet I prefer to call Ogieff. In fact, the imaginary planet doesn’t have an official name – it’s hosted at opengeofiction.net, which all the users call, simply, “OGF”. That initialism leads to my preferred name for the planet – just sound it out.

There is a real place called Castine – it’s a small town in Maine, USA. This is not that Castine.

I joined OGF in 2014, and Castine appeared and began evolving sometime in the year after that, I think – in 2015. I also became an admin on the opengeofiction.net website in that year.

During the period from 2015 to 2017, Castine became the locus of a kind of meta-proxy-war, where I used it as a stand-in for a never-ending argument I liked to have with my fellow OGF admins.

The issue in question was the rule about “verisimilitude”. I had long felt (and continue to feel) that OGF’s verisimilitude rule is a bad idea – it’s vague and impossible to enforce consistently. It has no objectivity. The principle is that mapping on the OGF world is supposed to be “realistic” in the sense that it eschews fantasy and sci-fi elements, and doesn’t contain cultural or cartographic artifacts that couldn’t reasonably exist in the real world. Hence, people who build 50 km bridges or tunnels are called out for violating verisimilitude, likewise more science-fictional elements like space elevators or fantasy elements like dens of dragons or nations of 1920’s-era talking sheep (all these examples really occurred at various times on the OGF planet).

Castine was (is) a borderline case of violating verisimilitude. Some users felt it violated the rule, others felt it was okay. My position was always something like: “since we can’t decide if this violates verisimilitude or not, but it’s really good mapping… c’cmon, people, let’s drop (or at least, fix) this stupid rule.”

Of course, this was an unpopular stance. And in the long run, I lost the battle to remove or even alter the verisimilitude rule on Ogieff, and I made my peace with it.

One way that I made that peace with it, was to create my own, separate planet! In 2016, I started the planet Arhet as a kind of alternative project to Ogieff. By 2018, it had several active mappers and its own emerging community. The principle concept behind Arhet is to be a kind of “libertarian” reinterpretation of OGF. It has very few rules: no verisimilitude rule, no assigned territories, etc. And somewhat to my own surprise, it sorta kinda works. The key to it working, I reckon, is that unlike OGF, Arhet is not “open” to any and all comers. There’s an application process to join, and although I enforce almost no rules for the planet, I do stand firm that arguments or disagreements between users that escalate to my remit will simply result in immediate banning of all parties. That keeps everyone participating on best behavior, I guess.

The irony is that then, in 2021, I took over the hosting of the original opengeofiction.net. So now I host a little federation of two imaginary planets, Ogieff and Arhet, which have substantially overlapping user communities but having quite different rule systems. And I’m okay about that. I inevitably yield to my fellow admins, whose hard work and dedication to the project I admire, when it comes to matters of rules and judgements on Ogieff. But off to the side, I run Arhet singularly, and I insist on its fundamentally anarchic state.

In around 2020, the creator of Castine (Ramasham) was banned from Ogieff – ultimately for violating another, different rule: the rule prohibiting direct upload of data copied from OSM. OSM is OpenStreetMap, which is a map of the Real World™ in the same technological vein as our two imaginary planets. This is the so-called “slippy map” paradigm, originally popularized by mapquest and perfected and dominated by google maps. OSM runs on and supports a whole complex ecosystem of software that is all open source, as a kind of alternative to google maps, and that’s why it’s easy (uh, “easy” in a financial sense, not “easy” in a technical sense) for us to use the same software to run OGF and Arhet.

Anyway, there is (and there has always been) a rule prohibiting copying OSM data into OGF. Ostensibly this is motivated by paranoia about copyright violation, but in fact copyright has little to do with it, in my own estimation – there are easy ways to avoid issues around copyright as long as you follow along with OSM’s “attribution and re-use” rules. The real motivation for the prohibition is legitimate, though: on OGF, we want to discourage mappers from spamming the planet’s map with cut-n-paste copies of real-world places. It’s low effort geofiction and discourages creativity.

That said, when I set up Arhet I decided to also not enforce OGF’s “no real-world (OSM) data” rule. And indeed I myself played around with cutting and pasting some data from OSM, including an ephemeral instance of country I called “Lingit Aani” (this is Tlingit language) – a copy of the islands of Southeast Alaska but minus any nearby continent, as an open-ocean archipelago. I later deleted this, but there are multiple copy-the-real-world geofiction projects going on in Arhet, these days, including clones of Sakhalin Island (Siberia) and Romania’s Bucharest, and at least two Polands – perhaps more.

I guess Castine’s creator, Ramasham, had been doing some copy-pasting of OSM data to increase the detail and complexity of Castine’s cartography. Notably, this airport is a modified cut-n-paste copy of one in the real world, with only the names of things altered. And so Ramasham was banned from OGF. Rules are rules, and that “no copy from OSM” rule is actually probably the most common reason for mappers to be banned from the site.

Now we come to February of this year (2022). The admin team at OGF, moving to “clean up” various abandoned territories around our (imaginary) globe, decided finally to delete Castine once and for all. And I had a moment of deep sadness and regret. Despite my having leveraged Castine back in 2016 as part of my proxy war with the other admins over the verisimilitude rule, in fact I really, really like Castine.

From a technical standpoint, Ramasham was at best a mediocre mapper. But the imaginary country is full of cartographic whimsy and playfulness, the naming is thorough and inventive and culturally intriguing, and the detail in some parts is quite incredible. I thought it was worth preserving.

So I considered: Ramasham’s ban from OGF was for violating the “No OSM data” rule; if there were any other issues with Castine, they were issues with the “verisimilitude” rule; so… hey – Arhet doesn’t have those rules!

The solution was obvious. I decided I’d move Castine to Arhet. And even more conveniently, the exact latitude and longitude of Castine’s old Ogieff location was open an unused on Arhet. I figured it should be quite easy to simply “cut-n-paste” the whole of Castine into Arhet.

Yikes! This turned out to be the far from the case – it was not easy. Not at all. Castine included almost 2 million distinct GIS objects: nodes, ways, relations. This was not trivial to simply cut, paste, and upload into the new site. And further, the data quality was quite poor, from a technical standpoint. Thousands of improperly stacked ways on shared nodes, hundreds of lazily-crafted or incomplete data relations, etc.

I have spent the last week in a kind of Alice-in-Wonderland nightmare of trying to rescue Castine and upload it to the Arhet planet. I think that as of this morning, that I have succeeded, but not before almost destroying the Arhet server altogether in the process.

Without going into a lot of detail, it seems that there were a couple of relations (a technical term in this case for a type of data object used in OSM GIS software) that were apparently so badly constructed that they broke the server’s database. Since I had to do a kind of trial-and-error search to finally identify these objects, it took a very long time. I’d upload some subset of the full Castine dataset, and watch to see if the database crashed or not. If it didn’t, fine, I’d try another set of data. If it crashed, I’d have to go back to the last backup of the server, restore it, and try again. I think I did a backup-restore cycle maybe 12 or 14 times over the last week on the Arhet server. It was painful, and tedious, and immensely frustrating.

The crash-provoking objects in question are puzzling. I still don’t understand why they crash the database. And given my difficulties in identifying them (and surviving them – see below), I probably won’t spend time, any time soon, trying to figure them out. They are “Giant Chessboards” – three of them. Interestingly, Castine also has other “Giant Chessboards” (e.g. here) that do not cause any kind of data problem. They are apparently implemented differently, in their details.

The problem was compounded yesterday, when, much to my shocked dismay, the server-level backup-restore functionality offered by my hosting provider, Linode – that I’d been so repeatedly abusing – suddenly and inexplicably failed to work.

So for a day (yesterday) the world was Arhetless. The server was down. I was in a panic because it seemed I’d have to fully rebuild the server from scratch. And it was only pure luck that I even had a copy of the map data, because I was still running a kludgey render engine (map drawing process) for Arhet on a different machine.

I wrangled with tech support at Linode, and they finally held my hand (or was it that they held my server’s hand?) through a successful if stressful restoration of the server’s image.

Let’s just say, these days Castine now has a quite colorful meta-history.

I reached out to the creator of Castine, sending an email to the address on record at OGF, announcing its restoration in Arhet. I would absolutely welcome and be pleased if that person would come back and take up work on the country, again – they won’t be constrained by the rules and regulations on Ogieff. Unfortunately I haven’t heard back. I speculate that there might be some bitterness about the whole business of having been first praised and then banned, back a few years ago.

The link to Castine-in-Arhet is here:

https://arhet.rent-a-planet.com/relation/10996

Please feel free to explore. I decided not to bother with adding extensive screenshots for this blog post – the point of having the Castine map hosted on the server is that you can explore easily directly on the website.

Happy mapping.

Music to view geofiction to: Dawg Yawp, “Lost At Sea.”

The State of the OGF Topo Layer

This will be a fairly long write-up, so depending on your interests, you can skip sections that aren’t useful to you.

First I will write about the OGF topo layer, and give a tour of its current status, with some discussion of how it works.

Then in subsequent sections I will try to discuss how to create or edit your own contributions to the OGF Topo layer.

Part I – Introducing The OGF Topo Layer

The term “topo” (short for topographic) refers to a style of map that shows the physical elevation of land, either through shading or drawing contour lines or both. There are many topo renders available for the OpenStreetMap platform. My own favorite for daily use is the Cycle Map layer on OSM – not for any specific reason except that it’s easy to get to – it’s included right there in the layer selector for the OSM site. The best known, however, and one of the oldest, is the separately built and maintained OpenTopoMap. The code for OpenTopoMap is freely available on github.

All the various implementations of topo style for OpenStreetMap rely on the easy availability of free data from the United States’ NASA. This agency did a planet-wide survey of the elevation of all the land in the world on an arc-second (approx. 30-meter) grid, using radar, and published that data, online, for anyone to use. This data is called “SRTM”. Developers implementing OpenStreetMap’s various topo layers are able to download and use this SRTM data to use as a raw source of the detailed topographic information needed to draw the map. So, in the case of OpenTopoMap, the SRTM data is loaded to the render machine, processed in various ways, and turned into the hillshading, contour lines, and map-elevation color shading that we all know and love.

For the real world, getting topo data is matter of launching a spacecraft or satellite with a little radar range-finder on it, and flying over the surface of the earth finding how high each part of the earth is relative to some baseline (i.e. mean high sea level). Easy! But OpenGeofiction is on a fictional planet. To repurprose Virginia Woolf’s famous line: There’s no there, there.

Instead we have to sit down and make up each square of elevation information – there’s got to be a 30-meter square grid for our entire imaginary planet, and we have to invent the elevation for each and every square! And just randomizing this isn’t going to work – real world topographic data betrays the evolved geophysical patterns of our planet, influenced by everything from plate tectonics to vulcanism to erosion to human landscaping. The data is going to have to either be “drawn” by creative human hands, or “drawn” by well-trained machine-learning AI’s. Both are possible in principle, but then we have the data problem.

The data problem is this: the NASA SRTM format was developed for storing real world data. But no one ever thought: we need a nice, user-friendly, SRTM file editor! Why? Because no one needs to edit real-world data. It came from a super-precise scientific gadget, who’s going to need to edit that? They could only mess it up.

So there simply does not exist any easy way to get the “imaginary” SRTM files that we need for our topo layer. Some geofiction genius is going to need to write a converter.

Luckily, OpenGeofiction has such a genius. The original creator of OpenGeofiction, Thilo, wrote a little program that turns a standard OSM XML format file (.osm) that has been filled with hand-drawn contour lines (tagged contour=elevation / ele=nnn) into a pseudo-SRTM format file (.hgt). Then we just need to install the standard OpenTopoMap software on our render server, copy the pseudo-SRTM files across, and voila, topo for OGF!

Still, we have a whole planet to cover. At this point in time (the time of writing this article, early 2022) the overall coverage for OGF’s topo layer is tiny, and of fairly low quality, on average, so far.

One of the main reasons I started my “OGF Mirror” project, last year, was because Thilo had decided he could no longer invest the time and energy required to support the topo layer for OpenGeofiction. I had a strong desire to see the topo layer continue to be supported and expand. Eventually that “OGF Mirror” became the migration project and I and user wangi and some other admin took over running OGF from Thilo, with his blessing. So it was my interest in topo that led to my taking on all this work for OGF, and therefore it remains of a focus of my efforts, moving forward.

Part II – The Current OGF Topo Zones

OpenGeofiction currently has 14 actively maintained topo zones. Here they are, in alphabetical order.

zone-alved

This zone has been created by mapper SwissCrusader. It’s a first draft, and clearly needs a lot of cleanup work, but it’s an excellent and ambitious start.

zone-gobras

This zone actually is two smaller areas that have been combined because of their close proximity. The older is the topo work for Khaiwoon, on the eastern tip of the Gobrassanya peninsula. The territory is one of the oldest on OGF. It was developed by user isleño with a lot of additional mapping by user histor. The contours were drawn as a collaboration between thilo and isleño. The other is the background topo for Gobras City, to the northwest. Gobras City is one of the most successful wholly public and collaborative projects on OGF, created by a very large number of different mappers over almost a decade, now. User Ūdilugbulgidħū volunteered to draw contours for the city, and in my opinion, they are quite excellent. Obviously, since the contours were drawn after much of the city, and since there is work in the city drawn without a strong awareness of the contours, there are areas where the human development (roads, etc) doesn’t really properly “match” the contours as drawn.

zone-gwynian

This zone is also actually two adjacent areas combined because of proximity. The older part, to the west, is my own work (user Luciano), drawn for my territory Ardisphere, in 2014-2016. I think it has well-done areas, but it’s quite noticeably incomplete, full of lacunae and missing expanses and weird glitches where the contour work is too sparse to properly cover the map. The newer part, currently in development by user Martinawa, is the largest single expanse of topo work on OGF, and still in early stages across most of the territory, but it’s got a lot of promise.

zone-hesperis

This is a quite small but complete set of contours for an island territory drawn by mapper Ūdilugbulgidħū.

zone-iscu

This zone includes several islands in a chain called Iscu, drawn by mapper Lithium-Ion. The contours are quite detailed and excellent.

zone-izaland

This zone includes a few of the southern islands of a the nation called Izaland, by the user Izaland Terramorphing Committee. I’m looking forward to seeing more of Izaland getting contours.

zone-january

I (user Luciano) drew these islands as part of a collaborative effort between myself and mapper deltanz. I later turned over the islands to deltanz. Personally I think they are my “most complete” set of contours among those I have done.

zone-kofuku

This is a new effort by mapper Huachachi. Kofuku is an immense territory and so far these are just a few islands off that country’s north coast. They are excellently done, and I’m looking forward to seeing more of Kofuku with contours.

zone-kojo

I (user Luciano) did a small spot of contours in collaboration with user Leowezy some years ago, in the southernmost spot of this zone, at the Joenji Peninsula. Leowezy has subsequently been gradually adding other areas of coverage for his territory, Kojo. Hopefully will come in the future.

zone-makaska

This zone actually overlaps not just the territory of Makaska, but also the one to the south, called Ooayatais. I (user Luciano) own both of these territories. The northern territory, Makaska, has a very complete set of contours and is work I’m quite proud of. The southern territory, Ooayatais is much lower quality and, for now, also quite incomplete. It’s my “current project” with contour mapping, however, and hopefully sometime later this year I’ll upload a major update.

zone-roantra

This is the oldest and, arguably, best of all the topo zones on OGF. User thilo, the original creator of OpenGeofiction, uploaded this topo zone when he first created the OGF server, for his territory Roantra. It is the archetype of OSM-style topo work.

zone-ruanerara

This is a small project done by user Toadwart for his outlying (overseas) territory of his country, KA076 (apparently this mapper recently decided to change the name of the country, and the new name hasn’t been decided on). Anyway, the contours are clearly still in development but I’m looking forward to seeing more progress.

zone-tapira

This zone, a small island, was created by mapper Ūdilugbulgidħū, but he subsequently gave up the territory. It’s someone’s chance to take on an already-existing topo project.

zone-tarrases

This is another of my own (user Luciano) territories. The Duchy of Tárrases is the one case where I quite successfully imposed a set of contours on an already-created map. I’m quite happy with the result. The surrounding islands used to be owned by me, too, but for now they are in a bit of limbo. For that reason I haven’t bothered to repair the mess of half-finished contours on those surrounding islands.

Part III – Create A New Topo Zone

There is quite a bit of work involved in creating a new topo zone for the ogf-topo rendering process. Multiple new working and data directories must be created, code must be added to xml layer specifications and data-processing scripts, and the proper raw contour templates need to be created and populated with initial draft contour data.

Therefore if you want to create a new topo zone for your OGF territory, you need to contact me and we can discuss, so I can get things set up and I can create the templates you’ll need to draw your contours (see next section). There is nothing wrong with diving in and drawing contours for yourself before contacting me, but bear in mind you’ll probably end up doing a lot of cutting and pasting and manipulation using the JOSM editor once I’ve created those templates for you, because I will no longer accept contour data drawn in files other than the pre-created templates.

If you’re trying to decide whether or not you want to create a topo zone, one good way to get a feel for what’s involved is to download one of the osm-squares from the server (see below) and just go ahead and open that up in JOSM and try to get a feel for how it works and what’s involved. Start with something small, complete and well-done, like the January Islands or Hesperis (see above).

Part IV – Edit or Improve An Existing Topo Zone

In the past, I’ve allowed users to simply submit new, hand-drawn “contour files” which I then clean up and process using the tools at hand. However, I have found that the first step – cleaning up the submitted files – can be quite tedious and time-consuming, and further, I often end up repeating work as newly submitted files have the same problems as previous iterations.

I would like to move toward a more reliable system of iterative improvement. To do this, if there is an already-existing topo zone and you want to add to it or improve it, you must download the existing osm-squares (contour files) from the ogfsrtm server, edit those, and send the edited copies of those files back to me. Do not try to bypass this process – I have special tags embedded in those contour files as found on the server, and I’ll know if you haven’t used those as a basis for your edits.

You can find downloadable individual squares for most topo zones at this location on the ogfsrtm server: https://ogfsrtm.rent-a-planet.com/osm-squares/

If a given square is missing from the server, contact me and I’ll create an initial template for that square and post it there, that you can then edit. Don’t try to create squares not found on the server.

Part V – An Alternative Pathway To Maintaining Your Own Topo Zone

If all this business of drawing vast numbers of contour lines in JOSM is not your style, there is, in fact, an alternative. You can directly submit NASA-SRTM format compliant .hgt files. This bypasses the whole contour conversion process, and would be quite a bit easier for me to do.

That said, so far, no one has done this successfully. There have been challenges to finding a program that produces .hgt files in the exact NASA-SRTM format expected by the OpenTopoMap rendering software. If, as a community, we find such a tool in the future, that may make a lot of what’s written above obsolete and we’ll have a better way to do things, moving forward.

Regardless, if you want to try doing it this way, you still need to contact me in advance, so I can set up the new zone on the render server – that aspect won’t change. Then you’d send me the pseudo-SRTM files (.hgt) so I can post them on the ogfsrtm server in the proper directory, since that’s where the render server expects to find them.

Music to draw contours to: Haddaway, “What is love?”

Real world geofiction

I’m going to make a rare reference to the “real world”.

At the geopolitical level, I think Putin and the Russian military have miscalculated.

Despite this (or preliminary to this), I should go on record that I actually agree with their “logic” on one key point: Ukraine, historically, is a part of Russia (or, depending on the point in history and the particular patch of land, Poland). Which is to say, Russian revanchist fantasies have some foundation in historical fact. The separate Ukrainian SSR was only carved off of Russia by Lenin in the 1920’s, and the Ukrainian national identity was essentially an artifice wrought by the half-hearted multiculturalist tendencies of the Soviet experiment. As Lenin said (hypocrisy alert), “The proletariat must demand the right of political secession for the colonies and for the nations that ‘its own’ nation oppresses. Unless it does this, proletarian internationalism will remain a meaningless phrase; mutual confidence and class solidarity between the workers of the oppressing and oppressed nations will be impossible.”

This Russian mistake, however, will be their undoing. If Ukraine lacked a “founding myth” and identity before now, Russia’s invasion is giving them one. From now on and far into the future, this Russian invasion of Ukraine will be the kind of foundational myth for Ukrainians that they never had before – and that will happen regardless of whether they win or lose the current war. If they win, then it will be a myth on the same level as George Washington and the American Revolution. If they lose, they become guerilla partisans like the Palestinians on the West Bank, and forge a distinct Ukrainian identity on that basis.

There is no scenario under which Ukraine fails to become a truly distinct nation in the geopolitical sense, as a direct consequence of Russia’s actions. And personally, I think that’s something that had been in doubt, until now. Putin’s “real-world geofiction” is not going to alter the map in the way that he hopes.

Here is the Ukrainian Highway Signs Agency, contributing to the information war:

The sign has been altered to say, loosely, “Fuck Off / Fuck Off More / Fuck Off Back to Russia”.

Music to forge nations by: Talking Heads, “Cities.”

Documentation of layers

I have been trying to develop some user-friendly documentation for the map layers on the main opengeofiction.net website. I have written quite a bit over the last 24 hours, and posted it to two articles on the OpenGeofiction wiki.

OGF Carto layer

OGF Topo layer

For those who might think I’m just a layabout and don’t do much, here is evidence to the contrary.

Music to write documentation to: Feduk & Allj – Розовое вино

On render “ghosts”

Thank you to everyone on the Opengeofiction server.

We had unexpected longer downtime last night and this morning.

The render server had some problem (not really clear what went wrong), and needed to be restored from backup (not part of the plan), and then the ogf-carto render was reset (which had been the plan).

Just want to say thank you to the community for your patience.

Those who have been struggling with “render ghosts” – I think they are all fixed now. You can see one really good example here:

For comparison, the ogf-topo render still has the ghosts (yes, topo needs maintenance work, I know!):

UPDATE: by the way, Happy mapping!

Music to fix servers by: Cake, “Long Line Of Cars.”

Advanced Conlanging…

“Conlanging” is the accepted name for the hobby of inventing new languages. I have been a conlanger since around age 7 – I remember inventing a language for my stuffed animals – specifically, a tribe of stuffed raccoons – at around that age, and though I don’t have my notes, I remember it being fairly sophisticated for something created at that age.

Like many conlangers, it’s only ever been a kind of side hobby, for me – though it dovetails nicely with another hobby I used to have as a child and that I resurrected in my post-cancer years and that has actually become a major avocation: geofiction (which is the whole point of the blog and website!). And of course, like many conlangers, I was, as a young adult, drawn to linguistics, where it eventually became one of my undergrad majors at the Univ of Minnesota. I don’t regret that, at all.

Anyway, this post is about conlanging, not geofiction. For some years now, there have existed some interesting websites and computer applications for inventing languages and storing the data. But I found a site yesterday that takes it to a new level. The site is called vulgarlang.com. I actually rather dislike the name, but I think it’s probably good marketing. Anyway, the site is created by people who clearly are quite knowledgeable on matters linguistic – to a level I’ve never seen before. I went ahead and paid the $25 “lifetime” access – we’ll see how that pans out, as I’ve seen many a website offering those terms that lasts 5-10 years before disappearing or radically altering its business model such that the guarantee doesn’t eventuate. But anyway, how could I resist. Let there be more conlanging, then – at a higher level of quality than ever before.

As an incidental, I haven’t posted much of my conlanging work online, at all, but a very incomplete exemplar can be found in this article about the Mahhalian language, which I created about 6 years ago originally.

On Mass Deletions in OGF

This morning, I posted this in response to a query on the User Diaries at OpenGeofiction:

General advice for all users:

When deleting large numbers of objects, please be careful. This is not a use-case that the OSM software is designed to handle (think about it – mass deletions are NOT common on OSM). Divide up your deletions to cover small numbers of objects (<1000) and small areas (so if something goes wrong you don’t mess up large areas).

I called attention to the post in the discord channel (“OGFC”), and so I decided to rant onward and provide some additional clarification and thoughts. I’ve distilled those, without much editing, below.

To the above, I would add that in my own practice, I have another step I take with respect to deletions. I almost never delete relations. You’ll notice in JOSM, it actually tries to warn you that “this is rarely necessary” when you try to delete a relation, and I think that’s a solid point. What you can do with relations you don’t need any more is repurpose them. That means you delete the members of the relation, and all of its tags, and then use it the next time you need a new relation for something.

user: What would be the potential harm of deleting relations?

Well I suspect that sometimes, a deleted relation can end up as a “ghost” object, or a “stranded” object, in the render database. Meaning it’s truly been deleted from the API database (the main rails port) but that deletion fails to transfer correctly to the render database (which stores things in a different format, and doesn’t actually have “relations” at all, but rather transforms a relation them into a special type of “closed way” I think)

By repurposing a relation instead of deleting it, you guarantee the render database will have to reclassify the linked object(s).

If you just delete it, the render database might not handle the disappeared relation correctly, and just leave the no-longer-linked objects lying around.

This is just speculation, because I don’t really understand it well. But just based on observed behavior

In general, I think that the OSM software is not designed to handle mass deletions. That’s the key point.

Because in OSM, who goes around deleting large number of objects? The real world doesn’t work that way.

You re-tag things, yes. You might move things slightly, adjusting position to improve accuracy. But things in OSM don’t “disappear”

Or if they do, they do so in very, very small sets. A building gets knocked down. A road gets torn out.

One object at a time.

So it seems very plausible to me that the OSM software actually hasn’t been designed to handle mass deletions, and hasn’t been tested for integrity in dealing with mass deletions.

For years now, I’ve been telling people, when you decide to rearrange your territory (and what geofictician doesn’t sometimes decide to rearrange their territory?!) … it’s better to move things than to delete/re-create things.

This prevents “ghosts” in the render.

In the long run, I suspect that we’ll have to just “reboot” the render now and then. But I’m not going to do it very often. (I mean “reboot” as in delete and re-create from zero, not “reboot” as in turn the machine off and on again – that happens every night).

I’d welcome comments on this rambling, vaguely rant-like discourse, for those who are knowledgeable about how the OSM apidb-to-renderdb pipeline works.

Music to make deletions to: 10cm, “오늘밤에.”

New server, same old city, but less old than before

Having worked furiously, all summer, in taking over the hosting for Opengeofiction, I have now finally reached a point where I feel like investing some time in the creative side – and actually work on the map again for a while.

I have added quite a bit of new area to the city of Ohunkagan. I’ve been saying that, within my “historical approach,” that I’ve brought it up to 1920, but I think there’s more I want to do before I call it officially caught up to 1920. So let’s call it 1917 or so. Also, some of the surrounding communities are still back before 1900 – e.g. Prairie Forge, Iyotanhaha, Riverton. They need to catch up, too. So here’s the new “work-in-progress” gif.

Here is the current snapshot.

[Technical note: screenshot taken at this URL (for future screenshots to match).] Here’s the wider area snapshot.

[Technical note: screenshot taken at this URL (for future screenshots to match).]

Music to map on a new server by: Lianne La Havas, “Forget.”

OGF Live

The OGFMirror became opengeofiction.net, over the weekend.

This is huge news.

So far the site is working okay. Not great, just okay. There are some issues.

The ActionController::InvalidAuthenticityToken is frequent and painful, for users.
Possibly related to the above – users keep getting forcibly logged out, over and and over.
The render is still playing catchup on the whole world map, and seems to be lagging around 2 hours for high zooms.
Incoming email is completely broken – possibly due to errors in the DNS tables at the opengeofiction.net registrar (and we’re waiting on the old host to fix this)
Outgoing email is problematic for some substantial portion of users due to over-aggressive anti-spam efforts by several major email providers, including Apple (icloud) and Microsoft (hotmail, outlook, live). I’m not even sure how to begin fixing this. I’ve implemented DKIM, but this also relies on fixing DNS errors which are not currently being fixed, and that might help. I’ve looked into a blacklisting of my email server by spamhaus.org and discovered it is due to my email server sharing the same IP range with a Nigerian Prince or somesuch in the server farm where it lives.

Work continues. Meanwhile, to all the users of OGF: “Happy mapping!”

Music to map happily to: Cake, “I Will Survive.”

OGFMirror

I’m super bad about posting to this blog. That’s partly because I feel a strong desire to report some actual, positive progress, which I haven’t felt enabled to do.

I have been very busy with HRATE technicalities. I am building – very, very slowly – a “mirror” for opengeofiction.net (OGF). I think if this is successful, then the owner of that site, who has expressed interest in “letting go” of having to continue to maintain it, will allow the mirror to take over for the site and a transition to a new hosting environment will be complete.

Someday, I intend to write up, in elaborate, technical detail, this process of setting up a mirror. But in broad outlines, here is what it involves (has involved, will involve).

Build a new Ubuntu 20.04 LTS server. This leads to lots of incompatibilities farther down the line, because the existing OGF server is an older version. Install the basics – apache, postgresql, etc.
Install an OSM rails port on the server.
Migrate the OGF data to this server. This was very, very hard – because the OGF data (in either .osm.pbf format, or in pg_dump format) proved to contain inconsistencies (data corruption). Some missing current nodes and ways had to be restored manually (text-editing .osm = .xml files). This ended up a 2-weeks-long process.
Set up incoming replication from the source apidb (OGF) to the new mirror (currently being called ogfdev).
Set up outgoing replication for the new ogfdev instance (to drive render, overpass, etc)
Set up a new primary render. This had some sub-parts.
- coastlines. This proved very difficult, because as far as I can figure out, the osmcoastline tool used to create the coastline shapefiles is broken on Ubuntu 20.04. An older version must be used. My current workaround: I’m actually running coastlines on an older server. I import a coastline-containing pbf file to the older server, run the osmcoastline tool, and post the shapefiles for consumption on the render server.
- I made a decision to run the renders on a different server than the apidb. I think this might involve a bit more expense, short term, but it makes the whole set of processes more scalable, long term. My experience with Arhet is that the render requires scaling sooner / more frequently than the apidb, as the user base grows. Installing the render software (mod_tile and “renderd”) proved difficult. It turns out that there are some lacunae and downright incorrect steps in the documented installation sequences on github.
- Set up incoming replication from the ogfdev database to the render database.
- There are substantial differences in recent versions of the openstreetmap-carto style – specifically, the special shapefiles are no longer stored in as datafiles in data folder in the render directory. Instead, the shapefiles are loaded to the database. Because non-standard shapefiles are used, this means rewriting the load procedures (python scripts) – the standard approach is to just grab the files for “Earth” (because who would run osm for some other planet?!). So that file-grabbing is hard-coded in the procedure.
Set up a new topo render. The topo render was shut down on OGF, so this will be the only working version. Unfortunately, I ran into a similar problem with some of the topo pre-processing as I ran into with osmcoastline, above. I suspect for the same reason – something in one of the dependencies they both have. So the topo pre-processing (turning the .hgt files into a contour database) is also being run on a separate, Ubuntu 18.04 server (just like the coastlines).
Set up appropriate changes and customizations for the front-facing rails port (osm website). This involves importing user data (done) but also user diaries (not done). These require ad hoc SQL coding that give me flashbacks to my job as DBA in the 2000’s. Another unfinished piece – internationalization. The current ogfmirror website looks okay, but only in English. Switch to another language, and it all reverts to OSM boilerplate. Why is internationalization done so badly on production software of this kind? I see no easy solution except manually editing each language’s .yml file in turn (OSM has a 100+ languages). Or building my own damn application to achieve that result.
Set up overpass and overpass-turbo. Overpass installs relatively painlessly, but I’m having trouble getting incoming replication to work correctly. overpass-turbo was quite difficult – the current version on github is flat-out broken, and so an older version (commit) must be compiled and installed. Further, the compilation and configuration process overwrites some of the parameters files, so the parameters files have to be modified after running the first steps of configuration, but before the last part. This is the step I am on right now.
Set up nominatim? – nice to have, but not urgent. Anyway nominatim doesn’t work on the existing OGF website
Implement some of the custom tools that are available on the OGF website: the “scale helper,” the “coastline helper,”…
What else? This is a work in progress…

So I’ve been busy. Here is a link to the site. Bear in mind, if you are reading this in the future, the link may not show you what I’m currently writing about, but rather some future iteration of it.

https://ogfmirror.com

I’m still working on some of those last steps. Open to hearing what else needs to be done.

Music to hack to: K-os, “Hallelujah.”