The Better Internet Dashboard is a project that has grown from a simple checker for my own benefit into a tool that’s used consistently by 1,000s of users. As this has happened, there have been multiple pieces of work taking place behind the scenes to ensure it’s both scalable and cost-effective.
Initially, the dashboard was a single page, with no input field and a fixed address displaying works and serviceability data for only a couple of providers. Now, that’s clearly not the case.
One of the very first things to be improved was the API itself. This went from a single process Python app to deployment on AWS Lambda and API Gateway, ensuring it can scale indefinitely with the added benefit of a very low operating cost.
As time has gone on, one dependency has clearly become a problem. The map itself.
I intend for this post to explain the thought processes behind the map tiles as well as provide someone with zero mapping knowledge (like myself) a good foundation to build their own static map tile library with a custom theme.
The Map Problem
This one caught me off guard, I didn’t expect the map itself to be problematic. However, Better Internet Dashboard first used Mapbox to serve map tiles. This worked well, they also have a generous free tier.
As the load picked up, I began to become extremely close to the limitation of the free tier. At this point, doubling the traffic from the free tier would cost an extra $100 a month to operate. Obviously, I don’t make this much money (in reality I make zero).
So what next? The obvious move here is over to OpenStreetMap. But this too presents issues.
OSM have several free to use tile servers. But if you take a skim through this page you can see the themes are quite substantially different and this is not something I am quite ready to give up.
Of course, if an emergency change was needed, I could pick up one of these servers and deal with it.
So if these themes don’t work for me, what do I do? Well, this is where things become unknown to me and the research and testing began.
Designing the Tiles
The main reason for even doing this was to somewhat preserve the design I had before with Mapbox, which brings us on to the first utility, Maputnik. This allows you to select elements of a map, set a style, and apply it. The result is a JSON file that can be used with the Mapbox GL style specification.
This specification appears to be widely used within the OpenMapTiles community and all the tools I needed to use, support it.
Generating the Tiles
Generating the map tiles turned out to be quite a lot more complex than I initially expected.
It’s possible to get the OpenStreetMaps dataset for specific regions (in this case, the United Kingdom) from Geofabrik. Alongside this, global coastlines are also required.
From these two datasets, TileMaker is used to
generate an .mbtiles file (this is the vector tile format, effectively an
SQLite database containing information for all the tiles) from the .osm.pbf
(the OpenStreetMap format) files that were fetched before.
The issue here is fairly obvious however, it’s not possible to use this several
gigabytes file for the tiles, since there’s no way a client can reasonably use
it.
Once the .mbtiles file has been produced,
TileServer GL can be used to host
the vector tiles.
Along with the design from Maputnik, the client can either render the tiles or
allow TileServer GL itself to render rasters (the client needs the style for
the former, the server needs the style for the latter).
Vector tiles can be served with MapBox GL, a Javascript client library that relies on WebGL for its processing. At current I use Leaflet, there’s an adaptor for using that and being backed by MapBox GL for the vector tiles layer.
Within the first days of using this, there were a couple of observations. It is far less responsive than Leaflet, it’s very jittery on mobile, and it is completely non-functional if you don’t enable WebGL. For these reasons I have started looking into generating PNG tiles like there used to be and allowing them to be served with plain Leaflet instead.
Serving the Tiles
Map servers are just HTTP servers that serve square images. It’s nothing more complex than that (or so I thought).
I knew it was possible to store and serve a vast amount of map tiles for next to zero operating cost. This is utilising a combination of Cloudflare and Backblaze B2 (there’s no bandwidth charge between the two).
Therefore, all I need is a set of PNGs and I can then push them using the S3 API.
Unfortunately, TileServer GL has no mechanism to batch generate static tiles.
Therefore, I have written a tool to read the .mbtiles data and fetch all the
required tiles through TileServer and push them up to B2.
This whole process requires a lot of computing power and therefore is running over a long period of time. However, since these tiles will not be updated frequently, this is absolutely fine.
By default, TileServer GL only produces 256 pixel tiles. I prefer to have the larger 512 pixel tiles (reducing the total number of tiles by 75%). There’s a fork I am using to do this.
Next Steps
So, as of right now, I am still generating the new tiles to serve. The vector tiles are still being served from the B2 setup.
Over the coming days, the map will be replaced with statically generated map tiles which should improve performance for all. Alongside this, I will release all the tools I have written to generate the map tilesets used for Better Internet Dashboard.
Without Mapbox, I don’t think I’d ever have gotten off the ground with Better Internet Dashboard. So although I no longer use it as I can’t justify the personal expense, I am very thankful for them and their tools!