This is Richard Knol, the developer of Sailaway. I’ve been behind the scenes more recently as I have been busy working on some big changes to Sailaway. The biggest of these tasks has undoubtedly been improving the accuracy of the map data so I thought I’d give you an insight into this process in my first developer blog!
Sailaway’s Map Data
The Sailaway maps are all rendered in real time based on the 3D model of the world that is stored on the Sailaway servers. This model was made with data from NASA and a few other sources and it holds the height of the terrain or sea floor at any location in the world.
There are a few issues with that:
- The NASA data only provides the height of the land, not of the sea floor. It also contained measurement errors and it has a resolution of about 10m2, 30m2 or 240m2, depending on the area.
- Most of the underwater height maps I found only have a resolution of about 240 meters
- The underwater data I found and could use is inaccurate and incomplete with regard to coastal waters
- Not all data sources use the same corrections for the fact that the earth is not a perfect sphere
So all in all a rather depressing list of issues when you want to run a worldwide sailing simulator.
Luckily there are nautical charts available that give more accurate info about water depths and much more. But charts are 2D and Sailaway uses a 3D model. Basically, a chart tells you that there are regions colored light-blue where the water depth is 0 – 10 meters, and regions in another shade of blue where the water depth is 10 – 50 meters. But what happens in between?
Following that data strictly would turn the entire seafloor into a giant staircase. Additionally there are soundings on those maps that give the exact water depth at a certain location. But they don’t say anything about if that point is a peak with a steep slope or a single measurement on a horizontal and flat surface. So here’s what I did to improve the Sailaway 3D model with the US Electronic Charts as provided by NOAA…
Deciphering the data
First the charts were downloaded. All 1280 of them. Next they needed to be read, or rather deciphered. The format in which they come is called S-57 and it was defined by the International Hydrographic Organization decades ago. It dates from the days when computer scientists tried to write data dictionaries and press various record types into a highly efficient binary format. A time when JSON and XML were never heard of yet! So I deciphered the gibberish and extracted the data that I needed for Sailaway and pushed that into a JSON format. This was several weeks work, but in the end I had a neatly organized collection of 1280 JSON files and
1280 small overview images of the actual charts.
The US charts come in 6 levels. At level 1 there are large overview charts, building down to level 6 where a specific port is shown in high detail. I wrote a program that would process each of these maps starting at level1. It would download a section of the Sailaway 3D model, match the Sailaway depths/heights with the data of the chart and make the necessary corrections.
Unfortunately that technique didn’t work, because it would create a staircase seafloor after the depth regions of the chart are applied. And it would litter the seafloor with spikes or pits at the locations of the soundings in the chart.
So instead I used the charts to verify our own data and make gradual adjustments where needed. If the chart says the seafloor depth is between 20 and 200 meters, and Sailaway has 20.5 or 198, I leave it untouched. If the Sailaway model says 18 meters, the floor is lowered and also the pixels around it are lowered gradually to smooth the sea floor. This way I kept as much of the shape and structure of the seafloor intact as I could.
Then there was the problem of turning land into water or water into land when the coastline would not match the chart. Turning water into land without any information on what this land would look like will result in ugly terrain. There is no fix for this. Sorry.
Implementing the data
So now that the algorithm seemed to work (again a few weeks work) I could put the computer to work. Starting with Level 1, here is how the numbers look:
The Sailaway model is made up of 0.25 x 0.25 degree tiles. That means (360 x 4) x (180 x 4) = 1,036,800 tiles. The tiles have a resolution of 900 x 900 pixels. Each pixel is represented by 2 bytes. That means 900 x 900 x 2 = 1,620,000 bytes per tile. Times 1 million tiles is 1.5 TB of data, or 800 billion terrain heights.
To process 1 chart of Level 1 (typically a region of 20 x 30 degrees) took about 2 days to compute.
In the meantime, I read and imported all the buoys, lights, rocks, shipwrecks and other vital information from those charts into the Sailaway database. This resulted in 120,000 new objects all along the US coast.
So my computer was humming and after many days, when it was already processing the charts for Level 3, I discovered a stupid mistake. Imagine a tiny volcanic island somewhere. A beautiful thing in the Sailaway maps, where you can see the slopes and the crater clearly. But this tiny island is not big enough to have made it on the Level 1 overview charts. Instead it falls in the big area indicated as 200-2000 meters deep. After processing the chart, the entire Island has vanished and has been turned into 200 meter deep water. Of course when the charts of level 3 are processed the island is reinstated, but all of it’s characteristic volcanic shape is gone. It is now a flat surface a few meters above sea level.
As a result, the algorithm had to keep a copy of the original shape at hand. And every time it wanted to alter the current height of the land or sea floor it had to refer back to the old data and see if it could use any of the original height info to recreate some of the original shape of the terrain.
The algorithm was started again from scratch. And after many weeks it is now ready with correct calculations. Yay!! The result is a more accurate seafloor, and a very ugly coast. The last is because after processing the 2D charts, a pixel is either land or water. Imagine a nice coast with some rocks, some distant mountains, small pebble beaches between the rocks. The chart may have drawn a line over that coast, dividing it into land on one side and water of at least 20 meters deep on the other. The result is a coast of steep cliffs, that plunge into 20 meter deep water. The pebble beaches are gone, and because of the size of the pixels the coast looks like a horrible collection of zigzag cliffs.
To fix this, or at least make the coast look somewhat more realistic, is my challenge for the coming weeks. Wish me luck!
Ongoing may improvements
I started work on these map improvements early August. It is now the end of October and I am not done yet and this was only the US coast! The European and Australian coast will be even more challenging. But it is great fun to work on this and see the maps improve. Sailaway becomes more mature with every step we take.
I see players asking for more detail in the terrain often; for ports & marinas, and even trees and buildings. These would be awesome to have, I fully agree. But when you realize that the current work in progress coastline, made up of 10 x 10 meter pixels, already measures 1.5 TB of data. Just imagine the amount of data required when this includes the rest of the world! All that data has to be collected, processed, checked, maintained, refined and find its way to your computer somehow… don’t forget, Sailaway is a persistent online world.
The world is a really big place and I hope that the process behind how I implement this accurate data has become much clearer. Please do not expect the same level of terrain detail in Sailaway that you see in other games. They do not have to carry the weight of the world with them.
Thank you to all of you who have joined us in early access and support this project! I’m sure you can see from the process above, this is a passion project for me and I’m thrilled to see so many people sailing the seas I’ve created.