Best practices for working with large datasets

[jokroese]

I have a website which involves visualising a large geojson file (~45MB, >100,000 points). I am trying to figure out ways to:

1. reduce the network requirements for visitors to the website
2. make the experience responsive once it loads

I am wondering if there are any best practices for working with large datasets in Maplibre GL JS?

In particular, I am wondering if there is a way to use a gzipped version of the geojson file with addSource()? My naive attempt of simply replacing the .geojson with .geojson.gz didn't work.

[jokroese]

So far, the most useful reference I have are these tips for working with large datasets in Mapbox GL JS.

I also came across the geobuf filetype (https://github.com/mapbox/geobuf) which sounded promising but two minutes after I saw this thread suggesting it wasn't as good as Mapbox claimed (#106 (comment)).

[acalcutt]

In my project at wifidb.net I have a couple million points i visualize.

For smaller lists below 50,000 points I dynamically generate geojson and maplibre is able to use that directly.

For bigger lists, like my full data exports used in my main map, i use https://github.com/mapbox/tippecanoe to pre-procese the the geojson into mbtiles. since i have so many points this is needed because it cuts down the points at higher zoom levels, which without doing the map exceeds some maximim (maximum points or objects i think). once i have a mbtles i can just use it in my tileserver and the client is able to download only what they need.

my Tippecanoe scipt to convert my geojson looks like this .. pretty basic.
https://github.com/acalcutt/wifidb-tileserver-gl/blob/master/convert/convert.sh

[jokroese]

Thanks @acalcutt! I'll check out these links tomorrow.

And great work with wifidb.net! So cool to see that many points. I've never seen a map of wireless networks like that.

[acalcutt]

Here an example of one of my dynamic pages. This is 50,000 points in geojson made on the fly. You can see it takes a bit to load, which is mostly because it is a LOT of geojson that it makes as the page loads (and I have a lot of info on each point).

This is 50,000 points of signal history from one access point
https://wifidb.net/wifidb/opt/map.php?func=exp_ap_sig&id=224383&from=0&inc=50000

The geojson is generated by an api I made.
https://github.com/acalcutt/WifiDB/blob/master/wifidb/wifidb/api/geojson.php

[acalcutt]

I also worked with someone in the past using this geojson server

https://github.com/TNOCS/geojson-tile-server

it is able to process the geojson on the server side, split it into pieces, and serve to maplibre as mvt vector tiles. which is faster because the client does not have to download all the geojson data, only the data in the tiles they need.