Downloaded a sample DEM tile from the NASA SRTM dataset, scaled it down to 1024x1024, and used it to control the height of a 1024x1024 grid of vertices. I am kind of new at this, but 2,093,050 is definitely a new personal record for number of triangles put on the screen… by a LOT. The normal generation needs a bit of work, but it is not too far off. Doesn’t make sense to waste too much time getting it nailed down for a proof of concept.
Before I start messing around with LOD algorithms, I need to get my hands on the entire dataset and figure out a pipeline to pre-process it in QGIS.
Yeah, I believe the way this is done is by creating holes in the heightmap (for caves) and positioning conventional 3D assets manually to match up with them (for caves or simple overhangs in general). In theory you should be able to encode holes right in the heightmap texture by encoding them as NaN, INT_MAX, or something along those lines, but a vertex shader is only capable of moving vertices, not deleting them. Like generating geometry, discarding geometry apparently requires the use of a tessellation shader, though some hacks can potentially be achieved in the vertex shader - e.g. swapping pairs of vertices to reverse the winding order of their triangles and trigger back-face culling. The trick is figuring out how to do that while avoiding conditional branches, which apparently ruin performance. Fortunately this isn’t my problem.