with gridScaleXY = 1 everything looks fine.
that’s a nice approach tonfilm!
so i think there a 4 possibilities to solve the problem:
1) calculate the formula for the normals in a mathprogram like mathematica and type this into your shader. (hard)
so if you have any function describing a 2d-surface which is dependant of 2 varaibles u and v, like
F:(u,v) -> (Px, Py, Pz)
which gives back cartesian coordinates Px,Py,Pz for every (u,v) pair, then you can use a math program to calc the normals like this:
At each point on the surface you need 2 tangent vectors along the surface, which you can cross to get the normal at this point.
The trick is: the tangent vectors are just the derivates of the function F in direction u and v.
so first you vary u (and get the derivative with fixed v and changing u)
t = dF / du
and then you fix u and vary v to get the binormal
b = dF / dv
these are called partial derivates for the function F:(u,v).
both t and b are again functions in this form:
t:(u,v) -> (tx, ty, tz)
b:(u,v) -> (bx, by, bz)
when you cross the t and b vector for your current (u,v) then you get the normal…
2) tonfilms more general approach, which is only slightly less accurate and only slightly slower, but much easier to program and it should also need less shader instruction slots.
it could get a little more accurate when substracting the points half-way to the neighbours from each other like
t = F(u+du/2, v) - F(u-du/2, v)
b = F(u, v+dv/2) - F(u, v-dv/2)
and if your graphics card still has power left, it is worth to think about doing these instructions (and the cross) in the pixelshader for even smoother results.
3) mess around with ddx and ddy functions in the pixelshader. these functions are astonishing. they can somehow calculate derivatives of any variables with respect to screenpixels. i.e. what happens with a particular function or variable when calculating the pixelshader for the neighbor screenpixels.
4) do it outside of the shader which is very slow when doing this every frame. however if when you want to calculate the geometry only once it should be faster to render it when normals are already stored in the mesh.
for now. chcheers