DX11 Layout Error

Hi Everyone,

To learn how to write DX11 shaders I’ve been rewriting some of my favourites.

I’m currently trying to rewrite Softshadow for Point Light by Microdee in The hope I can send the distance cubemap straight into the shadow shader without having to Render it out.

If I can get that to work it would be great. The problem so far is that while my Shader compiles it tells me that my layout is invalid.

Could anyone explain why?

Muchos Gracias,


TextureCube sMap:PREVIOUS;
SamplerState s0 : IMMUTABLE{Filter = MIN_MAG_MIP_LINEAR;AddressU=CLAMP;AddressV=CLAMP;};

Texture2D Tex0<string uiname="Texture";>;//I may nnot need this!!!
SamplerState s1:IMMUTABLE{Filter=MIN_MAG_MIP_LINEAR;AddressU=CLAMP;AddressV=CLAMP;};

float4x4 tW: WORLD;        //the models world matrix
float4x4 tV: VIEW;         //view matrix as set via Renderer (EX9)
float4x4 tP: PROJECTION;   //projection matrix as set via Renderer (EX9)

float3 LightPos <string uiname="Light Position";> = 0;
float bias <string uiname="Shadow Bias";> = 0.02;
float contrast <string uiname="Contrast";> = 5;
int samples <string uiname="Samples";> = 3;
float softness <string uiname="Softness";> = 0;
float blurring <string uiname="Softness Distance Multiplier";> = 1;

float4x4 tTex: TEXTUREMATRIX <string uiname="Texture Transform";>;

struct vs2ps
    float4 Pos : POSITION;
    float4 TexCd : TEXCOORD0;
    float4 wPos : TEXCOORD1;

vs2ps VS_Distance(
	float4 Pos: SV_POSITION,
    float4 TexCd : TEXCOORD0)
   vs2ps Out = (vs2ps) 0;
   Out.Pos = mul(Pos, tWVP);
   Out.TexCd = mul(TexCd, tTex);
   Out.wPos = mul(Pos,tW);
   return Out;

vs2ps VS_shadows(
    float4 Pos : SV_POSITION,
    float4 TexCd : TEXCOORD0)
    //inititalize all fields of output struct with 0
    vs2ps Out = (vs2ps)0;

    //transform position
    Out.Pos = mul(Pos, tWVP);

    //transform texturecoordinates
    Out.TexCd = mul(TexCd, tTex);
	Out.wPos = mul(Pos,tW);

    return Out;
float4 PS_Distance(vs2ps In) : SV_TARGET
	float3 p1 = In.wPos.xyz;
	float3 p2 = p1 - LightPos;
    float dist = sqrt(p2.x*p2.x+p2.y*p2.y+p2.z*p2.z);
    return float4(dist.xxx, Tex0.Sample(s1,In.TexCd).a+0.);
struct outcolor
	float4 c1 : COLOR0;
	float4 c2 : COLOR1;
outcolor PS_shadows(vs2ps In): SV_TARGET
    //In.TexCd = In.TexCd / In.TexCd.w; // for perpective texture projections (e.g. shadow maps) ps_2_0
	outcolor col = (outcolor) 0;
	float3 pp1 = In.wPos.xyz;
	float3 pp2 = pp1 - LightPos;
	float pdist1 = sMap.Sample(s0,normalize(pp2)).x;
    float pdist2 = sqrt(pp2.x*pp2.x+pp2.y*pp2.y+pp2.z*pp2.z);
	float dd = pdist2 - pdist1;
	col.c2.xyz = dd;
	float3 offs = -softness;
	for(int i=0; i<samples; i++) {
		offs.y = -softness;
		for(int j=0; j<samples; j++) {
			offs.x = -softness;
			for(int k=0; k<samples; k++) {
				float3 p1 = In.wPos.xyz+offs*(pdist2*blurring);
				float3 p2 = p1 - LightPos;
				float dist1 = sMap.Sample(s0, normalize(p2)).x;
    			float dist2 = sqrt(p2.x*p2.x+p2.y*p2.y+p2.z*p2.z);
				//float tmp = ((dist2-bias) > dist1);
				float tmp = min(max((dist2-dist1)*contrast-bias,0),1);
				col.c1.xyz -= tmp/pow(samples,3);
				offs.x += (softness/samples)*2;
			offs.y += (softness/samples)*2;
		offs.z += (softness/samples)*2;
	//col.c1.xyz = ((dist2-bias) > dist1) ? 0 : 1;
	float4 t = Tex0.Sample(s1,In.TexCd);
	col.c1.a = t.a + 0.5;
	col.c2.a = t.a + 0.5;
    return col;
technique11 _Shadow{
	pass P0{ 
	pass P1{

SoftShadows_1Pass.fx (3.4 kB)

i think that depends on the mesh you feed in the shader. maybe it’s missing TexCoords?

To clarify, the ‘Layout Message’ pin come back with
Geometry is missing semantics: SV_POSITION

Thanks, I figured it out.

The POSITION should have been an input for the VSshaders and the SV_POSITION an output, obviously. You live and you learn.

There’s one more question that I can’t find an answer to anywhere. Is it possible to generate a Cubemap as a pass without sending it through a Cubemap Renderer?

“There’s one more question that I can’t find an answer to anywhere. Is it possible to generate a Cubemap as a pass without sending it through a Cubemap Renderer?”
nop, that what cubemaprender for

Thanks Antokhio

Cube texture I mean ;-)
I’m beginning to think not. I haven’t found any online resources that show how I can annotate the type of output to pass.

Something like this (see string Type further down)

max uses wraped stuff, it’s not native hlsl, so prolly it does cubemaprender under the hood or something like that. That’s prolly can be implemented in vvvv also, but it’s still gonna create render somewhere from the line in shader code.

shader input should be :


and vs2ps should be:

struct vs2ps
float4 Pos : SV_POSITION;
float4 TexCd : TEXCOORD0;
float4 wPos : TEXCOORD1;

Yep, that fixed it.



(Follow up question another thread)