Hlsl dx11 back-face culling problem with diffuse shader: cull in ps shader, vs shader or in vvvv?



Hi guys,

Just starting to learn DX11 HLSL shader programming and I am trying to build fairly simple DIFFUSE shader from rastertek code and it seems to work but I am having issues with back-face culling. (It’s part of a spike ball shader which takes every 4 vertices and moves along normal, creating spike ball effect)
Anyway I know there is the rasterizer node for back-face culling, but I just can’t work it out. And also I am wondering shouldn’t I do the back-face culling in my pixel or vertex shader? I feel like in dx11 back-face culling is done in shader? or I am wrong? Sorry new to shader programming…
Thankx for the help guys here is my code and picture:

Texture2D texture2d <string uiname="Texture";>;

SamplerState linearSampler : IMMUTABLE
    Filter = MIN_MAG_MIP_LINEAR;
    AddressU = Clamp;
    AddressV = Clamp;
cbuffer cbPerDraw : register( b0 )
	float3 DiffuseLightDirection = float3(1, 0, 0);
	float DiffuseIntensity = 1.0;
	float SpikeIntensity = 0.5f;

cbuffer cbPerObj : register( b1 )
	float4x4 tW : WORLD;
	float4 cDif <bool color=true;String uiname="Color";> = { 1.0f,1.0f,1.0f,1.0f };
	float4 cSha <bool color=true;String uiname="Shadow";> = { 1.0f,1.0f,1.0f,1.0f };

struct VS_IN
	float4 PosO : POSITION;
	float4 TexCd : TEXCOORD0;
	float3 normal : NORMAL;
	uint iv : SV_VertexID;

struct vs2ps
    float4 PosWVP: SV_POSITION;
    float4 TexCd: TEXCOORD0;
    float3 normal : NORMAL;

vs2ps VS(VS_IN input)
        vs2ps output;
	input.PosO.w = 1.0f;
	if(input.iv%4==0 ){

        output.PosWVP  = mul(input.PosO,mul(tW,tVP));
	output.TexCd = input.TexCd;
	output.normal = mul(input.normal, (float3x3)wIt);
        output.normal = normalize(output.normal);
	return output;

float4 PS(vs2ps In): SV_Target
	float4 textCol = texture2d.Sample(linearSampler,In.TexCd.xy);
	float3 lightDir = DiffuseLightDirection;
        float lightIntensity = saturate(dot(In.normal, lightDir));	
	float4 color = saturate(cDif * lightIntensity * DiffuseIntensity);	
	color = color * textCol + cSha;
        return color;

technique11 Constant
	pass P0
		SetVertexShader( CompileShader( vs_4_0, VS() ) );
		SetPixelShader( CompileShader( ps_4_0, PS() ) );


Did check your Depth Buffer Mode on the renderer? Should be set to standard.


u can use raasterizer with no cull


THankx guys, @antokhio yeah I have been tinkering with the rasterizer node but as @everyoneishappy pointed, I simply had not seen the depth buffer mode drop down in herr inspektor. That’s IT! I knew I was missing something simple and silly. amazing thankx a lot guys!


Just as informative:

  • Default Rasterizer State is back cull indeed.
  • Default culling is performed at Rasterizer stage (not in shader)
  • You can’t cull in VS, since your vertices might be shared by different primitives, so triangle information is not available yet.
  • You can back/front cull in PS, but that’s not a good idea unless you have a very good reason to do so (you lose early depth rejection)
  • You can back/front cull in GS, but unless you want to do custom culling, let rasterizer to it for you it will be faster.