Depth-masking via stencil buffer?

Hi everyone!

I cant really wrap my head around this problem with so called depth-masking using shaders. My intention basically is to use it for whats called portal rendering, see image below:

The idea is to have an arbitrary mesh serving as a window through which one can see objects from another scene, as seen from current camera viewpoint, while the rest of the original scene renders normally around this hole.
I have been using this successfully in Unity 3D, whre it can be achieved like this - you make 2 shaders, one for the objects being viewed through the portal:

Pass{ 
  Stencil {
    Ref 1
    Comp Equal
...

and another fot the portal itself:

ZWrite Off
ColorMask 0
     
 Pass {
     Stencil {
         Ref 1
         Comp always
         Pass replace
     }
 }

(that is Unity 3d kind of hybrid HLSL/ShaderLab shader syntax i guess)

I can avchieve the effect of seeing into another space in DX9 even without writing a shader, just by using ZWriteEnable node with its func set to ‘Greater’ (connected to the shader dodes of the objects behind the hole), but obviously the objects revealed through the mask in this way then cannot sort their Z order with respect to each other correctly anymore (they actually seem to be rendered in reversein this way).

So I tried to port the above code to build my own DX9 shader, like this…

the objects:

technique TPhongPoint
{
    pass P0
    {
        //Wrap0 = U;  // useful when mesh is round like a sphere
        VertexShader = compile vs_3_0 VS();
        PixelShader = compile ps_3_0 PS();
    	//ZWriteEnable = false;
    	//ZFunc = Equal;
    	StencilEnable = true;
    	StencilRef = 1;
    	StencilFunc = Equal;
    }
}

and the mask:

technique TSimpleShader
{
    pass P0
    {
        
    	
    	//Wrap0 = U;  // useful when mesh is round like a sphere
        VertexShader = compile vs_3_0 VS();
        PixelShader  = compile ps_3_0 PS();
    	
        ColorWriteEnable = 0;
    	ZWriteEnable = false;
    	StencilEnable = true;
    	StencilRef = 1;
    	StencilFunc = Always;
    	StencilPass = Replace;
    }
}

But this neither gives the desired result, even though the code compiles sucessfully, so I suspect the stencil buffer functionality may not be supported for DX9 in 4v…?
Sooo I decided to move to DX11, after quickly checking that it supports some advanced stencil funcionality which was looking really promissing - first I tried to fiddle with the DepthStencil and DepthStencil (Advanced) nodes, with no luck, then finally when I tried to put my own DX11 shader together, I couldn’t even find a way to set the StencilRef value or its equivalent for the DepthStencilState in DX11:/
I believe there must be either another proper way to do this in DX11 which I just didn’t come around as my knowledge of shaders is pretty basic, or I am missing some essential shader concepts regarding 4v rendering…? Anyways for now I am out of ideas, I feel like I tried every possible way which makes sense to me so far.

Although I am curious if this could still be possible to do using DX9, as the project I am using it for is originally based on DX9 rendering and I dont want to mess up the two in one patch (as it is being discouraged on the forums generally), somehow I feel I should give the DX11 solution a higher priority for the sake of future development and performance optimizations.

Would anyone kindly push me in the right direction?

Cheers everyone, long live the community!:)

Well I managed to make this work in DX11 after all, using a custom DepthStencilState setup in the shader.

It really works as expected after all, it was my problem not knowing how to set the stencil ref value in DX11 - which is indeed done via a parameter in the funcion assigning the state - SetDepthStencilState(DepthStencilState state, uint refValue);
Also, the order in which the layers go to the renderer matters - the portal layer / shader needs to go first in the chain, either via group node or by feeding the objects layer into the portal layer and that one into the renderer.

So a am posting my shader code for both objects being viewed through the portal, and the portal mask itself(window), just for the case anyone might find it useful on a trip into parallel worlds while avoiding using blending-based mask:)

The objects:

  1. define stencil state

    DepthStencilState DepthMask
    {
    StencilEnable = TRUE;
    FrontFaceStencilFunc = EQUAL;
    };

  2. assign the state with Ref value as a param in the pass

    technique10 TPhongPoint
    {
    pass P0
    {
    SetDepthStencilState( DepthMask, 1 );
    SetVertexShader( CompileShader( vs_4_0, VS() ) );
    SetPixelShader( CompileShader( ps_4_0, PS() ) );
    }
    }

the window (mask):

  1. define stencil state

    DepthStencilState DepthMask
    {
    DepthEnable = FALSE;
    StencilEnable = TRUE;
    FrontFaceStencilFunc = ALWAYS;
    FrontFaceStencilPass = REPLACE;
    };

  2. assign it in the pass

    technique10 Constant
    {
    pass P0
    {
    SetDepthStencilState( DepthMask, 1 );
    SetVertexShader( CompileShader( vs_4_0, VS() ) );
    SetPixelShader( CompileShader( ps_4_0, PS() ) );
    }
    }

And voila!
{img src = "sites/default/files/screenshot1449579363.png" title = “test”}

Now I would really like to know if/how this is the hell possible in DX9:P

there is quite easier way to do that in post proc
don’t think u can write to stencil in current dx9 implementation

@antokhio:

Regarding DX9 - well thats what I was afraid of:)

But what would be the easier postprocessing solution then? Do you mean any blending or so?

hi, yea just render ur quad separate render with same camera then do the masking in pixel shader…

setting the Compare Function to GreaterEqual of ZWriteEnable (EX9.RenderState) can have a similar result, when you draw a close Quad first.

thanx guys for your feedback!

@velcrome: ok thats what I found before but then the problem of z-sorting of the individual objects in the virtual view arises, or is there another tweaking posible which I missed? I mean when I set the comp func to Greater / GreaterEqual, then the objects are rendered in front of the hole and masked by it as desired, but at the same time this breaks their relative Z-sorting huh… This is actually why i called this question as “depth-masking”, which is how it is called using this approach, just using the depth buffer but not the stencil.

@antokhio: sure that is an option but still I consider this approach more demanding speaking of resources, cause in this way i have to render all the objects without mask first (meaning drawing all not-culled faces of the mesh which can be rather complex, I hope) and then do another per pixel blending with the mask, while using the stencil I just discard the masked pixels as they are tested, am I right about this? Please correct me otherwise - I am really a newbie regarding shaders as I mentioned…
I realize the difference of the two method can be quite bearable when dealing with a simple scene… But, obviously I am doing this with several portals at a time, looking into different worlds from the a shared view, and because the installation should run in some reasonable resolution, i started to worry about this… Basically it is a kind of view-dependent projection (kind of augmented reality) running on 4 projectors assembled to cover the inner space of a 4m square box, so it needs at least some ~5000x1000px+ texture I guess. I might try bogrouping too I guess but it seems to be a overkill to use several computers for a rather ambient installation.

Anyways I already managed to remake the whole setup to use DX11 somehow, now I only need to correct the the lightning of the virtual scene sent to th eprojectors - the method of feeding the “real” camera view into the shader instead of using the renderers’ one (from PerspectiveLookAtRect) still gives some weird results:P

Thanks again for your time guys, actually even confirmation of the stencil-loss DX9 already saved me a lot of time;)