How to blur a jagged border in PS?

hi all,

this is related to this and that topic.


I want to blur the edge between my footage and my mapped-in faces. The faces are marked in the alpha channel and come with hard edges to not interfere with my baked-in uv maps.

What’s the best approach to blur those edges after compositing my faces?
I am thinking about derivatives or a kernel to find the edge, then dilate to increase the area to both sides of the border and then a fast blur.

Do you concur, dear Shader Cracks?

Am I trying to code FXAA?

Something like this maybe :