Toon/Cel shader, with video, code, blog

yojimbo2000 · May 2015

Code and explanation here:

https://puffinturtle.wordpress.com/2015/05/27/coding-3d-animation-5-toon-shaders/

Ignatz · May 2015

Good stuff! So drawing the back facing elements in black, slightly enlarged, creates the thick outline we need for cartooning. It seems a bit OTT to make two passes though.

I have seen different methods, but I can't remember how they did it.

Keep it up! =D>

yojimbo2000 · May 2015

Yeah, it's a shame the gl_FrontFacing variable is only available in the fragment shader. If you could access it in the vertex shader, then you could do it all in one pass. I guess there should be a way of working out whether the face is front-facing in the vertex shader though. After the normal has been multiplied by modelMatrix, would it just be a matter of checking whether z is positive? Could it really be as simple as that? Or would you need to multiply the normal by the modelViewProjection matrix?

Ignatz · May 2015

You have given me an interesting problem to play with. :-?

yojimbo2000 · May 2015

Don't tell me the answer straight away! Give me 8 hours to work on it. I'm guessing normal*modelViewProjection...

Ignatz · May 2015

@yojimbo2000 - you won't get far with that, swap them round (the matrix gets multiplied on the left)

I'll post my shader solution below, don't look if you want to

S = {
v = [[
uniform mat4 modelViewProjection;
attribute vec4 position;
attribute vec4 color;
attribute vec2 texCoord;
attribute vec3 normal;
varying lowp vec4 vColor;
varying highp vec2 vTexCoord;

vec4 v = vec4(0.0,0.0,1.0,0.0);

void main()
{
    vTexCoord = texCoord;
    vec4 n4 = vec4(normal,0.0);
    vec4 nView = normalize(modelViewProjection*n4);
    vec4 p=position;
    if (dot(v,nView)<0.0) {
        p.xyz = p.xyz*1.02; 
        vColor = vec4(0.0,0.0,0.0,1.0);
    }
    else vColor=color;
    gl_Position = modelViewProjection * p;
}
]],
f = [[
precision highp float;
uniform lowp sampler2D texture;
varying lowp vec4 vColor;
varying highp vec2 vTexCoord;

void main()
{
    lowp vec4 col = texture2D( texture, vTexCoord)*vColor;
    gl_FragColor = col;
}
]]}

yojimbo2000 · May 2015

@Ignatz Gosh, that's pretty close. The appearance and thickness of the line varies though as the camera moves around, so that it appears and disappears at certain points. I'm going to play with it, see if I can get the line to appear steady.

yojimbo2000 · May 2015

What is the dot doing in your version? I get the same results when I just query the z, like this if (nView.z<=0.)

I guess that what we're seeing here, is that because the decision about what is front-facing is being decided on the vertex shader, it creates a less smooth effect as the camera rotates, as entire stretches of the line will blink in and out of visibility. I guess this is because, even though a triangular face cannot be curved (so in theory a vertex-level decision about whether a face is front-facing should be the same as a fragment-level decision), because the normals are describing a curving surface (ie they are "smooth-shading" averaged-point normals), we are not getting a consistent decision from the vertex shader about whether a face is front-facing?? Maybe this technique would work better with face normals (which would have to be on another attribute slot. Or set of attribute slots if you're key framing ....) Or gl_FrontFacing is cleverer than we thought it was.

Another issue with doing it in one pass, and I think this is again to do with the normals being averaged-point normals rather than face-normals, is the model at certain points "bleeds" into the black outline, both in terms of colour and shape. If you made the calculation with face-normals, then I expect you'd get a sharp delineation between the outline and the coloured sections.

yojimbo2000 · May 2015

Did a quick test, and using face-normals rather than average-point normals does get rid of the line-bleeding problem in the one-pass method, but the line still appears and disappears in chunks as the camera rotates. I also can't get a thick line with this one-pass method: if I increase the amount that the rear-facing vertices are extended by, I get a thin line, and then a gap between the line and the main body. It's as if it's only drawing the faces which are perpendicular, but is still discarding the ones facing backwards.

Ignatz · May 2015

Yep, checking nView.z works. I was comparing nView with the camera angle, but that is not needed.

I think OpenGL is working as advertised when it discards backfacing vertices. I'm not sure there is a way to do it with one pass, especially if you are averaging normals.

yojimbo2000 · May 2015

Yeah, it seems the only way to get it to draw back-facing normals (needed to achieve line thickness) is to discard the front-facing ones in a 2-pass approach. Thanks for the suggestion though, it was fun trying.

yojimbo2000 · May 2015

In this GLSL ebook http://en.wikibooks.org/wiki/GLSL_Programming/Unity/Toon_Shading
they describe the Unity method (which I'm adapting with my 2-pass approach), but then they describe a method similar to your one (except they do this calculation on the fragment shader). The outline, in other words, is drawn out of the inside edge of the model, instead of being something which expands the model (the Unity approach). I'll investigate this later, if I have time:

if (dot(viewDirection, normalDirection) 
               < mix(_UnlitOutlineThickness, _LitOutlineThickness, 
               max(0.0, dot(normalDirection, lightDirection))))
            {
               fragmentColor = 
                  vec3(_LightColor0) * vec3(_OutlineColor); 
            }

yojimbo2000 · May 2015

Their version works! I'll post my adaptation of it soon. It's a subtly different effect, but it looks very cool.

yojimbo2000 · May 2015

OK, here's the single-pass version, adapted from the GLSL e-Book. Blog here:

https://puffinturtle.wordpress.com/2015/05/28/toon-shader-redux/

And the shader:

--# ToonShader3
FrameBlendNoTexToon = { --models with no texture image
    splineVert= --vertex shader with catmull rom spline interpolation of key frames
    [[ 
    uniform mat4 modelViewProjection;
    uniform mat4 modelMatrix;
    uniform float ambient; // --strength of ambient light 0-1
    uniform vec4 lightColor;
    const vec4 front = vec4(0.,0.,1.,0.);

    uniform int frames[4]; //contains indexes to 4 frames needed for CatmullRom
    uniform float frameBlend; // how much to blend by
    float frameBlend2 = frameBlend * frameBlend; //pre calculated squared and cubed for Catmull Rom
    float frameBlend3 = frameBlend * frameBlend2;

    attribute vec4 color;

    attribute vec3 position;
    attribute vec3 position1;
    attribute vec3 position2; //not possible for attributes to be arrays in Gl Es2.0 
    attribute vec3 position3;
    attribute vec3 position4;

    attribute vec3 normal;
    attribute vec3 normal1;
    attribute vec3 normal2;
    attribute vec3 normal3;
    attribute vec3 normal4;

    vec3 getPos(int no) //home-made hash, ho hum.  
    {
        if (no==0) return position;
        if (no==1) return position1;
        if (no==2) return position2;
        if (no==3) return position3;
        if (no==4) return position4;
    }

    vec3 getNorm(int no)
    {
        if (no==0) return normal;
        if (no==1) return normal1;
        if (no==2) return normal2;
        if (no==3) return normal3;
        if (no==4) return normal4;
    }

    varying lowp vec4 vAmbient;
    varying lowp vec4 vColor;
    varying lowp vec4 vNormal;
    varying lowp vec4 vPosition;

    vec3 CatmullRom(float u, float u2, float u3, vec3 x0, vec3 x1, vec3 x2, vec3 x3 ) //returns value between x1 and x2
    {
    return ((2. * x1) + 
           (-x0 + x2) * u + 
           (2.*x0 - 5.*x1 + 4.*x2 - x3) * u2 + 
           (-x0 + 3.*x1 - 3.*x2 + x3) * u3) * 0.5;
    }

    void main()
    {       

        vec3 framePos = CatmullRom(frameBlend, frameBlend2, frameBlend3, getPos(frames[0]), getPos(frames[1]), getPos(frames[2]), getPos(frames[3]) );
       vec3 frameNorm = CatmullRom(frameBlend, frameBlend2, frameBlend3, getNorm(frames[0]), getNorm(frames[1]), getNorm(frames[2]), getNorm(frames[3]) );

        vNormal = normalize(modelMatrix * vec4( frameNorm, 0.0 ));
        vPosition = modelMatrix * vec4(framePos, 1.);

        vAmbient = color * ambient;
        vAmbient.a = 1.; 
        vColor = color; 

        gl_Position = modelViewProjection * vec4(framePos, 1.);
    }

    ]],
    --frag shader with hard outline effect, option for posterization, specular highlights
    frag = [[ 
    precision highp float;
    uniform vec4 light; //--directional light direction (x,y,z,0)
    uniform vec4 eye; // -- position of camera (x,y,z,1)

    varying lowp vec4 vColor;
    varying lowp vec4 vAmbient;  
    varying lowp vec4 vNormal;
    varying lowp vec4 vPosition;

    //posterization
    const float posterize = 3.; //layers of posterization
    const float unposterize = 1./posterize;

    //specular highlights
    const float specularPower = 64.; // higher number = smaller highlight
    // const float shine = 8.; 

    //outline thickness
    const float litThickness = 0.2; //line thickness for well lit areas
    const float unlitThickness = 0.3; //line thickness for unlit areas

    void main()
    {   

        vec4 viewDirection = normalize(eye - vPosition);
        //diffuse
        vec4 nNorm = normalize( vNormal );
        float intensity = max( 0.0, dot( nNorm, light ));

        if (dot(viewDirection, vNormal) < smoothstep(unlitThickness, litThickness, intensity)) 
        {
            gl_FragColor = vAmbient; //vec4(0.,0.,0.,1.); //dark or black outline
        }
        else
        {
            vec4 col = vec4(vColor.rgb, 1.);
            float shine = vColor.a; //shininess is encoded on color alpha
            //specular blinn-phong. Uncomment and add "+ specular" to end of gl_FragColor line below
            vec4 halfAngle = normalize( viewDirection + light );
            float spec = pow( max( 0.0, dot( nNorm, halfAngle)), specularPower );
            vec4 specular =  spec * shine * 8. * col; //

            //gl_FragColor=vAmbient + col * ceil(intensity * posterize) *unposterize + specular ; //posterize colours.
            gl_FragColor=vAmbient + col * intensity + specular; //non-posterized colours. 
        }
    }

    ]]
    }

Ignatz · May 2015

can you provide some sample Codea code to show what the uniforms look like?

yojimbo2000 · May 2015

Here's the update source including the shader:

https://github.com/Utsira/examples/blob/master/Toon shader 2.lua

yojimbo2000 · May 2015

just spotted a bunch of mistakes in the shader (not normalizing the normals, normalizing vPosition by mistake, not adding colour to the specular component). Edit: errors fixed, in the code above, on the blog, and on github. I also added support for having different specular intensities on different parts of the model, by storing the intensity value in the alpha of the vertex colour.

Jmv38 · May 2015

@yojimbo2000 @-) (looking for a jaw-dropping smiley, but couldnt find any...)

yojimbo2000 · May 2015

Thanks, I'm just standing on others' shoulders!

speaking of which, this GLSL eBook is fantastic. It's aimed at Unity, but as with the code above, it's not too hard to port chunks of it to Codea:

http://en.wikibooks.org/wiki/GLSL_Programming/Unity

Howdy, Stranger!

Categories

In this Discussion

Toon/Cel shader, with video, code, blog

Comments