Introduction

For a while I have been trying to create toon shading that will look similar to the vector painting done with a Wacom tablet in an application such as Flash or Illustrator. A great example of that style is cover art for Grand Theft Auto series, espeically GTA III and Vice City (in latest sequel, San Andreas, there are more shades of color and the style is a bit different) – to see what I mean, head over to Rockstar North website and navigate to Dropout Center > Artwork.

The shading part of the this style wasn’t hard to figure out, especially since there already exist a nice shader to be used in toon style rendering: ramp shader. But what about the contours? Well, the contours are not much of the problem either, because of the integration of mental ray and Maya, which now involves minimal work to set up the contour rendering. I will not go in depth with the setup, because it is nicely explained elsewhere. I recommend the toon rendering tutorial on alias website (you will need to log in to access this page) – Edit: Since Autodesk acquired Alias, most of the tutorials that used to be free, now require silver membership that cost $20 per month! Oh well, at least there are some free tutorials right here ;).

If you don’t see option to create mental ray nodes in hyper shade, you can read instruction on how to set up the environment variables so that you will be able to access mental ray shading nodes inside hypershade. There is another nice tutorial that also covers options of mental ray’s contour shaders here. Unfortunatelly, that’s not all. We will need to set up additional two types of contours to render images in the above mentioned style, so let’s get going.

Setting up Ramp Shader

With assumption that you have some model to work with (and that model is not a plain primitive, but something more interesting, preferably a face – see the one I’m using pictured on the left – or whole human body model), start by creating a ramp shader in hypershade, multillister, or the perspective window (by right clicking on your model and choosing Material > Assign New Material > Ramp Shader). Assign the shader to your object and open its Attribute editor. There are couple of things to adjust, as seen on the snapshot. Mainly, we need to set Color Input to Brighrness and interpolation for color to None, so that we don’t have smooth transitions between different shades. I am working with just two colors, but you could add one more if you prefer. So pick two or more colors for your object. I recommend you to increase saturation a little bit for darker shades so that the object doesn’t seem to be too greyish in the shadow area. Make sure specularity is set to 0, as we don’t want the soft specular highlight. Also, important thing to set up for non-reflective object is the value of 0 for reflection and beware that this value is set to 1 or 0.5 for some materials by default, which will make your object highly reflective. If you know that you will not want to have any reflections in your scene, you could turn them off in mental ray render globals (under Raytracing tab, set reflections to 0, perhaps you could do the same for refractions as well), but you will probably want raytracing turned on, since raytraced shadows are a good choice for our rendering needs in this case since they will be colored with a shade we picked for dark values for our ramp shader. (Note: raytracing in mental ray is on by default, even in the draft mode).

The incadescence of the ramp shader will give us our first contour. The similar contours that incadescence produces could be otherwise created with the Sampler Info utility node, because they are calculated based on the facing angle between the surface and the camera. The way to set up these contours is to first set the interpolation for incadescence to none and set color at position 0 to be H: 0 B: 0 and V: -1 (the slider will not go to negative values, so you will need to type in -1). Then add another color – a regular black, with 0, 0, 0 for HSB, at position at about 0.25. There is one more thing to take care of before we test render the scene: go to camera’s attribude editor and in the Environment tab, set the background color to something different from black, so that you can see your black contours against that color (i used middle grey). Also, create a simple spotlight and enable raytrace shadows in Shadows tab. Now you are ready to do a test render. See mine.

As you can see, the results with ramp shader are not bad at all. We got a basic toon shading with nice painterly contours that really look like they have been drawn in vector paint program. You may want to play around with the position of second color you placed for incadescence color value, because it will determine how thick your lines are. The closer the second color is to the first one, the thinner the lines will be.

Adding mental ray contours

You will notice, however, that we are lacking some contour detail around the eyes and in the ear area. The way to add some more details to these areas is to add another kind of contours, which are calculated differently. Now mental ray contour shaders come into play. If you haven’t done so already, head over to the mental ray contour shader tutorials from the links above, to familiarize yourself with the properties these shaders have and results they produce. Briefly, what you need to do is following:

1. In render globals, mental ray tab, under contours tab, check the box next to Enable Contour Rendering.

2. Scroll down to the bottom of contour tab and open Custom Shaders section. Click the icon next to Contrast Shader and scroll down to Contour Contrast tab. Pick Contour_contrast_function_levels.
3. Back in render globals, assign the Contour_store_function to the Store Shader, the same way.

4. Open the attribude editor for Contour contrast shader (you can fiind it under utilities tab in hypershade, or you can get to it from render globals, where you assigned it to Contour Contrast). You will see that Store shader does not have any options, but Contour Contrast shader does. Adjust them as follows:

Leaving [ ] diff_index checked will render wireframe. If you want to have contours between different materials, leave [ ] diff_mat checked. You may want to experiment with the values of Zdelta and Ndelta to get a desired amount of contours, because these two are the main parameters that determine where on your object contours will be drawn. But lets first give our ramp shader a contour to work with.

Open the attribute editor for the ramp shader and get to its shading group tab by clicking the outputs button . In its mental ray tab, under custom shaders section, click next to Contour Shader to add a custom shader for contours. Now it’s time to pick the appropriate contour shader. Again, I will not explain here how each of them works, because it’s explained elsewhere. I prefer to use Conrour_shader_depthfade (second one in the left column, in Contour Shader tab), because it allows me to make the distant contours really thin or invisible. You will need to experiment with the settings for this shader as well, since near_z and far_z will depend on the shot and whether it is a closeup or full shot, because these two values determine how close to the camera the near width and near color will be reached and where in the distance the far_color and far_width will be reached. My settings usually look like this:

Notice that I decreased Near_width and Far_width values way down, as the default values would produce a very thick contour.

Finally, lets see what we got. See my render here. Notice that we now have contours around the eyelids and on couple more places. But what about those beautiful eyelbrows and wrinkles, you ask. Well, keep reading…

Setting up UVs

You are probably wondering why you need to set up UVs for your model, if all we want to do is render some contours and some flat areas of color. Well, not exactly. We will need some detail that would not be possible to get solely by relying on contours – or it will be possible by building very detailed geometry, which is, in my opinion, not the best solution for creating cartoonish character. However, it is not necessary to do a complete UV layout. You will need the layout of UVs only in the areas where you wish to add aditional line detail, in my case it will only be on the face. If you are not getting enough contour line detail on the hands, and especially fingernails, then you might want to do a layout of UVs of the hand as well, or model more detailed hands. But for now, just do a cylindrical projection around the head, fix problematic areas with too much stretching. See what my layout looks like. You could do multiple UV layouts for different parts of your character by using UV Sets. When creating UV snapshot, I recommend that you do it in Targa format and 2048 square size, because to get a vector paint look of the texture we will paint, we need a high resolution to work with so that the raster quality of the texture does not become noticeable or, in another words, so that the texture doesn’t look pixelated. We are after vector paint look and maybe one day we will be able to apply vector format images as textures, but for now large raster image texture is the only solution.

Paintiing a Texture

Now this is the fun part! To achieve the vector paint look, we will actually do some vector painting. Fire up your favorite Vector Paint program, such as Illustrator, FreeHand or, believe it or not, Flash, which I think has a wonderful vector paint capabilities, as it allows you to apply various degrees of smoothing to the shapes you paint. Now, for this part it’s imperative to use a tablet, because we want to paint the lines of varying width.

If you are using Adobe Illustrator, do the following:

1. Create a new document 2048 by 2048 pixels, then go to File > Place… and locate the UV snapshot you previously created. Check the Link and Template checkboxes and click OK. This will place your reference image on a locked layer, so you will not have to worry about accidentally moving it or drawing on it.

2. Now that your UV layout fits the document size, do one more thing before you go on. Create a square with a square tool and snap it to the edges of your UV image, because Illustrator will export only the minimum area where you have shapes, and we will need the exact 2048 square texture to match our UVs. I made a mistake here and it took me some time to figure out why my texture doesn’t match my UVs – because Illustrator exported the cropped image! You can easily snap the square to the corners of your imported image, by having smart guides turned on (ctrl + U, or View > Smart Guides).

3. Make sure the square has no fill, and has only a stroke (could be of any color, since it won’t be mapped onto your character, unless you have UVs that go over the edge, which is not recommended). Deselect the square and choose the brush you want to paint with. I usually use 10pt round brush, the third one in the default brushes palette. Double click on it and adjust it as follows:

Now we have set up the brush to make nice lines whose width will be determined by pressure. You can always come back and adjust the Variation and Diameter value to more or less, depending on your needs.

4. Select the Brush Tool and paint the wrinkles and other desired detail using your UV snapshot as a reference for line placement. See how my texture turned out.

5. Hide the template layer with the UV snapshot. Go to File > Export… and choose the Targa format. You can leave 72dpi (screen resolution), since you made the document big enough to hold all the detail. Leave Antialiased checkbox on, because aliased lines might look too jagged. You can choose either 24 or 32 bit. This should leave your transparency on. Finally, save an image where you can find it, such as Source Images folder inside your current Maya project folder.

If you are using Flash, follow along:

1. Fire up Flash, create new Flash document, give it a size of 2048 by 2048 pixels. Other document settings don’t matter.

2. Choose File > Import > Import to Stage… (or press Ctrl + R) and locate your UV snapshot. Flash will let you know that it doesn’t recognize the file format (if you saved your snapshot as Targa) and offer you to import it via QuickTime. Click OK. The snapshot will be placed on the stage.

3. Create a new layer by pressing in the timeline palette and lock the layer with your image so that you don’t accidentally draw on it or movie it.

4. Choose the Brush Tool (press B) and enable pressure, by clicking on . If you have Wacom Intuos or other tablet that has support for pen tilt, you can also enable tilt, by clicking on . Paint the wrinkles and other detail, using your UV snapshot on the locked layer below as a reference.

5. When you are done, delete the layer with UV snapshot, and export the image by going to File > Export > Export Image… You will not see many file types on the list. I usually choose to export flash linework as BMP file and then take it into Photoshop to fix the issues with transparency, if needed, because Flash will create an Alpha channel if you set Color Depth to 32 bit with alpha, but that may not work as desired in Maya. At least, I always have some issues with Bitmap file type, so I prefer it to bring it into photoshop and convert it to Targa. The export options should look like this:

Save the file where you can find it (preferably someplace inside your current Maya project folder).

That’s pretty much it with texture painting.

Assembling the Shading Network

We are now ready to go back to Maya and work with our shader. We will, however, need to “rewire” our shading network a little bit, but don’t worry, it will be easy.

First of all, we need to figure out how to keep the color of the ramp shader and combine it with our texture. Plugging in the texture we just created in previous step directly into Ramp Shader won’t work, so we need to figure out another way to handle this. There are at least two ways that we can do this, and I will explain both of them.

1. Using Layered Texture
Open Hypershade and graph the network of your ramp shader. We just need to have it in our work area for now. Create a Layered Texture Node by choosing Create > Layered Texture in Hypershade. Also create a Lambert material and select objects you assigned ramp shader to and assign this newly created Lambert Shader to them. Set Lambert’s Diffuse to 0 and its Ambient Color to white. Now connect the Layered Texture’s Out Color to Lambert’s Color (this is a default connection, so you can just middle-drag layered texture onto Lambert and choose Default or Color, same thing). Now connect your Ramp Shader (not its Shading Group) to Layered Texture node (middle-drag and drop and choose Default). Create a File node under 2D Texture section (make sure the texture mapping is set to Normal and not Projection or Stencil), open its attribute editor and choose the texture you just painted. Connect this file node to Layered Texture as you did with Ramp Shader, which should be default connect it to the next color input. Transparency is not necessary in this case, because we can set the Blend Mode for our file Texture to Multiply (see below), so that the white areas will become transparent and show the color of Ramp Shader that is underneath. But if you need to have transparency you will need to make sure you have alpha channel in your file texture and have the file’s out.alpha connected to layered texture’s inputs[1].alpha. Now, open the attribute editor for Layered Texture and make sure you only have two inputs, represented by blue rectangles, and that the one on the left is your file texture and the one on the right is your Ramp Shader. It should look like this:

If the rectangle that represents Ramp Shader is on the left, you can move it to the right by middle-mouse dragging it to the right of File node input. This order is important, because the file texture needs to be on the top of the Ramp Shader, or otherwise it wouldn’t show at all.

There is one more thing we need to reconnect, and that is the contour shader we previously assigned to Ramp Shader’s Shading Group, which needs to be connected to Lambert’s Shading Group instead. Also, if you have more than one UV Set, you might want to make sure that your file texture is linked to the proper one. With this setup you can pile up all your file textures in the Layered Texture node, on top of Ramp Shader, and connect them to appropriate UV Sets via UV linker (Window > Relationship Editors > UV Linking > Texture-Centric.. or UV-Centric). Locate texture on one side and connect it to appropriate UV Set on the other side – very simple. (Note: you need to select the object with textures and UV Sets for this to work). Check out the final shading network here.

2. Using Ramp Shader as Ambient Color
This setup is even easier than the previous one, because we have less nodes to work with. However, if we don’t use Layered Texture, we will not be able to use multiple textures for additional contours. Create, as before, Lambert material, set its Diffuse to 0 and connect your Ramp Shader to Lambert’s Ambient Color. Now assign the file texture with the line art you painted to the Lambert’s Color. And again, reconnect the contorur_shader_depthfade from Ramp’s Shading Group to Lambert’s Shading Group and assign Lambert to your object. And that’s it! See the shading network here. Finally, lets see what our last render looks like. Check it out!

Paint Effects Eyebrows

If you were wondering how I created the eyebrows that have third dimension (which shows in the wireframe, but not in the final render), you already know the secret: Paint Effects! Since version 5, Maya allows you to paint with Paint Effects brush directly on polygon object. Even more exciting addition to version 5 is the ability to convert the Paint Effects to polygons, which allows paint effects to be rendered in mental ray (paint effects are not supported in mental ray, they can only be rendered with Maya’s renderer). The connection between the stroke and polygon created from stroke is preserved so that you can use all Paint Effects features to animate the polygon mesh, including animated flow, growth, or Paint Effect forces, such as grass wind and turbulence.

To have paint effects work on polygons, we need to have all UVs laid out in 0 to 1 texture coordinate space and they can’t overlap. This is because Paint Effects use UV coordinates to map their position of the stroke on the polygon object. Since by default Paint Effects will use the default UV Set called map1 to position the strokes on, you should make sure this set has the complete layout that fits into 0 to 1 texture space and has no overlapping UVs. Press 8 to switch to the paint mode. Choose the brush you like. There are two eyebrow brushes in hair folder of the Visor. You may want to adjust the global scale of the stroke, and some of its other attributes, but you can do this later, since most of the stroke’s attributes can be changed after the stroke is created. All the strokes are linked to one UV set, which is, as mentioned above, default UV set called map1, but you can open the UV Linker by going to Window > Relationship Editors > UV Linking > Paint Effects/UV… and connect the Paint Effects strokes to different UV set; however, note that the stroke you already painted will most likely not be in place anymore if you switch the UV Set. Anyhow, all the new strokes you paint will be linked to the UV Set that you chose.

When you are happy with the position and overall look of your strokes (note: pen pressure works great with Paint Effects and you should use this feature to make the eyebrows thinner or thicker, longer or shorter; you could always readjust pressure mappings along the Paint Effects Curve in channel box or attribute editor), convert them to polygos, by selecting the strokes and choosing Modify > Convert > Paint Effects to Polygons. You will probably want to optimize this conversion, because if you have too many polygons, things can get really slow, of course. Maya has a handy tool to do this. With your strokes selected, in Rendering menu set go to Paint Effects > Paint Effects Mesh Quality… and move the sliders around. I recommend the lowest possible values that produce acceptable results. When the Paint Effects strokes are converted to polygons, they already have materials assigned to them and these are materials generated from the strokes, but you can easily change these materials (also note that polygons with different materials are created as different poly objects, so you can easily replace each material). In my exemple, I simply used black Lambert.

Also bear in mind that you can create hair using this Paint Effect technique. If you have Maya Unlimited 6 or above, you could use new powerful hair simulation, which also works with Paint Effects. At last, check out the finished rendered version of my character, with 3D eyebrows and glasses. The little dots around the mouth are caused by mental ray contour shader and I am not sure how to get rid of them. They usually don’t appear, but I just had bad luck when I rendered this.

Further Considerations

Toon shading can give some incredible results, but it is important to understand how to control it and how to use it for best results. Since toon shading puts so much emphasis on color, it is of greatest importance to pick your colors effectively. My tutorial on importing custom color palettes into Maya can help you with that process. Remember that lights do not have control over the color of your toon-shaded object. Lights only determine where bright and dark areas of colors are calculated, but you are in control of choosing the color. You don’t need to pick a single hue for one object. Try picking cooler hues (bluish, green or violet) for shadow, and a bit warmer for highlingts. See the following two examples:

The following image illustrates more extreme example of the hue difference between highlights and shadow areas:

Note how the dark areas are blue and bright areas are orange. The shadows are raytraced and shadow color is just another color on the Ramp Shader.

Please note: mental ray does not support Maya’s light effects, such as light glow or lens flare. It does, however, support shader glow, which can produce very similar results.. That is how all the glows are created in images above: build a simple geometry for lightbulbs and apply shader glow to it. Do not use Light Effects. Also note that you can only have one kind of Shader Glow, because this glow is in fact camera’s lens effect, and you cannot use different lenses in one shot – similarly, there is only one Shader Glow material that lets you control the appearance of the glow. If you wish different kinds of glow, you will have to render in passes.

I hope you got something out of this tutorial. If you have any questions, or would like particular explanation, please leave a comment.