This post is not exclusively about normal map generation in Maya, but I didn’t want to make the title too long. It’s about several issues that came up while I was generating normal map for my Aquadon model.
Here’s what happened: I imported high poly mesh into Maya and it didn’t exactly line up with my low resolution geometry, so I moved it to fit. When I generated normal map, it was out of place. The detail from upper arm transferred to lower arm, and I checked UVs couple of times – they were fine. I tried this again with broken up mesh – since Maya couldn’t quite handle 2+ million poly mesh, I broke it up to lower arm and upper arm, which is OK, since there is a texture seam in there anyways – but the same displaced normal map was generated again. Finally, I figured I should freeze the transformation of the high poly arm that was moved to fit – and voila! Conclusion: if you move your high or low poly meshes before generating normal map, remember to freeze transformations, because they matter to normal map calculation, and the map won’t be in the right if one of the meshes was moved.
Other thing that came up was this weird lighting artifacts happening on the low resolution arm. I first thought I might have baked vertex colors, because I tested some baking previously, but even when I flooded vertex color to middle grey, the messed up lighting effect was still happening. I figured it started happening after I have extracted upper arm faces and made it separate object. Both upper and lower arm had the same artifact. Smoothing edges didn’t do the trick. I knew it was something to do with vertices, and when I displayed their normals, I realized that the vertices that appear black have their normals reversed, or at least not visible from the outside. To fix this, I went to Edit Polygons > Normals > Set to Face – which creates the faceted look, as if all the edges were hard. Each vertex now has bunch of normals, each facing in the direction of normal of neighboring faces. To bring back the smooth look again, I selected all the vertices and went to Edit Polygons > Normals > Average Normals. Now all the vertex normals were snapped together and they create the smooth look.
A dumb, but useful reminder – normal maps won’t show in the viewport if the textures are not on (hotkey is 6), even when high quality rendering is on.
That’s all folks! Now off to finish high poly head of Aquadon.