Description
Goals:
-
Understand how to program shaders that run on the GPU.
-
Become familiar with setting up a modern GPU drawing pipeline.
-
Understand cartoon shading.
-
Understand texture coordinates.
-
Understand normal mapping.
-
Understand cube maps.
-
(optional) Understand how to compute a tangent-space from texture coordinates.
Background
- Topics: The Graphics Pipeline, Viewing, Illumination, Texture Mapping, Meshes
- Book (FoCG,4e): Chapter 17Â Using Graphics Hardware.
- Video: “Lecture 8: The Graphics Pipeline” and “Lecture 9: Texture Mapping”, “Assignment 6: Shaders aka The Graphics Pipeline”, and “Assignment 6: Shaders addendum”
- Quiz: Mesh Processing
(FoCG,4e is Fundamentals of Computer Graphics (4th edition) by Steve Marschner and Peter Shirley, 4th edition.)
Getting Started & Handing In:
-
Download or clone this code repository. I recommend cloning so you can easily integrate updates I release while you are working on the project. Don’t fork it on GitHub, or else your code will be visible to everyone.
-
Follow the instructions to install a working development environment:Â https://github.com/yig/graphics101.
-
Optional: Install the glfw library for creating an OpenGL window. If you don’t install it this way, theÂ
CMakeLists.txt
 will download and compileÂglfw
 on the fly.-
Mac with Homebrew package manager:Â
brew install glfw
. -
Ubuntu Linux:Â
sudo apt-get install libglfw3-dev
. -
Windows: It’s complicated. Just let theÂ
CMakeLists.txt
 compile it for you. If you really want to, you can use vcpkg. If you clonedÂvcpkg
 asÂC:\src\vcpkg
:git clone https://github.com/microsoft/vcpkg .\vcpkg\bootstrap-vcpkg.bat .\vcpkg\vcpkg install glfw3
if you are using a 64-bit Windows compiler toolchain, replace the last command with:
.\vcpkg\vcpkg install glfw3:x64-windows
and then copy the GLFW folderÂ
vcpkg
 installs to yourÂProgram Files
 or else add the following flag to yourÂcmake ..
:-DCMAKE_TOOLCHAIN_FILE=C:\src\vcpkg/scripts/buildsystems/vcpkg.cmake
-
-
-
The code will be written in GLSL (OpenGL shading language) and C++. You are encouraged to write helper functions as you like.
-
There are separate downloads for three large examples. Unzip them and move theÂ
head
,Âlemon
, andÂhercules
 directories into yourÂpipeline/examples/
 directory so that you end up withÂpipeline/examples/head
,Âpipeline/examples/lemon
, andÂpipeline/examples/hercules
. -
Copy theÂ
mesh.cpp
‘s functionsÂcomputeNormals()
,ÂnormalizingTransformation()
, andÂapplyTransformation()
 from theÂmesh.cpp
 you wrote for the previous homework on meshes. Don’t replace theÂmesh.cpp
 file with the one you previously wrote, because there’s a new function for this project and some functions aren’t needed (the half-edge support code is missing from this project). If you already turned in your meshes homework, you can email us to request a guaranteed-correct version ofÂmeshes.cpp
. -
Build and run the code. The code should compile. When it runs, it will ask you to choose a JSON file specifying a scene. It can also be run as a command-line program taking a single argument, the path to the JSON file. That way you can specify a command-line argument in Qt Creator and simply run. Run these two JSON scene files to verify that everything is working. They don’t depend on any code you write:
simplescene.json sphere.json
You should see:
Once you add the missing code toÂ
vao.cpp
,Âsphere.json
 will look as follows: -
ImplementÂ
vao.cpp
‘sÂmakeFromMesh()
 to upload normals and texture coordinates to the GPU as per-vertex attributes. Create your vertex shader and fragment shaders for each shader. Modify the JSON scene files as needed. Tangent-space normal mapping will require you to modify additional C++ code (mesh.cpp
‘sÂcomputeTangentBitangent()
 andÂfancyscene.cpp
‘sÂvaoFromOBJPath()
). -
Build and run and test that it is working correctly. If you run theÂ
pipeline
 program with theÂ-h
 orÂ--help
 flag, you should see the following:Usage: ./pipeline [--width pixels] [--height pixels] [--screenshot path/to/save.png] [path/to/scene.json]
-
Check your work with the autograder.
-
Here are the JSON scene files that will verify your output. Save screenshots by pressing theÂ
s
 key or by passingÂ--screenshot output.png
 to the executable. Save a screenshot for each of the JSON scenes into an output subdirectoryÂscreenshots
.phong_bunny.json
phong_sphere.json
cel_bunny.json
matcap_bunny.json
matcap_head.json
matcap_hercules.json
matcap_lemon.json
matcap_sphere.json
normalmap_cube.json
normalmap_head.json
normalmap_lemon.json
normalmap_hercules_bronze.json
normalmap_hercules_marble.json
- … and any others that you make. You are required to make at least one (discussed below).
-
There is a cmake targetÂ
screenshots
 that will do this for you, so you can just typeÂmake screenshots
 orÂcmake --target screenshots --build .
 in yourÂbuild
 directory. -
You will find reference output in theÂ
examples/
 directory. For eachÂ<filename>.json
 file, there is aÂ<filename>.png
 screenshot taken from the default point of view. -
Qt Creator has a great debugger for C++. For shaders, debugging is a bit trickier. When there is a GLSL error, you will see the shader code and the line number printed out to Application Output in Qt Creator.
-
Create a file namedÂ
Notes.txt
 in the folder. Describe any known issues or extra features or things you attempted but did not finish. Name people in the class who deserve a star for helping you (not by giving your their code!). -
When done, run the theÂ
cpack
 command from inside your build directory to generate an appropriate zip file of yourÂpipeline
 project. The zip file it creates,Âpipeline.zip
, will include theÂscreenshots
 subdirectory and yourÂNotes.txt
 file. It will ignore unneeded large and numerous directories (e.g.Âbuild
,Âexamples/head
,Âexamples/hercules
,Âexamples/lemon
). Upload yourÂpipeline.zip
 to Blackboard before the deadline. -
THIS IS AN INDIVIDUAL, NOT A GROUP ASSIGNMENT. That means all code written for this assignment should be original! Although you are permitted to consult with each other while working on this assignment, code that is substantially the same will be considered cheating. In yourÂ
Notes.txt
, please note who deserves a star (who helped you with the assignment).
Overview:
In this assignment, you will be writing GLSL shaders. You will write a Phong shader (the same lighting model as in your ray tracer). You will implement reflections with a cube map. You will load high resolution normals from a texture and implement tangent-space normal mapping. (For this, you will also need to implement some additional mesh processing to create additional per-vertex attributes.) You will write a cartoon shader. For bonus, you can also write additional shaders and write OpenGL code to setup a different rendering pipeline. Your output will look like this:
Rubric:
-
(40 points)Â Phong reflectance model. Your vertex shader only needs to transform the positions and normals. Your fragment shader is where the lighting is calculated. This is the same lighting model as you implemented for your ray tracer. However, you won’t be able to implement the global illumination terms the same way (shadow and reflection and refraction). In this assignment, you won’t be able to implement shadows at all. The simplified formula is:
KR IR + KT IT + ∑L ( KA IAL + [ KD IL ( N · L ) + KS IL ( V · R )n ] )
Read on for details. In this assignment, all the parameters are stored as uniforms, which are global variables for shaders programs. KA, KS, n, KR, and KT refer toÂ
material.color_ambient
,Âmaterial.color_specular
,Âmaterial.shininess
,Âmaterial.color_reflect
, andÂmaterial.color_refract
. If the booleanÂmaterial.use_diffuse_texture
 is false, then KD =Âmaterial.color_diffuse
. IfÂmaterial.use_diffuse_texture
 is true, use KD =Âmaterial.color_diffuse
 times the color stored in the texture map: KD =Âmaterial.color_diffuse*texture( material.diffuse_texture, fTexCoord )
. The summation is over allÂnum_lights
 of theÂLight
 structures; IAL and IL refer toÂlights[i].color_ambient
 andÂ.color
. The lightÂ.position
 should be interpreted as existing in eye-space, not world-space. See Tips below for converting between world and eye space.-
(10 points)Â Upload the normals and texture coordinates to the GPU as per-vertex attributes. To do this, you will implementÂ
vao.cpp
‘sÂmakeFromMesh()
. You will see in that function howÂmesh.positions
 are uploaded. Do the same forÂmesh.normals
 andÂmesh.texcoords
. -
(5 points)Â Vertex shaderÂ
phong.vs
. Use the provided uniformsÂuProjectionMatrix
,ÂuViewMatrix
, andÂuNormalMatrix
 to transform the input world-space positionÂvPos
 and normalÂvNormal
 into eye-space and pass them out to the fragment shader and set the output variableÂgl_Position
. Pass the texture coordinateÂvTexCoord
 out unchanged.-
fPos
,ÂfNormal
, andÂfTexCoord
 are output variables, which is how you pass along this information to the fragment shader, which is where you do your lighting calculations. You can pass alongÂfPos
 andÂfNormal
 in world-space or in eye-space, so long as you use a consistent space when you compute your lighting. (If you want to compute lighting in world-space, you will have to convert the light position and the eye position into world-space by multiplying them withÂinverse(uViewMatrix)
.) -
uViewMatrix
 is the matrix that converts a position in world-space into a position in eye-space:ÂuViewMatrix*p
 convertsÂp
 from world to eye-space.ÂuNormalMatrix
 is the matrix that converts a normal from world-space to a normal in camera-space. It is equal toÂtranspose(inverse(mat3(uViewMatrix)))
. -
uProjectionMatrix
 is the matrix that converts a position from camera-space to normalized-device coordinates. -
gl_Position
 is the one variable you are required to assign in any vertex shader, so that the GPU knows which pixels are inside the triangle. You must setÂgl_Position
 to the position in normalized device coordinates.
-
-
(20 points)Â Fragment shaderÂ
phong.fs
. Implement ambient, diffuse, and specular illumination.-
(2 points)Â Ambient lighting:Â KAÂ IAL.
-
(8 point) Diffuse lighting: KD IL ( N · L ). N is the (normalized) surface normal vector and L is the (normalized) vector from the surface position to the light’s position. Note that if this dot product is negative, then the light is behind the surface and you should not add diffuse OR specular lighting.
-
(10 points) Specular lighting: KS IL ( V · R )n. V is the (normalized) vector from the surface position to the eye position. In eye-space the eye is located at the origin. R is the (normalized) direction from the light position to the surface position, reflected across the surface normal. The formula for reflecting a vector across another vector is given below under Implementation Details (or you can use the GLSLÂ
reflect()
 function). Note that if the dot product is negative, then the light is reflected away from the viewer and you should not add any specular lighting. (You can use max( 0, V · R) instead of anÂif
 statement if you desire.) Also note that if the dot product used for diffuse lighting is zero (the light is behind the surface), then you also should not add specular lighting.
-
-
(5 points)Â Implement reflections with a cube map (in the fragment shader) ifÂ
material.reflective
. Use the world-space reflected view direction as the parameter to the cube map:Âtexture( uEnvironmentTex, direction )
. The resulting color is IR. If the direction you pass to the cube map is in eye-space, it’s as if the environment is rotating with the viewer, which means that the reflection will not change as you rotate the camera with the mouse. -
(bonus 5 points)Â Implement refraction ifÂ
material.refractive
. Use the same cube map texture as in reflections to get IT. You can get the refraction direction via the GLSLÂrefract()
 function.
-
-
(30 points)Â Cel or toon shading. You only need to write a fragment shaderÂ
cel.fs
, since the same vertex shader as the one you wrote for Phong shadingÂphong.vs
 can be re-used. Cel shading is the lighting effect used in games like Zelda: Wind Waker and Breath of the Wild, Jet Set Radio, and Naruto Shippuden. To achieve cel shading, we compute a scalar brightness value, round it to discrete levels, and then use that to scale the material color. The scalar brightness value F is computed with a simplified Phong reflectance in which no material colors K appear and the light color terms IAL and IL forÂ.color_ambient
 andÂ.color
 are simplified into scalar intensities:F = ∑L (IAL + [IL ( N · L ) + IL ( V · R )n])
Use the integer uniformÂ
material.bands
 to round F into discrete levels: Fdiscrete = min( floor( F * F *Âmaterial.bands
 ),Âmaterial.bands
 – 1 )/(Âmaterial.bands
 – 1). (Squaring F makes the bands more uniformly spaced.) The final color for a pixel is K * Fdiscrete, where K is the material color. If the booleanÂmaterial.use_diffuse_texture
 is false, then K =Âmaterial.color
. IfÂmaterial.use_diffuse_texture
 is true, use K =Âmaterial.color
 times the color stored in the texture map: K =Âmaterial.color*texture( material.diffuse_texture, fTexCoord )
. You will re-use yourÂphong.vs
 and only write a differentÂcel.fs
. -
(20 points + bonus 3 points)Â MatCaps with object-space normal mapping. You only need to write a fragment shaderÂ
matcap.fs
, since the same vertex shader as the one you wrote for Phong shadingÂphong.vs
 can be re-used. A MatCap (Material Capture) is a delightfully simple and flexible technique to create realistic or artistic lighting for a shape. The idea is simple. Different points on a shape look different because the normals are different. Think about your Phong reflectance model. The material parameters and lights are shared by every point on the object. The vector V from the point to the eye varies slightly, but we could approximate this with a constant direction. This is all to say that the appearance of an object is almost entirely dependent on its normals in eye space. A MatCap is a lookup table from a normal to a color. A sphere has normals pointing in all directions. All we need to do is render a sphere and save its colors. To render, we just need to look up the color of the sphere with that normal. In fact, we only render the part of the shape we can see, so we only need to worry about front-facing normals (with z>0 in eye-space). This gives us a very convenient way to store the lookup table:
We store the front half of the sphere as a circle. To look up the color from the matcap, we just use the x,y component of the normal scaled and shifted to lie in the range [0,1]² instead of [-1,1]²: color = texture( material.matcap_texture, n.xy*0.5 + 0.5 ).rgb
. We can create MatCaps any way we like. We can render a sphere with physically-based materials in a sophisticated lighting environment:
We can also create an artistic sphere:
Those MatCaps create the following bunnies:
You can see lots of MatCaps visualized on 3D shapes here.
-
(10 points)Â InÂ
matcap.fs
, set the output color to the color stored in the MatCap texture for the eye-space normalÂfNormal
. IfÂmaterial.use_diffuse_texture
, multiply the color by the diffuse texture colorÂtexture( material.diffuse_texture, fTexCoord ).rgba
. -
(10 points)Â Object-space normal mapping. Texture resolution is typically much higher than vertex resolution. This is why we typically store colors in a texture map and not as vertex attributes. Along these same lines, we can store normals in a normal map in order to have render super-detailed surfaces. Normal maps are typically stored in tangent-space, which is substantially more difficult to implement. (Tangent-space normal mapping is described below for bonus points.) InÂ
matcap.fs
, ifÂmaterial.use_normal_map
, read the object-space normal from the texture viaÂtexture( material.normal_map, fTexCoord ).rgb
. Normal components range from [-1,1], while colors range from [0,1], so each normal component is stored in the texture asÂ0.5*(n+1)
. Convert the color stored in the normalmap texture back to a normal viaÂ2*color-1
. This normal is in object-space. Convert it to eye-space usingÂuNormalMatrix
 and then use it instead ofÂfNormal
 to read from the MatCap. -
(bonus 3 points)Â Make your own MatCap! You can make a toon shader MatCap by taking a screenshot of a toon-shaded sphere. You can also make much cooler stuff. Have fun with this.
-
(10 points)Â Be creative! Create a time-varying artistic shader of your own design. Make use of the uniformÂ
uTime
, which stores the seconds since the program was launched. Declare it in your shader asÂuniform float uTime;
 Be sure to changeÂTimerMilliseconds
 in the scene JSON file to something like 16 (which corresponds to 60 frames-per-second). SeeÂsphere.vs
 for an example. -
(bonus 30 points)Â Normal mapping (
normalmap.vs
 andÂnormalmap.fs
). This is an extension of your Phong reflectance model shader. With normal mapping, the texture stores a normal vector. Because lighting is entirely determined by the normal vector, high resolution normals make a surface look incredibly detailed. The normals in a normal maps are typically stored as vectors in the tangent space of the surface. This technique requires you to compute a “tangent frame” for each vertex (tangent and bitangent vectors to accompany the normal) and upload that to the GPU as additional attributes. To do this, you will implementÂmesh.cpp
ÂcomputeTangentBitangent()
 to compute the tangent frame andÂfancyscene.cpp
ÂvaoFromOBJPath()
 to upload the additional attributes to the GPU. (Computing and uploading this additional data for this shader won’t get in the way of the other shaders.) Each face has a well-defined tangent frame derived from the texture coordinates of its vertices; see the accompanying handout for details. Just like with per-vertex normals, you will pre-compute each face’s tangent frame vectors and average them at the vertices. In the vertex shader, you will convert the tangent and bitangent vectors from world-space to eye-space (usingÂuViewMatrix
) and pass them to the fragment shader. In the fragment shader, you will extract the tangent-space normal from the texture and convert it to world-space with the tangent frame matrix. That will be the normal you use for your lighting calculations. (You can reconstruct the tangent-frame matrix from the tangent, bitangent, and normal. Don’t forget to normalize them.) Implementation note: Normal components range from [-1,1], while colors range from [0,1], so each normal component is stored in the texture asÂ0.5*(n+1)
. Convert the color back to a normal viaÂ2*color-1
.-
For a deeper explanation of the coordinate systems involved in tangent-space normal mapping, see this PDF.
-
For a video explanation of normal mapping, see this YouTube video.
-
-
(bonus variable points):
-
Image Processing (blur, edge detect, etc.). Draw a full-screen textured square. Your vertex shader should pass along the position and texture coordinates, ignoring the view and projection transformations. Your fragment shader accesses the texture value at the pixel and at nearby pixels. You can use the GLSL functionÂ
textureSize( tex, 0 )
 to get the texture width and height in pixels so that you can generate texture coordinates for the surrounding pixels. -
Ray Tracer or Signed Distance Field renderer. Implement ray tracing or ray marching (as in some ShaderToy examples). Just like with Image Processing, the vertex shader does nothing. The fragment shader implements an entire ray tracer or ray marching. All geometry is specified via uniforms.
-
Animation. You can create a sequence of transformation matrices and interpolate between them (linearly or with an easing function). You can go further and implement linear blend skinning, in which there are multiple sequences of transformation matrices animating in parallel, and each vertex has a set of associated “skin” weights. The vertex is transformed by the weighted average transformation matrix. Some of the infrastructure for this is already implemented.
-
Ambient Occlusion. This provides a much more accurate surface ambient color KA by computing the fraction of directions that are blocked by the mesh (a ray exiting the point with that direction would intersect the mesh). You can compute this in a pre-computation step or approximate it with a multi-pass shader. When computing it as a pre-process, you can store per-vertex attributes or a texture map.
-
Procedural textures. Generate a wood or marble color as a 3D function of position.
-
Geometry or tessellation shaders. We haven’t talked about these more exotic shaders, but the framework supports them.
-
Physically-Based Rendering. There are many ways to pursue this. Note that theÂ
lemon
 example comes with a gloss map. -
Something else!
-
Tips
-
If you see a blank screen (all background color and no shape), the problem is most likely in your vertex shader. This is assuming that your environment works (e.g. thatÂ
simplescene.json
 andÂsphere.json
 produce the expected output). -
The glm library is modeled after the OpenGL shading language GLSL. Anything involvingÂ
vec2
,Âvec3
,Âvec4
,Âmat3
,Âmat4
, etc. and virtually any GLSL functions should be accessible in C++ with theÂglm::
 prefix. There are many convenience constructors and math operations. Here is a nice reference: GLSL Programming: Vector and Matrix Operations. -
The GLSL (and glm) matrix constructors that take vectors use those vectors as the columns of the matrix, not the rows. (If you want to fill out the rows, fill out the columns and then callÂ
transpose()
.) In fact, all matrix indexing uses column-then-row (column-major) ordering, which is different from what mathematicians do. -
In C++ you can call the constructor of an object when you declare it, like so:
vec4 foo( 1.0, 2.0, 3.0, 4.0 );
GLSL does not allow this and you will get a compile error. Instead, you have to write:
vec4 foo = vec4( 1.0, 2.0, 3.0, 4.0 );
-
Unless you assign a value, a local variable you declare will be uninitialized. While some implementations will initialize, for example,Â
vec3 foo;
 to 0 by default, the GLSL specification does not require this. (The grader’s graphics driver will probably initialize to random values.) Manually initialize to 0,Âvec3 foo = vec3(0);
, if that’s what you want. -
There is a simple, sample 3D shader inÂ
sphere.json
 /Âsphere.vs
 /Âsphere.fs
. (The simplest possible shader is inÂsimplescene.json
 /Âsimplest.vs
 /Âsimplest.fs
 and is based off theÂsimplescene.h/cpp
 C++ setup code.) -
To convert a position from world-space to eye-space, left-multiply the provided uniformÂ
uViewMatrix
 by the position. To convert a normal from world-space to eye-space, use the provided uniformÂuNormalMatrix
, which is the inverse-transpose of the upper-left 3×3 portion ofÂuViewMatrix
. To convert a position or direction from eye-to-world, useÂinverse( uViewMatrix )
. To convert a direction (other than a normal) from eye-to-world, you can be more efficient and useÂtranspose( uNormalMatrix )
. -
The GLSL functionÂ
texture(...)
, which loads a value from a texture, returns aÂvec4
. Some overly permissive OpenGL drivers will allow you to use it as aÂvec3
, but that behavior is wrong. UseÂtexture(...).rgb
 to get the RGB components as aÂvec3
 or callÂvec3(texture(...))
. -
To reflect a (not necessarily normalized) vector v across a normalized vector n, the formula for the reflected vector r is r = v – 2(v·n)n.
-
All of the required portions of the assignment can use the same C++ setup code (defined inÂ
fancyscene.cpp
/fancyscene.h
) and differ only in which shaders and uniforms are specified in the JSON file. (Only normal mapping requires you to modify some of the C++ setup code.) -
You can achieve better code re-use in the shaders if you split them up. When you specify a shader in the JSON file, you can pass an array of file paths which will be concatenated together in sequence and then compiled. This allows you to write common helper functions in aÂ
common.glsl
 file if you so desire. The first line must always be aÂ#version
 directive, so I often split my complex shaders intoÂ[ shaderTop.vs, common.glsl, shaderBottom.vs ]
. It may also make sense to share a bottom but have different middles (e.g. normal mapping and Phong shading could share a bottom but have different middle files implementing an appropriateÂget_normal()
 function). -
You can create your own C++ scene if you want to customize the camera or attributes. You will need to duplicate the scene class (
SimpleScene
 orÂFancyScene
) and register it withÂpipelineguifactory.cpp
. For reference, you can see/study the simplest-possible OpenGL scene inÂsimplescene.h
 andÂsimplescene.cpp
. -
Almost everything in a JSON scene will live reload if using aÂ
FancyScene
, including shaders, uniforms, and textures. The only property that will not live reload isÂPipelineGUI
. -
You can find lots of Creative Commons environment (cube) maps on the website of Emil Persson, aka Humus:Â http://www.humus.name
-
You can find lots of MatCaps on this website:Â https://github.com/nidorx/matcaps. You can see all the MatCaps nicely visualized here:Â https://observablehq.com/@makio135/matcaps?ui=classic