Layered Rendering with Framebuffers

Sometimes, in a sketch, you might want to draw something to an image instead of directly to the screen. This lets you use the image as a texture on 3D shapes, repeat the image multiple times efficiently, read the image's data when drawing something else, and more.

In 2D mode, you can use createGraphics() to achieve this. While they also work in WebGL mode, for better performance and more features, you can use createFramebuffer().

What makes Framebuffers special?

Framebuffers live on your computer's GPU (Graphics Processing Unit), the part of your computer that specializes in drawing pixels of images as fast as possible in parallel. This means you can write to and read from Framebuffers really quickly, without your computer having to move lots of data around!

Framebuffers also can contain more information than a typical image or canvas. You can use higher-precision floating point numbers to store colors, letting you do more without running into weird visuals from rounding errors. They can also store 3D depth information about the contents drawn on them, which can help you create visual effects like blurs and shadows.

a line of spheres going off into the distance, showing first the color of the image, then a visualization of its depth, and then a version rendered with focal blur using both prior images

Framebuffer color

Framebuffer depth

Final image with focal blur using color+depth

Using Framebuffers

Framebuffers are like drawing targets. By calling .begin() on one, anything you draw will start going onto it instead of the main canvas. Calling .end() will make the main canvas be the draw target again, and you can then reference the framebuffer like an image.


let layer;

function setup() {
  createCanvas(windowWidth, windowHeight, WEBGL);
  layer = createFramebuffer();
}

function draw() {
  // Start drawing to the framebuffer
  layer.begin();

  clear();
  lights();
  noStroke();
  rotateX(millis() * 0.001);
  rotateY(millis() * 0.001);
  box(min(width/2, height/2));

  // Stop drawing to the layer so we can draw to the main canvas again
  layer.end();

  // Draw the layer to the main canvas
  clear();
  translate(-width/2, -height/2);
  for (let x = 0; x < 4; x++) {
    for (let y = 0; y < 4; y++) {
      image(
        layer,
        x*width/4, y*height/4,
        width/4, height/4
      );
    }
  }
}

Sketching with Feedback

Video feedback is a technique where the previous frame of a video gets used when drawing the next frame. This can look similar to when you stand between two mirrors.

To do this in a sketch, you typically draw to a layer so you have an image of the current frame. Then, when drawing to the next frame, you can add the previous frame to your drawing by using the image of the previous frame.

The fastest way to do this is to keep two layers around: one for the previous frame, and one for the next frame. At the start of draw(), swap the two layers so you can keep the image of the preview frame, clear the next frame, and draw a new image to it.


let prev, next;
function setup() {
  createCanvas(200, 200, WEBGL)
  prev = createFramebuffer({ format: FLOAT });
  next = createFramebuffer({ format: FLOAT });
  imageMode(CENTER);
}

function draw() {
  // Swap prev and next
  [prev, next] = [next, prev];
  
  // Clear next and draw a new next frame
  next.begin();
  clear();
  
  // Slightly rotate and scale the last frame
  push();
  rotate(0.01);
  scale(0.99);
  image(prev, 0, 0);
  pop();
  
  // Add a sphere on top
  translate(sin(frameCount*0.1)*50, sin(frameCount*0.11)*50);
  noStroke();
  normalMaterial();
  sphere(25);
  
  next.end();
  
  background(255);
  image(next, 0, 0);
}

A sphere moving around with previous copies of itself going off to infinity behind it

Framebuffers are especially good for feedback because they can store images as higher precision floating-point numbers, which you can specify with createFramebuffer({ format: FLOAT }). Normally, the red, green, and blue values of colors are stored as integers between 0-255. Each time you draw an image, the colors get rounded into that range, and a source frame can get drawn and redrawn many times in a feedback sketch, accumulating lots of rounding errors. That problem goes away when you use floats!

Sketching with Depth

When you draw to a Framebuffer, in addition to recording the color of the pixels, you are also recording their depth in space, stored as a number between 0 and 1. You can read this into a shader by looking at the depth property of a Framebuffer.

This can be useful if you want to change an image based on how far from the camera things are. One common effect that uses this is fog, where farther away objects are tinted more and more towards a fog color.


// sketch.js
let layer, fogShader, fog;
function preload() {
  fogShader = loadShader('fog.vert', 'fog.frag');
}

function setup() {
  createCanvas(200, 200, WEBGL);
  layer = createFramebuffer();
  fog = color('#b2bdcf');
  noStroke();
}

function draw() {
  // Draw a scene to a framebuffer
  layer.begin();
  clear();
  lights();
  fill('red');
  for (let i = 0; i < 12; i++) {
    push()
    translate(
      sin(frameCount*0.050 + i*1)*50,
      sin(frameCount*0.051 + i*2)*50,
      sin(frameCount*0.049 + i*3)*175-75
    );
    sphere(10);
    pop();
  }
  layer.end();
  
  // Apply fog to the scene
  shader(fogShader);
  fogShader.setUniform('fog', [red(fog), green(fog), blue(fog)]);
  fogShader.setUniform('img', layer.color);
  fogShader.setUniform('depth', layer.depth);
  rect(0, 0, width, height);
}


// fog.vert
precision highp float;
attribute vec3 aPosition;
attribute vec2 aTexCoord;
varying vec2 vTexCoord;
void main() {
  vec4 positionVec4 = vec4(aPosition, 1.0);
  positionVec4.xy = positionVec4.xy * 2.0 - 1.0;
  gl_Position = positionVec4;
  vTexCoord = aTexCoord;
}


// fog.frag
precision highp float;
varying vec2 vTexCoord;
uniform sampler2D img;
uniform sampler2D depth;
uniform vec3 fog;
void main() {
  gl_FragColor = mix(
    // Original color
    texture2D(img, vTexCoord),
    // Fog color
    vec4(fog/255., 1.),
    // Mix between them based on the depth.
    // The pow() makes the light falloff a bit steeper.
    pow(texture2D(depth, vTexCoord).r, 6.)
  );
}

A number of red spheres moving around in a foggy space, fading to grey as they get further away

If you want to customize how close or far to the camera an object needs to be to get depth values of 0 or 1, specify near and far values in perspective().

Conclusion

If you're sketching in WebGL mode and need to draw to an image, consider using createFramebuffer() as a way to make your sketch run fast and give everyone the best viewing experience possible.