Creating Smooth ASCII Shader Animations with OGL

A comprehensive guide based on building a rotating 3D ASCII sphere with smooth gradients.

Overview

This guide covers creating high-quality ASCII shader effects using WebGL (OGL library), focusing on achieving smooth gradients and avoiding common pitfalls like banding and uneven lighting.

Prerequisites

• Node.js and npm
• Basic understanding of shaders (GLSL)
• React/Next.js (for the example)

Installation

npm install ogl resolve-lygia

Note: Only install resolve-lygia if you need Perlin noise or other procedural functions. For simple geometric shapes like spheres, it's not required.

Core Architecture

Two-Pass Rendering System

1. First Pass: Render the 3D scene/effect to a texture
2. Second Pass: Convert the texture to ASCII characters

This separation allows you to:

• Create any visual effect in the first pass
• Apply ASCII conversion as a post-process
• Maintain high quality before ASCII conversion

Key Learnings & Best Practices

1. Smooth Gradients - The Critical Challenge

Problem: ASCII art uses discrete characters, which naturally creates banding/stripes.

Solution: Use BOTH character selection AND color modulation:

// ❌ WRONG: Only select characters
col = col * character(n, p);
fragColor = vec4(col, 1.0);

// ✅ CORRECT: Mix foreground/background colors
float charMask = character(n, p);
vec3 foreground = vec3(gray);  // Use actual gray value
vec3 background = vec3(1.0);
vec3 finalColor = mix(foreground, background, charMask);
fragColor = vec4(finalColor, 1.0);

Why this works:

• Character selection provides texture/pattern
• Color modulation creates the smooth gradient
• Even within one character type, colors vary continuously

2. Lighting Range Management

Critical: Control your lighting range carefully to avoid extremes.

// Calculate lighting with wrap for soft terminator
float wrap = 0.6f;
float wrapDiff = max(0.0f, (diff + wrap) / (1.0f + wrap));

// Map to narrow range - avoid pure black/white
float shade = wrapDiff * 0.25f + 0.48f;

// Clamp to prevent extremes
shade = clamp(shade, 0.48f, 0.73f);

// Double smoothing for extra-smooth transitions
shade = smoothstep(0.48f, 0.73f, shade);
shade = mix(0.48f, 0.73f, shade);

Key parameters:

• Wrap value (0.5-0.7): Softens the terminator line on spheres
• Minimum (0.40-0.50): Prevents overly dark shadows
• Maximum (0.70-0.80): Prevents blown-out highlights
• Range width: Balance contrast vs smoothness

3. Block Size & Character Spacing

Critical ratios:

vec2 blockSize = vec2(10.0f);     // Size of each ASCII block
vec2 p = mod(pix / 5.0f, 2.0f);   // Character spacing (half of blockSize)

If characters disappear or appear as black stripes:

• Block size is too small (minimum ~8-10 pixels)
• Character spacing doesn't match block size
• Rule of thumb: character_spacing = blockSize / 2

4. Renderer Configuration

For ASCII effects with white backgrounds:

const renderer = new Renderer({
  dpr: 2,      // Higher = sharper (2-3 recommended)
  alpha: false // Disable transparency
});

gl.clearColor(1.0, 1.0, 1.0, 1.0); // White background

DPR considerations:

• DPR 1: Low quality, better performance
• DPR 2: Good quality (recommended)
• DPR 3: Very high quality, may impact performance

5. Character Selection Strategy

Fewer is better for smooth gradients:

// ❌ Too many transitions (22) creates more banding
if(gray > 0.31f) n = 12816;
if(gray > 0.33f) n = 65600;
// ... 20 more levels

// ✅ Optimal (8-10) relies on color for smoothness
if(gray > 0.35f) n = 65600;
if(gray > 0.40f) n = 163153;
if(gray > 0.45f) n = 15255086;
if(gray > 0.50f) n = 15255054;
if(gray > 0.55f) n = 13121101;
if(gray > 0.60f) n = 15252014;
if(gray > 0.65f) n = 31599627;
if(gray > 0.70f) n = 11512810;

Why: With color modulation, you need fewer character transitions. More transitions without color variation just creates more visible bands.

Common Pitfalls & Solutions

Pitfall 1: Template String Parsing Errors

Problem: Using backticks in shader comments breaks JavaScript template strings.

// ❌ BREAKS: Backtick in comment
if(gray > 0.1f)
  n = 12816;  // ` character

// ✅ WORKS: Remove problematic comments
if(gray > 0.1f)
  n = 12816;

Pitfall 2: Visible Terminator Line

Problem: Sharp boundary between light and dark on spheres.

Solution: Wrap lighting technique:

float wrap = 0.6f;  // Adjust 0.4-0.7 for taste
float wrapDiff = max(0.0f, (diff + wrap) / (1.0f + wrap));

This simulates subsurface scattering and ambient fill.

Pitfall 3: Uneven Distribution

Problem: Some areas too bright, others too dark.

Solution:

1. Compress the lighting range (e.g., 0.48-0.73)
2. Use smoothstep for better distribution
3. Add ambient light component

Pitfall 4: Pure White Spots

Problem: Highlights become pure white (background).

Solution: Lower the background threshold:

// Check if truly background, not just bright sphere
if(gray > 0.85f) {  // High threshold
  fragColor = vec4(1.0f, 1.0f, 1.0f, 1.0f);
} else {
  // Render ASCII with color modulation
}

Complete Example: Rotating ASCII Sphere

Sphere Shader (First Pass)

#version 300 es
precision highp float;

uniform float uTime;
uniform vec2 uResolution;
in vec2 vUv;
out vec4 fragColor;

void main() {
  vec2 uv = (vUv - 0.5f) * 2.0f;
  uv.x *= uResolution.x / uResolution.y;

  float dist = length(uv);
  vec3 col = vec3(1.0f);

  if(dist < 1.0f) {
    // Calculate 3D position on sphere
    float z = sqrt(1.0f - dist * dist);
    vec3 normal = normalize(vec3(uv.x, uv.y, z));

    // Rotate normal (Y-axis rotation)
    float angle = uTime * 0.3f;
    float c = cos(angle);
    float s = sin(angle);
    vec3 rotatedNormal = vec3(
      normal.x * c - normal.z * s,
      normal.y,
      normal.x * s + normal.z * c
    );

    // Wrap lighting for soft terminator
    vec3 lightDir = normalize(vec3(0.5f, 0.3f, 1.0f));
    float diff = dot(rotatedNormal, lightDir);
    float wrap = 0.6f;
    float wrapDiff = max(0.0f, (diff + wrap) / (1.0f + wrap));

    // Map to controlled range
    float shade = wrapDiff * 0.25f + 0.48f;
    shade = clamp(shade, 0.48f, 0.73f);
    shade = smoothstep(0.48f, 0.73f, shade);
    shade = mix(0.48f, 0.73f, shade);

    col = vec3(shade);
  }

  fragColor = vec4(col, 1.0f);
}

ASCII Shader (Second Pass)

#version 300 es
precision highp float;

uniform vec2 uResolution;
uniform sampler2D uTexture;
out vec4 fragColor;

float character(int n, vec2 p) {
  p = floor(p * vec2(-4.0f, 4.0f) + 2.5f);
  if(clamp(p.x, 0.0f, 4.0f) == p.x) {
    if(clamp(p.y, 0.0f, 4.0f) == p.y) {
      int a = int(round(p.x) + 5.0f * round(p.y));
      if(((n >> a) & 1) == 1)
        return 1.0f;
    }
  }
  return 0.0f;
}

void main() {
  vec2 pix = gl_FragCoord.xy;
  vec2 blockSize = vec2(10.0f);
  vec2 blockCoord = floor(pix / blockSize) * blockSize;
  vec3 col = texture(uTexture, blockCoord / uResolution.xy).rgb;

  float gray = 0.299f * col.r + 0.587f * col.g + 0.114f * col.b;

  // Select character based on density (8 levels)
  int n = 4096;
  if(gray > 0.35f) n = 65600;
  if(gray > 0.40f) n = 163153;
  if(gray > 0.45f) n = 15255086;
  if(gray > 0.50f) n = 15255054;
  if(gray > 0.55f) n = 13121101;
  if(gray > 0.60f) n = 15252014;
  if(gray > 0.65f) n = 31599627;
  if(gray > 0.70f) n = 11512810;

  vec2 p = mod(pix / 5.0f, 2.0f) - vec2(1.0f);

  if(gray > 0.85f) {
    // Pure white background
    fragColor = vec4(1.0f, 1.0f, 1.0f, 1.0f);
  } else {
    // Color modulation for smooth gradient
    float charMask = character(n, p);
    vec3 foreground = vec3(gray);
    vec3 background = vec3(1.0f);
    vec3 finalColor = mix(foreground, background, charMask);
    fragColor = vec4(finalColor, 1.0f);
  }
}

Optimization Tips

1. Use highp precision for sphere calculations
2. Keep DPR at 2-3 max for performance
3. Resize handler should throttle or debounce
4. Cancel animation frame in cleanup to prevent memory leaks

Tuning Parameters Reference

| Parameter | Range | Effect | |-----------|-------|---------| | Wrap lighting | 0.4-0.7 | Higher = softer terminator | | Min brightness | 0.40-0.50 | Higher = lighter shadows | | Max brightness | 0.70-0.80 | Higher = brighter highlights | | Block size | 8-12px | Larger = more visible characters | | DPR | 1-3 | Higher = sharper but slower | | Character levels | 6-10 | Fewer = smoother (with color) | | Rotation speed | 0.2-0.5 | Lower = slower rotation |

Troubleshooting

Stripes/Banding visible

• ✅ Add color modulation (foreground/background mix)
• ✅ Reduce number of character transitions
• ✅ Apply double smoothing (smoothstep + mix)

Characters not visible

• ✅ Increase block size (minimum 8-10px)
• ✅ Check character spacing = blockSize / 2
• ✅ Verify DPR isn't too low

Harsh terminator line

• ✅ Add wrap lighting
• ✅ Increase smoothing iterations
• ✅ Compress lighting range

Areas too dark/light

• ✅ Adjust min/max brightness clamps
• ✅ Check lighting range matches character thresholds
• ✅ Add more ambient light

Conclusion

The key to smooth ASCII animations is:

1. Separate concerns: 3D rendering vs ASCII conversion
2. Color modulation: Don't rely solely on character selection
3. Controlled ranges: Avoid extreme values
4. Proper sizing: Block size and spacing ratios matter
5. Smooth lighting: Wrap lighting and double smoothing

With these principles, you can create beautiful ASCII effects with smooth gradients and minimal banding artifacts.