Pixel art antialiasing shader

[RNavega]

This is a pixel shader for use with modern pixel-art-style games that make use of smooth scrolling and scaling. It filters the edges of pixels, if needed, so they have an antialiased look. The size of the antialising is adjustable as a constant in the pixel shader in case you want it sharper or smoother, but the default value of 1.0 already leads to exact results.

All the code needed is in the main script:
https://github.com/RNavega/PixelArt-Ant ... r/main.lua

preview.gif (509.78 KiB) Viewed 13403 times

It also lets you optionally smooth the boundaries of images (for use with completely filled images like tiles etc.) without having to pad the images with redundant transparent pixels. It does this in the vertex shader, expanding the image quad, and again it's optional.

The algorithm in the shader was written by me and I'm licensing it as public domain.

Special thanks to these articles for the theory:
- https://colececil.io/blog/2017/scaling- ... roying-it/
- https://csantosbh.wordpress.com/2014/01 ... -in-webgl/
- https://medium.com/@michelotti.matthew/ ... 7a85d2dc43

[RNavega]

Someone on LÖVE's Discord channel suggested some tests.

If you draw your sprites scaled/rotated/subpixel-translated with 'linear' filtering onto a Canvas, and then draw that Canvas (scaled up or not) with 'nearest' filtering, the result you get looks blurry like with the standard 'linear' filter:



When using the shader, the benefit comes that you can scale/rotate/subpixel-translate your graphics with 'linear' filter and not get that blurry result:



So using the shader will not only look better but also avoid the extra memory and processing cost of drawing to an intermediary Canvas.

[zorg]

How about:
- Draw image to canvas with nearest filter, scaled up 14x, no rotation (nor translation)
- Draw canvas to screen with linear filter, rotated, translated (already using subpixels due to scaling)
Should have a similar, cleaner result. (Of course, doing it with a shader is more optimal w.r.t. memory usage)

[RNavega]

@zorg thanks for the suggestion, this was the closest test!
For a single static image the result is identical to when using the shader with a filter size of 1.0 (one pixel). The problem is when you add moving things, and pretty much all games would have those.

This is the exact same context (sprites, background, movements), rendered with the two methods. The footage was scaled 3x with nearest-neighbor so the pixels are easier to see:

1. https://www.dropbox.com/s/maj9dfujl3t2c ... .mp4?raw=1
   This is drawing the BG and sprites at 14x scale using nearest filter, to a canvas, then drawing that canvas w/ no scale but subpixel position/rotation and the linear filter so its contents are smoothed.
   The background (and anything else at integer coordinates or integer rotation like 180º, 90º or 45º) looks the same as in the shader, but anything with subpixel transforms don't. Also, the outer edges of the canvas are not antialiased as that's where its mesh ends.
   
   
2. https://www.dropbox.com/s/58r7q0904ip5x ... .mp4?raw=1
   This is using the shader. No canvas, everything is drawn directly at their subpixel transform.
   Changing the filter size to 1.5 pixel (a constant used in the shader code) led to an even silkier result.

This solidifies my trust in this. Filtering UVs on the shader leads to a better result when you need those subpixel transforms. If you only use integer coordinates and want the whole viewport to subpixel pan/zoom/rotate, then you can use a Canvas like you said.

This shader can also be used with 3D meshes with pixel art textures (and in other game engines of course, you just need to port it), this way you don't need to rely on hardware support for MSAA.
You'd think mipmaps would help with this, and they work fine for bi/tri-linear filtered textures, but they don't work well with nearest filtered textures on meshes (example), so a shader like this is essential for this aesthetic.

Edit: backup links to those videos, make sure to play at the highest quality to avoid compression.
A) https://drive.google.com/file/d/1XU8vk2 ... sp=sharing
B) https://drive.google.com/file/d/1a0en7S ... sp=sharing

[Davidobot]

Damn, those videos are crisp. I was just about to say that you can get similar results by cranking up MSAA normally, but then you go out and say that this allows for the same effect without hardware support. Solid!

[zorg]

MSAA won't do anything to color transitions -inside- a shape, only the edges of it get affected, as far as i remember.