Memory limits less than expected?

[bzSteve]

I'm running LÖVE 0.9.1 in OS X 10.10.2 on a Mac with 32gb RAM. Over 28gb of RAM are available to the app at run time.

My game is trying to load a lot of images (the .pngs are about 250mb on disk, not sure what that amounts to unpacked). After loading about two-thirds of the images, the program aborts without an error message. I've traced the line of code to where the image returned by love.graphics.newImage is stored in an array.

This got me thinking about memory limitations in LÖVE and a recent thread (viewtopic.php?f=3&t=79387) so I wrote a simple test:

Code: Select all

local mem = {}

for i = 1, 1000000000 do
	mem[i] = 1
	
	if i%1000000 == 0 then print (i) end
end

When I run this as my main.lua in LÖVE, it crashes after 67,108,864. When I run it in Lua 5.2, it finishes without error although there are some very short pauses while it runs.

Is this a limitation with LÖVE, LuaJIT2, or is it a bug?

[Doctory]

probably a limitation with luajit

[slime]

LuaJIT is limited to about 2GB of memory that it can allocate itself. That limit doesn't include images, which are created by LÖVE's C++ code.

Images use 4 * width * height bytes in memory, plus around 4 * width * height bytes in VRAM (e.g. a single 512x512 image uses 1MB of RAM and 1MB of VRAM.) If you use a compressed texture format such as DXT5, the image won't have to be decompressed at all in RAM or VRAM (so a single 512x512 DXT5 compressed texture will use 256KB in RAM and VRAM.)

[Azhukar]

How long does it take to load those images?

I'm thinking it terminates your app as not responding due to long load.

[slime]

OS X doesn't terminate unresponsive applications on its own.

[bizziboi]

As slime said - LuaJit is limited to addressing 2 gig (it uses the high bit for other purposes). You array example crashes because you're crossing that boundary. (one array entry is 4 bytes (assuming love2d is compiled to use floats, not doubles) so you would never hit 1 billion entries (500.000.000 at best if there were zero overhead).

Why you only reach 67 million - my guess would be that there's no garbage collection being called and this creates a ton of garbage resising the array). You could see if you can create more entries if you add a call to collectgarbage in that loop every once in a while (just for educational purposes).

But 2G lua memory is a hard limit that indeed is not present in vanilla Lua.

[slime]

one array entry is 4 bytes (assuming love2d is compiled to use floats, not doubles)

LÖVE just bundles a LuaJIT dynamic library (on platforms where it's not provided by the system), which is compiled to use double-precision floating point numbers like normal. Even if LuaJIT were compiled to use 32 bit floats rather than 64 bit floats each unique Lua value would still use at least 64 bits though.

[bizziboi]

Even if LuaJIT were compiled to use 32 bit floats rather than 64 bit floats each unique value would still use at least 64 bits though.

Cheers, I assumed floats in the array part were stored as floats. TIL.

[bzSteve]

This is helpful information everyone. Thank you.

I'll have to read up on DXT5 textures, slime. That's not something I'm familiar with.

The images load in just a few seconds. For my purposes, that's not a problem. The key is to make the game as quick as possible once running.

bizzibol, I think that you're right about garbage collection (I'll test that later today) because the Lua 5.2 test did sort of hiccup a few times while it was running. I'm guessing those pauses were GC.