Saving & Loading large files

[BOT-Brad]

Currently been testing with a few different serialisation libraries such as Serpent, tserialize and Ser. I'm trying to save a fairly large amount of data, of which it has been mostly cleaned up and any redundant data removed.

Right now, Ser runs the fastest by a comfortable distance, but produces slightly larger file sizes than Serpent and tserialize. Saving the game when around 200 years into the future in my football game currently leaves save file sizes of 12Mb in Ser, 10Mb in tserialize and 9Mb in Serpent. Ser takes around 2 seconds to run, tserialize around 3, but Serpent takes a good 15-20 seconds which is too long.

However, currently using Ser, I notice that when I try and load my save file back in, if the file size is above roughly 11,000kB, then my load_game() function fails.

The function

Code: Select all

function database.load_game()
	local data, size = love.filesystem.read("save/savedata1.data")
	print(data~=nil, size)
	data = loadstring(data)()
	assert(data, "Data could not be loaded!")
	database.team_list = data.team_list
	database.league_list = data.league_list
	database.nation_list = data.nation_list
	database.vars = data.vars
	--
	database.build_dict()
	--
	database.process()
	--
	g.notification:new("Game Loaded!\nData Size: " .. math.floor(size/1024) .. "kB")
end

The bit of code which fails

Code: Select all

data = loadstring(data)()

Code: Select all

attempt to call a nil value

It seems to me that loadstring(data) is producing a big fat "nil" value. Any ideas of ways around this? Possibly splitting up the save file?

EDIT: Just to note, this code runs DANDY and quite quickly if the filesize is below that 12,000kb mark!

[BOT-Brad]

I really should google a bit better first, huh?

So turns out loadstring returns a 2nd value that details the error message if things go wrong. Turns out I get this error;

Code: Select all

[string "local _ = {}..."]:110018: main function has more than 65536 constants

More than 65,536 constants! So maybe I should try splitting up my save file into multiple files or something, huh?

NOTE: Turns out it may/may not be some sort of bug -> URL LINKY.

[Guard13007]

Can I get a look at your data? I'm curious about this myself, especially since I just started a project with potentially unbounded amounts of data to save as well.

At anyone who knows more about LOVE's development: It looks like this was a bug that has since been fixed in LuaJIT, will LOVE update to the version that has fixed this? (Or have I misunderstood the issue? / Are there reasons why it might not be so simple to fix in the future?)

[Robin]

Hey BOT-Brad, is it only a problem with Ser? Or do the other serializers have this problem as well?

If the former, then I'd really like to take a look at your data. I try to make Ser as good as possible, so I'd like to see if there's something I can do about it.

[BOT-Brad]

So attached is the saved data produced by using Ser (savedata.ser) and Serpent (savedata.serpent).

Both saves were 'captured' at the exact same time (Well, right after one another), so are save-files of the exact same data. You can see that Ser is around ~11.3Mb in size, whereas Serpent is ~8.2Mb. I didn't time how long they took to save, but Ser was at least 3x faster. I'm willing to take that added speed-boost for slightly larger files at this point.

Using loadstring(data) with either save-file causes the aforementioned LuaJIT >65,536 constant error, causing LOVE to exit with an error.

Robin
It happens in Serpent as well (I couldn't get tserialize working again with my slightly changed data system), so definitely just an annoyance in LuaJIT it seems. Ser is great!

View savedata files at your own risk! You have been warned. I take no responsibility for loss of sanity, dismemberment, or even DEATH!

[Robin]

Ah, yes, I see the issue. It is unavoidable to have so many constants there, because there's just so many tables that all need to be numbered. I hope it is a fixed bug in LuaJIT, Lua 5.2 can load it just fine.

BTW, I could strip all extra whitespaces from Ser output, it saves some space, getting it to 8.6MB. One of the reasons Ser is both so time-efficient and so space-inefficient is that it gives every table a name (as you can see), which means a few wasted bytes per table, but also it doesn't have to do anything complicated to account for cycles. Apparently it's the whitespace. Imma do that right away. Who needs whitespace, right guys?

[slime]

Ah, yes, I see the issue. It is unavoidable to have so many constants there, because there's just so many tables that all need to be numbered. I hope it is a fixed bug in LuaJIT, Lua 5.2 can load it just fine.

Could it serialize tables like {{...}, {...}, foo="bar", [-1000]="baz"} rather than {[1]={...}, [2]={...}, foo="bar", [-1000]="baz"} ? (I assume that might need an additional processing step, so it would probably be a tradeoff – although it would also shrink the output size a bit.)

[s-ol]

Ah, yes, I see the issue. It is unavoidable to have so many constants there, because there's just so many tables that all need to be numbered. I hope it is a fixed bug in LuaJIT, Lua 5.2 can load it just fine.

Could it serialize tables like {{...}, {...}, foo="bar", [-1000]="baz"} rather than {[1]={...}, [2]={...}, [3]=etc, foo="bar", [-1000]="baz"} ? (I assume that might need an additional processing step, so it would probably be a tradeoff – although it would also shrink the output size a bit.)

It shouldn't bee too hard though should it?

Code: Select all

if type(x) == "table" then
  local max
  for i,v in ipairs(x) do
    -- append "v, "
    max = 1
  end
  for k,v in pairs(x) do
    if not max or type(k) ~= "number" or k > max then
      -- append "[k] = v, "
    end
  end
end

[Robin]

Could it serialize tables like {{...}, {...}, foo="bar", [-1000]="baz"} rather than {[1]={...}, [2]={...}, foo="bar", [-1000]="baz"} ?

That's not what I meant (and already what it does anyway). Ser works like this:

Code: Select all

> return serialise({{10}, {20}, {30}, foo='bar'})
local _={}
_[3]={30}
_[2]={20}
_[1]={10}
return {_[1],_[2],_[3],foo="bar"}

In this case it could have been serialised as

Code: Select all

return {{10},{20},{30},foo="bar"}

But you can't know that yet when you're at the outermost table. Maybe there's a cycle there somewhere, or a table that has multiple parents.

[BOT-Brad]

Ah, yes, I see the issue. It is unavoidable to have so many constants there, because there's just so many tables that all need to be numbered. I hope it is a fixed bug in LuaJIT, Lua 5.2 can load it just fine.

BTW, I could strip all extra whitespaces from Ser output, it saves some space, getting it to 8.6MB. One of the reasons Ser is both so time-efficient and so space-inefficient is that it gives every table a name (as you can see), which means a few wasted bytes per table, but also it doesn't have to do anything complicated to account for cycles. Apparently it's the whitespace. Imma do that right away. Who needs whitespace, right guys?

Nice, stripping the whitespace reduced a 5.6Mb save I had to around 4Mb, a decent saving for sure.

Still, has anyone got a clever way of avoiding this 65,536 constant issue? Now that my game saves a bit more data and the fact it will need to save even more, I am starting to worry about this!