[SOLVED] Fast JSON lib for big files ?

[Lapin]

Greetings, this is my first post on the Love 2D forums.

I've been coding GLua for 4 years, i started making my first Love2D project 1.5 month ago.
Here is the problem :

The game i'm working on use saves ... big saves (300kb-600kb), and the JSON lib i found is sslloooowwwwww as hell.
I tried about 4 JSON libs, all of them takes about 7 secs to load a 300kb file.

Here is the save http://puu.sh/gHZXA/5b409c692b.txt

And here is the code writing the save and deploying the save (merging with _G )

Code: Select all

function do_save_data()
	local tbl = {}
	local str;
		tbl.T_DAY = T_DAY
		tbl.T_SEM = T_SEM
		tbl.T_MONTH = T_MONTH
		tbl.T_YEAR = T_YEAR
		tbl.annualPayment = annualPayment
		tbl.Passed3mLimit = Passed3mLimit
		tbl.Money = Money
		tbl.Pub = Pub
		tbl.PassedEvents = PassedEvents
		tbl.DynaEvent = DynaEvent
		tbl.nbEmployees = nbEmployees
		tbl.nbInvestisseurs = nbInvestisseurs
		--tbl.tabInvestisseurs = tabInvestisseurs
		tbl.Popularity = Popularity
		tbl.Salary = Salary
		tbl.mainRate = mainRate
		tbl.Popularity = Popularity
		tbl.nbClients = nbClients
		tbl.PastMonthEvent = PastMonthEvent
		tbl.minimalInvestment = minimalInvestment
		tbl.middle1Investment = middle1Investment
		tbl.middle2Investment = middle2Investment
		tbl.maximalInvestment = maximalInvestment
		tbl.minimalTimeInvestment = minimalTimeInvestment
		tbl.middle1TimeInvestment = middle1TimeInvestment
		tbl.middle2TimeInvestment = middle2TimeInvestment
		tbl.maximalTimeInvestment = maximalTimeInvestment
		tbl.minimalRendement = minimalRendement
		tbl.middleRendement = middleRendement
		tbl.maximalRendement = maximalRendement
		tbl.minimal2Rendement = minimal2Rendement
		tbl.middle2Rendement = middle2Rendement
		tbl.maximal2Rendement = maximal2Rendement
		str = json.encode(tbl)
		love.filesystem.write("data.txt", str)
end


function do_restore()
	local str = love.filesystem.read("data.txt")
	local tbl = json.decode(str)
	table.Merge(_G, tbl)
	hook.Call("SaveRestored")
end

Here is the project ran under a profiler




I'm using JSON4LUA, available here -> https://github.com/craigmj/json4lua/blo ... n/json.lua


So what can i do ? Use CJSON ? (How can i find a precompiled lib, there is no doc about the vitual func if we wanna build a lib)
Compress the data before ? (how ?)


Regards,

-Lapin

[arampl]

Maybe just strip all whitespaces before save and turn off this "decode_scanWhitespace" function in json library you've found?
And if you really need spaces in your data just encode them as something unique (@@@ for example) and convert them back after loading or before using.

[szensk]

Why do you have to use JSON? Try serializing as Lua or msgpack. luamsgpack has the advantage of a very fast decoder, much faster than any pure Lua json library that I tried. Or by using Lua, you get a fast C parser with no extra dependencies.

Secondly, regardless of encoding, the data scheme of DynaEvent is questionable:
1) Don't store the numbers as strings.
2) Why are you storing "Money = Money - 31914"? Why not just store "func":31914? You can infer the `money = money -` part later.
3) Don't store the keys unless you must. Store them as arrays, in order. You're always storing date first and func second. It should be an array of arrays like so: [[421862, 25815.6], [351862,10942.2], ... ] Or perhaps more preferable ,as it retains keys but only once, something like this { "dates": [421862, 351862, ... ], "funcs": [25815.6, 10942.2, ...] }.

Applying all the ideas above, the file is reduced to 152kb and parses much faster. Using redundant keys (~230kb) and Lua, it parses in almost no time at all (<20ms).

[Lapin]

Thanks for the answers,

About the "Money = Money - 31914" :

I build two event system, one with preset events (will run some function at DD/MM/YY (in game time))
The second one is list the first but the function are not preset (Like : i bought something, let's reduce the cash after x days) , it means we have to save them if the player exit the game.


Thanks for the storing advices, i'll try "luamsgpack" ASAP.



EDIT : I had some lua errors under luamsgpack, so i just used some Gmod libs (table.ToString)

1.5mb -> about 0.5 sec to load


Well, storing data as JSON was not a good idea :v

Thanks again.

[szensk]

Glad it worked out for you. I've been rallying against using JSON in the Lua community for a while now. There's no reason* to use JSON unless you're dealing with a web API or program that uses JSON.

Also, if for some reason you must return to JSON, dkjson is about 2-3x faster than json4lua in my tests.

[Positive07]

Ser is as fast as you can get saving and loading data plus it is really simple

Code: Select all

local ser = require "ser" --Import Ser

local t = {your = "table", with = "data", 123912, 12+1, {"another", "table", 4}} --This is your data

local serialized = ser(t) --Serialize your data

love.filesystem.write("yoursavefile.lua",serialized) --Save it to a file

p = loadstring (serialized)() --Load from string

o = love.filesystem.load("yoursavefile.lua")() --Load from file

--t, p and o now hold all the same data

And since it is loaded as a lua chunk is REALLY fast! (Thanks Robin)