bitser: faster, harder, better serialization with LuaJIT

Showcase your libraries, tools and other projects that help your fellow love users.
User avatar
Robin
The Omniscient
Posts: 6506
Joined: Fri Feb 20, 2009 4:29 pm
Location: The Netherlands
Contact:

bitser: faster, harder, better serialization with LuaJIT

Post by Robin »

Oops I did it again
I played with serialization, got lost in the game
Oh baby, baby

bitser is the ultimate serialization library for LÖVE:
  • Uses LuaJIT to be superfast and superefficient
  • Just like Lady and binser, it supports registering resources (like Image, Source, anything) and classes (MiddleClass, SECL, hump.class, Slither)
  • It's secure: you can use it to serialize over teh interwebs
  • LÖVE bonus: you can load serialized values from FileData objects as well as from strings, which means loading gets even faster
  • Seriously, try it out, if you run the repo with LÖVE you get a benchmark with nice charts that you can compare if you download other serialization libraries to it.
Help us help you: attach a .love.
User avatar
qaisjp
Party member
Posts: 491
Joined: Tue Sep 04, 2012 10:49 am
Location: United Kingdom
Contact:

Re: bitser: faster, harder, better serialization with LuaJIT

Post by qaisjp »

Nice work! What kind of format does it serialize to?
Lua is not an acronym.
User avatar
Robin
The Omniscient
Posts: 6506
Joined: Fri Feb 20, 2009 4:29 pm
Location: The Netherlands
Contact:

Re: bitser: faster, harder, better serialization with LuaJIT

Post by Robin »

It uses a custom binary format that is optimized for size and writing time.

Here's an overview of each initial byte:

Code: Select all

0 * * * * * * * -> small int
1 0 * * * * * * -> small ref
1 1 0 * * * * * -> small string    [data:byte * rest]
1 1 1 0 * * * * -> small res       [label:byte * rest]
1 1 1 1 0 0 0 0 -> table           [len:number] [value:any] * len [keylen:number] ([key:any] [value:any]) * keylen
1 1 1 1 0 0 0 1 -> resource        [label:string]
1 1 1 1 0 0 1 0 -> instance        [class:string] [data:table]
1 1 1 1 0 0 1 1 -> reference       [idx:number]
1 1 1 1 0 1 0 0 -> string          [len:number] [data:byte * len]
1 1 1 1 0 1 0 1 -> long            [data:byte * 4]
1 1 1 1 0 1 1 0 -> float           [data:byte * 8]
1 1 1 1 0 1 1 1 -> nil
1 1 1 1 1 0 0 0 -> false
1 1 1 1 1 0 0 1 -> true
1 1 1 1 1 0 1 0 -> short           [data:byte * 2]
1 1 1 1 1 0 1 1 -> N/A
1 1 1 1 1 1 0 0 -> N/A
1 1 1 1 1 1 0 1 -> N/A
1 1 1 1 1 1 1 0 -> N/A
1 1 1 1 1 1 1 1 -> N/A
Type bytes 251-255 are reserved for future extension.
Help us help you: attach a .love.
prixt
Prole
Posts: 26
Joined: Sat Sep 12, 2015 5:53 am

Re: bitser: faster, harder, better serialization with LuaJIT

Post by prixt »

Is it possible to register custom classes that don't use libraries you mentioned? I prefer to create classes on a case by case basis, so don't use outside libraries.
User avatar
pgimeno
Party member
Posts: 2602
Joined: Sun Oct 18, 2015 2:58 pm

Re: bitser: faster, harder, better serialization with LuaJIT

Post by pgimeno »

Robin wrote:
  • It's secure: you can use it to serialize over teh interwebs
Yay! Security is the very reason I've used a JSON library instead of any of the other serializers out there, in my only project so far (which didn't use self-referential tables or any other stuff unsupported by JSON). Each and every serializer I've found that was able to save native Lua data in textual form, loaded the data through loadstring.

The fact that it outputs to binary only is a problem for me to adopt it, though. I'm not a big fan of obscurity in savegames in most situations. One should not need to use a savegame editor just to make a few changes.

However, I like the idea for complex games where there's a lot of data and storing it in a compact way is best.
monolifed
Party member
Posts: 156
Joined: Sat Feb 06, 2016 9:42 pm

Re: bitser: faster, harder, better serialization with LuaJIT

Post by monolifed »

does small int work like variable int in protobuf (uses less bytes for small integers)
User avatar
airstruck
Party member
Posts: 650
Joined: Thu Jun 04, 2015 7:11 pm
Location: Not being time thief.

Re: bitser: faster, harder, better serialization with LuaJIT

Post by airstruck »

pgimeno wrote:Each and every serializer I've found that was able to save native Lua data in textual form, loaded the data through loadstring.
You could use setfenv with loadstring, it should be reasonably secure.

Code: Select all

local f = loadstring(serialized)
setfenv(f, {})
local deserialized = f() 
I don't think it could do any real damage sandboxed like that, although it could try to be annoying by hanging the game or something (someone please correct me if I'm wrong).
User avatar
Robin
The Omniscient
Posts: 6506
Joined: Fri Feb 20, 2009 4:29 pm
Location: The Netherlands
Contact:

Re: bitser: faster, harder, better serialization with LuaJIT

Post by Robin »

prixt wrote:Is it possible to register custom classes that don't use libraries you mentioned? I prefer to create classes on a case by case basis, so don't use outside libraries.
It depends on how exactly you use it. Bitser has to heuristically detect how the class works and how to construct instances when loading. If the class is the metatable of instances, for example, it would work no problem, because bitser would think you're using hump.class. When registering a class you can also give a classkey and deserializer:

Code: Select all

bitser.registerClass('YourClass', YourClass, 'classkey', function(instance, class) return --[[make instance an instance of class]] end)
pgimeno wrote:The fact that it outputs to binary only is a problem for me to adopt it, though. I'm not a big fan of obscurity in savegames in most situations. One should not need to use a savegame editor just to make a few changes.
That's a good point. That's maybe a good idea for another library, that maximizes readability at the expense of size, speed and security. Originally I wrote Lady for that, but Lady is not very readable, making hand-editing it less fun.
pgimeno wrote:However, I like the idea for complex games where there's a lot of data and storing it in a compact way is best.
Yeah, that's more the intended use case. ;)
ingsoc451 wrote:does small int work like variable int in protobuf (uses less bytes for small integers)
Yes. Integers from -27 to 100 are stored in the type byte.
Help us help you: attach a .love.
User avatar
bakpakin
Party member
Posts: 114
Joined: Sun Mar 15, 2015 9:29 am
Location: Boston

Re: bitser: faster, harder, better serialization with LuaJIT

Post by bakpakin »

Wow. That is quality.

For luajit needs, binser might be obsolete. All that work I put into making sure numbers serialized correctly, and you go and just use ffi to copy memory.

The links for other libraries for the benchmark are broken, though. Replacing "github.com" in the urls with "raw.githubusercontent.com" should do the trick, I think.

Great work. This looks like it has applications outside of LOVE. I might use this with luvit.
((_((_CRAYOLA_((_((_> GitHub <_((_((_CRAYOLA_((_(()
User avatar
Robin
The Omniscient
Posts: 6506
Joined: Fri Feb 20, 2009 4:29 pm
Location: The Netherlands
Contact:

Re: bitser: faster, harder, better serialization with LuaJIT

Post by Robin »

Thanks!

Huh, the links worked for me. Maybe I should replace them then.
Help us help you: attach a .love.
Post Reply

Who is online

Users browsing this forum: No registered users and 14 guests