## bitser: faster, harder, better serialization with LuaJIT

Robin
The Omniscient
Posts: 6506
Joined: Fri Feb 20, 2009 4:29 pm
Location: The Netherlands
Contact:

### bitser: faster, harder, better serialization with LuaJIT

Oops I did it again
I played with serialization, got lost in the game
Oh baby, baby

bitser is the ultimate serialization library for LÖVE:
• Uses LuaJIT to be superfast and superefficient
• Just like Lady and binser, it supports registering resources (like Image, Source, anything) and classes (MiddleClass, SECL, hump.class, Slither)
• It's secure: you can use it to serialize over teh interwebs
• LÖVE bonus: you can load serialized values from FileData objects as well as from strings, which means loading gets even faster
• Seriously, try it out, if you run the repo with LÖVE you get a benchmark with nice charts that you can compare if you download other serialization libraries to it.
qaisjp
Party member
Posts: 491
Joined: Tue Sep 04, 2012 10:49 am
Location: United Kingdom
Contact:

### Re: bitser: faster, harder, better serialization with LuaJIT

Nice work! What kind of format does it serialize to?
Lua is not an acronym.
Robin
The Omniscient
Posts: 6506
Joined: Fri Feb 20, 2009 4:29 pm
Location: The Netherlands
Contact:

### Re: bitser: faster, harder, better serialization with LuaJIT

It uses a custom binary format that is optimized for size and writing time.

Here's an overview of each initial byte:

Code: Select all

0 * * * * * * * -> small int
1 0 * * * * * * -> small ref
1 1 0 * * * * * -> small string    [data:byte * rest]
1 1 1 0 * * * * -> small res       [label:byte * rest]
1 1 1 1 0 0 0 0 -> table           [len:number] [value:any] * len [keylen:number] ([key:any] [value:any]) * keylen
1 1 1 1 0 0 0 1 -> resource        [label:string]
1 1 1 1 0 0 1 0 -> instance        [class:string] [data:table]
1 1 1 1 0 0 1 1 -> reference       [idx:number]
1 1 1 1 0 1 0 0 -> string          [len:number] [data:byte * len]
1 1 1 1 0 1 0 1 -> long            [data:byte * 4]
1 1 1 1 0 1 1 0 -> float           [data:byte * 8]
1 1 1 1 0 1 1 1 -> nil
1 1 1 1 1 0 0 0 -> false
1 1 1 1 1 0 0 1 -> true
1 1 1 1 1 0 1 0 -> short           [data:byte * 2]
1 1 1 1 1 0 1 1 -> N/A
1 1 1 1 1 1 0 0 -> N/A
1 1 1 1 1 1 0 1 -> N/A
1 1 1 1 1 1 1 0 -> N/A
1 1 1 1 1 1 1 1 -> N/A
Type bytes 251-255 are reserved for future extension.
prixt
Prole
Posts: 26
Joined: Sat Sep 12, 2015 5:53 am

### Re: bitser: faster, harder, better serialization with LuaJIT

Is it possible to register custom classes that don't use libraries you mentioned? I prefer to create classes on a case by case basis, so don't use outside libraries.
pgimeno
Party member
Posts: 2602
Joined: Sun Oct 18, 2015 2:58 pm

### Re: bitser: faster, harder, better serialization with LuaJIT

Robin wrote:
• It's secure: you can use it to serialize over teh interwebs
Yay! Security is the very reason I've used a JSON library instead of any of the other serializers out there, in my only project so far (which didn't use self-referential tables or any other stuff unsupported by JSON). Each and every serializer I've found that was able to save native Lua data in textual form, loaded the data through loadstring.

The fact that it outputs to binary only is a problem for me to adopt it, though. I'm not a big fan of obscurity in savegames in most situations. One should not need to use a savegame editor just to make a few changes.

However, I like the idea for complex games where there's a lot of data and storing it in a compact way is best.
monolifed
Party member
Posts: 156
Joined: Sat Feb 06, 2016 9:42 pm

### Re: bitser: faster, harder, better serialization with LuaJIT

does small int work like variable int in protobuf (uses less bytes for small integers)
airstruck
Party member
Posts: 650
Joined: Thu Jun 04, 2015 7:11 pm
Location: Not being time thief.

### Re: bitser: faster, harder, better serialization with LuaJIT

pgimeno wrote:Each and every serializer I've found that was able to save native Lua data in textual form, loaded the data through loadstring.
You could use setfenv with loadstring, it should be reasonably secure.

Code: Select all

local f = loadstring(serialized)
setfenv(f, {})
local deserialized = f() 
I don't think it could do any real damage sandboxed like that, although it could try to be annoying by hanging the game or something (someone please correct me if I'm wrong).
Robin
The Omniscient
Posts: 6506
Joined: Fri Feb 20, 2009 4:29 pm
Location: The Netherlands
Contact:

### Re: bitser: faster, harder, better serialization with LuaJIT

prixt wrote:Is it possible to register custom classes that don't use libraries you mentioned? I prefer to create classes on a case by case basis, so don't use outside libraries.
It depends on how exactly you use it. Bitser has to heuristically detect how the class works and how to construct instances when loading. If the class is the metatable of instances, for example, it would work no problem, because bitser would think you're using hump.class. When registering a class you can also give a classkey and deserializer:

Code: Select all

bitser.registerClass('YourClass', YourClass, 'classkey', function(instance, class) return --[[make instance an instance of class]] end)
pgimeno wrote:The fact that it outputs to binary only is a problem for me to adopt it, though. I'm not a big fan of obscurity in savegames in most situations. One should not need to use a savegame editor just to make a few changes.
That's a good point. That's maybe a good idea for another library, that maximizes readability at the expense of size, speed and security. Originally I wrote Lady for that, but Lady is not very readable, making hand-editing it less fun.
pgimeno wrote:However, I like the idea for complex games where there's a lot of data and storing it in a compact way is best.
Yeah, that's more the intended use case.
ingsoc451 wrote:does small int work like variable int in protobuf (uses less bytes for small integers)
Yes. Integers from -27 to 100 are stored in the type byte.
bakpakin
Party member
Posts: 114
Joined: Sun Mar 15, 2015 9:29 am
Location: Boston

### Re: bitser: faster, harder, better serialization with LuaJIT

Wow. That is quality.

For luajit needs, binser might be obsolete. All that work I put into making sure numbers serialized correctly, and you go and just use ffi to copy memory.

The links for other libraries for the benchmark are broken, though. Replacing "github.com" in the urls with "raw.githubusercontent.com" should do the trick, I think.

Great work. This looks like it has applications outside of LOVE. I might use this with luvit.
((_((_CRAYOLA_((_((_> GitHub <_((_((_CRAYOLA_((_(()
Robin
The Omniscient
Posts: 6506
Joined: Fri Feb 20, 2009 4:29 pm
Location: The Netherlands
Contact:

### Re: bitser: faster, harder, better serialization with LuaJIT

Thanks!

Huh, the links worked for me. Maybe I should replace them then.