Blob.lua - binary serialization library

Showcase your libraries, tools and other projects that help your fellow love users.
grump
Party member
Posts: 947
Joined: Sat Jul 22, 2017 7:43 pm

Re: Blob.lua - binary serialization library

Post by grump »

Xii wrote: Fri Feb 12, 2021 4:54 pm Can I pass the ctype information to the library somehow? I have the string definition, after all.
Not currently supported. My current plan for transparent cdata support would require user code for all serialized ctypes:

Code: Select all

local ffi = require('ffi')

ffi.cdef('typedef struct { double x, y; } teststruct_t')
local ctype = ffi.typeof('teststruct_t')

ffi.metatype(ctype, {
	__index = {
		__typename = 'teststruct_t',

		__serialize = function(self, writer)
			return writer:number(self.x):number(self.y)
		end,

		__deserialize = function(ctype, reader)
			return ctype(reader:number(), reader:number())
		end,
	}
})
This solution still has some drawbacks: anonymous struct types are not supported (since they have no type name), and custom code would be required for all ctypes. ctypes from libraries with a metatype of their own, e.g. brinevector, wouldn't work.

I can implement support for this, but it will take a while.

Edit: cdata serialization as described above is now supported. Example code.

The __serialize() and __deserialize() functions are only required for complex structs with pointers, and for cases when ffi.sizeof() is not sufficient to get the true size of the data.
The __typename field is required, except when calling :cdata() manually with a type name, in which case the metatype is not required at all (e.g. when using brinevector).

Code: Select all

-- brinevector example (no __typename, __serialize, or __deserialize are defined in brinevector}
-- this works
writer:cdata('brinevector', vector)
...
vector = reader:cdata()

-- but this is not possible, because no type information is available (and metatypes are immutable)
writer:write(vector)
-- or
writer:table({ vector }) -- ditto
User avatar
Xii
Party member
Posts: 137
Joined: Thu Aug 13, 2020 9:09 pm
Contact:

Re: Blob.lua - binary serialization library

Post by Xii »

And here I was thinking I had been rudely demanding. You really delivered :3
grump wrote: Fri Feb 12, 2021 8:29 pm The __serialize() and __deserialize() functions are only required for complex structs with pointers,
So for plain old data structs, just need __typename ?
grump wrote: Fri Feb 12, 2021 8:29 pm and for cases when ffi.sizeof() is not sufficient to get the true size of the data.
As I were experimenting with this before, I noticed that ffi.sizeof() returns a padded size for my test struct. Will that be an issue?
User avatar
Xii
Party member
Posts: 137
Joined: Thu Aug 13, 2020 9:09 pm
Contact:

Re: Blob.lua - binary serialization library

Post by Xii »

Error

BlobWriter.lua:141: bad argument #2 to 'copy' (cannot convert 'table' to 'const void *')

Traceback

[C]: in function 'copy'
BlobWriter.lua:141: in function 'string'
BlobWriter.lua:156: in function '_writeTaggedPair'
BlobWriter.lua:323: in function 'write'
main.lua:31: in function 'load'
[C]: in function 'xpcall'
[C]: in function 'xpcall'

Code: Select all

local ffi = require("ffi")
local writer, reader = require("BlobWriter"), require("BlobReader")

ffi.cdef[[
	typedef struct
	{
		uint8_t one;
		uint16_t two;
		uint32_t four;
		uint64_t eight;
	}
	test;
]]

local meta =
{
	__index =
	{
		__typename = 'test',
	}
}
actual = ffi.metatype(ffi.typeof('test'), meta)

local data = {}
data.yksi = {1, 2, 3, 4}
data.kaksi = actual()

local blob = writer()
blob:write(data)
grump
Party member
Posts: 947
Joined: Sat Jul 22, 2017 7:43 pm

Re: Blob.lua - binary serialization library

Post by grump »

Xii wrote: Sat Feb 13, 2021 3:36 pm So for plain old data structs, just need __typename ?
Adding a __typename is sufficient for all data that is not a pointer and does not contain pointers. For stuff with pointers you need custom __serialize and __deserialize functions to write/reconstruct the data manually. Otherwise only the pointer's addresses would be serialized.
As I were experimenting with this before, I noticed that ffi.sizeof() returns a padded size for my test struct. Will that be an issue?
It can be an issue if the serialized data needs to be portable between different architectures and machine word sizes. In cases where portability is required, using #pragma pack() in the cdef call makes sure consistent padding is applied.

The bug you've encountered is now fixed. I forgot to add a test case for cdata in hash tables and of course there was a small mistake in that code.
User avatar
Xii
Party member
Posts: 137
Joined: Thu Aug 13, 2020 9:09 pm
Contact:

Re: Blob.lua - binary serialization library

Post by Xii »

Awesome! And beautiful! Look at how simple and succint this is thanks to your diligent efforts:

Code: Select all

local ffi = require("ffi")
local writer, reader = require("BlobWriter"), require("BlobReader")

function define_struct(name, memberstr)
	ffi.cdef("#pragma pack(1)\ntypedef struct{"..memberstr.."}"..name..";")
	local meta =
	{
		__index =
		{
			__typename = name,
		}
	}
	return ffi.metatype(ffi.typeof(name), meta)
end

actual = define_struct("test",
[[
	uint8_t one;
	uint8_t oneb;
	uint8_t onec;
	uint16_t two;
	uint32_t four;
	uint64_t eight;
]])

local data = {}
data.foo = {1, 2, 3, 4}
data.bar = actual()

local blob = writer()
blob:write(data)
assert(love.filesystem.write("test", blob:tostring()))
Don't forget to bump your copyright up to 2021 :nyu:
yal2du
Prole
Posts: 42
Joined: Wed Oct 13, 2021 5:41 pm

Re: Blob.lua - binary serialization library

Post by yal2du »

Is this library still available (somewhere)? The url github.com/megagrump/blob goes to 404.
User avatar
GVovkiv
Party member
Posts: 668
Joined: Fri Jan 15, 2021 7:29 am

Re: Blob.lua - binary serialization library

Post by GVovkiv »

yal2du wrote: Thu Dec 02, 2021 4:37 pm Is this library still available (somewhere)? The url github.com/megagrump/blob goes to 404.
https://github.com/megagrump/moonblob ?
yal2du
Prole
Posts: 42
Joined: Wed Oct 13, 2021 5:41 pm

Re: Blob.lua - binary serialization library

Post by yal2du »

GVovkiv wrote: Thu Dec 02, 2021 4:48 pm
yal2du wrote: Thu Dec 02, 2021 4:37 pm Is this library still available (somewhere)? The url github.com/megagrump/blob goes to 404.
https://github.com/megagrump/moonblob ?
Yes, thanks. :oops: I presumed it was moonscript but the lua source is there as well.
grump
Party member
Posts: 947
Joined: Sat Jul 22, 2017 7:43 pm

Re: Blob.lua - binary serialization library

Post by grump »

Yeah it was originally written in vanilla Lua and later rewritten in moonscript and renamed. The Lua code in the repo is there for convenience so you don't have to compile it yourself.

I've changed the link in the OP, thanks for the hint.
Post Reply

Who is online

Users browsing this forum: Ahrefs [Bot] and 21 guests