Page 3 of 3
Re: Blob.lua - binary serialization library
Posted: Fri Feb 12, 2021 8:29 pm
by grump
Xii wrote: ↑Fri Feb 12, 2021 4:54 pm
Can I pass the ctype information to the library somehow? I have the string definition, after all.
Not currently supported. My current plan for transparent cdata support would require user code for all serialized ctypes:
Code: Select all
local ffi = require('ffi')
ffi.cdef('typedef struct { double x, y; } teststruct_t')
local ctype = ffi.typeof('teststruct_t')
ffi.metatype(ctype, {
__index = {
__typename = 'teststruct_t',
__serialize = function(self, writer)
return writer:number(self.x):number(self.y)
end,
__deserialize = function(ctype, reader)
return ctype(reader:number(), reader:number())
end,
}
})
This solution still has some drawbacks: anonymous struct types are not supported (since they have no type name), and custom code would be required for all ctypes. ctypes from libraries with a metatype of their own, e.g. brinevector, wouldn't work.
I can implement support for this, but it will take a while.
Edit: cdata serialization as described above is now supported.
Example code.
The __serialize() and __deserialize() functions are only required for complex structs with pointers, and for cases when ffi.sizeof() is not sufficient to get the true size of the data.
The __typename field is required, except when calling :cdata() manually with a type name, in which case the metatype is not required at all (e.g. when using brinevector).
Code: Select all
-- brinevector example (no __typename, __serialize, or __deserialize are defined in brinevector}
-- this works
writer:cdata('brinevector', vector)
...
vector = reader:cdata()
-- but this is not possible, because no type information is available (and metatypes are immutable)
writer:write(vector)
-- or
writer:table({ vector }) -- ditto
Re: Blob.lua - binary serialization library
Posted: Sat Feb 13, 2021 3:36 pm
by Xii
And here I was thinking I had been rudely demanding. You really delivered
grump wrote: ↑Fri Feb 12, 2021 8:29 pm
The __serialize() and __deserialize() functions are only required for complex structs with pointers,
So for plain old data structs, just need __typename ?
grump wrote: ↑Fri Feb 12, 2021 8:29 pm
and for cases when ffi.sizeof() is not sufficient to get the true size of the data.
As I were experimenting with this before, I noticed that ffi.sizeof() returns a
padded size for my test struct. Will that be an issue?
Re: Blob.lua - binary serialization library
Posted: Sat Feb 13, 2021 4:09 pm
by Xii
Error
BlobWriter.lua:141: bad argument #2 to 'copy' (cannot convert 'table' to 'const void *')
Traceback
[C]: in function 'copy'
BlobWriter.lua:141: in function 'string'
BlobWriter.lua:156: in function '_writeTaggedPair'
BlobWriter.lua:323: in function 'write'
main.lua:31: in function 'load'
[C]: in function 'xpcall'
[C]: in function 'xpcall'
Code: Select all
local ffi = require("ffi")
local writer, reader = require("BlobWriter"), require("BlobReader")
ffi.cdef[[
typedef struct
{
uint8_t one;
uint16_t two;
uint32_t four;
uint64_t eight;
}
test;
]]
local meta =
{
__index =
{
__typename = 'test',
}
}
actual = ffi.metatype(ffi.typeof('test'), meta)
local data = {}
data.yksi = {1, 2, 3, 4}
data.kaksi = actual()
local blob = writer()
blob:write(data)
Re: Blob.lua - binary serialization library
Posted: Sat Feb 13, 2021 4:40 pm
by grump
Xii wrote: ↑Sat Feb 13, 2021 3:36 pm
So for plain old data structs, just need __typename ?
Adding a __typename is sufficient for all data that is not a pointer and does not contain pointers. For stuff with pointers you need custom __serialize and __deserialize functions to write/reconstruct the data manually. Otherwise only the pointer's addresses would be serialized.
As I were experimenting with this before, I noticed that ffi.sizeof() returns a padded size for my test struct. Will that be an issue?
It can be an issue if the serialized data needs to be portable between different architectures and machine word sizes. In cases where portability is required, using #pragma pack() in the cdef call makes sure consistent padding is applied.
The bug you've encountered is now fixed. I forgot to add a test case for cdata in hash tables and
of course there was a small mistake in that code.
Re: Blob.lua - binary serialization library
Posted: Sat Feb 13, 2021 8:47 pm
by Xii
Awesome! And beautiful! Look at how simple and succint this is thanks to your diligent efforts:
Code: Select all
local ffi = require("ffi")
local writer, reader = require("BlobWriter"), require("BlobReader")
function define_struct(name, memberstr)
ffi.cdef("#pragma pack(1)\ntypedef struct{"..memberstr.."}"..name..";")
local meta =
{
__index =
{
__typename = name,
}
}
return ffi.metatype(ffi.typeof(name), meta)
end
actual = define_struct("test",
[[
uint8_t one;
uint8_t oneb;
uint8_t onec;
uint16_t two;
uint32_t four;
uint64_t eight;
]])
local data = {}
data.foo = {1, 2, 3, 4}
data.bar = actual()
local blob = writer()
blob:write(data)
assert(love.filesystem.write("test", blob:tostring()))
Don't forget to bump your copyright up to 2021
Re: Blob.lua - binary serialization library
Posted: Thu Dec 02, 2021 4:37 pm
by yal2du
Is this library still available (somewhere)? The url github.com/megagrump/blob goes to 404.
Re: Blob.lua - binary serialization library
Posted: Thu Dec 02, 2021 4:48 pm
by GVovkiv
yal2du wrote: ↑Thu Dec 02, 2021 4:37 pm
Is this library still available (somewhere)? The url github.com/megagrump/blob goes to 404.
https://github.com/megagrump/moonblob ?
Re: Blob.lua - binary serialization library
Posted: Thu Dec 02, 2021 5:02 pm
by yal2du
Yes, thanks.
I presumed it was moonscript but the lua source is there as well.
Re: Blob.lua - binary serialization library
Posted: Thu Dec 02, 2021 5:08 pm
by grump
Yeah it was originally written in vanilla Lua and later rewritten in moonscript and renamed. The Lua code in the repo is there for convenience so you don't have to compile it yourself.
I've changed the link in the OP, thanks for the hint.