Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cattrs to unstructure/structure and allow (de)serialization with json or similar #181

Merged
merged 47 commits into from
Dec 16, 2021

Conversation

anthrotype
Copy link
Member

@anthrotype anthrotype commented Nov 18, 2021

this allows to do things like

import orjson
from ufoLib2.converters import default_converter as c
from ufoLib2.objects import Font

font = Font.open("NotoSans-Regular.ufo")

# dumping a UFO to json
data = orjson.dumps(c.unstructure(font))
with open("notosans-regular.json", "wb") as f:
    f.write(data)

# or loading a UFO from json
with open("notosans-regular.json", "rb") as f:
    data = f.read()
font2 = c.structure(orjson.loads(data), Font)

assert font == font2

Still WIP, as I need to add tests. Added tests

@anthrotype anthrotype marked this pull request as draft November 18, 2021 19:29
this way we don't get an extra {"data": {"foo": ...}}, but just {"foo": ...}, as 'data' is the only attribute/key for DataStore anyway.

reuse bytes encodings logic in misc, move lib specific private methods to lib.py module to avoid circular import.
Check if converter already supports natively bytes, in which case avoid the encoding/decoding to base85 string
@anthrotype anthrotype marked this pull request as ready for review November 19, 2021 18:38
@anthrotype
Copy link
Member Author

hm gotta figure out py37 compat (next week)

@anthrotype
Copy link
Member Author

anthrotype commented Nov 23, 2021

@madig I think this is ready for another review pass, at your leisure. Thanks

…ut it

the cattr.preconf.orjson converter does stuff we don't need/use:

- it serializes bytes as base85 (but we use base64 anyway as more widely compatible)
- it coerces mapping keys to str (but all our mappings use str anyway)
- has handler for datetimes (but we don't use any)
- coerces sets to lists (but we use no sets)
@anthrotype
Copy link
Member Author

By the way, all this is generic and not tied to json or any specific serialization format. Here's how it'd work with msgpack for example. The only difference below is, instead of using ufoLib2's default_converter, we create a new GenConverter, register our custom structure/unstructure hooks, and allow_bytes=True (for msgpack can encode those natively, unlike the our default_converter which is geared toward json and we need to encode bytes as base64).

import io
import cattr
import ufoLib2
import ufoLib2.converters
import msgpack

conv = cattr.GenConverter(omit_if_default=True)
ufoLib2.converters.register_hooks(conv, allow_bytes=True)

font = ufoLib2.Font.open("...ufo")

buf = io.BytesIO()

msgpack.pack(conv.unstructure(font), buf)

buf.seek(0)

font2 = conv.structure(msgpack.unpack(buf), Font)

assert font == font2

@anthrotype anthrotype changed the title Add cattrs to unstructure/structure and allow to dump/load as/from json Add cattrs to unstructure/structure and allow (de)serialization with json or similar Nov 24, 2021
after write() the _reader is set to None, if one calls unlazify after that, they'd get an assertion error.
AFAIK, this is the most unused of all glyph attributes, it's also marked optional in the UFO spec. fontTools.ufoLib happily skips it when it is None. Having all Glyph objects carry an unused image object seems like a waste. If one wants to use images, one can set one.
Fixes two bugs that affect us:
- cattr.gen.{make_dict_structure_fn, make_dict_unstructure_fn} now resolve type annotations automatically when PEP 563 is used. (#169)
- Fix an issue generating structuring functions with renaming and _cattrs_forbid_extra_keys=True. (#190)
…mit_if_default' global option

A value equal to None is pratically equal to it not being there, so there's no point in serializing those even if Converter has omit_if_default=False.

Add tests for different omit_if_default values
@madig
Copy link
Collaborator

madig commented Dec 10, 2021

The LayerSet does not seem to get unstructured?

import json

import ufoLib2
import ufoLib2.converters

u = ufoLib2.Font.open("tests/data/MutatorSansBoldCondensed.ufo/")
d = ufoLib2.converters.default_converter.unstructure(u)

j = json.dumps(d)

TypeError: Object of type LayerSet is not JSON serializable

@anthrotype
Copy link
Member Author

The LayerSet does not seem to get unstructured?

that is very weird, the tests pass, and I added more that explicitly test json (de)serialization. It must be a local setup issue of yours.

@madig
Copy link
Collaborator

madig commented Dec 15, 2021

It was a fluke, sorry for the noise.

Something different: How should other libraries handle serialized dumps? Consider e.g. intermediate build steps in a pipeline, where currently a Designspace might reference normal UFOs but should reference JSON dumps instead. Should e.g. designspaceLib be made aware of the new format or should this rather be done elsewhere?

@anthrotype
Copy link
Member Author

we are not designing any "new format" here. This is not tied to json particularly either. Clients of ufoLib2 can do whatever they like with this, this is just enabling them to more easily (de)construct ufoLib2 classes to something that can be (de)serialized. We can figure those thing out elsewhere

@madig
Copy link
Collaborator

madig commented Dec 15, 2021

Right, so we use it externally to the established libraries, where tooling has to deliberately make use of it.

@anthrotype anthrotype merged commit 58185a4 into master Dec 16, 2021
@anthrotype anthrotype deleted the cattrs branch December 16, 2021 13:17
anthrotype added a commit that referenced this pull request Dec 16, 2021
This is to continue supporting font.kerning = {(a, b): -10, ...} even after we typed Font.kerning as Kerning dict subclass to customize (un)structuring via cattrs in #181
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants