Thread by @teropa

Reading the very cool "Learned Representation for Scalable Vector Graphics" paper about learning a latent space of fonts, by @iraphas13, @hardmaru, @douglas_eck (thread)

arxiv.org/abs/1904.02632

This model deals with fonts as vector path data. That's what all OpenType fonts are actually represented as: TrueType or PostScript paths, which are essentially the same as SVG paths. Sequences of straight line segments and bezier curves.

These paths encode the outlines of character glyphs. For example, for an "o" character you have all the curve segments of both the outer and inner outlines of the o shape separately.

While font designers often (usually?) think in higher level abstractions than this, such as the strokes of a calligraphic pen, or the interplay of geometric shapes, this is inching towards that world.

Certainly this seems more readily useful for typographic tooling than the kind of pixel-based representations I've seen earlier (erikbern.com/2016/01/21/ana…,

https://twitter.com/cyrildiagne/status/1095603397179396098

)

The model learns a latent space of fonts using a Variational Autoencoder, trained on pixel renderings of glyphs using two neural nets: A convolutional encoder turns each image into a 32-dimensional latent vector, and a deconvolutional decoder turns it back into the original image

But then - and this is the coolest part - there's a separate "SVG decoder" that is trained to construct SVG paths (same as OpenType font paths) from the same 32-dimensional latent vectors learned by the pixel-based networks.

This is an LSTM network that models the SVG paths as sequences of commands: Draw line from A to B, then draw curve from B to C, etc.

Just like char-rnn does with text and many of Magenta's music models do with musical structures.

github.com/karpathy/char-… github.com/tensorflow/mag…

I assume the pixel-based rendering is what allows the model to learn the latent space of font "styles", which is difficult dealing purely with vectors: Two vector paths that look similar may have entirely different underlying commands. (There's no useful "gradient" to learn with)

What this model learns is a smooth latent space of font style, and with that you can do latent spacey things, such as explorable 2d/3d map visualisations of fonts using dimensionality reduction.

Or do interpolations between styles. There might be a useful way to construct interpolations for OpenType variable fonts in this.

typenetwork.com/brochure/opent…

Or build tools that make use of latent vector arithmetic: “Increase the boldness of this”, presumably also things like “add serifs to this”, “this but with more stroke contrast", etc...

The model is also conditioned on glyph, so it "knows" which characters it's dealing with. This means it can generalise style across glyphs: When you’ve obtained the latent vector for one glyph, you can decode it for a whole glyph set - generating a whole font with the same style.

Many interesting avenues to explore from this paper and its footnotes.

Also very much looking forward to the code. 🤩

https://twitter.com/iraphas13/status/1114011443429380096

I was also intrigued by the references to Hofstadter's "Fluid Concepts", which I've bought but not gotten around to reading yet. Hadn't realised it deals with typography.

amazon.com/Fluid-Concepts…