Replies: 7 comments 10 replies
-
Vello summaryAfter a first-pass reading through the code, here's my summary of how https://github.com/linebender/vello works, and what the issues are for us.
So, again, vello has some significant further development needed, and thus a Go port would then require significant updating in the future, and it may not render very good looking and / or fast text right now. Overall, the real strength of vello is in rendering complex graphical images with lots of overlap etc. This is not really our main use-case. Blitting font glyphs is our main use case. We can probably improve our CPU performance on that, in much less total time than doing the vello port. We'll continue to monitor it, and hopefully at some point it will be useful and super cool to get all this working in Go, but not right now. |
Beta Was this translation helpful? Give feedback.
-
|
It is also important to note that doing some kind of wrapper around vello itself, instead of a full Go port, is not useful because it would not work on the web: no cgo on the web! And this also presumes a C wrapper around vello which does not exist yet either! If we do the port, then we have pure Go calling WebGPU shaders through javascript via the webgpu javascript bindings: very nice. |
Beta Was this translation helpful? Give feedback.
-
|
By the way, we could theoretically link to C/Rust on the web using emscripten, but that would be a lot of work and would require a non-standard build system since Go doesn't automatically support it. (The big advantage of that would be that |
Beta Was this translation helpful? Give feedback.
-
Canvas issuesfloat32 vs. float64There are some "impedance mismatches" between canvas and paint, the main one being the use of
On balance, I would be very reluctant to move away from Renderer interfaceHere's the renderer interface in canvas, which is then the target api for all backends: canvas.go:
// Renderer is an interface that renderers implement. It defines the size of the target (in mm) and functions to render paths, text objects and images.
type Renderer interface {
Size() (float64, float64)
RenderPath(path *Path, style Style, m Matrix)
RenderText(text *Text, m Matrix)
RenderImage(img image.Image, m Matrix)
}The rasterization api all goes through // Path defines a vector path in 2D using a series of commands (MoveTo, LineTo, QuadTo, CubeTo, ArcTo and Close). Each command consists of a number of float64 values (depending on the command) that fully define the action. The first value is the command itself (as a float64). The last two values is the end point position of the pen after the action (x,y). QuadTo defined one control point (x,y) in between, CubeTo defines two control points, and ArcTo defines (rx,ry,phi,large+sweep) i.e. the radius in x and y, its rotation (in radians) and the large and sweep booleans in one float64.
// Only valid commands are appended, so that LineTo has a non-zero length, QuadTo's and CubeTo's control point(s) don't (both) overlap with the start and end point, and ArcTo has non-zero radii and has non-zero length. For ArcTo we also make sure the angle is in the range [0, 2*PI) and we scale the radii up if they appear too small to fit the arc.
type Path struct {
d []float64
// TODO: optimization: cache bounds and path len until changes (clearCache()), set bounds directly for predefined shapes
// TODO: cache index last MoveTo, cache if path is settled?
}The // PathCommand is the type for the path command token
type PathCommand fixed.Int26_6 //enums:enum -no-extend
// Human readable path command constants
const (
PathMoveTo PathCommand = iota
PathLineTo
PathQuadTo
PathCubicTo
PathClose
)
// A Path starts with a PathCommand value followed by zero to three fixed
// int points.
type Path []fixed.Int26_6And the more abstract Raster interface looks like this: // Raster is the interface for rasterizer types. It extends the [Adder]
// interface to include LineF and JoinF functions.
type Raster interface {
Adder
LineF(b fixed.Point26_6)
JoinF()
}
// Adder is the interface for types that can accumulate path commands
type Adder interface {
// Start starts a new curve at the given point.
Start(a fixed.Point26_6)
// Line adds a line segment to the path
Line(b fixed.Point26_6)
// QuadBezier adds a quadratic bezier curve to the path
QuadBezier(b, c fixed.Point26_6)
// CubeBezier adds a cubic bezier curve to the path
CubeBezier(b, c, d fixed.Point26_6)
// Closes the path to the start point if closeLoop is true
Stop(closeLoop bool)
}Note that For reference, in Skia, the class SK_API SkPath {
public:
/**
* Create a new path with the specified segments.
*
* The points and weights arrays are read in order, based on the sequence of verbs.
*
* Move 1 point
* Line 1 point
* Quad 2 points
* Conic 2 points and 1 weight
* Cubic 3 points
* Close 0 points
...StylingEvery framework has its own way of defining the stroke styles etc. Canvas is similar to our styles, but of course there are differences, which we need to examine. In our copy of // SetStroke set the parameters for stroking a line. width is the width of the line, miterlimit is the miter cutoff
// value for miter, arc, miterclip and arcClip joinModes. CapL and CapT are the capping functions for leading and trailing
// line ends. If one is nil, the other function is used at both ends. If both are nil, both ends are ButtCapped.
// gp is the gap function that determines how a gap on the convex side of two joining lines is filled. jm is the JoinMode
// for curve segments.
func (r *Stroker) SetStroke(width, miterLimit fixed.Int26_6, capL, capT CapFunc, gp GapFunc, jm JoinMode) {Images and textThese are not handled at all in srwiley/raster, and we just handle them separately in our Bottom lineWe should probably make an interface like that in canvas, and then have glue for each backend, etc. Can convert float32 to float64 etc to leverage the canvas api, but it almost certainly will not be the high-performance pathway. In any case, with our canvas bridge interface, we could directly leverage the other fancy backends like PDF, and probably we'd want to write our own more direct version of the html canvas backend for web. |
Beta Was this translation helpful? Give feedback.
-
|
It seems like we should fold canvas into Cogent Core, only taking the novel parts, instead of maintaining a compatibility layer that has to constantly convert between types. |
Beta Was this translation helpful? Give feedback.
-
|
And miss out on (or have to manually apply) upstream updates on canvas? |
Beta Was this translation helpful? Give feedback.
-
|
Some further points about the Raster interface:
Canvas does have the Context: // Context maintains the state for the current path, path style, and view transformation matrix.
type Context struct {
Renderer
path *Path
ContextState
stack []ContextState
}
|
Beta Was this translation helpful? Give feedback.
-
Text representationHere's the // Text holds the representation of a text object.
type Text struct {
lines []line
fonts map[*Font]bool
WritingMode
TextOrientation
Width, Height float64
Text string
Overflows bool // true if lines stick out of the box
}
type line struct {
y float64
spans []TextSpan
}
// TextSpan is a span of text.
type TextSpan struct {
X float64
Width float64
Face *FontFace
Text string
Glyphs []text.Glyph
Direction text.Direction
Rotation text.Rotation
Level int
Objects []TextSpanObject
}
// TextSpanObject is an object that can be used within a text span. It is a wrapper around Canvas and can thus draw anything to be mixed with text, such as images (emoticons) or paths (symbols).
type TextSpanObject struct {
*Canvas
X, Y float64
Width, Height float64
VAlign VerticalAlign
}
// Glyph is a shaped glyph for the given font and font size. It specified the glyph ID, the cluster ID, its X and Y advance and offset in font units, and its representation as text.
type Glyph struct {
SFNT *font.SFNT
Size float64
Script
Vertical bool // is false for Latin/Mongolian/etc in a vertical layout
ID uint16
Cluster uint32
XAdvance int32
YAdvance int32
XOffset int32
YOffset int32
Text rune
}
// Script is the script.
type Script uint32 <- this is a wrapper on go-text/typesetting/languageAha, and how I see that the Here's ours for reference: // Text contains one or more Span elements, typically with each
// representing a separate line of text (but they can be anything).
type Text struct {
Spans []Span
// bounding box for the rendered text. use Size() method to get the size.
BBox math32.Box2
// fontheight computed in last Layout
FontHeight float32
// lineheight computed in last Layout
LineHeight float32
// whether has had overflow in rendering
HasOverflow bool
// where relevant, this is the (default, dominant) text direction for the span
Dir styles.TextDirections
// hyperlinks within rendered text
Links []TextLink
}
// Span contains fully explicit data needed for rendering a span of text
// as a slice of runes, with rune and Rune elements in one-to-one
// correspondence (but any nil values will use prior non-nil value -- first
// rune must have all non-nil). Text can be oriented in any direction -- the
// only constraint is that it starts from a single starting position.
// Typically only text within a span will obey kerning. In standard
// Text context, each span is one line of text -- should not have new
// lines within the span itself. In SVG special cases (e.g., TextPath), it
// can be anything. It is NOT synonymous with the HTML <span> tag, as many
// styling applications of that tag can be accommodated within a larger
// span-as-line. The first Rune RelPos for LR text should be at X=0
// (LastPos = 0 for RL) -- i.e., relpos positions are minimal for given span.
type Span struct {
// text as runes
Text []rune
// render info for each rune in one-to-one correspondence
Render []Rune
// position for start of text relative to an absolute coordinate that is provided at the time of rendering.
// This typically includes the baseline offset to align all rune rendering there.
// Individual rune RelPos are added to this plus the render-time offset to get the final position.
RelPos math32.Vector2
// rune position for further edge of last rune.
// For standard flat strings this is the overall length of the string.
// Used for size / layout computations: you do not add RelPos to this,
// as it is in same Text relative coordinates
LastPos math32.Vector2
// where relevant, this is the (default, dominant) text direction for the span
Dir styles.TextDirections
// mask of decorations that have been set on this span -- optimizes rendering passes
HasDeco styles.TextDecorations
}
// Rune contains fully explicit data needed for rendering a single rune
// -- Face and Color can be nil after first element, in which case the last
// non-nil is used -- likely slightly more efficient to avoid setting all
// those pointers -- float32 values used to support better accuracy when
// transforming points
type Rune struct {
// fully specified font rendering info, includes fully computed font size.
// This is exactly what will be drawn, with no further transforms.
// If nil, previous one is retained.
Face font.Face `json:"-"`
// Color is the color to draw characters in.
// If nil, previous one is retained.
Color image.Image `json:"-"`
// background color to fill background of color, for highlighting,
// <mark> tag, etc. Unlike Face, Color, this must be non-nil for every case
// that uses it, as nil is also used for default transparent background.
Background image.Image `json:"-"`
// dditional decoration to apply: underline, strike-through, etc.
// Also used for encoding a few special layout hints to pass info
// from styling tags to separate layout algorithms (e.g., <P> vs <BR>)
Deco styles.TextDecorations
// relative position from start of Text for the lower-left baseline
// rendering position of the font character
RelPos math32.Vector2
// size of the rune itself, exclusive of spacing that might surround it
Size math32.Vector2
// rotation in radians for this character, relative to its lower-left
// baseline rendering position
RotRad float32
// scaling of the X dimension, in case of non-uniform scaling, 0 = no separate scaling
ScaleX float32
} |
Beta Was this translation helpful? Give feedback.
-
|
Here's gio structs. It adds a visualOrder slice that allows mapping from a standardized LR visual order to the input source order, which may get rendered RL. This is important for text selection logic, which does not appear to be supported in canvas. It does not appear to support vertical text. It does look like canvas supports vertical layout, and it also supports text decoration (underline etc), which Gio does not appear to support. It is unclear what kind of underlying representation of rich text is used, but we definitely need to separate the html part of it from the rich text aspect of things. Canvas also supports full knuth level line breaking, which is amazing, but also looks very complicated. It does not look like we could use the canvas code directly because it has its own distinct font representation, and doesn't have the key selection encoding, and uses float64 instead of 32, and actually text rendering needs to happen in fixed, though probably it makes sense to postpone that as long as possible, for other backend representations. We also handle SVG text rendering, which adds a rotation and path components. ugh. // document holds a collection of shaped lines and alignment information for
// those lines.
type document struct {
lines []line
alignment Alignment
// alignWidth is the width used when aligning text.
alignWidth int
unreadRuneCount int
}
// A line contains the measurements of a line of text.
type line struct {
// runs contains sequences of shaped glyphs with common attributes. The order
// of runs is logical, meaning that the first run will contain the glyphs
// corresponding to the first runes of data in the original text.
runs []runLayout
// visualOrder is a slice of indices into Runs that describes the visual positions
// of each run of text. Iterating this slice and accessing Runs at each
// of the values stored in this slice traverses the runs in proper visual
// order from left to right.
visualOrder []int
// width is the width of the line.
width fixed.Int26_6
// ascent is the height above the baseline.
ascent fixed.Int26_6
// descent is the height below the baseline, including
// the line gap.
descent fixed.Int26_6
// lineHeight captures the gap that should exist between the baseline of this
// line and the previous (if any).
lineHeight fixed.Int26_6
// direction is the dominant direction of the line. This direction will be
// used to align the text content of the line, but may not match the actual
// direction of the runs of text within the line (such as an RTL sentence
// within an LTR paragraph).
direction system.TextDirection
// runeCount is the number of text runes represented by this line's runs.
runeCount int
yOffset int
}
type runLayout struct {
// VisualPosition describes the relative position of this run of text within
// its line. It should be a valid index into the containing line's VisualOrder
// slice.
VisualPosition int
// X is the visual offset of the dot for the first glyph in this run
// relative to the beginning of the line.
X fixed.Int26_6
// Glyphs are the actual font characters for the text. They are ordered
// from left to right regardless of the text direction of the underlying
// text.
Glyphs []glyph
// Runes describes the position of the text data this layout represents
// within the containing text.Line.
Runes Range
// Advance is the sum of the advances of all clusters in the Layout.
Advance fixed.Int26_6
// PPEM is the pixels-per-em scale used to shape this run.
PPEM fixed.Int26_6
// Direction is the layout direction of the glyphs.
Direction system.TextDirection
// face is the font face that the ID of each Glyph in the Layout refers to.
face *font.Face
// truncator indicates that this run is a text truncator standing in for remaining
// text.
truncator bool
}
// glyph contains the metadata needed to render a glyph.
type glyph struct {
// id is this glyph's identifier within the font it was shaped with.
id GlyphID
// clusterIndex is the identifier for the text shaping cluster that
// this glyph is part of.
clusterIndex int
// glyphCount is the number of glyphs in the same cluster as this glyph.
glyphCount int
// runeCount is the quantity of runes in the source text that this glyph
// corresponds to.
runeCount int
// xAdvance and yAdvance describe the distance the dot moves when
// laying out the glyph on the X or Y axis.
xAdvance, yAdvance fixed.Int26_6
// xOffset and yOffset describe offsets from the dot that should be
// applied when rendering the glyph.
xOffset, yOffset fixed.Int26_6
// bounds describes the visual bounding box of the glyph relative to
// its dot.
bounds fixed.Rectangle26_6
}
// Range describes the position and quantity of a range of text elements
// within a larger slice. The unit is usually runes of unicode data or
// glyphs of shaped font data.
type Range struct {
// Count describes the number of items represented by the Range.
Count int
// Offset describes the start position of the represented
// items within a larger list.
Offset int
} |
Beta Was this translation helpful? Give feedback.
-
Benchmarking initial results
BenchmarkTable
|
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
Progress updatePer #1457 PR, at this point the new API is in place, with a full Interestingly, we gained basically nothing from tdewolf/canvas in terms of its rendering code which is notably slower than the amazing rasterx per above benchmarks and profiling, but at least it did provide a solid Path framework that is much richer than what I previously coded for the svg/path element. This Path representation is essential for all the backends, including future GPU, so it is a great piece of infrastructure. |
Beta Was this translation helpful? Give feedback.
-
Text plansFonts
Layout from fonts
|
Beta Was this translation helpful? Give feedback.
-
|
From an end user standpoint, supporting some kind of functional API for rich text would be great, and I think that using special unicode characters seems somewhat tenuous. |
Beta Was this translation helpful? Give feedback.
-
Text organizationHere are the different levels of text representations, what properties they have, and how we might want to organize them. I created a new overall top-level Sources:
Levels:
Uses:
Organization:
|
Beta Was this translation helpful? Give feedback.
-
|
Good link for complex international layout issues: https://www.w3.org/International/articles/vertical-text/ |
Beta Was this translation helpful? Give feedback.
-
Can now do the actual render in a separate thread!@kkoreilly just pointed out that once we create the full |
Beta Was this translation helpful? Give feedback.
-
This discussion is to organize plans around the 2D rendering infrastructure.
Currently, we're using a lightly modified version of https://github.com/srwiley/rasterx with the https://github.com/srwiley/scanx rasterizer backend, which was faster than the https://pkg.go.dev/golang.org/x/image/vector rasterizer in his testing (reported on the github readme for example).
This renderer is plenty fast for modern desktop systems, but when this CPU-based code runs via WASM on the web, we really start to see significant slowdowns, especially on mobile web platforms. Thus, one good strategy would be to directly leverage WebGPU, which we already have good infrastructure for, to do 2D rasterization, to avoid the CPU -> WASM slowdown. The https://github.com/linebender/vello framework in particular, written in Rust, provides very fast WebGPU-specific 2D rasterization, and was our nominal plan (by making a Go port and re-using their .wgsl shaders). However, in starting to investigate it, there are many issues that will be summarized in a subsequent sub-post here, and it is clear that this is a major undertaking with some significant drawbacks at this time.
We also looked at https://github.com/tdewolff/canvas which is all in Go and has lots of amazing features. Overall the code looks very compatible with our
stylesandpaintpackages, to the point where much of the actual code there is essentially redundant with that. The novel bits are mostly in the backends and in some of the more advanced text formatting functionality. The native Go render-to-an-image backend is none other than https://pkg.go.dev/golang.org/x/image/vector, so it is unlikely to represent a speedup relative to what we're currently using.However, it also has an html canvas backend, which suggests a different solution to the web rendering problem: just leverage the browser's own canvas engine (skia or whatever) directly, instead of doing everything in Go. This is not as fancy as WebGPU but is almost certainly a much easier and more well-supported path. It is unclear when WebGPU on the iPhone safari browser will actually be usable, but chrome on Android works now (as of very recently -- and maybe not on various older devices?)
That would just leave lower-performance mobile native devices with a sub-standard 2D rendering system (iPhone, Android) -- they get the WebGPU
Drawercompositing benefits, but not the basic 2D rendering.It is somewhat difficult to have comparable test cases across different rasterizing backends, and much likely depends on the details of the content being rendered, but a first priority is to try to get some benchmarks comparing our current
paintsetup withcanvason various realistic rendering cases (e.g., the equivalent of scrolling text and images, and some data-heavy plots, and some basic SVG images).Then we probably want to see if we can easily get the html canvas backend working, and see how that works (can hopefully use canvas to test that with our benchmarks first).
Eventually we may want to do the vello port, but probably not now.
The other major interacting topic here is text formatting on the way to rendering. Our current
paintpackage has an impl that works OK but lacks key international font rendering and layout support. https://github.com/go-text is used by Fyne and Gio for their text layout, and it has a direct Go translation of the "industry standard" https://github.com/harfbuzz/harfbuzz C library for cross-language support. Canvas also has its own reasonable support for this, and its framework is more compatible overall on first inspection with ourpaintimpl, so it might be easier to use for us. It also has latex line breaking and other support for latex (though not the native go version from star-tex?) There are some further layout things still pending: tdewolff/canvas#74 -- need to compare more directly with go-text.It is not 100% clear but it this wiki page: https://github.com/tdewolff/canvas/wiki/Planning suggests that perhaps there are further optimizations that need to happen on the text rendering -- we'll presumably find this out with our benchmarks. Our code uses rendered glyph images that get cached (which takes up memory!) but is presumably faster and worth it.
See also #1056 and #568 for our current issues on these things.
Beta Was this translation helpful? Give feedback.
All reactions