Canvas Animation / Render loop

[0pcom]

I'm trying to make a simple demonstration of animation using the canvas, as had previously been suggested.

However, it seems that what I want to do would require a render loop or perhaps setting something else. The documentation is lacking on this.

If it's possible to make an example of just alternating the color of a canvas over time, I can probably work with that.

but if I had some basic example ; such as this webgpu example but just using a canvas:

https://github.com/cogentcore/webgpu/tree/main/examples/cube

That would be even better. I don't need textures at this point.

What I aim to eventually do (among other things) is a canvas animation like this:

https://0magnet.github.io/wasm-stuff/
https://github.com/0magnet/wasm-stuff/blob/master/b.go

And I have an example here of an audio spectrogram display - implemented with 3 GUIs (gomobile, fyne, and wasm) and one TUI (tcell) - which I would really like to try to add an implementation that uses cogentcore:

https://github.com/0magnet/audioprism-go

[kkoreilly]

Here is a simple example of a canvas animation that I have added to the documentation in #1321:

t := 0
c := core.NewCanvas(b).SetDraw(func(pc *paint.Context) {
    pc.DrawCircle(0.5, 0.5, float32(t%60)/120)
    pc.FillStyle.Color = colors.Scheme.Success.Base
    pc.Fill()
})
go func() {
    for range time.Tick(time.Second/60) {
        c.NeedsRender()
        t++
    }
}()

That makes a circle with a changing size based on a time variable incremented in a render loop. If that example is not sufficient, I can definitely help you with the use cases you mentioned; they should absolutely be possible. I am improving the documentation for this in #1321 as mentioned above, and I will continue to work on the documentation.

[0pcom]

Fantastic.

cogentcore-globe-2024-12-30_17.06.09.mp4

Here's the code. Something like this in the canvas documentation / examples would be really eye catching.

package main

import (
	"math"
	"embed"
	"time"
	"cogentcore.org/core/base/mergefs"

	"cogentcore.org/core/core"
	"cogentcore.org/core/colors"
	"cogentcore.org/core/paint"
	"cogentcore.org/core/styles"

)

//go:embed mononoki/*.ttf
var mononoki embed.FS

//go:embed font-test.txt
var fontTest string

const (
	NumLatLines = 12 // Number of latitude lines
	NumLonLines = 24 // Number of longitude lines
	Radius      = 0.4
	CenterX     = 0.5
	CenterY     = 0.5
)

func main() {
	core.TheApp.SetSceneInit(func(sc *core.Scene) {
		sc.SetWidgetInit(func(w core.Widget) {
			w.AsWidget().Styler(func(s *styles.Style) {
				s.Font.Family = "mononoki"
				s.Text.LineHeight.Em(1)
				s.Text.WhiteSpace = styles.WhiteSpacePreWrap
			})
		})
	})

	b := core.NewBody()
	paint.FontLibrary.FontsFS = mergefs.Merge(paint.FontLibrary.FontsFS, mononoki)
	paint.FontLibrary.UpdateFontsAvail()
	core.NewText(b).SetText(fontTest)


	rotationX := 0.0
	rotationY := 0.0
	rotationZ := 0.0

	c := core.NewCanvas(b).SetDraw(func(pc *paint.Context) {
		sinX, cosX := math.Sin(rotationX), math.Cos(rotationX)
		sinY, cosY := math.Sin(rotationY), math.Cos(rotationY)
		sinZ, cosZ := math.Sin(rotationZ), math.Cos(rotationZ)

		projectPoint := func(x, y, z float64) (float32, float32) {
			ry := y*cosX - z*sinX
			rz := y*sinX + z*cosX

			rx := x*cosY + rz*sinY
			z = -x*sinY + rz*cosY

			x = rx*cosZ - ry*sinZ
			y = rx*sinZ + ry*cosZ

			screenX := CenterX + float32(x)
			screenY := CenterY + float32(y)
			return screenX, screenY
		}

		pc.StrokeStyle.Color = colors.Scheme.Primary.Base
		for lat := -math.Pi / 2; lat <= math.Pi/2; lat += math.Pi / float64(NumLatLines) {
			first := true
			var startX, startY float32
			for lon := 0.0; lon <= 2*math.Pi; lon += math.Pi / 100 {
				x := Radius * math.Cos(lat) * math.Cos(lon)
				y := Radius * math.Cos(lat) * math.Sin(lon)
				z := Radius * math.Sin(lat)
				screenX, screenY := projectPoint(x, y, z)
				if first {
					startX, startY = screenX, screenY
					first = false
				} else {
					pc.MoveTo(startX, startY)
					pc.LineTo(screenX, screenY)
					startX, startY = screenX, screenY
				}
			}
			pc.Stroke()
		}

		pc.StrokeStyle.Color = colors.Scheme.Success.Base
		for lon := 0.0; lon <= 2*math.Pi; lon += math.Pi / float64(NumLonLines) {
			first := true
			var startX, startY float32
			for lat := -math.Pi / 2; lat <= math.Pi/2; lat += math.Pi / 100 {
				x := Radius * math.Cos(lat) * math.Cos(lon)
				y := Radius * math.Cos(lat) * math.Sin(lon)
				z := Radius * math.Sin(lat)
				screenX, screenY := projectPoint(x, y, z)
				if first {
					startX, startY = screenX, screenY
					first = false
				} else {
					pc.MoveTo(startX, startY)
					pc.LineTo(screenX, screenY)
					startX, startY = screenX, screenY
				}
			}
			pc.Stroke()
		}
	})

	go func() {
		for range time.Tick(time.Second / 60) {
			rotationX += 0.01
			rotationY += 0.02
			rotationZ += 0.015
			c.NeedsRender()
		}
	}()

	b.RunMainWindow()
}

[kkoreilly]

Great! Thank you for sharing that code. We will consider featuring something like that on the website after we finish updating it to the new content structure in #1321.

[0pcom]

In another globe I have which compiles directly to webassembly, I'm using frag shader code and vert shader code like this

const vertShaderCode = `
attribute vec3 position;
uniform mat4 Pmatrix;
uniform mat4 Vmatrix;
uniform mat4 Mmatrix;
attribute vec3 color;
varying vec3 vColor;

void main(void) {
	gl_Position = Pmatrix*Vmatrix*Mmatrix*vec4(position, 1.);
	vColor = color;
}
`
const fragShaderCode = `
precision mediump float;
varying vec3 vColor;
void main(void) {
	gl_FragColor = vec4(vColor, 1.);
}
`

It basically just gives a red to blue color gradient to make the rotation of the globe more apparent.

How can this kind of thing be implemented using cogentcore ; in the code I posted in the previous reply?

...

I should probably also take the opportunity here to ask about mouse interactions with the canvas. Not that I'm needing to do that, currently, but to know if it's possible would be appreciated.

If it was possible, to grab the globe animation with the mouse at a certain point, and be able to drag it or spin it along whichever axis of rotation - that would be an interesting / fun example to interact with, at least.

Also if you can have animated effects from the mouse moving over the canvas, those tend to make really eye-catching examples as well.

...

Is it possible to play sound via a cogentcore application? Or is it simple enough to do that cross-platform already. I notice the video example, I assume it might be possible to do it via embedded audio files using the same method.

...

Another related question ; when is webgpu anticipated to be widely supported, or at least supported on linux? I've attempted to get some of those examples working in waterfox and brave browser, without success - regardless of the flags used on the browser which supposedly enable experimental support.

[kkoreilly]

Thank you for the update and apologies for the delayed response.

You can use any one of these three lines to create a red to blue gradient effect (I am currently writing website documentation on all of these gradient options):

pc.StrokeStyle.Color = gradient.NewLinear().AddStop(colors.Red, 0).AddStop(colors.Blue, 1)
pc.StrokeStyle.Color = gradient.NewLinear().AddStop(colors.ToBase(colors.Red), 0).AddStop(colors.ToBase(colors.Blue), 1)
pc.StrokeStyle.Color = gradient.NewLinear().AddStop(colors.ToUniform(colors.Scheme.Primary.Base), 0).AddStop(colors.ToUniform(colors.Scheme.Error.Base), 1)

I am not sure precisely what you want it to look like, but whatever you need should be possible with a gradient building on the examples above. You may want to use a rotation or another transformation on your gradient:

gr := gradient.NewLinear().AddStop(colors.ToUniform(colors.Scheme.Primary.Base), 0).AddStop(colors.ToUniform(colors.Scheme.Error.Base), 1)
gr.SetTransform(math32.Rotate2D(math32.Pi / 2))
pc.StrokeStyle.Color = gr

Please let me know if you need more help on how to implement the gradient you want. You can always make a custom type that implements the image.Image interface and use that as the color as well. I will respond to your other questions in subsequent replies.

[kkoreilly]

It is definitely possible to handle mouse interactions with a canvas, likely using the SlideStart, SlideMove, and SlideStop events for dragging, and MouseMove for hovering. If you do not currently need that functionality, it would be easier for me to document it on the website at a later point so that I can focus on other things now, but it should definitely be possible, and I can explain it once you need it. I will probably add some API functions to make things such as event coordinate normalization easier first.

For your audio question, we are planning to add audio playing helper functions and an audio player widget as part of #516. We will also add automatic dependency installation and inclusion where necessary for platform-specific audio dependencies at that time. Until we add that functionality, you should be able to use something such as beep or oto to play audio across platforms. If you have any issues with that, please let me know.

[kkoreilly]

For your final question, I am not completely sure when more browsers will support WebGPU. You can see https://caniuse.com/webgpu for more (somewhat inaccurate) information on current browser support, and I do believe that it is possible to get WebGPU working on some Linux browsers by running them from the command line with appropriate Vulkan flags, but I am not completely sure of that. We currently disable WebGPU support in Cogent Core on Linux web in favor of the 2D backup driver given the spotty support, but after we fix #1414, it should theoretically be possible to get it working with the right flags and drivers. See also #1144. I am sure that Linux web WebGPU support will be good at some point in the future; the question is how long it will take.

[0pcom]

After additional testing, I note that the web / wasm view of the same (60FPS) app fails to load.

If I reduce the frame rate to 30 frames per second, it works. Any higher and it just gets stuck on the loading / preview.

CPU usage is a bit higher for both web and desktop views than with other GUI frameworks I've used - at high frame rates.

I also note that in my case - with onboard graphics - the desktop version uses CPU directly in proportion to the size of the window, regardless of the size of the animation in the window.

I also experimented with adding the animation code to the live view of the canvas documentation here, since it's editable

https://www.cogentcore.org/core/widgets/media/canvases

The performance was lacking when I added the animation there. It seems that the more stuff on a page, the slower any animation is because (I guess) the whole page is being re-rendered at a certain point. That is the only way I can think of to explain it.

After adding and removing the animation there, it seems that the CPU usage stayed higher than it should be. That might merit investigation ; but I'm unsure such an issue would impact most people, as the setup for a live-editable code is not very common.

It may be worthwhile to handle such a case differently than it is currently handled, as it can cause the page to become unresponsive when the animation should be visible.

[kkoreilly]

Thank you for reporting that. We had similar issues with emergent before, and the fundamental cause of the issue is that there is no true concurrency (yet) on web with Go WASM. That means that goroutines actually don't run concurrently, so if one goroutine is constantly working, other goroutines will never get scheduled and the app will freeze / fail to load.

In your case, the maximum performance of the animation on web is probably around 30 FPS, so if you ask for anything higher, time.Tick will never block because it is always behind what you are asking for, so the app event loop and other key goroutines will never get scheduled, and the app will never load.

In the case of emergent, we were able to add short time.Sleep calls at semi-rare intervals to prevent the goroutines from being starved, but a quick attempt at that here didn't seem to work. The easiest solution is probably to compute the effective FPS and then update the requested FPS in the ticker to be less than that to prevent blocking.

It would be nice if Go could support true goroutines on web or have a better scheduling paradigm for unyielding goroutines on a platform that doesn't support concurrency; I saw golang/go#65178 recently, which is a similar issue, but the fix for it doesn't seem to work here. The key issue is that we have a single unyielding goroutine and there is no sysmon on web, so there is no way for it to be terminated. See also golang/go#60857.

As such, I'm not really sure if there is much that can be done here except for the automatic FPS adjustment solution I mentioned above. The only other thing I can thing of would be using requestAnimationFrame through syscall/js, which might actually work pretty well without much code. Please let me know if any of those approaches works. Also please let me know if you have any other ideas for solutions.

I filed #1416 for this, so we can continue discussion there.

[kkoreilly]

Thank you for raising the performance concerns. We are planning to turn to performance improvements at some point (#659), and GPU accelerated rendering (#1056) will greatly improve rendering performance capacity, especially for cases like this. Ideally we would make progress on those issues within the next few weeks, but other things may take higher precedence for the time being.

Until we implement hardware-accelerated 2D rendering, you may also want to consider using xyz for things like this globe, as it is already powered by WebGPU. Of course, that has lower browser support at this point as we discussed earlier in this discussion, so it isn't a complete solution.

We are currently rendering more backgrounds than we need to do, which is likely why the CPU usage increases with window size. When the canvas is re-rendered in an animation, only the canvas widget should be re-rendered by that NeedsRender call (it would be good to confirm that), although the scene image itself does need to be re-uploaded to the window. It is therefore somewhat strange that the animation would be significantly slower when there is other content; we will look into that. The persistence of the higher CPU usage after removing the animation is also somewhat unexpected, and we will also investigate that when we have the time.

We can continue discussion of these performance issues in #659.

[0pcom]

Overall it's very usable, and relatively easy to use compared to other gui frameworks. I don't think I've ever seen a globe animation in golang done with less code than the above implementation using core libraries.

This is only lacking a bit of optimization in some areas, to echo your sentiments.

I'm excited to witness the development, and I'm glad to provide constructive feedback on any issues I encounter ; and sincerely appreciative of your advice, guidance, your kind and thoughtful regards.

I would have never imagined it would be possible, in the first place, to compile the same golang GUI app to desktop, mobile, and web - before I knew about C.O.R.E. ; and I don't know of any other project with similar aims. So my sincerest thanks and compliments to the developers, maintainers, and contributors to this project.

[kkoreilly]

Thank you! We definitely appreciate all of your feedback, and we look forward to continuing development of Cogent Core, especially by improving documentation and performance.

[gedw99]

I suspect web workers are needed to scale out the work ?

[kkoreilly]

@gedw99 Thank you for the suggestion. Please see my reply in #1429. Please note that web workers are not necessarily needed to solve this; see my comment on #1416.

[0pcom]

I'm making some progress towards a spectrogram display using cogentcore libraries ; I just wanted to post this intermediate code which generates a color gradient for reference of others, or to potentially improve the documentation, as it took some time for me to figure out how to do a color gradient on the canvas like this

package main

import (
	"image"
	"image/color"
	"cogentcore.org/core/core"
	"cogentcore.org/core/paint"
	"cogentcore.org/core/styles"
)

func main() {
	b := core.NewBody()
	c := core.NewCanvas(b).SetDraw(func(pc *paint.Context) {
		width, height := 512, 512
		pc.Clear()
		for y := 0; y < height; y++ {
			for x := 0; x < width; x++ {
				r := uint8((x * 255) / (width - 1))
				g := uint8((y * 255) / (height - 1))
				b := uint8(255 - r)
				pixelColor := image.NewUniform(color.RGBA{R: r, G: g, B: b, A: 255})
				pc.StrokeStyle.Color = pixelColor
				pc.SetPixel(x, y)
			}
		}
	})
	c.Styler(func(s *styles.Style) {
		s.Grow.Set(1, 1)
	})
	c.NeedsRender()
	b.RunMainWindow()
}

[0pcom]

After figuring out setting pixels to the desired color, the implementation of the spectrogram display was not too difficult - and I should mention that it's the smallest codebase of any of the 4 other UIs with which I've implemented this.

package ui

import (
	"image"
	"image/color"
	"log"
	"time"

	"cogentcore.org/core/core"
	"cogentcore.org/core/paint"
	"cogentcore.org/core/styles"

	"github.com/jfreymuth/pulse"
	"github.com/0magnet/audioprism-go/pkg/spectrogram"
)

var (
	spectrogramHistory [][]color.Color
	historyIndex int
	width, height int
)

// Run initializes and starts the CORE application
func Run(wid, hei, fpsRate, bSize int, fpsDisp bool, wsURL string) { //nolint:revive
	width = wid
	height = hei
	b := core.NewBody()
	c := core.NewCanvas(b).SetDraw(func(pc *paint.Context) {
		pc.Clear()

		for x := 0; x < width; x++ {
			index := (historyIndex + x) % width
			for y := 0; y < height; y++ {
				invertedY := height - 1 - y
				pc.StrokeStyle.Color = image.NewUniform(spectrogramHistory[index][invertedY])
				pc.SetPixel(x, y)
			}
		}
	})

	spectrogramHistory = make([][]color.Color, width)
	for i := range spectrogramHistory {
		spectrogramHistory[i] = make([]color.Color, height)
		for j := range spectrogramHistory[i] {
			spectrogramHistory[i][j] = color.Black
		}
	}

	var stream *pulse.RecordStream
	var audioCtx *pulse.Client
	var err error
	audioCtx, err = pulse.NewClient()
	if err != nil {
		log.Fatal(err)
	}
	defer audioCtx.Close()

	stream, err = audioCtx.NewRecord(pulse.Float32Writer(processAudio), pulse.RecordLatency(0.1))
	if err != nil {
		log.Fatal(err)
	}
	defer func() {
		if stream != nil {
			stream.Stop()
		}
	}()

	c.Styler(func(s *styles.Style) {
		s.Grow.Set(1, 1)
	})

	stream.Start()
	defer stream.Stop()
	go func() {
		for range time.Tick(time.Second / 60) {
			c.NeedsRender()
		}
	}()
	b.RunMainWindow()

}

func processAudio(p []float32) (int, error) {
	start := 0
	step := spectrogram.FFTSize / 2
	for len(p)-start >= spectrogram.FFTSize {
		magnitudes := spectrogram.ComputeFFT(p[start : start+spectrogram.FFTSize])
		start += step

		currentRow := make([]color.Color, height)
		for y := 0; y < height; y++ {
			freq := float64(y) / float64(height) * 12000
			bin := int(freq * float64(spectrogram.FFTSize) / 44100)

			if bin < len(magnitudes) {
				magnitude := magnitudes[bin]
				currentRow[y] = spectrogram.MagnitudeToPixel(magnitude)
			} else {
				currentRow[y] = color.Black
			}
		}

		spectrogramHistory[historyIndex] = currentRow
		historyIndex = (historyIndex + 1) % width
	}

	return len(p), nil
}

CORE-spectrogram-2025-01-17_13.44.19.mp4

The caterpillar-like movement of the display is just due to the asynchronous updating with c.NeedsRender() in the goroutine.

I know you mentioned in #1416 that you would eventually add OnWindowPaint or similar function ; let me know in that ticket when you implemented it (or here) and I'll update my code to use that. It should smooth the display out considerably, and give me the desires FPS in the web / wasm context.

...

I've updated my project repo with the core option for the desktop gui here:

https://github.com/0magnet/audioprism-go

Sortof on that note, the current wasm version of this program does not use CORE and is compiled with tinygo; and that is actually embedded in the golang binary. Yet the wasm version of the core application will likely dwarf the size of the current audioprism binary, and bloat the current one to many times the size it was before, if included in the same way.

It's unlikely to matter very much for my particular use case, but I just mention this because sometimes it's convenient to embed the compiled wasm client code when client and server codebase are separate, but the size increase of doing that with CORE is substantial compared to some other frameworks. Which, you already know.

I'm not sure what the status of core working with tinygo is, though I've seen it mentioned a few times. Information on that would also be appreciated.

[gedw99]

Good discussion about perf.

I have been using cogent to do play with 2d, 3d rendering and compositing of these together to do 4d ( like a video editor , but a rendering editor essentially over time ).

I also use threejs etc too , because it has a mature architecture that worked out how to get perf on the web, via hard won trial and error.

maybe some of those constructs are needed in cogent .

in the end you can’t get perf , and lively UX , without some operation of computation at a formal level ?

in relation to web perf , web workers are your “threads”.
The golang compiler and channels will not automatically do this.

Thos is a simple example outputting to the console .

https://webgpu.github.io/webgpu-samples/?sample=worker

https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/transferControlToOffscreen Is the means to do this . But I think it’s meant for 2d. Not sure as it’s new to me.

https://web.dev/articles/offscreen-canvas Has demo and code and explains how it keeps the UX event loop alive.

So, if this is the way, In order to have an isomorphic architecture the concept of workers needs to be part of web and native.

https://github.com/magodo/go-wasmww is a simple golang /wasm example of workers for web that formalises this multi tiered architecture.

the golang compiler can’t do this . It has to be designed into the cogent architecture obviously . But it needs a shim so that both web and native can compile and run off the same code . The golang way to do this is build tags , rather than runtime reflection.

Compositing in real time is the way to use webgpu workers.

in the web up to 32 canvas are available ( like time I checked ) , with the main “ worker “ compositing the workers buffers together.

the web way of doing compositing is https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer which is in all browsers .

Correction. A shared array is what’s used with an offscreen canvas, because it works with a TransferableObject.

This all leads back to isomorphic architecture. The native and web versions need this structured stack formalised in order to get serious perf , with a way to pass the data around or a reference ( aka shared buffer ).

in the end your back to a queue and workers like we do for large scale computing on servers but applied to GPU. This is why I call it Compositing , which is the name for this in the graphics domain .

[0pcom]

This is some great info, thanks for posting.