Can RamaLama support Kubernetes-based inference clustering on a Mac Mini M4?

[hotwa]

I’m really impressed with the work being done on RamaLama and its ability to handle various models for inference. I have been exploring the possibility of leveraging Kubernetes (K8s) for distributed model in my local environment, specifically on a Mac Mini.

Is there currently support within RamaLama to deploy and manage model inference workloads using Kubernetes on a Mac Mini?
If not natively supported, are there any recommendations or best practices for integrating RamaLama with Kubernetes for local development/testing purposes?
Are there any known limitations or considerations when running Kubernetes-based inference clusters on ARM-based hardware (like the M4 chip) using RamaLama?

[ericcurtin]

It can be done, you need to install podman-machine with krunkit that's the first step https://podman.io/ . There's also kubelet generators via:

ramalama serve --generate

Most of the bits are there, just need someone to tie it all together.

[ericcurtin]

Would make a great blog post!

[rhatdan]

ramalama serve --generate kube MODEL

Will generate a k8s deployment for the containerized AI Model.

[hotwa]

Can Docker Desktop serve as a replacement for Podman in this setup?

[ericcurtin]

docker desktop should be relatively compatible, not that I've tried it personally. I recommend trying Docker VMM beta if you want accelerated GPU support on Docker Desktop (Podman Desktop is more tested with RamaLama though). Docker VMM beta uses krunkit https://github.com/containers/krunkit

https://docs.docker.com/desktop/features/vmm/

[rhatdan]

I am going to move this to a discussion.

[hotwa]

maybe podman is a better choice