HAMi (Heterogeneous AI Computing Virtualization Middleware) is an open-source solution for GPU virtualization and unified scheduling in Kubernetes clusters.

It enables compute power limiting and memory isolation for GPUs at the Kubernetes layer, allowing multiple workloads to share the same physical GPU without interference.

For AWS Neuron devices , HAMi integrates with the AWS Neuron Device Plugin to support shared access, while providing a unified scheduling layer for both GPU and Neuron resources.

Key capabilities:

GPU virtualization: Split NVIDIA GPUs by compute percentage and memory size, enabling fine-grained sharing for inference and training.

AWS Neuron integration: Works with AWS Neuron Device Plugin to enable shared usage of Inferentia and Trainium devices, with HAMi handling unified scheduling.

Topology-aware scheduling: Optimize workload placement based on NUMA and NVLink topology for better performance.

Works with popular AI stacks: Fully compatible with NVIDIA GPU Operator, Xinference, and vLLM Production Stack.

HAMi is ideal for AI inference, large model serving, and training workloads on Kubernetes, helping maximize hardware utilization across mixed GPU and AWS Neuron environments.