Run AI workloads on any cloud, store on Hugging Face: zero-egress storage with SkyPilot

For AI practitioners, the constant movement of massive datasets and model checkpoints between clouds has historically been a significant source of operational friction and wasted budget. This development effectively decouples compute from storage, allowing teams to prioritize GPU availability and cost-efficiency over data proximity. By removing these artificial barriers, engineers can now iterate faster, run experiments on any hardware, and avoid the vendor lock-in that typically dictates infrastructure strategy.

Hugging Face Storage is now a first-class backend for SkyPilot, accessible via the new hf:// scheme. This update allows users to mount Hugging Face buckets or repositories directly into their tasks using existing HF_TOKEN credentials, supporting both read-only model loading and read-write checkpointing. The integration leverages Xet-backed storage for efficient deduplication and utilizes a FUSE-based mount system that enables lazy loading, keeping GPUs productive while data streams in from the Hub.

Engineering leaders should audit their current cloud spend to identify opportunities for consolidating storage on the Hugging Face Hub, which now serves as a unified, provider-agnostic data layer. Teams should transition their existing SkyPilot configurations to utilize the hf:// scheme, as it simplifies secret management by replacing per-cloud bucket keys with a single authentication token. Moving forward, organizations should adopt this multi-cloud approach to gain greater bargaining power and operational flexibility when sourcing scarce GPU resources.