Senior / Software Engineer, Inference Platform & Agent Sandboxes

About Aleroy

Aleroy is building a unified platform for high-performance AI inference, making it easier for companies to deploy and run AI models at scale.
Through a single API, Aleroy provides access to ready-to-use and custom AI models across cloud infrastructure, dedicated capacity, and customers' own hardware. Our platform optimizes performance, cost, and deployment flexibility while giving enterprises control over where and how their AI workloads run.
We work in small teams, collaborate directly, and focus on solving complex engineering challenges to build reliable, scalable AI infrastructure.

About the Role

You build the platform our data-centre partners run. You bring it up on their clusters, keep each customer isolated, route and meter every request, and run the sandboxes where customers' AI agents execute code. This is systems work in Go on Kubernetes and Linux; it needs no ML research.
We're looking for a Senior / Software Engineer, Inference Platform & Agent Sandboxes. You work with our Tech Lead and CTO, and next to the engineer who owns model performance.

What You Will Own

• 
  Running on partner clusters

  • Bring-up on a new partner's hardware; failure detection and recovery

  • Deploying and scaling model servers on Kubernetes: autoscaling, rollouts, GPU placement

  • Isolation between customers: data, cache and limits

  Request path

  • The OpenAI-compatible API and token streaming, with cancellation and client disconnects

  • Cache-aware routing across replicas, models and backends; admission control and backpressure when request cost varies widely

  • Usage metering per customer and per partner, counted exactly once, because partners are paid on it

  Agent sandboxes

  • Isolated environments where customers' AI agents run code: fast start, snapshot and restore, resource limits, network policy

  • Running them on partners' CPU and GPU capacity, with the same isolation and metering as inference

  Training jobs and customer data

  • Post-training jobs on partner GPUs: multi-node scheduling, checkpointing and recovery

  • Customer data and model weights kept in the customer's own storage, with customer-managed encryption keys

About You

• 3+ years (mid) or 5+ years (senior) building and operating distributed systems in production

• Strong production experience in Go; TypeScript (Node.js) or Python is a plus

• Kubernetes in production: scheduling, autoscaling, rollouts; writing operators or controllers is a plus

• Linux internals: containers, cgroups, namespaces, networking

• Streaming APIs and request routing where request cost varies widely

• You have owned a production service: on-call, incidents, rollbacks

• For senior: you have led the design of a system end to end and explained its trade-offs to other teams

• Professional working English

What We Offer

• Real ownership of the backend that powers Aleroy inference platform

• Influence over backend architecture and product direction

• Hands-on work on hard problems in distributed systems, real-time streaming, and LLM serving

• A small, senior team with fast execution and low bureaucracy

• Competitive salary with performance-based incentives

• 24 days annual leave plus public holidays

• Health insurance

• Modern office in Yas Creative Hub

NICE TO HAVE

• Firecracker, gVisor, Kata Containers, KVM or QEMU; Rust

• vLLM, SGLang or llm-d in production; GPU scheduling or sharing (MIG, Kueue, HAMi)

• Usage billing for APIs; Envoy or another proxy

• RDMA or InfiniBand

• Bare-metal or on-prem operations; multi-node training infrastructure