Technical Lead - GPU Infrastructure

Tether.io
Cairo
Workplace: RemoteFull timeFunction: Administration & Executive AssistanceEducation: bachelorsSkills: ["Leadership","Engineering standards","Incident response","Architecture decision-making","Written and spoken communication"]

Own the architecture and delivery of a self-hosted GPU compute and managed inference platform for Cosmic AC. Lead a distributed engineering team to run bare-metal GPU scheduling via Slurm, operate a Kubernetes control plane, and enable NVIDIA GPU clusters. Drive reliability, observability, and incident response across control plane, GPU fleet, and application tiers while acting as the primary technical interface to infrastructure partners and vendors.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Tether.io
Tether.io
2 days ago

Technical Lead - GPU Infrastructure

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 7 hours agoStatus: Live

Job Summary

Own the architecture and delivery of a self-hosted GPU compute and managed inference platform for Cosmic AC. Lead a distributed engineering team to run bare-metal GPU scheduling via Slurm, operate a Kubernetes control plane, and enable NVIDIA GPU clusters. Drive reliability, observability, and incident response across control plane, GPU fleet, and application tiers while acting as the primary technical interface to infrastructure partners and vendors.
Location: Cairo
Workplace: Remote
Employment Type: Full time
Job Function: Administration & Executive Assistance
Seniority: Manager level

Key Responsibilities

  • •Own the platform architecture end to end (architecture proposals, high-/low-level designs, and maintaining them as the baseline).
  • •Lead and line-manage a distributed engineering team (backend, frontend, DevOps, QA, documentation) with engineering standards, code/design review, release gates, and one-to-ones.
  • •Design, build, and operate a managed Slurm scheduling layer on bare metal (controller/accounting, partitions/login nodes, node onboarding, driver/CUDA baseline and upgrades, stalled-job/node-health detection, drain/autohealing, storage visibility, identity/isolation).
  • •Own Kubernetes control plane and GPU enablement (cluster bootstrap/lifecycle on partner-provided bare metal, NVIDIA GPU Operator/Network Operator, VM-based GPU isolation, and day-2 ops like upgrades/backup/recovery/node replacement).
  • •Drive observability and operations across tiers (metrics/logging/alerting and SLOs), run incident response and post-incident reviews, and serve as the primary technical interface to partners/vendors.
Travel: Low travel

Key Requirements

  • •8+ years of hands-on engineering experience, including at least three years leading teams that build and operate infrastructure platforms, plus a bachelor’s or master’s degree (or equivalent practical experience).
  • •Hands-on Slurm operation for real users (e.g., slurmctld/slurmdbd): partitions, QoS/priority, accounting, prolog/epilog, node health scripting, and upgrades; ideally operated an HPC/GPU training cluster.
  • •Bare-metal GPU fleet operations: NVIDIA driver/CUDA lifecycle, Fabric Manager/NVSwitch behavior on SXM systems, DCGM health/utilization, MIG, and node burn-in/acceptance.
  • •High-performance interconnect experience: InfiniBand fabric/subnet config, RDMA, SR-IOV, and diagnosing multi-node NCCL performance problems.
  • •Production Kubernetes operations beyond deployment (control plane, upgrades, CNI/CSI, operators/custom controllers) plus Linux systems depth for compute workloads (e.g., vfio-pci, cgroups/namespaces).
Experience:Infrastructure platformsHPCGPU infrastructureKubernetesFintech
Education:Bachelor's
Skills:LeadershipEngineering standardsIncident responseArchitecture decision-makingWritten and spoken communication
Languages:English
Tech Stack:JavaScriptNode.jsReactKubernetesKubeVirtVFIOSlurmNVIDIA GPU OperatorNetwork OperatorCUDANVIDIA Fabric ManagerNVSwitchDCGMMIGInfiniBandRDMASR-IOVLinuxKernel modulesDrivers

Company Brief

Tether.io
Builds tools and services to help companies hire, onboard, and manage remote or distributed teams across borders, focusing on payroll, compliance, and global employment workflows.
Industry: HR Tech
Website