Senior Staff Applied AI Engineer - Context Retrieval

Databricks
Mountain View, San Francisco
Workplace: HybridFull timeFunction: Data Science & Machine LearningExperience: 10+ yearsSkills: ["Communication","Leadership","Problem-solving","Collaboration","Strategic thinking"]

Lead the design and build of a full retrieval stack and agent-aware search subsystems for enterprise SaaS data. Architect end-to-end context retrieval across structured/unstructured data, connectors to enterprise sources, and evaluation pipelines, enabling Databricks agents and humans to access accurate, high-quality context at scale.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Databricks
Databricks
2 months ago

Senior Staff Applied AI Engineer - Context Retrieval

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 6 hours agoStatus: Live

Job Summary

Lead the design and build of a full retrieval stack and agent-aware search subsystems for enterprise SaaS data. Architect end-to-end context retrieval across structured/unstructured data, connectors to enterprise sources, and evaluation pipelines, enabling Databricks agents and humans to access accurate, high-quality context at scale.
Location: Mountain View, San Francisco
Workplace: Hybrid
Employment Type: Full time
Job Function: Data Science & Machine Learning
Seniority: Sr. Manager level

Key Responsibilities

  • •Build the full retrieval stack from scratch, owning end-to-end system design for query understanding, content understanding and indexing, hybrid retrieval, ranking, and evaluation.
  • •Retrieve across heterogeneous data—structured and unstructured—and index/rank across assets like tables, SQL queries, dashboards, docs, tickets, and media.
  • •Connect to the SaaS surface area customers actually use by building connectors and retrieval adapters for enterprise data sources with appropriate freshness and permissions signals.
  • •Optimize retrieval for both LLM-driven grounded contexts and human-friendly discovery, aligning different signals to support both uses.
  • •Crack query understanding for agents through decomposition, intent classification, and entity resolution tuned for multi-turn workflows.

Key Requirements

  • •10+ years of software engineering experience, with significant time spent building production retrieval, search, or RAG systems at scale.
  • •Deep Information Retrieval (IR) expertise: lexical retrieval (BM25, Lucene/Elasticsearch/OpenSearch), dense retrieval (embeddings, ANN indexes — FAISS, ScaNN, HNSW), hybrid retrieval, and learning-to-rank.
  • •Hands-on experience with modern LLM-era retrieval: RAG architectures, query rewriting, re-ranking with cross-encoders, long-context strategies, and grounding techniques that reduce hallucination.
  • •Experience designing agentic systems on top of retrieval: search planners, multi-hop / iterative retrieval, self-reflection and sufficiency checks, tool-using agents that decide what to fetch and verify what came back.
  • •Strong grasp of relevance evaluation: nDCG, MRR, Precision@K, Recall@K; offline/online experimentation; LLM-as-judge frameworks; building human labeling pipelines.
Experience:10+ yearsEnterprise softwareAIInformation retrievalSaaS
Skills:CommunicationLeadershipProblem-solvingCollaborationStrategic thinking
Languages:English
Tech Stack:BM25LuceneElasticsearchOpenSearchFAISSScaNNHNSWRAGLLMEmbeddingsCross-encodersQuery rewriting

Company Brief

Databricks
Provides a unified data analytics platform powered by Apache Spark to simplify building, deploying, and scaling data engineering, data science, and machine learning workloads for enterprises.
Industry: Data Infrastructure
Company Size: Enterprise (1,001+ employees)
Revenue: USD 1B+
Growth: Scaleup
Valuation: Unicorn (USD 1B+)
Funding: Series E+
Headquarters: San Francisco, United States
Founded: 2013
WebsiteLinkedIn