Software Engineer, Web Crawling

Exa
United States
Workplace: OnsiteFull timeUSD 150,000 - 300,000 annuallyFunction: Software EngineeringSkills: ["Problem-solving","Collaboration","Communication"]

Join Exa to build Google-scale crawling infrastructure for a new search engine. You will scale web crawlers, work with high-performance languages, and optimize systems to deliver high-quality, rapidly indexed knowledge across massive datasets.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Exa
Exa
1 year ago

Software Engineer, Web Crawling

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 4 hours agoStatus: Live

Job Summary

Join Exa to build Google-scale crawling infrastructure for a new search engine. You will scale web crawlers, work with high-performance languages, and optimize systems to deliver high-quality, rapidly indexed knowledge across massive datasets.
Location: United States
Workplace: Onsite
Employment Type: Full time
Job Function: Software Engineering

Key Responsibilities

  • •Build and maintain crawlers capable of crawling large portions of the web at scale
  • •Collaborate to design and implement distributed crawling systems and prioritization
  • •Optimize crawl politeness, rate limits, and handling of dynamic content across domains
  • •Develop intelligent crawl scheduling and prioritization algorithms for maximum coverage
  • •Address anti-bot challenges and ensure robust handling of JavaScript-rendered content

Pay and Benefits

Salary: USD 150,000 - 300,000 annually
Equity and Bonus:Equity
Perks:Equity

Key Requirements

  • •Extensive experience building and scaling web crawlers, or eagerness to ramp up quickly
  • •Experience with high performance languages (C++, Rust, etc.)
  • •Familiar with TypeScript, Playwright, modern web design, CDP (Chrome DevTools Protocol)
  • •Ability to optimize a system to an exceptional degree
  • •Passion for finding high quality knowledge and its impact on the world
Experience:Web crawlingSearch
Skills:Problem-solvingCollaborationCommunication
Tech Stack:RustC++TypeScriptPlaywrightCDPJavaScript

Eligibility

Visa:OPTSTEM OPTH1BO1E3
Work Authorization:Sponsorship available.

Company Brief

Exa
Builds an embeddings-based neural search engine and web search API for AI applications, offering crawling, retrieval, and deep research tools to serve developers and AI agents with up-to-date web knowledge.
Industry: API Platforms
Company Size: Small (11 to 50 employees)
Growth: Scaleup
Valuation: USD 500M to 1B
Funding: Series B
Headquarters: San Francisco, United States
Founded: 2021
WebsiteLinkedIn