Software Engineer, Web Crawling

Exa is building a search engine from scratch to serve every AI agent. We build massive-scale infrastructure to crawl the web, train state-of-the-art embedding models to process it, and design super high performant vector databases in rust to search over it. If you like compute, we also own a $5M H200 GPU cluster (and soon 5x'ing that) and regularly spin up batchjobs with tens of thousands of machines. As a Web Crawler engineer, you'd be responsible for crawling the entire web. Basically build Google-scale crawling! WHO YOU ARE - You have extensive experience building and scaling web crawlers, or would be excited to ramp up very quickly - You have experience with some high performance language (C++, Rust, etc.) - You are familiar with TypeScript, Playwright, modern web design, CDP (Chrome DevTools Protocol) - You’re comfortable optimizing a system to an exceptional degree - You care about the problem of finding high quality knowledge and recognize how important this is for