Staff Software Engineer - Streaming

<p>At Databricks, we are passionate about enabling data teams to solve the world&39;s toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world&39;s best data and AI infrastructure platform so our customers can turn deep data insights into better business outcomes.</p> <p>We are the <strong>Spark Structured Streaming</strong> team, responsible for building stream processing into Apache Spark and the Databricks Data Intelligence Platform. Stream processing is still in its early days, and we&39;re here to build not only state-of-the-art streaming technology but also a best-in-class managed offering for customers to run their streaming workloads.</p> <h4>The role</h4> <p>We&39;re seeking an experienced Staff Engineer to drive the technical direction of Spark Structured Streaming, spanning both open source and Databricks-specific components. Your mission is to make Spark Structured Streaming the state-of-the-art stream processing engine — adding advanced capabilities such as sophisticated state management and new operators, while re-imagining the engine&39;s architecture to drive improvements for latency, throughput, and cost.</p> <p> </p> <p><strong>What you&39;ll do:</strong></p> <ul> <li>Set and drive the technical vision for Spark Structured Streaming across OSS and the Databricks Data Intelligence Platform</li> <li>Design and build core engine capabilities — state management, new operators, and architectural improvements to latency and throughput</li> <li>Raise the bar for engineering quality and operability, building software that is not just high quality but easy to run in production</li> <li>Make company-wide impact by driving stream processing adoption across the Databricks product portfo