Remote RAG and Evaluation Engineer job

Remote RAG and Evaluation Engineer job at LTS | Lensa More Log in Get free job alerts Get free job alerts Get free job alerts Log in RAG and Evaluation Engineer Full-Time Apply Report This Job Job Company Description Salary Skills Job Description LTS is seeking a RAG & Evaluation Engineer to join a small, senior engineering team applying frontier AI to one of the most consequential legacy systems still running in production today. The mission: build agents that read, translate, and modernize a decades-old codebase that millions of people quietly depend on. The work has executive backing, real users, and a customer who knows exactly what they're buying. Specifics shared once we're talking. The team is small by design. Every seat carries unusual leverage, and we hire people who are already deep in this work. We use AI tooling natively - agents in parallel, model as collaborator, no exceptions. What You'll Do: The RAG & Evaluation Engineer owns the knowledge surface and the eval harness. Ingestion pipelines for source code, structured metadata, technical documentation, patches, and additional corpora the customer provides. Retrieval quality across chunking, embeddings, hybrid retrieval, reranking, freshness. Benchmarks for translation accuracy, dependency-map correctness, and overall agent quality. The feedback loop from production usage back into evals and retrieval lives here. - Own the knowledge surface - ingestion pipelines for source code, structured metadata, technical documentation, patches, and additional corpora the customer provides. - Own retrieval quality - chunking, embeddings, hybrid retrieval, reranking, and freshness. - Own the eval harness - benchmarks for translation accuracy, dependency-map correctness, and overall agent quality. - Run A/B testing and regression detection across prompts, retrieval, and model changes. - Operate the feedback loop from production usage back into evals and retrieval. - Define wh