Senior LLM Systems Engineer

Senior LLM Systems Engineer at Risk Labs - Remote | Jobless Posted 2 months ago Senior LLM Systems Engineer RemoteRemoteFull-time AI Summary Senior LLM Systems Engineer responsible for improving the accuracy, performance, resilience, and observability of LLM-driven oracle automation systems. This is a production systems role focused on evaluations, fallbacks, and measurable model quality in real-world conditions. About this role Why This Role Exists: We are hiring a Senior LLM Systems Engineer to own and improve the LLM-driven components of our oracle automation stack. This person will focus on the accuracy, performance, resilience, and operational quality of the systems that use models to reason about wide ranging prediction market rules, evidence, and oracle outcomes. This is a production systems role, not a research-only or prompt-only role. You will build the evaluations, observability, tooling, fallbacks, and feedback loops that make LLM behavior measurable and dependable in real-world conditions. What You'll Own: LLM Accuracy: improve prompts, model selection, tool usage, structured outputs, retrieval, and evaluation coverage so the system gets more decisions right over time. System Performance: reduce latency, token usage, and cost while preserving decision quality and operational reliability. Resilience: design validation, retries, fallbacks, uncertainty handling, and human review paths for ambiguous, adversarial, incomplete, or conflicting inputs. Evaluation and Monitoring: build datasets, regression tests, dashboards, traces, and review loops that make model quality visible and prevent repeated failures. Agent and Tooling Architecture: Improve agent orchestration and tool use across internal services, APIs, search workflows, databases, and external data sources. Production Operations: help debug live issues, investigate regressions, improve runbooks, and reduce repeated operator friction. What Success Looks Like: The oracle