OverviewAt Microsoft AI, our health team is on a mission to help millions of users better understand and proactively manage their health and wellbeing. We’re responsible for ensuring that Microsoft AI’s models and services are useful, trusted and safe across diverse customer health journeys. We are assembling a world-class team of builders with backgrounds in healthcare, big tech, and frontier AI research labs. Our goal is to translate cutting-edge research into transformative products for millions of users across copilot.com and Microsoft’s consumer ecosystem. We are looking for software engineers with experience designing and building LLM-based products, and taking them through to production. You will be a key bridge between research and product, and play a pivotal role in establishing Copilot as the leader in safe, informative, trustworthy and useful health information. We are specifically looking for engineers with experience designing, building and running evaluations for LLMs. You will be expected to build eval pipelines, curate and synthesise datasets, design automated analyses and explain results to internal stakeholders.ResponsibilitiesBe a key cross-functional team member, and close collaborator to product teams. Make significant contributions from translating research into benchmarks to co-authoring product roadmap.Design and build evaluation systems that test LLM capabilities in the healthcare domain, and interpret and communicate results.Run experiments to determine how different prompting techniques affect results on internal and industry benchmarks.Improve the tooling that is used internally to implement, run and analyze evaluations.Design internal benchmarking and regression testing capabilities that capture model accuracy, safety and utility.QualificationsRequired Qualifications:Bachelor\'s or higher degree in Computer Science or related technical discipline and significant Python programming experience / machine learning research.Deep experience building with and around LLMs, and experience building tools for analysing and understanding their performance, including prompt and context engineering.Experience collaborating in cross-functional teams, and delivering high-quality results amidst ambiguity.0 to 1 years of experience with a bias towards shipping and learning, while maintaining a high-quality bar.Proven ability to collaborate and contribute to a positive, inclusive work environment, fostering knowledge sharing and growth within the team.Preferred Qualifications:Experience in healthcare technology or health domain.Experience with data engineering - handling text dataset sourcing, curation, and processing tasks at scale.Passionate about conversational AI and its deployment.Demonstrated written and verbal communication skills with the ability to work closely with cross-functional teams.Commitment to learning new technologies and staying up to date with AI trends and best practices.Seniority levelNot ApplicableEmployment typeFull-timeJob functionOtherIndustriesSoftware Development #J-18808-Ljbffr