LLM Evaluator (Model Response Analyst)

About the role

We are looking for a detail‑oriented LLM Evaluator to assess and improve the performance of large language models. You will analyze AI‑generated text for factual accuracy, coherence, safety, bias, and alignment with our guidelines, helping to make the models more reliable and trustworthy.

Key responsibilities

• Evaluate and rank model‑generated responses using complex rubrics covering factuality, coherence, safety, instruction‑following, and creativity.
• Compare multiple outputs for the same prompt, select the preferred one, and justify your choice.
• Provide concise feedback to modeling and training teams about recurring failure patterns.
• Design adversarial prompts to expose biased, harmful, or insecure model behavior.
• Collaborate with QA to refine evaluation guidelines for ambiguous edge cases.
• Participate in cross‑checking sessions to ensure inter‑rater reliability.
• Investigate underperforming outputs, hypothesize root causes, and flag novel behaviors.

Required profile

• Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP QA, or cognitive science.
• Bachelor’s degree in Computer Science or a related field.
• Proven ability to craft prompts that test model limits and to explain why outputs are good or bad.
• Experience with Reinforcement Learning from Human Feedback (RLHF) data collection and evaluation.
• Track record of maintaining consistency across evaluation teams and conducting calibration sessions.

Required skills

• Prompt engineering for LLMs
• RLHF data collection and analysis
• Dataset sourcing, cleaning, and annotation
• Understanding of data distribution impact on model performance
• A/B testing concepts applied to AI models