AI Benchmark Engineer
Full-time
LILT (Production)
Responsibilities
- Design and engineer Terminal-Bench tasks that evaluate coding agents on multilingual software challenges.
- Build realistic task environments, datasets, and files in Arabic without relying on English translations.
- Identify AI failure points through native-language prompting and translation work.
- Develop reference implementations and highly reliable deterministic verifier scripts.
- Analyze execution logs and calibrate task difficulty from Easy to Very Hard across Haiku, Sonnet, and Opus model tiers.
- Participate in creation, human review, calibration review, audit, and automated quality-control processes.
- Ensure benchmark tasks are fair, grammatically accurate, reproducible, and technically robust.
Requirements
- 5+ years of industry experience in software engineering.
- Proven experience at leading technology companies and/or graduation from top-tier engineering universities.
- Native or near-native Arabic fluency with deep knowledge of grammar, register, and phrasing, plus high English proficiency.
- Strong proficiency in Python, standard shell scripting, and data processing.
- Extensive experience with Terminal/CLI-based development workflows and familiarity with coding agents.
- Deep understanding of encoding and decoding, Unicode normalization, locale-dependent conventions, text I/O, toolchain interoperability, safe string operations, and Arabic bidirectional/RTL handling, font fallbacks, and rendering or typography issues.
- Reliable availability and commitment to on-time delivery after accepting tasks.
Benefits
- Independent contractor flexibility to choose when and how much to work.
- Remote freelance work with no fixed hours, check-ins, or micromanagement.
- Competitive rates and prompt payments.
- Access to diverse AI and language-technology projects and a global expert community.
- Work availability fluctuates with project demand; this is intended as supplemental income rather than a full-time primary role.
- Most tasks require at least 2 hours per day or 15–20 hours per week, though hours are not guaranteed.
- This is a 1099 contractor role without health insurance, paid time off, or retirement contributions; contractors are responsible for their own taxes.
- Geographic restrictions may apply in regions subject to international embargoes or sanctions.
Vacancy posted 14 days ago
Similar jobs that could be interesting for youBased on the AI Benchmark Engineer in Remote vacancy
- ...We are seeking experienced native-speaking software engineers to design, build, and validate these benchmarks. You will create high-signal, high-quality tasks that... ...& Translation: finding failure points where AI does not work, in your native language - Implementation...
- ...Breakout, xStocks, Payward Services and CF Benchmarks – has spent the last 15 years building... ...Transformation owns Kraken’s internal AI platform. AI tooling is deployed at scale... ...the governed integration gateway as an engineering surface. You’ll handle connector onboarding...
- ...ourselves to inspire others. About the role: We are scaling and our specialists are leveling up. Now we’re looking for an AI Engineer to build AI agents and solutions that transform engineering workflows. Your impact: Build AI agents for engineers that...
- ...your expertise meets the highest-priority AI problems. You should expect to move... ...and pull external datasets, models, and benchmarks from published research and open-source releases... ...code and design review, and bring other engineers up on eval practice. – Some of the technical...
- ...privacy and security laws and regulations. For more information, please see our Privacy Policy We’re looking for Senior AI Engineers to design, build and run LLM-powered agents and AI features in production. You’ll work across the full lifecycle: from scoping a...
- ...applications and next steps. Our partner is looking for an AI Research Engineer (Kernel & Inference Optimization) based in Saudi Arabia. You... ..., and vision transformers. Your work will involve rigorous benchmarking, production testing, and iterative optimization to...
- ...on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Enterprise AI Governance & Trust Layer Engineer based in Saudi Arabia. This is a remote engineering role focused on securing and governing enterprise use of...
- ...This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a AI Trainers Network - Maltese based in Saudi Arabia. Join a global contributor network helping shape the next generation of artificial intelligence...
- ...This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a AI Trainers Network - English based in Saudi Arabia. As part of a global AI contributor network, you will help shape smarter, safer, and more...
- ...on behalf of a partner company, who manages all applications and next steps. Our partner is looking for an Artificial Intelligence (AI) Technician based in Saudi Arabia. This is a remote opportunity for a technically skilled AI professional looking to make a tangible...
$130000 - $195000 per year
...ubiquitous. We build the foundation for agent engineering in the real world, helping developers move from prototypes to production-ready AI agents that teams can rely on. We began... ...5M raised at Series B from IVP, Sequoia, Benchmark, CapitalG, and Sapphire Ventures, we’re...- ...partner is looking for a Senior Software Engineer, Quality based in Saudi Arabia. This... ...generation of quality engineering through AI-powered testing and intelligent automation... ...experience with LLM evaluation or benchmarking, open-source software development, or developer...
$21393 per year
...About this role: Are you an experienced engineer at the stage of your career where, above... ...on who you are: ~ We are all in on AI where it allows us to get more $ to people... ...Kenya, we will share an estimated salary benchmark for the country you are based in during the...- ...ubiquitous. We build the foundation for agent engineering in the real world, helping developers move from prototypes to production-ready AI agents that teams can rely on. We began... ...5M raised at Series B from IVP, Sequoia, Benchmark, CapitalG, and Sapphire Ventures, we’re...
- ...Breakout, xStocks, Payward Services and CF Benchmarks – has spent the last 15 years building... ...investors and institutional clients. Engineering Operations is the operational engine of... ...and tooling that remove friction – Use AI and automation in your day to day work...
- ...We're hiring: AI Facilitators/Instructors across MENA Manara is looking for experienced AI Facilitators/Instructors to help educators across the MENA region build the confidence and skills to use AI in their classrooms. As part of a large-scale, multi-year AI education...Remote job
- ...throughput, low-latency Flutter applications while elevating engineering standards across our development teams. What you’ll drive:... ...clear decisions on system trade-offs. Establish engineering benchmarks for app startup times, memory footprint, and frame rates....
- ...specialists are leveling up. Now we’re looking for talented DevOps Engineer ready to grow with us. Your impact: Manage and optimize... ...-oriented approach to problem-solving ~ Regular use of AI tools, understanding of their limitations, and responsible handling...
- ...Ensure GIS solutions comply with enterprise architecture, security, engineering, and deployment standards. Troubleshoot production issues,... ...~ Experience with Datadog is preferred. ~ Experience using AI tools or AI agents to improve engineering productivity is...
- ...Our partner is looking for a Coding Specialist (Fluent in Dutch) - Freelance AI Trainer Project based in Saudi Arabia. As a Coding Specialist and AI Trainer, you’ll use your software engineering expertise and Dutch fluency to help improve the capabilities of advanced AI...
$50 per day
...At Raya, DevSecOps Engineers are passionate about building and maintaining robust, scalable infrastructure that empowers product engineering... ...and resource utilization. You’re enthusiastic about leveraging AI to enhance your own infrastructure workflows, automate complex...$8 - $65 per hour
...of a partner company, who manages all applications and next steps. Our partner is looking for a Welsh Language Specialist - Freelance AI Trainer Project based in Saudi Arabia. This freelance opportunity is designed for an experienced Welsh language professional who...$2000 per year
...markets. Our platform, Ubuntu, is very widely used in breakthrough enterprise initiatives such as public cloud, data science, AI, engineering innovation, and IoT. Our customers include the world's leading public cloud and silicon providers, and industry leaders in many...- ...Kraken, NinjaTrader, Breakout, xStocks, Payward Services and CF Benchmarks – has spent the last 15 years building one of the most modern... ...and institutional clients. Our Platform and Core Engineering teams build the systems that make Kraken secure, resilient, and...
$60 per hour
...applications and next steps. Our partner is looking for a SWE-Bench AI Task Auditor - Freelance AI Trainer Project based in Saudi... ...This freelance opportunity is designed for experienced software engineers who want to contribute their technical expertise to the development...- ..., data platforms, and third-party services. Partner with Data, Engineering, Product, Legal, and Compliance to apply privacy-by-design and... ...AWS and GCP. – Assess customer data shared with third-party AI model vendors and define minimization, approved use, retention,...
- ...contributions to open-source projects, especially in P2P or decentralized technology. Benefits ~100% remote work with a globally distributed team. ~ Opportunity to work on AI-assisted experiences, P2P technology, decentralized technology, and digital finance products....
$1000 per year
...Anywhere Jobs (FlexJobs, 2025). We are looking for Senior CV Engineer . Your main tasks will be: Train and fine-tune generative... ...kernel and framework-level work Optionally: train multimodal AI companions that combine CV and NLP for deeper visual...- ...Overview We are seeking a highly experienced Senior Linux Infrastructure Engineer with deep expertise in Linux administration, bare metal infrastructure, enterprise storage, and next-generation AI Factory / GPU infrastructure platforms . This role is focused on...
- ...company, who manages all applications and next steps. Our partner is looking for a Hungarian Trilingual Language Specialist - Freelance AI Trainer Project based in Saudi Arabia. This freelance opportunity is ideal for a multilingual language specialist passionate about...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Benchmark Engineer. Be the first to apply!
