Senior Software Engineer - Crawler
Full-time
ZoomInfo
Responsibilities
- Design and implement scalable, fault-tolerant web crawling and extraction pipeline components.
- Write clean, production-grade Java and Python code.
- Build and operate large-scale ETL/ELT pipelines for data extraction and transformation.
- Work with GCP and AWS cloud infrastructure, primarily on GKE.
- Improve observability, reliability, and operational excellence across contributed systems.
- Partner with product and data science teams to deliver solutions.
- Contribute to code reviews, documentation, and knowledge sharing.
- Keep current with web technologies, anti-crawling mechanisms, and AI-powered extraction approaches.
Requirements
- At least 5 years of professional software engineering experience building production systems.
- Strong knowledge of algorithms, data structures, concurrency, and distributed systems.
- Proficiency in Java and/or Python.
- Experience owning features end-to-end from design through deployment and operation.
- Experience making component-level architectural decisions.
- Hands-on experience with cloud data warehouses such as BigQuery or Snowflake.
- Experience designing and operating large-scale ETL/ELT pipelines.
- Experience with orchestration tools such as Apache Airflow.
- Experience with streaming or event-driven systems such as Apache Kafka.
- Production experience on GCP or AWS, with GCP preferred.
- Hands-on experience with Kubernetes, including GKE or EKS, for distributed workloads.
- Familiarity with infrastructure-as-code tools such as Terraform.
- Strong communication, pragmatic problem-solving, adaptability, and self-direction.
- Web crawling at scale, Scrapy, proxy infrastructure, anti-bot evasion, SERP extraction, AI/LLM-based extraction, or B2B data-company experience is preferred.
Benefits
- Remote work arrangement is indicated by the LI-REMOTE designation.
- Comprehensive benefits are offered, along with holistic mind, body, and lifestyle well-being programs.
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Senior Software Engineer - Crawler in Remote vacancy
- ...external code, and deployed instrumentation. Apply and coach engineering best practices for robust, testable, and composable code.... ...technologies. Requirements At least 5 years of experience as a software engineer, data scientist, or ML engineer; most successful...
- ...excellence matter every day. We're looking for a hands-on Senior Corporate IT Engineer to own and scale the technology that supports our workforce... ...About the Role House Rx is hiring a Senior Security Software Engineer to help us build, review, and harden security-sensitive...
- ...and production inference workflows. Partner with application engineering teams to integrate AI into code review, test generation,... ...technical leadership. Requirements ~6+ years of professional software engineering experience or equivalent, including shipping production...
- ...and working with brilliant people. You are an accomplished senior Python programmer that enjoys challenging projects in mission-critical... ...one of the industry's pioneers in open source with intelligent engineers at every level from engineer to CTO and CEO level. At Canonical...
$2000 per year
...across PC and IoT technologies. Our teams partner with specialist engineers from major silicon companies to integrate next-generation... ...integration for the latest IoT and server-class hardware platforms and software stacks Work with partners to deliver a delightful,...$2000 per year
...Canonical is a leading provider of open source software and operating systems to the global enterprise and technology markets. Our platform... ...initiatives such as public cloud, data science, AI, engineering innovation, and IoT. Our customers include the world's leading...- ...30%+ at ~$2B ARR is that we need great engineers everywhere, and we have the room to shape... ...NYSE: IOT) sits at the center of hardware, software, AI, and the physical world. The... ...is broad and flexible: we're hiring from Senior through Principal, across platform and product...
- ...company founded by former Google Cloud leaders, architects, and engineers. We are seeking candidates with significant experience in... .... What we need... ~5+ years of experience working in a Software Engineering capacity ~ Excellent knowledge and experience with...
- ...We’re looking for a Software Engineer who would contribute to PVcase software and would not be afraid of heights (at least career ones)! You will play an important role in our exciting journey while accelerating the global transition to clean energy. Participate and...
- ...for task authoring, expert annotation, review, quality control, and structured data collection. Collaborate with researchers and engineers to ensure evaluation data and published outputs are accurate, consistent, and integrated. Requirements ~4+ years of...
- ...our platform scales globally, we're looking for people who can turn complex systems into practical, working solutions. As a Software Engineer in Partner Engineering, you’ll play a pivotal role in shaping technical solutions that directly impact our top-tier partners....
- ...with customers and subject-matter experts, and take ownership of product outcomes. Requirements Four or more years of software engineering experience, most of it in frontend work, at companies with strong engineering practices. Strong frontend capability in either...
- ...Your Next Successful Chapter with Veta Virtual! Are you a seasoned full-stack engineer with a passion for building secure, high-performance systems? Veta Virtual is hiring a Full-Stack Software Engineer for an important fintech company. This role is ideal for someone...
$2000 per year
...Canonical is a leading provider of open source software and operating systems to the global enterprise and technology markets. Our platform... ...initiatives such as public cloud, data science, AI, engineering innovation, and IoT. Our customers include the world's leading...- ...About The Role Join the cutting-edge team building the data platform that powers fleet-scale humanoid robotics. As a Staff Software Engineer on our Data Platform team, you will play a pivotal role in designing, developing, and maintaining the infrastructure that enables...
- ...Improve application performance, code quality, accessibility, and developer experience. Provide technical guidance to Software Engineers I and II through code reviews, pairing, and design feedback. Collaborate with Product, Design, and QA to align technical implementation...
$2000 per year
...Canonical is a leading provider of open source software and operating systems to the global enterprise and technology markets. Our platform... ...initiatives such as public cloud, data science, AI, engineering innovation and IoT. Our customers include the world's leading...- ...Canonical is a leading provider of open source software and operating systems to the global enterprise and technology markets. Our platform... ...initiatives such as public cloud, data science, AI, engineering innovation, and IoT. Our customers include the world's leading...
$2000 per year
...Canonical is a leading provider of open source software and operating systems to the global enterprise and technology markets. Our platform... ...initiatives such as public cloud, data science, AI, engineering innovation, and IoT. Our customers include the world's leading...$2000 per year
...Canonical is a leading provider of open source software and operating systems to the global enterprise and technology markets. Our platform... ...initiatives such as public cloud, data science, AI, engineering innovation, and IoT. Our customers include the world's leading...$2000 per year
...Canonical is a leading provider of open source software and operating systems to the global enterprise and technology markets. Our platform... ...initiatives such as public cloud, data science, AI, engineering innovation, and IoT. Our customers include the world's leading...$2000 per year
...Canonical is a leading provider of open source software and operating systems to the global enterprise and technology markets. Our platform... ...initiatives such as public cloud, data science, AI, engineering innovation and IoT. Our customers include the world's leading...- ...Use AI-assisted development tools, including GitHub Copilot. Participate in production support and on-call rotations. Mentor engineers while remaining hands-on with development. Requirements At least 8 years of Java development experience. Strong expertise...
- Responsibilities Design, automate, validate, and deliver the complete lifecycle of customer compute environments from provisioning through upgrades, recovery, and decommissioning. Use AI to automate and accelerate infrastructure delivery and operations. Provision...
- ...with the security team to align the application with organizational security standards. Collaborate directly with the client engineering team as an embedded engineer. Requirements Strong C# and .NET experience, including advanced C# features and nullable reference...
- ...mismatch three systems deep. The client who calls because a number looks wrong and it takes an hour of digging to prove it isn't. Most engineers want to build the shiny new thing and hand off the boring maintenance to someone else. We need the opposite: someone who wants to...
- ...focused on two new major product lines coming to market in the next few months. Join us!! The Role We're hiring a Staff/Senior Security Engineer to lead our signing and treasury security program across wallets and custodian accounts. This is a high-impact, mission-...
$2000 per year
...chance to be a part of that as a Community Engineer at Canonical . We are building... ...multiple positions available, at different seniority levels. All applications and candidates... ...who is passionate about open source software, Linux, and sustainable community building...$2000 per year
...Canonical is a leading provider of open source software and operating systems to the global... ...as public cloud, data science, AI, engineering innovation, and IoT. Our customers include... ...works in Python, across all levels of seniority. Apply here if you believe you are...$2000 per year
...the entire internet. The role of Security Engineering Manager directly impacts the safety and... ...of engineers, ranging from graduate to senior Coach, mentor, and offer career development... ...Knowledgeable and passionate about software and application security Solid...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior Software Engineer - Crawler. Be the first to apply!
