Architect

Role Overview

As a Senior Data Engineer / Tech Lead, you will design, develop, and optimize scalable, enterprise‑grade cloud‑native data platforms and pipelines on the Microsoft Azure ecosystem. You will possess deep technical ownership and be responsible for implementing modern data architectures (such as Lakehouse and Medallion architecture). This role requires close collaboration with architects, business analysts, and downstream data consumers to translate complex business requirements into high‑performing and secure data solutions.

Key Responsibilities

1. Data Pipeline Design & Orchestration (ADF)
   • Design, build, and maintain complex, scalable ETL/ELT pipelines using Azure Data Factory (ADF) to ingest heterogeneous data from diverse sources (APIs, SFTP, Oracle Fusion, SQL Server, Event Hubs, etc.)
   • Implement parameter‑driven dynamic pipelines, custom activities, and robust scheduling triggers
   • Configure and manage Integration Runtimes (Self‑Hosted and Azure IR) for secure and efficient on‑premise to cloud data movement
2. Advanced Big Data Processing (Azure Databricks)
   • Develop high‑performance batch and real‑time streaming data pipelines using Azure Databricks, PySpark, Spark SQL, or Scala
   • Implement Medallion Architecture (Bronze, Silver, and Gold layers) and maintain modern Lakehouse architectures utilizing Delta Lake
   • Optimize Spark compute usage through parameter‑driven jobs, proper cluster scaling, and Databricks serverless compute configurations
   • Perform deep performance tuning, query optimization, and memory management of heavy Spark workloads
3. Data Warehousing & Modeling
   • Design and refine data models including dimensional modeling, star schemas, and Slowly Changing Dimensions (SCDs)
   • Optimize databases by writing complex, performant T‑SQL queries, stored procedures, and schemas on Azure SQL Database or Azure Synapse Analytics
4. Data Governance, Security & DevOps
   • Enforce data governance, lineage, and access controls across the platform utilizing Unity Catalog
   • Secure data architectures utilizing Azure Key Vault, Managed Identities, Service Principals, and VNET‑integrated environments
   • Implement CI/CD pipelines using Git and Azure DevOps to automate database and pipeline deployments
5. Leadership & Collaboration
   • Provide technical mentorship and guidance to junior developers and QA teams
   • Enforce engineering discipline through standard code reviews, structured pre/post‑deployment validations, and thorough documentation

Required Skills & Qualifications

• Overall Experience: Minimum of 8+ years of professional experience in Data Engineering or Data Warehousing
• Azure Experience: 4+ years of dedicated, hands‑on experience designing and operating modern data platforms on Microsoft Azure
• Programming Languages: Proficient in Python/PySpark, SQL (advanced query optimization, indexing, stored procedures), and optionally Scala
• Core Azure Stack: Extensive knowledge of Azure Data Factory, ADLS Gen2, Azure Databricks (Delta Lake), Azure SQL Database, and Azure Synapse Analytics
• Big Data Concepts: Robust understanding of Apache Spark architecture, distributed computing, and the Hadoop ecosystem
• Data Integration: Strong background in CDC (Change Data Capture) mechanisms, API integrations, and event‑driven data streaming (e.g., Event Hubs/Kafka)
• Soft Skills: Excellent analytical capabilities, strong documentation skills, and the ability to articulate technical strategies to non‑technical business partners

Preferred / "Good-to-Have" Qualifications

• Certifications: Microsoft Certified: Azure Data Engineer Associate (DP‑203), Azure Solutions Architect Expert, or Databricks Certified Associate/Professional
• Cloud Platforms: Exposure to multi‑cloud environments (e.g., Google Cloud Platform or AWS)
• IaC: Experience using Infrastructure as Code (IaC) tools like Terraform or ARM Templates to deploy Azure resources