Lead Data Engineer

Description:

Aadhya Technologies Inc specializes in delivering cutting-edge IT consulting and top-tier recruitment services to help businesses thrive in a rapidly evolving digital world. We focus on quality, agility, and innovation to solve complex tech challenges and build high-performing teams for our clients across various industries.

Role: Lead Data (Databricks) Engineer

Location: Toronto, ON or Mississauga, ON

Job Type: Full-time | On-site (Face-to-Face Interview Required)

Experience Required: 8 to 10+ years

Job Summary

We are seeking a highly skilled Azure Data Engineer with deep expertise in Databricks to join our data engineering team. The ideal candidate should bring 8 to 10+ years of experience in designing, building, and optimizing large-scale data pipelines. You'll work closely with cross-functional teams to deliver cutting-edge data solutions with a focus on scalability, reliability, governance, and performance.

Key Responsibilities

Data Pipeline Development

• Build and maintain scalable ETL/ELT pipelines using Databricks
• Use PySpark, Spark, and SQL to transform and process large datasets
• Integrate data from Azure Blob Storage, ADLS, and other structured/unstructured sources

Collaboration & Analysis

• Partner with business and BI teams to deliver tailored data solutions
• Understand data requirements and translate them into efficient technical implementations

Performance & Optimization

• Optimize Databricks workloads for performance and cost efficiency
• Proactively monitor and troubleshoot data pipelines

Governance & Security

• Implement and enforce data access and governance policies using Unity Catalog
• Ensure compliance with data security and organizational standards

Deployment & CI/CD

• Use Databricks Asset Bundles for seamless deployment of jobs and notebooks
• Collaborate using GitLab for code versioning and DevOps automation

Technical Skills

• Strong experience with Databricks (Delta Lake, Unity Catalog, Delta Live Tables, Runtime, etc.)
• Proficient with Azure Cloud Services (ADLS, ADF, Blob Storage, etc.)
• Expertise in Spark, PySpark, and SQL
• Familiarity with Databricks Asset Bundles and version control via GitLab
• Solid understanding of data architecture and distributed processing frameworks

Preferred Experience

• Working knowledge of Databricks Runtime configurations
• Experience with real-time streaming tools like Spark Streaming
• Building real-time or near-real-time data processing pipelines

Certifications(Preferred, not mandatory)

• Azure Data Engineer Associate
• Databricks Certified Data Engineer Associate

Additional Requirements

• 8 to 10+ years of relevant experience in data engineering
• Must attend an in-person face-to-face interview in Toronto or Mississauga office