Description:
Adastra is seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines and cloud-based data platforms that support advanced analytics, reporting, and AI initiatives. This role is responsible for developing high-performance ETL/ELT solutions using PySpark and modern data engineering technologies, enabling efficient processing of large-scale structured and semi-structured data across enterprise environments.
The ideal candidate is a hands-on data engineering professional with strong expertise in distributed data processing, cloud data platforms, and data pipeline orchestration. This individual will collaborate closely with data architects, data scientists, analysts, and business stakeholders to deliver reliable, scalable, and high-quality data solutions that drive business value and support data-driven decision-making.
Primary Location: Toronto, ON
Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using PySpark for batch and near-real-time data processing
- Develop and optimize Spark jobs using DataFrames, RDDs, and Spark SQL for large-scale data transformations and aggregations
- Build reusable data ingestion frameworks to support integration from databases, APIs, flat files, and streaming sources
- Optimize Spark application performance through partitioning, caching, broadcast joins, and cluster resource tuning
- Collaborate with data scientists, analysts, and business stakeholders to deliver clean, curated datasets for reporting and advanced analytics
- Implement automated data quality checks, validation rules, and monitoring processes to ensure data reliability
- Develop and maintain workflow orchestration using Airflow, Oozie, or similar scheduling tools
- Integrate data pipelines with cloud data lakes and data warehouse platforms such as Snowflake, Redshift, Delta Lake, S3, and HDFS
- Troubleshoot and resolve pipeline failures, production issues, and distributed processing bottlenecks
- Maintain technical documentation, version control, and deployment processes for data engineering solutions
- Participate in code reviews, testing, and CI/CD initiatives to improve solution quality and delivery efficiency
- Support the continuous improvement of data engineering frameworks, standards, and best practices
Qualifications, Skills & Experience
- Bachelor's degree in Computer Science, Engineering, Information Technology, Data Analytics, or a related field
- 5+ years of experience in Data Engineering, ETL/ELT development, or Big Data environments