Description:
We are seeking an experienced Data Engineer to design, develop, and maintain scalable data solutions across cloud platforms. The ideal candidate will have strong expertise in Python, SQL, and modern data engineering practices, with hands-on experience in AWS and Azure/Microsoft Fabric ecosystems. This role requires a strategic thinker who can design robust data pipelines, optimize analytical queries, and deliver actionable insights through advanced analytics and visualization tools.
Key Responsibilities
Data Architecture & Design
- Design and implement scalable data architectures using modern data stack technologies
- Develop and maintain data models, schemas, and metadata frameworks
- Create data architecture blueprints for enterprise-wide data solutions
- Establish data governance policies and best practices
Data Engineering
- Build and optimize ETL/ELT pipelines using Python, SQL, and cloud-native tools
- Develop data ingestion frameworks for batch and real-time processing
- Implement data quality checks and monitoring solutions
- Optimize database performance and query execution plans
Cloud Platform Development
- Design and deploy data solutions on AWS (S3, Glue, Athena etc.)
- Implement data platforms on Azure/Microsoft Fabric (Data Factory, Lakehouse)
- Leverage cloud-native services for data storage, processing, and analytics
Analytics & Reporting
- Write complex SQL queries and analytical queries for business intelligence
- Develop interactive dashboards and reports using Power BI or equivalent tools
- Create data models and semantic layers for self-service analytics
- Collaborate with analysts and stakeholders to deliver data-driven insights
Development & Collaboration
- Work with Jupyter notebooks and similar tools for data exploration and prototyping
- Collaborate with cross-functional teams including data scientists, analysts, and engineers
- Document data pipelines, architectures, and technical specifications
- Mentor junior engineers and promote best practices
Required Qualifications
Core Technical Skills
- Python Programming
- Notebooks & Data Exploration
- SQL & Analytical Queries
- AWS Cloud Platform
- Azure / Microsoft Fabric
- Data Architecture & Engineering
Supplementary Skills (Preferred)
- Power BI / Business Intelligence Tools
- Additional Technical Skills
- Experience with Apache Spark (PySpark) for big data processing
- Knowledge of version control systems (Git, GitHub, GitLab)
- Familiarity with CI/CD pipelines for data workflows
- Experience with containerization (Docker, Kubernetes)
- Understanding of machine learning concepts and ML pipelines
- Knowledge of data quality frameworks (Great Expectations, yoda data)
Education and Experience
- Bachelor or Master degree in Computer Science, Information Systems, Data Engineering, or related field
- 5+ years of experience in data engineering, data architecture, or related roles
- 3+ years of hands-on experience with cloud platforms (AWS and/or Azure)
- Proven track record of designing and implementing large-scale data solutions
- Strong problem-solving skills and attention to detail
- Excellent communication and collaboration abilities
- Experience working in Agile/Scrum environments