Description:
Lead Databricks Engineer
Type of hiring – Subcon/Fulltime
Job Description:
Key Responsibilities:
- Data Pipeline Development
- Develop and fine tune data processing pipelines within Databricks.
- Implement performance optimization techniques.
- Integration with Analytics Teams
- Work closely with data science and analytics teams to integrate data solutions.
- Support data architecture initiatives that incorporate Databricks technologies.
- System Support and Troubleshooting
- Provide support for Databricks based tools.
- Resolve operational and performance issues, ensuring system reliability.
- Documentation and Knowledge Sharing
- Document technical processes and system behaviors.
- Lead knowledge transfer sessions to train team members on data processing techniques and tools.
- Governance and Security Implementation
- Establish and enforce data governance and security protocols within Databricks.
- Ensure compliance with regulatory requirements related to data management.
Required Qualifications
- Bachelor’s Degree in Computer Science, Information Technology, Engineering, or a related field.
- Strong hands-on experience with Databricks and Apache Spark.
- Proven experience building and optimizing large-scale ETL/ELT data pipelines.
- Experience working with cloud platforms such as AWS, Azure, or GCP.
- Solid understanding of data architecture, distributed systems, and performance tuning.
- Experience implementing data governance and security controls.
- Strong troubleshooting, analytical, and problem-solving skills.
- Excellent communication skills with the ability to collaborate across technical and business teams.