Description:
The Data Architect / Data Solutions Architect (Databricks) will lead the design, governance, and strategy of the company’s data systems, encompassing the data warehouse, data lake, and integrations between systems. This role ensures the scalability, security, and performance of the data ecosystem, aligning it with business objectives. The ideal candidate combines technical expertise with strategic vision, leveraging tools like Databricks and GCP/Azure to build a robust and future-proof architecture.
Responsibilities:
Key responsibilities include, but are not limited to:
Architecture Design & Strategy
- Design and oversee the architecture of the company’s Data Warehouse and LakeHouse using Databricks and Azure/GCP services.
- Develop and maintain best practices for data modeling, storage, and system integration.
- Define the roadmap for enhancing the data ecosystem, focusing on scalability, innovation, and business alignment.
Strategic Oversight
- Act as a key advisor to business and IT leaders, ensuring data systems support organizational goals.
- Lead initiatives to integrate emerging technologies (e.g., MLFlow, Unity Catalog) into the company’s data strategy.
- Ensure compliance with data governance, security, and privacy regulations.
Data Integration Leadership
- Provide guidance on the design and optimization of ETL/ELT pipelines using Databricks, Delta Live Tables, and Azure Data Factory/Dataflow/DataProc.
- Collaborate with SMEs and analytics teams to streamline data ingestion and enhance reliability.
- Supervise the integration of data from structured, semi-structured, and unstructured sources.
Performance & Cost Optimization
- Oversee resource utilization and performance tuning of Databricks clusters and Spark jobs.
- Implement strategies to reduce cloud costs while maintaining high performance and scalability.
Mentorship & Team Collaboration
- Mentor and guide data engineers, analysts, and developers to implement architectural standards.
- Foster collaboration between cross-functional teams to align data solutions with business needs.
- Drive organization-wide adoption of data management best practices.
Security, Compliance, & Governance
- Establish and enforce data security frameworks, including RBAC, encryption, and secure key management.
- Collaborate with IT and compliance teams to maintain audit readiness and adhere to industry standards.
- Designing and building reference architectures.
- Creating how-to guides and demo applications.
- Integrating Databricks with 3rd-party applications.
- Guiding customers through evaluating and adopting Databricks.
- Supporting customer operational issues.
- Working with Databricks technical teams, project managers, and customer teams.
- Providing product and implementation feedback.
Qualifications & Requirements:
- Advanced experience with Databricks features, such as Unity Catalog, MLFlow, and Delta files.
- Strong understanding of data architecture principles, data modeling, and data integration.
- Proficiency in Databricks Unified Data Analytics Platform, including Spark, Delta Lake, and MLFlow.
- Experience with cloud platforms (e.g., Azure, AWS, GCP).
- Knowledge of data security and compliance best practices.
- Experience with data pipelines and ETL processes.
- Experience in Financial industry is nice to have