Responsibilities:
* Design, deploy, and manage Databricks clusters on Google Cloud Platform (GCP), ensuring optimal performance, scalability, and reliability.
* Provide expertise in Databricks architecture, configuration, and administration, following best practices and industry standards.
* Collaborate with data engineering, data science, and analytics teams to understand their requirements and provide technical guidance and solutions.
* Optimize Databricks workloads and processes for performance, cost-efficiency, and scalability.
* Monitor and troubleshoot Databricks clusters, identifying and resolving issues proactively.
* Implement security measures and ensure compliance with data governance and privacy regulations.
* Design and implement data pipelines and workflows, integrating Databricks with other GCP services and tools.
* Stay up to date with the latest advancements in Databricks, GCP, and data engineering technologies, and evaluate their applicability to our environment.
* Mentor junior team members, providing guidance and support in Databricks administration and architecture.
Requirements:
* Minimum of 5 years of experience as a Databricks Admin/Architect, with a strong focus on Google Cloud Platform (GCP).
* Extensive experience in designing, deploying, and managing Databricks clusters, notebooks, jobs, and libraries.
* Deep understanding of Databricks architecture, configuration, and performance optimization techniques.
* Strong knowledge of GCP services, including Compute Engine, BigQuery, Cloud Storage, Dataflow, and Pub/Sub.
* Proficiency in scripting languages like Python or Scala for automation, terraform, and data engineering tasks.
* Experience with data engineering principles and tools, such as ETL, data pipelines, and data modeling.
* Strong troubleshooting and problem-solving skills, with the ability to identify and resolve complex issues in a timely manner.
* Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams.
* GCP certifications, such as Google Cloud Certified - Data Engineer or Google Cloud Certified - Professional Cloud Architect, are highly desirable.