Job Opportunity: Data Engineer (Databricks Specialist)
Considering making an application for this job Check all the details in this job description, and then click on Apply.
Location: Manchester | Employment Type: Full-time.
Are you a passionate Data & AI expert with a strong Databricks background? A leading consulting firm is seeking a person skilled in Data Engineering to join their growing team. In this senior role, you’ll work with clients to design and implement scalable, high-performance Databricks-based platforms that drive impactful business outcomes
Key Responsibilities
As a Data Engineer, you will be responsible for designing, implementing, and optimizing data workflows that drive critical insights across the business. You'll work with cutting-edge technologies to build scalable, high-performance data solutions that meet the evolving needs of our clients.
* Data Pipeline Development: Design and implement robust ETL/ELT workflows using Databricks to process and transform large datasets efficiently, enabling real-time insights and decision-making.
* Data Integration: Develop scalable data ingestion processes from multiple sources and integrate them into Delta Lake or data lakes for unified storage and analysis.
* Collaboration: Partner closely with data architects, data scientists, and business analysts to understand requirements and deliver data solutions that align with business objectives and enhance analytics capabilities.
* Performance Optimization: Optimize Databricks workflows for performance, scalability, and cost efficiency, including Spark tuning and cluster management.
* Cloud Integration: Build cloud-native data engineering solutions on leading cloud platforms like Azure, AWS, or GCP to ensure seamless, secure, and efficient data management.
* Governance & Quality: Implement and maintain data governance best practices, ensuring data consistency, quality, and security across the organization.
* Automation: Automate repetitive tasks, develop robust CI/CD pipelines, and streamline data workflows to improve operational efficiency and reduce manual effort.
* Documentation: Create and maintain clear, detailed documentation of processes, workflows, and architectures to ensure knowledge sharing and promote best practices within the team.
Skills & Qualifications
Required:
* Experience:
* 3+ years of experience in data engineering, with hands-on expertise in Databricks.
* Proven ability to deliver large-scale data pipelines in consulting or enterprise environments.
* Technical Expertise:
* Proficiency in Databricks, Delta Lake, and Apache Spark.
* Strong programming skills in Python and SQL, with optional experience in Scala.
* Experience with cloud platforms such as Azure (preferred), AWS, or GCP.
* Strong understanding of relational and non-relational databases, data lakes, and real-time data processing.
* Soft Skills:
* Strong problem-solving skills and a proactive approach to addressing complex challenges.
* Excellent communication skills and the ability to collaborate effectively with cross-functional teams.
Preferred:
* Experience with tools like MLflow or other AI/ML frameworks.
* Familiarity with data visualization tools such as Power BI or Tableau.
* Certification in Databricks or cloud platforms (e.g., Azure Data Engineer Associate).
* Experience in automation and CI/CD practices for data engineering pipelines.
Why Join Us?
* Challenging Projects: Work on diverse and impactful projects for clients across various industries, contributing to real-world data solutions.
* Growth Opportunities: Gain exposure to cutting-edge technologies and continue developing your technical expertise within a fast-paced, innovative environment.
* Supportive Culture: Join a collaborative team that values creativity, innovation, and continuous learning.
* Competitive Package: We offer an attractive salary, comprehensive benefits, and flexible working options, including remote work opportunities.
If you're passionate about data engineering, enjoy tackling complex challenges, and want to be part of a forward-thinking team, we want to hear from you!