Job Title: Data Engineer (Azure, Databricks, and Microsoft Fabric)
Location: London ideally but open to Birmingham, Manchester or Newcastle
Duration: Perm
Overview:
We are looking for a highly skilled Data Engineer with expertise in Azure, Databricks, and Microsoft Fabric to design, build, and optimize data pipelines and infrastructure. This permanent role involves working with cutting-edge cloud technologies, enabling scalable and efficient data processing, analytics, and business intelligence solutions.
Key Responsibilities:
* Design, develop, and maintain end-to-end data pipelines using Azure Data Services, Databricks, and Microsoft Fabric.
* Implement Lakehouse architecture using Databricks Delta Lake and Microsoft OneLake.
* Develop and optimize ETL/ELT workflows using Azure Data Factory, Databricks, and Fabric Dataflows.
* Work with Azure Synapse Analytics and Fabric Data Warehouses for large-scale data processing.
* Ensure data quality, security, and governance across cloud platforms.
* Develop and manage real-time and batch data processing using Apache Spark and Databricks workflows.
* Collaborate with data analysts, scientists, and business teams to enable data-driven decision-making.
* Automate data pipelines using Python, SQL, and Spark for efficiency and scalability.
* Monitor and troubleshoot data performance issues, ensuring high availability.
* Stay updated with the latest advancements in Azure, Databricks, and Microsoft Fabric technologies.
Required Skills & Qualifications:
* 3+ years of experience in data engineering with Azure and Databricks.
* Expertise in Azure Data Services (Azure Data Lake, Azure Synapse, Azure SQL).
* Hands-on experience with Databricks (Delta Lake, Spark, MLflow, Notebooks, Workflows).
* Strong knowledge of Microsoft Fabric (OneLake, Data Factory, Synapse, Power BI).
* Proficiency in SQL, Python, PySpark, or Scala for data transformation and automation.
* Strong problem-solving skills and ability to work in a fast-paced, agile environment.
* Stakeholder Management
Priyanka Sharma
Senior Delivery Consultant
Office: 02033759240
Email: psharma@vallumassociates.com