Staines, United Kingdom | Posted on 18/11/2024
Job Title: Data Engineer
We are seeking an experienced Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, with expertise in SQL, Python, and Microsoft Azure. You will be responsible for designing, building, and maintaining robust data pipelines and storage solutions to support our data-driven initiatives. This role requires a minimum of 5 years of experience in data engineering and 3 years of hands-on experience with the specified technologies.
Key Responsibilities
1. Design, develop, and optimize data pipelines to support data integration and processing workflows.
2. Write efficient and scalable queries using SQL to extract, transform, and load data.
3. Develop and maintain data processing scripts and tools using Python.
4. Leverage Microsoft Azure services such as Azure Data Factory, Azure Databricks, Azure SQL Database, and Azure Data Lake to build cloud-based data solutions.
5. Ensure data quality and integrity through testing, monitoring, and validation processes.
6. Collaborate with stakeholders to gather requirements, translate them into technical solutions, and deliver actionable insights.
7. Implement best practices in data security, governance, and compliance in alignment with organizational standards.
8. Monitor and troubleshoot data workflows to ensure optimal performance and reliability.
Qualifications and Skills
1. 5+ years of experience as a Data Engineer, with a track record of delivering data solutions in complex environments.
2. 3+ years of hands-on experience in:
1. Writing complex queries and optimizing performance in SQL.
2. Developing data processing scripts and solutions in Python.
3. Designing and implementing data workflows using Microsoft Azure services.
3. Proficient in building scalable data architectures and handling large datasets.
4. Strong problem-solving skills and ability to troubleshoot data pipeline and system issues.
5. Experience working with Agile methodologies.
6. Knowledge of data visualization tools such as Power BI or Tableau.
7. Familiarity with DevOps practices, including CI/CD pipelines for data workflows.
8. Certifications in Microsoft Azure or related technologies.
9. Understanding of distributed computing frameworks such as Spark.
#J-18808-Ljbffr