Job Title: Data Engineer
Location: Remote
Contract: £500 Per Day
Role Overview
Our client is seeking a Data Engineer with a solid foundation in both data engineering and business intelligence, capable of understanding and addressing business needs. This role involves managing and optimizing AWS-based data processes, orchestrating data flows, and supporting business intelligence capabilities. The ideal candidate will not only bring technical expertise but also a strong business acumen to facilitate effective collaboration with non-technical stakeholders.
Key Responsibilities
* Data Pipeline Management:
* Database Integration: Maintain MySQL databases on AWS (approx. 25TB) as primary data sources feeding the data pipeline.
* Real-Time Data Ingestion: Utilize Apache Kafka to manage data flows from MySQL into AWS S3 buckets, enabling real-time streaming.
* Data Processing: Implement ETL processes using AWS Glue and PySpark to build data lakes within S3 for further analysis.
* Business Intelligence (BI):
* Facilitate data availability for BI purposes, supporting daily updates and aiming to move from overnight batch jobs to near-real-time 15-minute refreshes.
* Collaborate on BI dashboard and reporting creation to provide actionable insights.
* Complex Data Reconciliation:
* Reconcile transaction records across multiple merchants, currencies, and network operators, addressing both straightforward and complex data relationships.
Technical Requirements
* Data Engineering Expertise:
* Proficiency with MySQL, Apache Kafka, AWS Glue, PySpark, and AWS S3 for managing data pipelines and ETL processes.
* Business Intelligence Skills:
* Experienced in designing and maintaining BI reports and dashboards with frequent data refresh capabilities.
* Data Reconciliation Experience:
* Skilled in reconciling transactional data across diverse systems, operators, and currencies to ensure data accuracy.
Business & Communication Skills
* Business Acumen:
* Ability to translate business requirements into data solutions and work collaboratively with non-technical stakeholders to develop reports and dashboards tailored to business needs.
* Communication:
* Strong communication skills to bridge technical and business needs, working closely with stakeholders to ensure data solutions support strategic objectives.