Senior Data Engineer opportunity to join a well-known financial services company working on a Remote first basis (once a month in the office on average).
Senior Data Engineer - Remote First (offices in lake district) - Up to £65,000 + Great Benefits
Purpose of Job
Advise on and ensure the maximum return of value from the company's data assets; through use of best practice data engineering techniques that aid the delivery of the company's data management strategy and roadmap and align to regulatory requirements.
Contribute towards the support, maintenance, and improvement of the company's platform inc data pipelines, and ensure data quality and governance.
Responsibilities
1. Advise on the Design, implementation, and maintenance of complex data engineering solutions.
2. Ensure the implementation, and maintenance of complex data engineering solutions to acquire and prepare data and that it adheres to Best Practice. (Extract, Transform, Load)
3. Contributes towards and ensures the delivery of the creation and maintenance of data pipelines to connect data within and between data stores, applications, and organisations.
4. Contributes towards and ensures the delivery of complex data quality checking and remediation.
5. Identifying data sources, data processing concepts and methods
6. Evaluating, designing and implementing on-premise, cloud-based and hybrid data engineering solutions
7. Structuring and storing data for uses including - but not limited to - analytics, machine learning, data mining, sharing with applications and organisations.
8. Harvesting structured and unstructured data
9. Integrating, consolidating, and cleansing data
10. Migrating and converting data
11. Applying ethical principles and regulatory requirements in handling data
12. Ensuring appropriate storage of data in line with relevant legislation, and the company requirements.
13. Guiding and contributing towards the development of junior & trainee Data Engineers.
14. Providing Technical guidance to Data Engineers
Knowledge, Experience and qualifications
15. Excellent level of knowledge & experience in Azure Data, including DataLake, Data Factory, DataBricks, Azure SQL (Indicative experience = 5yrs+)
16. Build and test processes supporting data extraction, data transformation, data structures, metadata, dependency and workload management.
17. Knowledge on Spark architecture and modern Datawarehouse/Data-Lake/Lakehouse techniques
18. Build transformation tables using SQL.
19. Moderate level knowledge of Python/PySpark or equivalent programming language.
20. PowerBI Data Gateways and DataFlows, permissions.
21. Creation, utilisation, optimisation and maintenance of Relational SQL and NoSQL databases.
22. Experienced working with CI/CD tooling such as Azure DevOps/GitHub including repos, source-control, pipelines, actions.
23. Awareness of Informatica or similar data governance tools (Desirable)
24. Experience of working in agile (SCRUM) and waterfall delivery-type teams.
25. Experienced with Confluence & Jira.
26. Experience in a financial services or other highly regulated environment.
27. Any experience or interest in AI is desirable