Data Engineer
An exciting opportunity has arisen for a skilled Data Engineer to support a government department in transitioning a successful proof-of-concept (PoC) business intelligence and data warehousing solution into a fully operational, production-ready platform.
Over a four-year period, a PoC was developed to explore technology options aligned with the department’s long-term digital strategy. Through agile delivery and close collaboration, the organisation significantly improved its understanding of internal data assets and the Microsoft technology ecosystem, enhancing its reporting capabilities and data-driven decision-making.
The next phase of this programme will involve expanding and maturing the platform into a scalable, secure, and well-supported system. This will be delivered through phased agile delivery cycles, with one or more specialist teams contributing to key technical milestones along the roadmap.
Key Responsibilities:
The successful candidate will be responsible for designing and implementing robust data engineering solutions within a Microsoft Azure environment. Key tasks will include optimising data pipelines, enhancing security configurations, and contributing to the overall architecture of the enterprise data platform.
Technical Skills:
Azure Data Factory (ADF):
1. Incremental creation of data dimensions to optimise processing, reduce overheads, and improve cost efficiency across compute, storage, and networking.
2. Enhancing pipeline security by replacing basic authentication with managed identities or service principals.
3. Introducing parallelism into pipeline workflows to improve performance and scalability.
4. Refactoring existing pipelines into reusable templates to support the onboarding of new datasets.
Azure SQL:
1. Integrating database access with Microsoft Entra ID for improved identity and access management.
2. Designing and implementing STAR schema models to support analytical workloads.
3. Defining consistent naming conventions across all database objects.
4. Migrating data processing logic from Power BI into Azure SQL for better scalability and maintainability.
5. Segregating datasets into separate schemas or views to improve manageability and performance.
6. Applying data protection.
Desirable:
1. Previous experience working on public sector or government data programmes.
2. Familiarity with agile delivery in a multi-team or scaled agile environment.
#J-18808-Ljbffr