Job Title - Lead Data Engineer (SC Cleared)
Mainly remote - may require occasional onsite visits to London
Duration - 12 Months
Rate - Up to £500 per day (Outside IR35)
Overview:
We are seeking an experienced and highly skilled Lead Data Engineer with active SC clearance.
Required Skills and Experience:
* Active SC clearance is mandatory
* Proven experience working within public sector or central government environments, understanding the unique challenges and compliance requirements.
* AWS, Apache Spark, Kafka Stream, Glue and Kubernetes.
* Develop scalable cloud-based data architectures, data lakes, and pipelines for structured/unstructured data.
* Ensure data availability, consistency, and optimization across ingestion-to-consumption stages using ETL/ELT processes.
* Deploy data solutions on AWS
* Experience managing teams of data engineers
* Strong experience with Apache Spark or similar big data processing frameworks.
* Solid background in designing and implementing ETL pipelines, data modeling, and managing large datasets.