- Sector: Public sector / Government - Clearance required: SC clearance - Contract length: 6 months - Rate: 400-500/d - IR35 status: Outside - Onsite requirement: 2.d a week - Role title: Data engineer Job Description: · Build streaming data ingestion pipelines using AWS MSK and batch data ingestion pipelines using APIs · Prepare data for prescriptive & predictive modelling. · Experience working with various data formats e.g., parquet, avro, json · Experience in building ETL pipelines using Lambda, Step Functions and Glue · Experience working with various AWS database services (e.g., RDS, DynamoDB and Redshift) · Technical expertise with various data modelling techniques · Hands-on experience with Python and SQL · Kafka Streaming data experience to build the new data pipelines. Supporting Skills · 5 years' experience in an iterative and Agile development environment · Appreciation of Software Development Environments & Languages · Appreciation of Business Analysis Activities Appreciation of other scripting language, eg JavaScript, Lua, Perl, PHP, Unix