Role Type: Permanent Location: Telford, UK Hybrid – 2 days’ work from Telford office Salary: 70K GBP/Annum AWS Data Engineer • Design, develop, and maintain real-time data streaming pipelines using AWS Kinesis for processing and analyzing high-velocity data streams. • Collaborate with stakeholders to gather requirements and design efficient data streaming architectures for real-time analytics, monitoring, and alerting. • Build and optimize Kinesis Data Streams, Kinesis Data Firehose, and Kinesis Data Analytics to process real-time data from various sources. • Develop and implement ETL processes to transform, enrich, and aggregate streaming data for downstream consumption in data lakes or data warehouses such as AWS S3, Redshift, and DynamoDB. • Integrate Kinesis with other AWS services like Lambda, S3, and EMR for real-time data ingestion, processing, and storage. • Optimize streaming applications for performance and scalability, ensuring efficient resource usage and minimal latency. • Develop and maintain robust error handling, data validation, and retry mechanisms within Kinesis pipelines to ensure data consistency and reliability. • Implement security best practices for data streaming, including encryption in transit and at rest, access control using IAM roles and policies, and data retention policies. • Monitor and troubleshoot AWS Kinesis streams using CloudWatch metrics, alarms, and logs, ensuring high availability and minimal downtime. • Collaborate with DevOps teams to automate the deployment and scaling of streaming pipelines using AWS CloudFormation, Lambda, and CI/CD tools. • Work closely with data analysts and data scientists to ensure real-time data is processed and delivered for immediate insights and reporting. • Implement data partitioning, sharding, and key management strategies to handle high-throughput data streams and ensure scalability. • Optimize the cost of real-time data processing by tuning the configurations of Kinesis and associated AWS services like Lambda and Firehose. • Stay current with AWS updates and best practices for streaming data, recommending improvements to the architecture as new features and services become available. • Provide guidance and mentorship to junior engineers in real-time data processing, ensuring best practices in streaming technologies and AWS cloud services. • Create and maintain detailed technical documentation for data pipelines, including data flow diagrams, stream processing logic, and operational guidelines.