We have an exciting job opportunity for AWS Data Engineer role at Telford, UK - Permanent.
Title: AWS Data Engineer
Location: Telford, UK
Duration: Permanent
Job Description:
* Design, develop, and maintain Real Time data streaming pipelines using AWS Kinesis for processing and analysing high-velocity data streams.
* Collaborate with stakeholders to gather requirements and design efficient data streaming architectures for Real Time analytics, monitoring, and alerting.
* Build and optimize Kinesis Data Streams, Kinesis Data Firehose, and Kinesis Data Analytics to process Real Time data from various sources.
* Develop and implement ETL processes to transform, enrich, and aggregate streaming data for downstream consumption in data lakes or data warehouses such as AWS S3, Redshift, and DynamoDB.
* Integrate Kinesis with other AWS services like Lambda, S3, and EMR for Real Time data ingestion, processing, and storage.
* Optimize streaming applications for performance and scalability, ensuring efficient resource usage and minimal latency.
* Develop and maintain robust error handling, data validation, and retry mechanisms within Kinesis pipelines to ensure data consistency and reliability.
* Implement security best practices for data streaming, including encryption in transit and at rest, access control using IAM roles and policies, and data retention policies.
* Monitor and troubleshoot AWS Kinesis streams using CloudWatch metrics, alarms, and logs, ensuring high availability and minimal downtime.
* Collaborate with DevOps teams to automate the deployment and scaling of streaming pipelines using AWS CloudFormation, Lambda, and CI/CD tools.
* Work closely with data analysts and data scientists to ensure Real Time data is processed and delivered for immediate insights and reporting.
* Implement data partitioning, sharding, and key management strategies to handle high-throughput data streams and ensure scalability.
* Optimize the cost of Real Time data processing by tuning the configurations of Kinesis and associated AWS services like Lambda and Firehose.
* Stay current with AWS updates and best practices for streaming data, recommending improvements to the architecture as new features and services become available.
* Provide guidance and mentorship to junior engineers in Real Time data processing, ensuring best practices in streaming technologies and AWS cloud services.
* Create and maintain detailed technical documentation for data pipelines, including data flow diagrams, stream processing logic, and operational guidelines