Senior Data Engineer - Python / Data Pipelines / Data Platform / AWS - is required by fast growing, highly successful and tech focused organisation.
About the job
You will play a crucial role in designing, building, and maintaining their data platform, with a strong emphasis on streaming data, cloud infrastructure, and machine learning operations.
Key Responsibilities:
1. Architect and Implement Data Pipelines:
* Design, develop, and maintain scalable and efficient data pipelines
* Optimize ETL processes to ensure seamless data ingestion, processing, and integration across various systems
2. Streaming Data Platform Development:
* Lead the development and maintenance of a real-time data streaming platform using tools like Apache Kafka, Databricks, Kinesis.
* Ensure the integration of streaming data with batch processing systems for comprehensive data management
3. Cloud Infrastructure Management:
4. Utilize AWS data engineering services (including S3, Redshift, Glue, Kinesis, Lambda, etc.) to build and manage our data infrastructure
5. Continuously optimize the platfor...