Senior Data Engineer - Python / Data Pipelines / Data Platform / AWS - is required by fast growing, highly successful and tech focused organisation.
About the job
You will play a crucial role in designing, building, and maintaining their data platform, with a strong emphasis on streaming data, cloud infrastructure, and machine learning operations.
Key Responsibilities:
* Architect and Implement Data Pipelines:
* Design, develop, and maintain scalable and efficient data pipelines
* Optimize ETL processes to ensure seamless data ingestion, processing, and integration across various systems
* Streaming Data Platform Development:
* Lead the development and maintenance of a real-time data streaming platform using tools like Apache Kafka, Databricks, Kinesis.
* Ensure the integration of streaming data with batch processing systems for comprehensive data management
* Cloud Infrastructure Management:
* Utilize AWS data engineering services (including S3, Redshift, Glue, Kinesis, Lambda, etc.) to build and manage our data infrastructure
* Continuously optimize the platform for performance, scalability, and cost-effectiveness
* Communications:
* Collaborate with cross-functional teams, including data scientists and BI developers, to understand data needs and deliver solutions
* Leverage the project management team to coordinate project, requirements, timelines and deliverables, allowing...