Lead Data Engineer (AWS)
Permanent - Up to £108,000 per annum plus excellent benefits and pension scheme.
Fully remote working
Digital Skills are working with a growing tech company that builds and provides tools and services to a range of end clients. The company specializes in designing, building and continuously improving data-enabling products within highly governed sectors.
We are currently recruiting for a Data Engineer who will assist in the development of a secure batch/Real Time data platform for a number of clients leveraging the latest commercial and open-source technologies.
This involves getting data from a variety of different sources, getting it into the right format, assuring alignment to target models, and ensuring that data can be made available to downstream consumers.
Responsibilities;
* Code, test, and document new or modified data pipelines that meet functional/non- functional business requirements
* Conduct logical and physical database design
* Expand and grows data platform capabilities to solve new data and analytics problems
* Conduct data analysis, identifying feasible solutions and enhancements to data processing challenges
* Ensure that data models are consistent with the data architecture (eg entity names, relationships and definitions)
* Perform root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement
Qualifications needed;
* Data pipeline development using data processing technologies and frameworks
* Agile or other rapid application development methods
* Datamodelling and understanding of different data structures and their benefits and limitations under particular use cases
* Experience in Public Cloud services, such as AWS. Practical experience with core services such as EC2, RDS, Lambda, Athena & Glue would be even better!
* Configuring and tuning Relational and NoSQL databases, including both query processing and query planning, or other data processing infrastructure
* Programming or Scripting languages, such as Python
* Exposure to Kafka would be ideal.
* Test Driven Development with appropriate tools and frameworks
Please don't hesitate to apply if this role seems right for you.