PERMANENT ROLE WITH SMART DCC
Based in Manchester
Competitive Salary plus benefits
Role
In our Data and Analytics team, we're focused on using data to drive our organization forward. Our organization is in search of a DataOps Engineer, tasked with revolutionizing and streamlining our data workflows, pipelines, and architectures to bolster our advanced analytics, machine learning, and artificial intelligence efforts. In this critical role, you're expected to harness your engineering prowess to enhance data process efficiencies, ensuring data is seamlessly, reliably, and efficiently managed across our analytics platforms. Your contributions will be vital in enabling quick access to insights, promoting a culture of continuous improvement, and driving innovation within our data practices.
As a DataOps Engineer, your responsibilities will span the development and implementation of automated solutions for data integration, quality control, and continuous delivery. This role demands a solid grounding in software engineering principles, fluency in programming languages such as Python or Scala, and an adeptness with DevOps tools. You'll play a crucial role in constructing and maintaining sophisticated data pipelines that support the organization's data science and analytics ambitions.
Collaboration is a cornerstone of this position. You will work closely with teams across the organization, assimilating their data requirements and challenges, and crafting agile, robust data solutions. Your efforts in implementing best practices in DataOps will aim to eliminate bottlenecks, elevate data quality, and ensure that data management processes are in tight alignment with our strategic analytics and decision-making objectives.
In this role, automating data pipelines and implementing scalable solutions will be just the beginning. You will also ensure data availability and integrity through effective governance, advocate for DataOps methodologies alongside IT and data teams, and continuously monitor, troubleshoot, and optimize data systems for superior performance.
This DataOps Engineer position is perfectly suited for those with a passion for leveraging technology to tackle data challenges, a desire to enhance data analytics capabilities, and a drive to influence business outcomes through streamlined data operations. If you're eager to contribute to the data infrastructure backbone of our organization and possess a robust background in software engineering and data pipeline automation, this role offers a compelling opportunity to advance within our Data and Analytics team.
What will you be doing?
1. Detailed ETL Expertise: Design and implement complex ETL processes to efficiently extract, transform, and load data from disparate sources into target data stores, ensuring data quality and integrity.
2. Streaming Data Processing: Develop real-time data processing pipelines using technologies like Apache Kafka or cloud-native streaming platforms, enabling low-latency data ingestion and analysis for time-sensitive applications.
3. Batch Processing Optimization: Optimize batch processing workflows using distributed computing frameworks such as Apache Spark or Apache Flink, fine-tuning parallelism, resource allocation, and data partitioning strategies for performance and scalability.
4. Infrastructure as Code (IaC): Implement infrastructure automation using tools like Terraform or Ansible to provision, configure, and manage cloud resources, data clusters, and containerized environments, ensuring reproducibility and consistency across environments.
5. Adopting Cloud-Native Services: Evaluate, select, and integrate cloud-native services and technologies (e.g., AWS, Azure) to address DataOps challenges efficiently and cost-effectively to reduce effort and improve consistency.
6. Integration with Data Lakes and Data Warehouses: Integrate cloud-native data services with existing data lakes, data warehouses, and analytics platforms to establish seamless data pipelines, support hybrid architectures, and enable cross-cloud data integration for multi-cloud deployments.
7. Continuous Integration and Deployment (CI/CD):
8. Automated Testing: Develop automated test suites and integration tests for data pipelines and analytics applications, validating data transformations, business logic, and data quality constraints throughout the CI/CD pipeline.
9. Pipeline Orchestration: Orchestrate CI/CD pipelines using tools like Jenkins, GitLab CI/CD, or Apache Airflow, automating build, test, and deployment tasks for data engineering artifacts and workflows.
10. Monitoring and Alerting:
11. Comprehensive Monitoring: Implement comprehensive monitoring solutions using tools like Prometheus, Grafana, or Datadog to monitor data pipeline health, performance metrics, and system resource utilization in real-time.
12. Anomaly Detection: Configure anomaly detection algorithms and threshold-based alerting mechanisms to proactively identify and address issues such as data latency, pipeline failures, or resource contention.
13. Cross-functional Collaboration: Collaborate with DevOps teams, data engineers, data scientists, and business stakeholders to align DataOps practices with broader DevOps principles, fostering a culture of collaboration, transparency, and shared responsibility across teams.
14. Agile Methodologies: Embrace agile methodologies such as Scrum or Kanban, participating in sprint planning, backlog grooming, and retrospective meetings to prioritize tasks, iterate on development efforts, and adapt to evolving requirements.
15. Performance Optimization:
16. Query Optimization: Profile and optimize SQL queries, data processing workflows, and distributed computing jobs to improve query performance, reduce latency, and minimize resource utilization across data infrastructure components.
17. Resource Management: Monitor and optimize resource allocation, container resource limits, and cluster configurations to maximize resource utilization, minimize costs, and ensure optimal performance under varying workloads.
18. Professional Development: Stay abreast of industry trends, emerging technologies, and best practices in DataOps through continuous learning, participation in conferences, webinars, and online communities, and pursuing relevant certifications.
19. Process Optimization: Identify opportunities for process improvements, tool enhancements, and automation initiatives to streamline DataOps workflows, reduce cycle times, and enhance productivity, leveraging feedback from stakeholders and retrospectives to drive iterative improvements.
What are we looking for?
* Advanced proficiency in database technologies such as SQL Server, Oracle, MySQL, or PostgreSQL for data management and querying.
* Expertise in implementing and managing data pipelines.
* Strong understanding of data warehousing concepts, data modelling techniques, and schema design for building and maintaining data warehouses or data lakes.
* Proficiency in cloud platforms such as AWS, Azure, or Google Cloud for deploying and managing scalable data infrastructure and services.
* Knowledge of DevOps principles and practices for automating infrastructure provisioning, configuration management, and continuous integration/continuous deployment (CI/CD) pipelines.
* Strong scripting and programming skills in languages like Python, Bash, or PowerShell for automation, data manipulation, and orchestration tasks.
* Ability to collaborate with cross-functional teams including data engineers, data scientists, and business stakeholders to understand requirements, design data solutions, and deliver projects.
* Excellent communication skills to effectively convey technical concepts to non-technical stakeholders and collaborate with team members.
* Strong problem-solving skills to troubleshoot data issues, optimize performance, and improve reliability of data pipelines and infrastructure.
* Ability to stay updated with emerging technologies, trends, and best practices in the field of DataOps and data engineering.
* Initiative and drive to continuously improve skills, automate repetitive tasks, and streamline data operations processes for increased efficiency and productivity.
* University degree or equivalent experience in business or a STEM subject.
* Experience in a relevant regulated industry.
About the DCC:
At the DCC, we believe in making Britain more connected, so we can all lead smarter, greener lives. That desire to make a difference is what drives us every day and it wouldn’t be possible without our people. Each person at the DCC brings a special kind of power to the business, and if you join us, we’ll give you the means to unleash yours. Here, we depend on each other and hold each other accountable. You have the power to challenge and make change, to take the initiative and enjoy real responsibility. Whether it’s doing purposeful work, helping us grow or building the career you want – we’ll give you the support to do it all. Our secure network for smart meters is transforming Britain’s energy system and helping the country’s fight against climate change: we want you to be part of our journey.
Company benefits:
The DCC’s continued success depends on our people. It’s important to us that you enjoy coming to work, and feel healthy, happy and rewarded. In this role, you’ll have access to a range of benefits which you can choose from to create a personalized plan unique to your lifestyle.
If there are any questions you’d like to ask before applying, please contact Stephanie.Owen@peregrineresourcing.com to complete your application, so we can learn more about you. Your application will be carefully considered, and you’ll hear from us regarding its progress.
Join the DCC and discover the power of you.
What to do now
Choose ‘Apply now’ to fill out our short application, so that we can find out more about you. If you have any questions you’d like to ask before applying, please contact Stephanie.Owen@peregrineresourcing.com.
#J-18808-Ljbffr