Job Title: Senior Data Engineer
Job Summary:
We're seeking a skilled Senior Data Engineer to join our team at NRG. As a key member of our data engineering group, you'll play a pivotal role in designing, developing, and maintaining our cloud-based data infrastructure.
Key Responsibilities:
* Design and implement end-to-end data pipelines on Microsoft Azure.
* Develop, optimize, and maintain Azure Data Lake, Azure Synapse, Azure Databricks, and other cloud-based data platforms.
* Implement data transformation processes using Azure Data Factory and Databricks to support analytics and business intelligence needs.
* Ensure high performance, availability, and scalability of our data pipelines and systems.
* Collaborate with data scientists, analysts, and other stakeholders to deliver clean, reliable, and accessible data.
* Manage and monitor cloud resources, ensuring security, compliance, and best practices in data governance.
* Automate workflows and optimize data processes to minimize manual intervention and improve efficiency.
* Lead data engineering initiatives, mentoring junior team members, and sharing best practices in data architecture and development.
Requirements:
* 5+ years of experience in data engineering, with a focus on cloud-based solutions.
* Strong expertise in Microsoft Azure services such as Azure Data Factory, Azure Synapse Analytics, Azure Data Lake, Azure Databricks, and Azure SQL.
* Proficiency in SQL, Python, or Scala for data processing.
* Experience with ETL/ELT design patterns and building scalable, distributed data architectures.
* Deep understanding of data modelling, data warehousing, and data integration best practices.
* Knowledge of Azure DevOps and infrastructure automation tools like Terraform or ARM templates.
* Strong problem-solving skills and ability to work independently or in a team.
* Experience with CI/CD pipelines and containerization technologies such as Docker and Kubernetes is a plus.
Preferred Qualifications:
* Experience with big data technologies like Spark, Hadoop, or Kafka.
* Knowledge of data governance, security, and compliance frameworks in the cloud.
* Azure certifications such as Azure Data Engineer Associate or Azure Solutions Architect.