We are seeking a highly skilled Scala Engineer with a passion for building scalable, data-intensive systems. In this role, you will design, develop, and optimize high-performance data pipelines, working closely with Big Data technologies, distributed systems, and cloud platforms. If you enjoy solving complex data challenges and working with large-scale streaming and batch processing systems, we want to hear from you!
Key Responsibilities:
* Design and develop scalable, high-performance data processing applications using Scala.
* Build and optimize ETL pipelines for handling large-scale datasets.
* Work with Apache Spark, Kafka, Flink, and other distributed data frameworks to process massive amounts of data.
* Develop and maintain data lake and warehouse solutions using technologies like Databricks Delta Lake, Apache Iceberg, or Apache Hudi.
* Write clean, maintainable, and well-documented code.
* Optimize query performance, indexing strategies, and storage formats (JSON, Parquet, Avro, ORC).
* Implement real-time streaming solutions and event-driven architectures.
* Collaborate with data scientists, analysts, and DevOps engineers to create robust, scalable, and fault-tolerant data solutions.
* Ensure data governance, security, and compliance best practices in all pipelines.
* Conduct code reviews, mentor junior developers, and contribute to best practices in functional programming.
Required Skills & Qualifications:
* Bachelor’s or Master’s degree in Computer Science Engineering, or a related field.
* 10+ years of experience in software development with Scala.
* Hands-on experience with Apache Spark (batch & streaming) using Scala.
* Experience in developing and maintaining data lake and warehouses using technologies like Databricks Delta Lake, Apache Iceberg, or Apache Hudi.
* Experience working with open-source data formats JSON, Parquet, Avro, etc.
* Deep understanding of distributed computing and parallel processing.
* Experience working on cloud platforms (preferably AWS) and services like S3, EMR, Glue, Redshift, BigQuery.
* Proficiency in CI/CD pipelines, Docker, Kubernetes, and Terraform.
* Knowledge of Java, Python, or other JVM-based languages is a plus.
* Experience with RESTful APIs, GraphQL, and microservices architecture.
* Strong understanding of functional programming concepts (immutability, monads, higher-order functions).
* Experience with SQL and NoSQL databases (PostgreSQL, Cassandra, DynamoDB, HBase, MongoDB).
* Experience with Kafka, Flink, or other streaming frameworks.
* Experience with data security, lineage, and governance tools.
* Experience working in Agile environment.
Preferred Experience:
* Experience of working as a Scala developer/engineer within the technology function of a Global Investment Bank.
* Experience migrating data applications from Hadoop to AWS Databricks.
Seniority Level:
Mid-Senior level
Employment Type:
Full-time
Job Function:
Finance and Information Technology
Industries:
IT Services and IT Consulting and Banking
#J-18808-Ljbffr