Please note that this role requires security clearance at SC level. You must be SC cleared to be considered for the role.
Tasks and Responsibilities:
Engineering:
* Ingestion configuration.
* Write python/pyspark and spark SQL code for validation/curation in notebook.
* Create data integration test cases.
* Implement or amend worker pipelines.
* Implement Data validation/curation rules.
* Convert data model data mapping to technical data mapping.
* Implement Integrations.
* Data migration implementation and execution.
* Data analysis (profiling etc).
* Azure DevOps/GitHub configurations for ADF and Databricks code.
Our ideal candidate:
Strong experience in designing and delivering Azure-based data platform solutions, technologies including:
1. Azure ADF and Databricks, Azure Functions, App Service, Logic App, AKS, Azure App Service, Web App.
2. Good knowledge in real-time streaming applications, preferably with experience in Kafka real-time messaging or Azure Functions, Azure Service Bus.
3. Spark processing and performance tuning.
4. File formats partitioning (e.g., Parquet, JSON, XML, CSV).
5. Azure DevOps/GitHub.
* Hands-on experience in at least one of Python with knowledge of the others.
* Experience of synchronous and asynchronous interface approaches.
* Knowledge in data modeling (3NF/Dimensional modeling/Data Vault2).
* Work experience in agile delivery.
* Able to provide comprehensive documentation.
* Able to set and manage realistic expectations for timescales, costs, benefits, and measures for success.
Nice to Have:
* Experience with integration and implementation of data cataloging tools like Azure Purview/Collabra.
* Experience in implementing and integrating visualization tools like Power BI/Tableau, etc.
* Experience in C# application development (ASP.NET MVP).
#J-18808-Ljbffr