Our client, a leading insurer, is seeking a Senior Data Engineer with strong expertise in Azure Databricks to join their team. This position focuses on building and optimizing scalable data pipelines, enhancing data workflows, and ensuring data reliability and performance.
The ideal candidate will bring hands-on experience with Azure Databricks, Delta Lake, and Apache Spark, along with strong coding skills in Python or Scala. This role presents an opportunity to work on cutting-edge data engineering challenges within a dynamic, technology-driven organization.
Key Responsibilities
Data Architecture & Engineering:
- Develop and maintain end-to-end data pipelines using Azure Databricks, Delta Lake, and other Azure data services.
- Architect and implement scalable, high-performance data solutions to support business needs.
- Apply best practices for data modeling, using both relational and dimensional techniques.
Platform Development and Ownership:
- Act as the technical lead for the organization's cloud-based analytics platform, encompassing the data warehouse, data lake, and operational data stores.
- Partner with cross-functional stakeholders to ensure the platform supports analytics, reporting, and decision-making requirements.
Mentorship and Collaboration:
- Guide and mentor junior engineers to foster technical growth within the team.
- Collaborate with business analysts, architects, and other team members to define and deliver data solutions.
Data Quality and Transformation:
- Design and implement ETL/ELT workflows to transform raw data into clean, usable datasets.
- Ensure data quality by implementing validation, monitoring, and governance practices.
Cloud Technology SME:
- Stay informed on emerging cloud technologies and incorporate them into the organization's data strategy
Required Qualifications
Technical Expertise
Azure Cloud Expertise:
- Deep knowledge of Azure Databricks, Azure Data Factory, Azure Data Lake, Azure SQL, and related Azure services.
- Experience in ETL frameworks and data pipeline orchestration.
Programming and Data Processing:
- Proficiency in Python and PySpark for developing scalable data solutions.
- Advanced knowledge of SQL for querying and transforming data.
- Experience with distributed computing frameworks such as Apache Spark.
Data Modeling and Transformation:
- Experience designing and implementing data models using dimensional and relational approaches.
- Strong knowledge of data transformation techniques for building analytics-ready datasets.
Automation and DevOps Integration:
- Experience with DevOps, version control, and CI/CD pipelines.
Experience
- Bachelor's degree in Computer Science, Data Science, or a related field.
- 8+ years of experience in data engineering, with a focus on cloud-based solutions.
- Proven experience delivering enterprise-scale data platforms, including data lakes and warehouses.
- Exposure to end-to-end software development lifecycle (SDLC), from design to deployment.
