What you do:

The Senior Data Engineer – AI & Analytics Platform will be a key role in driving Allianz Life’s AI and Analytics capabilities on the Azure/Databricks platform. You’ll leverage your hands-on data engineering and Databricks expertise to build and maintain data pipelines and automation that power advanced analytics and AI solutions. You’ll have a hand in defining and executing data ingestion and data architecture with a focus on operational excellence, security, and performance. Collaboration will be key, as you partner with IT, data science, analytics engineering, and vendor teams to move and manage data on a robust platform that accelerates innovation and business value.

Data Engineering & Operations (60%)

Develop, deploy, and maintain platform components, including: data connections, data pipelines, Databricks Lakeflow jobs, related GitHub Actions and Databricks Declarative Automation Bundle (DABs) pipelines, and various monitoring frameworks.

Create and maintain data engineering templated project repositories enabled and simplified for the internal platform team with custom CLI. Create/manage Databricks DAB Infrastructure as Code repositories as needed.

Ensure high availability, performance, and observability of data pipelines and data quality & cost monitoring through proactive monitoring and incident response.

Maintain team documentation for the internal platform and for external business users of the platform (i.e. developers).

Collaboration & Governance (30%)

Mentor junior data engineers within the internal team, including reviewing PRs and ensuring code quality are standardized and rigorously reviewed on a platform level.

Partner with other members of the internal team, IT, data science, and analytics engineering teams to support end-to-end solution delivery, including building and promoting data pipelines, provisioning the related Databricks DAB infrastructure for data ingestion & transformation, and supporting DevOps pipelines (GitHub Actions) and processes.

Provide support and guidance across the SDLC, including CI/CD, release management, and environment promotion.

Provide ad hoc support for application deployment for Databricks Apps space and Azure App Services. Kubernetes application experience preferred.

Innovation & Enablement (10%)

Help evaluate emerging Azure and Databricks services and AI/ML capabilities to drive innovation and continuous improvement.

Contribute to data engineering platform governance, including data ethics, compliance, and security best practices.

Mentor technical teams and promote architectural standards and reusable patterns.

Utilization of artificial intelligence tools and resources (e.g. generative AI)

What you bring:

6+ years of experience in a combination of Data Engineering, DevOps, and App Development roles.
Experience configuring and maintaining Databricks Workspaces, ACLs, and other Workspace artifacts
Experience working with Databricks (Lakehouse architecture, Spark, Unity Catalog, Delta tables).
Experience with DevOps, and SDLC practices in a regulated enterprise environment.
Experience creating GitHub Actions Workflows for code promotion across environments.
Experience creating data ingestion and data transformation pipelines in Pyspark, preferably within Databricks.
You must be legally authorized to work in the U.S. without requiring immigration sponsorship now or in the future. This includes holders of H-1B, H4, L-1, L-2, TN, OPT, CPT, and other nonimmigrant visas.

2 Jobs for Delta lake in the United States

Senior Data Engineer - AI & Analytics Platform - Hybrid

VP, Product Engineer, Alternatives