Senior Data Engineer

Data Pipeline Development

• Design, build, and maintain ETL/ELT pipelines in Databricks to ingest, clean, and transform data from diverse product sources.
• Construct gold layer tables in the Lakehouse architecture that serve both machine learning model training and real-time APIs.
• Monitor data quality, lineage, and reliability using Databricks best practices.

AI-Driven Data Access Enablement

• Collaborate with AI/ML teams to ensure data is modeled and structured to support natural language prompts and semantic retrieval using 1st and 3rd party data sources, vector search and Unity Catalog metadata.
• Help build data interfaces and agent tools to interact with structured data and AI agents to retrieve and analyze customer data with role-based permissions.

API & Serverless Backend Integration

• Work with backend engineers to design and implement serverless APIs (e.g., via AWS Lambda with TypeScript) that expose gold tables to frontend applications.
• Ensure APIs are performant, scalable, and designed with data security and compliance in mind.
• Utilize Databricks and other APIs to implement provisioning, deployment, security and monitoring frameworks for scaling up data pipelines, AI endpoints, and security models for multi-tenancy.

Qualifications

• 3+ years of experience as a Data Engineer or related role in an agile, distributed team environment with a quantifiable impact on business or technology outcomes.
• Proven expertise with Databricks, including job and workflow orchestration, change data capture and medallion architecture.
• Proficiency in Spark or Scala for data wrangling and transformation on a wide variety of data sources and structures.
• Practitioner of CI/CD best practices, test-driven development and familiarity with the MLOps / AIOps lifecycles.

• Proven ability to work in an agile environment with product managers, front-end engineers, and data scientists.

Additional Information

Preferred Skills

• Familiarity with AWS Lambda (Node.js/TypeScript preferred) and API Gateway or equivalent serverless platforms, knowledge of API design principles and working with RESTful or GraphQL endpoints.
• Exposure to React-based frontend architecture and the implications of backend data delivery on UI/UX performance – including end-to-end telemetry to measure performance and accuracy for the end-user experience.
• Experience with A/B testing, experiment and inference logging and analytics.