SS
Sigma Software
Data Engineer
Senior Data Engineer (Databricks Migration)
On-siteSeniorData Engineerposted 5d ago
Role summaryAI-generated
Lead the migration of a large-scale analytical platform from BigQuery to Databricks, designing scalable Lakehouse architectures and optimizing Spark-based data pipelines. Collaborate with cross-functional teams to implement data governance with Unity Catalog and develop reusable platform components.
Skills required
About this role
- Participate in the migration of a large-scale analytical platform from BigQuery to Databricks
- Design and implement scalable Lakehouse architectures using Databricks and Delta Lake
- Analyze existing ETL / ELT workloads and define migration approaches
- Develop and optimize data pipelines processing large volumes of retail and analytical data
- Implement incremental processing strategies and scalable transformation frameworks
- Build and maintain Spark-based data processing solutions using PySpark
- Design and maintain medallion architecture layers including Bronze, Silver, and Gold
- Implement data governance and security best practices using Unity Catalog
- Collaborate with Data Science, Analytics, Product, and Customer Engineering teams
- Participate in architecture discussions and technical solution design
- Develop reusable data platform components and engineering standards
- Conduct code reviews and contribute to platform reliability and maintainability
- Troubleshoot and optimize complex SQL and Spark workloads
- Support production deployments and platform modernization activities
Qualifications
- 5+ years of professional experience as a Data Engineer
- Strong programming skills in Python and advanced SQL
- Hands-on commercial experience with Databricks
- Strong knowledge of Apache Spark, primarily PySpark
- Experience designing and building modern cloud-based data platforms
- Experience developing ETL / ELT pipelines and large-scale data processing solutions
- Hands-on experience with Delta Lake
- Experience with Spark Declarative Pipelines
- Experience with cluster monitoring, metrics analysis, and performance optimization
- Strong understanding of distributed data processing architectures
- Solid understanding of data warehousing concepts and dimensional modeling
- Experience with Airflow or similar orchestration tools
- Experience optimizing complex analytical SQL workloads
- Experience implementing CI / CD practices for data engineering platforms
- Strong troubleshooting and performance optimization skills
- Ability to work collaboratively in cross-functional international teams
- Upper-Intermediate or higher English level
WILL BE A PLUS
- Experience working with GCP cloud services
- Experience with AWS or Azure cloud platforms
- Experience in retail analytics or pricing optimization domains
- Experience supporting machine learning or AI-related data workloads
- Experience with platform modernization and cloud migration initiatives
Additional Information
PERSONAL PROFILE
- Strong analytical and problem-solving mindset
- Proactive and ownership-driven approach
- Ability to work independently and collaboratively
- Good communication and stakeholder collaboration skills
- Passion for scalable data engineering and modern data platforms
- Interest in continuous learning and technology innovation
Apply on Sigma Software →Opens in new tab
score your resume against this role