Senior Data Engineer (m/w/d)

You will be responsible for ensuring high-quality access to data sources, with a strong focus on data quality, standardization, qualification, and governance to enable effective use by Data Analysts and Data Scientists.

You will contribute to the definition and implementation of data policies and to the structuring of the data lifecycle, ensuring compliance with regulatory requirements in collaboration with key data governance stakeholders.

You will oversee data management and processing systems, including Big Data and IoT platforms. You will ensure the integration of structured and unstructured data from multiple sources and maintain high data quality within the Data Lake through rigorous testing, validation, and deduplication processes.

Data Qualification & Data Management:

• Capture structured and unstructured data generated by internal applications and external sources.
• Integrate and consolidate data from multiple systems and environments.
• Structure data through semantic modeling and standardization practices.
• Map available data assets and maintain data consistency.
• Clean and enrich datasets, including duplicate elimination and quality remediation.
• Validate data integrity and business consistency.
• Create and maintain data repositories where required.

Datastream Team Scope:

Support data management activities across Downstream and Upstream domains, with a particular focus on data used for forecasting purposes and business-critical analytics.

Technical Skills & Expertise:

Data Engineering & Development

• Proficiency in Python and .NET/C# for script development and ETL implementations.
• Strong knowledge of software development best practices and version control using Git.
• Expertise in modern data acquisition, preparation, integration, and transformation techniques.

Databases & Data Warehousing

• Strong understanding of relational databases and SQL.
• Ability to design efficient data models and optimize query performance.
• Experience working with enterprise data warehouses and large-scale data platforms.

Big Data & Cloud Technologies

• Experience with Databricks, including ETL development, Spark cluster management, and SQL optimization.
• Experience with PySpark and Apache Spark ecosystems.
• Knowledge of AWS cloud services and data platform solutions.
• Experience building and orchestrating ETL workflows using Airflow and DAGs.

Qualifications

• Master's Degree in Computer Science, Data Engineering, Data Science, Information Systems, Applied Mathematics, or a related field.
• 8-10 years of experience
• Strong knowledge of Data Governance, Data Quality, Data Warehousing, and Data Lake architectures.
• Experience working in Agile environments and large-scale enterprise ecosystems.
• Excellent communication, stakeholder management, and problem-solving skills.

Additional Information

All our positions are open to both women and men and are, of course, open to people with disabilities.