I
Inetum
Data Engineer
DATA ENGINEER
On-siteSeniorData Engineerposted 1w ago
Role summaryAI-generated
The candidate will lead the migration of the existing Hadoop environment to a cloud-native stack using Kubernetes Engine, Spark as a service, and Airflow as a service. They will also design, automate, and test robust data pipelines while maintaining documentation and ensuring data quality.
Skills required
About this role
Hybrid work mode in Madrid. This service could imply a willingness to travel occasionally within Italy.
The main responsibilities are:
- Migrate the existing Hadoop infrastructure to cloud infrastructure on Kubernetes Engine, COS, Spark as a service, and Airflow as a service.
- Implement data transformation and quality to ensure data consistency and accuracy. Utilize programming languages such as Scala and SQL and tools like Spark for data transformation and enrichment operations.
- Set up CI/CD pipelines to automate deployments, unit testing and development management.
- Write and conduct unit and validation tests to ensure accuracy and integrity of code developed.
- Automate data pipelines and streamline data ingestion through the implementation of different orchestrators and scheduling processes (Airflow as a Service mainly).
- Writing technical documentation (specifications, operational documents) to ensure knowledge capitalization.
- Collaborate with cross-functional teams to understand data requirements and deliver solutions.
- Foster a culture of continuous learning and improvement within the team.
Qualifications
Good knowledge of:
- Spark on Scala
- CI/CD tools (Gitlab, Jenkins…)
- HDFS and structured databases (SQL)
- Apache Airflow
- S3 storage/COS and parquet (and ORC) format
- Strong knowledge of SQL and NoSQL databases
- Hadoop
Full understanding of:
- Oozie
- Shell script
- Software Development Lifecycle (SDLC) awareness
- Agile principles and ceremonies
- Design effective prompts to leverage Gen AI tools across IT domains (e.g., development, testing, data generation, documentation) during the development stage.
Some knowledge of:
- Kubernetes containerization
- Dremio as tool to virtualize data
Optionally/ as a plus:
- Elasticsearch and Kibana
- Streaming process (Kafka, event steam…)
- HVault
Additional Information
BUSINESS AND TRANSVERSAL SKILLS
- Knowledge of banking industry and processes
- Problem-Solving and Decision-Making person
- Business / IT relationship (including IT OPS)
- Ability to understand, explain and support change
- Ability to Deliver / Results driven
- Ability to collaborate / Teamwork with data squads and business teams
LANGUAGES
- English level C1
- Optionally Fench
Apply on Inetum →Opens in new tab
score your resume against this role