Python Developer (Data Engineer)
This role focuses on improving legacy data engineering systems in Lisbon by refining Python-based pipelines, enforcing clean code standards, and ensuring compatibility with modern Python 3 frameworks. The candidate will collaborate closely with a small team to optimize data workflows while maintaining high-quality testing and modular architecture.
Skills required
About this role
Ability to work independently and take initiative to solve technical problems.
Excellent communication skills to effectively collaborate with technical and non-technical teams.
You will be part of a team of 5 developers and will report to the Data domain manager.
- Analyze and refactor existing code primarily developed by data engineers.
- Establish and maintain modular and clean development standards.
- Experience with code reviews and provide feedback to improve code quality and development practices.
- Write and execute unit tests, integration tests, performance tests, and regression tests to ensure code robustness and quality.
- Ensure migration and compatibility with recent versions of Python 3 while managing the existing code.
- Apply the SOLID principles of object-oriented design to ensure code quality and maintainability.
- Work with tools and environments such as Cloudera to manage and optimize data.
- Understand and apply distributed computing principles to optimize data processing and computation.
DevOps responsibilities:
- Experience with GitLab CI/CD for automation of build and deployment processes.
- Collaborate with data engineering teams to understand and improve existing data pipelines.
- Implement Continuous Integration (CI) using Jenkins and Jenkins files/groovy, SonarQube
- Familiarity with SonarQube for code quality analysis.
- Develop, deploy, and manage containerized applications using Docker.
- Understanding of deployment strategies using Virtuozzo Containers and Docker.
- Knowledge of Cloudera platform and its integration with Python applications.
- Experience with Jira API for task and project management.
- Familiarity with Nexus Repository for artifact management.
- Strong troubleshooting skills and ability to optimize performance.
- Knowledge of package management with PyPI and pip.
- Auto feed DEV, SIM, UAT envs
Methodology:
Work in an Agile methodology, participating in daily stand-ups, sprint planning, and retrospectives.
Qualifications
Proven experience (3+ years) in Python development, with a strong understanding of object-oriented and modular programming principles.
- Good knowledge of Python 3 and the challenges associated with migrating from older versions.
- Deep understanding of the SOLID principles of object-oriented design.
- Expertise in code refactoring and improving code quality.
- Proficiency in unit testing, integration testing, performance testing, and regression testing.
- Experience with distributed computing principles and optimizing data processing (for example: Spark).
- Understanding of data modelling and creating data structures on Hadoop.
- Experience with big data environments, particularly Cloudera, is a significant plus.
DevOps skills:
- CI/CD Pipeline Automation
- Version Control & Branching
- Deployment Automation
- Artifact Management
- Containerization & Orchestration
- Automation & Monitoring
Similar open roles
