QG
Quest Global
Data Engineer

Data Engineering Specialist

On-siteSeniorData Engineerposted 2w ago
✦Role summaryAI-generated

The role focuses on designing and maintaining scalable ETL pipelines using Spark, NiFi, and Airflow on the Cloudera/Hadoop ecosystem. Collaboration with architects and stakeholders ensures reliable data ingestion, transformation, and automation across batch and workflow processes.

Skills required

About this role

Job Requirements
  • · Design and develop robust data ingestion and ETL pipelines for batch and workflow-based processing requirements.
  • · Build, optimize, and maintain data processing solutions using Apache Spark, PySpark, and Apache NiFi.
  • · Develop, schedule, monitor, and manage workflows in Apache Airflow to ensure reliable orchestration.
  • · Work on the Cloudera Data Engineering platform and the broader Hadoop ecosystem, including HDFS and Hive.
  • · Write efficient Hive SQL and Python code for data transformation, validation, and operational automation.
  • · Support, troubleshoot, and optimize production data pipelines for performance, reliability, and scalability.
  • · Collaborate with data architects, analysts, platform teams, and business stakeholders to understand requirements and deliver data solutions.
  • · Follow engineering best practices for code quality, version control, documentation, deployment, and monitoring.


Work Experience

· 8+ years of relevant experience in data engineering, big data platforms, and production pipeline delivery.

· Strong hands-on experience in designing data pipelines using NiFi, Spark/PySpark, Hive SQL, and Python.

· Good understanding of Hadoop concepts, HDFS storage patterns, data partitioning, and distributed processing.

· Experience managing Airflow DAGs, dependencies, retries, monitoring, and failure handling.

· Ability to identify pipeline bottlenecks and perform performance tuning across ingestion, transformation, and orchestration layers.

Strong problem-solving mindset with the ability to support critical production workloads.



✕ position closed

This role is no longer accepting applications. It’s kept here for reference — check out the similar open roles below.

Similar open roles

PT
NEW

(SG) Data Engineer

Poet Technologies·Singapore, Singapore, Singapore
On-siteMidData Engineer
yesterday
D
NEW

Staff Data Engineer - Onsite

Danaher·San Jose, California, United States of America; Indianapolis, Indiana, United States of America
On-siteStaffData Engineer
$150k – $200k USD
2d ago
JC
NEW

Software Engineer III - Data Engineering

JPMorgan Chase·Plano, TX, United States
On-siteMidData Engineer
2d ago
JC
NEW

Software Engineer II - Data Engineering

JPMorgan Chase·LONDON, United Kingdom; BOURNEMOUTH, DORSET, United Kingdom
On-siteMidData Engineer
2d ago
JC
NEW

Software Engineer II - Data Engineering

JPMorgan Chase·BOURNEMOUTH, DORSET, United Kingdom
On-siteMidData Engineer
2d ago
JC
NEW

Software Engineer III - Data Engineering

JPMorgan Chase·BOURNEMOUTH, DORSET, United Kingdom
On-siteSeniorData Engineer
2d ago