Lead Data Engineer (AWS)

The purpose of this role is to provide technical guidance and suggest improvements in development processes. Develop required software features, achieving timely delivery in compliance with the performance and quality standards of the company.

Job Description:

Business Title

Technical Lead

Years of Experience

Min 6 and max upto 10.

Job Description

Looking for a hands‑on AWS Technical Lead – Data Engineering with 6 to 10 years of total experience to design and deliver scalable, secure, and high‑performance data platforms on AWS.
This role focuses on strong individual contribution with technical ownership, working closely with global teams, architects, and clients to deliver enterprise‑grade data engineering solutions. The position requires deep expertise in AWS data services, SQL, and Python, and the ability to build and optimize reliable data pipelines for analytics and business use cases.

Key responsibiltes

Data Platform Design & Development
Design and implement AWS‑based data engineering solutions aligned to enterprise standards
Build and optimize batch and streaming data pipelines using AWS native and open‑source tools
Develop SQL‑driven transformations and Python‑based data pipelines for analytics use cases
Design efficient data models for performance, scalability, and cost effectiveness

Delivery & Quality Ownership
Own data engineering deliverables from development through production support
Perform performance tuning, cost optimization, and capacity planning
Troubleshoot complex data pipeline and production issues, including root‑cause analysis
Ensure solutions meet requirements for security, reliability, and scalability

Collaboration & Client Engagement
Work closely with architects, product owners, and client stakeholders
Translate business and analytics requirements into robust AWS data engineering solutions
Provide clear technical inputs, estimates, and implementation trade‑offs
Contribute to solution discussions and technical design reviews

Engineering Best Practices
Follow and contribute to coding standards, documentation, and data engineering best practices
Participate in code reviews and continuous improvement initiatives
Ensure adherence to AWS, security, and compliance guidelines

Must have skills

Cloud & Data Engineering (AWS)
Strong hands‑on experience with AWS data services, including:
Amazon S3, AWS Glue, Athena, Redshift
Experience designing cloud‑native data lakes and data warehouse architectures on AWS
Deep understanding of batch and streaming data pipelines
Experience building scalable, fault‑tolerant data ingestion and transformation workflows

SQL & Python (Mandatory)
Strong SQL expertise
Writing complex SQL for transformations, aggregations, performance tuning, and analytics
Hands‑on experience handling large‑scale datasets in Redshift / Athena

Strong Python programming skills (mandatory) for data engineering use cases

PySpark / Spark‑based processing
Building reusable ETL components, utilities, and data pipelines
Strong understanding of data modeling, transformations, and performance optimization

Data Processing & Engineering
Proven hands‑on experience with distributed processing frameworks such as Spark / PySpark
Experience working with structured, semi‑structured, and unstructured data
Solid understanding of schema design, partitioning, and query optimization

DevOps & Platform Engineering
Experience with Infrastructure as Code using Terraform and/or CloudFormation
Hands‑on experience building and maintaining CI/CD pipelines for data platforms
Exposure to containerized workloads (Docker, ECS/EKS where applicable to data workloads)

Collaboration & Ownership
Strong ownership mindset for solution quality, performance, and production stability
Excellent communication skills to collaborate with architects, DevOps, QA, and business stakeholders

Good to have skills

Experience with real‑time/streaming technologies (Kinesis, Kafka, MSK)
Exposure to Lakehouse architectures and modern data platform patterns
Experience integrating AWS data platforms with BI and analytics tools
Knowledge of data governance, data quality, and metadata management
Familiarity with FinOps practices for optimizing AWS data platform costs
Exposure to marketing, customer, or analytics data domains (CDP / MarTech)
Experience working in Agile delivery models with global delivery exposure

Education Qulification

1. Bachelor’s or Master’s degree in Computer Science, Information Systems, Data Engineering, or a related field.

Certification If Any

AWS Data Analytics / Solutions Architect
Any two of the above
Databricks, Snowflake, or other cloud data platform certifications are a plus.

Shift timing

12 PM to 9 PM and / or 2 PM to 11 PM - IST time zone

Location:

DGS India - Pune - Indiqube Orchid

Brand:

Merkle

Time Type:

Full time

Contract Type:

Permanent