Data Engineer - Chennai / Bengaluru

On-siteMidData Engineerposted 2mo ago

✦Role summaryAI-generated

This role focuses on building scalable data pipelines and orchestration systems using Airflow, PySpark, and cloud platforms like Snowflake, with a strong emphasis on custom data connectors and CI/CD automation. The candidate will collaborate with cross-functional teams to design robust solutions for structured and unstructured data workflows while ensuring reliability and performance.

Skills required

Airflow Spark Hive trino Snowflake SQL Python Elt pipelines Data connectors Data cicd Data ingestion Data pipelines ELT CI/CD Data engineering Data governance

About this role

Must Needed Skills: Apache Airflow, Python, PySpark ,Hive(Trino), SQL and Snowflake

Develop and maintain data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python, PySpark ,Hive(Trino) and Snowflake to ensure the efficient and reliable delivery of data.
Design and implement custom connectors to facilitate the ingestion of diverse data sources into our platform, including structured and unstructured data from various document formats .
Collaborate closely with cross-functional teams to gather requirements, understand data needs, and translate them into technical solutions.
Design and implement data CI/CD pipelines to enable automated and efficient data integration, transformation, and deployment processes.
Monitor and troubleshoot data pipelines, proactively identifying and resolving issues related to data ingestion, transformation, and loading.
Conduct data validation and testing to ensure the accuracy, consistency, and compliance of data.
Stay up-to-date with emerging technologies and best practices in data engineering.
Document data workflows, processes, and technical specifications to facilitate knowledge sharing and ensure data governance.