Data Engineer (AWS+Pyspark)

On-siteSeniorData Engineerposted 4d ago

✦Role summaryAI-generated

This role focuses on building and optimizing PySpark applications on AWS services such as EMR, Glue, and Athena, handling large-scale data pipelines. Candidates must have extensive experience with Python, Spark DataFrames, and AWS storage and compute services, along with version control and columnar data formats.

Skills required

Python Spark Aws emr Aws glue Aws athena Aws lambda Aws s3 Git Parquet Snappy AWS Data warehousing Snowflake Redshift

About this role

Data Engineer (AWS + pySpark)

Having 6+ yrs years of relevant experience, which includes hands on experience in Big Data technologies.
Mandatory - Hands on experience in Python and PySpark.
Build pySpark applications using Spark Dataframes in Python.
Worked on optimizing spark jobs that processes huge volumes of data.
Hands on experience in version control tools like Git.
Worked on Amazon’s Analytics services like Amazon EMR, Amazon Athena, AWS Glue.
Worked on Amazon’s Compute services like Amazon Lambda, Amazon EC2 and Amazon’s Storage service like S3 and few other services like SNS.
Good to have knowledge of datawarehousing concepts – dimensions, facts, schemas- snowflake, star etc.
Have worked with columnar storage formats - Parquet etc. Well versed with compression techniques – Snappy, Gzip.
Good to have knowledge of AWS databases (atleast one) Aurora, RDS, Redshift, ElastiCache, DynamoDB.am