Here's an interesting job that we think might be relevant for you -
Job Title: Cashfree Payments - Senior Data Engineer - Spark/Apache Flink (5-8 yrs)
Click here to Apply
Job Description :
Responsibilities :
- Design and Develop : Architect and implement scalable data pipelines using Spark, Flink, Airflow, and Apache Hudi to process large-scale data efficiently.
- Database and Storage Optimization : Work extensively with Redshift, ClickHouse, and other databases, optimizing query performance, storage, and cost efficiency.
- Data Lake and ETL Development : Build and manage data lakes and ingestion frameworks using AWS services such as S3 Glue, and EMR.
- Data Modeling and System Design : Design and implement robust data models for structured and semi-structured datasets, ensuring performance, scalability, and maintainability.
- Real-time and Batch Processing : Develop and maintain both real-time and batch data pipelines using technologies like Flink, Spark Streaming, and Kafka.
- AWS and Cloud Infrastructure : Leverage AWS services (S3 Redshift, Lambda, ECS, etc. ) to build a scalable and cost-efficient data ecosystem.
- Automation and Orchestration : Manage workflow automation and orchestration using Apache Airflow.
- Mentorship : Guide and mentor junior engineers, conduct code reviews, and contribute to best practices for data engineering.
- Monitoring and Optimization : Implement observability (monitoring, alerting, logging) for data pipelines to ensure reliability and performance.
Requirements :
- 6+ years of experience in data engineering, big data processing, and cloud-based data architectures.
- Strong expertise in Redshift, ClickHouse, and other distributed databases.
- Experience with Apache Spark, Flink, Apache Hudi, and Airflow.
- Deep knowledge of data modeling, schema design, and database optimization.
- Hands-on experience in building data pipelines on AWS (S3 Glue, Redshift, EMR, Lambda, etc. ).
- Proficiency in Python, SQL, and Scala for data engineering tasks.
- Experience in mentoring and leading junior engineers in best practices.
- Strong problem-solving skills and ability to work in fast-paced environments.
Preferred Qualifications :
- Experience with CI/CD pipelines for data engineering workflows.
- Familiarity with streaming data technologies (Kafka, Kinesis, Pulsar, etc. ).
- Prior experience with Terraform, Kubernetes, or serverless architectures.
- Exposure to data governance, security, and compliance best practices.
To view the job description, Please refer to the below link:
https://www.hirist.com/j/cashfree-payments-senior-data-engineer-sparkapache-flink-5-8-yrs-1460326.html?ref=tm
PS: Please ignore this email if you have already applied or not interested in this job.
Best regards,
Team hirist.com
info@hirist.com
