Job Description

Designing, developing, and maintaining data processing pipelines using  Apache Spark
• Hands-on with very strong experience at
o Apache Spark: Spark’s core architecture, including RDDs, DataFrames, and Datasets. Experience in building, tuning, and deploying Spark applications for both batch and stream processing using Structured Streaming
• Implement data ingestion, transformation, and storage solutions that support real-time and batch processing
• Strong programming skills in Java, Scala, R and Python
• Experience in data processing services with cloud platforms like (AWS, Azure, Google Cloud)
• Fair understanding of machine learning frameworks and libraries
• Good experience with real-time data analytics, transformation, and storage
 
Nice to have:
• Good experience with data integration and messaging systems like Apache Kafka, RabbitMQ
• Fair knowledge of Docker containerization data processing applications
• Message brokers like RabbitMQ / ActiveMQ, and their integration with data processing pipelines