What makes you a great fit:

• Design, build, and maintain scalable data pipelines using Apache Spark and Apache Kafka.
• Develop and optimize Spark jobs for data processing, analysis, and transformation
• Implement real-time data streaming solutions using Kafka.
• Work closely with data scientists and analysts to understand data requirements and translate them into technical solutions.
• Collaborate with cross-functional teams to integrate data from various sources and ensure data quality and consistency.
• Optimize and tune Spark and Kafka configurations for performance and reliability.
• Monitor and troubleshoot data pipelines to ensure uninterrupted data flow.
• Design and implement data storage solutions, including data lakes and data warehouses.
• Implement security and access controls for data pipelines and storage solutions.
• Stay up-to-date with emerging technologies and industry trends in big data and distributed systems.