We are seeking talented Scala Spark Developer to create innovative solutions. Our partner's team manages large data stores of customer interactions data, providing a foundation for machine learning analytics and contextual, omni-channel interaction routing.
Main duties and responsibilities:
- You would be responsible for creating & maintaining large data sets and streaming solutions within their Hadoop environment, along with the read/write services that enable the collection and retrieval of the data. You will work in team to design and implement Hadoop based data extraction, transformation; processing for load or presentation into visualization and reporting tools. Your duty is to work with data from traditional databases & data warehouses, web services, MongoDb /Kafka sources process; transform and augment the data structuring it into views suitable for integration with tools (e.g. visualization and reporting).
- Proven expertise w/leveraging big data components to build large scale data processing systems. Proven experience in building in Data Driven applications using a combination of Java/Scala and the Spark framework. Sound knowledge about Hadoop platform and Data Streaming (e.g. Kafka). Good understanding of Apache Spark architecture and having hands-on experience. Experience building a distributed Apache Hadoop multi-node large cluster. Build data pipelines and ETL using heterogeneous sources to Hadoop using Kafka, Flume, Sqoop, Spark Streaming etc. Tuning performance optimization of Spark data pipelines. Experience in batch or real time data streaming
- Knowledge of design strategies for developing scalable, resilient, always-on data lake.
- Must be very comfortable with reading and writing Scala, Python or Java code. Have strong experience in designing and programming in Java/Scala in Unix/Linux environment.
- Develops programming and development standards and procedures as well as programming architectures for code reuse. Has in-depth knowledge of state-of-the art programming languages and object-oriented approach in designing, coding, testing and debugging programs.
- Ability to write scalable, reliable, high performance distributed application
- Experience in agile development methodology