Development of a big data platform in Hadoop using Spark, Oozie, and related;
Implement and support big data tools, frameworks such as HDFS, HIVE and streaming technologies such as Kafka and Spark;
Set standards for warehouse and schema design; Write HIVE Queries to process and manage data in external tables;
Automating task of loading the data into HDFS by developing workflow jobs in Oozie; Develop/build Maven scripts to integrate them with Jenkins and automate the compilation, deployment and testing of web applications through XL Release; feed machine learning models in big data environment;
Developing, optimising hadoop/Spark jobs and automation of hadoop/spark jobs using shell scrip, event engine tool;
Developing scala, python and java based rest API for batch processing, Real time processing of streaming requests through API in big data platform;
Responsible for unit testing, debugging, verifying splunk logs and deploying the production ready code and various micro services for US and International market projects.
Migrating big-data projects to new architectural platform and configuration driven applications from old platform using technologies – Java, Hadoop Distribution, Map-Reduce, HIVE, Shell Scripting, SQL, REST Services;
Developing and maintaining the big-data pipelines (batch and real-time) from end-to-end flow.
Education Required: Masters in Info.Sys/CS/Inf.Tech/related plus 6 months experience as Programmer/System Analyst/Soft.Dev/Soft.Engg/related.
