Senior Data Engineer - Spark expertise - NYC or Remote North America at SecurityScorecard (New York, NY) (allows remote)

Position Summary

The Senior Data Analytics Engineer will build meaningful analytics that inform companies of security risk.  You will be working closely with our Data Science team, implementing algorithms and managing the analytic pipeline. We have over 1 PB of data, so the ideal candidate will have experience processing and querying large amounts of data.  

This role requires senior level experience in Spark, SQL and Scala. Our interview process will include live coding using these technologies!

Responsibilities

• Manage the analytic pipeline using Spark, Hadoop, etc 


• Leverage cutting-edge technologies to support new and existing and services and processes.


• Quickly and efficiently design and implement in an agile environment


• Work with other team members to implement consistent architecture


• Drive projects through all stages of development


• Actively share knowledge and responsibility with other team members and teams


• Improve the effective output of the engineering team by managing quality, and identifying inconsistencies.  

Skills and Experience:

• Bachelor's degree (CS, EE or Math preferred) or equivalent work experience as well as interest in a fast paced, complex environment.


• 5+ years of experience Scala preferred in a commercial environment 


• Expert in Spark, experience with the Hadoop ecosystem and similar frameworks


• Expert in SQL


• Familiarity with various tools such as AWS and Docker and an instinct for automation


• Strong understanding of Software Architecture principles and patterns.


• Experience working with 3rd party software and libraries, including open source


• Experience with Postgres

Traits:

• Quick-thinker who takes ownership and pride in their work


• A commitment and drive for excellence and continual improvement 


• A strong sense of adventure, excitement and enthusiasm.


• Excellent systems analytical, problem solving and interpersonal skills

Interview Process:

• Initial Conversation with a SecurityScorecard Talent team to learn more about your experience and career objectives


• Technical Interview with 1- 2 data engineers. This will include live coding in SQL, Spark, Scala.


• Coding Exercise - take home exercise


• Final Interview: Meet 1-2 engineering leaders

by via developer jobs - Stack Overflow