Big Data Engineer at Rakuten (Tokyo, Japan)

• 
  
• 
  

  Description

  
  

  As a Data Engineer, you will be designing and developing large scale data processing solutions for Rakuten Group companies. Be a member of a team that develops and implements data pipelines and algorithms that extract, classify, merge, and deliver new insights and business value out of heterogeneous structured and unstructured data sets. Designing and developing code, scripts and data pipelines that leverage structured and unstructured data integrated from multiple sources. Looking at setting platform components like Presto, Nifi, Hbase, Kafka, Spark clusters, etc.
  You will have a chance to learn and work with multiple technologies in the Big Data space and petabytes of data.

  
  
• 
  
• 
  

  Job Qualifications:

  
  
  • 
    
  • Experience programming in Java, Python, SQL or C/C++
  • 
    
  • Experience with Unix
  • 
    
  • Experience with SQL, NoSQL, relational database design and methods for efficiently retrieving data
  • 
    
  • Strong experience in working with and managing large distributed computing cluster (combining various technologies & frameworks like Hadoop, Nifi, streamsets, spark, kafka etc.).
  • 
    
  • Strong analytical skills and creative problem solver
  • 
    
  • Professional or academic background that includes computer science, mathematics, statistics etc.
  • 
    
  • Excellent verbal and written communications skills
  • 
    
  • Strong team player capable of working in a demanding start-up like environment building the next generation platform.
  • 
    
  
  

  Preferred Knowledge, Skills and Abilities:

  
  
  • 
    
  • Firm understanding of when to use interfaces vs abstract classes, subclassing, designing classes for re-use, static string constants rather than in-line constants, use of auto-closing resources of finally blocks
  • 
    
  • Familiarity with JUnit, TestNG, Mockito, JMockit, etc.
  • 
    
  • Fluid understanding of aggregates, group by’s, inner joins, outer joins, etc.
  • 
    
  • Ability to create a java project, maven project or similar with multiple packages an referenced dependencies, ability to load a project into an IDE without submitting IDE config files to source control
  • 
    
  • Basic knowledge on using fit, adding and committing, and branching
  • 
    
  • Unix permissions (chmod, chgrp, sudio, bitmask) basic unix commands (ls, top, find, xargs, grep, wc) unix processes (kill, ps).
  • 
    
  • Ability to edit config files, enable syntax highlighting, display or hide line numbers, use vim keyboard combinations like navigating to the end of a line, copy/paste using vim keyboard combinations
  • 
    
  • Ability to edit config files, enable syntax highlighting, display or hide line numbers, use emacs keyboard combinations like navigating to the end of a line, creating macros using emacs keyboard combinations
  • 
    
  
  .
• 
  

by via developer jobs - Stack Overflow