The Applecart Graph Team is at the epicenter of Applecart’s technology-- everything we do at Applecart is contingent upon the veracity of our core product, the Social Graph. To date the Social Graph has mapped out over 4.5 billion relationships across the US. The Graph Team works dailly to build and incorporate new social connections into the Social Graph.
As a Data Engineer on our Graph Team, you will be responsible for sourcing, enhancing and integrating data sources to our Social Graph while providing optimizations to the graph for creation of powerful machine learning applications.
Responsibilities:
- Collaboratively architect, build, launch and maintain new Social Graph components that enhance profiles, increase coverage and edge accuracy.
- Create, maintain, and scale data pipelines for and between data ingesters, the Social Graph, machine learning predictors, client deliverables, and data warehousing.
- Interact cross-functionally with a wide variety of people and teams. Work closely with client services leads and data scientists to identify opportunities and assess improvements to Applecart products and deliverables.
- Integrate systems for monitoring of streaming and batch data processing (e.g. DataDog, Nagios). Track data quality and consistency.
- Evangelize solid coding practices (e.g. unit & integration testing, code reviews, continuous deployment, automated linting, staging environments), and mentor junior engineers.
- Contribute to the architectural designs and decision making around data stores, schemas, data security and cloud storage.
- Rapidly prototype proof-of-concept data pipelines for ROI determination, then replace them with modular productionized versions.
- Keep abreast of industry trends, best practices, and emerging methodologies.
- Support quality assurance as a part of the engineering process and collaboration with product managers such as producing sampled outputs, supporting KPIs, outlining PR limitations and future improvements.
Qualifications:
- BS or MS degree in Computer Science, Math, Statistics or other technical field.
- 3-5+ years of applied software engineering experience (especially startups, big data, Python).
- 1+ years as a team lead or managerial role.
- Python Expertise: classes & inheritance, generators, decorators, docstrings, pylint, pytest, etc.. Numpy/pandas experience preferred.
- SQL/Hive Expertise: where clauses, joins, group bys, windowing functions, exploding.
- Spark Expertise: SparkSQL, pyspark, Caching, Checkpointing, Dataframes, RDDs.
- Expertise in building, monitoring and maintaining reliable ETL pipelines.
- Ability to write well-abstracted, extensible, object-oriented code components.
- Enjoy working in a fast-paced, highly collaborative and ambitious startup work environment.
- Basic understanding of probability & statistics; experience evaluating data quality at scale.
- Experience with Amazon Web Services (RDS, S3, EC2, EMR, Data Pipeline), PyCharm, Github, JIRA (or equivalents).
Prefered Qualifications:
- Experience with open source search platforms such as Solr, ElasticSearch or the like.
- Experience with Unix/OSX CLI (bash)
- Background in data wrangling various structured & unstructured data sets, consuming APIs (e.g. rate limiting and exponential back-offs) and the like.
- Knowledge of graph storage and computation frameworks (e.g. GraphX, TitanDB, Neo4J).
- Familiarity with Scala and/or Java, Apache Spark internals and job optimization.
- Significant interest or background in big data, politics, advertising or finance technology
- Experience working directly with data scientists; basic knowledge of common machine-learning techniques
- Experience with agile development or similar methodologies for continuous development of product and technology.
by via developer jobs - Stack Overflow
No comments:
Post a Comment