Lead Infrastructure Engineer - Infrastructure Team

About source{d}

source{d} builds the open-source components that enable large-scale code analysis and machine learning on source code.

Our powerful tools can ingest all of the world’s public git repositories turning code into ASTs ready for machine learning and other analyses, all exposed through a flexible and friendly API.

We are paving the way for the future of the software development life cycle, where code becomes analyzable data powering the next generation of developer tools.

Position

Engineering consists of five different teams that represent the architecture of our product:

• Applications (Scala, Go, Python, and Frontend tools): Builds CLI/Web applications combining ML research with our stack
• Machine Learning (C++ and Python): Performs R&D for Machine Learning on Source Code
• Data Retrieval (Scala and Go): Builds the technology that finds, fetches, stores and analyzes over +60M Git repositories
• Language Analysis (Go and another +15): Focused on Babelfish, the universal code parsing server
• Infrastructure (Go and Python): Manages a cluster of on-prem bare metal servers with Kubernetes and CoreOS, and GCP for user facing applications

Your qualification

This position is for a Lead Engineer in the Infrastructure team. The team has currently 3 members.

The Infrastructure team manages multiple clusters:

• Pipeline clusters, built on bare metal servers at a hosting provider. It has more than 1000 threads, more than 6TB of RAM and 500TB of storage, and it is backed by CoreOS and Kubernetes. It has two main goals: on the one hand, it stores all the available public code in a distributed filesystem and on the other hand, it runs intensive computation jobs over the stored data on top of Apache Spark
• Machine Learning research cluster, built on bare metal servers at our Madrid office. It is backed by CoreOS and Kubernetes too and it has GPUs available on every server to run deep learning algorithms
• Multiple Google Kubernetes Engine clusters for public-facing services

All clusters are managed with Terraform, Kubernetes and Helm.

The team also maintains several services such as databases, queues, continuous integration, monitoring, logging, etc.

We are looking for someone with background on Linux, networking and containers, passion for automation and experience working at scale. Finally, knowledge of at least one backend/scripting language who cares about best development practices.

What we offer

• We go to conferences and other developer events!
• Open Source Days, every second Monday, you are encouraged to work on any OSS project you choose
• Flexible hours, set your own schedule that fits you
• Free books. We will buy any books that help you learn & grow
• If you choose to work from one of our offices, you will enjoy a comfortable and spacious environment
• Annual summer and winter Christmas parties and a hackathon retreat are held in Madrid and all team members are flown over for it 
• We also have our own, Open Source craft beers
• We offer visa and relocation support for those wanting to work in the Madrid or San Francisco office