Senior/Lead Data Engineer

About Hero.io

Hero.io is an AI-powered platform designed to make your crypto decisions smarter.

We prioritize truth, transparency, and the power of data. Hero.io provides you with hard facts and clear insights, enabling you to make informed decisions free from the common misinformation in the blockchain space.

Hero.io offers a comprehensive suite of tools to verify crypto projects and profiles, manage your assets securely, process payments effortlessly, and navigate the Web3 space with privacy and confidence.

Position

We are seeking a motivated and enthusiastic Senior/Lead Data Engineer to be an integral part of our data engineering team, responsible for supporting the design, development, and maintenance of our data infrastructure and pipelines.

Key Responsibilities:

• Develop and maintain web-scraping scripts that support data pipelines  (Python + SQL and related technologies)
• Understand Data Modeling, and how to implement transformations and cleansing techniques to validate data from multiple streams including manual data entry sources
• Monitor and troubleshoot API Endpoints to ensure data accuracy, completeness, and reliability
• Contribute to the optimization of data storage and retrieval processes
• Assist in the development and maintenance of data warehouse architecture
• Participate in code reviews, documentation, and knowledge sharing sessions

Your qualification

• 3+ years of experience with one or more programming languages, especially Python and SQL
• Knowledge of ETL processes, web-scraping, handling unstructured data
• Experience with scripting, scraping, and data wrangling on unstructured datasets, with proper error handling
• Ability to implement and test RESTful APIs (FastAPI, Flask)
• Understanding relational database design concepts and data modeling (SQL + PostgreSQL, TortoiseORM, SQLAlchemy)
• Experience with version control systems (Git)

Bonus Qualifications:

• Experience with writing and maintaining DAGs + workflows (Apache Airflow)
• Knowledge of Proxy server rotation and utilizing cloud platforms (AWS, Azure, or Google Cloud Platform)
• Familiarity with containerization (Docker + Docker Compose)
• Visualizing and Monitoring tasks within Workflows and profiling bottlenecks