Data & Back-End Engineer
- Experience working in agile software development teams: 5+ years;
- Experience with SQL language: 3+ years;
- Python experience: 3+ years;
- Experience working with cloud platforms like Amazon, Azure or Google Cloud Platform: 2+ years;
- Experience working with Docker: 2+ years;
- Experience in software release management: Git, CI/CD;
- Implement and maintain different data ingestion/processing pipelines for existing and new projects, extending and improving our existing data pipeline stack built using Airflow, Cloud Dataflow, and Bigquery.
- Handle architectural and design considerations such as performance, scalability, reusability, and flexibility constraints.
- Evaluate the data integrity from each source, finding the way to ingest, curate, and normalize it.
- Document and communicate changes inline code, PR, tickets and release notes with the engineering, product and data team.
- Work jointly with the research team to make prototypes out of proof-of-concepts and operationalize prototype pipelines.
- Work jointly with the product team to understand the needs of the output datasets and find interconnections between projects.
- Work jointly with the front-end and data to make sure the data is being used correctly.
- Implement uptime monitoring for the multiple ingestion pipelines and set up continuous QA metrics on processing pipelines.
- Debug data issues working closely with team members from other areas of the organization (subject matter experts or ML scientists).
- Modify/Fix/Implement new API endpoints owning the full development cycle. Including supporting QA for test automation and product for feature release and documentation.