Job Description
As our Senior Data Engineer, you will optimize our data integration at scale. Our Data team is responsible for designing pipelines and warehouses to model data from multiple sources that will allow us to derive business insight. The team uses Azure and other open-source technologies, such as Azure Data Factory and PySpark.
Responsibilities:
- As a Senior Data Engineer, you will be responsible for the roll-out and continuous optimization of the respective data management platform
- Build data pipelines and Python-based ETL tools for acquiring, processing, and delivering data
- Develop data models and schemas in our data warehouse that enable performance analysis and intuitive analysis
- Handle the challenges that come with managing terabytes of data
- Collaborate with business leaders and analysts to define key metrics and build reports to monitor and understand company performance
- Develop the server applications and APIs that are used by our Data Team
- Mentor other developers on their development work and peer review their activities as per department/enterprise guidelines
- Working as technical lead of various projects and working with PM, analysts, and architects.
Basic Qualifications:
- Bachelor’s degree or higher in Computer Science, Computer Engineering, Information Technology, or related field – Master’s degree is preferred
- Fluent in several programming languages such as Python, R, or Scala
- 7+ years of work experience in building ETL pipelines in production data processing and analysis
- Prior experience in managing technical associates
- Expert in designing SQL tables, choosing indexes, tuning queries, and optimizations across different functional environments
- 7+ years of hands-on experience in writing complex SQL queries and using a BI tool
- Experience with Data lakes and designing and maintaining data solutions using Spark and Azure serverless services such as ADF
- Experience with data ingestion APIs, data sharing technologies, and warehouse infrastructure and development
- Good verbal and written communication skills, having leadership qualities, and following best practices in project documentation.