Job Title: Data Engineer
Location: Remote
Contract: 500 Per Day
Role Overview
Our client is seeking a Data Engineer with a solid foundation in both data engineering and business intelligence, capable of understanding and addressing business needs. This role involves managing and optimizing AWS-based data processes, orchestrating data flows, and supporting business intelligence capabilities. The ideal candidate will not only bring technical expertise but also a strong business acumen to facilitate effective collaboration with non-technical stakeholders.
Key Responsibilities
- Data Pipeline Management:
- Database Integration: Maintain MySQL databases on AWS (approx. 25TB) as primary data sources feeding the data pipeline.
- Real-Time Data Ingestion: Utilize Apache Kafka to manage data flows from MySQL into AWS S3 buckets, enabling real-time streaming.
- Data Processing: Implement ETL processes using AWS Glue and PySpark to build data lakes within S3 for further analysis.
- Business Intelligence (BI):
- Facilitate data availability for BI purposes, supporting daily updates and aiming to move from overnight batch jobs to near-real-time 15-minute refreshes.
- Collaborate on BI dashboard and reporting creation to provide actionable insights.
- Complex Data Reconciliation:
- Reconcile transaction records across multiple merchants, currencies, and network operators, addressing both straightforward and complex data relationships.
Technical Requirements
- Data Engineering Expertise:
- Proficiency with MySQL, Apache Kafka, AWS Glue, PySpark, and AWS S3 for managing data pipelines and ETL processes.
- Business Intelligence Skills:
- Experienced in designing and maintaining BI reports and dashboards with frequent data refresh capabilities.
- Data Reconciliation Experience:
- Skilled in reconciling transactional data across diverse systems, operators, and currencies to ensure data accuracy.
Business & Communication Skills
- Business Acumen:
- Ability to translate business requirements into data solutions and work collaboratively with non-technical stakeholders to develop reports and dashboards tailored to business needs.
- Communication:
- Strong communication skills to bridge technical and business needs, working closely with stakeholders to ensure data solutions support strategic objectives.
