About OMG Digital + Annalect
In the summer of 2020, we connected the central Annalect & Digital teams to create a more efficient & effective central model. OMG Digital and Annalect is the Digital, Data, Performance, Product, Technology and Analytics division of OMG, sitting as a foundation of expertise. We operate at the group level within OMG as well as embedded in the individual agencies. In-agency, the teams bring the OMG Digital and Annalect proposition to life for clients across our full range of data, technology and analytics services. Data management, R&D and leadership sit at the centre to fuel capabilities across the group.
Objective of role
OMG Labs is building a Group wide data layer for the UK servicing Reporting, Data science, Programmatic and Product development initiatives. The Data Layer is the cornerstone of these efforts. A large number of pipelines are being built using a formulaic technology framework to facilitate and standardise data ingestion at scale. To help deliver this ambition we require a Senior Data Engineer to join our team, learn our data processing Framework and lead support, development and innovation activities across the data ingestion landscape.
What you'll do:
- Be an integral part of the Annalect Data Engineering team and Lead development & support efforts to ensure timely delivery of data to Reporting, Data science, Programmatic and Product development teams.
- Manage relationships with Agency stakeholders and the Data Science and Product teams to align delivery of data pipelines with expectations.
- Take ownership of the support of data pipelines in production ensuring they run successfully as per schedule.
- Proactively work with the wider OMG Labs team to introduce processes and solutions to improve the monitoring and reliability of all our production pipelines.
- Support Cost tracking policies and implementation
- Work with the Product team to support requirements definition and product lifecycle evolution and cadence of delivery.
- Be responsible for the data pipeline inventory, support documentation and the service transition of pipelines into production.
- Line management and coaching of junior engineers.
- Delivering new ETL pipeines using a mix of Airflow, Python & PySpark
What we need:
- Experience of working on the end-to-end development of large-scale data processing solutions, taking responsibility for non-functional needs of ETL/ELT data processing pipelines. Focussing on the design, code, test defect resolution and operational readiness, and includes setting the standards for these activities.
- Comfortable with senior stakeholder management and communication to Business Users/ teams.
- Ideally a background as BI/ ETL Developer with experience in building data pipelines in line with Business stakeholder requirements.
- Strong Python & SQL experience
- Software development experience with distributed data storage and processing technologies including Hadoop and Spark. Pipeline development using PySpark/Scala (ideally PySpark)
- Experience building pipelines using orchestration tools such as Airflow/DBT (ideally Airflow)
- Experience building data lakes and data marts using columnar storage formats such as Parquet.
- Experience in working with relational databases such as MSSQL, Postgres, etc.
- Good experience using cloud platforms such as AWS/GCP (Ideally AWS)
- Knowledge or experience using Kubernetes is a plus
We want everyone to make the most of the opportunity to shine and showcase your talents and we are happy to make adjustments in the recruitment process so you can be your best. Please discuss any specific requirements with your dedicated Talent Acquisition team member or if you would feel more comfortable, you can email us confidentially at to let us know how we can support you.
At OMG, our vision is to be an agency where difference is valued and everyone is able to thrive in a culture of equality, inclusion and belonging. We are committed to providing a truly inclusive environment that reflects today's society, where everyone is able to bring their true selves to work, and where diverse voices and backgrounds are valued, heard, and well-represented
We will process your personal data in accordance with our Recruitment Privacy Notice which is available on www.omgukcareers.com
