Are you looking for Big Data Jobs or Candidates? Please go to our WORK section
The Big Data Engineer builds what the big data solutions architect has designed. Big data engineers develop, maintain, test and evaluate big data solutions within organisations. Most of the time they are also involved in the design of big data solutions, because of the experience they have with Hadoop based technologies such as MapReduce, Hive MongoDB or Cassandra. A big data engineer builds large-scale data processing systems, is an expert in data warehousing solutions and should be able to work with the latest (NoSQL) database technologies.
A big data engineer should have sufficient experience in software engineering before the move can be made to the field of big data. Experience with object-oriented design, coding and testing patterns as well as experience in engineering (commercial or open source) software platforms and large-scale data infrastructures should be present. Big data engineers should also have the capability to architect highly scalable distributed systems, using different open source tools. He or she should understand how algorithms work and have experience building high-performance algorithms.
A big data engineer should embrace the challenge of dealing with petabyte or even exabytes of data on a daily basis. A big data engineer understands how to apply technologies to solve big data problems and to develop innovative big data solutions. In order to be able to do this, the big data engineer should have extensive knowledge in different programming or scripting languages like Java, Linux, C++, PHP, Ruby, Phyton and/or R. Also expert knowledge should be present regarding different (NoSQL or RDBMS) databases such as MongoDB or Redis. Building data processing systems with Hadoop and Hive using Java or Python should be common knowledge to the big data engineer.
A big data engineer generally works on implementing complex big data projects with a focus on collecting, parsing, managing, analysing and visualizing large sets of data to turn information into insights using multiple platforms. He or she should be able to decide on the needed hardware and software design needs and act according to the decisions. The big data engineer should be able to develop prototypes and proof of concepts for the selected solutions.
Additional qualifications should include:
- To enjoy being challenged and to solve complex problems on a daily basis;
- To have excellent oral and written communication skills;
- To be proficient in designing efficient and robust ETL workflows;
- To be able to work with cloud computing environments;
- To have a Bachelors or Masters degree in computer science or software engineering;
- To be able to work in teams and collaborate with others to clarify requirements;
- To be able to assist in documenting requirements as well as resolve conflicts or ambiguities;
- To be able to tune Hadoop solutions to improve performance and end-user experience;
- To have strong co-ordination and project management skills to handle complex projects.
The big data engineer is a technical job that requires substantial expertise in a broad range of software development and programming fields. The big data engineer should especially have sufficient knowledge of big data solutions to be able to implement those on premises or in the cloud.