Are you looking for Big Data Jobs or Candidates? Please go to our WORK section
Big Data Solutions Architecture is an architecture domain that aims to address specific big data problems and requirements. Big data solutions architects are trained to describe the structure and behaviour of a big data solution and how that big data solution can be delivered using big data technology such as Hadoop. He or she needs to have hands-on experience with Hadoop applications (e.g. administration, configuration management, monitoring, debugging, and performance tuning).
The Big Data Solutions Architect is required in any organisation that wants to build a big data environment on premises or in the cloud. They are the link between the needs of the organization and the big data scientists and the big data engineers. The big data solutions architect is responsible for managing the full life-cycle of a Hadoop solution. This includes creating the requirements analysis, the platform selection, design of the technical architecture, design of the application design and development, testing, and deployment of the proposed solution.
A Big Data Solutions Architect generally should have a lot of experience gained in normal solutions architecture before making the move to big data solutions. 10-15 years of working experience is very common for this position. Obviously, he or she needs to have experience with the major big data solutions like Hadoop, MapReduce, Hive, HBase, MongoDB, Cassandra. Quite often they also need to have experience in big data solutions like Impala, Oozie, Mahout, Flume, ZooKeeper and/or Sqoop.
In addition to big data solutions, a big data solutions architect needs to have a firm understanding of major programming/scripting languages like Java, Linux, PHP, Ruby, Phyton and/or R. As well as have experience in working with ETL tools such as Informatica, Talend and/or Pentaho. He or she should have experience in designing solutions for multiple large data warehouses with a good understanding of cluster and parallel architecture as well as high-scale or distributed RDBMS and/or knowledge on NoSQL platforms.
When the big data solution will be developed in the cloud, the big data solutions architecture should have experience with one of the large cloud-computing infrastructure solutions like Amazon Web Services or Elastic MapReduce.
As may be clear by now, the Big Data Solutions Architect is a very skilled architect with cross-industry, cross-functional and cross-domain know-how. He or she sketches the big data solution architecture, then monitors and governs the implementation. The design of the Big Data Architecture is the basis of a big data platform and therefore the big data solutions architect should understand and have experience with data security and privacy concerns that could arise and that should be taken care of from the start.
The role of a big data solutions architect is a very technical one, but he or she should also have some other skills that are important in designing the right architecture for the right need:
- To be able to benchmark systems, analyse system bottlenecks and propose solutions to eliminate them;
- To be able to clearly articulate pros and cons of various technologies and platforms;
- To be able to document use cases, solutions and recommendations;
- To have excellent written and verbal communication skills;
- To be able to explain the work in plain language;
- To be able to help program and project managers in the design, planning and governance of implementing projects of any kind;
- To be able to perform detailed analysis of business problems and technical environments and use this in designing the solution;
- To be able to work creatively and analytically in a problem-solving environment;
- To be a self-starter;
- To be able to work in teams, as a big data environment is developed in a team of employees with different disciplines;
- To be able to work in a fast-paced agile development environment.
Of course, the perfect big data solutions architect with all the above skills, experience and know-how is hard to find. He or she should however have at least an understanding of a variety of hardware platforms including mainframes, distributed platforms, desktops, and mobile devices as well as a deep understanding of databases, data in storage and data in motion. In the end, the big data solutions architect is responsible for the overall design and development of a vision that underlies a projected big data solution.
