In this article I would like to outline, at a high level, the overall thought process for “architecting” big data solutions. The process of developing an architecture is a very personal thing. I don’t propose to say that this is the only way – this is simply a framework from which you could launch your own effort.
In this article I would like to outline, at a high level, the overall thought process for “architecting” big data solutions. The process of developing an architecture is a very personal thing. I don’t propose to say that this is the only way – this is simply a framework from which you could launch your own effort.
Stage 1: Determine the Business Goal
The starting point for every architecture must be the business.
Some of the questions that should be answered are:
“What business goal or goals am I trying to achieve with this initiative?” This question must give you the problem that you are trying to solve in the language of the business. Something like “What are people saying on the internet about product X in the last 30 days.” presents a clear gold that can be acted on.
“How will the solution data, process, and systems integrate into or change the business process? By defining these characteristics we can determine the effect the solution will have, any inconsistencies or conflicts with existing business processes. This activity will generate additional clarity for all involved.
“Have I confirmed that the proposed change is required, beneficial, and consumable by the business?” One of the questions that we don’t ask enough is if the change is going to benefit the company or is it just change for the sake of change. Many times we can develop a solution that an excellent technical solution but does not fit with the organization. Benefit generation and Organizational fit provide solutions that cause less resistance to change.
Aligning your design efforts with the business is paramount to success. With Big Data it is doubly important in that the data and the analysis are significantly more impacting, especially when they will be directly influencing executive thinking. When we use Big Data to generate analytic data, executive dashboards or decision support systems we are presenting or executive leaders with decision making tools we must never take this privilege lightly.
Business alignment of IT projects creates a more efficient IT operation, better return on investment, reduced risk, and better clarity of vision.
IT operations are more efficient when aligned with business goals since only the projects that return business value are undertaken reducing wasted effort and erosion of the business’ trust. Trust takes a moment to lose and a lifetime to regain.
Return on investment is critical in demonstrating that IT understands the business drivers and is being a good steward of the business’ investment. We must always strive to demonstrate our understanding and unwavering support for the business.
IT can reduce business risk by thoroughly validating the risk factors for a solution and exposing them clearly to the business. Many businesses will accept these risks if the perceived payoff is large enough. When IT does not expose these risks to business we are being irresponsible and eroding trust.
Stage 2: Determine the Source and Type of Data
Now that we have a clear understanding of the business goals and how the business will utilize the data, we are ready to take a look at the source and type of data that you will be utilizing.
The source and type of data will drive the architectural decisions around integration, access, security, and extract-transform-load (ETL). We must determine how we will get the data, how to handle the data and how to secure the data to have a clear vision of how we will design the proposed solution. This activity will also provide us with an understanding of what is possible and not possible.
For example, if we are mining social media for comments on the company’s products we will have a much different design and implementation than if we were to collect data from the company’s web site clicks.
Stage 3: Develop & Document the Proposed Solution
Now that we have a clear understanding of the Business Goals and the Data we will be working with we can begin to develop the solution in line with the business goals, organizational capabilities, and enterprise standards.
Every aspect of the proposed solution should be tied to the business goals that we are supporting. This linkage will provide a strong indication to the business that we are thinking about their needs and now this system will support those needs.
As architects, system designers and business analysts we need to consider the organizational structure that we are designing solutions for so that we make good decisions that fit. For example a designer specifying a solution using Java in a dot net shop is likely to be taken out and shot. (well maybe not)
One of the places that we as designers can add a lot of value during the architecture process is when we are thinking through the architecture. When an innovative thought hits and we can predict some future requirement, issue or opportunity. We can present those in the architecture for discussion with the business. Everyone wins even if the lightning bolt was really a wet noodle at least we have thoroughly thought through the idea, communicated it and rationally included or excluded it from the solution documentation.
Conformance with Enterprise Standards presents another opportunity to support the operational efficiency of the business by seeking reuse and intellectual capital development. If we follow the enterprise standards and along the way develop additional capabilities that other units can use I have invested in the company for the future.
Documentation is something that must be done intelligently. We can definitely stall a project if we try to create too much documentation. Too little documentation will leave ambiguity in the solution that may come back to bite us. My general rule of thumb is to document to the point where ambiguity is removed and clarity of business and technical purpose are achieved.
Stage 4: Validate the Proposed Solution
This stage is critically important. This is the time that we are both validating the appropriateness of the proposed solution to the business and selling the solution to those who are not currently on board.
Call a meeting with the stakeholders and present the solution to them in their language. Demonstrate how this meets the business goals provided. Describe the risks and any mitigation strategies that we have developed for those risks. Clear up any lingering ambiguity that may still be present and get them input.
This activity will help the project to gain traction and garner buy in from your business stakeholders. Generating a win / win situation is the only way to drive a project to a successful solution.
Stage 5: Design the Final Solution
Now all of the input has been collected we are in a position to get down to the details of how we are going to technically implement the solution. We need to complete the systems and technology architecture design and documentation. Develop a team profile with disciplines, skills and experiences to assist with team selection and staffing. We need to work with the Project Manager to create a project and staffing plan. And finally work with the Implementation Teams to get the solution build, implemented and tested.
Also posted on IT Advisor