One of the newest trends in the BI world is “End User/Self Service BI”.
One of the newest trends in the BI world is “End User/Self Service BI”. When I’ve been involved as an IT participant in Self Service BI, there have been some obstacles to overcome, as well as a few pains in moving to a new way of producing Business Intelligence reporting.
The concept is being driven in part by vendors such as MicroStrategy, Tableau and QlikView. Business users are interested in it since they no longer are dependent upon IT’s schedules, delays and procedures, plus they can slice and dice the data however they see fit.
On the other hand, IT is concerned because they no longer “control” the reports or their quality. Political issues abound as control and responsibility moves from one group to the other. Haphazard metric creation or improper usage of data can create conflicting or incorrect reports, thus leading to poor business decisions.
Even though IT is not producing the actual reports, that is no excuse to toss out the rest of the reporting effort (like the ETL or data warehouse support) or lessen the quality that is expected. In fact, it’s critical that IT support the business users in ways that were not necessary when IT produced the reports.
• IT still manages data storage, movement and how it’s presented to the Business User for his report development, so we need to insure that it’s presented correctly and in a proper manner.
• IT needs to help educate the Business User on why our procedures exist, how to apply them to their own areas, and why they will benefit in the long run from doing so.
• IT can now act as mentors for report creators in order to ensure best practices are developed and enforced
As a partner in the new Self Service environment, IT has many opportunities to support the Business Users and help ensure their success, and along with them the Organization as a whole. There are essentially three areas of roles and responsibilities to consider.
IT should:
• Ensure that common datasets are used where possible when multiple groups need access to the same data. For example, do Finance and Marketing both need access to Sales data? If so, then have only one set of sales tables. Database Views can be used to provide custom data presentation using a single source while providing appropriate security.
• Ensure all parties agree on report accuracy. It is less important now which reports are “more important”, be they Month End Financial or Daily Summary. IT should partner with Business Users in during development so these reports can be used as the basis for both IT and User Acceptance testing to confirm the accuracy of the data.
• Develop and recommend a proper naming convention. This is to help ensure that data is named in a standard, consistent way which helps lead to proper usage. Field and Metric Names should be descriptive, and not cryptic, and use mixed/camel case. Allowed abbreviations should be documented and clear, like “Cd” for “Code”, or “Qty” for “Quantity”. Spell out the remainder of the words. For example, which is clear: GRAT_AMT or GratuityAmt?
• Performance Testing and Monitoring should be part of the IT Staff’s responsibility. This includes:
o Regular server monitoring to ensure that reporting servers and databases are not overtaxed.
o Periodicly reviewing of how much data each report is returning as well as how often it is being executed.
o Conduct reviews on potentially caching or posting frequently executed reports that are not filtered by user
o Appropriately governing specific users or groups in order to prevent excessive amounts of data to be retrieved from the database.
o Limiting the number of reports a user can simultaneously execute or store in memory
• Set up and maintain Security Roles in order for users or groups to only access information that is appropriate for their role. For example, Finance users typically have no need to access HR data, and vice-versa. Likewise, Finance users may only need to see data for their own group rather than data for the entire company.
Business Users should:
• Reports should have a description, included in the report Metadata if possible. This is more than “This report shows Sales for the Current Month”. It should include:
o details on who uses the report and for what reason
o a description of the data in the report
o how often the report is generated
o any filters or constraints on the data
o any sorting of the data, and on which columns
• Partnering with IT to agree up front where responsibilities lie and where they cross team lines and are joint responsibilities.
o Who will create the metrics, attributes and facts?
o Who is responsible for user management, such as adding new users, removing or changing user permissions?
o Who is responsible for moving reports from QA to Production?
o How are schema changes communicated and implemented?
o Who should be involved when the data does not seem to be correct?
o How often should server and report performance reviews take place?
IT and Business should jointly:
• Create and maintain the BI project documentation. This includes detailed descriptions and usage of each dimension, report, fact, attribute and metric. Descriptions should be detailed (as noted above). Here are two descriptions of “CustomerNumber” as discussed by Tom Krieger in his very informative blog:
o Good: The CustomerNumber uniquely identifies a Customer. The first character identifies whether the customer is a direct customer or via a broker (D or B), the second character indicates the nationality of the customer (U=USA, C=Canada, M=Mexico). The remainder of the characters are numeric and system generated. The Sales Department uses CustomerNumber to track customers and specific products they are purchasing, Marketing tracks advertising directed at specific customers, Finance uses CustomerNumber to track Accounts Receivable.
o Bad: The unique identifier of a customer.
• Implement a Dev->QA->Production process to ensure that new or modified reports are appropriately tested and approved before being placed into a Production environment. This may be difficult, and viewed by the Business as unnecessary, but it is critical to their own success.
o The process for creating Ad Hoc or one-time-only reports should be agreed upon. Clearly doing this “development in a production environment” could have serious performance implications.
o This process also should enforce a level of control on metrics in order to ensure their accuracy and eliminate duplication.
• Develop and conduct regular training classes for new users, including fact, attribute and metric overviews, reviews of project documentation and its maintenance.
In summary, Self Service BI is an opportunity for both IT and the business users to concentrate on what they do best. The key to a successful effort is ensuring a proper understand of roles and responsibilities is in place.
The concept is being driven in part by vendors such as MicroStrategy, Tableau and QlikView. Business users are interested in it since they no longer are dependent upon IT’s schedules, delays and procedures, plus they can slice and dice the data however they see fit.
On the other hand, IT is concerned because they no longer “control” the reports or their quality. Political issues abound as control and responsibility moves from one group to the other. Haphazard metric creation or improper usage of data can create conflicting or incorrect reports, thus leading to poor business decisions.
Even though IT is not producing the actual reports, that is no excuse to toss out the rest of the reporting effort (like the ETL or data warehouse support) or lessen the quality that is expected. In fact, it’s critical that IT support the business users in ways that were not necessary when IT produced the reports.
- IT still manages data storage, movement and how it’s presented to the Business User for his report development, so we need to insure that it’s presented correctly and in a proper manner.
- IT needs to help educate the Business User on why our procedures exist, how to apply them to their own areas, and why they will benefit in the long run from doing so.
- IT can now act as mentors for report creators in order to ensure best practices are developed and enforced
As a partner in the new Self Service environment, IT has many opportunities to support the Business Users and help ensure their success, and along with them the Organization as a whole. There are essentially three areas of roles and responsibilities to consider.
IT should:
- Ensure that common datasets are used where possible when multiple groups need access to the same data. For example, do Finance and Marketing both need access to Sales data? If so, then have only one set of sales tables. Database Views can be used to provide custom data presentation using a single source while providing appropriate security.
- Ensure all parties agree on report accuracy. It is less important now which reports are “more important”, be they Month End Financial or Daily Summary. IT should partner with Business Users in during development so these reports can be used as the basis for both IT and User Acceptance testing to confirm the accuracy of the data.
- Develop and recommend a proper naming convention. This is to help ensure that data is named in a standard, consistent way which helps lead to proper usage. Field and Metric Names should be descriptive, and not cryptic, and use mixed/camel case. Allowed abbreviations should be documented and clear, like “Cd” for “Code”, or “Qty” for “Quantity”. Spell out the remainder of the words. For example, which is clear: GRAT_AMT or GratuityAmt?
- Performance Testing and Monitoring should be part of the IT Staff’s responsibility. This includes:
- Regular server monitoring to ensure that reporting servers and databases are not overtaxed.
- Periodically reviewing of how much data each report is returning as well as how often it is being executed.
- Conduct reviews on potentially caching or posting frequently executed reports that are not filtered by user
- Appropriately governing specific users or groups in order to prevent excessive amounts of data to be retrieved from the database.
- Limiting the number of reports a user can simultaneously execute or store in memory
- Set up and maintain Security Roles in order for users or groups to only access information that is appropriate for their role. For example, Finance users typically have no need to access HR data, and vice-versa. Likewise, Finance users may only need to see data for their own group rather than data for the entire company.
Business Users should:
- Reports should have a description, included in the report Metadata if possible. This is more than “This report shows Sales for the Current Month”. It should include:
- details on who uses the report and for what reason
- a description of the data in the report
- how often the report is generated
- any filters or constraints on the data
- any sorting of the data, and on which columns
- Partnering with IT to agree up front where responsibilities lie and where they cross team lines and are joint responsibilities.
- Who will create the metrics, attributes and facts?
- Who is responsible for user management, such as adding new users, removing or changing user permissions?
- Who is responsible for moving reports from QA to Production?
- How are schema changes communicated and implemented?
- Who should be involved when the data does not seem to be correct?
- How often should server and report performance reviews take place?
IT and Business should jointly:
- Create and maintain the BI project documentation. This includes detailed descriptions and usage of each dimension, report, fact, attribute and metric. Descriptions should be detailed (as noted above). Here are two descriptions of “CustomerNumber” as discussed by Tom Krieger in his very informative blog:
- Good: The CustomerNumber uniquely identifies a Customer. The first character identifies whether the customer is a direct customer or via a broker (D or B), the second character indicates the nationality of the customer (U=USA, C=Canada, M=Mexico). The remainder of the characters are numeric and system generated. The Sales Department uses CustomerNumber to track customers and specific products they are purchasing, Marketing tracks advertising directed at specific customers, Finance uses CustomerNumber to track Accounts Receivable.
- Bad: The unique identifier of a customer.
- Implement a Dev->QA->Production or equivalent process to ensure that new or modified reports are appropriately tested and approved before being placed into a Production environment. This may be difficult, and viewed by the Business as unnecessary, but it is critical to their own success.
- The process for creating Ad Hoc or one-time-only reports should be agreed upon. Clearly doing this “development in a production environment” could have serious performance implications.
- This process also should enforce a level of control on metrics in order to ensure their accuracy and eliminate duplication.
- Develop and conduct regular training classes for new users, including fact, attribute and metric overviews, reviews of project documentation and its maintenance.
In summary, Self Service BI is an opportunity for both IT and the business users to concentrate on what they do best. The key to a successful effort is ensuring a proper understand of roles and responsibilities is in place.