Dark data is the material a company collects, stores and processes during its daily operations, but typically does not use for purposes that could be beneficial. There are numerous ways dark data is impacting the ways entities analyze and use data.
1. It’s Helping Historical Data Further Current Scientific Research
Data collected as part of publicly funded research occurring in the United States and some other nations must be kept in public databases. However, one research study investigated how materials collected in the 1970s and 1980s about zooplankton existed during a time when the technology was not available to publicize that data.
Scientists know this kind of dark data could give valuable information about how the ocean has changed since those past generations due to climate change. Therefore, there’s a modernized effort to tap into that dark data and use it to motivate ongoing research.
2. It’s Making Businesses Consider How Dark Data Fits Into GDPR Regulations
In May 2018, the General Data Protection Regulation (GDPR) will come into effect and have implications for all businesses in the European Union that store data or companies outside of it that collect data from or market to EU citizens.
One of the most interesting components of it is that consumers can exercise the so-called right to be forgotten and ask companies to delete their data under certain circumstances. However, fulfilling such requests could become time-consuming and costly if businesses continue to ignore dark data.
Statistics collected in 2016 by IBM found 80 percent of all data was the dark, unstructured variety. Not surprisingly, analysts have pointed out that under GDPR regulations, business representatives who don’t pay attention to dark data could be at increased risks for GDPR fines if dark data gets exposed during a breach. That’s in addition to the obvious challenges of a customer requesting removal of his or her dark data.
3. It’s Creating More Uses for Innovative Technologies
Data virtualization is a process that connects all of a company’s data sources and presents them in a unified dashboard, allowing people to see the compiled material in real time, or as close to it as possible. It also brings agility to data processing, and some product representatives believe their data virtualization solutions could bring visibility to dark data.
Another technological advancement that could facilitate helping businesses start to make dark data structured and usable is artificial intelligence (AI). When companies rely on AI to shed light on dark data, the associated algorithms and natural language processing could make the task much more achievable than if humans worked on it without such technologies.
4. It’s Enabling Establishments to Know Even More About People
A report from Deloitte titled Dark Analytics: Illuminating Opportunities Hidden Within Unstructured Data Tech Trends 2017 discusses how some medical facilities could use dark data to take more all-encompassing approaches to patient care.
For example, during consultations, doctors may take handwritten notes and capture voice recordings, plus make notes in emails or cloud-based applications. Collecting it all and making it accessible could improve treatments and insights, reducing the instances of incorrect diagnoses or interventions that don’t work as well as more appropriate options.
People in the health care field are also hopeful that dark data could make it possible to analyze population groups. The previously unused data could potentially make predictions about future needs and illness trends that could ultimately affect individuals’ interactions with health professionals and help local health departments understand the situations their staff members will most likely encounter.
The Deloitte report spotlighted retailers that are investigating how dark data could help them understand more about what customers want, as well. After getting permission to do so, an online clothing subscription company called Stitch Fix carried out a customer survey to get details about attire preferences, but then looked at those respondents’ social media feeds and Pinterest boards to dig for more data.
Most companies already realize they have accessible content that helps them learn about customers. By broadening their approaches and looking at dark data too, they could develop a complete picture of an audience and learn things that would otherwise remain unknown. For example, dark data could provide content that strongly suggests a product concept wouldn’t fit market needs by highlighting a pattern not captured within the structured data.
5. It’s Increasing Data Security Risks If Left Unstructured
It’s crucial to realize companies may not know the worth of their dark data, but that doesn’t mean hackers aren’t targeting it. Cybercrime experts say there’s been a shift in hacking strategies, resulting in unstructured data getting pinpointed by cybercriminals more than ever before.
There are other security risks associated with unstructured data, too, such as the inability to know what it contains and which employees at an organization should access it. Both of those issues could leave companies liable if unexpected circumstances unfold.
Furthermore, data held by a business may include sensitive details companies are legally provided to submit for court subpoenas, but it’s not always easy to find that in-demand data if it’s classified as dark. In one situation, a company spent $6 million searching for dark data requested for a court case, illustrating how costly it can be not to prioritize figuring out ways to classify dark data in practical ways.
By making plans to investigate dark data instead of acting like it doesn’t exist, companies can adopt proactive, rather than reactive, strategies. Otherwise, they could miss opportunities and experience unnecessary complications.