"Big data" - the data generated from the use of the internet, social media, healthcare records, and purchasing history - is exploding. Within the next 4 years, over 5 billion people will have internet access and so the velocity of data collection continues to rise.
Let's put this into context. All of the data from recorded history to the year 2003 amounts to approximately 5 billion gigabytes.1 In 2011, it took two days to generate that same volume of data. And in 2015, that same amount of data was available every 10 seconds - every 10 seconds! And it continues to increase with every minute of every day.
"Every day, we create so much data that 90% of the data in the world today has been created in the last two years alone".2
Just how much data is generated every minute? The numbers are astonishing. Using quantities we can relate to, every minute Google translates 69,500,000 words; Netflix subscribers stream 86,805 hours of video; and YouTube users share 400 hours of new video.3
Can we effectively make sense of such large quantities of data? Yes - to some extent.
Several companies have translated user activity into profitable gains. But as these companies and researchers have acknowledged, the sheer volume of data is larger than the tools currently available to analyze them.4 This limits our ability to answer questions based on big data in a cost- or time-effective manner. But methodologies and systems able to analyze such large quantities of data are in continuous development.
Big data is clearly profitable in the private sector - can this usefulness be extended further? The possibilities of big data are vast and include many public health arenas.
Meaningful healthcare information on a population and even an individual level may be used to improve patient outcomes, reduce costs, and increase quality of life.5,6 It is exciting that such enormous quantities of data can be brought to the level of the individual patient.7,8
Big data provides an opportunity to use mountains of information from millions of patients to better understand and treat medical illnesses.
With a rapid increase in the number of devices which can collect data, including watches and mobile phones, more data than ever is being generated and used for a multitude of purposes including to monitor and improve our understanding of the course of progressive diseases like Parkinson’s9 and Alzheimer’s.10
There have also been huge advances in the way that patients' medical information is recorded. According to the US Centers for Disease Control, in 2013 nearly 80% of physicians used some form of electronic medical record system, compared to just 18% in 2001.11 A database of electronic medical records will allow researchers to assess more data - increasing our understanding of population health.
Think of the benefit all of this data could have for an individual patient. Imagine if a doctor could input a patient’s medical information and out pops a recommendation for care - based on a database of thousands of relevant scientific journal articles as well as tens of thousands of similar patients. This possibility is starting to become a reality.
Companies have developed technology platforms that are able to answer questions using a combination of natural learning processing and machine learning based on large amounts of unstructured data.12 In recent years, these platforms have been adapted in an effort to transform the healthcare industry. Cognitive-computing systems are unlocking the power of big data in an effort to aid physicians in the diagnosis and treatment of patients fighting cancer.13
These systems provide a completely objective patient assessment based on their medical and social history. They allow for a virtual, collaborative effort between physicians and researchers worldwide, and fill gaps in healthcare shortages.
With further advances, these systems could be expanded to combat other complex diseases, including mental health disorders.
The clinical utility of systems medicine and cognitive-computing systems is exciting.
Ethical questions regarding the way in which data is collected and used are central to big data discussions.14,15 The way in which massive volumes of data are obtained throughout the world should be beneficial to society, and also comply with a set of best ethical principles. This will ensure the greatest degree of mutual benefit for those using the data and those from whom the data is collected.