As depicted in the Chart, above, the early results from our short survey to Financial Services firms show that structured internal data management is maturing with active projects planned or underway, while the state of unstructured internal and external data is less mature with fewer projects—even though there is an awareness that change is needed.
The maturity of internal structured data is reflective of the growth of data management initiatives over the last ten years to control the cost of acquiring data and also meet the growing regulatory reporting mandates for data and investment transparency. Automating data governance is the final phase of the maturity curve, with many firms currently evaluating options in this space.
Big Data often references the 3 V’s: Volume, Velocity and Variety. Whereas Structured Data in Financial Services often contains very large data sets and real time pricing and transaction data, it’s really the huge Volume (such as log surveillance) and Variety of unstructured data that adds complexity to traditional relational databases and opportunities for those firms who master it.
Anecdotally, we have seen that only the “early adopter” top tier firms -- buy side, sell side and banks – have started to invest in Big Data projects to enhance their unstructured internal and unstructured external data, and that the majority of smaller firms are aware they need to do something yet have not begun or are not sure where to start. This is despite the often quoted saying that 80% of Enterprise data is unstructured, and the well-publicized rapid growth of social analytics.
Unstructured internal data often contains valuable business insight into investment research documents (street and internal), staff on-boarding, knowledge management and client communications. Thought leading firms in many industries are investing to harvest this internal information, yet it seems to be a lower priority for financial services firms, perhaps due to the current investments in structured data projects.
Unstructured data from the web and social media is where the 3 V’s of Big Data are gaining a lot of traction with early adopters within the Financial Services industry.
- Marketers are creating highly segmented campaigns and using predictive analytics to increase conversion rates online and via social media optimization.
- Traders and portfolio managers are gaining broader insight into their investments, often before traditional data providers are aware of news or can update their feeds.
- Social media activity prompted the SEC to rule on April 2, 2013 that companies can use social media outlets to disclose material non-public information.
- The power of data science and predictive analytics working on vast data sets are being used to provide competitive advantage to investors.
- Very large research databases can be managed and mined more effectively on Big Data platforms such as Hadoop and NoSQL than on traditional relational platforms.
It’s rewarding to be involved in such an exciting time of rapidly changing technologies.