Harnessing the power of big data – the future is now

The future is now

Poor data and analytics were identified as weaknesses during the global financial crisis. When Lehman Brothers collapsed in 2008, financial regulators, private sector managers and customers were unable to assess quickly the extent of market participants’ exposure to Lehman or to explore quickly and fully how the vast network of market participants were connected to one another. 

This article brings to fore the perspective that irrespective of the availability of multi-petabytes (1 petabyte = 1,000,000 gigabytes) of data, the limits of human cognition may prevent banks from making effective use of the data without application of technologies like ‘big data’ to enhance human information processing capabilities. According to industry pundits, the future would predominantly rely on the way data is looked upon or analysed by embedding smart cognitive and computational techniques. However distant it may seem, the future is now.

What is big data?

Data has always played a critical role in driving business decisions and strategies. Industries, organisations and clients increasingly mine large amounts of data for structural/operational efficiencies or gaining a competitive advantage in the market. This, however, is changing exponentially.

We live in the era of data explosion:

  • Every day we create 2.5 quintillion bytes of data (enough to fill 10 million Blu-ray discs)1 

  • 50,000 GB/s is the estimated rate of global Internet traffic by 2018 

  • 90% of generated data is ‘unstructured’ 

  • USD 3.1 trillion is the estimated amount of money that poor data quality costs the US economy per year 

  • One in three investors don’t trust the information they use to make decisions.

The term ‘big data’ thus refers to data sets whose volume, variety, velocity, and complexity make it extremely difficult for traditional tools and techniques to store, manage and process. By unlocking insights at a geometric rate, big data is changing the way organisations of all sizes across all industries perceive their business.

The real value from big data is expected to be in the form of high-end analytics, predominantly using data mining, statistics, optimisation, and forecasting capabilities. The analytics ­­proactively turn the large volume and variety of data into intelligence, to drive business benefits and better decision-making capabilities while also revealing hitherto untapped trends, patterns and correlations. 

Big data being vastly different from traditional data processing, a whole new set of tools and techniques (Hadoop, streaming analytics, machine learning to name a few) are also evolving as a ‘big data ecosystem’. These are designed to economically extract value by enabling massively huge storage and computing capabilities, high-velocity capture, discovery and/or analysis.

Big data in the financial industry

Today due to social media and heavy regulatory and performance pressure, the data sets have grown immensely in terms of size, type and complexity, and are awkward to work with using traditional database management tools. Many large financial and banking institutions are reaching the upper limits of their legacy systems and are now seeking fresh analytics and framework solutions. The key drivers of big data growth are shown in Chart 1.

Chart 1: Key drivers of big data growth
Chart 1: Key drivers of big data growth
Source: TCS internal research

The biggest secret of big data is that it doesn’t have a big secret. Instead, it has a number of little secrets, information that it puts together one vector at a time. Customers across industries are benefitting from this technology that transitions data to information and information to insights in a matter of seconds. 

Big data in action – Tata Consultancy Services perspective

The adoption of big data started slowly but with maturity has today reached a stage where decision-making is real-time and efficient. 

Tata Consultancy Services (TCS) has rich, progressive experience in handling tough business scenarios/challenges with big data technologies. Listed are a few of many pronounced scenarios where big data is delivering instantaneous value:

  • Economic value of data – within a large European investment bank 

    A big data platform coupled with advanced analytics techniques helped the bank derive hidden relationship patterns and trends from its immense pool of intrinsic and extrinsic data ecosystems. This solution also assisted in making its clients regulatory-compliant (FATCA/MiFID II/MLD 4, etc.)

  • Cognitive machine-learning-based predictive analytics solution to identify trade fails – one of top 10 US banks

    A big-data-based solution was used that employed elements of predictive analytics in conjunction with cognitive machine learning techniques to create propensity models that would assign confidence levels to a trade that might fail or not settle. This helped significantly reduce settlement risk.

  • Social media and sentiment analytics – used across industries

    Deriving insights from seamlessly raw pool of dark data sets originating from social media data. Analytics combined with big data was used to assess a variety of data sets, including those from clickstream, emails, credit and debit card transactions, Twitter, and Wikipedia. 

  • Anti-financial crime – market regulators, clearing houses, banking giants

    Big data technologies are used to process vast amounts of data to identify individual behaviour that could reveal risks or openings to make money or to identify potential rogue traders, who could conceivably bring massive losses. Big data does not replace banks’ current analytical infrastructure but simply extends its scope.

Case in point: Portfolio optimisation using big data analytics

Portfolio optimisation is the investment decision-making process to hold a set of financial assets to meet various criteria of the investors. In general terms, the criteria are maximising return and minimising risk. A big data analytics framework2, 3 (Chart 2), developed by the Indian Institute of Technology, integrates unstructured and structured data, that leads to an informed investment decision (which can further be predictively analysed using a behavioural bias algorithm). The framework would have the evolving elements shown in Chart 2.

Chart 2: Big data analytics framework for portfolio optimisation 2, 3
Chart 2: Big data analytics framework for portfolio optimisation
Source: Indian Institute of Technology Delhi, Tata Consultancy Services adapted from sources in footnotes 2 and 3

Previously, this accurate and predictive portfolio optimisation framework/service was only available to high-net-worth individuals and even then only on an ad hoc basis. However, with big data this information can be made available to all.

Big data technologies are enabling today what the industry pundits have been calling as the future of technology for the past decade. The power of efficient and timely insights into the market, the future – or for that matter: our daily needs – is now at our fingertips.

From choosing the best mobile handset to what hybrid exotic products to invest in, decisions now are way more quicker and better-informed, not to mention insightful.  

We believe that although there are not going to be definitive answers or perfect solutions for many of the future problems and questions relating to big data, the advent of big data technologies is going to be the starting point for a new breed of data-centric organisations. 

The future is now, we live in the future.

1 'Extracting business value from the 4 V’s of big data’, IBM, http://www.ibmbigdatahub.com/sites/default/files/infographic_file/4Vs_Infographic_final.pdf

2 'A Heuristic Stock Portfolio Optimization Approach Based on Data Mining Techniques’, Negar Koochakzadeh, Department of Computer Science, University of Calgary, Alberta, March, 2013, http://theses.ucalgary.ca/bitstream/11023/569/2/ucalgary_2013_koochakzadeh_negar.pdf

3 'A Big Data Analytical Framework For Portfolio Optimization’, Department of Management Studies, Indian Institute of Technology Delhi, http://epic.is.cityu.edu.hk/wibf14/docs/WIBF14-Papers/wibf14_submission_14.pdf

Commerzbank Disclaimer

The views expressed in this article are those of the author and may differ from the published views of Commerzbank Corporate Clients Research Department, the communication has been prepared separately of such department. No representations, guarantees or warranties are made by Commerzbank with regard to the accuracy, completeness or suitability of the data.