Categories
Concept Explainers

Big Data

Big Data – the lifeblood for quantitative trading strategies is a buzzword for large amounts of data.

What is big data?

Technical definition: Big Data refers to datasets whose size is beyond the ability of typical database software tools to capture, store, manage and analyze.*

Simple definition: Big Data is a buzzword for large amounts of data.

The Vs of Big Data:
1. Volume (High Quantity)
2. Velocity (High speed of capture)
3. Variety (High complexity)
4. Veracity (Trustworthiness)
5. Value (Meaningful)
6. Visualisation (Graphical representation)

Uses for Big Data in Finance:
-Financial modelling
-Real time analytics
-Algorithmic trading
-Machine learning

Example of big data being used in trading:
A Hedge Fund called Coniecto Technologies uses Big Data to estimate its investment risks. For this it gathers a lot of data to make more precise estimates.

1. Volume – Past data of market variables (stock indices, FX, Bonds, Commodities, Crypto price and volumes) since inception
2. Velocity – Hourly crypto price and volume data
3. Variety – Analyst reports, news feeds, sentiment analysis of Twitter
4. Veracity – Official data from trustworthy exchanges
5. Value – Helps to understand relationships between market variables, say, exchange rates and interest rates
6. Visualisation – Heat map indicating volatility of assets (Colours low to high volatility using a green to red gradient)

Challenges in using Big Data:
1. Access to data (Could be expensive or out of reach)
2. Inconsistent and incomplete data (For example – Missing values)
3. Heterogeneity of data (Diversity) (prices, reports, social media analytics etc)
4. Data privacy and protection (ethical considerations)

Some Big Data tools:
R
Apache Hadoop

*https://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/big-data-the-next-frontier-for-innovation

By sanika

Founder at Coniecto.

Leave a Reply

Your email address will not be published. Required fields are marked *

1 × 3 =