What is big data?
Technical definition: Big Data refers to datasets whose size is beyond the ability of typical database software tools to capture, store, manage and analyze.*
Simple definition: Big Data is a buzzword for large amounts of data.
The Vs of Big Data:
1. Volume (High Quantity)
2. Velocity (High speed of capture)
3. Variety (High complexity)
4. Veracity (Trustworthiness)
5. Value (Meaningful)
6. Visualisation (Graphical representation)
Uses for Big Data in Finance:
-Financial modelling
-Real time analytics
-Algorithmic trading
-Machine learning
Example of big data being used in trading:
A Hedge Fund called Coniecto Technologies uses Big Data to estimate its investment risks. For this it gathers a lot of data to make more precise estimates.
1. Volume – Past data of market variables (stock indices, FX, Bonds, Commodities, Crypto price and volumes) since inception
2. Velocity – Hourly crypto price and volume data
3. Variety – Analyst reports, news feeds, sentiment analysis of Twitter
4. Veracity – Official data from trustworthy exchanges
5. Value – Helps to understand relationships between market variables, say, exchange rates and interest rates
6. Visualisation – Heat map indicating volatility of assets (Colours low to high volatility using a green to red gradient)
Challenges in using Big Data:
1. Access to data (Could be expensive or out of reach)
2. Inconsistent and incomplete data (For example – Missing values)
3. Heterogeneity of data (Diversity) (prices, reports, social media analytics etc)
4. Data privacy and protection (ethical considerations)
Some Big Data tools:
R
Apache Hadoop
