IB Digital Society 3.1F Big Data and Data Analytics Notes

3.1F Big Data and Data Analytics

What Makes Data Big

1. The scale of big data goes beyond sheer size. It arrives in such quantity, at such speed, and in so many forms that ordinary tools cannot store or process it. Handling it needs specialised systems spread across many machines.
2. Scale changes the stakes. It opens up things that were possible only with huge datasets, like tracking a whole population in real time. It also magnifies mistakes: a single error can repeat billions of times.
3. The classic description is the four Vs. Volume is the sheer amount (billions of records). Velocity is the speed new data streams in, often real time from sensors, clicks, and transactions.
4. Variety is the mix of forms (tidy tables plus messy text, images, audio, video) that older databases were never built to hold together.
5. The fourth V, veracity, asks how trustworthy the data is. Noisy, incomplete, or biased data is not reliable just because there is a lot of it; it is the V most often forgotten.

Analytics: Extracting Insight at Scale

1. Data analytics examines raw data to find patterns and turn them into something a person can act on; at big-data scale this is done by machines that sift more than any human could read.
2. The work moves through stages: messy data is cleaned of errors and gaps, then analysed for patterns, and only then does a pattern become an insight guiding a decision.
   1. Skipping the cleaning stage lets errors and gaps pass straight into the analysis, so the conclusion looks confident but rests on faulty data.
3. Big-data analytics can spot faint patterns invisible at small scale. A trend you would miss in a hundred customers can show up clearly across a hundred million. This is a main reason organisations gather so much data.

Predicting from Patterns

1. Predictive analytics uses patterns in past data to forecast what is likely next (which customers will leave, which machines will fail, which patients are at risk), letting an organisation intervene early.
2. The logic is simple: fit a pattern to what happened, then extend it forward. Big data sharpens forecasts, but a forecast only ever gives a probability, not a guarantee of what will happen.
3. The stakes rise once a prediction becomes an action. A lender predicting default from a postcode can entrench disadvantage, and a biased forecast can harm someone over an outcome that would never have happened.

The Trap of Spurious Correlations

1. Searching millions of variables guarantees some line up by chance, producing a spurious correlation: a strong statistical link with no real connection. Scale multiplies coincidences as readily as real signals.
2. Correlation is not causation. Ice cream sales and shark attacks rise together, but neither causes the other. Hot weather drives both.
3. At scale the trap is industrial. The discipline is to ask whether a plausible mechanism links the variables and to test a found pattern on fresh data before trusting it.

Veracity and the Power of Aggregation

1. With enough data, quality does not stop mattering. If a large share is inaccurate, out of date, or biased, scaling multiplies the error while the polished result hides how shaky its foundation is.
2. Bias is the sharpest version. Data drawn from mostly one group fails everyone else, and a huge biased sample can look authoritative while being systematically wrong. Before trusting a result, check whose data it rests on and whether that group represents the population, not just how many records there are.
3. The deepest issue is aggregation. One location ping or purchase reveals almost nothing; combined with thousands, the same points build an intimate profile of habits, health, relationships, and home that was never knowingly handed over.
4. This gives real power to whoever holds aggregated data: they can know and predict people better than they know themselves, the engine behind targeted advertising and large-scale surveillance.
5. Big data is a shift in power. Aggregation lets a hospital spot an outbreak early, and it also lets a company or state watch a population in fine detail. Weigh both in any specific case.