Lecture 4
Big Data
February 3, 2025
Big Data
Big Data
Big data and analytics are likely to be significant components of future careers across various fields.
Big data refers to enormous and complex data collections that traditional data management tools can’t handle effectively.
Five key characteristics of big data (5 V’s):
- Volume
- Velocity
- Value
- Veracity
- Variety
Big Data
1. Volume
- In 2017, the digital universe contained an estimated 16.1 zettabytes of data.
- Expected to grow to 163 zettabytes by 2025.
- Much new data will come from embedded systems in smart devices.
Big Data
1. Volume
| Name | Symbol | Value |
|---|---|---|
| Kilobyte | kB | 10³ |
| Megabyte | MB | 10⁶ |
| Gigabyte | GB | 10⁹ |
| Terabyte | TB | 10¹² |
| Petabyte | PB | 10¹⁵ |
| Exabyte | EB | 10¹⁸ |
| Zettabyte | ZB | 10²¹ |
| Yottabyte | YB | 10²⁴ |
| Brontobyte* | BB | 10²⁷ |
| Gegobyte* | GeB | 10³⁰ |
Note: The asterisks (*) next to Brontobyte and Gegobyte in the original image have been preserved in this table. These likely indicate that these units are less commonly used or are proposed extensions to the standard system of byte units.
Increase in size of the global datasphere
Big Data
2. Velocity
- Refers to the rate at which new data is generated.
- Estimated at 0.33 zetabytes each day (120 zetabytes annually).
- 90% of the world’s data was generated in just the past two years.
Big Data
3. Value
- Refers to the worth of data in decision-making.
- Emphasizes the need to quickly identify and process relevant data.
- Users may be able to find more patterns and interesting anomalies from “big” data than from “small” data.
Big Data
4. Veracity
- Measures the quality of the data.
- Considers accuracy, completeness, and currency of data.
- Determines if the data can be trusted for good decision-making.
Big Data
5. Variety
- Data comes in various formats.
- Structured data: Has a predefined format, fits into traditional databases.
- Unstructured data: Not organized in a predefined manner, comes from sources like documents, social media, emails, photos, videos, etc.
Sources of an Organization’s Data
Free Sources of Useful (Big) Data
Economics/Finance
| Data Source | Description | URL |
|---|---|---|
| Bureau of Labor Statistics (BLS) | Provides access to data on inflation and prices, wages and benefits, employment, spending and time use, productivity, and workplace injuries | BLS |
| FRED (Federal Reserve Economic Data) | Provides access to a vast collection of U.S. economic data, including interest rates, GDP, inflation, employment, and more | FRED |
| Yahoo Finance | Provides comprehensive financial news, data, and analysis, including stock quotes, market data, and financial reports | Yahoo Finance |
| IMF (International Monetary Fund) | Provides access to a range of economic data and reports on countries’ economies | IMF Data |
| World Bank Open Data | Free and open access to global development data, including world development indicators | World Bank Open Data |
| OECD Data | Provides access to economic, environmental, and social data and indicators from OECD member countries | OECD Data |
Free Sources of Useful (Big) Data
Government/Public Data
| Data Source | Description | URL |
|---|---|---|
| Data.gov | Portal providing access to over 186,000 government data sets, related to topics such as agriculture, education, health, and public safety | Data.gov |
| CIA World Factbook | Portal to information on the economy, government, history, infrastructure, military, and population of 267 countries | CIA World Factbook |
| U.S. Census Bureau | Portal to a huge variety of government statistics and data relating to the U.S. economy and its population | U.S. Census Bureau |
| European Union Open Data Portal | Provides access to public data from EU institutions | EU Open Data Portal |
| New York City Open Data | Provides access to datasets from New York City, covering a wide range of topics such as public safety, transportation, and health | NYC Open Data |
| Los Angeles Open Data | Portal for accessing public data from the City of Los Angeles, including transportation, public safety, and city services | LA Open Data |
| Chicago Data Portal | Offers access to datasets from the City of Chicago, including crime data, transportation, and health statistics | Chicago Data Portal |
Free Sources of Useful (Big) Data
Health, Climate/Environment, and Social Data
| Data Source | Description | URL |
|---|---|---|
| Healthdata.gov | Portal to 125 years of U.S. health care data, including national health care expenditures, claim-level Medicare data, and other topics | Healthdata.gov |
| CMS Open Payments | Centers for Medicare & Medicaid Services (CMS) Open Payments houses a publicly accessible database of payments that reporting entities, including drug and medical device companies, make to covered recipients like physicians. | CMS |
| World Health Organization (WHO) | Portal to data and statistics on global health issues | WHO Data |
| National Centers for Environmental Information (NOAA) | Portal for accessing a variety of climate and weather data sets | NCEI |
| NOAA National Weather Service | Provides weather, water, and climate data, forecasts and warnings | NOAA NWS |
| FAO (Food and Agriculture Organization) | Provides access to data on food and agriculture, including data on production, trade, food security, and sustainability | FAOSTAT |
| Pew Research Center Internet & Technology | Portal to research on U.S. politics, media and news, social trends, religion, Internet and technology, science, Hispanic, and global topics | Pew Research |
| Data for Good from Facebook | Provides access to anonymized data from Facebook to help non-profits and research communities with insights on crises, health, and well-being | Facebook Data for Good |
| Data for Good from Canada | Provides open access to datasets that address pressing social challenges across Canada | Data for Good Canada |
Free Sources of Useful (Big) Data
General Data Repositories
| Data Source | Description | URL |
|---|---|---|
| Amazon Web Services (AWS) public data sets | Portal to a huge repository of public data, including climate data, the million song dataset, and data from the 1000 Genomes project | AWS Datasets |
| Opportunity Insights | Harvard-based team offers big data to solve economic and social problems | Opportunity Insights |
| Gapminder | Portal to data from the World Health Organization and World Bank on economic, medical, and social issues | Gapminder |
| Google Dataset Search | Helps find datasets stored across the web | Google Dataset Search |
| Kaggle Datasets | A community-driven platform with datasets from various fields, useful for machine learning and data science projects | Kaggle Datasets |
| UCI Machine Learning Repository | A collection of databases, domain theories, and datasets used for machine learning research | UCI ML Repository |
| United Nations Data | Provides access to global statistical data compiled by the United Nations | UN Data |
| Humanitarian Data Exchange (HDX) | Provides humanitarian data from the United Nations, NGOs, and other organizations | HDX |
| Democratizing Data from data.org | A platform providing access to high-impact datasets, tools, and resources aimed at solving critical global challenges | Democratizing Data |
| Justia Federal District Court Opinions and Orders database | A free searchable database of full-text opinions and orders from civil cases heard in U.S. Federal District Courts | Justia |