Research Guides: Statistics & Data: Datasets

What is a Dataset?

A dataset, or data set, is simply a collection of data.

The simplest and most common format for datasets you’ll find online is a spreadsheet or CSV format — a single file organized as a table of rows and columns. But some datasets will be stored in other formats, and they don’t have to be just one file. Sometimes a dataset may be a zip file or folder containing multiple data tables with related data.

How Are Datasets Created?

Different datasets are created in different ways. Some data sets will machine-generated data. Some will be data that’s been collected via surveys. Some may be data that’s recorded from human observations. Some may be data that’s been scraped from websites or pulled via APIs.

Whenever you’re working with a dataset, it’s important to consider: how was this dataset created? Where does the data come from? Don’t jump right into the analysis; take the time to first understand the data you are working with.

Dataset Sources

Awesome-Public-Datasets on Github
This github hosts a library of awesome, public datasets! They are all sorted by category and link you straight to the hosting website.
BuzzFeed News
BuzzFeed makes the data sets, analysis, libraries, tools, and guides used in its articles available on Github.
Data.gov
Data.gov allows you to download and explore data from multiple US government agencies. Data can range from government budgets to climate data.
data.world
data.world describes itself at ‘the social network for data people’, but could be more correctly describe as ‘GitHub for data’. It’s a place where you can search for, copy, analyze, and download data sets. In addition, you can upload your data to data.world and use it to collaborate with others.
Earthdata Search [NASA]
Earthdata Search enables data discovery and access to more than 33,000 Earth observation data collections from NASA’s EOSDIS, as well as from our U.S. and international partner agencies. Key features include a free text search and the ability to filter by science area (keyword), platform, instrument, or by day/time and spatial region. Customization (e.g., subsetting, regridding, reformatting) for select data collections is available.
FiveThirtyEight
FiveThirtyEight, sometimes rendered as 538, is an American website that focuses on opinion poll analysis, politics, economics, and sports blogging.
Google Dataset Search
Google Dataset Search is a search engine from Google that helps researchers locate online data that is freely available for use.
Kaggle
Kaggle is an online community of data scientists and machine learning practitioners. Kaggle allows users to find and publish data sets, explore and build models in a web-based data-science environment, work with other data scientists and machine learning engineers, and enter competitions to solve data science challenges.
Pew Research Center datasets
The Pew Research Center’s mission is to collect and analyze data from all over the world. They cover all sorts of topics like politics, social media, journalism, the economy, online privacy, religion, and demographic trends. While they do their own nonpartisan, non-advocacy research and analysis, they also offer their raw data for public access. Access simply requires a brief registration on the site and credit to Pew Research Center as the source of the data, with a waiver that Pew is not responsible for alternative data conclusions.
Planetary Data System [NASA]
The Planetary Data System (PDS) is a long-term archive of digital data products returned from NASA's planetary missions, and from other kinds of flight and ground-based data acquisitions, including laboratory experiments.
Quandl
Quandl is a repository of economic and financial data. Some of this information is free, but many data sets require purchase. Quandl is useful for building models to predict economic indicators or stock prices. Due to the large amount of available data sets, it’s possible to build a complex model that uses many data sets to predict values in another.
Socrata
Socrata hosts cleaned open source data sources ranging from government, business, and education data sets.
UCI Machine Learning Repository
The UCI Machine Learning Repository is one of the oldest sources of data sets on the web. Although the data sets are user-contributed, and thus have varying levels of documentation and cleanliness, the vast majority are clean and ready for machine learning to be applied. UCI is a great first stop when looking for interesting data sets.