The elements of big data analytics has roots in statistics, knowledge management, and computer science. Many of the data mining terms below appear in these disciplines but may have different connotation or specialized meaning when applied to our problems. The problems of massive parallel processing and the specialized algorithms employed to perform analysis in a distributed computing environment are enough to require specialized treatment.

Machine Learning Terms

Term Definition
Accuracy A measure of a predictive model that reflects the proportionate number of times that the model is correct when applied to data
Bias Difference between expected value and actual value
Cardinality The number of different values a categorical predictor or OLAP dimension can have. High cardinality predictors and dimensions have large numbers of different values (e.g. zip codes), low cardinality fields have few different values (e.g. eye color).
CART Classification and Regression Trees. A type of decision tree algorithm that automates the pruning process through cross validation and other techniques.
CHAID Chi-Square Automatic Interaction Detector. A decision tree that uses contingency tables and the chi-square test to create the tree. Classification. The process of learning to distinguish and discriminate between different input patterns using a supervised training algorithm. Classification is the process of determining that a record belongs to a group
Cluster Centroid most typical case in a cluster.  The centroid is a prototype. It does not necessarily describe any given case assigned to the cluster.
Clustering The technique of grouping records together based on their locality and connectivity within the n-dimensional space. This is an unsupervised learning technique.
Collinearity The property of two predictors showing significant correlation without a causal relationship between them
concentration of measure any set of positive probability can be expanded very slightly to contain most of the probability the average of bounded independent random variables is tightly concentrated around its expectation
Conditional Probability The probability of an event happening given that some event has already occurred. For example the chance of a person committing fraud is much greater given that the person had previously committed fraud
Confidence The likelihood of the predicted outcome, given that the rule has been satisfied.
convergence of random variables a sequence of essentially random or unpredictable events can sometimes be expected to settle down into a behaviour that is essentially unchanging when items far enough into the sequence are studied
correlation number that describes the degree of relationship between two variables
Coverage A number that represents either the number of times that a rule can be applied or the percentage of times that it can be applied
Cross-validation The process of holding aside some training data which is not used to build a predictive model and to later use that data to estimate the accuracy of the model on unseen data simulating the real world deployment of the model.
Data Mining Process Define the problem. Select the data. Prepare the data. Mine the data. Deploy the model. Take business action.
Discrete Fourier Transform Concentrates energy in first few coefficients
Entropy A measure often used in data mining algorithms that measures the disorder of a set of data
Error Rate A number that reflects the rate of errors made by a predictive model. It is one minus the accuracy
Expectation–maximization algorithm for estimating parameters where there exist significant missing or inferred values
Expectation-Maximization (EM) Solves estimation with incomplete data. Iteratively use estimates for missing data and continue until convergence
Expert System A data processing system comprising a knowledge base (rules), an inference (rules) engine, and a working memory
Exploratory Data Analysis The processes and techniques for general exploration of data for patterns in preparation for more directed analysis of the data
Factor Analysis A statistical technique which seeks to reduce the number of total predictors from a large number to only a few “factors” that have the majority of the impact on the predicted outcome.
Fuzzy Logic A system of logic based on the fuzzy set theory
Fuzzy Set A set of items whose degree of membership in the set may range from 0 to 1
Fuzzy System A set of rules using fuzzy linguistic variables described by fuzzy sets and processed using fuzzy logic operations
Genetic Algorithm Optimization techniques that use processes such as generic combination, mutation, and natural selection in a design based on the concepts of  revolution
Genetic Operator An operation on the population member strings in a genetic algorithm which are used to produce new strings
Gini Index A measure of the disorder reduction caused by the splitting of data in a decision tree algorithm. Gini and the entropy metric are the most popular ways of selected predictors in the CART decision tree algorithm
Hebbian Learning One of the simplest and oldest forms of training a neural network. It is loosely based on observations of the human brain. The neural net link weights are strengthened between any nodes that are active at the same time.
Hill Climbing A simple optimization technique that modifies a proposed solution by a small amount and then accepts it if it is better than the previous solution. The technique can be slow and suffers from being caught in local optima
Hypothesis Testing The statistical process of proposing a hypothesis to explain the existing data and then testing to see the likelihood of that hypothesis being the explanation
ID3 Decision Tree algorithm
Intelligent Agent A software application which assists a system or a user by automating a task. Intelligent agents must recognize events and use domain knowledge to take appropriate actions based on those events.
Itemset An itemset is any combination of two or more items in a transaction
Jackknife Estimate estimate of parameter is obtained by omitting one value from the set of observed values. Allows you to examine the impact of outliers.
Kernel a function that transforms the input data to a high-dimensional space where the problem is solved
k-Nearest Neighbor A data mining technique that performs prediction by finding the prediction value of records (near neighbors) similar to the record to be predicted
Kohonen Network A type of neural network where locality of the nodes learn as local neighborhoods and locality of the nodes is important in the training process. They are often used for clustering
Latent variable variables inferred from a model rather than observed
Lift A number representing the increase in responses from a targeted marketing application using a predictive model over the response rate achieved when no model is used
Machine Learning A field of science and technology concerned with building machines that learn. In general it differs from Artificial Intelligence in that learning is considered to be just one of a number of ways of creating an artificial intelligence
maximum likelihood method for estimating the parameters of a model
Maximum Likelihood Estimate (MLE) Obtain parameter estimates that maximize the probability that the sample data occurs for the specific model. Joint probability for observing the sample data by multiplying the individual probabilities.
Mean Absolute Error AVG(ABS(predicted_value – actual_value))
Mean Squared Error (MSE) expected value of the squared difference between the estimate and the actual value
Memory-Based Reasoning (MBR) A technique for classifying records in a database by comparing them with similar records that are already classified. A form of nearest neighbor classification.
Minimum Description Length (MDL) Principle The idea that the least complex predictive model (with acceptable accuracy) will be the one that best reflects the true underlying model and performs most accurately on new data.
Model A description that adequately explains and predicts relevant data but is generally much smaller than the data itself
Neural Network A computing model based on the architecture of the brain. A neural network consists of multiple simple processing units connected by adaptive weights
Nominal Categorical Predictor A predictor that is categorical (finite cardinality) but where the values of the predictor have no particular order. For example, red, green, blue as values for the predictor “eye color”.
Ordinal Categorical Predictor A categorical predictor (i.e. has finite number of values) where the values have order but do not convey meaningful intervals or distances between them. For example the values high, middle and low for the income predictor
Outlier Analysis A type of data analysis that seeks to determine and report on records in the database that are significantly different from expectations. The technique is used for data cleansing, spotting emerging trends and recognizing unusually good or bad performers
overfitting The effect in data analysis, data mining and biological learning of training too closely on limited available data and building models that do not generalize well to new unseen data. At the limit, overfitting is synonymous with rote memorization where no generalized model of future situations is built
Point Estimation estimate a population parameter. May be made by calculating the parameter for a sample. May be used to predict value for missing data.
Predictive model model created or used to perform prediction. In contrast to models created solely for pattern detection, exploration or general organization of the data
Predictor The column or field in a database that could be used to build a predictive model to predict the values in another field or column. Also called variable, independent variable, dimension, or feature.
Principle Component Analysis A data analysis technique that seeks to weight the importance of a variety of predictors so that they optimally discriminate between various possible predicted outcomes
Prior Probability The probability of an event occurring without dependence on (conditional to) some other event. In contrast to conditional probability
Purity/Homogeneity the degree to which the resulting child nodes are made up of cases with the same target value
Radial Basis Function Networks Neural networks that combine some of the advantages of neural networks with those of nearest neighbor techniques. In radial basis functions the hidden layer is made up of nodes that represent prototypes or clusters of records
Receiver Operating Characteristic (ROC) The area under the ROC curve (AUC) measures the discriminating ability of a binary classification model. The larger the AUC, the higher the likelihood that an actual positive case will be assigned a higher probability of being positive than an actual negative case. The AUC measure is especially useful for data sets with unbalanced target distribution (one target class dominates the other).
Regression A data analysis technique classically used in statistics for building predictive models for continuous prediction fields. The technique automatically determines a mathematical equation that minimizes some measure of the error between the prediction from the regression model and the actual data
Reinforcement Learning A training model where an intelligence engine (e.g. neural network) is presented with a sequence of input data followed by a reinforcement signal
Root Mean Squared Error SQRT(AVG((predicted_value – actual_value) * (predicted_value – actual_value)))
Sampling The process by which only a fraction of all available data is used to build a model or perform exploratory analysis. Sampling can provide relatively good models at much less computational expense than using the entire database
Segmentation The process or result of the process that creates mutually exclusive collections of records that share similar attributes either in unsupervised learning (such as clustering) or in supervised learning for a particular prediction field
Sensitivity Analysis The process which determines the sensitivity of a predictive model to small fluctuations in predictor value. Through this technique end users can gauge the effects of noise and environmental change on the accuracy of the model
Simulated Annealing An optimization algorithm loosely based on the physical process of annealing metals through controlled heating and cooling
Sparsity This means that a high proportion of the nested rows are not populated.
Statistical Independence The property of two events displaying no causality or relationship of any kind. This can be quantitatively defined as occurring when the product of the probabilities of each event is equal to the probability of the both events occurring
Stepwise Regression Automated Regressions to identify most predictive variables.  1st regression finds most predictive, 2nd regression finds most predictive given 1st regression.
Supervised Algorithm A class of data mining and machine learning applications and techniques where the system builds a model based on the prediction of a well defined prediction field. This is in contrast to unsupervised learning where there is no particular goal aside from pattern detection.
Support The relative frequency or number of times a rule produced by a rule induction system occurs within the database. The higher the support the better the chance of the rule capturing a statistically significant pattern.
Time-Series Prediction The process of using a data mining tool (e.g., neural networks) to learn to predict temporal sequences of patterns, so that, given a set of patterns, it can predict a future value
Unsupervised Algorithm A data analysis technique whereby a model is built without a well defined goal or prediction field. The systems are used for exploration and general data organization. Clustering is an example of an unsupervised learning system
Visualization Graphical display of data and models which helps the user in understanding the structure and meaning of the information contained in them

 

A “Runbook” or “Run Book” by any other name…

We use a runbook template to document what processes occur automatically or on demand. Typically these events are either date-triggered or event-triggered. An example of a date-triggered process might be a scheduled email sent nightly at midnight. An example of an event-triggered process might be to send an email when any disk on a server is 90% full.a run book captures processes

Configuration Management

Use this template as a starting point for building your own. It should be a key element of your Configuration Management Database (CMDB) used to document and operate your enterprise.

Quickstart

The simple runbook template below features three tabs. The first is to list all your automated processes–those that occur without assistance. The second is operational process, which captures tasks your team may perform either on demand or based on some event. The last is a lookup tab for dropdown values in the other tabs. You can ignore this.

The tabs are pretty self-explanatory. Fill in each task that your group performs, how often, how long it normally takes, and any accompanying notes. Feel free to add more columns to capture other configuration items such as who to contact in case of failure, email distribution groups who support the process, organization the process supports, etc.

https://augmentedintel.com/wp-content/uploads/2013/05/operational-runbook-template.xlsx

 

Please note that you are free to use this for any purposes, public or private.  I submit it as a (hopefully) helpful tool back to the community of professionals who have given me the advice and feedback and trialed it in real-world operations.

computer vision training images

Below are many downloadable free machine learning datasets. They cover click data, air traffic control data, surveys, temporal datasets of various types, crime data, employee pay data, map data, law data, and many other types.

 

I am a huge fan of SSPS, scikit-learn, opennlp, and other mainstream libraries but for quick analysis and visualization don’t forget about Pentaho data mining (
https://wiki.pentaho.com/display/DATAMINING/Pentaho+Data+Mining+Community+Documentation ) based on University of Waikato’s Weka (
https://www.cs.waikato.ac.nz/ml/index.html ). It can also be used with Pentaho Kettle to submit to a hadoop cluster and perform advanced multi-step analysis.

Public data resources: research-quality, free data mining data sets

All datasets with keywords

Search entries:

Announcing the Article Search API – Open Blog – NYTimes.com

Text mining` article` api` text` corpus` newspaper

Information Extraction: The RISE Repository of Information Sources

Text mining` information` text mining` extraction` reviews` jobs

Repositories

Text mining` links` text mining` books` rdf` ocr` documents

API Documentation — BackType

Text mining` api` blog` comments` text mining` stream` trends` backtype` queryminer

Free book usage data from the University of Huddersfield » “Self-plagiarism is style”

Text mining` books` library` borrowing` recommender` isbn` recommendation` collaborative` filtering

ICWSM 2009 – International AAAI Conference on Weblogs and Social Media

Text mining` blog` crawl` corpus` network` web` link, data mining data sets

Change.gov: The Obama-Biden Transition Team | Join the Discussion: Healthcare

Text mining` textmining` opinion` comment` topic` government` queryminer

Opinion Extraction, Opinion Mining, Sentiment Analysis, Summarization of Customer Reviews

Text mining` sentiment` mining` classification` machine learning` reviews` recommender` text mining` links

http://www.yr-bcn.es/semanticWikipedia

Text mining` wikipedia` named entity` tagged` text ming

Building a (fast) Wikipedia offline reader

Text mining` django` wikipedia` compressed` text mining` howto

Reddit’s Secret API

Text mining` reddit` api` json`

phishingcorpus [JoseWiki]

Text mining` phising` corpus` text` email` text mining` nlp` mail` security

Wikipedia Datasets for the Hadoop Hack | Cloudera

Text mining` wikipedia` hadoop` textmining` links

Main Task QA Data

Text mining` question` answering` trec` nlp` machinelearning

The New York Times Annotated Corpus « YooName – named entity recognition

Text mining` named entity` nytimes` corpus` people` organizations` locations

ADL Gazetteer Development

Text mining` named entity` location` place names` geo` nlp` natural language processing

Beautiful Data – WikiContent

Text mining` book` data` wiki` via:jhammerb

Web FAQ collection | ILPS

Text mining` faq` question_answering` questions` web` crawl` corpus` xml` textmining

Wikipedia:Lists of common misspellings/For machines – Wikipedia, the free encyclopedia

Text mining` spelling` mispelling` wikipedia, data mining data sets

build.kiva: Blog – Introducing the Kiva API

Business and Finance` finance` api` social` kiva` microlending` lending

Visualizing the Growth of Target, 1962-2008 | FlowingData

Business and Finance` visualization` retail` finance` gis` map` location` store` via:magnetbox

The Economy According To Mint

Business and Finance` finance` commercial` consumer` mint` spending

Best Buy Remix – Welcome to the Best Buy Remix Developer Network

Business and Finance` retail` data` api` product` bestbuy

Behavioral Targeting, Analytics and Advertising Service for Publishers, Ad Networks

Business and Finance` analytics` audience` segmentation` toolbar` commercial` sem` search` advertising

Executive PayWatch Database

Business and Finance` ceo` compensation` pay` economics` business` labor

TradingSolutions – Data Sources

Business and Finance` trading` finance` s` api` list

Netflix API – Welcome to the Netflix Developer Network

Business and Finance` netflix` api` movie` mashup` netflixprize` ratings

Open beats Closed: Best Buy’s new APIs – O’Reilly Radar

Business and Finance` retail` bestbuy` api

Tickermine

Business and Finance` custom` research` retail` finance` market` service` analyst`

University of Arkansas – Daily Headlines

Business and Finance` retail` dillards` uark

developerWorks Interviews: Massive data mining and the resurgent mainframe

Business and Finance` price` retail` transaction` sams_club` dillards

opentick :: market data

Business and Finance` opentick` nasdaq` finance` stock` data mining data sets

U.S. Company Filings and Annual Reports

Business and Finance` finance` links` sec

FTP Information – EDGAR Database

Business and Finance` edgar` finance` sec` filing` ftp` instructions

Data Mining For Investing

Business and Finance` investing` finance` datamining` announcement` sec` filing` links

UN General Assembly Voting Data

Government` un` voting` statistics` government

Research Datasets :: CID Data :: Center for International Development at Harvard University (CID)

Government` economics` international` development

Subsidyscope.com

Government` government` banking` csv` tarp` bailout

Data Catalog

Government` dc` government` feeds` transparency` opendata

Announcing the New York Times Campaign Finance API – Open – Code – New York Times Blog

Government` nyt` api` campaign` donations` fec`

Voter registration data; or, HERE IS YOUR HOPE, YOU FOOLS! « The Edge of the American West

Government` voter` registration` politics`2008

import/parse/fec.py at master from aaronsw’s watchdog — GitHub

Government` fec` python` parser` government` campaign

The Watchdog Project: volunteer

Government` government` transparency` parsing` election` python

Dataset of the day: Where are the Obamacans? | Off the Map – Official Blog of FortiusOne

Government` obama` goverment` mashup` gis` geo` map` campaign` donations

Normalized Campaign Contribution Data

Government` cmu` politics` campaign` donations` fec` via:jhammerb` government

Crime data bonanza!!!

Government` timeseries` crime` statistics` publicdata

Ohio voter registration data

Government` voter` voting` politics` government` name` address` registration

Voter List Data Files – Election Department, Clark County, Nevada

Government` voting` voter` registration` name` address` data` election` politics

UNdata

Government` UN` publicdata` government` statistics

RealClearPolitics – Election 2008 – Democratic Presidential Nomination

Government` polls` politics

Crime in the United States 2006

Government` crime` fbi

Daily Kos: Obama helps us track $1,000,000,000,000 of federal spending

Government` corruption` government` politics` finance`

Welcome to USAspending.gov

Government` government` money` politics`

Campaign Finance Reports and Data

Government` campaign` politics` elections

ERS/USDA Data – International Macroeconomic Data Set

Government` usda` economics` population` cpi` gdp` income

State Agency Databases – GODORT

Government` government` directory` links` wiki` states

National Bureasu of Economic Research: Data

Government` economics` links

Bureau of Labor Statistics Data

Government` economics` lumber` building` materials` homedepot

NBI ASCII Files – Bridge – FHWA

Government` government` bridges` safety

Twitter API Wiki / REST API Documentation: Social Graph Methods

Network Analysis` graph` network` api` social` twitter

Using the Wikipedia link dataset — Henry Haselgrove

Network Analysis` graph` network` link` wikipedia` pagerank

twibs : find the businesses on twitter

Network Analysis` directory` businesses` twitter` companies

Massive Scrape of Twitter’s Friend Graph « blog.infochimps.org – Organizing Huge Information Sources

Network Analysis` textmining` twitter` network` socialnetwork` pagerank` graph` queryminer

Twitter Scrape (rough draft) – get.theinfo | Google Groups

Network Analysis` twitter` socialnetwork` graph

wiki.dbpedia.org : Downloads 32

Network Analysis` wikipedia` named_entity` rdf` ontology

ICWSM 2009 – International AAAI Conference on Weblogs and Social Media

Network Analysis` blog` crawl` corpus` network` web` link

Linked Movie Data Base

Network Analysis` rdf` movies` movie` api

YouTube Dataset

Network Analysis` youtube` research` crawl` socialnetwork` network` graph` web

API Documentation – Twitter Development Talk | Google Groups

Network Analysis` twitter` text` api

CRAWDAD

Network Analysis` wireless` RF` radio` signal` dartmouth` network

Yahoo! Music API – YDN

Network Analysis` api` yahoo` music` artists

Lookery Developer Network – Lookery Developer Resources

Web Analytics` web` analytics` api` traffic` advertising` demographics` lookery

True Marble Imagery – Free Download

Spatial Analysis` gis` geo` map` mapping` images` satellite

Zillow – Labs – Neighborhood Boundaries

Spatial Analysis` neighborhoods` geo` gis` maps

Full Examples — PyMVPA Home

Image Analysis and Video Analysis` fmri` neuroscience` python` neuralnetwork

HumanScan : BioID : Downloads : BioID Face Database

Image Analysis and Video Analysis` face` detection` image

Face Detection

Image Analysis and Video Analysis` facerecognition` opencv` face` links

NORB Object Recognition Dataset, Fu Jie Huang, Yann LeCun, New York University

Image Analysis and Video Analysis` image` 3d

LIFE photo archive hosted by Google

Image Analysis and Video Analysis` images` photo` pictures` search

Activity Recognition: Datasets, Bibliography and others

Image Analysis and Video Analysis` activity` recognition` intent

Frontal Face Databases

Image Analysis and Video Analysis` facerecognition` face` image` recognition

Copyright Free and Public Domain Media

Image Analysis and Video Analysis` images` audio` publicdata` maps` video` free

Databases you can use for benchmarking

Image Analysis and Video Analysis` image` vision` recognition`

2007 IEEE AVSS Detection and Tracking Algorithm Datasets

Image Analysis and Video Analysis` tracking` video` detection` image` recognition` vehicle` pedestrian`

OTCBVS

Image Analysis and Video Analysis` image` recognition` detection` pedestrian` thermal` tracking` facerecognition` illumination

Carnegie Mellon University – CMU Graphics Lab – motion capture library

Image Analysis and Video Analysis` gait` pedestrian` walk` motion

public domain sounds | free sound library

Audio Analysis` sound` publicdomain` audio

Full Examples — PyMVPA Home

Bioinformatics` fmri` neuroscience` python` neuralnetwork

CinC Challenge 2000 data sets

Medical Informatics` timeseries` machinelearning` ecg` health` medical` sleep` apnea

UC Berkeley. Sheldon Margen Public Health Library. Statistical/Data Resources

Healthcare Analytics` health` links` resources` publichealth` berkeley

Google Flu Trends | How does this work?

Healthcare Analytics` google` health` trends` search` prediction` epidemiology` biodefence` queries

Eigenvector Research, Inc. : Data Sets Available to Download

Chemoinformatics` NIR` spectra` chemistry` semiconductor` pharmaceutical` matlab`

Vaccines: IIS/Tech/Deduplication Test Cases

Healthcare Analytics` duplicate

Health Data Tools and Statistics

Healthcare Analytics` health` information` public` publicdata

Cardiac MRI dataset – York University

Healthcare Analytics` mri` cardiac

NACDA: Search Holdings

Demography` aging` statistics` studies

Poverty Data Sets General Information

Demography` poverty` statistics

Pew Internet & American Life Project

Demography` internet` demographics` online` web

The 2000 U.S. Census: 1 Billion RDF Triples

Demography` gis` census` rdf` semantic` sparql

Download Database – baseball1.com

Sports Analysis` baseball` database` publicdata` statistics` sports

It’s a Pitch-by-Pitch Scouting Report, Minus the Scout – New York Times

Sports Analysis` baseball` gameday

BART – For Developers

Network Analysis` urban` transportation` feeds` public` sanfrancisco` bart` api`

Tim Davis: UF Sparse Matrix Collection : sparse matrices from a wide range of applications

Matrices` spare` matrix

Idealware: Mapping Blues: Where is the Data?

Pre-processing` resources` links`mapping

Amazon Web Services Public Datasets » Data Wrangling Blog

Amazon Web Services` amazon` ebs` ec2` s3` publicdata` hadoop

Amazon Web Services (AWS) Hosted Public Data Sets

Hosted Datasets` amazon` ebs` publicdata

WSCD09: Workshop on Web Search Click Data 2009

Web Analytics` workshop` search` web` microsoft` log`

downloading – flossmole – Google Code – How to get FLOSSmole data for your own use

Google` opensource` project` activity` mysql` dump

Multi-Domain Sentiment Dataset

Supervised` sentiment` review` product` amazon

Chris Pound’s Name Generation Page

 bizzare` scifi` phrase` name` word` generators` random` perl

Big Huge Thesaurus API: Access 145,000 Words and Phrases

Phrases` webservice` api` thesaurus` textmining` nlp` rest`

Search Query Performance report – Google AdWords Help Center

Performance` adwords` ppc` search` metrics` webanalytics` sem` query` queryminer

Wordze Keyword Research Tool

Web Analytics` queryminer` keyword` tool` research` commercial` search` adwords

Searchable Catalogs of Data

Network Analysis` links` catalogs` social

radiohead – Google Code

Audio Analysis` lidar` visualization` radiohead` google` video

80 Million Tiny Images

Image Analysis and Video Analysis` images` words` english` search` visualization` imagemap

Time Series Center | Harvard University

Temporal Analysis` timeseries` anomaly` detection` astronomical` physics

OpenVisuals – Open Source Visualization Framework

Image Analysis and Video Analysis` visualization` community` design` processing

BGN: Domestic Names – State and Topical Gazetteer Download Files

Demography` gis` usgs

Datasets

Random` benchmark` clustering` regression` machinelearning` list` statistics` mathematics

Isomap Datasets

Image Analysis and Video Analysis` nonlinear` dimensionality` reduction` faces` digits` images` manifold

Yahoo! Search Blog: BOSS — The Next Step in our Open Search Ecosystem

oss` api` open` search` yahoo` BOSS` queryminer

Download the Database – IP Address Lookup – Community Geotarget IP Project

Network Analysis` geocoding` geoip` internet` ip` ipaddress` mysql

Airline Data Project

Government` airline` statistics` finance` revenue` location` travel

Show Us a Better Way: What public data is already available?

Government` statistics` census` uk` school` news` publicdata

NGA: Country Files

Government` country` cities` geo

OHPI – Traffic Volume Trends

Government` government` traffic` statistics` trends` transportation

Semantic Search the US Library of Congress

Government` via:inkdroid` libraries` mashup` rdf` semantic` search` semanticweb` books

reddit.com: Ask Reddit: Where to download a DB dump of Reddit?

Text mining` reddit` socialnetwork` news` web

Collaborative filtering dataset – dating agency

Text mining` collaborative` filtering` dating` rating` profiles` czech

About Us – Predictify

Business and Finance` predictionmarket` tool` finance` buzz` advertising` marketing` startup` mmds

VGChartz.com | Video Games, Charts, News, Forums, Reviews, Wii, PS3, Xbox360, DS, PSP

Business and Finance` sales` ranking` videogames` retail

Store Level Information

Business and Finance` retail` finance` sales` store`

Code for querying and downloading Flickr images

Image Analysis and Video Analysis` image` python` code` flickr` matlab` recognition

Image Parsing Datasets

Image Analysis and Video Analysis` image` recognition

TAGora » Data

Network Analysis` tag` tagging` s

TAGora » Data

Network Analysis` netflixprize` imdb` sparql

Quality of Life Grand Challlenge Dataset: Kitchen Capture

Image Analysis and Video Analysis` machinelearning` motion` capture` sensor

Summize Twitter Search API

Text mining` api` buzz` opinion` trends` text` twitter` summize` search

2008 IEEE InfoVis Contest Dataset

Image Analysis and Video Analysis` visualization` contest` scalability` motion` tracking` pedestrian` sensor

IMDb Pro : Scary Movie 4: Box office

Business and Finance` movie` revenue` sales` box_office` imdb` commercial` movie_study

Spider-Man 2 (2004) – Daily Box Office Results

Business and Finance` movie` revenue` box_office`

Live Search : xRank™ Celebrity — check out who’s hot and who’s not!

Network Analysis` search` query` volume` trends` celebrity` prediction` buzz` named_entity

IMDbPro.com Free Trial Signup

Business and Finance` movie` revenue` timeseries` imdb` commercial` subsription

Free time-series and micro-data to download

Business and Finance` economics` links

PyGTrends: Python API for Google Trends Data

 google` trends` search` web` analytics` api` code` python` hack

Official Google Blog: A new flavor of Google Trends

 google` trends` search` query` api` csv` keyword` timeseries

Open Research – the Data: Lastfm-ArtistTags2007 – Duke Listens!

 last.fm` music` tagging` artists` tags` collaborative` filtering

i2b2: Informatics for Integrating Biology & the Bedside

 medical` obesity`

Tiger Data Set Lecture

 tiger` gis` lectures

Google To Launch Large Scale Geo-Services

 geo` google` gps` location` geolocation` cell` wifi` api` gis

Last.fm’s Playground

 celebrity` misspelling` spelling` names

ImportGenius.com : U.S. Customs Database and Competitive Intelligence Tools

 commercial` shipping` imports` exports` finance` datamining

Directory Listing of Betfair price files

 betting` prediction` betfair` price` csv` predictionmarket

Reuters Spotlight – Article and Media API

 news` text` articles` api` content` media` xml` images` publicdata

DataSets – Scikits – Trac

 scipy` python` machinelearning` statistics` resource

[Wikitech-l] page counters

 wikipedia` pageviews` trends` textmining` seo` topic

Wikipedia article traffic statistics

 via:chl` wikipedia` web` analytics` seo` topic` textmining` traffic

Yahoo! Internet Location Platform – YDN

 yahoo` geo` geocoding` location` landmarks` gis

How to find images on the internet « Random knowledge

 images` links` lists` archive`

Yahoo offers geographic data to Web sites | Tech news blog – CNET News.com

 gis` webservice` yahoo` api` location` landmark

Instructions for Obtaining Search Engine Transaction Logs

 query` search` log` excite` altavista` alltheweb` transaction

TechTC – Technion Repository of Text Categorization Datasets

 datamining` textmining` categorization` classification` odp` directory` text

The TechTC-100 Test Collection for Text Categorization

 textmining` classification` category` odp` directory

FEC Election Contributions: Download Detailed Files by Election Cycle

 individual` donations` government` election` publicdata` fec

Juiced Google Analytics Python API: Juice Analytics

 search` statistics` keywords` analytics` api` python` web` seo` google

Country Name and ISO 3166 Code MySQL Import File

 mysql` states` countries` isocode

geocoded Hotels « GeoNames Blog

 hotels` geonames`

GeoNames webservice and data download

 locations` cities` countries` gis

Index of /download/worldcities

 cities` gis

ualberta dependency based thesaurus and word count data

 corpus` text` similarity` terms

CommonCrawl – About

 web` crawler` bot`

Data sets and corpus / corpora for biological literature and text mining  

 bioinformatics` text` corpora` domainspecific` genomics` corpus`

Office of Defects Investigation (ODI), Flat File Downloads

 defect` recall` automobile` fightclub` nhtsa` saefty

p2psim – kingdata : DNS server latency network distance matrices

 distance` matrix` network` p2p` dns` latency` nmf` queryminer

Sep Kamvar / Personalization /

 pagerank` web` matrix` matlab

beta.opentick.com

 opentick` trading` beta` feeds` finance

WikiXMLDB: Querying Wikipedia with XQuery

 wikipedia` xml` ec2

kiwitobes.com » Blog Archive » Walmart Growth Video

 walmart` visualization` video` freebase` store` retail` locations` opening

Open Cell Id dataset – phone geolocation from GSM cellids

 gis` mobile` geolocation

The Cornell Web Lab – The Cornell Web Lab

 cornell` web` archive` hadoop` crawl

im2gps: estimating geographic information from a single image

 imagerecognition` via:csantos` gis` cmu` gps` imageprocessing` paper` hack` freaking_awesome

Datasets: MUSCLE WP2 Evaluation, Integration and Standards

 image` video` audio` currency` sports` imagerecognition

Open Economics – Store – Index

 economics` list

welcome @ omdb

 free` movie` database` netflixprize

Cogblog » Blog Archive » Cogmap APIs

 api` cogmap` person` name` organization` record_linkage

Wal-Mart : Freebase – The World’s Database

 retail` locations` stores

Cogmap: The Org Chart Wiki

 record_linkage` identity` name` organization` orgchart` marketing

German English Parallel Corpus “de-news”, Daily News 1996-2000

 german` translation` corpus` english` text` via:maxme

Welcome to the CRCNS data sharing activity website — CRCNS

 neuroscience` patch` clamp` recordings` neuron` timeseries` patchclamp` data` neural

Infochimps.org: Free Redistributable Rich Data Sets

 aggregator` links

Frequent Itemset Mining Dataset Repository

 retail` clickstream` traffic` web` links` sales

Dolores Labs Blog » Blog Archive » Our color names data set is online

 colormap` color` mechanicalturk

TeradataUniversityNetwork.com -> Registration

 teradata` retail` transactional` database

Pascal Learning Challenge Large Datasets

 large` competition` challenge` svm` machinelearning` scalability

ECIS 2007 – The 15th European Conference on Information Systems

 retail` dillards` sams_club

Alexa Web Search

 alexa` aws` web` search` api`

State and Federal Case Law

 creativecommons` court` legal` law` via:inkdroid

Access to Web Research Collections VLC2/WT10g/WT2g

 blog` web` text

Lyricsfly Lyrics API, database access to search for music artist and song title

 song` lyrics` database` api`

99 Wikipedia Sources Aiding the Semantic Web » AI3:::Adaptive Information

 links` directory` record_linkage` extraction` wikipeida` named_entity` recognition` textmining` semanticweb

AudioScrobbler Data

 audioscrobbler` recommendation` collaborative` filtering` music

The Linking Open Data dataset cloud

 directory` rdf` semantic` data` soup` graph

Free Economic Data | Economic, Financial, and Demographic Data

 finance` economics` portal` links

::MLSP 2008::: MLSP competition

 machinelearning` trading` competition` backtest` matlab` code` finance` via:DeliciousRob

Computer Vision Test Images

 computer` vision` image` ray` trace` fingerprint` stereo` detection` via:chl

The Dataverse Network Project | The Dataverse Network Project

 statistics` repository` harvard

DVN – Home

 harvard` repository` social` science` research` portal` links

Temperature data (HadCRUT3 and CRUTEM3)

 climate` temperature` netcdf

MNIST handwritten digit database, Yann LeCun and Corinna Cortes

 handwriting` mnist` image` recognition

LFW : Labelled Faces in the Wild

 facerecognition` face` recognition` umass` image

Making random contacts – (37signals)

 generator` names

Test (Sample) Data Generators

 generator` tools` list` via:jd

Compete – Compete Developer Resources

 compete` api` web` statistics` traffic` analytics` mashup

Machine Learning (Theory) » The Peekaboom Dataset

 peekaboom` vision` image` large` human` computation` machinelearning` recognition

Ocean Processes and Modeling: Ocean Data

 links` oceanography` satellite

BlogoCenter data sets

 blog` ucla

Tagged datasets for named entity recognition tasks

 nlp` corpus` tagged` named_entity` recognition` list

del.icio.us stats – deli.ckoma

 del.icio.us`

The Financial Data Finder A – G

 finance` links

Freebase Wikipedia Extraction (WEX)

 wikipedia` xml` structured` corpus

The arXiv.org API

 arxiv` api` open` paper` academic`

England Football Results Betting Odds | Premiership Results & Betting Odds

 gambling` soccer` football` excel` statistics

HughesData – Main – Hughes Lab

 rna` bioinformatics` microarray` expression` gene` machinelearning

Stanford MicroArray Database

 bioinformatics` microarray` expression` gene` machinelearning` stanford

ArrayExpress Home

 bioinformatics` microarray` expression` gene` machinelearning

Gene Expression Omnibus (GEO) Main page

 bioinformatics` microarray` expression` gene` machinelearning

Index of /courts.gov

 corpus` text` legal` law` court` ruling` opensource` publicdata

Welcome to Openvest

 python` finance` edgar` pylons` matplotlib` sec` webservice` via:jolby

Statistical Science Web: Data Sets

 links` statistics

Text Mining, Visualization and Social Media

 crawler` blog` corpus

Aleix Face Database

 facerecognition` machinelearning` face` image

Data Repository Evaluation

 umd` links` statistics` government` sports` via:rickladd

PMC FTP Service

 biology` medicine` articles` text` journal` authors

“uspop2002” data set

 music` similarity` machinelearning

Internet Archive: Details: Amazon ASIN listing and similarity graph

 ASIN` amazon` recommendation` collaborative` filtering` via:keyvowel

European Climate Assessment Daily Weather Data

 weather` europe` ascii` netcdf

StatLib—Datasets Archive

 machinelearning` datamining` cmu` link` collection

National Household Travel Survey (NHTS) Data

 driving` transportation` publicdata

Nielsen BookScan USA

 books` sales` commercial

Home – Numbrary

 finance` data`

About – Numbrary

 searchengine` search` tagging` aggregator` numeric` extraction` tables` collaboration` web2.0

Main Page – OpenTextMining

 textmining` open` nature` standards` search

Metafilter Infodump

 metafilter` comments` network` via:chl

WEBSPAM-UK2007 | Datasets | Web Spam Detection

 web` search` spam` crawler` yahoo

Trust network datasets – TrustLet

 socialnetwork` trustnetwork` trust

TaskForces/CommunityProjects/LinkingOpenData/DataSets – ESW Wiki

 opendata` semantic` rdf` collaboration

Some Datasets Available on the Web » Data Wrangling Blog

 publicdata` links

XML.com: GovTrack.us, Public Data, and the Semantic Web

 semanticweb` rdf` congress` politics` government

CiteULike: Available datasets

 networks` research` graph` tags` paper` record_linkage

Archive-It.org

 archive` internet` web` index`

Challenge: Synopsis – Causality Workbench

 competition` machinelearning` forecasting` contest

Natural Language Processing

 microsoft` text` paraphrase` corpus

LDC – Linguistic Data Consortium – Obtaining Data Resorces

 nlp` text` corpus` ngram` google` commercial` license

1990 Census Name Files

 census` names` identity` frequency` record_linkage

Given Name Frequency Project: Analysis of Given Name Popularity

 name` record_linkage` text` identity` code

Email Datasets

 enron` names` identity` text` record_linkage

ZoomInfo – Welcome to the ZoomInfo Developer API

 api` identity` people` webservice` record_linkage

Name Discrimination Data Named Entity Resolution /  Entity Disambiguation

 record_linkage` corpus` nlp` names

Developers Area – eBay Market Data Documentation – eBay Market Data Documentation

 ebay` api` retail` price` code

New SwetoDblp RDF dataset released with 11M triples

 name` authorship` rdf` record_linkage

LSDIS : SwetoDblp

 bibliography` rdf` ontology` duplicate` name` record_linkage

StrikeIron Super Data Pack Web Service 1.0 – StrikeIron Marketplace

 webservice` publicdata` datacleaning

Duplicate Detection, Record Linkage, and Identity Uncertainty: Datasets

 duplicate` detection` record_linkage` datacleaning` text

INFO 747 – Social and Economic Data

 datacleaning` record_linkage` video` lectures` course` cornell` economics` finance` publicdata

Overstock.com Affiliate Program

 retail` overstock` sales` api` product` price` forecasting

Amazon Web Services Developer Connection : Can Alexa WS provide detailed …

 finance` alexa` amazon` tech

Market Data — eBay Developers Program

 ebay` retail` pricing` sales` api` product

Machine Learning and Data Mining – Datasets

 face` image

GIS for Schools

 epidemiology` gis` health

Google Trends API coming soon | Tech news blog – CNET News.com

 google` trends` api`

MIT Media Lab: Reality Mining

 social` activity` location` cell` gis

RL Competition 2008 – Home

 machinelearning` reinforcement` agent` competition`

Vehicle Routing Data Sets

 optimization` vehicle` routing

EIA – Petroleum Data, Reports, Analysis, Surveys

 oil` energy` statistics` economics` petroleum

DMOZ100k06 – Michael G. Noll

 search` pagerank` text` tags` content

Grading

 machinelearning` CMU` course` projects` graphicalmodel` code` paper

Financial Forecast Center’s Historical Economic and Market Data

 exchangerate` dollar` economics`

Browse Business Cycle Indicators Data

 economics` indicators` time` series

The Numbers Guy : Aspiring to Be the Wikipedia of Numbers

 finance` numberpedia` mechanicalturk` textmining` statistics

Social characteristics of the Marvel Universe

 socialnetwork` graphs` comicbooks

SourceForge.net: Word Lists Collection

 dictionary` words

See Who’s Editing Wikipedia – Diebold, the CIA, a Campaign

 wikipedia` authorship`

Dataset Generator – Perfect data for an imperfect world.

 tools` generator

Entree Chicago Recommendation Data

 recommender` collaborative` restaurant

community resource guide: i’ve been here before – show me the links

 demographics` maps` gis` statistics` links

Social Science Data on the Net

 economics` social` government` health` labor` links

List of films: A – Wikipedia, the free encyclopedia

 netflix` netflixprize` movie` index` wikipedia`

The arXiv on your harddrive

 paper` corpus` arXiv

Insanely Useful Websites | Sunlight Foundation

 links` transparency` government` politics` congress` reference

Technophilia: Where to find public records online – Lifehacker

 public` records` links

Junk email project

 corpus` email` spam` textmining

Enron Email Dataset

 enron` corpus` email` text` social` network

ftp://ftp.bls.gov/pub/special.requests/cpi/cpiai.txt

 finance` cpi` inflation` data

GOS – Geospatial One Stop

 health` gis` epidemiology` links

CIA Factbook Grep in Python

 cia` population` python` code` grep

Miller Center of Public Affairs – Richard Nixon – Oval Office Recordings

 nixon` speech` tapes` audio` mp3` wav` flac

Deborah Jeane Palfrey Legal Defense Fund

 phone` politics

UC San Diego Data Mining Competition – 2007 – Datasets

 housing` refinance` mortgage`

Retail Industry Financial Ratios & Benchmarks

 retail` finance` sales` sqft`

Retail Industry Financial Ratios & Benchmarks

 retail` finance` sales` sqft

stores | POI Factory

 retail` location` poi

GpsPasSion Forums – ** INDEX OF POI COLLECTIONS **

 retail` poi` location` gis` gps

GPS POI US : Home > Retail Stores

 retail` location` gis

Collective Dynamics Group

 smallworld` networking` socialnetwork` graph

Jester Data download page

 collaborative` filtering` jokes

TricTrac: Video Dataset

 video`

Premium Business Information Databases – AlacraWiki

 links` finance` commercial

Index of /edgar

 finance` xml` edgar` sec` code` perl

Mail Index

 EDGAR` sec` mail` text

metafy / Anthracite Idioms

 finance` SEC` scrape` parse` commercial

Retail and Food Services – Time Series Data/Seasonal Factors

 retail` sales` census

TDT

 categorization` textmining` detection` tools

Volume of retail sales: Social Trends 33

 retail` sales` uk

generatedata.com

 tools` generator` random

Melissa DATA – Lookups

 consumer` data` database` api

FactSet: Data Maven – Kiplinger.com

 factset` finance`

IBES (Demo)

 finance` ibes` analyst` forecast` wharton

Thomson Financial I/B/E/S Data

 finance`

Historical Quotes – Yahoo! Finance

 yahoo` finance` stock` price`

Network data

 network` links

Bureau of Labor Statistics Home Page

 statistics` labor` government` consumer

NAR: Research: EHS Data

 housing` sales` finance

RFA – The Industry – Industry Statistics

 ethanol`

Chain Store Guide – Retail Locations

 retail` finance` store` locations` gis

Press Releases – Directions Magazine

 retail` gis` store` locations

Energy Information Administration – EIA – Official Energy Statistics from the U.S. Government

 finance` government` energy` historical` forecasts` fuel` oil

Databases you can use for benchmarking

 links

UPC Database: Downloads

 product` upc` database`

Web Crawling / Crawl Datasets at Tobias Escher at the OII

 crawler` benchmark` search` web` links

TechTC – Technion Repository of Text Categorization Datasets

 corpus` text

TMC data archive download site

 traffic` data`

http://www.volvis.org/

 volume rendering

Computational Vision: Archive

 vision` caltech` image recognition

DC Pedestrian Classification Benchmark

 pedestrian` image` classification` detection

opentick :: home

 finance` economics` feed` free` stock` trading` opentick` opensource

Web as Corpus

 textmining` corpus` concordance` wordlist` n-gram

.:[ packet storm ]:. – http://packetstormsecurity.org/

 dictionary` hack` security` wordlist` password

Enron Dataset

 data` mysql` email` energy` text` social network

Splog Blog Dataset

 blog` corpus` spam

Home Page for 20 Newsgroups Data Set

 corpus` text` newsgroup

White Glove Tracking

 crowd sourcing` image` processing` algorithm` collaborative` distributed` web2.0` code` opensource

NOAA Paleoclimatology Program – Coral and Sclerosponge Data

 paleo climatology` climate` oceanography` coral` sponge` biology

NAICS — North American Industry Classification System

 finance` economics` naics` industry` classifications

Saving Democracy With Web 2.0 –

 democracy` web2.0` mashup` government` funding` article

Congresspedia – Congresspedia

 collaborative` wiki` government` congress` politics` elections` web2.0` directory

Population Estimates Data Sets

 census` data` population` statistics

CRAN Task View: Machine Learning & Statistical Learning

 statistical learning` machine learning` code` R` libraries` cran`

Data for Data Mining

 linkd` datamining` timeseries` text` extraction` socialnetwork

PAIDA – Pure Python scientific analysis package

 python` visualization` library

SUBDUE – Graph Based Knowledge Discovery

 machine learning` network` graph`

AOL search data mirrors

 aol` search`

Python Cheese Shop : shakespeare 0.4

 python` text`

AG’s corpus of news articles

 corpus` nlp` machine learning` textmining

Sampling Techniques for Massive Data – Google Video

 video` machine learning` statistics` matrix` sampling` large` sparse` algorithm` experiment_design

metachronistic » Mirror the Wikipedia

 wikipedia` laptop` install` dump

LETOR: Benchmark Datasets for Learning to Rank

 ranking` search

CN710: Comparative Analysis of Learning Systems (Spring 2006) – Class Project

 machinelearning` algorithm` ogi` bu` greyhound` finance

UrbanSim Home

 python` urban` software` simulation` opensource` GIS` census`

System One – Wikipedia³

 wikipedia` rdf`

System One – Labs

 wikipedia` rdf` tools

Face Recognition Homepage – Databases

 face` algorithm` facere cognition` data` image

CBCL SOFTWARE Face data set

 face` seung` algorithm` recognition` image

Text Analytics Solutions from ClearForest

 extraction` finance` semantic` semanticweb` text

23C3 – Mining Search Queries – Google Video

 aol` search` video` talk` algorithm` information retrieval` datamining` machinelearning

Digital History Hacks: Keywords and Clues

 aol` search` query` analysis

Digital History Hacks: Searching for History

 aol` search` query` analysis

The Tom Kyte Blog: An interesting data set…

 aol` search` oracle` database` code

KDD 2005 – KDD Cup 2005: Aug 21-24, Chicago, IL. USA

 query` categorization` algorithm` google

Statistical NLP / corpus-based computational linguistics resources

 corpus` machine learning` text

Ph.d.-student Rasmus Elsborg Madsen

 text` machine learning` context` matlab

Intelligent Web Search and Mining: Tools & Resources

 machine learning` code` links

PageRank Datasets and Code

 pagerank` code` algorithm

Official Google Research Blog: All Our N-gram are Belong to You

 linguistics` google` ngram` nlp` record_linkage

Hyper-threaded Java – Java World

 clustering` algorithm` java` parallel

Statistical Modeling, Causal Inference, and Social Science

 blog` econometrics` finance` machine learning` math` statistics

Structural Analysis of Discrete Data and Econometric Applications,

 books` econometrics` economics` finance` ebook

Kris Brower » Archives » Google Onpage Search Results Analysis

 google` ranking` aol` search` analytics

CSE 250B Fall 2006

 netflixprize` machine learning` course`

Matrix Market

 matrixmarket` matrix`

Estimation of mean values, covariance matrices and imputation of missing values

 imputation` matlab` missing` EM` machinelearning

Face Detection

 face` image

CSE 250B Project 4, Fall 2006

 subset` netflix prize` dimensionality` reduction

G3DATA

 extract` from` graphs` hack` google` trends

cwm – a general purpose data processor for the semantic web

 python` processor` semantic` web` rdf

WebBase Project

 link` analysis` structure` web` crawler` stanford

sam roweis : data

 machine` learning` matlab` python` hackers` image

Flight Data and Weather Data

 flight data` airplane data` weather data` airline route data` aircraft flight data` in-flight analysis` airline on-time data

Survey of Patients Hospital Experiences

 healthcare analytics` subjective outcomes` healthcare customer service` quality of care` patient care` patient surveys

Largest collection of longitudinal hospital care data in the US

 healthcare analytics` healthcare big data` research datasets` national in-patient statistics` local healthcare statistics` in-patient statistics` hospital cost data` hospital use data

Ambulatory healthcare data

 physician visits data` doctor visit data` outpatient care data` private practice physician data` non-federal healthcare data` physician office data

Index of /data/sequence/mnist

 mnist` xml` format

MNIST handwritten digit database

 mnist`

Book-Crossing Dataset

 data` set` collaborative` filtering` datamining` books` movie

allmovie

 movie` netflix prize` source`netflix

Submissions Guidelines for the Collectorz.com Online Movie Database

 movie` source

cinema.com

 plot` synopsis` movie` netflix prize` prize

LUMIERE

 netflixprize` prize` european` movie` revenue`

Data dumps – Meta

 mediawiki` wikipedia` import` mysql` sql

“phone ***” ” address *” “e-mail” intitle:”curriculum vitae” – Google Search

 resume` google

generatedata.com

 random` generator` database` sql

Lending Club Loan Data

Finance`Loans`business`investing

SMS Spam Collection

spam`email`text analysis

Data Sets | Pew Research Center’s Internet & American Life Project

demography

Flickr personal taxonomies

flickr`taxonomy`images

Yahoo Data for Researchers

yahoo

DBLP Computer Science Bibliography

bibliographies`text mining

ICWSM Spinnr Challenge 2011 dataset

weblog`blog`social media`network analysis

Quantum Chaotic Thoughts: Facebook100 Data Set

facebook`network analysis`social

Public Data Sets on Amazon Web Services (AWS)

Amazon Web Services` amazon` ebs` ec2` s3` publicdata` hadoop

The ClueWeb09 Dataset

human language`text mining

Data | The World Bank

government`finance`economy

ImageNet

images

What is Twitter, a Social Network or a News Media? – WWW’10

twitter`text mining`social

dotbot | DotNetDotCom.org

spider`web analytics

arXiv.org help – arXiv Bulk Data Access – Amazon S3

Amazon Web Services` amazon` ebs` ec2` s3` publicdata` hadoop

YouTube Dataset

youtube`image analysis`video analysis

Face Recognition Homepage – Databases

face recognition`facial recognition`image analysis

UCI Network Data Repository

data repositories

Datasets for “The Elements of Statistical Learning”

learning

MovieLens Data Sets | GroupLens Research

movies`video analysis`business

Translation Task – EMNLP 2011 Sixth Workshop on Statistical Machine Translation

translation`human language

Project Gutenberg

books`text mining

About WordNet – WordNet – About WordNet

wordnet`corpus

Aligned Hansards of the 36th Parliament of Canada

canada`parlaiment`government`text mining

CRCNS – Collaborative Research in Computational Neuroscience – Data sharing

Bioinformatics` fmri` neuroscience` python` neuralnetwork

USENET corpus

usenet`text mining

UniGene

bioinformatics

ChEMBLdb

chemoinformatics

UCI Machine Learning Repository

algorithms

Gene Expression Omnibus (GEO) Main page

genetics`bioinformatics

Social Science Data

social science

IMDB dataset

business

Stanford Large Network Dataset Collection

network analysis

Google Books n-gram dataset

books`text mining

Million Song Dataset | scaling MIR research

audio analysis

Belly Button Biodiversity 2.0

health informatics`bioinformatics

Datasets – Modeling Online Auctions

auctions

2gb of photos of cats

image analysis`pets`cats

Click Dataset | Center for Complex Networks and Systems Research

web analytics

The Electric Rice Cooker — One year of deleted weibos archive

text mining

Registered meteorites that has impacted on Earth visualized – AnalyticBridge

meteorites`atmosphere

GeoJSON files for real-time Virginia transportation data.

road`traffic`accidents`transportation

NYPD Crash Data Band-Aid

road`traffic`accidents`transportationccidents`transportation

State Department of Education datasets

student performance`school demographics`standardized test performance`school quality`education

Open Source Virtual Data Lake

Before data virtualization was well-known I was presented with the scenario of a large company that had purchased several other companies and needed to fuse their data. I had already presumed that I would be introducing the Hadoop ecosystem into the portfolio because that was some large part of why they had hired me. Rather than take the approach that has become common now and build a data lake I proposed we use data virtualization along with a custom lightweight layer that performed the major functions of a Multidimensional Online Analytical Processing (MOLAP) engine to allow us to provide immediate business benefit by using their data in-place while providing an abstraction that allowed us to evolve the systems that held the data. Some of the key requirements of this platform was that it must be easily changed, provide query abilities as good as or better than the underlying data stores, and scale linearly.   I recommended an architecture that is still unique within our global domain, the concept of a virtual data lake.

Virtual Online Analytical Processing (VOLAP) design

The product we built in 2014 and continue to evolve is unique in the marketplace with the exception of perhaps Denodo. The key difference is that it uses entirely free and open source technologies that can be swapped for others as needed. Currently we have a small java app, Jboss Teiid (swappable for Apache Calcite), and the many data technologies the company uses–(many built based on my reference architecture template) HBase, MySQL, Oracle, Impala, Hive, MongoDB, Cassandra, SalesForce, and external REST APIs.

Teiid architecture

We also designed our our approach to operate as a virtual OLAP (VOLAP) system–an approach that doesn’t currently exist–that adds cost-based routing to make the query run as fast as possible by selecting the fastest data source automatically and executing all possible operations at the data source.  The large tech company now leverages this platform as their enterprise data layer–a virtual data lake that performs as a virtualized hybrid OLAP “database”–VOLAP.

The basic premise
REST APIs are arguably the most universal of data interchange methods and for this reason we elected a REST api as our interface for any application that needed data. We support jdbc and odbc as well but discourage their use because we can provide better service through REST (a swagger UI for exploration, traffic smoothing of queries, circuit breakers, security injection, complex in-flight transformations, etc).

I hold that a business expert (mathematical analyst, account manager, HR manager) need not to know the location, context, or formulas to have basic utility of data. They need simple access to data and they add the expertise over that data to provide business value. To that end data virtualization provides a universal adapter and the java application adds a universal language and the business knowledge to the data at query time automatically in a way that is easily governed.

How it works — the request sequence
With more in-depth explanation below, here is how a request for data proceeds in its simplest form:
java app-
REST API->request parser->logical field replacement->data source selection->query writer->
data virtualization-
Query splits into multiples if going to multiple data stores->data store queried->rows buffered and if necessary, aggregated->data streamed back

Injection points exist in our java application that you may consider for such options as plugging in other ways of data sources selection other than cost-based (like semantic, quality of service, or other algorithmic approach), circuit breakers (fail if sql injection attempted, some business rule not passed), and traffic smoothing algorithms such as token bucket for DDOS protection.

Applied example
Assume a web application or spreadsheet wants to determine which customer had the most sales this month in Canada. The application sends a simplified form of the request to a URL that looks very much like a web page using a simplified SQL language:
http://myserver?query=SELECT customer, sales where sale_date between 01/01/2019 and 02/03/2019

  1. receive the request
    This request goes to the java application
  2. extract the logical fields we’ll need to return–customer, sales, and sale_date. These are LOGICAL fields and may or may not correspond to the name of the columns in any of the underlying data storage.
  3.  get the formulas for each field. Assume in your underlying data sources you have identified and mapped a series of sources that contain these fields. In this case assume in the data virtualization layer we have set that customer is an attribute column that is just the column “customer_name”, sales is a metric column that corresponds to the formula “sum(amount_spent – tax_collected)” and sale_date is called “date_of_transaction”.
  4. use the metadata from the data virtualization layer to identify all data sources with all of these columns–customer_name, amount_spent, tax_collected, and sale_date. There are four because it requires for columns to get the required data. Use the cost-based (or other plug-in) rule to identify which will be quickest. Lowest cardinality of the underlying fact table is usually a good first guess at the lowest cost (fastest) data source.
  5. write a query to be sent to the virtualization layer that will retrieve these data: select customer_name as customer, sum(amount_spent – tax_collected) as sales from myvirtualtable where date_of_transaction between ’01/01/2019′ and ’02/03/2019′
  6. send query to virtualization layer, which splits the query into several queries if there are multiple data sources involved (data federation).
  7. underlying data store is queried and rows sent back to data virtualization layer. If the underlying data provider can’t perform some part of the query natively (called a pushdown operation), such as summing values, the values are returned to the virtualization layer and performed there.
  8. data are streamed back from the virtualization layer to the java application where it is sent back to the caller

Next, architecture and code.

Planning and Communicating Your Cluster Design

When creating a new Amazon Web Services (AWS) hadoop cluster it is overwhelming for most people to put together a configuration plan or topology.  Below is a Hadoop reference architecture template I’ve built that can be filled in that addresses the key aspects of planning, building, configuring, and communicating your hadoop cluster on AWS.

I’ve done this many times and as part of my focus on tools and templates thought I’d add a template you can use as a basic guideline for planning your Cloudera big data cluster.  The template includes configurations for:

  • instance basics
  • instance list
  • storage
  • operating system
  • CDH version
  • the cluster topology
  • metastore detail for hive, YARN, hue, impala, sqoop, oozie, and Cloudera Manager
  • high-availability
  • resource management
  • and additional detail for custom service descriptors (CSD) for Storm and Redis

No Warranty Expressed or Implied

It’s not meant to be exhaustive as there are many items not covered (AWS security groups, network optimization, dockerization, continuous integration, monitors, etc.) but it is an example of a real-world cluster in AWS (details of instance and AZ changed for security).

Screenshot of the roles and services in the big data design template

Example list of EC2 instances for the cluster plan

Cloudera hadoop reference architecture  configuration template for Amazon Web Services (AWS)

AWS_topology_template

Please feel free to let me know how it works for you and if you have any improvements for it.

NOTE: This content is for archive purposes only.  With generation 4+ EBS volumes big data IO performance no longer requires volume prewarming.

Fresh Elastic Block Storage volumes have first-write overhead

At my employer I architect Big Data hybrid cloud platforms for global audience that have to be FAST.  In our cluster provisioning I find we frequently overlook doing an initial write across our volumes to reduce write time during production compute workloads (called pre-warming the EBS volumes).  Per Amazon (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-prewarm.html) failure to pre-warm EBS volumes incurs a 5-50% loss in effective IOPS.  Worst case that means you could DOUBLE the IO portion of your HDFS writes until each sector has been touched by the kernel.  Amazon asserts that this performance loss, amortized over the life of a disk, is inconsequential to most applications.  For one of our current clusters we have a portion with 8-1TB drives in each of 10 compute nodes as a baseline.  Our estimated pre-warm time is 30 hours on each mount point so if done sequentially that’s 2,400 hours to touch each drive block.

What does this imply?  Without pre-warming we would have added as much as 2,400 additional hours of write latency during initial HDFS writes and that latency could appear in many different places in the stack (HDFS direct writes, Hive postgresql/mysql metadata writes and management, log writes, etc.)

Steps to optimize your EBS writes

Read the AWS document above carefully as it will ERASE EVERYTHING ON THE DISK if you use the first method in their article.  The steps below will execute this safely on disks with existing content.

To pre-warm the drives on your cluster:

    1. stop the cluster services
    1. ssh into each server
    1. execute lsblk and note the mount points (they likely start from /dev/xvdf and go down from there increasing the letter at the end, such as /dev/xvdg, /dev/xvdh, etc.)
    1. unmount each one at a time with sudo umount /ONEMOUNTPOINT
    1. Continue until all mount points are unmounted, meaning there’s nothing shown after the ‘disk’ column as below:
    1. CAUTION:  DO NOT DO THE FOLLOWING ON A MOUNTED DISK AND MAKE SURE YOU USE THE SAME MOUNT FOR BOTH if AND of
    1. execute the following, changing the if= and of= to the same mount pointsudo dd if=/YOURMOUNTPOINT of=/YOURSAMEMOUNTPOINT conv=notrunc bs=1MExample:  sudo dd if=/dev/xvdf of=/dev/xvdf conv=notrunc bs=1M
    1. Wait.  It’ll be a few minutes for a 32GB drive as shown in the Amazon write-up above or 1 day+ for a 1TB drive.
  1. After ALL the processes on the server complete, reboot the server

If you’d like to check the process or if your ssh session has expired and you want to ensure you’re still warming execute ps aux|grep YOURMOUNTPOINT , example:  ps aux |grep /dev/xvdf

A far better approach, of course, would be to automate this as part of your cluster deployment process using Chef or equivalent infrastructure automation tool.

Ref: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html

Scenario 1:  There was data, the logs say Namenode not formatted, the dfs.data.dir (check your config to see where it is) is empty

Cause:  The data was emptied out of your namenode directory.

Things to try (in order):

    1. FSCK (see scenario 2 below)
    2. recover the namenode
      1. hadoop namenode start -recover
      2. If the output says some directories are missing, create them, chgrp to hadoop, chown to hdfs, chmod 755, then run again
    3. Import the fsimage from a non-corrupt secondary namenode
      1. hadoop namenode -importCheckpoint
      2. If the output says some directories are missing, create them, chgrp to hadoop, chown to hdfs, chmod 755, then run again
    4. Brute force it
      1. Find out in the config where the snn checkpoint is kept (fs.checkpoint.dir)
      2. SCP down ALL the files in the fs.checkpoint.dir to your local machine
      3. SCP up ALL the files you just downloaded to the dfs.name.dir
      4. For all those files chgrp to hadoop, chown to hdfs, chmod 755
      5. Start your HDFS service as usual through the cluster manager and think optimistic thoughts.

Scenario 2:  There was data, the logs point to corrupt blocks

Cause:  Probably a bad termination signal during copy or high volume data movement with bad network

Things to try (in order):

    1. FSCK
      You can use

        hadoop fsck /
      

      to determine which files are having problems. Look through the output for missing or corrupt blocks (ignore under-replicated blocks for now). This command is really verbose especially on a large HDFS filesystem so I normally get down to the meaningful output with

        hadoop fsck / | egrep -v '^\.+$' | grep -v eplica
      

      which ignores lines with nothing but dots and lines talking about replication.

      Once you find a file that is corrupt

        hadoop fsck /path/to/corrupt/file -locations -blocks -files
      

      Use that output to determine where blocks might live. If the file is larger than your block size it might have multiple blocks.

      You can use the reported block numbers to go around to the datanodes and the namenode logs searching for the machine or machines on which the blocks lived. Try looking for filesystem errors on those machines. Missing mount points, datanode not running, file system reformatted/reprovisioned. If you can find a problem in that way and bring the block back online that file will be healthy again.

      Lather rinse and repeat until all files are healthy or you exhaust all alternatives looking for the blocks.

      Once you determine what happened and you cannot recover any more blocks, just use the

        hadoop fs -rm /path/to/file/with/permanently/missing/blocks
      

      command to get your HDFS filesystem back to healthy so you can start tracking new errors as they occur.

Scenario 3: Secondary Namenode can’t checkpoint the namenode

the SNN logs show checkpoint failed, probably with missing txid=####

  1. Change /etc/fstab and set the mount point to allow fsck on boot
    1. vi /etc/fstab as root
    2. Change the last zero in the first line to one, so change:
      1. LABEL=cloudimg-rootfs / ext4 defaults 0 0

        to

      2. LABEL=cloudimg-rootfs / ext4 defaults 0 1
    3. Save the file and exit
  2. Change the FSCKFIX in /etc/default/rcS to yes
    1. vi /etc/default/rcS as root
    2. Find the line that says #FSCKFIX=no
    3. Change it to FSCKFIX=yes (make sure you remove the commenting # at the beginning)
    4. Save and exit
  3. Check and record the last FSCK run
    1. execute and record the output of
      sudo tune2fs -l /dev/xvda1 | grep “Last checked”
  4. Reboot (use AWS instance reboot or do it from ssh)
  5. Check that FSCK ran on boot
    1. execute and verify that the date changed using
      sudo tune2fs -l /dev/xvda1 | grep “Last checked”
  6. Reverse the changes you made in steps 1 and 2
  7. Reboot

Jboss enterprise has a free data virtualization (NOT server virtualization) platform called Teiid.  Capabilities of this include service of data from multiple technologies (jdbc, odbc, Thrift, REST, SOAP, etc.), merging/transformation of data, fault tolerance, scalability, and other capabilities one would require of an enterprise service.  This can stand in the technology portfolio as part of an Enterprise Service Bus (ESB) to abstract big data and make it APPEAR to be relational (among other benefits)  To set up a Teiid server to expose Hive data:

Install the Jboss EAP

  1. Download jboss-eap-6.4.0.zip (or latest) from jboss downloads
  2. Unzip to c:\programfiles\jboss\ on windows , /etc/jboss on linux

 Overlay Teiid on top of EAP

  1. Download teiid-9.0.1-jboss-dist.zip (or latest)
  2. Unzip on top of jboss you just installed:  c:\programfiles\jboss\jboss-eap-6.4.0

 Add the Teiid web console to jboss

  1. Download teiid-console-dist-1.x.zip (or latest)
  2. Unzip on top of jboss you just installed:  c:\programfiles\jboss\jboss-eap-6.4.0

In \jboss-eap-6.3\standalone\configuration\standalone-teiid.xml add to the drivers section:

<driver name="hive" module="org.apache.hadoop.hive">
     <driver-class>org.apache.hive.jdbc.HiveDriver</driver-class>
</driver>

Find on your cluster the following files and add them to <jboss install dir>\modules\org\apache\hive\main   This path is VERY important and is mis-documented at present on the jboss site.

commons-logging-1.1.3.jar
hadoop-common-2.5.0-cdh5.3.0.jar
hadoop-core-2.5.0-mr1-cdh5.3.0
hive_metastore.jar
hive_service.jar
hive-common-0.13.1-cdh5.3.0.jar
hive-jdbc-0.13.1-cdh5.3.0.jar
hive-metastore-0.13.1-cdh5.3.0.jar
hive-serde-0.13.1-cdh5.3.0.jar
hive-service-0.13.1-cdh5.3.0.jar
httpclient-4.2.5.jar
httpcore-4.2.5.jar
HiveJDBC4.jar
libfb303-0.9.0.jar
libthrift-0.9.0.jar
log4j-1.2.14.jar
ql.jar
slf4j-api-1.5.11.jar
slf4j-log4j12-1.5.11.jar
TCLIServiceClient.jar

Navigate to the EAP bin directory and execute ./standalone.sh -c standalone-teiid.xml

Additional versions:  http://tools.jboss.org/downloads/overview.html

I’ve attached an excel file for a full-featured Big Data (hadoop) Production topology with a good starting place for an architecture that supports full Lambda architecture (streaming for seconds-old recency, batch for heavy lifting, and services to logically merge the two on demand).  The cluster is composed of 21 AWS instances with EBS backing.  The HDFS layer can be partitioned with the older data (those more than 1 year for example) are on cheaper S3 storage while still fully query-able.

The use cases covered in this architecture:

  1. Accessibility
    1. Data miner support through SQL and machine learning libraries into the raw data
    2. Ad-hoc querying through SQL in a dimensional model
    3. REST, thrift, and other API access with load balancing, data merging (from any data technology), and efficient data source routing
    4. OLAP cubes with perspectives (through data marts) for business analysis
  2. Technical
    1. Open source, free licensing model
    2. Fault tolerance and re-entrance on failure
    3. Scalable design with massive parallelism
    4. Cloud design for flexibility