The US National Institute of Standards and Technology (NIST) kicked off their Big Data Working Group on June 19th 2013. The sessions have now been broken down into subgroups for Definitions, Taxonomies, Reference Architecture, and Technology Roadmap. The charter for the working group:
NIST is leading the development of a Big Data Technology Roadmap. This roadmap will define and prioritize requirements for interoperability, reusability, and extendibility for big data analytic techniques and technology infrastructure in order to support secure and effective adoption of Big Data. To help develop the ideas in the Big Data Technology Roadmap, NIST is creating the Public Working Group for Big Data.
Scope: The focus of the NBD-WG is to form a community of interest from industry, academia, and government, with the goal of developing a consensus definitions, taxonomies, reference architectures, and technology roadmap which would enable breakthrough discoveries and innovation by advancing measurement science, standards, and technology in ways that enhance economic security and improve quality of life. Deliverables:
Develop Big Data Definitions
Develop Big Data Taxonomies
Develop Big Data Reference Architectures
Develop Big Data Technology Roadmap
Target Date: The goal for completion of INITIAL DRAFTs is Friday, September 27, 2013. Further milestones will be developed once the WG has initiated its regular meetings.
Participants: The NBD-WG is open to everyone. We hope to bring together stakeholder communities across industry, academic, and government sectors representing all of those with interests in Big Data techniques, technologies, and applications. The group needs your input to meet its goals so please join us for the kick-off meeting and contribute your ideas and insights.
Meetings: The NBD-WG will hold weekly meetings on Wednesdays from 1300 – 1500 EDT (unless announce otherwise) by teleconference. Please click here for the virtual meeting information.> Questions: General questions to the NBD-WG can be addressed to BigDataInfo@nist.gov
I’ve started a meetup for local professionals in the decision science field around the Tampa Bay area to come together and learn about what’s happening in our area. If you are a data science professional, come join us and be a part of making the Tampa-St. Petersburg metro area the southeast center of excellence in big data and analytics. Visit http://www.meetup.com/Analytics-Professionals-of-Tampa/ to find events and to join. I hope to see you there.
Artificial Intelligence for the Creation of Competitive Intelligence Tools
Introduction
Often in prioritizing business development activities it is helpful to determine who is able to influence a decision and how they are related to those in the market space. To make a defensible and actionable strategy it is useful to perform Influence Analysis and Network Analysis, which can form the kernel of a competitive intelligence analysis strategy. The data required for analysis must be obtained by identifying and extracting target attribute values in unstructured and often very large (multi-terabyte or petabyte) data stores. This necessitates a scalable infrastructure, distributed parallel computing capability, and fit-for-use natural language processing algorithms. Herein I will demonstrate a target logical architecture and methodology for accomplishing the task. Influence and Network analysis by machine learning algorithm (naïve bayes or perceptron for example) will be covered in a later supporting article.
Recognizing Significance
Named-Entity Recognition is required for unstructured content extraction in this scenario. This identification scheme may or may not employ stemming but will always require tokenizing, part-of-speech tagging, and the acquisition of a predefined model of attribute patterns to properly recognize and extract required metadata. A powerful platform with these built-in capabilities is the Apache openNLP project, which includes typed attribute models for the name finder, an extensible name finder algorithm, an API that exposes a Lucene index consumer, and a scalable, distributed architecture. The Apache Stanbol project in the incubator (http://stanbol.apache.org/) shows promise at semantic-based extraction and content enhancement but hasn’t been promoted outside the incubator yet.
Apache openNLP attribute recognition models are available in only a few languages with the original and largest being English. The community publishes models in English for the Name Finder interface for dates, location, money, organization, percentage, person, and time (date). Each is an appropriate candidate for term extraction for competitive intelligence analysis.
Logical Architecture
openNLP in four node Hadoop cluster
The controlling requirement for the task of metadata extraction from massive datasources is the processing of massive datasets to extract information. For this Hadoop provides a flexible, fault-tolerant framework and processing model that readily supports the natural language processing needs. The logical architecture for a small (<1TB) 4-node clustered Hadoop solution is as follows:
Process Flow
As below, the process to execute is standardized on the map/reduce patterns Distributed Task Execution, Union, Selection, and Intersection. Pre-processing using a Graph Processing pattern in a distinctly separate map phase would likely hasten any Influence Analysis to be performed post-process.
Multi-node Sequence Diagram for openNLP with Map Reduce on Hadoop
The primary namenode initiates work and passes the data and map/reduce execution program to the task trackers, who in turn distribute it among worker nodes. The worker nodes execute the map on HDFS-stored data, provide health and status to the task tracker, who reports it to the primary namenode. On node map completion the primary namenode may redistribute map work to the worker node or order the reduce task, each by way of the task tracker. The reduce task selects data from the HDFS interim resultset, aggregates, and streams to a result file. The result file is then used later for analysis by the machine learning algorithm of choice.
File Structures
The input file is of a machine-readable ASCII text type and is unstructured. Example:
From: Amir Soofi
Sent: Thursday, December 06, 2012 2:37 AM
To: Aaron Macarthur; Hugo Cruz
Cc: Donald Krapohl
Subject: RE: Language Comparison
Hugo,
FYI, Rick Marshall unofficially approved a 3-day trip for one person from the Enterprise team down to Jacksonville, FL to assist in the catalog reinstall.
I’ll be placing it in the travel portal soon for the official process, so that the option becomes officially available to us.
I think together we’ll be able to push through the environment differences better in person than over the phone.
Let us know whether your site can even accommodate a visitor, and when you’d like to exercise this option.
Respectfully,
Amir Soofi
Principal Software Engineer, Enterprise
The output of the openNLP Name Find algorithm map task on this input:
FYI, <namefind/person>Rick Marshall</namefind/person> unofficially approved a 3-day trip starting <namefind/date>14 November</namefind/date> for one person from the Enterprise team down to <namefind/location>Jacksonville, FL</namefind/location> to assist in the catalog reinstall.
I’ll be placing it in the travel portal soon for the official process, so that the option becomes officially available to us.
I think together we’ll be able to push through the environment differences better in person than over the phone.
Let us know whether your site can even accommodate a visitor, and when you’d like to exercise this option.
Respectfully,
<namefind/person>Amir Soofi</namefind/person>
Principal Software Engineer, Enterprise
The output of an example reduce task on this output:
{DocumentUniqueID, EntityKey, EntityType}
{234cba3231, Amir Soofi, Person}
{234cba3231, Thursday, December 06, 2012 2:37 AM, Date}
{234cba3231, Aaron Macarthur, Person}
{234cba3231, Hugo Cruz, Person}
{234cba3231, Donald Krapohl, Person}
{234cba3231, Rick Marshall, Person}
{234cba3231, 14 November, Date}
{234cba3231, Jacksonville/,FL, Location}
{234cba3231, Amir Soofi, Person}
A second reduce pass might yield combinations for network analysis (link strength below being calculated on instances of co-existence across unique documents):
{EntityKey, LinkedEntity, LinkStrength}
{Amir Soofi, Donald Krapohl, 6}
{Amir Soofi, Aaron Macarthur, 15}
{Amir Soofi, Jacksonville/, FL, 1}
The data may then be consumed into the analysis tool of choice, such as RapidMiner, WEKA, PowerPivot, or SQL Server/SQL Server Analysis Services for further analysis.
Conclusion
openNLP on Hadoop can provides good metadata extraction for key information in unstructured data. The information may be retrieved from competitor websites, SEC filings, Twitter activity, employee social network activity, or many other sources. The data pre-processing and preparation steps in metadata extraction for competitive intelligence applications can be low relative to that of other analytical problems (contract semantic analysis, social analysis trending, etc.). The steps outlined in this paper demonstrate a very high-level overview of a logical architecture and key execution activities required to gather metadata for Influence Analysis and Network Analysis for competitive advantage.
