How is Apache Hadoop used Big Data Analytics and Inteligence

Big Data Analytics - Usage of Apache Hadoop / What is the use of Apache Hadoop in an Analytical Project

There are various ways Apache Hadoop is used in a Big Data Project.  +Hortonworks latest report

Apache Hadoop Patterns of Use: Refine, Enrich and ... - Hortonworks categorizes it into

• Data Refinery Pattern
• Data Exploratory Pattern
• Application Enrichment Pattern

In Data Refinery - Hadoop is used for Cleansing up the data and sending the output (probably as aggregation / refinement) to the Enterprise Data Warehouse which might be +Oracle +SQLServer etc

In Data Exploration pattern - data is analysed in Hadoop environment, using Hive etc. and the results are shared with the Applications. There are many Big Data Visualization Tools that provide good reports. There are some Open Source tools like Pentaho Business Intelligence Studio that offers Big Data Visualization and Reports

In Application Enrichment Pattern the entire data is stored in Apache Hadoop. For example, the Web session is stored in Hadoop and appropriate actions are taken based on the user's web navigation .


These are some broad classification of the use of +Apache Hadoop

Free Tools for Data Mining

I got to know about RapidMiner and thought of testing it out (under progress). Tried K-Means clustering. The selection of raw data / input file was pretty easy, then I tried to link the process to K-means.  +RapidMiner threw an error that there were some missing values (also suggested to select a method/process that is available for missing value imputation.

I added the process (refer figure below) and then linked the process back to the +KMeans. So far so good. Wanted to know more from statisticians and analysts who have used  Open Source Software - like R,  and also SAS,   +Microsoft Dynamics ERP and SPSS

Segmentation of American Teens Internet Usage

American Teens - Segmentation

Smartphone adoption among American teens has increased substantially and mobile access to the interTeens and Technology 2013 findings. +Teens of America are +Hyperconnected
net is pervasive  by Pew Research Center's

This also shows the predictions for 2020 by +Pew Research Center and +Internet Provider.org

How to Measure Effectiveness of a Campaign using Marketing analytics (Banking)

Marketing Analytics - Measure Effectiveness of Email Campaign for Bank


In our Case Study: Customer Analytics for Telecom Operator to Cross-Sell and Up-Sell we had a look at the Telecom Analytics .. Now let's have a look at how Analytics has changed the communication between the bank and its customers

Banks have a good understanding of customer as the customers part with various data points (SSN #, Tax #, Address, Employee, Salary etc.) at different points of time (Enrolment, Address Change, Mobile Banking etc..)

This information is used for personalized marketing through email (This is the Channel for analysis - multiple outbound channels can also be fed) .

Processes in Marketing Analytics for Personalized Promotions for Banking

The processes involved are as follows

1. Gather Data
2. Analyze Data , Document Discovery and Create Model
3  Execute Campaign
4. Collect Responses
5. Measure the Results (Efficiency)
6. Fine tune / Optimize the model
7. Refresh Model and Segment

Since the necessary data for understanding the customer is already available / collected, the data is segmented to understand different customer segments that are available and their profiles

Each segmented is treated in unique way based on their behavioural patterns

The segments are divided into control and test groups. The customers in the test groups are subjected to the personalized marketing campaign while the control segment customers are part of regular campaigns

The most important step after sending those mailers is to collect responses. This is a key differentiator between a good and bad campaign management

The responses are subjected to Response Segmentation analysis and the results are measured. What are the parameters that are clearly visible in the responders? What are the feature of the non-responders .. this is an area where business needs to play some role rather than IT / Analysts

The Bank's Personalized Campaign has created a 500% increase in response compared to the generic mailer (Refer Figure)

The model is recreated with the understanding from the new findings - this is called Model Recalibration. The model is refreshed in campaign management system and the process continues



The Next Best Action (here the Mailer) is a continuous learning process

+Microsoft Dynamics ERP   was used as the CRM for the Bank. Third party software was used for Mail blast and custom code was used for model development \


There are also custom tools/solutions for campaign / personalized promotion. For example, Manthan Systems' TargetOne tool uses data from various touch points like point-of-sale (POS) and store vicinity, during ecommerce transactions, email click-throughs and social media engagements to ensure right conversation is initiated with the right customer at the right time through the right medium

See also:
Payment Card Industry - Crucial for Real-Time Next Best Action (NBA) - Big Data

How to Predict Employee Churn using analytics

How to Know if an Employee is going to Quit before he does

Internal Spies! Yes this was one of the credible source of information. The spies are equipped with some extraordinary talent to grasp information from various sources / relate and then report / conclude (usually she/he leaves this to the boss)

With Lean management .. the human source of information is trimmed and put into better use (really!)

There are various data points that need to be captured to Analyse Employee Churn (We had already spoken a lot about Customer Churn in Customer Attrition Modeling (Analytics))

• Current and Previous Appraisals (this might be a major source)
• Current Pay ?(vis-à-vis Peers and Market)
• Grievances / Issues Reported
• Deviation in Time spent in Office
• Frequency (change) of swipe pattern changes
• Frequency (change) in access to internal information (Intranet etc.)
• Job sites / Social Sites visit patterns (some offices have banned them)
• Feedback from Peers
• Feedback from Supervisors

This list will go on .. Social Media data ...

Employee Churn is costly for the company. It takes months / years to replace the employee with a new one by adequate training etc

Thanks to data analysis, there are tools that help the companies to understand the employees, predict churn and help the manager in retaining the employee (that is the ultimate goal)

SumTotal's Talent Management System is one such system that helps the HR 

Utlimate Software (
www.ultimatesoftware.com) has a tool called Retention Predictor
(TM) for predicting customer Churn

 

How to Get Sales Leads from Analyzing Web Data

Web Pattern Analysis for Identifying Prospects and Products

Web is an abundant source of information. The trails left behind by online shoppers / browsers are as part of Web Log, Click Stream data etc.

This data along with Social Media is fed to the Analytical model which analyses the trend and identifies potential customers / products that can be promoted

There are many companies /retailers that are using Special algorithms / tools to crawl the web and derive useful insights.


Some specialized companies like Palo Alto based Infer (https://www.infer.com/) offer an analysis software that can be linked to CRM system like SalesForce (http://www.salesforce.com/)a and rank the customer



Infer's ranking system uses more than 150 signals , which includes many data feeds - Census, Job Boards, Tweets , Facebook Likes and Comments



There are many algorithms that help in analysing the Facebook data

OpenGraph API is one of them


 See Also:

How to Segment Social Media Users (Influencers / Detractors / Recommenders)

How to measure Price Sensitity of Customers

Segmentation Standards / Framework

Payment Marketing Analytics - Something to Watchout in 2013

How to Push Messages Real-Time at Point-of-Sale or Payment

Payment Marketing is catching up now. This provides the Bank / Retailer to push messages / offers on real-time to their customers.

What is Payment Marketing?

Payment marketing is a mode of marketing using the Customer's payment card information and provide the customers with relevant offers / promotions etc. with the help of a layer built on top of the existing POS Terminals.

We have already seen the importance of Payment card companies in the field of marketing. This is another layer of it

How it Works?

The lifecycle of Payment Marketing revolves around understanding the customer through analytics. Swipely, a Payment marketing technology integrates with the payment network and collects data whenever the payment is made. The data is analysed and the insights are fed back to the marketing
system, which can be used to

• Send Personalized Messages (Thank you message etc)
• Cross-Sell products/Services
• Inform corrective actions

The above can be expanded based on NBA strategy. The advantages of Swipely is the ease of upgrading - no hardware / software is required and no changes in customer's payment m

echanism

DriveItNow uses the Credit, Criteria and Collateral to provide real payments

How does External Factors Affect Next Best Action (NBA) - Timely Promotions

What is (NOT) the Right Time for Pushing Promotional Messages

We have already had a look at the methods to identify the right time from customer segmentation or from propensity models

However, the real-time delivery of messages is a key in today's world. The customer might be in a different location or might be scanning for important news in his Facebook.

For example, pushing a promotional message during Boston Marathon is an example for NOT pushing promotions.

Business Rules and Next Best Action

The above example is a good one for using a Rules Engine to filter / time the delivery of the messages. There are open source Rules engine like Drools that can be used to create and maintain business rules .