Von Westendorp Approach Using Python

 

Von Westendorp Approach Using Python

(Artificial Intelligence Series)

Strategic pricing is a critical process of Product Management, when it comes to launching a new product. There are many methods to pricing your product and one of the most popular ones is the Von Westendorp approach. The beauty of this approach is that it provides a glimpse in the form of a “Price Sensitivity Meter”. Once the graph is plot, we get the following price points at the intersections:

a.       OPP (Optimal Price Point) – The ideal price where the consumer is most willing to pay.

b.      IPP (Indifference Price Point) – The price where the consumer is least willing to pay.

c.       PMC (Point of Marginal Cheapness) – The price beyond which the consumer will consider the product to be too cheap and might not consider buying.

d.      PME (Point of Marginal Expensiveness) – The price beyond which the consumer will consider the product to be too expensive and might not consider buying.

From the above four price points, we get the RAI (Range of Acceptable Index); which can be used to price our product. All said and done, let’s get our hands dirty now!

A survey is conducted to assess the likely price points with questions like:

·         At what price would you consider the product to be too expensive and out of reach? (Too Expensive)

·         At what price would you consider the product to be expensive but still worth buying? (Fleecing)

·         At what price would you consider the product to be a cheap and worth buying? (Bargain)

·         At what price would you consider the product to be too cheap that quality would be doubtful? (Too Cheap)

All the responses are collected and the dataset is arranged in the following manner:

Please note that the first two columns are sorted in descending order while the last two are sorted in the ascending manner. Using this dataset, one can plot the Von Westendorp graph in excel itself. While this might seem quite easy and does not require any other dependencies like a Python IDE; I personally detest doing it in excel. I will explain the reasons at the end of this article.

I’ve used Miniconda CLI, Jupyter notebook and .csv data format for this project. You may choose other IDE and tools that you are comfortable with like Spyder, NoSQL, etc.

Launch a fresh notebook and import the necessary libraries. Now, load the data and check the shape as shown below:

Next step is to remove the intransitive price preferences* and compute the cumulative frequencies, as shown below:

Once the above is done, we need to define a table where all the array values can be tabulated.

 

Now, we need to compute the results and get the values, as shown below:

The last step is to plot the graph:

The output graph will look like this:

So, that’s it, pretty simple, isn’t it? Well, no. It wasn’t that easy when I did it for the first time. Let me share my learnings with you. Prior to plotting it in Python, I did it with excel. I’m presenting a comparison table of the values that I got:

One can easily note that the values arrived at via Python are higher than those from excel. The reason being the following:

·         In Excel, it will accept all price points from the dataset as it is, in order to plot the graph while in Python, I have the freedom to “remove intransitive price preferences*”. If I had skipped this step, values from Excel would match those from Python.

·         It is imperative to remove the intransitive price preferences as not doing so would present a skewed graph, as well as, generate incorrect values.

·         As a result, my data shape dropped to 16 from 39; meaning entire 23 rows were invalidated.

The obvious question that would arise in the minds of data analysts and machine learning enthusiasts is – Why not scale or normalize the dataset?

The answer is plain simple here – applying normalization to the dataset would replace the suspect values (intransitive, duplicate, etc.) with the average or mean. Since, this is akin to “changing the respondent answers”, it would amount to gross manipulation of data and render the purpose of survey useless.

Thus, the key takeaways from this exercise are:

1.       Statistical inferences are better accomplished with Python than with Excel

2.       Respondents are prone to error and one might need to drop off the incorrect answers

3.       Adequate sample size is a must to ensure that there are enough data points to suffice for the inference

4.       Scaling or normalization is not a ‘one size fits all’ solution and must be applied with sufficient caution

5.       Intent needs to be preserved even if it is at the cost of the content

Note: *intransitive price preferences – the price points that are not linear but are concurrent. This can happen where a respondent might choose the same price point for more than one question. For e.g. Rs. 8000 selected as ‘bargain value’ as well as ‘too cheap value’.

P.S. You can contact the author to know more about this article. The sample python code is available at https://github.com/southpau79/humintonline .

For those lazy to read & have scrolled to find the end of page, here's the video:

OPINION MINING & JUDGEMENT ERRORS

OPINION MINING & JUDGEMENT ERRORS

(Artificial Intelligence Series)

Video link for this article:

Machine learning is a great tool for developing predictive models. One of the most interesting applications is its use in opinion mining. Opinion mining too is a broad field with numerous use cases ranging from product reviews to personality assessment. It is often argued by advocates of AI that human judgment is always biased and machines can perform better in this area. However, the bone of contention here is this – Who is training those machines? Well, without getting into the complexities involved let me bring your attention towards the psychological / technical basis of judgment and the error-prone systems that we face each day.

Products

How do we judge any given product? Without understanding this, training a machine to analyze huge chunks of data and generate results – be it a Recommender System or a UX dashboard; the value to effort ratio remains below unity. We have tried everything from the basic “queuing theory” to “gamification” and even the “hyper-game theory”, but still struggle to find the golden ratio. In an ideal scenario, the following equation will hold good:

When emotions overtake the intellect fueling our experience for the cause or reason-to-believe that goes beyond the expectation of the product itself, we make the decision to buy. Whether it is a need or a want, whether it is justified or not, whether it comes with a positive or negative feeling (e.g. vaccination is received out of fear), people buy something if and only if the perceived value is greater than the perceived effort. Great marketers and advertisers know this too well and they constantly exploit the emotions to evoke the decision to buy. 

Just look at the film industry and compare the movie reviews from the pre-covid and post-covid eras. Until two years ago, when the public could watch their choice of movie inside a theatre, their review would mostly largely positive where even mediocre movies would gain 3.5-4 stars on a scale of 5. Post-covid, when people are able to watch movies mostly on the OTT platform, their reviews have become more critical where everyone is seeking for quality content amidst binge-watching. Many starrer movies are being rated below 3 stars and seemingly underdog movies are being rated higher. The reason for this shift is that the pandemic has bitten into the overall “movie-going” experience, thereby letting the watchers use their intellect more rather than being swayed by their emotions that were previously being fueled with anticipation via teasers, trailers, promotions, campaigns and advertisements.    

Thus, no amount of “sentiment analysis” of movie review data can predict anything with accuracy; without understanding the basic premise or context. 

Processes

What happens before any output is generated? If we do not understand the process behind a given outcome, we are playing a “trial-and-error” game. Obviously one can’t play a game where the rules, the stakes and the quitting time remain unknown. This is one of the main reasons that domain knowledge is an absolute pre-requisite for delivering any kind of technological solution viz. machine learning in the current context. Understanding a process involves going beyond the process itself, as shown below:

With the kind of technological advancement we have today, one can create a robotic bartender and automate the process of serving drinks at the local pub. But, without considering a “process approach” that includes the process, the environment, the human factor and other variables, one will end up with nothing more than a “talking vending machine”. 

Training a machine simply by mimicking what a human being does, does not add any value to the process per se. If your algorithm can either augment or compliment what a human being or an existing machine can, then it is definitely worth it to take the leap forward. This is the main reason as to why the adoption of “Conversational AI” like chat-bots, digital assistant, etc. has not picked up on a massive scale. I know that many AI enthusiasts and experts will disagree with me and probably refer to several hypotheses and modern theories like AGI, ASI, Swarm Intelligence, etc.; but the harsh truth is that we are centuries far from it.            

People 

Why do people perceive something the way they do? The world we are living in today is primarily system-driven, be it the rule of democracy or online shopping. That’s what most of us are made to believe. But, if you do a reality check at the grassroots level, everything depends on people. End of the day, it is people who affect both processes and products, either directly or indirectly. Organizations with the best processes and products in place can fail if it is not support by the right kind of people. In terms of leadership, people factor is of prime importance. 

For the sake of our understanding, if we ignore the Fortune 500 list; all other companies including startups and SMEs heavily and solely depend on their leadership structure for their success, growth and / or survival. So, when it comes to selection of leaders or rewarding top performers, people make huge judgmental errors. Similar to management of products and processes, there are numerous theories for people management like BEI (Behavioral Event Interviewing), Psychometrics, Adam’s Equity Theory, Hertzberg’s Motivation Theory, Jay Hall Conflict Management Theory, etc. 

Most of these judgmental errors can be attributed to the misunderstanding of two most basic theories: 

Dunning-Kruger Effect – A hypothetical cognitive bias stating people with low ability at a task overestimate their ability. 

Imposter Syndrome – A psychological pattern of perceived fraudulence, where an individual doubts his / her skills, talents, or accomplishments despite having external evidence of competence.

It is simply illogical to compare Guy Martin (famous pit stop wheel changer) with Michael Schumacher (famous F1 racer). If one were to get into their minds, Guy would be found suffering from the Imposter Syndrome and Michael Schumacher facing the Dunning-Kruger Effect. Now, if one were to reverse their roles, what would be the effect? While this may seem to be an extreme example; such scenarios occur often in our day-to-day routines. An example that everyone can relate to is your appraisals. More than the actual ability and past performance, more weightage is given to attitude. Those who rate themselves high during self-appraisal are the ones who usually get the highest rating in the final appraisal. The fact remains that the lot who hold themselves in high esteem and exuberate confidence are actually part of the Dunning-Kruger Effect, while those who are self-doubtful lie somewhere in the Imposter Syndrome region. 

Thus when it comes to people analytics or consumer research, even the best of experts can go wrong when the primary data is based on opinions prone to judgment errors.  

The following quote sums up my point too well:

Summarizing the whole article, I can say that the following factors need to be accounted for before embarking on a decision-making journey based on analytics:

• Value created or projected (product-centric)

• Variables both known and unknown (process-based)

• Viewpoint of everyone involved (people-centric)

I help improve people, processes and products.. Reach out to me to know more. 🙂 

DIGITAL TRANSFORMATION (DX) V/S FAKE TRANSFORMATION (FX)

 

KNOW THE DIFFERENCE

For those who don't like to read, here's the video:

In this new era that is ushering in the 4^th industrial revolution, almost every organization is seeking to adopt “Digital Transformation”. But today’s world is filled with fake transformation projects. A mere technology upgrade or change management do not account for authentic transformation. Organizations and leaders both in the public and private sector have been caught unaware with the “delusion of digital transformation”. Many of the projects that CEOs and other leaders undertake are akin to moving from traditional systems to paperless office or shifting from legacy to cloud infrastructure. They are victims of this grand delusion and will eventually lead their organizations to an early grave.

Digital Transformation (DT or DX) is the adoption of digital technology to transform services or businesses, through replacing non-digital or manual processes with digital processes or replacing older digital technology with newer digital technology. Digital solutions may enable - in addition to efficiency via automation - new types of innovation and creativity, rather than simply enhancing and supporting traditional methods. (Source: https://en.wikipedia.org/wiki/Digital_transformation)

Digital Transformation is application of digital capabilities to processes, products, and assets to improve efficiency, enhance customer value, manage risk, and uncover new monetization opportunities. (Source: https://www.cio.com/article/3199030/what-is-digital-transformation.html)

Researchers have analyzed some digital transformation strategy examples and trends of recent years.

Eventually, some of the key predictions were:

·         By 2023, investments in digital transformation will grow from 36% in 2019 to over 50% of all information and communication technology investments.

·         Investments in direct digital transformations are rapidly growing at an annual rate of 17.5%. They’re expected to approach $7.4 trillion over 2020-2023.

·         By 2024, artificial intelligence-powered companies will respond to their customers and partners 50% faster than their peers.

(Source: IDC FutureScape: Worldwide Digital Transformation 2020 Predictions, October 2019)

The idea is to create whole new products, services or business models, not just improve old ones. Companies that go through digital transformation are said to be more agile, customer centric and data driven. DX can have different blueprints depending upon the company and industry. But, in all cases it needs to follow these basic steps:

In terms of the People-Process-Technology framework, a definite “cultural shift” is a prerogative to achieve DX in its truest sense. Businesses must learn to push boundaries, experiment, and accept the associated failures. This potentially involves abandoning well-established processes for new ones – ones that are often still being defined.

For real DX, one needs to separate from the herd involved in FX.

Look at the chart given below:

It doesn’t matter if you find yourself in quadrant one, but never stray from the course that turns out to be an FX instead of genuine DX. A few good examples of DX are given below for you to understand and get inspired:

Anheuser-Busch (AB) InBev has looked at how digital transformation can be applied all through the business while retaining focus on serving its consumers. They have achieved it via the following –

·         Developed a mobile application called B2B with an inbuilt algorithm that makes specific replenishment suggestions, creating opportunities for sales staff to talk about new brands and products with store owners.

·         Created a tech innovation lab, Beer Garage, to explore ways that artificial intelligence (AI), machine learning (ML) and the internet of things (IoT), among other technologies can be used to improve experiences for consumers and retailers alike.

DHL is well known for its excellent stock management and supply chain but that did not stop them from improving. Their stock management and supply chain systems are easy to use and automated, but they want to take things to the next level. For this they decided to team up with Ricoh and Ubimax and –

·         Developed application for smart glasses. By pairing smart glasses with these applications, it can be used for reading bar codes, streamline pickup and drop off and reduce the chances of errors. Their stock price doubled from 20 euros in 2016 to 40 euros in 2018.

Honeywell has helped many companies improve their digital presence and capabilities. In 2016 the company began transforming itself digitally by introducing new technologies like data-centric, internet-connected, offerings and devices. They have leveraged digital solutions like this –

·         Using new digitized internal solutions and customer data, the company now offers its customers more technology solutions and has reinvented its industrial process control. As a result, in the past four years, Honeywell’s stock per share price has gone from $95 to $174.

As a leader what you must be doing is listed below:

·         Develop competency – Invest in talent and upgrading skills of employees in the organization. Digital and analytics skills are very critical for DX and go a long way in bringing about real transformation.

·         Plan and prioritize – Assess the current scenario and develop a roadmap. An efficient plan makes a solid foundation for any achievement. Pick relevant themes and prepare a business case.

·         Commitment – Absolute commitment along with appropriate investment is crucial to bring about DX. Always look at the tangible as well as intangible benefits of the project.

A snapshot for a typical CDO is provided below:

Hence, the message is to steer clear of all types of fake transformation projects and drive real digital transformation projects that truly alter the arena. I can conclude this article with an allegorical anecdote:

Resume statement (DX)	Reality (FX)
Surpassed targets by 60% through implementing energy saving initiatives and loss prevention strategies via leading digital transformation projects across the organization, thereby contributing to the bottom line of the company.

Note: This blog including all articles are copyrighted by the author. Wherever, external content is used, the relevant sources are posted in separate links or the images itself. 

PREDICTIVE ANALYTICS FOR THE ENERGY SECTOR

 

Machine Learning Business Use Case

One of the major areas where AIML can truly change the landscape is the energy sector. Let us first understand the “Analytics Maturity Model” before attempting to consider the implications:

Almost all the OEMs and principals of the energy sector have in-built objective analytics (e.g. enterprise pipeline management solution) along with ICS / DCS since ages that address the essential process requirements like vibration monitoring, condition monitoring, cathodic protection etal. pertaining to asset integrity. Then, we also have the major players that have developed real-time monitoring solutions in pace with the 4^th industrial revolution viz. advanced telemetry, imaging systems, distributed monitoring systems (vibration, acoustics, temperature), etc. However, when it comes to predictive analytics there is a huge gap since it is mostly done in hind-sight and is not real-time. There are very few players in the market who have indeed rolled-out real-time predictive analytics either as a SaaS model or a standalone offering. The GTM feasibility of many other similar products is underway as the global MLaaS market is forecasted to reach USD 117.19 billion by 2027 (source: https://www.fortunebusinessinsights.com/machine-learning-market-102226).   

Use-case Scenario

For the energy sector, there can be several use-case scenarios as listed below:

•  Process Optimization
•  Reducing MTBF / MTTR
•  Loss Prevention (process losses, leakages, etc.)

Here, I will attempt to present a use case scenario for “real-time predictive analytics” for a typical natural gas pipeline. A typical high-level production process flow is given below:

 

In this example, let us presume that we have a simple process flow for pipeline data management as shown below:

Data Scientists / Analysts must be aware of the entire data science life-cycle before attempting to initiate the solution as shown below:

Once the logged data is collected, the machine learning process is initiated as per the process flow shown below:

It is to be noted that supervised learning models are preferred since OEM design parameters need to be considered for baseline / target values. However, semi-supervised algorithms may be used while using a classification approach to get better results.

Building the model

Remember that the processed data will be humongous with hundreds of columns to accommodate all the relevant parameters collected via numerous sensors required to maintain pipeline integrity. Here, I will be focusing only on semi-supervised graph-based algorithms that can be validated and finally selected for deployment. For ease of business, I’m only considering distributed temperature monitoring dataset in this case.

Label Spreading

It is based on the normalized graph Laplacian:

This matrix has each a diagonal element l_ii equal to 1, if the degree deg(l_ii) > 0 (0 otherwise) and all the other elements equal to:

The behavior of this matrix is analogous to a discrete Laplacian operator, whose real-value version is the fundamental element of all diffusion equations. To better understand this concept, let's consider the generic heat equation:

This equation describes the behavior of the temperature of a pipeline section when a point is suddenly heated. From basic physics concepts, we know that heat will spread until the temperature reaches an equilibrium point and the speed of variation is proportional to the Laplacian of the distribution. If we consider a bidimensional grid at the equilibrium (the derivative with respect to when time becomes null) and we discretize the Laplacian operator (∇² = ∇ * ∇) considering the incremental ratios, we obtain:

Therefore, at the equilibrium, each point has a value that is the mean of the direct neighbors. It's possible to prove the finite-difference equation has a single fixed point that can be found iteratively, starting from every initial condition. In addition to this idea, label spreading adopts a clamping factor α for the labeled samples. If α=0, the algorithm will always reset the labels to the original values (like for label propagation), while with a value in the interval (0, 1], the percentage of clamped labels decreases progressively until α=1, when all the labels are overwritten.

The complete steps of the label spreading algorithm are:

•         Select an affinity matrix type (KNN or RBF) and compute W
•         Compute the degree matrix D
•         Compute the normalized graph Laplacian L
•         Define Y⁽⁰⁾ = Y
•         Define α in the interval [0, 1]
•         Iterate until convergence of the following step –

It's possible to show that this algorithm is equivalent to the minimization of a quadratic cost function with the following structure:

The first term imposes consistency between original labels and estimated ones (for the labeled samples). The second term acts as a normalization factor, forcing the unlabeled terms to become zero, while the third term, which is probably the least intuitive, is needed to guarantee geometrical coherence in terms of smoothness.

Python code for Label Spreading

We can test this algorithm using the Scikit-Learn implementation. Let's start by creating a very dense dataset:

from sklearn.datasets import make_classification

nb_samples = 5000 nb_unlabeled = 1000

X, Y = make_classification(n_samples=nb_samples, n_features=2,

n_informative=2, n_redundant=0, random_state=100)

Y[nb_samples - nb_unlabeled:nb_samples] = -1

We can train a LabelSpreading instance with a clamping factor alpha=0.2. We want to preserve 80% of the original labels but, at the same time, we need a smooth solution:

from sklearn.semi_supervised import LabelSpreading

ls = LabelSpreading(kernel='rbf', gamma=10.0, alpha=0.2)

ls.fit(X, Y)

Y_final = ls.predict(X)

The result is shown, as usual, together with the original dataset:

As it's possible to see in the first figure (left), in the central part of the cluster (x [-1, 0]), there's an area of circle dots. Using a hard-clamping, this aisle would remain unchanged, violating both the smoothness and clustering assumptions. Setting α > 0, it's possible to avoid this problem. Of course, the choice of α is strictly correlated with each single problem. If we know that the original labels are absolutely correct, allowing the algorithm to change them can be counterproductive. In this case, for example, it would be better to preprocess the dataset, filtering out all those samples that violate the semi-supervised assumptions. If, instead, we are not sure that all samples are drawn from the same pdata, and it's possible to be in the presence of spurious elements, using a higher α value can smooth the dataset without any other operation.

Similarly, one can also use “Label Propagation based on Markov Random Walks” to find the probability distribution of target labels for unlabeled samples given a mixed dataset by simulation of stochastic process.

It is to be noted that the above two algorithms can be applied against design parameters adjusted against the best achieved values based on historical data. In any other scenario, the predictions may go totally wrong since the iterations are based on absolute correctness of the labels (e.g. derived from control limits or stability tests).

Once the model has been validated with good level of accuracy and deployed for production, the ICS (Industrial Control System) interface (API integration) for real-time predictive analytics can be set-up that will trigger alarms at the set control points and provide deeper insights for loss prevention. As the ML system matures over time, one may even be able to move to the next level i.e. “Prescriptive Analytics”.

Conclusion

Given below is a estimated annual cost for the minimum length (100 km) and minimum diameter (12 inch) of a large company:

Component	Segment A (small)	Segment B (large)	Total
Periodic Inspection	$ 121550	$ 297000	$ 418550
Scheduled Pigging	$ 40000	$ 80000	$ 120000
Leak Surveys (@95% of total)	$ 13000	$ 22000	$ 35000
Repair Backlog (@annual cost of rule)	$ 103000	$ 197000	$ 300000
Total	$ 277550	$ 596000	$ 873550

Source: Greene’s Energy Group, LLC (2013), updated to 2015 dollars using the Bureau of Labor Statistics US All City Average Consumer Price Index (2013=233.5; 2015=237.8).

Note: The above table is a simplified one and does not include several factors like terrain, regulated / unregulated, mobilization costs, compliance costs, etc.

Cost-benefit Analysis

It is to be noted that even a 1% of loss prevention against an average production of 6 MMSCFD (169512.82 m3/d or 1066203.53 bbl/d) goes a long way in cost optimization for an oil & gas producer.

For this particular example, a savings of US$ 5400 / km can be easily achieved for a combined loss prevention strategy (gathering / process loss & maintenance / inspection cost).

Note: The above is a highly conservative estimate and actual savings may be 4x higher in actual practice. 

Reference Standards: ISO 55001:2014 Asset Management, ISO 31000:2018 Risk Management, ISO 14224:2006 Petroleum, petrochemical and natural gas reliability & maintenance, ISO/IEC CD 22989 AI Terms & Concepts, ISO/IEC AWI TR 5469 AI Functional Safety