We kicked off a discussion in the last lesson - Objective function - AI is nothing but an optimization problem. We discussed the definition of objective function in machine learning. Essentially, it is a scorecard that is required for a Machine Learning Model to know how well it is doing at a task. We also briefly discussed how important it is to align the Business Objective with the Machine Learning Objective function but quite often it does not happen, why so?

When companies kick off a machine learning project, especially during today’s times when there is a gold rush to establish oneself as an AI company, the tech and product teams start with AI in mind rather than starting from where it should have started - A user or a business problem.

‍

Solving the right problem is more important than solving a problem right away

‍

In this lesson, we will learn to do it the right way and we will also introduce the concept of PRD - Product Requirement Document, a wildly misused or unused tool that is needed to align people on a common mission.

‍

Even though this stage is tagged as Business understanding, it can very well be called User / Business understanding. It is captured in a Product Requirement Document (PRD). I will write a detailed article at some other time to walk you through various components of a PRD but for now, let us quickly look at the main components of a PRD. This should be the focal point of the rest of the framework.

1. User/Business problem: It all starts with creating a one-pager that outlines the User / Business problem.  A succinct and clearly worded User problem helps align large cross-functional - geographically dispersed teams towards a common vision. If it is an internal focused problem then it is worded as a business problem.This section is the most important part of the PRD as it helps align Technical and Non-Technical teams to agree on problems that they intend to solve and the problems that they do not want to solve.
2. Business Value (ROI analysis):  Provide a detailed analysis of the anticipated Return on Investment (ROI) associated with the product. This may include estimating the potential financial benefits, such as increased revenue, cost savings, or efficiency gains, compared to the investment required to develop and deploy the product.Outline any assumptions or factors that underpin the ROI analysis, such as market size, pricing strategy, adoption rate, or operational costs. Be transparent about the basis for the ROI calculations to ensure credibility and facilitate discussion.ROI analysis helps stakeholders make data-driven decisions by comparing the expected benefits of an investment to its costs. It provides clarity on whether the investment is worthwhile and aligns with the organization's objectives. This step will ensure we do not get into a “feature trap - A situation where a product development team becomes overly focused on adding new features to a product without considering their actual value or alignment with user needs”.
   
   Below is a sample of an ROI Analysis. Note - By no means this is exhaustive or plug-and-play but just guidance on how to start thinking about an ROI analysis.

3. User/Business Metric: How would you measure the success criteria when the problem gets solved? Or how would you know whether the AI/ML solution actually ended up solving the problem? Let us take an example of a recommender system for an eCommerce platform. The user problem statement could be “As a user of eCommerce platform XBay, I struggle to find the right electronics product for my home office among thousands of products on the platform”.
   Suppose we believe that we can build a recommender system that looks at customers’ preferences and recommends the right electronics product. We surely can do that.But how do you know by developing a recommendation system, the user problem will be solved?
   a) One way is to measure the Conversion ratio - How many users came to the platform and ended up buying something on the platform. After all, if the recommendation system is doing its job, then it should make the task of searching easier by increasing the relevance of the shown product on the front page. If the relevance goes up then the conversion should go up. So looks like Conversion or %converted user is a good metric.
   b) But maybe conversion is not only a function of relevance but also a function of price. While we increased relevance, maybe conversion did not go up. But what if instead of focusing on conversion we look at the users who bought the product, what was the time taken to find the purchased product? Again, if the recommender system is able to show the right recommendation then it should cut down the time to find the product for the user. Similarly, we can develop other metrics also. Choosing the right metric that ties back to the user problem is essential before diving into developing an AI/ML product.
   ‍
4. ‍Hypothesis:  A hypothesis is a statement or an assumption that can be tested. It includes two things - a change you are making (independent variable) and an expected outcome from that change (dependent/outcome variable).
   
   A hypothesis is an assumption, an idea that is proposed for the sake of argument so that it can be tested to see if it might be true.
   
   Many leaders phrase their feature ideas as I strongly believe that if we launched this feature, it would increase user adoption by X%. What is wrong with it? ‘Belief is your identity’ — When you say “I believe that …” — You associate yourself with the idea.
   
   This makes it difficult for your team members to challenge your idea. After all, challenging your idea is challenging your belief. As a result, what happens is that the engineering team becomes a feature company - a lot of releases but little to show for it in terms of business value.
   
   'Hypothesis’ on the other hand has an inherent notion that it needs to be validated. It thus separates ideas and opinions from facts by opening the way for experimentation. The hypothesis is a great equalizer in any product-business-technology team.
   
   A sample hypothesis is listed below for your reference
   If {this happens} then {this will happen} which will result in an x% increase of the user/business metric M by y% in T time
   
   A thought exercise - How many of your executives talk like this? Somehow leaders have been asked to appear confident and they have interpreted confidence as being ‘Know-it-all’. Hope this small reminder will help you change your company culture.

5. While other aspects of PRD are outside the scope of this article, another important component is when the Product Lead starts phrasing the User/Business or user problem into a machine learning problem. The ability to be able to deeply understand User/Business problems and their intersections with possible AI/ML algorithms is what has given rise to the field of AI Product Management.
   
   So let us take a look at how AI product managers can phrase a business problem into an ML problem.

‍

Framing the Machine Learning Problem into a Machine Learning Problem

‍
Step 5.1. Breaking down the User/Business problem into sub-problems to answer is it solvable.

In the previous section, we identified the problem statement (User or Business problem) that needs to be solved. However, time and again you and your team will pick up problem statements that may seem unsolvable. While a situation like this is not specifically tied to AI/ML, this problem becomes more daunting in the context of AI/ML.  Why so? It is because:

1. It has always been done this way’ - A resistant culture: Sometimes your team will conclude that certain problem statements can't be solved with AI/ML and can only be done by humans. This could be a symptom of a culture of resistance to the adoption of AI. This could be because people lack the required technical skills to understand, hence embrace AI or there is a fear of being replaced by AI. We will discuss this more in later modules.
2. Right experts not at the table: Unlike its conventional program counterpart, AI/ML is not something that a data science team should solve in silos. It requires a true collaborating partnership between Non-Technical Field Experts, Tech Teams, and Sponsors from conception to deployment. Not including the right person may lead to a poor understanding of the problem statement and in turn assumed non-solvability.
3. Failures in previous attempts may fuel negative feelings about the capabilities of AI as people are still coming to terms with seeing true value being delivered by AI.
4. Fear of the unknown: Poor understanding of recent advancements in AI space could limit a team from seeing a connect between what is to what could be.

When faced with this situation, it helps a lot to apply the power of the first principle and break down problem statements into subproblems.
The First Principle approach encourages us to break down complex problems into their fundamental truths, questioning assumptions, and re-examining the underlying principles. By stripping away layers of complexity, we gain clarity and insight into the problem at hand. This will eventually help us convert the problem into more management subproblems. This lends itself to tackling each aspect individually, gradually building towards a solution.

Each sub-problem becomes a stepping stone on the path to solving the larger issue. With careful analysis and iteration, we address one sub-problem at a time, leveraging the power of AI and machine learning algorithms where applicable.

An example of an AI/ML problem statement that was solved by employing first principles thinking and breaking down the problem into smaller components is "Autonomous Vehicle Navigation in Complex Urban Environments."

Initially, engineers faced the challenge of creating AI/ML algorithms capable of safely navigating autonomous vehicles through densely populated cities with numerous obstacles, unpredictable traffic patterns, and diverse road conditions. The problem seemed overwhelming, as conventional approaches struggled to address the complexities involved.

However, by applying first principles thinking, engineers broke down the problem into smaller, more manageable components:

1. Sensor Fusion: Instead of relying solely on a single sensor type (e.g., cameras or LiDAR), engineers combined data from multiple sensors (e.g., cameras, LiDAR, radar) to provide a comprehensive view of the vehicle's surroundings. This sub-problem involved developing algorithms to integrate and process sensor data in real time.
2. Object Detection and Recognition: Engineers focused on developing AI/ML models capable of accurately detecting and recognizing various objects in the vehicle's environment, such as vehicles, pedestrians, cyclists, and traffic signs. This sub-problem involved training deep learning models to identify objects from sensor data with high precision and reliability.
3. Path Planning and Decision Making: Engineers designed algorithms to analyze the surrounding environment, predict the behavior of other road users, and generate optimal paths and trajectories for the autonomous vehicle to follow. This sub-problem involved incorporating decision-making algorithms that consider factors such as traffic rules, road conditions, and pedestrian behavior.
4. Real-time Control and Execution: Engineers developed control algorithms to translate planned trajectories into precise steering, acceleration, and braking commands in real time. This sub-problem involved ensuring smooth and safe vehicle operation under varying conditions.

By breaking down the autonomous vehicle navigation problem into these smaller components and addressing them individually, engineers were able to overcome the challenges associated with navigating complex urban environments. This approach enabled them to develop robust AI/ML solutions capable of safely and efficiently guiding autonomous vehicles through city streets, contributing to advancements in self-driving technology.

‍

Step 5.2. Sequence the sub-problems & pick the first sub-problem to solve

Once we have broken down the problem statement into smaller sub-problems, we can sequence them in the order of dependency or complexity.

‍

Step 5.3. Re-assess whether the problem is solvable otherwise go back to Step 5.1 and repeat this. If the sub-problems are not solvable, in all likelihood we did not apply the first principle right in Step 5.1.

Step 5.4. Framing Machine learning function by mapping Input to Output given Data D such that Loss Function is satisficed

We know from Module 1 Lesson 3 here that Machine learning is essentially finding a mapping function F(x) so that F(x) maps Input to the desired output.

This step involves:

5.4.1 Understanding the Input (I): For example when we wanted to predict at what price a house would sell for, we needed some input. Do you remember what it was? It was the identifier of the house say House Address. But was that enough? We also needed to provide the features of the house such as SQ Ft, #BR, #BA, Location, etc.

5.4.2 Outlining the Desired Output (O). In this case, it was ‘The price at which the house will sell for in $’.

5.4.3 Understand the required context Data (D): All AI model requires Data, which acts as a context from which the machine learns. In the above case, we would need data of comparables, houses with similar features (Similar SQ Ft, #BR, #BA, Location, etc.), and prices for which they got sold in the market. We will revisit Data in Module 2 Lesson 2. But at the first stage, we should have a pretty good idea of whether we have the context data D required for the machine to learn the function. If we don't have this knowledge then we need to increase our intimacy with in-house data.

‍

5.4.4 Framing the Scenarios S: I → O that maps input to output s.t. we minimize a Loss (Objective) Function L(S): The next part is to put in writing that we need to develop a function such that for every Scenario S there is a mapping between I and O. However, this mapping should be under a mathematical constraint that we are going to measure by Loss Function L(S).We discussed this in Module 2 Lesson 1 here. Let us understand this with two examples:

1. In the above case, for the pricing recommender engine, we can have a simple objective functionL1 = Sum |Predicted Price of house i from the model - Actual price of house i from the model|We know from Module 1 Lesson 1 that this is called Absolute Error. We can instead take MSE (Mean Squared Error) whereL2  = Sum (Predicted Price of house i from the model - Actual price of house i from the model))^2 / #Houses
2. Now, let us suppose that we are building a recommender engine that recommends houses on our real estate website. So we need to rank houses in a particular order in which they will appear on our website. One question that Product and Business leaders should ask is which one of these rankings is the best? How do we answer this question?

                We will answer this question later in this series as we need to build some more foundational understanding of the Machine learning models.

Step 5.5. Rule-based Model or AI? Evaluate whether ML is the right approach

We already know that we should not be looking to implement the ML model just for the sake of it. Some business problems don’t need ML as simple business rules can do a much better or equally good job. For other business problems, there might not be sufficient data to apply ML as a solution or ML is an overkill when Rule based model can do the job. We discussed this in detail in Module 1 Lesson 3 here.

• So how do you know whether you can use AI to solve the problem?
  Here is a simple thumb rule that we recommend "If humans can do something and can do it consistently, then Machines leveraging AI can do it better" ‍


• So how do you know whether you should use AI?
  If you have already exhausted the based Model and either a Rule-based model can't be used to solve the problem or there is a much higher lift we are expecting from a Machine learning model then we can leverage ML.

Step 5.6. ML Objective → Business Objective such that Business Objective is satisficed

Map the technical outcome to a business outcome. Let us revisit the business outcome in the above two examples:

1. What could be a business objective of a Pricing Recommendation Engine? Profitability sounds like a decent choice. But is there a connection between the Mean Square Error objective function that we created for our ML model and the business metric profitability?
   
   Yes, if we are somehow we are able to keep MSE low, this would mean we are able to predict the house pricing with a very high degree of accuracy. For a business, this would open many opportunities to arbitrage if somehow the market is pricing the house at a higher price than the actual price in the market.
   

2. What about the business objective of a Ranking Recommendation Engine? how about % Converted users? After all the higher the percentage of users converting on our real estate website, the more revenue (Pay Per Lead or commission on successful sales) the business can make. Irrelevant ranking could turn off customers hurting the business. In a survey, people indicated that 79% of people who don't like what they find on one site will go back and search for another site (Source). Thus, Higher relevance of search results ⇒ Better Ranking ⇒ Improved Revenue

One thing to note is that above we said ‘Satisficing Criteria’. Many decisions in Machine learning problems in a real-world setting will fall into what we call a Satisficing decision rather than a satisfying decision.

A satisficing metric is a measurement or criterion used in decision-making that aims to find an acceptable solution rather than an optimal one. Unlike optimization metrics, which strive to find the best possible outcome, satisficing metrics focus on identifying solutions that meet a predefined threshold of acceptability or sufficiency.

We discuss this at length in the article - Metrics: ‘Satisficing’ Metric — Not all metrics need to be optimized. This acts as a reminder to the product and technology team to not optimize the model unnecessarily if there is a marginal improvement in business metrics.

‍

Step 5.7. Align Data Engineering, Data Science, and Business Executives on 1-5 so that DS/DE leads can take on the next part of the framework

Many AI projects fail not because of a lack of a technical toolkit to solve the problem but because of a lack of alignment between internal stakeholders. The last phase in the stage is a reminder to all leaders and participants to ensure a collaborative approach in the journey. As they

When you have to walk fast go alone, when you have to go far go together. Walking together is always the right answer in the long run so build alignment early and often.

In the next lesson, we will discuss the next phase in our Machine Learning Development Framework - Data. Data is the new currency so let us see what needs to be in place to extract the maximum out of it.

‍