Outliers are data points that deviate significantly from the rest of the dataset and can have a profound impact on business decision-making. While statistical techniques are essential for identifying outliers, diving straight into analytics without first understanding the business context often leads to misinterpretation. This article outlines a structured approach to outlier detection with a focus on practical business applications. Real-life examples from various industries highlight how companies can identify and manage outliers to drive performance improvements.

Step 1: Understanding the Business Context

Before employing any outlier detection techniques, it is critical to understand the business purpose behind the data. Knowing why the data is being collected and what it represents allows you to set the correct expectations for what is normal and what constitutes an outlier.

1.1 Why the Data is Collected

Data is typically collected to drive key business decisions, whether it’s customer behavior analysis, supply chain optimization, or risk assessment. Understanding the reason behind the data collection can help clarify what anomalies might look like and whether they are actual problems or simply natural variations.

Example: Banking Industry – Fraud Detection

In retail banking, transactional data is collected to monitor customer behavior and detect fraudulent activities. A sudden, large withdrawal from a customer’s account might be flagged as an outlier. However, knowing the business context is crucial. If the transaction happens during the holiday season, it may align with expected high spending. But if a similar transaction happens in an unusual location without prior high-value transactions, it may signal potential fraud. Hence, understanding the temporal and geographic context helps distinguish between regular and fraudulent activities.

1.2 What the Variable Represents

Every variable in a dataset has a specific meaning, and what constitutes an outlier in one variable may not be the same for another. For example, sales revenue might have outliers at both the low and high ends, and depending on the business context, either or both could indicate important trends or issues.

Example: Manufacturing Industry – Equipment Maintenance

In the manufacturing sector, machine performance data is constantly monitored. One variable might be the temperature of a critical machine component. If the normal operating range is between 70°F and 120°F, temperatures outside this range could indicate an issue. A sudden spike to 150°F might signal a pending equipment failure, while a drop to 50°F might indicate a sensor malfunction. In this case, outliers can help predict and prevent costly machine downtime, but only if the variable’s context is well-understood.

Step 2: Statistical Exploration of Data

Once the business context has been established, statistical methods can be employed to detect outliers in numerical or categorical data. Let’s delve into some real-world applications where these methods bring immense value.

2.1 Outlier Detection for Numerical Variables

For numerical variables, statistical methods like summary statistics, Z-scores, and the interquartile range (IQR) are commonly used to detect outliers. These methods can pinpoint unusual data points that may require further business validation.

2.1.1 Summary Statistics

By calculating mean, median, standard deviation, and range, organizations can identify outliers that deviate significantly from typical data points. However, these summary statistics should be interpreted within the business context.

Example: Healthcare Industry – Patient Vital Signs Monitoring

In healthcare, vital sign data such as heart rate, blood pressure, and oxygen saturation is continuously monitored for patients. For instance, a heart rate of 200 bpm would immediately stand out as an outlier. However, this may or may not be a cause for alarm depending on the patient’s condition. For a high-performance athlete, this could be normal during intense activity, but for an elderly patient in a hospital bed, it would signal a serious health issue that requires immediate intervention.

2.1.2 Z-Scores

Z-scores can highlight how far a particular data point is from the mean in terms of standard deviations. Z-scores greater than 3 or less than -3 are typically considered outliers.

Example: Retail Industry – Sales Data Analysis

In retail, sales data can vary greatly based on time, location, and product categories. Let’s say a national retail chain wants to assess the performance of its Black Friday promotions. While most stores see a 20-40% increase in sales, one store shows a 150% increase. This data point has a high Z-score, indicating it is a significant outlier. Investigating this further, it’s revealed that the outlier is due to a localized promotion that wasn’t included in the overall campaign plan. Identifying this outlier helped the company replicate the success of that promotion chain-wide.

2.1.3 Interquartile Range (IQR)

The interquartile range (IQR) is useful for identifying extreme values in skewed data. The IQR is the difference between the first and third quartiles, and outliers are typically identified as data points falling outside 1.5 times the IQR above the third quartile or below the first quartile.

Example: Financial Services – Investment Portfolio Management

For an investment firm managing a portfolio of stocks, bonds, and other assets, return on investment (ROI) data might typically vary between 2% and 10% per year. However, one particular stock in the portfolio returns 30%, an outlier identified using IQR. Further investigation reveals that this stock experienced rapid gains due to a merger announcement. While initially flagged as an outlier, this data point now informs future investment decisions for similar scenarios, helping the firm refine its investment strategy.

2.2 Outlier Detection for Categorical Variables

For categorical variables, outliers often appear in the form of rare or unexpected categories. Detecting these outliers requires a combination of frequency analysis and business rules to ensure accurate detection.

2.2.1 Frequency Distribution

Analyzing the frequency distribution of categorical variables allows businesses to identify rare or unexpected categories.

Example: E-commerce Industry – Product Returns

In an e-commerce business, return reasons are captured in a categorical format (e.g., “Size Issue,” “Damaged Item,” “Not as Described”). A sudden spike in “Not as Described” returns for a particular product category may be an outlier. Upon investigation, the company discovers that a recent update to product descriptions contained errors, leading to confusion among customers. The outlier detection helps the company fix the descriptions quickly, thereby preventing further customer dissatisfaction and lost sales.

2.2.2 Business Logic

Outliers in categorical data can also result from data entry errors or inconsistent coding.

Example: Insurance Industry – Policy Claims Categorization

An insurance company tracks policy claims under different categories like “Accident,” “Natural Disaster,” and “Theft.” Suddenly, a new category appears: “Miscellaneous.” Upon further investigation, the company discovers that several claim handlers have been misclassifying claims that didn’t fit neatly into existing categories. These outliers lead to a review of the classification system, ensuring more accurate data entry in the future.

Step 3: Applying a Structured Approach to Detecting Outliers

The process of outlier detection should not be rushed. It requires a systematic approach that combines business context with statistical rigor.

3.1 Business Context First, Algorithms Second

Always start with understanding the business context. Jumping straight into statistical methods without understanding what the data means often results in misinterpretation. Outliers can sometimes represent significant business opportunities or risks, and dismissing them too quickly can lead to missed insights.

Example: Telecom Industry – Customer Churn Prediction

A telecom company uses machine learning models to predict customer churn. During the data analysis phase, several customers are identified as outliers because they’ve stayed with the company for more than 10 years without upgrading their plans. Initially flagged for exclusion, these outliers are later re-examined, leading to a new strategy targeting long-term customers with personalized offers. This outlier analysis helps the company improve customer retention rates for a valuable segment.

Step 4: Actionable Insights and Next Steps

Once outliers have been detected, the next step is to determine how to handle them. Not all outliers should be removed from the dataset. Some may represent valuable business insights that should be further investigated.

4.1 Handling Outliers

– Exclude: Remove outliers that are clearly errors or irrelevant to the analysis.

– Correct: Address outliers that are the result of data entry errors.

– Investigate: Dig deeper into outliers that could indicate new trends or opportunities.

Example: Supply Chain Management – Inventory Optimization

In a global supply chain, an outlier is detected where a particular warehouse reports consistently higher inventory levels than other locations. This outlier prompts further investigation, revealing inefficiencies in the warehouse’s reorder point settings. Correcting this issue leads to significant cost savings and improved inventory turnover.

Conclusion

Outlier detection is more than just applying statistical techniques—it’s about understanding the business context behind the data. By combining this understanding with the right analytical methods, businesses can uncover valuable insights, drive efficiencies, and mitigate risks. Whether detecting fraud in banking, optimizing inventory in supply chain management, or preventing equipment failures in manufacturing, outliers provide crucial information that can transform business outcomes.

What does generative AI mean for the future of organizations?

If you haven’t had the chance to read ‘Predicting the Generative AI Revolution Requires Learning from the Past’ by Winston Chang, Global Public Sector CTO at Snowflake, we will summarize and unpack Chang’s framework for technological revolutions, experiment with his predictions for the future, and identify the place in which you, and your organization can explore modernization in this era of near-constant change.

In summary, the article explores the evolution of technology in the business landscape, focusing on the GenAI revolution and its impact on modernizing companies. It emphasizes the importance of learning from past technological advancements to predict future trends accurately. By leveraging platforms like Snowflake’s GenAI, businesses can streamline operations, enhance decision-making processes, and stay competitive in today’s data-driven world.

As Chang states, “AI is out of the barn.”

Aligning framework to Logic Pursuits’ Client Case Study

Chang shares a macro-framework referencing how technology has created revolutions in the past. He then applies this to the future with AI/ML and postulates that in this case, data is the infrastructure, algorithms are the product, and the X factor is more widely accessible complex functions.

(Infrastructure + Products = Revolution in X).

Let’s explore this theory in real-world terms. Our B2C client faced challenges primarily within their Marketing function due to heavy reliance on third-party data sources, prompting a shift towards digital marketing strategies. They grappled with the absence of real-time data, limited flexibility, and excessive costs associated with data access.

Our initial focus was to establish a robust data access strategy, enabling in-house data management for enhanced flexibility, model building, and personalized marketing efforts. By implementing a Snowflake environment, we facilitated real-time access to data, personalized customer experiences, and reduced data access costs.

The infrastructure component represents the establishment of a robust data access strategy, focusing on in-house data management through platforms like Snowflake. This step addresses challenges such as the absence of real-time data, limited flexibility, and excessive costs associated with third-party data sources. By investing in modern data infrastructure, the client lays the foundation for future innovation and growth.

+

Integration of Snowflake’s GenAI feature further enhanced customer understanding and shopping behavior analysis.

The product’s component comes into play with the integration of advanced features like Snowflake’s GenAI, which enhances customer understanding and enables more sophisticated analysis of shopping behavior. This represents the product aspect of the equation, where technological tools and capabilities are leveraged to drive change and create value.

=

Addressing customer experience challenges, particularly regarding chargeback processes, necessitated housing Finance data within Snowflake, ensuring regulatory compliance such as Sarbanes–Oxley (SOX) and bolstering customer satisfaction. This journey towards data modernization not only improved operational efficiencies but also strengthened regulatory compliance, aligning with an overarching goal of achieving more with less while operating within a controlled environment. This represents the path toward establishing a contemporary data foundation, which in turn created avenues for integrating AI and Machine Learning, emphasizing the foundational role of data infrastructure to drive technological advancements.

The revolution, as per Chang’s equation, occurs when the infrastructure and products converge to address specific challenges and unlock new opportunities. In the case of our client, this revolution manifested in the transformation of their marketing and finance functions, leading to improved customer experiences, streamlined processes, and enhanced regulatory compliance.

The example demonstrates how a strategic focus on both infrastructure and products can lead to significant revolutions within organizations, enabling them to adapt to changing market dynamics, drive innovation, and achieve strategic objectives. Or in Chang’s terms, we call this Revolution in X.

Bridging the Gap

So, how do you become an organization that is an active participant in the AI revolution?

For starters, future-ready architecture does not tie to any specific industry; AI transcends industries. We believe focusing instead on the core business functions where it can have the greatest impact. Identifying the initial use case that presents the greatest challenge, compounded by the eagerness of everyone in the organization to be the first to adopt.

Our team at Logic Pursuits engages with leaders across a variety of business functions; Marketing and Sales, Finance, Accounting and Controllership, Procurement and Supply Chain operations, Risk & Compliance, Internal Audit along with collaborate closely with Enterprise Data Management leaders who facilitate dialogue between business unit stakeholders, ensuring alignment between business needs and technological capabilities. Our success lies in bridging the gap between business users and data professionals.

What’s in it for you?

As Chang mentions in his article, there’s a massive opportunity for GenAI and LLMs to have a positive impact on businesses and the world at large. Pushing for modernization and, in turn, giving more users access to powerful, easy-to-use models brings advanced analytics and AI into the hands of more people.

As noted by Chang, a significant opportunity awaits for GenAI to profoundly influence businesses and the global landscape. Advocating for modernization and consequently granting broader access to potent, user-friendly models democratizes advanced analytics and AI, placing these tools within reach of more individuals.

Rest assured, what may appear daunting and futuristic is well within reach. We’re here to assure you that the journey towards modernization within your organization is closer than you think. Remember, the journey to AI mastery doesn’t require an overnight transition from your current state. Essentially, backing GenAI leads to being ahead of the curve, igniting creativity, and transforming your organization. AI paves the way to unlocking new opportunities for you.

About Logic Pursuits

As a data consulting firm dedicated to empowering business leaders to achieve more with less, our mission is to assist you in surpassing revenue growth barriers. We specialize in identifying opportunities, offering expert guidance, and suggesting the most suitable tools, technologies, and cutting-edge techniques, such as LLMs, ML, and GenAI, to enhance and expedite your business objectives.

Join forces with us to leverage your current business data to its fullest potential and embark on the journey towards intelligent automation, robust data management and governance, and data visualization.

With Chang’s anticipated AI/ML revolution now imminent, it’s a good time to prepare for the data-driven era ahead. If the journey towards a future-ready architecture involves solving for X through infrastructure plus product, then the revolution lies within reach.

So, what’s stopping you from taking the next step forward?