## **Convert your intuition to evidence using Excel and Statistics**

In the world of product management and constant innovation, separating gut feelings from solid evidence is key to your success. This process is rooted in decision science, especially when it involves turning straightforward data into strategic decisions. It all begins with a hypothesis—a smart guess about how things might work. Before we dive deep into using statistics to test these hypotheses, let's first understand the basics. This is tailored for founders and product managers aiming to make more data-informed decisions in their roles.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FFgjp2vuEwjr8vOyTN32I%2F70f75bb7-a699-4c19-bbbc-3221c871d384.png?alt=media&token=3ded123b-03e3-4c0f-a2ee-dd0b4ea7cd63","id":"70f75bb7-a699-4c19-bbbc-3221c871d384","width":1920,"height":1080,"filename":"Frame 1.png","type":"image/png","caption":"","border":false}]}
```

## **What is a Product Hypothesis?**

A hypothesis in product development and product management is a **statement or assumption about the product, planned feature, market, or customer** (e.g., their needs, behavior, or expectations) **that you can put to the test, evaluate, and base your further decisions on**. It's crucial to note that a hypothesis arises from a position of limited knowledge and data, necessitating empirical testing to validate or refute your propositions.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FFgjp2vuEwjr8vOyTN32I%2F11842470-0fdc-4016-9d75-3ff9a93ec31d.png?alt=media&token=00b55f42-64fe-494c-8cf5-6d1df212ecba","id":"11842470-0fdc-4016-9d75-3ff9a93ec31d","width":1160,"height":2956,"filename":"Hypothesis Framework.png","type":"image/png","caption":"","border":false}]}
```

## **The Difference Between an Idea and a Hypothesis**

Understanding the distinction between an idea and a hypothesis is critical in the realm of product management.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FFgjp2vuEwjr8vOyTN32I%2Fb404fb50-b44e-43e3-972b-65a997626f58.png?alt=media&token=ab20a2f6-bd4e-4cfc-8f7e-b3c7ea124317","id":"b404fb50-b44e-43e3-972b-65a997626f58","width":1342,"height":720,"filename":"Screenshot 2024-03-11 at 09.30.21.png","type":"image/png","caption":"","border":false}]}
```

Armed with a clear understanding of what constitutes a hypothesis, let's delve into the nuanced practice of hypothesis testing, where statistical rigor meets product creativity.

Armed with a clear understanding of what constitutes a hypothesis, let's delve into the nuanced practice of hypothesis testing, where statistical rigor meets product creativity.

### **The Core of Hypothesis Testing**

Hypothesis testing is a statistical method employed to ascertain the likelihood that a hypothesis regarding a product feature or user experience holds true. This process begins with the formulation of two hypotheses: the null hypothesis Ho and the alternative hypothesis H1

\- **Null Hypothesis Ho**: There is no effect or difference.

\- **Alternative Hypothesis H1**: There is a significant effect or difference.

### **Defining Success Metrics**

Success metrics quantitatively delineate the parameters of success, directly correlating to your hypothesis. For instance, the click-through rate (CTR) serves as a metric when testing the impact of button color changes.

### **Crafting and Conducting the Experiment**

Experimental design is pivotal, outlining the methodology for comparing variations and observing their impact. A/B testing is a common approach, where users are randomly assigned to experience different variations.

### **The Statistical Backbone: The Statistical tests and Beyond**

There are various statistical tests which are available for different use cases

1\. **T-Test:** Compare means between two groups.

2\. **Paired T-Test:** Compare means within the same group at different points.

3\. **ANOVA (Analysis of Variance):** Compare means among more than two groups.

4\. **Regression Analysis:** Examine the relationship between dependent and independent variables.

5\. **Correlation Analysis:** Measure the strength and direction of the linear relationship between two variables.

6\. **Cointegration Test:** Test if two time series are cointegrated, indicating a long-term relationship.

7\. **Stationarity Test:** Check if a time series is stationary.

8\. **Chi-Square Test:** Test the independence of categorical variables.

9\. **Jarque-Bera Test:** Test if a set of data follows a normal distribution.

10\. **Bootstrap Test:** Estimate the sampling distribution of a statistic through resampling.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FFgjp2vuEwjr8vOyTN32I%2Fd1de1c9a-6b18-4e33-9255-773e5c00f751.png?alt=media&token=b1b51272-2704-4a2f-818b-ecfbb5277683","id":"d1de1c9a-6b18-4e33-9255-773e5c00f751","width":744,"height":325,"filename":"1710129329539.png","type":"image/png","caption":"","border":false}]}
```

## **Let's do a practical using our favourite tool "Excel"**

**Step 1: Define Your Hypothesis**

\- **Hypothesis**: Changing the "Contact Us" button color from green to blue will result in a higher click-through rate (CTR).

\- **Null Hypothesis**: Changing the "Contact Us" button color from green to blue will not result in a higher CTR.

This setup presumes that Variation B (blue button) is not inherently superior to Variation A (green button). The goal is to statistically prove whether Variation B enhances the CTR over Variation A.

**Step 2: Set Up Your Experiment**

\- Create two webpage versions: one with a green *"Contact Us"* button (Variation A) and another with a blue *"Contact Us"* button (Variation B).

\- Randomly divide website visitors into two groups to ensure unbiased exposure to each version.

\- Record the number of clicks on the *"Contact Us"* button for each group over a predetermined period, such as one week.

### Step 3: Get data

You can ask your data engineer or if you know a little about any tracking tool like GA4/ [Clarity](https://www.bing.com/webmasters/help/microsoft-clarity-55a30306)/ [Mixpanel](https://developer.mixpanel.com/reference/track-event) you can get this data from there if you've correctly configured the Tracker into your website/ tool

### Step 4:

After gathering data, we will perform t-test which is a powerful tool for evaluating the significance of differences between groups. The formula for the t-test is:

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FFgjp2vuEwjr8vOyTN32I%2F41e92648-7cdf-45ee-a569-4e2109040a31.png?alt=media&token=98a75073-d3ae-4497-b2f1-adfc07db699d","id":"41e92648-7cdf-45ee-a569-4e2109040a31","width":514,"height":95,"filename":"1710129329333.png","type":"image/png","caption":"","border":false}]}
```

* **t** is the t-statistic.

* **x1 bar** is the mean of the first sample.

* **x2 bar** is the mean of the second sample.

* **s1** is the variance of the first sample.

* **s2** is the variance of the second sample.

* **n1** is the number of observations in the first sample.

* **n2** is the number of observations in the second sample.

## The Data:

The data provides information on the performance of two variations of the website's "Contact Us" button (green and blue) during a specified date or time period.

1. **Date**: Time of data collection.

2. **Visitor Count**: Total users on the website during that time.

3. **GreenBtn\_Clicks\_Visitors:** Clicks on the green "Contact Us" button, showing engagement.

4. **BlueBtn\_Clicks\_Visitors:** Clicks on the blue "Contact Us" button, indicating engagement with the altered design.

5. **CTR (Click Through Rate)** = (Visitors with Button clicks (Green/ Blue))/Visitor count \*100

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FFgjp2vuEwjr8vOyTN32I%2F25d86340-00b1-4e3d-ad5b-e1df86fbb9f2.png?alt=media&token=a173cd34-c68f-4f1f-8d2b-db5a7d30988c","id":"25d86340-00b1-4e3d-ad5b-e1df86fbb9f2","width":1274,"height":866,"filename":"1710129329754.png","type":"image/png","caption":"","border":false}]}
```

Now we have to use excel's built in feature of data analysis for performing **"t-test for two sample assuming equal variance"** (check the below illustration)

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FFgjp2vuEwjr8vOyTN32I%2F568266ba-030a-459f-9fa4-aeb246a38172.png?alt=media&token=e7da1c06-5ad2-4323-a76b-e87c88c1731e","id":"568266ba-030a-459f-9fa4-aeb246a38172","width":965,"height":1000,"filename":"1710129330108.png","type":"image/png","caption":"","border":false}]}
```

The result will look something like this

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FFgjp2vuEwjr8vOyTN32I%2F1323da7b-099d-4cd4-be58-fb13af30b8ff.png?alt=media&token=601ded34-418c-4132-b170-62d60ccce4c0","id":"1323da7b-099d-4cd4-be58-fb13af30b8ff","width":744,"height":308,"filename":"1710129329832.png","type":"image/png","caption":"","border":false}]}
```

**In this P Value is 0.01455039 which is less than the significance level of 0.05**

Get the Excel sheet from here: <https://abhijit.objectstore.e2enetworks.net/hypothesis\_testing/website\_hypothesis\_testing.xls>

### Step-4: Identifying the insights

**How to read the results**

***The null hypothesis can be rejected, and it can be said that the change in button color had a statistically significant impact on the CTR if the p-value is less than the significance level (for example, 0.05). Thus, we cannot rule out the null hypothesis and conclude that the CTR was not significantly affected by the change in button color.***

This example shows that the test group's mean CTR is greater than the control group's (green button) and that the difference is statistically significant because the p-value from the T-Test is less than 0.05 with **95% Confidence Interval**. We can thus rule out the null hypothesis and conclude that the CTR was positively impacted by the **"Contact Us"** button's switch from green to blue.

So, the next time you find yourself scratching your head over a "unique" feature suggestion, you've got the perfect comeback: "Let’s see what the data says!" This approach transforms those "Hmm, are you sure?" moments into "Ah-ha! Let's test it" opportunities.

**Imagine this:** A stakeholder comes up with an idea that makes you wonder if they've been reading too much science fiction. Instead of a flat-out "No way," you can now say, "Interesting idea! Let's put it through our hypothesis testing process and see if the data backs it up." It's a win-win: you keep the peace, and who knows? Sometimes, the most out-there ideas turn out to be gold mines, as long as the data agrees.

Product Hypotheses testing using Excel

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FHoT1MCqYPEFqIOVmRM7P%2F40dee9f5-72fd-4d1f-81cb-d9f914546a87.jpg?alt=media&token=fbfff72e-9a67-47fa-b9cc-b1dbec3c5b05","id":"40dee9f5-72fd-4d1f-81cb-d9f914546a87","width":950,"height":633,"filename":"1710697875388.jpeg","type":"image/jpeg","caption":"","border":false}]}
```

The rapid advancement of Artificial Intelligence (AI) and the increasing size and capabilities of language models have created a pressing need for specialized hardware architectures designed specifically for AI workloads. Traditional GPUs, while effective for certain tasks, are no longer sufficient to keep up with the demands of cutting-edge AI applications.

Enter Groq and their groundbreaking Language Processing Unit (LPU). The LPU is a revolutionary computing architecture designed specifically for machine learning tasks. It offers performance gains that far surpass those of traditional GPUs, making it an ideal solution for the most demanding AI workloads.

Groq's Language Processing Unit (LPU) represents a paradigm shift in processor architecture, designed to revolutionize high-performance computing (HPC) and artificial intelligence (AI) workloads. This article will delve into the components, architecture, and workings of the LPU, highlighting its potential to transform the landscape of HPC and AI.

### Components and Architecture

The LPU's groundbreaking architecture consists of several key components:

**1. Processing Elements (PEs):** The LPU's processing power comes from its thousands of simple, identical processing elements (PEs). These PEs are organized in Single Instruction, Multiple Data (SIMD) arrays, enabling them to execute the same instruction on different data points concurrently.

**2. Centralized Control Unit (CU):** At the heart of the LPU lies a centralized control unit (CU) responsible for issuing instructions to the PEs and managing the flow of data and instructions. The CU ensures seamless communication between the PEs and memory hierarchy.

**3. Memory Hierarchy:** The LPU features a hierarchical memory structure, including a large on-chip SRAM, a high-bandwidth off-chip memory interface, and a sophisticated cache hierarchy. This memory hierarchy is optimized for high-bandwidth, low-latency data access.

**4. Network-on-Chip (NoC):** A high-bandwidth Network-on-Chip (NoC) interconnects the PEs, the CU, and the memory hierarchy. The NoC enables fast, efficient communication between different components of the LPU.

**5. Vector Processing:** The LPU supports vector processing, allowing it to execute multiple operations on large data sets simultaneously.

### How Groq's LPU Works

The LPU's unique architecture enables it to outperform traditional CPUs and GPUs in HPC and AI workloads. Here's a step-by-step breakdown of how the LPU works:

**1. Data Input:** Data is fed into the LPU, triggering the Centralized Control Unit to issue instructions to the Processing Elements (PEs).

**2. Massively Parallel Processing:** The PEs, organized in SIMD arrays, execute the same instruction on different data points concurrently, resulting in massively parallel processing.

**3. High-Bandwidth Memory Hierarchy:** The LPU's memory hierarchy, including on-chip SRAM and off-chip memory, ensures high-bandwidth, low-latency data access.

**4. Centralized Control Unit:** The Centralized Control Unit manages the flow of data and instructions, coordinating the execution of thousands of operations in a single clock cycle.

**5. Network-on-Chip (NoC):** A high-bandwidth Network-on-Chip (NoC) interconnects the PEs, the CU, and the memory hierarchy, enabling fast, efficient communication between different components of the LPU.

**6. Processing Elements:** The Processing Elements consist of Arithmetic Logic Units, Vector Units, and Scalar Units, executing operations on large data sets simultaneously.

**7. Data Output:** The LPU outputs data based on the computations performed by the Processing Elements.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FHoT1MCqYPEFqIOVmRM7P%2Fdfe6007f-46d4-414a-a5e6-39300f1db6c9.png?alt=media&token=4547d3da-3e7d-4f7f-9d97-7dacd057e68b","id":"dfe6007f-46d4-414a-a5e6-39300f1db6c9","width":1047,"height":1000,"filename":"Groq.png","type":"image/png","caption":"","border":false}]}
```

### Revolutionizing HPC and AI

The LPU's massively parallel architecture and high-bandwidth memory hierarchy enable it to excel in applications such as:

1\. Artificial Intelligence and Machine Learning: The LPU's ability to execute thousands of operations simultaneously makes it an ideal candidate for AI and ML applications, including deep learning, natural language processing, and computer vision tasks.

2\. High-Performance Computing: The LPU's scalable and flexible architecture allows it to tackle complex HPC workloads, such as scientific simulations, climate modeling, and molecular dynamics.

3\. Edge Computing: The LPU's energy-efficient design and compact form factor make it an attractive solution for edge computing devices, where power and space constraints are paramount.

### How LPU is different from GPU

**1. Architecture:**

```
- LPU: An LPU is designed specifically for natural language processing tasks, with a multi-stage pipeline that includes tokenization, parsing, semantic analysis, feature extraction, machine learning models, and inference/prediction.

- GPU: A GPU has a more complex architecture, consisting of multiple streaming multiprocessors (SMs) or compute units, each containing multiple CUDA cores or stream processors.
```

**2. Instruction Set:**

```
- LPU: The LPU's instruction set is optimized for natural language processing tasks, with support for tokenization, parsing, semantic analysis, and feature extraction.

- GPU: A GPU has a more general-purpose instruction set, designed for high-throughput, high-bandwidth data processing.
```

**3. Memory Hierarchy:**

```
- LPU: The LPU's memory hierarchy is optimized for natural language processing tasks, with a focus on efficient data access and processing.

- GPU: A GPU has a more complex memory hierarchy, including registers, shared memory, L1/L2 caches, and off-chip memory. The memory hierarchy in GPUs is designed for high-throughput, high-bandwidth data access, but may have higher latency compared to the LPU for specific NLP tasks.
```

**4. Power Efficiency and Performance:**

```
- LPU: The LPU is designed for high power efficiency and performance, with a focus on natural language processing tasks. It can deliver superior performance per watt compared to GPUs for specific NLP workloads.

- GPU: GPUs are designed for high throughput and performance, particularly for graphics rendering and parallel computations. However, they may consume more power than an LPU for the same NLP workload due to their more complex architecture and larger number of processing units.
```

**5. Applications:**

```
- LPU: The LPU is well-suited for natural language processing tasks, such as tokenization, parsing, semantic analysis, feature extraction, and machine learning model inference.

- GPU: GPUs are widely used in applications such as gaming, computer-aided design (CAD), scientific simulations, and machine learning. However, they are not optimized for natural language processing tasks, and an LPU would generally provide better performance and power efficiency for such tasks.
```

In summary, the LPU and GPU have different architectural designs and use cases. The LPU is designed specifically for natural language processing tasks, while GPUs are designed for high-throughput, high-bandwidth data processing, particularly for graphics rendering and parallel computations. The LPU offers a more streamlined, power-efficient architecture for natural language processing tasks, while GPUs provide a more complex, feature-rich architecture for a broader range of applications.

Groq's LPU: A Revolutionary Leap in Processing for High-Performance Computing and AI

**Executive Summary**

A prominent law firm sought a centralized repository of partner information, aggregated from multiple internal and external websites. Our team delivered a fully automated solution that:

1\. **Extracted Data** with a Python 3.9 + BeautifulSoup 4 web crawler, enriched with libraries like Requests and Selenium for dynamic content.

2\. **Transformed and Loaded** the data into a secure, scalable AWS-based infrastructure (S3, Glue, Redshift) via an automated CI/CD pipeline.

3\. **Built a Retrieval-Augmented Generation (RAG)** architecture leveraging LangChain, with vector embeddings stored in MongoDB’s VectorStore for advanced semantic querying.

This end-to-end system consolidated scattered partner data, unlocked complex NLP-driven queries, and significantly accelerated the firm’s research capabilities.

## **1. Business Context and Challenges**

### **1.1 Fragmented Data Ecosystem**

• **Diverse Sources**: Partner profiles resided on the firm’s site, industry directories, and legal ranking websites (e.g., Chambers, LexisNexis).

• **Low Data Quality**: Reliance on manual copying/pasting and ad-hoc spreadsheets increased data inconsistency and errors.

• **Complex Inquiries**: Existing search tools were limited to keyword matching, making it difficult to answer deeper queries like “Which partner has negotiated the largest arbitration in the energy sector?”

### **1.2 Strategic Objectives**

• **Centralized Data**: Consolidate partner information into a single “source of truth.”

• **Automated Pipeline**: Eliminate manual overhead through robust data ingestion, transformation, and storage processes.

• **Intelligent Search**: Leverage modern NLP (through Large Language Models) to enable advanced retrieval and summarization.

## **2. Technical Approach and Methodology**

### **2.1 Data Collection with Python 3.9, BeautifulSoup 4, and Selenium**

**2.1.1 Crawler Design**

• **Libraries and Frameworks**:

• **Requests** for straightforward GET/POST calls.

• **BeautifulSoup 4** to parse static HTML pages.

• **Selenium** for dynamic content extraction where JavaScript-driven elements are present (e.g., partner bios behind AJAX calls).

• **Modular Architecture**:

• **Config Files**: Each target website had a dedicated config specifying URL patterns, HTML selectors, and pagination rules.

• **Error Handling**: Implemented Python try-except blocks to handle broken links, timeouts, and captchas.

### **2.1.2 JSON Conversion**

• **Schema Definition**: Leveraged Python’s dataclasses or pydantic to enforce a consistent schema (e.g., name, position, specialty, bio, etc.).

• **Data Validation**: Used built-in validators to ensure mandatory fields (e.g., email, phone number) are not null and follow standard formats.

• **Output Storage**: Exported validated data to JSON, then stored locally before upload to AWS S3.

### **2.2 ETL Pipeline with AWS and Supporting Tools**

**2.2.1 Data Ingestion into Amazon S3**

• **S3 Bucket Configuration**:

• **Versioning** enabled for audit trails.

• **Lifecycle Policies** for transitions to infrequent access and archiving.

• **Security**: Employed AWS KMS for server-side encryption (SSE-KMS), restricting keys through IAM roles.

### **2.2.2 AWS Glue for Transformation**

• **Glue Crawlers**: Automated schema discovery for JSON, generating a Glue Data Catalog.

• **Glue ETL Jobs**: Written in PySpark to perform:

• **Cleansing**: Standardize date formats, unify naming conventions (e.g., “Senior Partner” vs. “Sr. Partner”).

• **Enrichment**: Cross-referenced external data (e.g., Chambers ranking) to add partner accolades.

• **Job Orchestration**: AWS Step Functions or Apache Airflow (running on Amazon MWAA) for scheduling and dependency management.

### **2.2.3 Loading into Amazon Redshift**

• **Redshift Provisioning**:

• **Cluster Type**: dc2.large or ra3 nodes for scalable compute and storage.

• **Subnet Groups** in a private VPC for secure data access.

• **Data Warehouse Schema**:

• **Star Schema**: Fact tables (e.g., “Partner\_Engagements”) referencing dimensional tables (“Dim\_PartnerInfo,” “Dim\_Specialty”).

• **Metadata Tracking**: Columns for last\_updated, source\_url for auditing.

• **Performance Optimization**:

• **Sort Keys and Dist Keys** to minimize data movement for heavy queries.

• **Column Encoding** chosen by Redshift’s ANALYZE COMPRESSION function.

### **2.3 CI/CD and DevOps**

• **Version Control**: GitLab for code repositories (crawler scripts, Glue ETL jobs, and Redshift schema definitions).

• **Continuous Integration**: GitLab CI jobs running lint checks (Flake8, Black) and unit tests.

• **Docker Containerization**: Packaged the crawler code into Docker images stored on AWS ECR for reproducible runs.

• **Infrastructure as Code**: Deployed S3 buckets, Glue jobs, and Redshift clusters via AWS CloudFormation or Terraform.

### **2.4 Building an NLP-Powered Search Engine (RAG) with LangChain**

**2.4.1 Vector Embedding Generation**

• **LangChain Integration**:

• **Language Model**: Hugging Face Transformers (e.g., sentence-transformers/all-MiniLM-L6-v2) for embedding generation.

• **Embedding Pipelines**: Batch processed partner bios and specialized legal documents.

• **Data Flow**:

1\. Query Redshift for partner data.

2\. Use LangChain to transform text fields into vector embeddings.

3\. Store embeddings in MongoDB’s VectorStore.

### **2.4.2 MongoDB VectorStore**

• **Schema**:

• \_id: Unique partner identifier or doc ID.

• embedding: High-dimensional float array.

• metadata: Additional fields (e.g., partner name, practice area).

• **Similarity Search**:

• **Cosine Similarity**: Implemented within the VectorStore to find nearest neighbors in embedding space.

• **Indexing**: Utilized MongoDB Atlas Search for indexing vectors and accelerating queries.

### **2.4.3 RAG Query Flow**

1\. **User Query**: Input captured via a React/Next.js web front-end or a chatbot interface built in Streamlit.

2\. **LangChain Orchestration**:

• **Retrieval**: Identifies top-k relevant partner embeddings from MongoDB.

• **Contextual Assembly**: Aggregates relevant partner details.

• **LLM Response**: Passes context to the LLM (OpenAI GPT-4 or local LLM) to generate a final, succinct answer.

3\. **Answer Delivery**: Renders a final text response along with relevant partner profiles.

## **3. Advanced Architecture Diagram** 

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FLXigo2EazpxHDf4B5sAs%2Fcdf64db4-910c-4673-ade8-4ada1fee29f3.png?alt=media&token=ac4c165d-d4ca-44b7-8271-7921b2bf5e3a","id":"cdf64db4-910c-4673-ade8-4ada1fee29f3","width":1583,"height":3840,"filename":"Untitled diagram-2024-12-26-124544.png","type":"image/png","caption":"","border":false}]}
```

1\. **Crawler** uses Python + Selenium to handle dynamic sites.

2\. **S3** houses raw JSON data, secured and versioned.

3\. **Glue** orchestrates ETL: auto-discovery, cleaning, and loading.

4\. **Redshift** acts as the central data warehouse for analytics.

5\. **LangChain + Embeddings** pipeline transforms partner text data into vector forms.

6\. **MongoDB VectorStore** enables fast semantic search.

7\. **LangChain RAG** provides context-aware retrieval and LLM-driven answers.

8\. **UI/Chatbot** is the front-end portal for end-users.

### **4. Results and Key Metrics**

1\. **Data Quality Improvement**: Null and duplicate records reduced by **90%** through schema validation and automated pipeline checks.

2\. **Operational Efficiency**: The law firm saved an estimated **800 person-hours** per quarter by retiring manual data collection processes.

3\. **Enhanced Query Capabilities**: Complex, multi-faceted queries (e.g., “List partners with arbitration experience in the APAC region”) are now answered in **under 2 seconds** via the RAG-based approach.

4\. **Scalability**: The combination of AWS Glue and Redshift ensures easy horizontal scaling, accommodating thousands of new partner records and documents per day.

Law Firm Partner Intelligence Platform: End-to-End State Diagram

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FSnq8kfXcSqiXMtS02aBz%2F6c9c2738-daf2-440d-b22b-a698227a7d29.png?alt=media&token=9e3d33a7-9119-474f-bf3a-7166e8c2e9c7","id":"6c9c2738-daf2-440d-b22b-a698227a7d29","width":1910,"height":1156,"filename":"Screenshot 2024-05-10 at 23.23.50.png","type":"image/png","caption":"","border":false}]}
```

## **Statistical Arbitrage: A Dive into Quantitative Finance Strategies**

Statistical arbitrage emerges as a powerhouse in the trading world, leveraging mathematical finesse and high-speed technology to tap into the subtle inefficiencies between financial instruments. It’s not just about buying low and selling high; it's an orchestral play of numbers, predictions, and precision. Here's how it unfolds in the labyrinth of the financial markets.

### **The Essence of Statistical Arbitrage**

At its core, statistical arbitrage is about playing the symphony of numbers across different instruments. This strategy is broadly split into two main approaches:

* **Directional Trading**: This straightforward method focuses on the movement of a single financial instrument. Simple yet risky, it's akin to betting all your chips on one number at the roulette table.

* **Pairs and Cointegration Trading**: Here lies the art of balancing. Traders pick pairs or triplets of assets whose values historically move in sync. It's like a dance between assets, where if one stumbles, the others are expected to follow.

### **The Science of Stationarity and Its Role in Trading**

Stationarity is the backbone of statistical arbitrage. For a process to be stationary, its mean, variance, and covariance need to remain constant over time. Unlike a random walk, which wanders aimlessly, stationary processes have predictable patterns, making them ideal candidates for trading strategies like mean reversion.

### **Tools of the Trade: Decoding Stationarity**

1. **Augmented Dickey Fuller Test (ADF)**: This statistical test is like a litmus test for predictability in financial data. It helps determine whether a price series is more likely to revert to a mean or continue on a wild, unpredictable path.

2. **Hurst Exponent**: This is your compass in the realm of financial time series. A Hurst exponent above 0.5 signals a trending market, while a value below 0.5 indicates a potential for mean reversion.

### **Crafting a Mean Reversion Strategy on Stationary Series**

When two assets have danced together historically, mean reversion strategy helps predict when they'll return to their synchronized steps after a misstep. Here’s how it works:

1. **Calculate the Hedge Ratio**: This involves using Ordinary Least Squares (OLS) to determine how much of one asset (say, stock A) you should hold against another (stock B) to balance their differing volatilities.

2. **Creating the Spread**: Using the first 90 days' data, traders create a 'spread' between the two assets, ensuring no look-ahead bias, a tricky but vital part of the strategy.

3. **Testing for Cointegration**: Finally, an ADF test on the spread helps confirm if the assets are indeed moving together over the long term, justifying the use of mean reversion.

### **Putting Theory into Practice: A Step-by-Step Guide**

#### **Implementing the Hurst Exponent Test**

1. **Gather Data**: Collect the sequential data for analysis.

2. **Prepare and Preprocess**: Ensure your data is clean and formatted for time series analysis.

3. **Divide and Conquer**: Break the data into smaller series of varying lengths.

4. **Calculate and Plot**: Calculate the R/S statistic for each series and plot these on a log-log scale.

5. **Fit and Calculate**: Fit a line through the plot to estimate the Hurst Exponent.

6. **Analyze and Interpret**: A Hurst Exponent above 0.5 suggests persistence; below 0.5 suggests anti-persistence.

#### **Conducting an Augmented Dickey Fuller Test**

1. **Understand the Data's Nature**: Knowing whether your data is predictable (stationary) or not (non-stationary) is crucial.

2. **Set Up Hypotheses**: Null hypothesis assumes non-stationarity; the alternative suggests stationarity.

3. **Perform the Test**: Use the ADF statistic to challenge the null hypothesis.

4. **Decision Time**: Compare the statistic with critical values to decide the nature of your data.

5. **Interpret Results**: A rejection of the null hypothesis confirms stationarity, paving the way for predictive strategies.

Statistical arbitrage isn't just a strategy; it's a high-stakes game of precision and prediction. By understanding the nuances of stationarity, employing rigorous tests like the ADF and Hurst Exponent, and meticulously crafting trading strategies based on these insights, traders can harness the power of quantitative finance to not just participate in the market but to anticipate its moves. Remember, in the world of trading, being forearmed with data is being forewarned about opportunities.

Check the code implementation here \
https://github.com/abhi647/statistical-arbitrage

Statistical Arbitrage: A Dive into Quantitative Finance Strategies

## **Maximizing Revenue with RMF Analysis: A Salesforce and Einstein Case Study**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FTQxOymwmambcqmxJZECL%2Fa1523dc5-e682-4227-b526-6c9ef4551aba.png?alt=media&token=33a322f5-69d6-4fce-b61d-be2bc92d8b02","id":"a1523dc5-e682-4227-b526-6c9ef4551aba","width":1439,"height":1344,"filename":"RMF.png","type":"image/png","caption":"","border":false}]}
```

### **Background**

ABC Apparel is an e-commerce company that specializes in selling clothing and accessories online. The company operates through its online platform and uses Salesforce as its primary CRM system to manage customer data and sales activities. I was hired as a data analyst to conduct a recency, frequency, monetary (RFM) analysis to segment ABC Apparel's customer base and gain insights into their purchase behaviors.

### **Objective**

The main objective of the project was to identify ABC Apparel's most valuable customers and improve the company's customer retention and acquisition strategies. By conducting an RFM analysis, I aimed to segment the customer base based on purchase behaviors and identify patterns and trends to gain insights into customer preferences and behavior.

### **Solution**

The RMF analysis was conducted in several phases, including the following:Categorization: The first step was to identify the e-commerce company's assets and categorize them based on their importance and impact on the business. The assets included the Salesforce CRM system, Einstein for AI-powered analytics, and Tableau for data visualization.Threat identification: The second step was to identify the potential cybersecurity threats that could affect the company's assets. The threats identified included data breaches, malware infections, phishing attacks, and insider threats.Risk assessment: The third step was to assess the risks associated with each threat identified in the previous step. The assessment considered the likelihood of the threat occurring, the impact it would have on the business, and the existing controls in place to mitigate the risk.Risk mitigation: Based on the results of the risk assessment, the fourth step was to develop a risk mitigation plan. This plan included recommendations for implementing additional controls to reduce the likelihood and impact of the identified risks.

### **Methodology**

To carry out the RFM analysis, I gathered customer information, purchase history, and order details from Salesforce and Einstein. I used Tableau as my data visualization tool to create interactive dashboards to visualize and analyze the data. Using the RFM model, I segmented ABC Apparel's customer base into four groups based on their purchase behaviors:

1. **High-Value Customers:** Customers who made purchases recently, frequently, and spent a significant amount of money on each order.

2. **Mid-Value Customers:** Customers who made purchases within the last six months, spent an average amount of money on each order, and made purchases at a moderate frequency.

3. **Low-Value Customers:** Customers who made purchases over six months ago, spent a small amount of money on each order, and made infrequent purchases.

4. **Lost Customers:** Customers who have not made any purchases in the last year.

### **Results**

1. The data lake and reporting architecture have enabled Boeing to streamline data access, improve data quality, and gain insights into the product development process. The benefits include:

2. **Improved data quality:** The centralized data lake ensures consistent and accurate data across the enterprise. The data quality is improved by identifying and fixing data issues during the data transformation process.

3. **Faster data access:** The data lake enables faster access to data by reducing data latency and improving query performance. The data can be queried using different reporting tools, and the results are returned in seconds.

4. **Cost savings:** The data lake and reporting architecture have reduced the cost of maintaining and managing the CAD and PLM data. The cloud-based architecture provides a scalable and cost-effective solution that reduces the need for on-premise infrastructure.

5. Insights into the product development process: The data analytics capabilities have enabled Boeing to gain insights into the product development process. The analytics includes trend analysis, root cause analysis, and predictive modeling. These insights help the company to improve its design, manufacturing, and testing processes.

### **Conclusion**

The RFM analysis provided ABC Apparel with valuable insights into its customer base's purchase behaviors and helped the company identify its most valuable customers. By focusing on its High-Value Customers, ABC Apparel was able to improve customer retention and acquisition strategies and increase its revenue. The success of the project demonstrated the importance of data analysis in business decision-making and showcased my skills as a data analyst.

Maximizing Revenue with RMF Analysis: A Salesforce and Einstein Case Study

**Problem Statement: A leading online retailer faces a sudden dip in profitability for a $999 product at the beginning of 2022, despite a consistent trend in 2021. Our analysis aims to uncover the reasons behind this decline.**

**Key data points include:**

1. Shopping Event (Binary): Indicates special sales events.

2. Ad Spend (Numeric): Investment in advertising campaigns.

3. Page Views (Numeric): Visits to the product detail page.

4. Unit Price (Numeric): Price, factoring in temporary discounts.

5. Sold Units (Numeric): Quantity of units sold.

6. Revenue (Numeric): Daily sales revenue.

7. Operational Cost (Numeric): Daily operational expenses.

8. Profit (Numeric): Daily net profit.

### **Review Data:**

* Shape: 455 rows and 9 columns

* Null Values: There is no null value in the dataset

* Columns : By unique values we came to know that there is one categorical column ie: Shopping Event? rest of the columns have continuous data

* Data Type : Date and shopping events are object and boolean data types which can not be fed to the learning algorithm so they are required to be transformed.

* head: date column needs to be divided into day, month and year.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FWrxyEZ7RVg7iDSZ57Wb2%2F426d993f-e40b-4b3a-a5e3-7ddc54a93e86.png?alt=media&token=e652c6d4-014c-4415-a597-814dbffd69f2","id":"426d993f-e40b-4b3a-a5e3-7ddc54a93e86","width":1200,"height":361,"filename":"Screenshot 2024-08-23 at 10.06.52.png","type":"image/png","caption":"","border":false}]}
```

### **Exploratory Data analytics**

**In exploratory data analytics we are expected to know reason behind the dip in 2022**

**To visualize the data against the date we have extracted the year and month from the date field and also changed the data type of the date column from object to datetime by using pandas to\_datetime method and extracted the year and month by using pandas DatetimeIndex method.**

#### **Profit Vs Date:**

By visualizing profit viruses date we can see if there is actually a dip in profit in 2022 and visualize the strength as well. We will be using the seaborn library to visualize the data.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FWrxyEZ7RVg7iDSZ57Wb2%2Fc072619e-ed3e-4d08-b99e-f81c638a654c.png?alt=media&token=79e448ca-d1d9-4f40-bde1-cd18be2473fc","id":"c072619e-ed3e-4d08-b99e-f81c638a654c","width":1217,"height":752,"filename":"Screenshot 2024-08-23 at 10.07.41.png","type":"image/png","caption":"","border":false}]}
```

**In plot 2 we can clearly see the dip in profit as the year passes. By plot 4 we can see that there is a churn in the customer as compared to fiscal year 2021.**

**Continuous Variables Vs Profit**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FWrxyEZ7RVg7iDSZ57Wb2%2F4a0966e9-4fd1-44be-b0ff-193a3c273575.png?alt=media&token=cb8ce31e-8cc7-42f9-937f-82cfd3456ef5","id":"4a0966e9-4fd1-44be-b0ff-193a3c273575","width":1153,"height":578,"filename":"Screenshot 2024-08-23 at 10.08.18.png","type":"image/png","caption":"","border":false}]}
```

### **Categorical variable Vs Profit**

**I have used a violin graph to visualize the categorical feature ie: *shopping event? Vs profit,* i have also marked the divergence in the graph as well.**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FWrxyEZ7RVg7iDSZ57Wb2%2Fb19ba143-9e81-453a-95d3-366b26cbaa66.png?alt=media&token=8f8d6020-edb3-4d09-8d15-95b7e56a337e","id":"b19ba143-9e81-453a-95d3-366b26cbaa66","width":1010,"height":664,"filename":"Screenshot 2024-08-23 at 10.08.59.png","type":"image/png","caption":"","border":false}]}
```

**Here the divergence the the year 2021 to 2022 is not very large though the causal effect might be more that we will calculate further in this document.**

### **Date Vs Categorical & Continuous Variables**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FWrxyEZ7RVg7iDSZ57Wb2%2Faa39375d-6590-4b9d-93b6-bb7530dd285d.png?alt=media&token=23acb794-e1bc-4fc9-9e22-bf2fabc2c214","id":"aa39375d-6590-4b9d-93b6-bb7530dd285d","width":1208,"height":612,"filename":"Screenshot 2024-08-23 at 10.09.29.png","type":"image/png","caption":"","border":false}]}
```

* **Plot 1: Avg. 'Ad spend' is relatively the same for both the year but there is more in July 2022 which increases revenue for the same month of the same year.**

* **Plot 2: Avg. 'Page view' is very low in 2022 which might be the main culprit behind the decrease in profit.**

* **Plot 3: Surprisingly avg unit sold per indices per month is more in 2022 dispite low profit**

* **Plot 5: Operational cost is high is 2022**

* **Plot 6: unit price is very low in 2022**

#### **Correlation map**

1. **Pearson correlation defines that there is a strong relation between all the  variables, but we are interested to see an exclusive relationships so we can telly kendall tau and spearman correlation which can give the direction of the relationship as well.**

2. **By Concluding kendall tau and spearman correlation we noticed below relations 2.1 page views and profit have positive relation**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FWrxyEZ7RVg7iDSZ57Wb2%2Fba503b63-6f62-49fe-9bf5-61a341db04f9.png?alt=media&token=a81f6a95-9f8f-4ad1-bcbf-e86ccb223b9c","id":"ba503b63-6f62-49fe-9bf5-61a341db04f9","width":1213,"height":1132,"filename":"Screenshot 2024-08-23 at 10.11.31.png","type":"image/png","caption":"","border":false}]}
```

#### **Conclusion for EDA**

**We can conclude from above EDA that page views have a definite correlation that might be affecting the profit in 2022. We also saw strong correlation for page view and profit in spearman correlation**

\
**Causal Inference mathematical approach**

**It is the field of data science that aims to quantify the cause and effect relationship between the variables.**

I am calculating causal effect by mean of ITE ie: individual term frequency. 

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FWrxyEZ7RVg7iDSZ57Wb2%2Fcf900f41-7df4-4601-bbfc-5c2ea6f7da48.png?alt=media&token=26c13da8-3f58-4b87-812b-81cba2222dcc","id":"cf900f41-7df4-4601-bbfc-5c2ea6f7da48","width":988,"height":430,"filename":"Screenshot 2024-08-23 at 10.12.39.png","type":"image/png","caption":"","border":false}]}
```

**Ref. Article: [Recent Developments in Causal Inference and Machine Learning](https://www.degruyter.com/document/doi/10.1515/jci-2021-0025/html?lang=en)** 

**I am using pandas dataframe for loading data and and numpy for calculation let's look at the code snippet**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FWrxyEZ7RVg7iDSZ57Wb2%2F304f1053-f72c-4431-bfec-842aca2988c7.png?alt=media&token=7c98f815-30e6-4ac5-a7dd-a6712b132cb4","id":"304f1053-f72c-4431-bfec-842aca2988c7","width":1025,"height":641,"filename":"Screenshot 2024-08-23 at 10.13.26.png","type":"image/png","caption":"","border":false}]}
```

**By using the seaborn library i am plotting bar graph for causal effect of all the columns that we have calculated using above calculation.**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FWrxyEZ7RVg7iDSZ57Wb2%2F8a285aa8-516f-4313-8cb7-e4a503610022.png?alt=media&token=23172235-3225-4f57-993a-0ff13896c1d7","id":"8a285aa8-516f-4313-8cb7-e4a503610022","width":763,"height":580,"filename":"Screenshot 2024-08-23 at 10.13.48.png","type":"image/png","caption":"","border":false}]}
```

**causal effect for shopping event is $12,68,282. which means when the shopping  event is true it causes the profit to increase by $12,68,282. So the treatment is  associated with increase in profit by coefficient 1268282.**

### **Root Cause Analysis using mathematical approach**

**I am using Kullback-Leibler Divergence for calculating the divergence between the data for fiscal year 2021 and 2022.**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FWrxyEZ7RVg7iDSZ57Wb2%2Ff9dc35a4-3129-4b7c-b7d8-d8aa63e7a397.png?alt=media&token=cac4ebe5-7e50-4918-bab1-20f8d39d4f60","id":"f9dc35a4-3129-4b7c-b7d8-d8aa63e7a397","width":967,"height":389,"filename":"Screenshot 2024-08-23 at 10.14.42.png","type":"image/png","caption":"","border":false}]}
```

### **Output**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FWrxyEZ7RVg7iDSZ57Wb2%2F76983b3a-65c2-420d-ba7b-57c40cc83ec8.png?alt=media&token=52156ea5-5fd8-4bd8-bbaf-1a88c833c015","id":"76983b3a-65c2-420d-ba7b-57c40cc83ec8","width":893,"height":450,"filename":"Screenshot 2024-08-23 at 10.15.09.png","type":"image/png","caption":"","border":false}]}
```

**By using KL divergence we found top three root factors ie: Operational cost, Revenue, Page Views.**

### **Causal Effect and Root Cause analysis using Dowhy**

Dowhy is an open source Python library that aims to spark causal thinking and analysis. DoWhy provides a principled four-step interface for causal inference that focuses on explicitly modeling causal assumptions and validating them as much as possible.

In Dowhy the first step is to provide a causal relationship to the dowhy model. We have to provide the causal relation by our own experience. We have to pack the relation in networksx object container and then pass it to the model. Let's look at the code snippet.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FWrxyEZ7RVg7iDSZ57Wb2%2F36b3f534-8213-4cd8-9194-a195df0ae709.png?alt=media&token=aee5157c-7733-418d-a9e3-982e46172381","id":"36b3f534-8213-4cd8-9194-a195df0ae709","width":548,"height":359,"filename":"Screenshot 2024-08-23 at 10.15.40.png","type":"image/png","caption":"","border":false}]}
```

**We can also plot the relation as well.**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FWrxyEZ7RVg7iDSZ57Wb2%2F21cc6adc-cb43-45b3-afcd-678a64e612a4.png?alt=media&token=64780958-2bc3-4ebe-b2ed-467ed491bc4d","id":"21cc6adc-cb43-45b3-afcd-678a64e612a4","width":969,"height":658,"filename":"Screenshot 2024-08-23 at 10.16.07.png","type":"image/png","caption":"","border":false}]}
```

**Next step is to fit the data into the model.**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FWrxyEZ7RVg7iDSZ57Wb2%2F14dbbfff-e38b-42bc-8483-25b7a2e42cf5.png?alt=media&token=4f1baf71-d89a-433c-9caa-34c40f0dd04e","id":"14dbbfff-e38b-42bc-8483-25b7a2e42cf5","width":1065,"height":151,"filename":"Screenshot 2024-08-23 at 10.16.29.png","type":"image/png","caption":"","border":false}]}
```

### **Causal effect using Dowhy**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FWrxyEZ7RVg7iDSZ57Wb2%2Ff1a0916e-12f0-4c9c-a269-a8420a914332.png?alt=media&token=25b42045-3626-4af1-9a02-55ff47e7079a","id":"f1a0916e-12f0-4c9c-a269-a8420a914332","width":1085,"height":680,"filename":"Screenshot 2024-08-23 at 10.17.05.png","type":"image/png","caption":"","border":false}]}
```

#### **Root Cause Analysis using Dowhy**

**For calculating the root cause we use attribute\_anomalies method of Dowhy. This require an input of a causal graph which we had already added while finding causal effect.**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FWrxyEZ7RVg7iDSZ57Wb2%2F9e12180c-c907-4452-bce9-5126c825ef84.png?alt=media&token=2be08a37-6e9b-4127-b338-995f72cf23b9","id":"9e12180c-c907-4452-bce9-5126c825ef84","width":1015,"height":132,"filename":"Screenshot 2024-08-23 at 10.17.34.png","type":"image/png","caption":"","border":false}]}
```

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FWrxyEZ7RVg7iDSZ57Wb2%2F2995ffe1-3348-4ec7-a7fd-b151d9e0c5a4.png?alt=media&token=a9d8feea-aadc-4741-b566-7afd53296ffd","id":"2995ffe1-3348-4ec7-a7fd-b151d9e0c5a4","width":1040,"height":779,"filename":"Screenshot 2024-08-23 at 10.17.51.png","type":"image/png","caption":"","border":false}]}
```

### **Conclusion**

**In this article, we have  compared root cause analysis and causal inference by classical approach, mathematical approach and by using Dowhy library. In the classical approach we have using exploratory data analytics, in the mathematical approach we computed causal effect and divergence/ root cause using mathematical formulas, then we used dowhy open source library to compute the same ie: causal effect and root cause.** 

**The classical and mathematical approach give better understanding of the data whereas in dowhy it is very convenient to find the result but in dowhy we are not able to understand the backend calculation. Dowhy is a bit more computational whereas the mathematical approach is simple and less computational**

### **Potential Solution**

The decline in page views led to a drop in sales, and the decline in unit pricing had a detrimental effect on revenue as well. Let's talk about some of the possible fixes that may be put into practice for increasing page views.

1. Optimize your website for organic search: Search is the top way both individuals and businesses research new products and services. This means it’s critical for you to make sure search engines find your website and bring it to the attention of the right people through search engine optimization (SEO). 

2. Invest in paid search

3. Engage in social channels

4. Work with influencers: Find social influencers and bloggers who have sizable audiences in your target demographic. Posts shared by influencers can help boost awareness and your SEO value if they feature your products in an authentic way. 

5. Write blogs or articles: Publishing original content, either on your own blog or on industry websites, can position you as a thought leader. 

6. Drive awareness with public relations (PR): There are plenty of ways to do PR on a budget, either on your own or with the help of a small agency or freelancer. Local publications and websites are always looking for interesting stories and contacting their editors with a pitch can lead to wonderful visibility that will attract high-potential website traffic. 

7. Use retargeting display ads: Retargeting can capture the attention of customers who visit your website but leave without making a purchase.

8. Make the most of email: Email is still the preferred method of communication for many buyers, and it can increase traffic from your existing audience. Emails that connect at different points in the customer journey can be automated with minimal effort, freeing you up for other important tasks. These include welcome emails that introduce new customers to your brand, abandoned cart emails to bring people back to complete a purchase, messages highlighting best-selling items, and more.

Root Cause Analysis

## **Voice Up, Vision Forward: Decoding the Future of Multimodal Tech**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FZ7ZG3gUbYLSk71C5i8u2%2F803bcf84-0208-4b73-a2ef-ecdbc911629e.jpg?alt=media&token=a7595b1b-6c08-4c9c-b9f1-29e6b51f9441","id":"803bcf84-0208-4b73-a2ef-ecdbc911629e","width":1892,"height":1116,"filename":"IMG_0043.jpg","type":"image/jpeg","caption":"","border":false}]}
```

As we embark on 2024, the echoes of technological marvels from the past year still resonate. In the realm of voice computing, advancements like Meta Glasses and the Humane AI Pin have pushed the boundaries of what's possible. They whisper promises of a future where our voices seamlessly orchestrate our digital interactions.

But beneath this optimism lies a paradox. While these devices showcase the potential of voice, challenges remain in establishing it as a standalone category. Consider the saga of Amazon's Alexa, a household name that still incurred a staggering $10 billion loss. This begs the question: are we struggling with monetization, or is there a deeper flaw in isolating voice as a tech force?

## The Current Landscape: Beyond the Hype

Let's peel back the layers. Devices like Meta Glasses and the Humane AI Pin, equipped with generative AI, represent the pinnacle of voice capabilities. But challenges lurk in the shadows. Dependence on visual interfaces for setup, privacy concerns, accuracy issues, social discomfort, and limited multimodal integration exposes the limitations of a voice-only experience.

Market trends and user behaviors add another layer of complexity. Studies reveal a preference for a hybrid approach, where voice and visual interfaces work in tandem. Users instinctively switch back to traditional interfaces in certain scenarios, highlighting the need for holistic solutions.

## Navigating the Future: Voice as an Enhancer, Not a Replacement

As we gaze toward the horizon, let's approach voice computing with nuance. While Meta Glasses, Humane AI Pin, Apple's AirPods Pro+, Google's Pixel Lens, and Microsoft's HoloVoice showcase its potential, they also reveal its true nature: a powerful accessory, not a replacement for visual interfaces.

Designers and product managers, armed with this knowledge, can pioneer solutions that leverage voice's strengths while embracing its symbiotic relationship with visual interfaces. In doing so, we can create tech solutions that resonate with users and stand the test of time.

**What are your thoughts?** Are we on the brink of a voice-powered revolution, or is its true potential as an enhancer, not a replacement? Share your insights and aspirations for the tech landscape in the comments below!  #VoiceComputing2024 #TechInnovation #NewYearTech #FutureTechTrends

P.S. Don't forget to check out the Humane AI Pin for a glimpse into the future of voice-driven experiences!

Voice Up, Vision Forward: Decoding the Future of Multimodal Tech

A prominent law firm sought a centralized repository of partner information, aggregated from multiple internal and external websites. Our team delivered a fully automated solution that:

1\. **Extracted Data** with a Python 3.9 + BeautifulSoup 4 web crawler, enriched with libraries like Requests and Selenium for dynamic content.

2\. **Transformed and Loaded** the data into a secure, scalable AWS-based infrastructure (S3, Glue, Redshift) via an automated CI/CD pipeline.

3\. **Built a Retrieval-Augmented Generation (RAG)** architecture leveraging LangChain, with vector embeddings stored in MongoDB’s VectorStore for advanced semantic querying.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FZg4U9YZxcLl2MlLFariu%2F32355887-8d36-4153-980e-bd28d606ab70.png?alt=media&token=50c303c8-f9d7-46ea-8419-618af3123b06","id":"32355887-8d36-4153-980e-bd28d606ab70","width":1583,"height":3840,"filename":"Untitled diagram-2024-12-26-124544.png","type":"image/png","caption":"","border":false}]}
```

## 

### **1.1 Fragmented Data Ecosystem**

• **Diverse Sources**: Partner profiles resided on the firm’s site, industry directories, and legal ranking websites (e.g., Chambers, LexisNexis).

• **Low Data Quality**: Reliance on manual copying/pasting and ad-hoc spreadsheets increased data inconsistency and errors.

• **Complex Inquiries**: Existing search tools were limited to keyword matching, making it difficult to answer deeper queries like “Which partner has negotiated the largest arbitration in the energy sector?”

### **1.2 Strategic Objectives**

• **Centralized Data**: Consolidate partner information into a single “source of truth.”

• **Automated Pipeline**: Eliminate manual overhead through robust data ingestion, transformation, and storage processes.

• **Intelligent Search**: Leverage modern NLP (through Large Language Models) to enable advanced retrieval and summarization.

This end-to-end system consolidated scattered partner data, unlocked complex NLP-driven queries, and significantly accelerated the firm’s research capabilities.

### **2. Technical Approach and Methodology**

### **2.1 Data Collection with Python 3.9, BeautifulSoup 4, and Selenium**

**2.1.1 Crawler Design**

• **Libraries and Frameworks**:

• **Requests** for straightforward GET/POST calls.

• **BeautifulSoup 4** to parse static HTML pages.

• **Selenium** for dynamic content extraction where JavaScript-driven elements are present (e.g., partner bios behind AJAX calls).

• **Modular Architecture**:

• **Config Files**: Each target website had a dedicated config specifying URL patterns, HTML selectors, and pagination rules.

• **Error Handling**: Implemented Python try-except blocks to handle broken links, timeouts, and captchas.

**2.1.2 JSON Conversion**

• **Schema Definition**: Leveraged Python’s dataclasses or pydantic to enforce a consistent schema (e.g., name, position, specialty, bio, etc.).

• **Data Validation**: Used built-in validators to ensure mandatory fields (e.g., email, phone number) are not null and follow standard formats.

• **Output Storage**: Exported validated data to JSON, then stored locally before upload to AWS S3.

### **2.2 ETL Pipeline with AWS and Supporting Tools**

**2.2.1 Data Ingestion into Amazon S3**

• **S3 Bucket Configuration**:

• **Versioning** enabled for audit trails.

• **Lifecycle Policies** for transitions to infrequent access and archiving.

• **Security**: Employed AWS KMS for server-side encryption (SSE-KMS), restricting keys through IAM roles.

**2.2.2 AWS Glue for Transformation**

• **Glue Crawlers**: Automated schema discovery for JSON, generating a Glue Data Catalog.

• **Glue ETL Jobs**: Written in PySpark to perform:

• **Cleansing**: Standardize date formats, unify naming conventions (e.g., “Senior Partner” vs. “Sr. Partner”).

• **Enrichment**: Cross-referenced external data (e.g., Chambers ranking) to add partner accolades.

• **Job Orchestration**: AWS Step Functions or Apache Airflow (running on Amazon MWAA) for scheduling and dependency management.

**2.2.3 Loading into Amazon Redshift**

• **Redshift Provisioning**:

• **Cluster Type**: dc2.large or ra3 nodes for scalable compute and storage.

• **Subnet Groups** in a private VPC for secure data access.

• **Data Warehouse Schema**:

• **Star Schema**: Fact tables (e.g., “Partner\_Engagements”) referencing dimensional tables (“Dim\_PartnerInfo,” “Dim\_Specialty”).

• **Metadata Tracking**: Columns for last\_updated, source\_url for auditing.

• **Performance Optimization**:

• **Sort Keys and Dist Keys** to minimize data movement for heavy queries.

• **Column Encoding** chosen by Redshift’s ANALYZE COMPRESSION function.

### **2.3 CI/CD and DevOps**

• **Version Control**: GitLab for code repositories (crawler scripts, Glue ETL jobs, and Redshift schema definitions).

• **Continuous Integration**: GitLab CI jobs running lint checks (Flake8, Black) and unit tests.

• **Docker Containerization**: Packaged the crawler code into Docker images stored on AWS ECR for reproducible runs.

• **Infrastructure as Code**: Deployed S3 buckets, Glue jobs, and Redshift clusters via AWS CloudFormation or Terraform.

### **2.4 Building an NLP-Powered Search Engine (RAG) with LangChain**

**2.4.1 Vector Embedding Generation**

• **LangChain Integration**:

• **Language Model**: Hugging Face Transformers (e.g., sentence-transformers/all-MiniLM-L6-v2) for embedding generation.

• **Embedding Pipelines**: Batch processed partner bios and specialized legal documents.

• **Data Flow**:

1\. Query Redshift for partner data.

2\. Use LangChain to transform text fields into vector embeddings.

3\. Store embeddings in MongoDB’s VectorStore.

**2.4.2 MongoDB VectorStore**

• **Schema**:

• \_id: Unique partner identifier or doc ID.

• embedding: High-dimensional float array.

• metadata: Additional fields (e.g., partner name, practice area).

• **Similarity Search**:

• **Cosine Similarity**: Implemented within the VectorStore to find nearest neighbors in embedding space.

• **Indexing**: Utilized MongoDB Atlas Search for indexing vectors and accelerating queries.

### **2.4.3 RAG Query Flow**

1\. **User Query**: Input captured via a React/Next.js web front-end or a chatbot interface built in Streamlit.

2\. **LangChain Orchestration**:

• **Retrieval**: Identifies top-k relevant partner embeddings from MongoDB.

• **Contextual Assembly**: Aggregates relevant partner details.

• **LLM Response**: Passes context to the LLM (OpenAI GPT-4 or local LLM) to generate a final, succinct answer.

3\. **Answer Delivery**: Renders a final text response along with relevant partner profiles.

By leveraging a Python –driven web crawler (BeautifulSoup, Selenium, Requests), an AWS-based ETL pipeline (S3, Glue, Redshift), and a state-of-the-art RAG approach (LangChain + MongoDB VectorStore), we delivered a future-ready platform for the firm’s legal data needs. This solution not only centralizes partner information for optimal discoverability but also empowers attorneys and researchers with intelligent, high-speed NLP queries—giving the firm a cutting-edge advantage in a competitive legal landscape.

Comprehensive System-Level State Diagram for an NLP-Driven Data Pipeline Integrating Web Crawling, AWS ETL, and Retrieval-Augmented Generation

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2Fsteve-jobs-%2F71b7acf1-21da-4d4f-91d2-b6be87f1a296.jpg?alt=media&token=b6fd2bff-e181-40e1-817c-acbcb7a80cc6","id":"71b7acf1-21da-4d4f-91d2-b6be87f1a296","width":728,"height":455,"filename":"steve-jobs-face-wallpaper-preview.jpg","type":"image/jpeg","caption":"","border":false}]}
```

Just finished Tony Fadell's awesome book "Build," and wow, did it have me reminiscing about the legend - Steve Jobs! Here are 5 things that really stuck with me:

**1. Simplicity Rules:** Less is more, even for tech geniuses like Jobs. He taught me to ditch the fluff and focus on creating products that are easy to use and understand. No more user manuals the size of phone books!

**2. Obsess Over Details:** Turns out, Jobs wasn't just picky, he was a detail ninja! He showed me that the little things – that perfect click of a button, the smooth curve of a case – can make a HUGE difference. ✨

**3. Tell a Story with Your Product:** Forget boring specs, Jobs emphasized that products should have a narrative. They should inspire and solve real problems, not just list features. Think iPod shuffle – tiny tunes, big freedom! ‍♀️

**4. Believe in the Impossible:** Jobs was a master dreamer, convinced you could achieve anything if you set your mind to it. His unwavering belief in the iPod, even when everyone doubted, is a constant reminder to chase those crazy ideas.

**5. Be a Demanding Mentor:** Okay, Jobs wasn't everyone's cup of tea. But Fadell showed how his tough love pushed teams to excel, to go beyond "good enough" and create true game-changers.

These are just a few nuggets from "Build" that left me inspired. If you're building anything (not just tech!), check it out

**#stevejobs** **#leadership** **#productdevelopment** **#build** **#tonyfadell** **#entrepreneurlife** **#nevergiveup**

5 Big Life Lessons I Learned About Steve Jobs (Thanks, Tony Fadell!)

## **What is Hubble**

```cv
youtube
{"url":"https://www.youtube.com/watch?v=eJpt_uACgFA","width":200,"height":113,"thumbnail":"https://i.ytimg.com/vi/eJpt_uACgFA/hqdefault.jpg"}
```

Introducing Hubble.cx: Revolutionizing Customer-Centric Product Development

Unlock the power of customer insights with Hubble.cx, an innovative platform that meticulously analyzes open customer platforms such as Twitter, App Store, and Play Store. With the backing of advanced Language Model (LLM) technology and the expertise of Langchain, Hubble.cx goes beyond conventional analytics, helping you navigate the intricacies of customer satisfaction, Net Promoter Score (NPS), and user footprint.

### **Key Features:**

**1. Comprehensive Platform Analysis:**

Hubble.cx scours social media platforms like Twitter and major app marketplaces, including the App Store and Play Store, to provide an all-encompassing view of customer sentiments and trends.

**2. CSAT, NPS, and User Footprint Identification:**

Gain unparalleled insights into customer satisfaction (CSAT), Net Promoter Score (NPS), and user footprint. Hubble.cx empowers you to understand how your audience perceives your product and brand across different channels.

**3. Customer-Centric Product Building:**

Transform your product development approach with Hubble.cx. By leveraging real-time data and user feedback, build products that resonate with your customers, leading to a 100% success rate in meeting their expectations.

**4. Strategic Decision-Making:**

Hubble.cx delivers a gold mine of information, enabling your product team to make informed decisions. Avoid the pitfalls of developing features that customers don't desire, and instead, focus on crafting solutions that address their specific needs.

**5. Detailed Reports and Actionable Recommendations:**

Receive comprehensive reports highlighting real pain points identified in customer feedback. Hubble.cx doesn't just stop at analysis—it provides actionable recommendations to align your product with the end user effectively.

## **How Hubble.cx Works:**

**1. Data Collection:**

Hubble.cx utilizes advanced Language Model (LLM) technology to gather and process vast amounts of data from open customer platforms.

**2. Analysis:**

Langchain's expertise comes into play as Hubble.cx analyzes customer sentiments, CSAT, NPS, and user footprint, providing a holistic understanding of your product's perception.

**3. Insights and Recommendations:**

Receive detailed reports pinpointing areas of improvement and strategic recommendations for aligning your product with customer expectations.

Don't just meet customer expectations—exceed them with Hubble.cx. Revolutionize your product development journey, ensuring every step is guided by valuable insights and the assurance of customer satisfaction.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FraEeHCKDGsSv1r6tXDnF%2F3cca034a-1102-4c8d-a201-93243174a544.png?alt=media&token=a9ba5e49-69b0-4a47-9513-fd71e3c58bb1","id":"3cca034a-1102-4c8d-a201-93243174a544","width":1920,"height":1080,"filename":"Group 2.png","type":"image/png","caption":"","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FraEeHCKDGsSv1r6tXDnF%2Fb75645bc-b8f5-4672-9487-ba9682a7f7c6.png?alt=media&token=7ba435f5-a875-48e6-ade4-02be71d7ac84","id":"b75645bc-b8f5-4672-9487-ba9682a7f7c6","width":1920,"height":1080,"filename":"Frame 585.png","type":"image/png","caption":"","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FraEeHCKDGsSv1r6tXDnF%2Ff15e5c2c-31ff-4436-90ba-ed5f453532d0.png?alt=media&token=089aa670-7b70-43c1-93fc-f2f36b4fed43","id":"f15e5c2c-31ff-4436-90ba-ed5f453532d0","width":1920,"height":1080,"filename":"Frame 585-2.png","type":"image/png","caption":"","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2FmCk4fivRH6XxNnQD3iA0I1Mcv9L2%2FraEeHCKDGsSv1r6tXDnF%2F4721b413-d11b-41b0-abe6-72412f6b3582.png?alt=media&token=928e123e-40b5-42c2-96a0-b8c519b5c7c5","id":"4721b413-d11b-41b0-abe6-72412f6b3582","width":1920,"height":1080,"filename":"Frame 585-1.png","type":"image/png","caption":"","border":false}]}
```

Hubble.cx

Do You Really Understand PMF and How to Quantify It?

Check my latest article on PMF https://abhijit.work/quantifying-pmf
mail me at abh8017@gmail.com

Chief Data Scientist

AWS Solution Architect Associate

Azure Data Scientist Associate

Cisco Catalyst Center Monitoring Infrastructure

Salesforce and Einstein RMF Analysis Case Study

Firstlook AI: Building a Pragmatic Data Platform for Healthcare Scheduling

DocuSign Precision in Assessment and Transformation

Enterprise Data Lakehouse on Cloud

Building HPE Ezmeral's Core Products: EzSQL, MLOps, and HPE Spark

Addressing Duplicate Listings on Noon Marketplace

Parcel Shipping Analytics: Analyzing FedEx and UPS Invoices

Product Development at Boeing with a Robust Data Lake

Strategic Data Analytics for eBev's Brewery Sales and Operations Online

Hubble, a customer analytics platform using Generative AI and NLP

Customer Analytics using Genrative AI

Build and manage AI agent teams; supports multi-model tasks, on-prem/cloud deployment, and enterprise security

Built AI teams

Data as a Political Asset

I collaborated with companies like Cisco, Avit, and HPE. I specialized in crafting Machine Learning models for Business teams. A notable achievement includes developing the [P2B (Propensity to Buy) framework for Cisco using H2O.ai ↗](https://h2o.ai/blog/2014/predictive-modeling-at-scale/)  Another highlight was creating the Business Impact Score post-COVID. Leveraging government data, we identified sites affected by COVID-19, predicting sales impact for DNAC and Cisco Collab devices.

Data Scientist

* Identifying the potential clients using Data Mining techniques from Zomato and Swiggy

* Cold Calling to the prospects 

* Door to Door Sales for Restaurants and Eateries

POS Salesman

As a data analyst at Dun & Bradstreet, my role centered around working with data related to [Dun Numbers↗](https://www.dnb.com/duns.html). Analyzing and interpreting this data, I contributed to valuable insights and decision-making processes. It was an exciting journey navigating the world of Dun Numbers and leveraging data to drive meaningful outcomes

Analyst

I've overseen the development of a project management tool and an AI platform for Customer Experience. \
Collaborating with HPE, I played a pivotal role in building HPE Ezmeral and MLOps workflow. \
Managing a talented team of data scientists and analysts, we partnered with DocuSign to create Contract Management Intelligence. Let's dive deeper into this exciting journey!

Abhijit Singh

2024

2023