Introduction
Hi, I am Akira, the editor-in-chief of Data Without Code. In our previous Use Case, we successfully blended Google Analytics web traffic with CRM sales data. You now have a perfect view of your internal customer journey.
But business does not happen in a vacuum. No matter how perfectly you optimize your own marketing, if your biggest competitor drops their price by 20% overnight and you do not notice, your sales will plummet.
How do you currently track competitor pricing? Do you have a junior analyst manually visiting five different websites every Monday morning, copying the prices, and pasting them into an Excel tracker? As a DX manager, I consider that a massive waste of human potential.
In this tutorial, we are going to combine several skills we have already learned to build a fully automated Competitor Price Monitoring Dashboard in KNIME.
Step 1: Scrape the Competitor Prices
The first step is to get the data automatically. We do not want to visit the websites manually.
If you remember our guide on how to extract data from a website using KNIME, we are going to use the exact same logic. You will need a GET Request node to download the competitor’s product page, followed by an XPath node to extract the specific HTML element that contains the price tag.
If you are tracking multiple products across different competitors, you can put this process inside a loop! Use the CSV Reader node to load a list of competitor URLs, and run them through a Loop Start and Loop End process to scrape them all at once.
Step 2: Clean the Price Data
When you scrape a price from a website, it usually comes back as messy text, like "$ 1,299.00 USD". You cannot do math on this text or plot it on a chart.
We need to clean it up using our Data Prep skills:
- Use the String Manipulation node to remove the dollar signs, commas, and letters. You can use the
removeChars()function to strip them out. - Once you have a clean string like
"1299.00", pass it through the String to Number node to officially convert it into a Double (decimal number).
Step 3: Build the Monitoring Dashboard
Now that you have a clean list of today’s competitor prices, we need to visualize it so your pricing strategy team can make quick decisions.
Let’s use the exact same technique we used when we built our first automated sales dashboard.
- Add a Bar Chart node to show the current price of your product versus your top 3 competitors.
- If you save this scraped data every day to a SQL database, you can also add a Line Plot node to show how the competitor’s price has fluctuated over the last 30 days.
Select both charts, right-click, and select Create Component. You now have a beautiful, interactive price monitoring dashboard that updates automatically every time you run the workflow.
Akira’s DX Hack: Add Automated Price Alerts
Why stop at a dashboard? The ultimate goal of automation is to manage by exception. You only want to be notified if something goes wrong.
Add a Rule Engine node to compare your product’s price against the competitor’s scraped price. Write a simple rule: If Competitor Price < Our Price => “ALERT”.
Then, connect a Row Filter to isolate only the “ALERT” rows, and route them to a Send Email node. Now, if a competitor suddenly launches a flash sale, KNIME will instantly email your pricing manager so they can react immediately.
Conclusion: Your Next Steps
Congratulations! You have just built a system that gives your company a massive competitive advantage. You are no longer reacting to the market; you are monitoring it in real-time without spending a single minute on manual data entry.
We have covered so much ground in this course. From basic data cleaning to web scraping, from HR analytics to competitive intelligence. But there is one final business problem we must solve. It is the most critical metric for any subscription or SaaS business: Customer Retention.
If you want to predict which of your customers are going to cancel their contracts next month, you cannot afford to miss our final, masterclass tutorial. Join me as we build a Churn prediction model in KNIME: Step-by-step for beginners!
