Install and Use ChangeDetection

Posted on In , , Word count in article: 451 Reading time ≈ 2 mins.

Introduction

Recently, I was waiting for a notification from a website, so I kept checking it periodically. Eventually, I found this too cumbersome and considered writing a web scraper to monitor it automatically. However, I soon discovered that there are many existing platforms available (so there’s no need to write any code). Most of these platforms offer free plans, but they only monitor static web pages, and you have to pay if you want to handle JSON data. So I started looking for open-source alternatives. This led me to the topic of this post — ChangeDetection.

ChangeDetection has two main functions: monitoring changes in URL content and notifying you of those changes. These two features cover most daily use cases.

Docker compose

The basic configuration for ChangeDetection is as follows, with a simple port mapping and a volume.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
version: "3"

services:
  changedetection:
    image: dgtlmoon/changedetection.io:0.46.04
    container_name: changedetection
    restart: unless-stopped
    environment:
      - TZ=Asia/Shanghai
    ports:
      - 55000:5000
    volumes:
      - changedetection-data:/datastore

volumes:
  changedetection-data:
    external: true

ChangeDetection also offers a visual interface, allowing non-programmers to select the content they want to monitor, similar to how they would use a browser. Key features include:

  • Browser Steps: This can record a series of user actions.
  • Visual Filter Selector: This allows you to choose a specific part of a web page to monitor.

These features are powered by Playwright, which requires a WebDriver. The official docker compose file includes the configuration needed to run an additional Chrome container for Playwright. Since I don’t need this feature, I didn’t configure it.

If you plan to monitor a website frequently, it’s recommended to use a proxy to avoid getting your IP blocked, which could prevent you from completing your tasks.

Usage

Using ChangeDetection is straightforward since the interface has only a few buttons.

Filter

The “Filters & Triggers” section is what we need to focus on, as this is where data cleaning is handled. There are many types of filters available, but I mainly use CSS and JQ. I recommend having GPT write specific rules for you.

Email Notifications

In the “Settings | Notifications” section, you can configure notification services. I currently use email notifications, which essentially sends an email to yourself from your own account. The configuration rule is as follows:

1
mailto://{user}:{password}@qq.com

Keep in mind that the password varies depending on the email provider. For Gmail, it’s your account password, but for QQ Mail, you need to generate an authorization code in the “Account and Security | Security Settings” section (similar to an API TOKEN).

The email content uses a Jinja2 template, making it easy to customize.

After configuring everything, test whether it works properly.

References