Exclude webpage sections

Websites often contain a lot of boilerplate information that may not be relevant to the AI, such as navigation menus, footers, or promotional banners.

When you add webpages to the knowledge base, you can explicitly exclude certain sections of the page from being crawled and indexed by the AI. This is highly recommended for two reasons:

1. It removes irrelevant or confusing information, keeping the AI’s responses tightly focused.
2. It prevents unnecessary characters from being counted towards your workspace’s storage limits.

How to exclude sections of a webpage

To configure exclusion rules when scraping:

1. Navigate to the Knowledge Base from your Agent dashboard.
2. Click the Add new or New Data Source button.
3. Select Website, Sitemap, or URL list as your data source type.
4. Click the Advanced Settings button to reveal the scraping options.

Defining exclusion rules

You can exclude specific sections by entering the HTML classes or IDs of the DOM elements you want to exclude (e.g., #footer, .nav-menu, .promo-banner). Note: You must press Enter after typing each class or ID to lock it into the list.

Additionally, you can quickly select generic HTML tags you want to remove globally, such as <header>, <footer>, or <nav>.

Once your rules are configured, you can proceed with entering the website, sitemap, or list of URLs that you want to ingest. The crawler will intelligently exclude all the sections you specified during the extraction process!