Types of data sources

Data sources are the foundation of your AI Agent’s intelligence. Together, they provide the exact context and information the AI needs to respond accurately to user queries without hallucinating.

You can add several different types of data sources to your Knowledge Base to cover all of your business information:

1. Website

When you add a website, our crawler will visit the domain and extract all the internal pages it finds. You can then select exactly which pages you want to ingest into the Knowledge Base. (Note: The standard website crawler can extract up to 1,000 pages from a single domain. If your site contains more pages than that, you must use the Sitemap data source).

2. Sitemap

Add a sitemap URL (e.g., https://yourdomain.com/sitemap.xml) to systematically fetch all the pages of your website. The sitemap is the most reliable method for ingesting large websites with more than 1,000 pages. Once the sitemap is fetched, you can select which pages to add to the Knowledge Base.

3. URL List

Add a manual list of individual URLs to the Knowledge Base. This is extremely useful when you only want to ingest specific blog posts, help center articles, or deep links instead of scraping an entire website domain.

4. Document

Upload offline documents such as PDF, DOCX, and TXT files. The AI will parse the document, extract the text, and use it seamlessly to generate responses.

5. Text

You can add raw text snippets directly to the Knowledge Base. This is incredibly useful when you want to add custom, private, or ad-hoc information that isn’t available on your public website or in any existing document.

6. FAQs

You can add a structured list of questions and answers that users frequently ask. This is useful when the information is not easily available through other data sources. You can also use FAQs as a highly effective way to fine-tune the AI responses (for example, explicitly providing the correct Q&A format when the AI responds incorrectly to certain edge-case questions).