When you add a website to a knowledge base using Site maps, DialLink crawls the site to collect its pages. Sometimes the crawl can’t be completed. This article explains the most common reasons and what you can do in each case.
Bot protection
Some websites use security services, such as firewalls or CAPTCHA checks, that block automated visitors. When this happens, DialLink’s crawler can’t access the site’s pages.
What to do: ask your website administrator or web team to allow DialLink’s crawler through the site’s bot protection settings. If that isn’t possible, you can add individual pages using Web pages, or copy the content into a text snippet or upload it as a file.
Blocked by robots.txt
Most websites have a robots.txt file that tells crawlers which pages they can and can’t visit. If the file blocks DialLink’s crawler from some or all pages, those pages won’t be added to your knowledge base.
What to do: if you own the website, you can verify your domain and turn on Crawl despite robots.txt (see below). If you don’t own the site, DialLink respects its robots.txt rules, so add the content another way, such as uploading a file or adding a text snippet.
No pages found
Sometimes the crawler reaches the website but doesn’t find any pages to collect.
This can happen if:
The URL is incorrect or points to a page with no links to other pages.
The site’s content loads only after the page opens in a browser, so the crawler can’t read it.
The crawl depth or page limit is set too low to reach the content you need.
What to do: check that the URL is correct, then try increasing the crawl depth or page limit. If the problem continues, add the pages individually using Web pages.
Crawl Your Own Website Despite robots.txt
If you’ve verified that you own a website, you can turn on Crawl despite robots.txt. This lets DialLink’s crawler collect pages from your site even if its robots.txt file would normally block them.
This option is available only for verified website owners, since only the owner can decide to override the site’s own crawling rules.