Skip to content

UfrontKnowledgeBot

You found UfrontKnowledgeBot in your server logs. This page says who it is and how to stop it.

What it fetches

UfrontKnowledgeBot fetches only web pages that a Ufront customer added as a knowledge source for their own AI agent, and the robots.txt of those sites. It also fetches property data feeds (JSON or XML) that a customer set up for their own listings; robots.txt is not read for those feeds. It does not follow links and does not crawl the web. It may fetch the same page again later to keep the agent up to date, and then asks only for changes (If-None-Match, If-Modified-Since).

How it fetches

It uses HTTPS only, reads text and HTML pages up to 2 MB, waits at most 10 seconds and follows at most 5 redirects. It identifies itself with the User-Agent UfrontKnowledgeBot/1.0.

robots.txt

Before every page (not feed), and again after every redirect, it reads /robots.txt of that site. It obeys the Disallow lines in groups for UfrontKnowledgeBot and for *. Rules are plain path prefixes: Allow lines, * and $ wildcards are not read. If robots.txt answers with a server error, it does not fetch the page and tries again later.

How to block it

Add these lines to your robots.txt. To block only part of the site, use that path instead of /.

User-agent: UfrontKnowledgeBot
Disallow: /

Contact

Questions or a request to stop fetching your site: write to hello@ufront.eu

UfrontKnowledgeBot - the Ufront knowledge crawler | UFRONT