Ten tips for better behaved bots

How to keep your credit union’s AI agents from being blocked from your website

More and more credit unions are starting to use AI for all sorts of things, such as answering questions for members, optimizing copy and content, looking for compliance issues, helping generate ideas, and much more. Many times, these AI agents or bots need to use your credit union’s website as a source of knowledge.

However, AI agents that don’t behave themselves as they scan your site can quickly find themselves blocked. Here are some of the issues we’ve seen, and some of the principles for ensuring that your AI agents can access the information they need.

What’s the problem?

Rapidly surging use of AI bots has quickly become THE leading security and resource management concern for everyone who manages CU websites, and strong automated protections are vital. Although some AI bots are malicious, the main problem is that the vast majority of bot traffic is simply parasitic, wasting tremendous bandwidth and processing power through incessant rapid-fire requests, and providing no value or human traffic in return.

The good news is that the anti-bad-bot systems are themselves using AI, and rapidly becoming far more intelligent about allowing access to useful AI agents. In a nutshell, we’re finding that well-behaved AI agents usually don’t have a problem.

Instead of getting involved in the complex and ever-changing details of this AI vs. AI arms race, here are some overall principles to help make sure your bot stays on the good side of the gatekeepers.

Building better behaved bots

1) Check in with your website and AI pros.

Before you unleash your new AI pal, check with your website developers about what they would recommend, the tools they might have to log and monitor activity, and what to do if your agent gets blocked. It may not be possible to broadly whitelist a specific agent or server while keeping the vital automated protections in place.

2) Give yourself an “off” switch and a fallback.

Make sure you have a way to quickly switch off your AI agent’s website scanning if it starts having a problem or the results are unexpected, and that you have a fallback plan for the service the AI is performing.

3) Rate-limit your agent.

There’s not a specific published rate limit, but instruct your agent to limit itself to a reasonable request rate (less than 60 to 100 per minute is a common recommendation). Most credit union sites have fewer than 100 pages, and don’t change that rapidly, so a slow request rate has little real-world impact.

4) Only re-scan the site as needed.

Target only changed content, and only when needed. In other words, retain the information from the initial scan; don’t re-scan the site from scratch with every query.

5) Limit your agent’s scope and file types to those necessary.

Most AI agents only need to gather the site’s HTML text or perhaps PDFs for a knowledge source. There’s usually no need for these to examine other files like fonts, Javascript, CSS, or many kinds of images.

6) Avoid looking dishonest.

Don’t use evasion tactics, tricks, or anything that could be interpreted as an attack, hack, or an attempt to subvert protections. There’s a lot of gray area here; for example IP hopping or simulating user-agent strings to look like “real” people are currently common tactics, but can also be seen as dishonest. This is evolving at AI speed, but the underlying principle is that agents should be up-front about what they’re doing and who they work for.

7) Stay aware and in control of what your bot is doing.

This includes understanding the amount of traffic and rate of requests your agent is generating, and the resources and AI credits or tokens it’s consuming. It’s very easy to lose track of this, so make sure limits and reporting are built in to everything you do with AI, and that there’s always a human in the loop.

8) If your bot is blocked…

Try reviewing and changing its behavior and try updating later. Most automated blocks are active and intelligent, and usually not permanent. If your agent is still getting blocked, ask your web developers to review the logs for clues and discussion.

9) Register as a Verified Bot.

Cloudflare filters more internet traffic than any other company, and registering your agent or bot might help ensure that it’s is seen as one of “the good guys” because they know who is operating the bot and why, and the scope of what it can do: https://developers.cloudflare.com/bots/concepts/bot/verified-bots/

This is also evolving and changing at AI speed, and of course Cloudflare isn’t the only game in town. There are many, MANY other companies and projects working on ways to register known “good” bots, and for websites to allow and disallow specific bots and agents.

10) Experiment, and remember the humans.

Things in the AI world move very quickly, so it’s important to keep an open mind and experiment. And of course, leverage your credit union superpowers: keep your mission and principles in sight at all times by doing what’s best for the human members you serve.

Brian Wringer
Latest posts by Brian Wringer (see all)