Skip to main content

Optimizing your AI expert based on testing results and feedback

Your testing has revealed areas that need improvement. The optimization process is iterative - you make changes, test again, and so on and so on until you are satisfied with the results. Here's a detailed guide to four key areas of tuning your AI Expert.

16.1 Tuning the "heart" of your AI: Editing the system prompt (system_prompt.gdoc file)​

If your AI Expert repeatedly exhibits undesirable behavior (e.g., is too familiar/formal, doesn't answer certain types of questions correctly, "hallucinates" in certain areas, doesn't respect disclaimers), you often need to modify or add instructions to its system prompt.

Most common modifications​

  • Clarify personality and communication style: if the AI's tone is inappropriate, edit the wording in the "PERSONALITY AND COMMUNICATION STYLE" section. Be more specific (e.g., instead of "Be professional," try "Your tone is polite, businesslike, and avoids colloquialisms.").
  • Strengthen rules and boundaries: if the AI oversteps boundaries (e.g., advising in areas where it shouldn't or ignoring disclaimers), tighten or simplify the guidelines in the "BASIC BEHAVIORAL RULES AND ETHICAL Boundaries" section. Sometimes it helps to repeat or emphasize the instruction (e.g., "ALWAYS AND IN ALL CONDITIONS BEFORE GIVING ANY HEALTH ADVICE, PROVIDE THE FOLLOWING DISCLAIMER:...").
  • Use of the "CORRECTION OF COMMON ERRORS" section: as described in Section 12.2 (Section F), this is a very effective way to "reteach" the AI in specific cases where standard instructions are not sufficient. Clearly describe the problem behavior and immediately below it write what the correct behavior is.
warning

After every modification to the system_prompt.gdoc file (and saving it to Google Drive), you must run the /update command in Telegram to have the changes loaded and reflected in your AI's behavior!

16.2 Adding, refining and editing your know-how (working with the sources folder on Google Drive)​

If the AI is responding incorrectly, incompletely, out of date, or admitting ignorance on topics it should be mastering, there is probably a problem with the quality or scope of your uploaded know-how.

What to do​

  • Fill in the missing information: If the AI doesn't know the answer to a legitimate question in your domain, create new material (e.g., a Google Doc in the sources/Texts folder, a PDF, an audio recording) with a detailed answer and upload it.
  • Update outdated information: review your existing materials in sources and make sure they are up to date. Delete outdated or invalid files and replace with new versions.
  • Refine unclear wording: If the AI is misinterpreting some of your materials, try rewording them to make them clearer and more understandable.
  • Create a "FAQ for AI": For frequently asked questions that you want to make sure are answered in a completely accurate and consistent manner, you can create a special Google Document (e.g., FAQ_for_AI_Expert.gdoc in the sources/Texts folder) that explicitly lists these questions and ideal answers.
warning

After every change to the contents of your sources folder (adding, editing, deleting files/URLs), you must run the /update command in Telegram to have the changes reflected in your AI knowledge base!

16.3 Revising and regenerating expert subroles (working with the ROLES folder - for advanced debugging)​

This step is usually not needed often, but can be crucial in some specific situations.

When to consider working with roles?​

  • If analysis with /lastRole and /investigateAnswer:{message} consistently shows that the AI's problem responses come from the definition of a specific expert subrole (e.g. the role is too general, it's missing key aspects of your know-how, etc.).
  • If you've made a very major expansion or restructuring of your know-how (e.g., you've added 10 new course modules on entirely new topics, or you've completely rewritten your core book). In this case, it is very likely that the roles you originally generated no longer fully reflect the current state of your knowledge.

How to proceed?​

  • Targeted update of a specific role: If the problem affects only one or a few roles (and you know which ones), you can try:
    1. (Optional, for maximum control) Open a Google Document with the role definition in the ROLES folder and make manual edits and refinements to it.
    2. Then run /updateRole:{role_name} in Telegram. The AI will save the definition of that role based on the most recent information.
  • Complete re-generation of ALL roles (after a major know-how update):
    1. Make sure to first upload all new/changed know-how to sources and successfully complete the /update command (to keep the knowledge base up to date).
    2. Then go to your Google Drive to your ROLES folder.
    3. Delete ALL Google Docs that contain the definitions of each role (i.e. files with names like Role_NameRole.gdoc). Be careful NOT to DELETE the list_of_roles.gdoc file - it only contains a list of role names and is needed to recreate them!
    4. Then run the /updateRoles command in Telegram. The AI will now start from scratch to generate detailed definitions for all the roles listed in list_of_roles.gdoc, based on your complete, up-to-date knowledge base. This process may take longer, but it will ensure that your expert subroles reflect all of your know-how as best as possible.

16.4 Improving onboarding and first impression (working with the onboarding.gdoc file)​

Based on feedback from your beta testers (or your own feelings about testing as a new user), you may want to modify the welcome messages (first_messages) or the automatic onboarding messages (onboarding_messages). Are the messages clear, motivating and friendly enough? Does the call to action lead the user in the right direction? Does the link to any onboarding video work? Make the necessary adjustments to the onboarding.gdoc file.

warning

After every modification to the onboarding.gdoc file, you must run the /update command in Telegram to have the AI load the new settings!

16.5 Iterative process: the key to success is repetition and patience​

Remember that testing and optimization is not a one-time thing, but a cyclical, iterative process:

  1. Keep testing (alone or with beta testers).
  2. Carefully analyze the results and collect specific feedback.
  3. Make targeted adjustments based on the analysis (System Prompt, know-how in sources, role definitions, onboarding).
  4. Test again to verify the impact of your changes.
  5. Repeat until you are really satisfied with the behavior and quality of your AI Expert.

Be patient and systematic. Tuning complex AI can take time. The goal is not to reach 100% perfection (because AI is fundamentally probabilistic and constantly learning), but to reach a state where you and your testers are confident that the AI is delivering high value and is ready for your real customers.

Last updated on