>>534970870
I swapped to d&d dice and it works MUCH better fairness wise, you can see it call the API and then describe the result, even if there is some strange behavior I have yet to fix where it dumps context into the chat, pic related.

Lots of (mostly fixable) issues you need to constantly remind the AI of, as expected, miscalculating character sheets etc. Sticking to the AP is so-so, not very good at keeping the campaign moving between scenes. I've actually read the first chapters of Rise of the Runelords and it's done some interesting things where it has replaced the roles of certain characters with others.

For example, it kept Lonjiku alive and killed off Tsuto instead, giving his role in the ap to his father. Very interesting choice.

Overall storytelling wise I'd say that the AI is a competent, if samey and spineless GM. It does pretty well with tying even things it's hallucinated back into the story and the lore, impressive improv bullshit.
It's bad at comprehending the rules of pathfinder, you have to remind it of a lot things to get a good experience.
You also have to understand that no matter what, it's pretty bad at saying "no, you can't roll for that."