I suspect you’re holding it wrong. What kind of product is it, and how do you use AI? I can probably help if you want?
Well, my prod support toolkit pulls data from anywhere from 6 to 16 different systems to local jsonl files with PII obfuscated so it doesn’t get sent to the AI. Then it has to examine where data integrity was violated, assign it to the system or user that mishandled the data, discover the root cause — because if it’s an automated system then either a business rule went wrong, data was previously polluted, or an agent mis-entered data in some upstream system. Lastly, it has to identify how to fix the data and request permission to resolve the issue, or provide instructions to other operators with permissions in systems I can’t reach into to fix, such as prod database or Salesforce, which is the domain of other teams.
It diagnoses anything from user error to system instability to networking issues to user error to business requirements that work cross-purposes. It queries three different data stores, two different log providers, and ten or so APIs.
It has several playbooks for processing “standard” problems, which include instructions to attempt to falsify its conclusions. It also has documentation of each system and specific conventions/apis/etc. And to the degree possible, it executes everything deterministically with small python scripts for each system, “flow” bash scripts for executing standard sets of calls, and transforming and validation. It also executes multiple adversarial agents that attempt to disprove its conclusions or rewrite unclear wording and remove all the little one-time notes it leaves for itself.
I don’t want to suggest this toolkit isn’t extraordinarily helpful, because it has saved me countless hours, but it gets conclusions wrong all the time and I have to continually ask it whether it performed this check or that one, or why it decided this was the root cause when there isn’t enough information to make that determination or there is clearly evidence that points in another direction.
It is written pretty much exactly as I envisioned and when I need to add another tool for another system, it does that itself in minutes. There are likely some areas that could be improved.
I recently added the PII obfuscation because someone would push back hard on that in a security review once we explain what we’re doing. It would work better if every system wasn’t written by separate teams to separate standards, but it does really well. I can compare tokens to see if they match without decoding them, and I also have a script for turning tokens back to clear text so that I can see the values without sharing them with the cloud.
Let me know if you see any gaps there other than operational because I don’t get to dictate business requirements. I think it’s extraordinarily robust, but that doesn’t help it diagnose the right problem.
Point is: if it has all of that and can’t reliable reason about how systems work together, what are the chances it’s misunderstanding how the code needs to go together?
You didn’t describe exactly what your setup is (or even what your product is, but I’m guessing some kind of CRUD web system…). Here’s what I would do if I was in your position:
Set up a sandbox with Opus 5.5 and an agent harness in it (e.g. Codex, Pi or OpenCode).
Most importantly, give it access to your source code, and a way to run a dev environment (set up the database etc.).
Feed it all the info with PII removed as you are (good work btw).
Write some skills, based on your previous experience - you mentioned having to tell it to check things repeatedly, you can just write those things down for it.
The absolute most important things are that it has access to the source code (and documentation if you have any lol), and an environment where it can actually try things out. If you think about it, that makes things way easier for humans too so it’s not really that surprising. If I gave you a bunch of C++ and said “where’s the syntax error”, it’s a much easier task for both humans and AI if you have access to a C++ compiler.
It’s in the hospitality industry. I have to make sure the separate guest profile systems align between bookings, crm, identity, and authentication. Each has a unique data store with different requirements (booking data is allowed to be bullshit because we just want that sale with zero friction) but eventually the systems must be aligned, so the system has to join identities between all systems based on matching a preponderance of PII, while also allowing someone to get a name change through marrying, move to a different location, go by a nickname, or someone made up a birthdate because they couldn’t guess. Also we have to not incorrectly merge because that creates a data spillage.
Well, my prod support toolkit pulls data from anywhere from 6 to 16 different systems to local jsonl files with PII obfuscated so it doesn’t get sent to the AI. Then it has to examine where data integrity was violated, assign it to the system or user that mishandled the data, discover the root cause — because if it’s an automated system then either a business rule went wrong, data was previously polluted, or an agent mis-entered data in some upstream system. Lastly, it has to identify how to fix the data and request permission to resolve the issue, or provide instructions to other operators with permissions in systems I can’t reach into to fix, such as prod database or Salesforce, which is the domain of other teams.
It diagnoses anything from user error to system instability to networking issues to user error to business requirements that work cross-purposes. It queries three different data stores, two different log providers, and ten or so APIs.
It has several playbooks for processing “standard” problems, which include instructions to attempt to falsify its conclusions. It also has documentation of each system and specific conventions/apis/etc. And to the degree possible, it executes everything deterministically with small python scripts for each system, “flow” bash scripts for executing standard sets of calls, and transforming and validation. It also executes multiple adversarial agents that attempt to disprove its conclusions or rewrite unclear wording and remove all the little one-time notes it leaves for itself.
I don’t want to suggest this toolkit isn’t extraordinarily helpful, because it has saved me countless hours, but it gets conclusions wrong all the time and I have to continually ask it whether it performed this check or that one, or why it decided this was the root cause when there isn’t enough information to make that determination or there is clearly evidence that points in another direction.
It is written pretty much exactly as I envisioned and when I need to add another tool for another system, it does that itself in minutes. There are likely some areas that could be improved.
I recently added the PII obfuscation because someone would push back hard on that in a security review once we explain what we’re doing. It would work better if every system wasn’t written by separate teams to separate standards, but it does really well. I can compare tokens to see if they match without decoding them, and I also have a script for turning tokens back to clear text so that I can see the values without sharing them with the cloud.
Let me know if you see any gaps there other than operational because I don’t get to dictate business requirements. I think it’s extraordinarily robust, but that doesn’t help it diagnose the right problem.
Point is: if it has all of that and can’t reliable reason about how systems work together, what are the chances it’s misunderstanding how the code needs to go together?
You didn’t describe exactly what your setup is (or even what your product is, but I’m guessing some kind of CRUD web system…). Here’s what I would do if I was in your position:
The absolute most important things are that it has access to the source code (and documentation if you have any lol), and an environment where it can actually try things out. If you think about it, that makes things way easier for humans too so it’s not really that surprising. If I gave you a bunch of C++ and said “where’s the syntax error”, it’s a much easier task for both humans and AI if you have access to a C++ compiler.
It’s in the hospitality industry. I have to make sure the separate guest profile systems align between bookings, crm, identity, and authentication. Each has a unique data store with different requirements (booking data is allowed to be bullshit because we just want that sale with zero friction) but eventually the systems must be aligned, so the system has to join identities between all systems based on matching a preponderance of PII, while also allowing someone to get a name change through marrying, move to a different location, go by a nickname, or someone made up a birthdate because they couldn’t guess. Also we have to not incorrectly merge because that creates a data spillage.
Fun times.