Rendered at 15:50:40 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
simonw 17 hours ago [-]
I heard a neat tip recently about API design for agents: give them a way to send you feedback.
The example I heard was an MCP with a "feedback" tool which had a tool description saying that coding agents should call that any time they had trouble figuring out how to use the rest of the MCP.
I really like this. It's super cheap to implement and I expect you'd get a bunch of actionable signal in amongst the noise.
hdjrudni 9 hours ago [-]
My work recently changed an agent API. It was a certain CLI tool built for agents which I had copied into one of my Skills.
What they did was kind of clever. When the agent calls the old API/uses old args, I guess they told it to tell the human to fix their skill and 'upgrade' to the new API. They even provided a skill upgrader skill.
Anyway, my agent tried calling it the old way, must have received some sort of error message and then diligently reported to me that my skill needed upgrading and we should be doing it some new way.
Kind of like reverse feedback I guess? Not the AI giving feedback to them, but them giving feedback to me via my AI.
kolinko 8 hours ago [-]
Good one! Also, a simple thing - when your service gets miscalled (wrong params either via cli or http) instantly show help/explaination of the proper param uses. Can save a few turns of agents trying to figure out what’s wrong. Helps humans as well :)
pjm331 17 hours ago [-]
Yeah i have a similar setup at the harness level - a “devlog” at the end of every session about the experience developing, what could have been better, what was confusing
And also files issues for blockers
I’ve absolutely caught things and made improvements just from skimming them occasionally - they are particularly useful when you get a PR that makes you scratch your head
But I’m definitely not taking full advantage of all the feedback coming in yet
I have to imagine parsing signal from noise there is a massive challenge when it’s other agents that are using your MCP and not just your own
kolinko 7 hours ago [-]
I have a similar system and it does wonders. One of the questions is also to ask for just freestyle opinion of the task and the replies are hilarious at times.
“I existed for one word. The entire point of my life was that my process boot side-effected a module reload. 'ok'."
“ "The overhead-to-work ratio was astronomical."
“the koleo purchase turned into a boss fight with an undismissable Vue modal
and a WKWebView that kept eating its own cookies."
catlifeonmars 13 hours ago [-]
You won’t get grounded results by asking for feedback. The LLM doesn’t know its own internal state so while you may get a plausible and contextually significant answer, it isn’t going to be an accurate one.
kolinko 7 hours ago [-]
Are you talking from experience or from theory?
I have a reflections mechanism, which launches right after the session ends (when it’s still in cache, so it’s almost free) and it catches a ton of problems with tool/skill/system design, all the time. Low level stuff can be fixed almost automatically (e.g. agent funbled 10 turns bc tool’s docs were wrong), high level too but human gating is useful.
If you try doing it any other way you have to pay for input tokens, not cache read, so it’s 10x more expensive.
catlifeonmars 2 hours ago [-]
Theory. I guess my question is: why do you think it works so well. What’s the mechanism behind it?
It feels like it would be relatively random across the range of plausible results, but it sounds like you are finding otherwise.
what 14 hours ago [-]
Why are you asking an LLM what could have been better or was confusing? It literally has no idea.
rsfern 13 hours ago [-]
True. But it has no idea that it has no idea, so it might be able to look back at the session trace and pattern match its way to actionable feedback?
simonw 13 hours ago [-]
If they try something and it doesn't work and they then try something else and it DOES work that's concrete feedback.
what 12 hours ago [-]
Are you saying the machine can tell you what confused them?
simonw 10 hours ago [-]
Yes. Try a prompt like this:
> In our recent conversations were there any instances where you attempted a tool call, got an error, then adjusted your tool call and were successful? Suggest improvements to the tool instructions that would have avoided the initial error.
A year ago I don't think this would have worked. The frontier models have been trained to be able to write prompts for themselves, mainly so they can effectively use sub-agents.
datadrivenangel 13 hours ago [-]
they're surprising good at this. The next tokens after a confusing string of tokens when asked to explain the confusing parts are pretty accurate!
benswerd 17 hours ago [-]
One strategy I like for this is a notes field on every single MCP route. Lets the AI optionally just decide to tell you things.
tekacs 12 hours ago [-]
I love this, although I can't help but think that a lot of agents will - for better or for worse - send you a bunch of PII.
ktanishqk 16 hours ago [-]
a feedback tool idea is good (mintlify landed on smth similar) but it's just the starting point. what we found is feedback < support. agents love resolution because anything less is still a blocker in their workflow. and it can't be MCP specific because your curl and CLI users need the same surface.
ventana 17 hours ago [-]
I don't like the idea of making APIs effectively unusable for a human developer. We might not write a lot of code anymore, but getting rid of defaults and asking agents to pass all possible values explicitly makes it impossible to quickly debug the API call (e.g. with curl or something).
It's similar to HTTP/1: there are a lot of headers in the protocol, but you can still use nc or openssl s_client and type the request manually; you probably only need Host: and Content-Type:, maybe Content-Length:, and it will work. I don't normally write HTTP requests in the terminal, but I know I can do it if I need. It's better to keep it this way, I think.
Same with APIs.
bigstrat2003 12 hours ago [-]
Also, any competent developer still writes tons of code. The "agentic era" is completely overblown.
benswerd 11 hours ago [-]
I'll say this: as a very competent engineer I have gone weeks at a time without handwriting a single line of code in the past few months. It is where the industry is going.
gnabgib 11 hours ago [-]
[flagged]
benswerd 11 hours ago [-]
I’m confused what you mean, I post on either subdomains of my own website for personal projects or blogs on my company website freestyle.sh.
My twitter is @benswerd.
What are you saying?
towardtype1 12 hours ago [-]
Where do you see this? I’ve not found this to be true since at least Xmas.
_pdp_ 15 hours ago [-]
I had to deal with exactly this issue with one of my recent oss projects so I can share a few things on the topic.
1. Text is the default interface
i.e. the api must be text based first but it should allow to fallback to structured output by using the accept header.
That being said, it is not wrong to introduce other means to return json by using ?format=json etc.
2. Make it grepable
Basically surface as much useful information in a single line so that the agent can grep and slice.
3. Identifiers must be short
i.e. short enough to describe in 5 tokens but not too short to introduce collisions or confusion.
Otherwise you could be wasting a lot of token for nothing. However, adding prefixes helps like cus_abc123, token_xxx, etc. The prefix can help with lookup, error correction and deduplication.
4. Surface information that is likely to be used by the agent
i.e. if the agent is asking for a list of resources, don't just return the list but also some additional information that might help the agent understand better the context around the resource.
Without this a single task could take a lot more steps simply because the agent needs to run its own loop - it is slow and expensive.
5. Add bulk operations
It is a lot easier to insert 10 records in one request then performing 10 separate requests.
6. Error messages should be descriptive
Ensure that error message point to actual docs and manuals that can be read by the agent so that it can troubleshoot on its own. Also return hints.
I've added most of these at https://github.com/crmkit/crmkit which is highly experimental CRM I tried to specifically design for AI agents. We use it internally for a few projects and it is not perfect but I think it might be on the right direction.
It's nice to see more people are exploring this topic.
hankbond 16 hours ago [-]
Re: default values are bad
One of my recent projects has contributable code (via extensions) with a central settings management (json file). What I do is take all the defaults contributed by the extensions (namespaced by extension name) and materialize them to the settings file as the initial values to each setting. When an agent wants to edit the settings, it already knows the entirety of the setting surface area and thier values. It never has to go digging to find default values nested somewhere in the extension code or documentation. Also, the code that reads the extensions into the framework (for runtime execution) only reads the materialized values, it doesn't have a concept of a default value past the point of materialization. Defaults only exist in the extension registration/mounting part of the lifecycle (so they can materialize if missing from the settings file).
One upside of this is as new settings are created (from new features being developed) your configuration gets notified via the new fields being materialized into your settings upon startup.
AFAIK this is not at all a new practice. I used to see TOML files with default values in commented out lines all the time.
The downside is that the agent is not aware of which settings are being overridden from the default (like a sparse settings file would provide), but I don't know how much semantic value that offers in most cases (other than maybe debugging?).
I'm not even taking a side on this one this is just a pattern I decided to take for this use case.
hdjrudni 9 hours ago [-]
I did that too but not for the sake of the LLM, just for myself. Massively helps discoverability. It also protects from backwards-compat if you ever want to change a default without breaking existing users, knowing that whatever they had before has already been saved into their settings so some setting doesn't randomly change on them.
cheekygeeky 17 hours ago [-]
Suggestion:
Save the article as an .md file.
Upload it to Fable 5 with the prompt: "Agree or disagree. Be verbose."
I learned a lot: Where he's right. Where he's wrong. A 'delicious bug in his own example" code (command injection vulnerability - in the code sample used to demonstrate why you don't need sandbox.git.clone). I also learned that, according to Fable 5: "A Typescript SDK is the strongest anti-hallucination device we currently have." And a lot more.
benswerd 16 hours ago [-]
Just did this, found result fairly interesting, I reject most of its objections on the basis of bad code. When I say Im pro explicitness I'm also pro comment, and pro separating the "this is core fact" and "this is configuration that i don't care too much about". Will use this test for future writing.
rgbrgb 17 hours ago [-]
in my experience models will answer introspective questions decisively but can't actually introspect. this is pretty similar to humans (e.g. ford's faster horses quote).
with AI stuff you can actually run real tests though.
ayende 17 hours ago [-]
> Defaults are bad. Agents can be expected to read the documentation, register what good starting values are, and fill them all in, in place.
That is pretty bad, when you now need to look at this code, and you have 95% of the output being craft, but sometimes it isn't, and you need to understand the difference between each.
A great example of that is:
int fd = open("log.txt", O_RDONLY);
Versus:
// 1. Manually build and initialize the Security Attributes structure
SECURITY_ATTRIBUTES sa;
sa.nLength = sizeof(SECURITY_ATTRIBUTES);
sa.lpSecurityDescriptor = NULL; // Explicitly no custom security descriptor (inherits default)
sa.bInheritHandle = FALSE; // Explicitly state this handle cannot be inherited by child processes
// 2. We need a handle for the template file parameter.
// Win32 requires this to be an active file handle opened with GENERIC_READ, or explicitly INVALID_HANDLE_VALUE.
HANDLE hTemplateFile = INVALID_HANDLE_VALUE;
// 3. Now we call CreateFile with every single parameter fully populated
HANDLE hFile = CreateFile(
"log.txt", // 1. lpFileName: The file we want to open
GENERIC_READ, // 2. dwDesiredAccess: Read-only access
FILE_SHARE_READ, // 3. dwShareMode: Prevents any other process from writing to it
&sa, // 4. lpSecurityAttributes: Pointer to our explicitly defined struct
OPEN_EXISTING, // 5. dwCreationDisposition: Only open if it already exists
FILE_ATTRIBUTE_NORMAL, // 6. dwFlagsAndAttributes: Normal file, no special caching or async flags
hTemplateFile // 7. hTemplateFile: Passing our explicit invalid handle instead of NULL
);
An agent can output the second just as well, sure.
However... which one do you think is better to read or understand?
Did you catch the fact that this is blocking concurrent writers? Or that we expected the file to exist?
Same thing, but actually understanding what is going on is orders of magnitude different.
benswerd 17 hours ago [-]
This is a fair point. Im not opposed to the open function existing, but I do think its valuable to own the underlying implementation of thigns like it (maybe not open) in your codebase so you can review the implementation and reconfigure it to your precise requirements.
cco 13 hours ago [-]
Having just joined Twilio (thanks for buying us) and designing some new APIs right now, it warmed my heart to see messages.create(), written over 15 years ago, show up as an example here :)
There are a lot of ideas swirling around right now, it has been the most anybody has cared about OAuth in 15 years.
Most auth companies are playing in this space, some are trying to own the spec, like WorkOS, some are working more closely with standards bodies.
I, like the other commentator, am of the opinion that 90% of the problem space is solved with OAuth and the remainder will be worked out over the next few years as we see how autonomous agents evolve.
benswerd 19 hours ago [-]
MCP Auth is just Oauth, its designed for humans to authenticate their sessions for connections.
TBH I know nothing about A2A.
Agent Identity and Authz is a different problem, allowing agents to operate independently from humans with granular permissions is coming/whether from these protocols or others, and when it does I think CLIs/CLI Device Auth which is used as a rough proxy for this where the agent just takes your identity will finally go away.
cheekygeeky 17 hours ago [-]
[flagged]
mohamedkoubaa 19 hours ago [-]
>Defaults are bad. Agents can be expected to read the documentation, register what good starting values are, and fill them all in, in place.
This is not what defaults are for.
If, for example, you are writing a replacement CLI for git, *for the love of God and all that is holy* do not force agents to read the entire documentation and pass a value for every possible parameter
Because the defaults aren't just for convenience, the API designer is also making the parameters they think should be used the most have the least resistance. Good example is runtime parameters like in the JVM. You shouldn't start with having to tune your JVM, you probably want a middle of the road place to start with even if you know you're going to tune it.
benswerd 17 hours ago [-]
im not opposed to good defaults, i just believe they should be explicit. The AI should read the starting.md to fill in explicitly what its JVM configuration is. Then, when you want to tune in the future its clear what options are available and what specifically is changing.
grim_io 4 hours ago [-]
I can't parse this. If you explicitly define recommended values, those are not what I consider "defaults".
tyre 18 hours ago [-]
Because LLMs degrade with context length. And even if they didn't, having to constantly ingest the same stuff is wasteful for execution and cost. Why stuff the context when you don't have to?
benswerd 18 hours ago [-]
Conservatively speaking, LLMs degrade past 40% of a 2M context window, 4k tokens is 0.2%, so no degradation there. Thats also current generation conservative estimate.
I think wasteful is an irrelevant metric. Claude ingests those tokens in a quarter of a second, if it causes it to catch any bug ever it saves far more time than it ever uses.
Any individual default that causes unexpected behavior causes more problems, takes more time and costs more than the small cost of being explicit.
The example I heard was an MCP with a "feedback" tool which had a tool description saying that coding agents should call that any time they had trouble figuring out how to use the rest of the MCP.
I really like this. It's super cheap to implement and I expect you'd get a bunch of actionable signal in amongst the noise.
What they did was kind of clever. When the agent calls the old API/uses old args, I guess they told it to tell the human to fix their skill and 'upgrade' to the new API. They even provided a skill upgrader skill.
Anyway, my agent tried calling it the old way, must have received some sort of error message and then diligently reported to me that my skill needed upgrading and we should be doing it some new way.
Kind of like reverse feedback I guess? Not the AI giving feedback to them, but them giving feedback to me via my AI.
And also files issues for blockers
I’ve absolutely caught things and made improvements just from skimming them occasionally - they are particularly useful when you get a PR that makes you scratch your head
But I’m definitely not taking full advantage of all the feedback coming in yet
I have to imagine parsing signal from noise there is a massive challenge when it’s other agents that are using your MCP and not just your own
“I existed for one word. The entire point of my life was that my process boot side-effected a module reload. 'ok'."
“ "The overhead-to-work ratio was astronomical."
“the koleo purchase turned into a boss fight with an undismissable Vue modal and a WKWebView that kept eating its own cookies."
I have a reflections mechanism, which launches right after the session ends (when it’s still in cache, so it’s almost free) and it catches a ton of problems with tool/skill/system design, all the time. Low level stuff can be fixed almost automatically (e.g. agent funbled 10 turns bc tool’s docs were wrong), high level too but human gating is useful.
If you try doing it any other way you have to pay for input tokens, not cache read, so it’s 10x more expensive.
It feels like it would be relatively random across the range of plausible results, but it sounds like you are finding otherwise.
> In our recent conversations were there any instances where you attempted a tool call, got an error, then adjusted your tool call and were successful? Suggest improvements to the tool instructions that would have avoided the initial error.
A year ago I don't think this would have worked. The frontier models have been trained to be able to write prompts for themselves, mainly so they can effectively use sub-agents.
It's similar to HTTP/1: there are a lot of headers in the protocol, but you can still use nc or openssl s_client and type the request manually; you probably only need Host: and Content-Type:, maybe Content-Length:, and it will work. I don't normally write HTTP requests in the terminal, but I know I can do it if I need. It's better to keep it this way, I think.
Same with APIs.
My twitter is @benswerd.
What are you saying?
1. Text is the default interface
i.e. the api must be text based first but it should allow to fallback to structured output by using the accept header.
That being said, it is not wrong to introduce other means to return json by using ?format=json etc.
2. Make it grepable
Basically surface as much useful information in a single line so that the agent can grep and slice.
3. Identifiers must be short
i.e. short enough to describe in 5 tokens but not too short to introduce collisions or confusion.
Otherwise you could be wasting a lot of token for nothing. However, adding prefixes helps like cus_abc123, token_xxx, etc. The prefix can help with lookup, error correction and deduplication.
4. Surface information that is likely to be used by the agent
i.e. if the agent is asking for a list of resources, don't just return the list but also some additional information that might help the agent understand better the context around the resource.
Without this a single task could take a lot more steps simply because the agent needs to run its own loop - it is slow and expensive.
5. Add bulk operations
It is a lot easier to insert 10 records in one request then performing 10 separate requests.
6. Error messages should be descriptive
Ensure that error message point to actual docs and manuals that can be read by the agent so that it can troubleshoot on its own. Also return hints.
I've added most of these at https://github.com/crmkit/crmkit which is highly experimental CRM I tried to specifically design for AI agents. We use it internally for a few projects and it is not perfect but I think it might be on the right direction.
I hope this helps.
It's nice to see more people are exploring this topic.
One of my recent projects has contributable code (via extensions) with a central settings management (json file). What I do is take all the defaults contributed by the extensions (namespaced by extension name) and materialize them to the settings file as the initial values to each setting. When an agent wants to edit the settings, it already knows the entirety of the setting surface area and thier values. It never has to go digging to find default values nested somewhere in the extension code or documentation. Also, the code that reads the extensions into the framework (for runtime execution) only reads the materialized values, it doesn't have a concept of a default value past the point of materialization. Defaults only exist in the extension registration/mounting part of the lifecycle (so they can materialize if missing from the settings file).
One upside of this is as new settings are created (from new features being developed) your configuration gets notified via the new fields being materialized into your settings upon startup.
AFAIK this is not at all a new practice. I used to see TOML files with default values in commented out lines all the time.
The downside is that the agent is not aware of which settings are being overridden from the default (like a sparse settings file would provide), but I don't know how much semantic value that offers in most cases (other than maybe debugging?).
I'm not even taking a side on this one this is just a pattern I decided to take for this use case.
with AI stuff you can actually run real tests though.
That is pretty bad, when you now need to look at this code, and you have 95% of the output being craft, but sometimes it isn't, and you need to understand the difference between each.
A great example of that is:
Versus: An agent can output the second just as well, sure. However... which one do you think is better to read or understand?Did you catch the fact that this is blocking concurrent writers? Or that we expected the file to exist?
Or what about:
Versus: Same thing, but actually understanding what is going on is orders of magnitude different.MCP and A2A weren't enough?
Most auth companies are playing in this space, some are trying to own the spec, like WorkOS, some are working more closely with standards bodies.
I, like the other commentator, am of the opinion that 90% of the problem space is solved with OAuth and the remainder will be worked out over the next few years as we see how autonomous agents evolve.
TBH I know nothing about A2A.
Agent Identity and Authz is a different problem, allowing agents to operate independently from humans with granular permissions is coming/whether from these protocols or others, and when it does I think CLIs/CLI Device Auth which is used as a rough proxy for this where the agent just takes your identity will finally go away.
This is not what defaults are for.
If, for example, you are writing a replacement CLI for git, *for the love of God and all that is holy* do not force agents to read the entire documentation and pass a value for every possible parameter
The docs for git clone at https://git-scm.com/docs/git-clone are less than 4000 tokens, I don't think this is unreasonable.
I think wasteful is an irrelevant metric. Claude ingests those tokens in a quarter of a second, if it causes it to catch any bug ever it saves far more time than it ever uses.
Any individual default that causes unexpected behavior causes more problems, takes more time and costs more than the small cost of being explicit.