Test/verification technique effectiveness with Astra

(danluu.com)

19 points | by luu 23 hours ago

3 comments

  • reedf1 17 minutes ago
    The really powerful thing you've done here is begin to create a taxonomy of agent patterns. Given we are pretty much all engineers in this space; I am shocked at the lack of engineering taking place. People are picking patterns they find intuitive; for various invented reasons, this is fine in the beginning, but we need to start objectively measuring their performance. It needs to be said that there are an infinite number of plausible agent patterns, and we should view all (from skills to MCP to Audit, whathaveyou) as something that may go the same way as netscape until there is real justification.
  • kooi 28 minutes ago
    Nice post. Good motivation to look more at formal verification. There was that Aristotle post today which seems like a prompt driven, plug and play verifier using lean.

    Makes me think that LLM assisted formal requirements generation for a roll your own "Aristotle" is possible.

    Just thinking about our own projects in the signal processing domain. A nice path could be: human language -> temporal logic specs for black and white box run-time testing.

    There are code generators which will turn formal logic into a run time monitor. In the past, I've never used them past toy examples because its so verbose.

    > The point of this post is to demonstrate that it’s possible in the era of AI coding to still make decisions based on evidence rather than vibes.

    That's the difference between an engineer and a script kiddie.

    Domain expertise will always be valuable.

  • ArtRichards 11 minutes ago
    I use what I would say is a TDD skill, but I couldn't find anywhere in the article what you meant by TDD? How did you prompt it to use TDD? Why was it less effective? I feel targeted ;)