Trade-offs/Experience

[matu3ba]

Usually I try to find the trade-offs of learning yet another tool before using it as to not waste time during debugging.
What classes of problems work and until which problem size/complexity etc?

Otherwise, it would be nice that no data has been collected so far on that.
From what I know so far with LLMs figuring out problems of an existing diff works great and/or if the commit is small and simple enough.
Or what is your experience?

[nicovank]

ChatDBG is an extension to your usual debugger, so it has a low barrier to entry. If you already run GDB or LLDB, with ChatDBG installed, just try running why when encountering a crash before other commands. It costs (almost) nothing and may either fix the issue or give a good lead as to where to start.

Our paper has evaluation and more details on our experience: https://arxiv.org/pdf/2403.16354
In the paper, we evaluate on both smaller student Python programs, and larger C/C++ programs with memory issues.
I would say that results depend more on the class of bug encountered rather than size of project, so YMMV for various types of issues.

Please report bugs or success stories if you use this tool!