Rendered at 07:20:14 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
goranmoomin 10 hours ago [-]
I do have a hunch in that we might be able to utilize tiny LLMs to figure out parts that might possibly be brittle and combine them with traditional generation/mutation-based fuzzing to generate fuzz targets that are more likely to trigger an edge case.
I did not think of applying LLMs on fuzzing at all until I saw llvm-hackme[0] which does both traditional mutation fuzzing as well as LLM-generated targeted regression test cases, where the LLM is pretty effective in understanding the PR and targeting edge cases! It was pretty impressive and I keep getting to think on how we can actually combine LLMs to make fuzzing much more efficient & effective.
// Sorry about the yet-another-LLM-comment. I really love PLs and I'm terribly sorry that I'm contributing yet another LLM-related content (instead of the more interesting stuff!)
I love good programming languages, if that counts for anything, and I think there's a huge amount of room and value in "using an LLM to translate one programming language to another".
I also think your instincts are on a useful path. Perhaps using a very small and stupid LLM to generate plausible-sounding-but-probably-wrong programs might be a path to generating interesting test cases?
WalterGR 7 hours ago [-]
Any fuzzer needs to compare its results to AFL (American Fuzzy Lop), "a free software fuzzer that employs genetic algorithms in order to efficiently increase code coverage of the test cases." https://en.wikipedia.org/wiki/American_Fuzzy_Lop_(software)
At one point it was considered state-of-the-art. As a project it's since been superseded by AFL++ - https://aflplus.plus/ .
daniellionel 13 minutes ago [-]
very cool! will check that out. (hi, author here)
nwellnhof 7 hours ago [-]
> We can compare the output of the same program for both targets and flag any differences.
This is called differential fuzzing and is one of the most powerful methods to find bugs in all kinds of software.
stephenlf 15 hours ago [-]
Fantastic article. Thanks for sharing. I love the honest take on LLM-based fuzzing. It’s exactly something I would do with a similar problem.
slowhadoken 11 hours ago [-]
Writing code with an LLM seems like a deal with the devil but debugging logical errors or type coercion bugs is hell.
I did not think of applying LLMs on fuzzing at all until I saw llvm-hackme[0] which does both traditional mutation fuzzing as well as LLM-generated targeted regression test cases, where the LLM is pretty effective in understanding the PR and targeting edge cases! It was pretty impressive and I keep getting to think on how we can actually combine LLMs to make fuzzing much more efficient & effective.
// Sorry about the yet-another-LLM-comment. I really love PLs and I'm terribly sorry that I'm contributing yet another LLM-related content (instead of the more interesting stuff!)
[0]: https://github.com/dtcxzyw/llvm-hackme
I also think your instincts are on a useful path. Perhaps using a very small and stupid LLM to generate plausible-sounding-but-probably-wrong programs might be a path to generating interesting test cases?
At one point it was considered state-of-the-art. As a project it's since been superseded by AFL++ - https://aflplus.plus/ .
This is called differential fuzzing and is one of the most powerful methods to find bugs in all kinds of software.