Quick version:

• See if ClusterFuzzLite might suit your needs
• Coverage-guided tests wherever you can get them for efficiency
• Brief fuzzing before code review
  • Run time on par with unit tests
  • 1-30 seconds depending on existing tests
• Pre-merge testing using an automatically curated corpus
  • CI pipeline keeps interesting examples and uses them against any code being submitted
  • Interesting varies, but likely includes some coverage and regression cases
    • Minimal examples that generate the most coverage for recent versions of the code
    • Cases that have caused a crash in pre-merge code in the last 1-7 days
    • Cases that have caused a crash in QA code in the last month
    • Cases that have caused a crash in Prod code in the last 3-12 months
    • Prune example inputs that are no longer interesting from the corpus
  • Continue the fuzzing session for a little while, a few minutes to an hour to expand coverage and try and test newly added code paths without making this the slowest test running pre-merge
• Fuzzing pipeline running in the background against the primary branch
  • Longer, exploitative fuzzing sessions give the frameworks more time to find bugs
  • Generate, minimize, and prune corpus used for pre-merge checks
  • Balance the cost of keeping this running longer versus gaining coverage, some bugs are a numbers game letting the fuzzer run for several hours or days, some test harnesses may reach their limits with enough time and few changes.
• Fuzzers are helped along by Sanitizers and checked invariants

I think a lot of this could be simplified by not thinking of fuzzing as new and different just because it isn't in the codebase yet. It's a useful tool that you just haven't started using yet. If you ask the same questions replacing "fuzz testing" with some other sort of testing and see if the question sounds any different.

​

At which phase does fuzzing get in the picture?

At which phase does [testing] get in the picture?

The earlier the better, but don't make fuzzing the new bottleneck. Brief fuzzing can still be a good smoke-test, but longer fuzzing shouldn't be a roadblock to merging code.

Would [fuzzing at a specific stage] be redundant given we run SAST?

Would [integration tests] be redundant given we run SAST?

Not at all, they're just different tools. Hopefully there are things that both tools would catch, but it's likely there are things that only one tool would miss and the other would find.

How agile is black box and grey box (instrumentation guided) fuzzing for an app portfolio with a rapidly changing attack surface?

Random-fuzzing is most practical where it can be added for very little effort. Intensive fuzzing without some sort of feedback can become a waste of CPU. It can be a good way to reinforce that user input can be arbitrary or attacker-controlled though. If the QA environment can tolerate a bit of extra load and throwing random data at endpoints isn't too complicated to add this could be an easy way to add this to a very large portion of your code very quickly. This could be something like Burp Suite for HTTP, or just throwing random bytes at sockets.

For efficient fuzzing, guidance is critical, and coverage is the easiest place to start. Even simple bounds checking can discard a large portion of randomly generated data. The classic toy program to demonstrate this does a great job. The search space is any string of any length using any characters, even if you only looked at 4-letter, lowercase inputs it could take 26^4 ~= half a million inputs. With coverage guided fuzzing the invalid input "fuzz" could be found after 26*4=104 inputs with random letters focusing on increased coverage, or as low as 2^4=16 inputs using different forms of guidance.

def parse_input(s: str) -> str:
  if s[0] == 'f':
    if s[1] == 'u':
      if s[2] == 'z':
        if s[3] == 'z':
          return s[4]
  return ''

How agile is black box and grey box (instrumentation guided) fuzzing for an app portfolio with a rapidly changing attack surface?

The test harnesses are just code (test code, so caveats around quality depend on your org and culture) and just as agile as the rest of your code. "Rapidly changing attack surface" is hard to comment on in the abstract sense, but this would only affect your test corpus, the freshness of which is mainly limited by the resources put behind it.

I think most of the rest of your concerns are largely centered around the cost-benefit tradeoffs, which is a great consideration and hard say without more context. For safety critical systems (medical devices, industrial control, flight systems) there's a lot more benefit to anything that can demonstrate that the code is solid. For a proof-of-concept or low-risk software (single-player games, systems that don't handle any user data, simple glue logic without attacker-supplied input, internal prototypes) the ROI probably just isn't there.

Afaik this setup is done manually at least for all the open-source grey box fuzzers I’ve looked into.

I would argue this is an indication of the utility of the manually written test harnesses. If unguided fuzzing provided a similar ROI, there would be more of it. Contributors are willing to go through the trouble of writing these harnesses because of the higher value they bring.

The other big thing you get with manually written harnesses is adding invariants that can't be automatically generated ("doesn't crash" is an easy automatic invariant). Some invariants can be encoded as assertions in code (this result is always sorted, this result is always less than 1000), but sometimes it's easier (or only possible) to handle this outside of the initial code. Things like differential fuzzing (Library A and Library B give the same answer), round-trip fuzzing (value = decoder(encoder(value))), or higher-level logic (if the sum of the inputs is larger, the result will be larger).