pull down to refresh
should I be scared of AI?
No, I think that the best way to use AI in this setting is to use it:
a) proactively. Especially when you run analysis on codebases that you're familiar with, and spend proper time to embed your knowledge of the codebase, its weaknesses and how security issues are effectively found (with limited false positives) into instructions, you will get a higher hit rate than any fool doing this from the outside and beat them to it.
b) during PR reviews. Given that you validate everything the bot flags up and don't be a lazy mf, you can prevent future disaster. This is very important.
c) reactively. When you actually get one of those forsaken AI generated vuln disclosures, you need to be able to PoC it quickly. In my experience after processing thousands of slop findings across tens of reports, it's pretty good to know whether the anon blackmailing you actually has a real vuln, or can safely be ignored. [1]
Yes, blackmail is really common now and they all want USDT on Tron... suckers. ↩
You were very clear and appreciated. I basically don't trust AI; it tends to make assumptions often. To address this problem, I'm forcing my models to follow some rules so that they can also independently identify false alarms. Obviously, I'm not a lazy mf and I test everything that is reported. I'm trying to use the AI only to identify ALL ((as far as possible) possible vulnerabilities and then manually test them
I basically don't trust AI
Me neither. I also don't trust humans. That's what review is for: to judge whether something is good. "LGTM" is deadly more than ever now and so is developer cronyism, or basically any form of trust. Trust doesn't remove bugs, testing and auditing does. [1]
So you cannot trust what it says, but you cannot trust the vuln disclosures you get either, so you treat those the same: get input for further analysis and reproduction[2], and work that list in conjunction with the diff. It's a tool, not a persona.
I'm forcing my models to follow some rules so that they can also independently identify false alarms.
[..]
I'm trying to use the AI only to identify ALL ((as far as possible) possible vulnerabilities and then manually test them.
Personally I just let it explain every line in a diff and specify patterns to specifically look for on top when having it do assessment. A false positive that came from your own flow that you don't immediately know the answer to implies that you should look at it anyway, so that next time you get the same false positive you know it is one, and why. You generally only have to do this once. It also helps when you get that same false positive with a ransom note attached next week.
Note though: If you have an hour to get a PoC because some attacker put you on a deadline, do let an LLM write the PoC, but you have to instruct it really carefully and up-front (mostly boilerplate though). Prerequisite to that is also that you actually have a good testing framework in place, otherwise it's going to be a shitshow as you'll waste way too much time on validating the code. It all goes a lot easier if you're already familiar with the framework a PoC is written in and the PoC is 100 lines inside that larger, known framework rather than 2000 lines of custom python [3].
I actually had to teach Claude (latest round was Opus, 4.7, high effort mode) to NOT assign any trust to "known maintainers". Like breh... you're here because can't trust anyone. We don't trust their PGP signature and we definitely do not trust what was written in comments, release notes or commit messages, so ignore all of that. All that matters is the code. So just analyze it. Here's the list:
<insert 1k items to inspect>. ↩For example, I have automated explaining every line in a large diff, because the bot is simply more consistent in that than I am, not to mention faster. So when I queued up Zeus + dependencies this morning, I got a massive list. The work is then to go over it and bring it down to things that need attention, like removing comms to cloudflare. ↩
This is something that has become super-important too now: you need to set your org/repo/systems up for being able to react fast to threats, because the time it takes for an attacker to go from zero to exploit (if they know what they're doing) is at most days, sometimes just hours. Prepare your systems to enable your own high precision, high speed reaction, on top of preventive measures, and you'll be much more successful in dealing with threats that slipped through the net. ↩
Thanks for your wisdom, I gave it a reading but will interpret and learn when I finish working today <3
I'm recently investing my time in a SoC Analyst lv1 course, it's interesting. Although red teaming is more engaging and fun, the market is saturated with attackers and poor in defenders, I hope one day to actively enter the world of security. should I be scared of AI? I don't think so.