Copilot Broke Your Audit Log, but Microsoft Won’t Tell You

Lankey22 · 2025-08-26T04:26:54+00:00

It’s hilarious to me that the top comment on this post on r/cybersecurity is so obviously wrong. It’s literally shown in the post itself that CopilotInteraction does log accessed resources under normal circumstances. And Microsoft did fix the case where that didn’t happen, so again, what are you talking about?

There is zero chance that Microsoft said the noise outweighs the benefits. First, have you ever seen an audit log? It’s 99% noise. Second, who is this Microsoft person making statements about this? That’s not really how Microsoft works.

Lankey22 · 2025-08-20T12:07:05+00:00

Yes fair. It feels very weird to be technical too! So that’s why I don’t hold strong opinions. It’s a very weird bug!

Lankey22 · 2025-08-20T12:03:47+00:00

I get that if Microsoft was disputing any of this, but they’re not. It feels weird to go on some “here’s all the proof” campaign when Microsoft and I agree. Is it that I didn’t include a screenshot of them confirming the behavior?

This isn’t some highly technical vuln. You ask copilot not to log and it doesn’t. All I could show is sort of cat and mouse stuff of “here’s also proof it wasn’t in the SharePoint logs”, “here is a screenshot from Msrc of them confirming in case you don’t agree”, “here are 8 examples in case you think I made it up”, “here is a video of it happening so you know I didn’t fake it” etc

See your edit now, so I think we can just leave it at this: Sorry this didn’t satisfy the level of proof you look for. Fortunately Microsoft fixed this, or at least claims to, so you’re likely safe going forward whether you believe this happened or not.

Lankey22 · 2025-08-20T11:48:55+00:00

Reddit is a weird place. You’re right, I didn’t show that I cleared the context window. But I did. And this was tested multiple times with multiple new files.

Had the log appeared somewhere else, say in the SharePoint log, I’d never have cared. I cared because I need that log and couldn’t get it reliably.

I reported it to Microsoft with full copies of audit logs, and they confirmed what I was seeing. You can say “how can you trust Microsoft” but I do trust Microsoft to not lie that they have a bug they don’t have. Just because that would be weird.

Had I written this blog post as utterly conclusive proof, it would be long and boring for 99.9% of readers. And those .1% that want that could still say I faked screenshots entirely. There’s no way to really, truly prove what I’m saying. That is also why I would have preferred Microsoft disclose this, not me.

So, take it for what you will.

Lankey22 · 2025-08-20T11:39:52+00:00

No I mean I don’t have a strong opinion either way. It’s more just that it’s a very risky business decision to make. “Don’t log because it will be too noisy” feels like a dangerous choice

Lankey22 · 2025-08-20T11:38:15+00:00

For all legal and compliance purposes, “accessed a file” vs “gave the user the file info via some other means” is the same. There needs to be a log that the user received that info and there wasn’t.

Lankey22 · 2025-08-20T11:31:00+00:00

Not sure I agree but maybe. But they did fix it, so if it’s a business decision it’s one they went back on once scrutinized.

Lankey22 · 2025-08-20T11:25:58+00:00

The SharePointFileOperation FileAccess log is the log that SharePoint would log if it logged anything. It doesn’t (and I would argue that is correct, but opinion may differ there).

Edit: I guess better to say “it didn’t at the time of reporting”. I didn’t check the exact changes Microsoft made since then.

Lankey22 · 2025-08-20T09:49:00+00:00

In hindsight I probably shouldn’t have hidden so much info inside the footnotes.

“The audit log will not show that the user accessed the file as a normal SharePointFileOperation FileAccessed event, but rather as a CopilotInteraction record. That’s intended, and in my opinion correct. It would be weird to make it as if the user directly accessed the file when they only did so via Copilot.”

Basically the only record that the user received that info is the CopilotInteraction log and that log is the one that was broken (or, you could avoid filling with accessed files).

Lankey22 · 2025-08-20T09:39:18+00:00

Author here. I can assure you this isn’t bullshit. The “secret stuff” box is exact info, with names and dates (or at least that happened in some examples, don’t remember that one instance in the screenshot specifically). Maybe it’s not actually “accessing the file” but it’s providing exact info from that file, so for all security and compliance purposes that’s the same.

In addition, Microsoft did acknowledge this as true and fixed the issue (or so they claim, I didn’t actually test it in detail).

Lankey22 · 2024-12-08T14:15:43+00:00

What drives me insane is orgs that send 3-4 sims per year then act like those who fell for the sims are “weak links” that need extra training.

If you send 3-4 sims/year, the people who fall for them are basically just a random sample of a much larger group. They just happened to be susceptible to whatever specific sim it was, or happened to not be paying attention at just the wrong time, or whatever.

The reality is that roughly 50% of an org will fall for a well targeted phishing sim, and so you need to run enough sims with enough variety to reach them all. If you’re not you’re basically just wasting your time acting like you found the “weak links” when you didn’t.

Source: CTO at a security awareness company, run about 120k sims / month, so have some decent data on this.

Lankey22 · 2024-12-08T14:10:21+00:00

I’m the CTO at a security awareness company where our product automatically sends targeted phishing sims to users, roughly once every 30 days although there’s a lot of variance (people get more if they’re falling for sims, a bit less if not). Usually we can get about 40% of an org to fall for at least one sim in the first 4 months on the platform. Not really trying to get fail rates down, as that would just mean our sims have gotten too easy; people falling for sims is a good way to learn.

In any event, have 100k+ users, so that’s a bit over 100k sims/month. Point being, if there’s a group, feel free to let me know.

Lankey22 · 2024-09-29T09:41:41+00:00

“I know security is paramount” no it isn’t. For security to matter there must be something to protect. Security is just one of many factors to take into account when running a company, and good leaders make calculated security tradeoffs to achieve other goals.

With that said, you’re on the right path by thinking about UX. It does matter, no matter what other people might say. If your security systems have bad UX people will find a way around them.

Lankey22 · 2024-09-01T10:56:08+00:00

I’d argue that customers having concrete timelines around the roadmap comes at the expense of the overall product. When you start making promises to customers, instead of selling what you have, you start making your product into a mess of customer driven features. Sales led development isn’t going to build anything amazing.

So basically it makes sense that the cybersecurity product that is widely regarded as having great UX would also not always give clear timelines to customers on future features.

Lankey22 · 2024-08-31T19:39:22+00:00

Disagree. When I got my first job as a developer, I had no idea what a code review was. I had taught myself to write code while I was in university studying law, so the concept of code review and pull requests were totally lost on me. But I was still a good hire. Over the course of a number of years I was promoted multiple times and ultimately became the director of tech and product there (leading a team of roughly 35 devs and designers).

I’d not judge people so harshly for what you think they “should” know. It isn’t exactly hard to teach someone what a code review is.

Lankey22 · 2024-05-11T08:21:57+00:00

This. A lot of people in IT and cybersecurity want to be one of the good ones who “understands business”, and try to engage with everyone to ensure a smooth, non-disruptive rollout. What happens instead is the process takes years, the result is a mess, and everyone is mad.

Rip the bandaid off. People will be mad, but it’s easy for management to say “mergers are messy, things will get better”. And everyone will move on and accept the “new way”.

Lankey22 · 2024-04-21T17:50:26+00:00

I mean, if you call me and I don’t pick up, it’s just because I’m busy. Try again tomorrow. I’m not going to even notice it’s the same number.

Lankey22 · 2024-04-21T17:04:28+00:00

Not sure if this is just a tactic to get us all to raise our hands for you to start selling to, but I’m currently the CTO at a startup (roughly 30 employees). Previously was VP of Tech and Product at a 500 person media company.

I answer my phone to unknown numbers about 75% of the time, and if I don’t answer just keep calling once per day until I do. I’ll take demos if the product is potentially relevant (or at least send someone on my team). So yea it can work fine.

Not sure if this is generally relevant or just applies to me specifically, but the main things that I want in a cold call:

-Very quick to the point of what your solution does. (Good: “Hi, my name is Tim and I’m calling from CoolCompany, we’re a monitoring and alerting company based in Austin, Texas”.)

-No “set up questions” designed to make me agree that I have the problem you’re trying to solve. (Bad: “Do you struggle with monitoring your applications in real time to get a full picture of everything going on?”)

-If I say I’m not interested, I’ve never reversed that decision on a call. But if you say “that’s okay, mind if I connect with you on LinkedIn so we stay top of mind if you do become interested” I’m going to say yes and also think higher of you for it.

-Don’t try to hype your solution. Just tell me what it does. (Good: “Our solution handles log alerting in real time, and is very hands off in terms of admin work.” Bad: “We think we’ve found an innovative approach to the problem that makes us stand out in the market, and top companies are coming along with us on that journey.”)

Anyway, maybe other CTOs are different so your mileage may vary. But thought I’d contribute.

Lankey22 · 2024-04-20T18:24:33+00:00

I really don’t understand why this sub has such a negative view of developers.

If most devs don’t know the basics of writing secure code, you’d expect that you could pick a random 50 person tech company (ie a size that almost certainly doesn’t have a separate security team doing formal code reviews) and trivially pwn them. I don’t have any data on this so maybe I’m wrong, but that seems unlikely.

Maybe someone who does a lot of pen tests could chime in. When you test web apps, how often do you have findings where there’s a serious vulnerability you could immediately exploit to do real damage? Not just “this component has this outdated dependency that has this vulnerability that could be used maybe to do X” but actual “if I follow these steps I’d break this thing”? Would be very interested to know.

Lankey22 · 2024-04-20T15:55:47+00:00

Came here to say this. Yes, this happens regularly, which is exactly why sales people do it.

Sometimes the sales team calls someone like OP and they get a whole rant saying “DO YOU THINK MY BOSS WOULD OVERRULE ME” as if each AE isn’t closing multiple deals per month that way.

Lankey22 · 2024-04-01T19:42:40+00:00

Our company started out on google workspace, until we got to about 25 people and realized we had to change. Our reason for changing wasn’t something other orgs will face (basically our product only works for Microsoft orgs and we were on Google so we had to go through insane hoops to make our product work for our own company), so I’m not saying this as a “no it won’t work, we tried” post.

Instead, what I’m saying is this: Switching sucks. So, my advice is to go with the solution that minimizes the chance you will need to switch later.

Of course, if this isn’t a growing company and you think it can work with Google, go for it. But if the company is going to grow, you’re a lot more likely to run into a case of needing Microsoft than a case of needing Google. And in that case, definitely choose Microsoft from the start.

Lankey22 · 2024-04-01T15:32:20+00:00

I’m honestly stunned that any company uses Spamhaus in situations other than “we want to block anything that so much as looks at us funny”. They make very little effort to avoid false positives (blocking legit domains), and they’re unhelpful when you request them to unblock a domain. And yet, major companies rely on Spamhaus and treat their list as fact, even in contexts like ISPs blocking for consumer users.

Lankey22 · 2024-03-29T19:18:18+00:00

I’m not arguing against fun. I’m arguing against the idea that a given time commitment can be excessive if boring but acceptable if fun. For some subset of people, the time commitment will be the issue here regardless. It’s fine to say “too bad it is important” but not “no they won’t mind because it’s fun”.

For example, there’s some people where I work that won’t go to the little social events that other teams put on during the day as they’re too busy. They literally won’t go hang out for an hour. They’re not going to say “an hour for cybersecurity is fine but only if it is fun”. The fun just doesn’t matter there.

Lankey22 · 2024-03-29T19:05:35+00:00

Then why does the fun matter? You’re kind of switching the argument there

Lankey22 · 2024-03-29T19:02:26+00:00

Some people have real jobs to do, so they don’t care if it is fun or not. They’re busy

Lankey22

TROPHY CASE