We all have seen it, and some of us are living through it, but every organization and many users are making use of AI systems in their day-to-day work. Within Microsoft’s security realm, we have Copilot for Security. I previously provided some insights into how you can deploy and set up Copilot for testing (however, the same steps can be used for production as well). In this post, we’ll discuss how touchy the wording of your queries can be and how a word or two can affect the outcomes of your results.
As always, please be aware of my blog disclaimer
Your Copilot Query Wording Matters
In my honest opinion, what we call AI isn’t truly artificial intelligence. At least not what we have access to. To me, artificial intelligence should be the ability of a program, platform, entity, or whatever you want to reason or guess what you are saying. There is so much more to AI than just that, but for the purpose of this post, that’s one feature I’d like to focus on. Where I am going with this is that right now, an AI interface is really just some highly superior language models sitting on top of even more complex programming with access to data and the ability to skim over that data and provide it back to others, usually in a format that helps the requestor understand the response.
I know I am oversimplifying these tools’ capabilities, but I intend to put them into perspective for those who may have the background.
Because Copilot for Security can’t reason what you are asking, it is going to use best guesses to determine the intent. Unfortunately, these are not always accurate. To illustrate, I was looking for roles in Purview that I could use to give a user access to the information protection solution. Initially, I used more natural language (to me, at least) and asked Purview in the following manner: “Which roles affect Microsoft Information Protection”. If I were to ask that of a colleague, they would understand that I meant which roles in purview would grant access to Information Protection if I added a user to them. Sadly, Purview did not have the same understanding my colleague would have:
As you can see, Purview provided some fantastic information but completely missed the mark of my question. I then thought that perhaps Copilot for Security was not meant for these sorts of prompts. Then I realized that Purview called them role groups, not just roles. So, I reworded my question a bit: “Which role groups in Microsoft Purview grant the ability to create and edit sensitivity labels?” As you can see, I now called them role groups and told Purview what it was I wanted the user to do. Much better results this time:
So then I wondered if it was the use of role groups or that I had spelled out what I wanted the user to be able to do. So instead of using role groups in my prompt, I changed it to security groups. I received the same result:
My next test then checked to see if role groups would work if I was a bit more ambiguous about the access: “Which role groups affect Microsoft Information Protection”
As you can see, the answer was very similar to the others, but this time, Copilot threw in Information Barriers because I was asking about information protection in general.
The End Result?
As anyone can tell you, we have come a long way in AI-related technologies in a very short time, but we still have a lot to go. You need to take results a bit cautiously. In this case, I knew the answer, but what if it was something I didn’t? I would want to validate the results. Copilot helps you by providing the references it used to generate the answer. I still think that Copilot is important and is a great deal of help to those who use it. Just be sure you are asking the right questions correctly.
By the way, I used a total of 1.607 SCUs for the testing in this post.
Thanks for reading!
Leave a Reply