Copilot for Microsoft 365 How it ACTUALLY Works!

In this, the last of 3 special episodes featuring the Director of Microsoft 365 and the host of Microsoft mechanics, Jeremy Chapman and I discuss Microsoft 365 Copilot. Answering all of your questions, we’ll discuss everything you need to know about Copilot. Including how it works, what it can do and how to prepare your business and get ready for the AI revolution. This is a deep dive session. That aims to ask some hard hitting questions that goes beyond the fluffy marketing.
Visit Jeremys KZbin channel / @deployjeremy
Microsoft mechanics - / @msftmechanics
For more on me visit me on LinkedIn or at Andymalone.org
If you want access to exclusive content and more why not consider supporting me and join me on Patreon. / andymalonemvp
Timecodes
00:00 Introductions
02:53 Introducing Jeremy Chapman
03:53 How Copilot Works
04:31 Microsoft 365 Copilot Demo
05:54 How Copilot Works from the Backend
07:03 Understanding Copilot Architecture
09:00 Customising Copilot Prompts
10:30 Building your own Copilot with Azure AI Studio
18:30 Customising Copilot Thresholds & Filters
19:42 Security & Compliance Controls
21:51 How to Prepare your Organisation for Copilot
24:30 Auditing Copilot
26:35 How do I get started and learn more?
30:19 Check out the NEW Copilot Dashboards
31:11 Session conclusions and next steps

[@nocodecreative]

Excellent Andy, some of the best content I've seen to date on the subject!

[@YN-wv1rd]

Great overview, thank you! This is the information I was looking for.

[@Daniel_alejandro39]

Finally an employee that doesn't take lunch breaks or coffee mornings.

[@yankeeee80]

Great summary of copilot in general and copilot for M365 in particular. Thank you both!

[@DeployJeremy]

Glad you liked it. 😊

[@1.618Golden]

Thank you for this. I have been trying to figure this out for a while now.

[@saurabhvkadam]

Thanks mr. Andy, for the educational video on copilot, we have purchase license for copilot and soon delivered to india. Happy see ms mechanic giving such info abt copilot. Waiting for some deep dive from you.

[@Sarakness]

That was great. Thank you!

[@MrSam_Derp_Man]

A lot to learn, a lot to build!

[@ankurakashady]

Thanks for this Andy

[@xyberfunk]

Congrats on the Silver Award! Look forward to many more tech videos. Thank you.

[@AndyMaloneMVP]

Many thanks

[@UR_Great]

Thank you for the information.

[@driver288]

We were supposedly the first company to get Copilot licenses through CSP in Europe and are doing a pilot now 😊

[@mukundthaker794]

Excellent insights into the subject. Thanks so much

[@AndyMaloneMVP]

Glad it was helpful!

[@DeployJeremy]

Thank you!

[@Nimitz_oceo]

We did it!!!

[@tonyelliott5045]

wait, @ 7:23 when the architect diagram is presented, I see that step 3 is to take the massaged prompt and put it into the LLM. However the disclaimer on the LLM is that it does not learn from prompts, responses or grounding data. So how do LLM's get improved to operate better? How does the RAI get improved to run better? A see a circular argument occurring with LLM and RAI maintainers: "Boss: I'd like you to make the XYZ LLM run more efficiently and to be able to handle more complex requests" "Maintainer: sure, give me some real world data models to study" "Boss: oh no, we don't use actual user prompts, responses, etc to study". I feel it would be easy enough to de-identify user metadata to allow enough information to be used by LLMs and RAIs.

[@iamalfofficial]

Congratulations on your 100k subs and yt award.

[@AndyMaloneMVP]

Thank you so much 😀

[@earlthompson6212]

Thank you. Just subscribed. E

[@abe6215]

I've been using it, and I find it way less powerful that chat GPT plus, no way to easily build custom gpt's, copilot studio feels basic and more tuned for public facing gpt's

[@starfish258852]

Thank you

[@user-tj8ub7cz1g]

Nice video. Hate that noise at 22:11 though!

[@Teramos]

Would love to see Semantic Index in action, is it indexing the files metadata or the content within the file also? For example, Word or PDF Files in a SharePoint library.

[@DeployJeremy]

We just published two short Semantic Index explainer videos on @msftmechanics - one with how it works and another with a demonstration - and I'd also recommend watching our show on vector search with Pablo Castro. I can't add links unfortunately or KZbin will remove this comment. Semantic indexes are built at both the tenant level and the user account level to add more context and personalization used with search results and as part of info retrieval during Copilot orchestration.

[@DeployJeremy]

We just published two short Semantic Index explainer videos on Microsoft Mechanics - one with how it works and another with a demonstration - and I'd also recommend watching our show on vector search with Pablo Castro. I can't add links unfortunately, due to comment filters. Semantic indexes are built at both the tenant level and the user account level to add more context and personalization used with search results and as part of info retrieval during Copilot orchestration.

[@7350652]

thanks

[@georgedubberc]

At 28.26 Jeremy talks about public web content. Forgive me if this is a silly question; I am not tech qualified/ an admin. If the business is buying Copilot for Microsoft 365 and wants to keep its data within the M365 service boundary, can you (1) explain the relationship between "Copilot for M365" and "Microsoft Copilot formerly known as Bing Chat Enterprise"? and (2) if you check the box on Jeremy's slide in the Admin centre "Allow Copilot to reference public web content" are you thereby enabling web connection via M365 Chat or are you using Bing Chat Enterprise (which has different T&Cs)? (3) Aren't you losing some of your M365 protection of your data?

[@DeployJeremy]

Good question. The checkbox in the Copilot admin center page is enabled/checked by default. It enables information retrieval from the web, in addition to the Microsoft Graph, as part of the retrieval process to gather data. It functions similar to how a plugin would in this case, traversing outside the Microsoft 365 boundary for retrieval. In this case, when the prompt is interpreted, it's running a sub query and would not need to use the full user prompt to retrieve what's required from the web.

[@DataSpook]

Using Copilot studio is the only way to create the bot? If so I saw it’s $200 mo. Is that the only option? Thanks! Helpful vid.

[@DeployJeremy]

You can create chat style apps with several tools, AI Studio just makes it easier to build the orchestration, add data for RAG, content filters, and monitoring into your app.

[@DavidGodin]

It feels to me that all of the KZbin videos and content I look for seem to brush over the odd thing that the 365 version does not use Chatgpt 4 Turbo underneath, but the Pro version does. I am using the former, and I keep hoping that the 365 version does the upgrade.

[@AndyMaloneMVP]

Me too

[@bogzero]

I bought the business version of copilot in all office apps the copliot button is shown but in teams chat the copilot button is missing. Which config is wrong ? In Apps the Copilot is shown but i dont can use them in chats

[@DeployJeremy]

Do you have Copilot Pro (which is for personal use with for example a hotmail account), or Copilot for Microsoft 365 (which is for work use with a work account, aka Microsoft Entra ID)?

[@AhmadZahida]

Hi Andy thanks for this amazing vedio I have two questions if you allow me. Does Microsoft graph need to be configured to work with CoPilot or does it work out of the box. Copilot can be used for education organizations to build content, including personalizing the learning experience .

[@AndyMaloneMVP]

Yes, copilot can be configured to use Microsoft graph. But it’s already configured by default copilot for education will no doubt be released in due course. Watch this space for more details.

[@johncarr123]

Now we know how big brother AI works to access all our private data.

[@AndyMaloneMVP]

Dude, wouldn’t you listening!

[@vap0rtranz]

Great walk-thru of custom bot building in Studio. Most vids never step through building a custom copilot (small "c"), which is the REAL power of Copilot & Graph. What Jeremy steps-thru is RAG (Retrieval Augmented Generation) and a lot of AI research is saying Hybrid semantic search with re-ranking and backed by a vectorDB of embedded document contents and a frontend custom prompt is THE most accurate response (aka. least LLM hallucinations). I thought Copilot Studio would be the no-code alternative to AI Studio for Business/Enterprise customers?

[@AndyMaloneMVP]

I’ll leave this one to Jeremy 😀

[@DeployJeremy]

Copilots use retrieval augmented generation. It's actually one of the most common misnomers I hear when someone says "the LLMs in Copilot are trained on your data." The personalization in the responses is based on search and session state only. The LLMs are a shared service, which do not retain data from any session state or inference interactions, so you have to add the information to your prompt. As token limits continue to increase, it makes fine tuning or custom training increasingly less justifiable, because you can just add it in real time via augmented prompts. Also, even though vector databases and calculating embeddings for them is not a trivial exercise, it's more efficient once complete compared doing multiple retrieve-read-retrieve turns in RAG orchestration to achieve the same result and the better the retrieval is, the more you can load balance LLM completions.

[@DeployJeremy]

Sorry, I forgot to answer the question of using Copilot Studio for low code RAG apps. It does allow you to add data sources from various locations, but isn't yet quite as rich as Azure AI Studio for hybrid search, prompt flows, content filters, monitoring, etc. The core elements are there though. Check out the blogs on this topic from Carlotta Castellucio (I can't link them directly, or KZbin will remove this comment).

[@vap0rtranz]

@@DeployJeremy That makes sense because Azure AI Studio has been around longer. It's good to hear that there's capability in the works. I'm checking out Carlotta's blogs, ty Jeremy!

[@DeployJeremy]

@@vap0rtranz Azure AI Studio was purpose built for building RAG style apps using lots of LLM options, too. Copilot Studio is a nice evolution of Power Virtual Agents and to some extent, the Azure Bot Framework to add Copilot capabilites and integration, where you might say the conversation progressions and branches were a bit more codified than ChatGPT style interactions to for example step through a workflow, like booking an appointment or travel. And it continues to work for bots that have been built on it over the past few years. If you think about it, the ChatGPT style interactions are simpler in terms of workflow or sequences of tasks comparatively, because currently the objective is to generate content. Will be interesting to see if generative AI does eventually evolve partially to improve process automation in a more conversational way.

[@dancnabest]

He didn't talk about the set u how to get. Which license does it work for tutorial for setting it up

[@AndyMaloneMVP]

Ah great point I’ll cover this in an upcoming video

[@tH3steg]

My biggest gripe of the release of co-pilot is the cost. Working with in education the benefits of this solution is unparalleled. But at a cost of £30 per FTE per month is just not doable for most schools. Sadly trying to keep schools in the Microsoft ego system as the costs increase against Google’s offering is becoming a weekly struggle.

[@AndyMaloneMVP]

I’m m sure there will be a plan for education soon

[@user-jv7el4zx4b]

Thank you for your chanel. My Q is, Is Microsoft 365 itself a LLM that learn?

[@AndyMaloneMVP]

No, Microsoft, 365 is not a large language model. It’s based on a number of technologies, including the graph API, security graph, API, Entra ID, SharePoint, exchange and so on. Copilot, however, can integrate with all of these technologies. For more details on Microsoft Copilot check out learn.microsoft.com and I’m delighted to have you on board 👍

[@DeployJeremy]

In this scenario, a great way to think about it is that the LLM is trained on public content (training). It doesn't learn as processes interact with it (inference). Imagine the times when someone has asked you to write a report on a topic that you are only somewhat familiar with. So you would need to find relevant information from the Internet, library, or files you've worked on in Microsoft 365, then you read that content and write a better report. The difference here is that the LLM doesn't retain any of that temporary knowledge it used to write the report, it's more like a calculator in that sense. The calculator stopped training, too, once it left the factory.

[@mohamedifham3195]

Is it posssible to use copilot api to search and get results from a 3rd party application or postman

[@AndyMaloneMVP]

Yea copilot can be extended using numerous connectors I’m preparing a new copilot video this week 👍

[@pookapookyoyo3095]

Does anybody know where i can get a more in depth explanation of the architecture/processflow?

[@AndyMaloneMVP]

All documentation is on learn.microsoft.com

[@jaygutierrez7230]

Dose this version of copilot work the same way in Dynamics?

[@AndyMaloneMVP]

There is a dynamics version

[@gthofm]

Exactly what I needed. Thank You!

[@605man]

Another question that needs to be asked is how does Microsoft handle companies compliance needs? Such as NIST 800, PUB 1075, HIPA etc....

[@AndyMaloneMVP]

They are fully compliant with audit reports available

[@user-js4mt5cd2t]

Would be nice if it actually did some real work for a change.

[@juanpablocortesl-iv7ze]

Explains the random emails about updates I never asked for.

[@davidjones7144]

How did I get this get this forced into my MS update? I dont want it, killed it already, now I'm getting all these "how great coplit is" vids in my feed. If I wanted AI on my PC, I would seek it out and install it.

[@605man]

Sorry, somehow i missed this section in your video....

[@ThomasAnderson-MetaCortex]

Even Microsoft employees don't call it Entra lol

[@getl0st]

Imagine all of the other Corporate Data Microsoft is analysing from it's Customers that no-one knows anything about....

[@AndyMaloneMVP]

Please don’t be paranoid. I can tell you that there are numerous protections in place to prevent this. That’s why we have compliance a complex governance and auditing tools to prevent this.

[@1zcott]

Microsoft forced subscription models suck huge doney

[@AndyMaloneMVP]

They have pros and cons

[@mattcunliffe2807]

Man - no fluff ? Show some real life examples - from scratch - you need plugins? 😅 gee man relate to the audience

[@DeployJeremy]

We didn't use plugins in this the case for the demo, it was the native orchestration with Microsoft 365.

[@Seanpfree]

Scammers all over the globe rejoice! This is terrible for vulnerable people. Microsoft, google, KZbin cant even keep the scammers out of the comments section of EVERY video, AI is just going to help scammers.

[@AndyMaloneMVP]

Chat GPT yes. Not Copilot for Microsoft 365 as its internal

[@TheVineOfChristLives]

@@AndyMaloneMVPsad because the OP doesn’t know older transformer models have been doing this for years.

[@nicolasblanc5885]

We don't want Copilot

[@AndyMaloneMVP]

And those cavemen never wanted fire either

[@nicolasblanc5885]

@@AndyMaloneMVP Fire was not invented by man... Beyond the joke, Copilot poses many problems under the GDPR: finality of processes, legal basis, proportionality....Moreover, audits are going to be very complicated and Microsoft places the responsibility on the data controller. Who is going to configure the thresholds and filters discussed in the video, which raises questions in terms of subjectivity

[@AndyMaloneMVP]

I totally agree. It is scoped only to your tenant UNLESS you enable it to work outside. You are in control. Can I ask have you actually tried it yet?

[@DrFuzzyDuck]

​@@nicolasblanc5885recommend you don't buy it. Eventually a competitor will solve these problems and out compete those who avoid these tools.