Microsoft Copilot Generated Image

Copilot for Vision on Azure

So, I built my own Copilot on Azure (custom).

As the above AI generated image suggests, a human worker (Pilot) is collaborating with digital worker (Copilot) in AI augmented style – for informed decision making as well as execution. That’s the philosophy of Copilot ecosystem.

Background:

There are two tools, both offer graphic developer environments to build a Copilot on your data using generative AI. I call them two paths journey, you can choose any or both.

Fig.1

Path1:

Azure AI studio might be a natural choice for those who have some familiarity with the Azure OpenAI service or the OpenAI playground and wish to have higher control over the LLM used by the Copilot, including the ability to evaluate and compare different model versions as well as to design the model prompt.

Path2:

Microsoft Copilot Studio might be a smoother transition for those who are familiar with the Power Virtual Agent suite and need the flexibility to pre-define some closed dialogue journeys for frequently asked questions and then use generative answers for fallback.

Use Case Overview:

I decided to create my custom Copilot for Vision with Azure AI Studio using Azure AI Vision.

The Video search and summarization capability of Azure AI Vision, uses a combination of natural language processing (NLP) and computer vision techniques to analyze the content of a video. It can quickly and concisely summarize the main points of a video, and it also allows you to search for specific moments within the video, making it easy to find relevant content.

Here is the workflow, I followed to build it end-to-end:

Fig. 2

At the end of the execution of this workflow, I had my Copilot for Vision ready to be used by me.

During the deployment of Azure AI Studio, several key workspaces were created by design as shown below:

Fig.3

Also, with Azure AI Studio, any LLM deployments are declaratively integrated with evaluation, prompt flow etc. by design.

That is cool!

Use Case Walkthrough:

Fig. 4

Deployed Azure AI Studio. Created project: byocopilot. Selected Build your own copilot, as shown in Fig.4.

Fig.5

Within ByoCopilot project, opened Playground and Deployments, there is no deployment of any LLM services currently. This is how the Build section here looks like (Fig.5). I could use GPT-4 Turbo Vision LLM here for the use case.

Within ByoCopilot project, under Explore section, selected Azure AI Vision capabilities. This is where I have to use Azure AI Vision from.

Fig.6

Following are the Azure AI Vision capabilities. I was particularly interested in Video retrieval for my Copilot for Vision.

Fig.7

Video retrieval capability of Azure AI Vision uses a combination of natural language processing (NLP) and computer vision techniques to analyze the content of a video. It can quickly and concisely summarize the main points of a video, and it also allows you to search for specific moments within the video, making it easy to find relevant content.

When I opened it, it was empty — no video data ingested and indexed as shown here (of course).

Fig. 8

I took a video of a grocery shop in my neighbourhood for this use case. IMG_2778.MOV is my video data. With the data ingestion and indexing, my data appeared in the library as shown in the Fig.9.

Fig.9

And, my Copilot for Vision was ready here and so was I to chat with it.

I asked my Copilot the following:

• Price for Tayto crisp: it spot-on showed the clip and price 2.95.
• Show me Rice Cakes: it spot-on showed the clip and price.

Fig.10

• Show me Great Value aisle: it spot-on showed the clip of the Great Value section. Showed all the clips, with different time stamps where Great Value sections had appeared.

Fig.11

• Show me Koka Chicken: it spot-on showed the clip of the Koka Chicken noodles. Showed all the clips, with different time stamps where it had appeared.

Fig.12

Conclusion:

• You can also build your own Copilot for AI applications.
• You can use GPT-4 Turbo Vision also for such use cases.
• Various custom Copilots can be built with choice of LLMs, Azure AI capabilities, and leveraging your own data.
• Azure AI Vision based Copilots can be built for ethical surveillance, event reporting and monitoring related use cases.

Join me on Medium for more practical hands-on based stories. You can contribute by clapping, sharing and commenting.

Thanking you,

Vishal Anand.

Disclaimer: Views are my own and personal ones.