Microsoft Copilot Gains Image-Reading Capability
Microsoft's Copilot can now support image reading and optical character recognition (OCR), expanding its reach into unstructured data. This capability enables data extraction from scanned forms, medical images, and PDFs. The development is relevant for healthcare analytics, though extracting contextual meaning from visuals remains a challenge.
- The image analysis capabilities in Copilot are powered by Azure AI Vision, which provides access to advanced algorithms for tasks like Optical Character Recognition (OCR) and object detection. For developers, this is accessible through the Azure AI Vision ImageAnalysis client SDK, allowing for integration into custom applications and data pipelines. - A common architectural pattern for integrating this into healthcare data workflows involves a multi-stage pipeline. This typically includes a classification stage to identify the document type (e.g., a lab report vs. a prescription), followed by an extraction stage using services like Azure Document Intelligence (formerly Form Recognizer) to pull out structured data, and finally a resolution stage for validation and integration into a structured database like a lakehouse in Microsoft Fabric. - For analytics engineering, this means that unstructured data from scanned documents or images can be programmatically transformed into structured data in a Fabric lakehouse. A Fabric notebook can be used to call the Azure AI service, process the extracted data, and append it to a Delta table, making it available for analysis in Power BI. - In the context of healthcare analytics, AI-powered OCR can be used to analyze medical imagery like X-rays, CT scans, and MRIs to help identify abnormalities. This technology also aids in extracting data from clinical trial forms and patient records to accelerate research and development. - When building these systems, it is crucial to address compliance requirements such as HIPAA. This involves using HIPAA-eligible Azure services, signing a Business Associate Agreement (BAA) with Microsoft, and ensuring that data is encrypted both at rest and in transit. Microsoft Purview can be used to implement data governance and compliance frameworks. - The M365 Copilot APIs provide a way to integrate these AI capabilities into custom applications while adhering to Microsoft's compliance and security standards. This allows developers to build specialized assistants that can retrieve information from an organization's documents in a secure manner. - Looking ahead, the roadmap for Power BI Copilot includes the ability to answer more complex "why" questions about data, indicating a deeper integration of AI for data analysis. The availability of Copilot across all Microsoft Fabric capacity sizes is intended to broaden access to these AI-powered features for more users. - For custom solutions, Copilot Studio allows developers to build agents that can respond to queries based on uploaded images and documents. This enables the creation of tailored AI assistants for specific business needs, such as interpreting invoices or other forms in an image format.
