Microsoft released their new AI Invoice Processing capability in preview last month. I took it for a test run using Power Apps and Power Automate. How do I rate it? Read on to find out!
AI Invoice Processing
I’ve been looking out for a solution to extract data from invoices for some time. I’ve tried using the existing Power Platform AI modules to do the job but it’s fair to say that to date, none have hit the spot:
- Text Recognition extracts all the text from the invoice image, kind of useful but this still leaves the important elements of the text still to be identified, which is far from easy
- Key Phrase Extraction is good for free format text, but much less so for structured documents like invoices
- By far the closest of the existing AI production modules to do the job is Entity Extraction. However, because Entity Extraction scans documents for structured text fields, it requires all the documents to be in a similar if not identical format. We all know that invoices come in many shapes and sizes contain varying terminology, so this isn’t a viable solution either
Consequently, I was hopeful that the new AI Invoice Processing module would take the best from each of the existing modules and combine it into a simple solution. I couldn’t wait to give it a go!
My requirement is to identify the following information from an invoice, extract the content and display it on screen:
- Customer Name
- Vendor Name
- Invoice Date
- Invoice Number
- Sub (Pre-Tax) Total
- Tax Total
- Invoice Total
I also want to know the level of confidence from AI Invoice Processing that it has extracted the correct data
That’s a pretty demanding set of requirements!
How to use the AI Invoice Processing Module
There is a Microsoft document here to help you set up the AI Invoice processing Flow and trigger it from Power Apps. This is a great place to start and will get you up and running quickly. Both the Flow and the basic Power App are not complex to build
What really got me fired up that AI Invoice Processing could be the real deal was that all the data fields I require (together with a number of others) are available as pre-defined outputs. Simply pick the output you require when creating the Flow and give it a name
Having built the Flow and the corresponding Power App, I tested against a number of different invoices, all of which were typed, had a different layout and terminology and were in differing currencies
The Results Are In:
Overall the results were very good! The large majority of the fields were correctly identified and the correct data extracted. Across all the invoices I used for the test, 83% of fields were correctly returned. A further 11% returned partially correct information, most of which was additional superfluous data being returned. For example, Customer Name sometimes also included the first line of the customer’ address. This sometimes happened even though the address was on separate line to the customer name
What Worked Well and Not So Well
The quality of the image didn’t need to be perfect. Even under mediocre lighting conditions the results remained very good, though obviously, the better the image, the more accurate the results. Even taking an image of an invoice upside down didn’t cause a problem!
If the invoices had sub totals, that was sometimes caused an issue. As did partially paid invoices, though that wasn’t unexpected. In every single case though, the amount of tax paid and the total value of the invoice was correctly identified. Knowing these 2 values, the Sub Total (Pre-Tax) value of the invoice can be easily calculated
The large majority of invoice dates were identified correctly. However, where a date is in the European format with a 2-digit year (DD/MM/YY), then usually a null value was returned as the format YY/MM/DD was expected. With 4-digit year though, the European date format was translated just fine. Extracting the data by Text rather than using Date in the Flow completely avoided this issue. Clearly though, the data would require further processing if required in a pure date format
Where an item of information was missing, it was correctly recognised as missing. When this was the case, a result of 0% confidence was returned, as the data hadn’t been found. On the one hand this makes sense. On the other, If the data is missing and this is recognised, then a confidence value of 100% would avoid manual intervention to look for non-existent data
The time taken from starting to upload the image to returning the output to Power Apps was extremely consistent at 10-12 seconds. Not blazingly fast, though in partial mitigation I should declare that my Internet connection speed is no more than mediocre
All in all though, an 83% success rate is pretty damn good! Especially so when the additional 11% partial success rate is also taken into account. As the invoice processing is using Microsoft Artificial Intelligence, we can expect the success rate to only improve in the future
Below is an example with a dummy invoice which was printed and ‘roughed up’ a little to reflect reality. The image was taken on a white background and the results are shown in the animation below. I’ve built the Power App to allow manual editing. Where manual editing takes place, the confidence level is changed to 100% for the updated fields
Hand-Written Invoices
The success rate on computer generated invoices was impressive, but what about the curved ball of hand-written invoices?
Overall, the results were not as good, which is not unexpected. Even so, the majority of the numbers were picked up correctly, but not the text. My handwriting isn’t great so with help from my wife, who is considerably neater, we got the overall success rate to near 50%. Not bad at all!
AI Invoice Processing as a Business Solution
In my app I made the Text boxes editable so the returned values can be amended or changed. I’d recommend you do the same
I also used a traffic light system based on the returned confidence value:
- 75%+ Assume the returned value is correct
- 50%-75% Probably correct but worth an eyeball to confirm
- 0-50% Requires manual validation
- 0% The required data is probably not on the invoice and that is why the field is blank, but check it is not there on the invoice under a different description or title
The 10-12 seconds or so to return the output would be a constraint to efficient working. The best enterprise solution would be one where the scanning is automated and the images are upload to file storage. This could be easily carried out with Power Automate uploading the images to Azure BLOB storage. Overnight the images could be processed using AI. All those with a full set of green results would be automatically accepted. Those with potential queries would be retained for manual review. The following day, those invoices could be reviewed displaying the extracted data alongside the invoice image on a desktop or laptop. All of which is easily achievable with Power Apps!
Finally, don’t forget that AI Invoice Processing is still in preview so it’s not yet production ready. The functionality could still change before full release
If you have your own thoughts or experience of AI Invoice Processing, please share them below. Also checkout out my other posts on the Power platform here
Below is the Flow I created. For the numerical fields I required, I tested both extracting them as Text or Numbers. The Text option proved more reliable at extracting the content, but obviously for numerical fields would require reformatting. I used the Text option for the confidence levels before converting to a Number, but using a Number would work equally well
Interesting article. I’ve been using AI for reading Receipts but not getting anything like your success rate
Hi Marcus, I too struggled with Receipt Processing. With plain receipts it worked fine but with anything more complex such as receipts with promotions, advertising, etc, the results are disappointing
Having looked into this again, the standard Receipt processing model is caveated to be optimised for US receipts. That’s probably why it’s results aren’t that great for UK receipts