Comparison between luis.ai vs api.ai vs wit.ai?
One aspect of this question is how efficient are these tools at understanding natural language. In a recent benchmark we (Snips, a French AI company) just published, we have tested the built-in natural language engines of Alexa (Amazon), SiriKit (Apple), Luis (Microsoft), and API.ai (Google).
We tested their ability to understand natural queries like “Find me a salad bar I can go to for my lunch meeting”, “Order a cab for 6 people”, as well as 326 other queries.
The overall conclusion is that all solutions are imperfect.
More precisely, they all have similar levels of noise in their responses (between 60% and 90% precision), but there are significant differences in the breadth of language they can support. From this perspective, Luis performs the most poorly: on every use case we tested, it understood less than 14% of the queries. API.ai performs better, although not very reliably: it understands between 0 and 80% of the queries we tested, depending on the use cases. The highest levels of recall can be observed for Alexa (42% and 82% recall) and Siri (61% recall).
More details, and the raw data behind these results can be found in our blog post, Benchmarking Natural Language Understanding Systems
This blogpost has a really good analysis and comparison of Luis, Wit.ai, Api.ai, Amazon Alexa and IBM Watson services. It also has a nice background on why you would want to build a conversational bot in the first place and some of the challenges that come with that. It's written by the people behind YumiBot (a bot that gives you price quotes for app development).
The general gist is that Wit.ai and Luis are great choices if you're experimenting and just want to get something out for free. Api.ai has a great service and user experience but isn't free. Same with IBM Watson, the latter priced more for enterprise work. Alexa's API is great but only works with Alexa (but given that they have a huge userbase, isn't a bad deal).
Their advice is also to not rely too much on one provider:
We would recommend you store all data needed for your model in a structured way in your own code repository. So later you can retrain the model from scratch, or even change the language understanding provider if needed. You just don’t want to be in a situation when a company shuts down their service and you are completely unprepared. Do you remember Parse?
I hope this helped a little! I think the best way to make a choice is to just give these services a try. Given that a lot of them are still heavy under development and adding features/changing pricing models, you should try coming at them with a specific use-case and see which one can get you to where you need the quickest.
wit.ai vs Dialogflow vs luis.ai
╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗
║ wit.ai vs api.ai(Dialogflow) vs luis.ai ║
╠══════╦════════════════════════════════════╦═════════════════════════════════════════════╦════════════════════════════════════╣
║ S.No ║ Wit.ai ║ Api.ai(Dialogflow) ║ Luis.ai ║
╠══════╬════════════════════════════════════╬═════════════════════════════════════════════╬════════════════════════════════════╣
║ 1 ║ Wit.ai API is completely free ║ Api.ai Has a paid enterprise option ║ LUIS is in beta and free to use ║
║ ║ with no limitations on ║ which allows for this to be run on a ║ 10K transactions per month ║
║ ║ request rates. ║ private cloud internally and more ║ and up to 5 requests per second ║
║ ║ ║ from their services team., After google ║ for each account. ║
║ ║ ║ acquisition they are providing free ║ ║
║ ║ ║ services by integrating google cloud ║ ║
║ ║ ║ services. ║ ║
╠══════╬════════════════════════════════════╬═════════════════════════════════════════════╬════════════════════════════════════╣
║ 2 ║ Provides a nice combination ║ Speech to Text and Text to Speech ║ LUIS uses machine learning ║
║ ║ of both voice recognition and ║ capabilities, along with machine ║ based methods to analyze ║
║ ║ machine learning for developers. ║ learning. ║ sentences. To perform machine ║
║ ║ ║ ║ learning, LUIS breaks an ║
║ ║ ║ ║ utterance into "tokens". ║
╠══════╬════════════════════════════════════╬═════════════════════════════════════════════╬════════════════════════════════════╣
║ 3 ║ Has two main elements to it ║ Support of Intents, Entities, actions ║ Supports Intents, Entities ║
║ ║ that you set up within your ║ and one key focus area is its “Domains”. ║ and actions. ║
║ ║ app – intents and entities. ║ ║ ║
║ ║ Actions are separated to ║ ║ ║
║ ║ use as a combined operations. ║ ║ ║
╠══════╬════════════════════════════════════╬═════════════════════════════════════════════╬════════════════════════════════════╣
║ 4 ║ Has pre-build entities like ║ Has pre-build entities like @sys.date, ║ Has pre-build entities ║
║ ║ temperature, number, URLs, ║ @sys.color, @sys.unit-currency… etc. ║ builtin.intent.alarm, ║
║ ║ emails, duration… etc. ║ ║ builtin.intent.calendar, ║
║ ║ ║ ║ builtin.intent.email… etc. ║
╠══════╬════════════════════════════════════╬═════════════════════════════════════════════╬════════════════════════════════════╣
║ 5 ║ Doesn’t have integration module ║ Has integration module to connect ║ Has integration to Microsoft ║
║ ║ to directly communicating with ║ directly to Facebook messenger and ║ Azure and other services, can be ║
║ ║ Facebook messenger or other ║ other messenger api’s. Has support for ║ deployable in any supported ║
║ ║ messenger APIs. but has web ║ deploying in to heroku server, enterprise ║ servers. ║
║ ║ service api to hook services. ║ paid environment. ║ ║
╠══════╬════════════════════════════════════╬═════════════════════════════════════════════╬════════════════════════════════════╣
║ 6 ║ Early in 2015, joined Facebook ║ Created by a team who built personal ║ LUIS was introduced together with ║
║ ║ and opened up the entire platform ║ assistant app for major mobile platforms ║ Microsoft Bot Framework and Skype ║
║ ║ to be free for both public and ║ with speech and text enabled conversations. ║ Developer Platform which can be ║
║ ║ private instances. ║ acquired by google (sept 2016). ║ used to create Skype Bots. ║
╠══════╬════════════════════════════════════╬═════════════════════════════════════════════╬════════════════════════════════════╣
║ 7 ║ Wit.ai API for developers of iOS, ║ Api.ai has SDKs for Android, iOS, ║ LUIS allow building applications ║
║ ║ Android, Node.js, Raspberry Pi, ║ the Apple Watch, Node.js, Cordova, ║ by using the LUIS web interface. ║
║ ║ Ruby, Python, C, Rust and ║ Unity, C#, Xamarin, Windows Phone, ║ No coding needed other than the ║
║ ║ Windows Phone. It even ║ Python and JavaScript. It also can be ║ ability to interpret and use the ║
║ ║ has a JavaScript plugin for ║ integrated with Amazon’s Echo and ║ returned JSON in application. ║
║ ║ front end developers. ║ Microsoft’s Cortana. ║ It is also possible to use the ║
║ ║ ║ ║ LUIS REST API for ║
║ ║ ║ ║ automation of applications. ║
╚══════╩════════════════════════════════════╩═════════════════════════════════════════════╩════════════════════════════════════╝
Update: API.AI is now Dialogflow. Learn more here.
We have recently published an evaluation study of seven NLU API-enabled services: API.ai, Amazon Lex, Microsoft LUIS, IBM Watson Conversation, wit.ai, Recast.ai and Snips.ai.
A brief summary of our findings:
- IBM Watson intent detection is the best one, especially on smaller training datasets (although when trained on over 2000 samples the difference is indistinguishable).
- API.AI is free, the performance on big enough training set matches IBM Watson and Microsoft LUIS.
- Microsoft LUIS works significantly faster than others in our tests. wit.ai has somewhat worse performance and response time than the three above, but it’s free and it provides the best language coverage (some 50 languages).
- Amazon Lex has quite strict API limits (the training set size is limited to 200K symbols, which may be insufficient to reach a good intent detection quality for a multi-intent assistant; also it requires all training utterances to be labeled by entities, which complicated preparation of the dataset.