SpeakRight provides an automated tester SRAutoTester. It runs your callflow in a test harness where user input comes from strings that you provide. It checks the progress through the callflow to validate the application logic. Tests can be run directly on a developer's machine, no VoiceXML platform is needed.
SRAutoTester uses the same test harness as SRInteractiveTester, so you can test manually (using the keyboard) and auto-test, and vice versa.
The format of the string you give is a set of commands separated by semi-colons, where each command's format is: cmd[~ExpectedCurrentFlowObject]
The commands are
- "e" echo. toggle echo to log of VoiceXML on/off
- "g" go. simulate the current VoiceXML page ending and posting back. Causes SpeakRight to proceed to the next flow object. Can contain user input, such as "go chicago" or if a slot is set use ":" like this "go city:chicago"
- "q" quit
The commands for testing a good login are: g;g 4552;g 1234;g~MainMenu;q
Let's break that down:
- "g" means run the first flow object, which is the welcome prompt
- "g 4552" is the user id
- "g 1234" is the password
- "g~MainMenu" validates that we're at the MainMenu flow object
Note that these tests are high-level tests of the callflow. They allow requirements such as "A bad login does not proceed to the Main Menu". Testing the details of the VUI prompts and grammars need to be done as well; either with another SpeakRight tester (TBD) or on the VoiceXML platform itself.
5 comments:
My apologies if this is considered spam, but the message is relevant to your topic. If it is against your posting policies, simply remove.
My company recently announced a new VoiceXML test product called Voiyager (www.Voiyager.com) that will be available later this year. In short, it automatically tests an entire VoiceXML application without requiring scripts, like the one you show in your blog. Of course, it isn’t OSS either.
The problem with scripts is getting developers to write them and then keeping them up to date. I typically find people will write a few complex go right paths, but rarely test the error paths. In particular, developers rarely test paths where people can get stuck (continuous recognition errors or even the continuous screaming of help). But even worse than poor error path testing is limited go right testing. Enterprise applications being built today for insurance, FI and other verticals with a large amount of logic organized consumer plans and even call experience personalization. There just isn’t time to see how the call flow is altered across a large number of accounts using manual and scripted testing.
The short pitch:
Voiyager has a VoiceXML browser built into it. It fully executes the VoiceXML and SRGS grammar documents. As it processes the documents, it chooses every unique call path in an application until it discovers the entire call flow. While doing this, it checks for missing resources, errors in the resources and a few different common usability problems. For grammars with large number of unique choices (filler and synonyms are automatically ignored) users can provide test data. Once the process has completed, the call review component allows you to analyze the different calls Voiyager made by organizing by state, test data (account #) and other state level attributes. This allows business logic to be quickly validated.
Thanks for the information on Voiyager. It looks like a powerful tool for doing thorough coverage testing. A good testing tool has to be much smarter than the app it tests, in order to handle timing variations of the app under load.
Self-discovery has one drawback though. It tests what it finds. It doesn't validate that the app operates as required. If the VoiceXML for a login, for example, only retries twice, but the requirements specified three retries, then Voiyager won't catch this. So a combination of unit testing, such as SpeakRight provides, and something like Voiyager is required.
>It doesn't validate that the app operates as required.
That is correct. Voiyager can't tell you what is right or wrong in the flow. And, after spending more than the last decade writing large enterprise IVR applications, that business logic is where most of your challenges are experienced.
To address that issue in Voiyager, we implemented a component we call the State Validator. The state validator gives you a recognition state-by-recognition state view of your application. Within this view, we show you the unique set of call experiences that were experienced. So, for example, you could exercise your application with several test accounts and then look at the behavior of the account/pin dialog states for each of those accounts (account 1 fails, account 2 succeeds). Imagine a state where a bank balance or history is played. Run Voiyager with 100 accounts and quickly zip through the results, comparing against an expected results sheet. A human is still needed, but their time is spent reviewing results, not placing calls.
What we're trying to do is peel back the different layers of testing and automate as much as possible. Where we can't automate, we try to find ways to present the large amounts of data in a way where an analyst can quickly scan the results and look for incorrect behavior.
Voiyager doesn't in anyway replace SpeakRight nor unit testing (though our automation APIs could also be used to drive calls and check results via unit tests). In particular, writing VoiceXML applications, in particular rich VUI applications, still seems to require too much work and frameworks like SpeakRight, I believe can help.
Thank you for the opportunity to talk about my product. If this in anyway crosses the line becoming spam let me know and I will stop.
Sounds like Voiyager is doing true acceptance testing. That's a great way to support agile development and to de-risk upgrades. One still needs to test user experience by listening to prompts, volume, timeouts, and flow. But an automated way to validate that core business layer actions are working correctly is a great regression test. Probably a pretty good initial system test too!
You say "rich VUI applications, still seems to require too much work". I agree, but can't really decide where the biggest inefficiencies lie. Could be the VUI design (prompts, error handling, flow), which needs to be carefully done. Could be development -- rich interactions are still tricky in VoiceXML. Could be testing. Could be tuning.
Thoughts?
Good info about voiyager. Combine it with Hammer or Jigsaw (www.gtn-tech.com) should make VXML app testing much more efficient.
Post a Comment