What is SpeakRight?

SpeakRight is an open-source Java framework for writing speech recognition applications in VoiceXML. Unlike most proprietary speech-app tools, SpeakRight is code-based. Developers write their application in Java using SpeakRight classes and their own extensions. Java IDEs such as Eclipse provide great debugging, and fast Java-aware editing and refactoring.

SpeakRight is based around the flow object which has a similar role to controls in GUI frameworks. A flow object manages presentation (prompts, grammars, and retry logic), and control flow. In MVC terms, a flow object is both the view and the controller. Flow objects can be customized (by setting properties), or extended using sub-classing. SpeakRight provides built-in objects for standard data types (time, date, alphanum), and for standard flow algorithms (forms, menus, list traversal, confirmation).

Flow objects can contain sub-flow objects. This nesting continues all the way out to a single application flow object. At runtime SpeakRight executes the application flow object. It may decide that the first thing to do is to execute a sub-flow. The sub-flow may execute its sub-flow. And so on. Nesting continues until a leaf flow object that actually produces VoiceXML content is encountered. At this point, a flow stack has been built up of all the active flow objects. The topmost item in the stack is the leaf. Only one flow object executes at a time. SpeakRight persists itself across HTTP requests (using serialization) . The results of VoiceXML page arrive back at the servlet as a new HTTP request. SpeakRight gives the results to the currently executing flow object. It can do one of three things:

• execute again, and produce another VoiceXML page
• return null to indicate it has finished. It will be popped off the flow stack and the next flow object will be executed.
• "throw" an event. SpeakRight uses a Throw/Catch style of event handling. A thrown event causes a search down the stack looking for a handler.
  

SpeakRight prompts use a powerful formatting system. A prompt string such as "The current interest rate is {$M.rate} percent. Please wait while we get your portfolio {..}{music.wav}" represents:

• A TTS (text-to-speech) utteran "The current interest rate is"
• a model value
• more TTS "percent. Please wait while we get your portfolio.."
• silence (500 msec)
• audio file

Applications can use in-place prompts, such as the one shown. Or they can use prompt ids, such as "id:sayInterestRateAndWait" which is a reference to an external XML file containing the prompt string. Multi-lingual apps are possible by deploying language-specific prompt XML files.

SpeakRight grammars are... GRXML files. I've totally punted in this early version. Dynamic and GSL prompts later, but for now, get the wonderful Grammar Editor that comes (free) with the Microsoft Speech Application SDK (SASDK for MSS 2004 R2).

MVC architecture requires a Model and SpeakRight has one. The model is used to share data between flow objects. User input (or data retrieved from a database) that needs to be used later in the call flow should be put into the model. SpeakRight provides binding so this occurs automatically. In SpeakRight you generate the model using code generation -- a tool called MGen. You write a simple XML file specifying your model fields (such as "Date departureDate") and MGen generates a type-safe Model.java file. Your application can now use intellisense (called CodeAssist in Eclipse) to inspect the model variables. Each model variable has get, set, and clear methods.

Applications need to have side effects. Their purpose is to interact with some back-end database to transfer funds, buy a train ticket, or vote. SpeakRight calls these transactions. Each flow object, upon finishes, can invoke a transaction. If the transaction is invoked asynchronously, SpeakRight pauses the callflow until the transaction completes.

Of course, before we accept data we need to validate it. Flow objects have a ValidateInput method that is called whenever user input has occurred. The method can accept or reject the input. If input is rejected, the flow object is executed again.

Another part of speech recognition is confirmation. This is done whenever the recognition confidence level is below a confirmation level. SpeakRight uses pluggable confirmation strategies. Some of the strategies are: None, YesNo, ConfirmAndCorrect, and ImplicitConfirm. You can basically plug any confirmation strategy into any flow object.

Finally, and most importantly, SpeakRight provides an extensible architecture for re-usable components called SRO (SpeakRight Objects). These speech objects provide standard mechanisms for common tasks. There are (to come) SROs for input of common data types such as time, date, currency, and zip code. There are also SROs for common control flow algorithms such as list traversal. SROs are highly configurable: prompts and grammars can be customized or replaced. Prompts can be customized at several levels, including pluggable formatters for rendering model values.

Re-usability is a key technical challenge in the project. It should be easier (and better) to re-use existing objects than rolling your own. Prompts, for example, need to be highly flexible. Users do not like prompts that switch back and forth between TTS and pre-recorded audio. A single voice is important. SpeakRight provides an audio matching feature, where all prompt segments are looked up in an audio match XML file that contains a replacement audio file for that segment. Of course, prompt coverage tools are needed to help build the list of prompts to be sent for professional recording. These tools are (cough) TBD.

Tutorial

This tutorial uses Eclipse 3.2.1 to build a simple SpeakRight application

1. Getting Started
   
2. Creating a Hello App
3. Adding a Menu

Tutorial 2 - Adding a Menu

Let's add a menu to our Hello app.

add the menu class to app1 (as inner class)

here's the grxml file

import org.speakright.core.*;

public class App1 extends SRApp {
public App1(){super("app1"); //give the app a name (useful for logging);
add(new PromptFlow("Welcome to SpeakRight!"));
}
}

This flow will simply say "Welcome to SpeakRight!" and hang up.

There are a number of ways to run your app. Eventually you'll embed it in a Java servlet and point a VoiceXML browser at it. But for now, let's use the ITest console utility that comes with SpeakRight. Switch back to Hello.java, which should have been created when you made the project.

import org.speakright.itest.*;
public class SRHello {public static void main(String[] args){
System.out.println("SRHello");
SRInteractiveTester tester = new SRInteractiveTester();
App1 app = new App1();
tester.run(app);
System.out.println("Done.");}
}

OK. Let's run the app. In the Package Explorer, find your class file Hello.java, right-click and select Run As and Java Application. ITest runs in the java console. You should see this in the Console view

The "1>" indicates this is the first turn. That is, your app is about to create it's first VoiceXML page. Type "go" and press Enter. You'll see some logging and the VoiceXML:

welcome to SpeakRight 


 
 

When a SpeakRight app is in a java servlet, this VoiceXML would be returned to the VoiceXML server, where it would be executed. The results of that execution would be sent back to SpeakRight. In this case, the page simply plays some text-to-speech audio, so there is no user input to send back. In ITest, we are the VoiceXML platform and we send back (using the "go" command) the results of executing the most recent page. So type "go" and press Enter again. The app has nothing more to do so it finishes, along with ITest.

That's it. You've created your first SpeakRight app.

Tutorial 1 - Creating an Hello app

This tutorial uses Ecplise 3.2.1

First let's create our flow class. Each app will have a top-level flow object. In the Package Explorer, right-click on your project Hello and select New / Class. Name it App1. Here's the Package Explorer (ignore Model.java for now)

You should then be able to edit the App1.java file, adding the following:

import org.speakright.core.*;

public class App1 extends SRApp {
public App1() {
add(new PromptFlow("Welcome to SpeakRight!"));
}
}

This flow will simply say "Welcome to SpeakRight!" and hang up.

There are a number of ways to run your app. Eventually you'll embed it in a Java servlet and point a VoiceXML browser at it. But for now, let's use the ITest console utility that comes with SpeakRight. Switch back to Hello.java, which should have been created when you made the project.

import org.speakright.itest.*;

public class SRHello {public static void main(String[] args) {
System.out.println("SRHello");

SRInteractiveTester tester = new SRInteractiveTester();
SRFactory factory = new SRFactory();
SRRunner runner = factory.createRunner("", "", "", null);
App1 app = new App1();
tester.run(run, app);

System.out.println("Done.");}
}

For a simple app we pass empty strings to createRunner. For more information on the parameters to pass to createRunner, see Initialization.

OK. Let's run the app. In the Package Explorer, find your class file Hello.java, right-click and select Run As and Java Application. ITest runs in the java console. You should see this in the Console view

The "1>" indicates this is the first turn. On each turn your app will create a VoiceXML page. Type "go" and press Enter. You'll see some logging and the VoiceXML:

When a SpeakRight app is in a java servlet, this VoiceXML page would be sent to the VoiceXML server (as part of an HTTP reply). The server would execute the page, and send back the results (as a new HTTP request) . In ITest, we take the place of the VoiceXML server. In the "go" command we specify the results of executing the most recent page. In this case, the page is output only; it only plays text-to-speech audio, so there is no user input to send back. So type "go" and press Enter again. The app has nothing more to do so it finishes, along with ITest.

That's it. You've created your first SpeakRight app. To see how a fully-functioned app is written see Simpsons Demo, or get more documentation at Table Of Contents.

Getting Started

After reading What is SpeakRight and perhaps some other articles in Table of Contents, you're ready to install and try it out.

Installation

Download the code from here. I used Eclipse 3.2.1 which uses Java 1 (jre/jdk) 1.5.0.11 and Tomcat 5.5.

You'll also need

• log4j 1.2.8 (comes with Eclipse)
  
• StringTemplate 3.0 from here which includes antlr 2.7.7
  

This tutorial assumes you are using Eclipse 3.2.1. Any Java IDE should work.

Start Eclipse and select File / New / Project and select New Java Project. Name it "SRHello".

In the Package Explorer (on the left-hand side of your screen), right-click and select Properties. Select Java Build Path and then the Libraries tab.

Select Add External Jars and add SpeakRight, Log4J, StringTemplate, and antlr. Here's the dialog box. Ignore JUnit and XMLUnit; they're not needed at this time.

Now we're ready to write some code! On to the SRHello app.

Frequently Asked Questions

What is SpeakRight?

SpeakRight is an open-source Java framwork for writing speech recognition applications in VoiceXML.SpeakRight apps live inside a web server, such as a JSP servlet. The SpeakRight runtime executes a set of flow objects, and generates VoiceXML pages, one a time. SpeakRight is stateless so it can work across multiple HTTP requests.

How is it licensed?

SpeakRight is licensed under the OSS-approved Eclipse Public License (EPL). Under this license anyone can download and use SpeakRight. There are no license fees for use within a commercial product. EPL's main restriction is that if you modify SpeakRight source files and distribute them, then you must release those files under the EPL. Of course, your own source files (that may extend or use SpeakRight classes) are excempt.

Where does the name SpeakRight come from?

The name comes from the term "speak right into the microphone". Interviewers used this phrase in the early days of TV. People being interviewed didn't know how to behave with this new technology, and often froze on camera. The goal of the SpeakRight Framework is to help make people comfortable with with speech recognition applications.

Why use SpeakRight?

I have written a number of speech apps on different SALT and VoiceXML platforms. After trying a number of different approaches (raw VoiceXML, drag-and-drop toolkits), I came to believe that the complexity of speech application VUIs required a code-based approach (see Benefits of a Code-Based Approach)

What's the current status?

The current status is alpha. Single-slot questions are supported, with NBest support. Audio can be audio files, TTS, or generated values (such as today's date, or an application variable). Inline, built-in, and external grammars are supported. Also supported: control flow, event handling, the model, DTMF-only-mode, and confirmation. Transfer and Record too.

Testing can be done from JUnit, a console app, from a web browser (apps can generate an HTML version of themselves), or from a VoiceXML platform.

Not supported: mixed initiative, hot words (link tag).

What platforms does it run on?

Currently has only been tested on Voxeo's free site. SpeakRight uses a template engine (see String Template template engine) so changing the output is often as easy as changing the template file.

What is SpeakRight?

SpeakRight is an open-source Java framework for writing speech recognition applications in VoiceXML.Unlike most proprietary speech-app tools, SpeakRight is code-based. Developers write their application in Java using SpeakRight classes and their own extensions. Java IDEs such as Eclipse provide great debugging, and fast Java-aware editing and refactoring.

The basic class is the flow object which has a similar role to controls in GUI frameworks. A flow object manages presentation (prompts, grammars, and retry logic), and control flow. In MVC terms, a flow object is both the view and the controller. Flow objects can be customized (by setting properties), or extended using sub-classing. SpeakRight provides built-in objects for standard data types (time, date, alphanum), and for standard flow algorithms (forms, menus, list traversal, confirmation). Your application can combine and customize these built-in flows, or create your own.