Speech in Mobile and Pervasive Environments

Inhaltsverzeichnis

About the Series Editors xiii

List of Contributors xv

Foreword xvii

Preface xix

Acknowledgments xxiii

1 Introduction 1

1.1 Application design 3

1.2 Interaction modality 3

1.3 Speech processing 4

1.4 Evaluations 5

2 Mobile Speech Hardware: The Case for Custom Silicon 7

2.1 Introduction 7

2.2 Mobile hardware: Capabilities and limitations 11

2.2.1 Looking inside a mobile device: Smartphone example 11

2.2.2 Processing limitations 14

2.2.3 Memory limitations 16

2.2.4 Power limitations 19

2.2.5 Silicon technology and mobile hardware 22

2.3 Profiling existing software systems 24

2.3.1 Speech recognition overview 24

2.3.2 Profiling techniques summary 25

2.3.3 Processing time breakdown 27

2.3.4 Memory usage 29

2.3.5 Power and energy breakdown 30

2.3.6 Summary 32

2.4 Recognizers for mobile hardware: Conventional approaches 32

2.4.1 Reduced-resource embedded recognizers 33

2.4.2 Network recognizers 35

2.4.3 Distributed recognizers 36

2.4.4 An alternative approach: Custom hardware 38

2.5 Custom hardware for mobile speech recognition 38

2.5.1 Motivation 38

2.5.2 Hardware implementation: Feature extraction 40

2.5.3 Hardware implementation: Feature scoring 41

2.5.4 Hardware implementation: Search 44

2.5.5 Hardware implementation: Performance and power evaluation 47

2.5.6 Hardware implementation: Summary 49

2.6 Conclusion 49

Bibliography 50

3 Embedded Automatic Speech Recognition and Text-to-Speech Synthesis 57

3.1 Automatic speech recognition 57

3.2 Mathematical formulation 58

3.3 Acoustic parameterization 60

3.3.1 Landmark-based approach 64

3.4 Acoustic modeling 64

3.4.1 Unit selection 64

3.4.2 Hidden Markov models 66

3.5 Language modeling 69

3.6 Modifications for embedded speech recognition 71

3.6.1 Feature computation 71

3.6.2 Likelihood computation 75

3.7 Applications 77

3.7.1 Car navigation systems 77

3.7.2 Smart homes 78

3.7.3 Interactive toys 78

3.7.4 Smartphones 79

3.8 Text-to-speech synthesis 79

3.9 Text to speech in a nutshell 80

3.10 Front end 81

3.11 Back end 84

3.11.1 Rule-based synthesis 84

3.11.2 Data-driven synthesis 86

3.11.3 Statistical parameteric speech synthesis 90

3.12 Embedded text-to-speech 91

3.13 Evaluation 92

3.14 Summary 94

Bibliography 94

4 Distributed Speech Recognition 99

4.1 Elements of distributed speech processing 100

4.2 Front-end processing 101

4.2.1 Device requirements 103

4.2.2 Transmission issues in DSR 104

4.2.3 Back-end processing 105

4.3 ETSI standards 106

4.3.1 Basic front-end standard ES 201 108 107

4.3.2 Noise-robust front-end standard ES 202 050 107

4.3.3 Tonal-language recognition standard ES 202 211 107

4.4 Transfer protocol 108

4.4.1 Signaling 109

4.4.2 RTP payload format 109

4.5 Energy-aware distributed speech recognition 110

4.6 ESR, NSR, DSR 111

Bibliography 113

5 Context in Conversation 115

5.1 Context modeling and aggregation 115

5.1.1 An example of composer specification 121

5.2 Context-based speech applications: Conspeakuous 122

5.2.1 Conspeakuous architecture 124

5.2.2 B-Conspeakuous 125

5.2.3 Learning as a source of context 125

5.2.4 Implementation 127

5.2.5 A tourist portal application 130

5.3 Context-based speech applications: Responsive information architect 132

5.4 Conclusion 133

Bibliography 134

6 Software: Infrastructure, Standards, Technologies 137

6.1 Introduction 137

6.2 Mobile operating systems 139

6.3 Voice over internet protocol 140

6.3.1 Implications for mobile speech 141

6.3.2 Sample speech applications 142

6.3.3 Access channels 142

6.4 Standards 143

6.5 Standards: VXML 144

6.6 Standards: VoiceFleXML 145

6.6.1 Brief overview of speech-based systems 147

6.6.2 System architecture 148

6.6.3 System architecture: VoiceFleXML interpreter 150

6.6.4 VoiceFleXML: Voice browser 155

6.6.5 A prototype implementation 159

6.7 SAMVAAD 163

6.7.1 Background and problem setting 165

6.7.2 Reorganization algorithms 166

6.7.3 Minimizing the number of dialogs 167

6.7.4 Hybrid call-flows 171

6.7.5 Minimally altered call-flows 172

6.7.6 Device-independent call-flow characterization 174

6.7.7 SAMVAAD: Architecture, implementation and experiments 175

6.7.8 Splitting dialog call-flows 180

6.8 Conclusion 188

6.9 Summary and future work 188

Bibliography 189

7 Architecture of Mobile Speech-Based and Multimodal Dialog Systems 191

7.1 Introduction 191

7.2 Multimodal architectures 193

7.3 Multimodal frameworks 195

7.4 Multimodal mobile applications 196

7.4.1 Mobile companion 197

7.4.2 MUMS 199

7.4.3 TravelMan 200

7.4.4 Stopman 203

7.5 Architectural models 206

7.5.1 Client-server systems 207

7.5.2 Dialog description systems 208

7.5.3 Generic model for distributed mobile multimodal speech systems 210

7.6 Distribution in the Stopman system 211

7.7 Conclusions 214

Bibliography 214

8 Evaluation of Mobile and Pervasive Speech Applications 219

8.1 Introduction 220

8.1.1 Spoken interaction 220

8.1.2 Mobile-use context 222

8.1.3 Speech and mobility 223

8.2 Evaluation of mobile speech-based systems 224

8.2.1 User interface evaluation methodology 225

8.2.2 Technical evaluation of speech-based systems 226

8.2.3 Usability evaluations 227

8.2.4 Subjective metrics and objective metrics 228

8.2.5 Laboratory and field studies 230

8.2.6 Simulating mobility in the laboratory 231

8.2.7 Studying social context 232

8.2.8 Long- and short-term studies 232

8.2.9 Validity 233

8.3 Case studies 235

8.3.1 STOPMAN evaluation 235

8.3.2 TravelMan evaluation 240

8.3.3 Discussion 247

8.4 Theoretical measures for dialog call-flows 248

8.4.1 Introduction 248

8.4.2 Dialog call-flow characterization 250

8.4.3 (m,q,a)-characterization 251

8.4.4 (m,q,a)-complexity 253

8.4.5 Call-flow analysis using (m,q,a)-complexity 254

8.5 Conclusions 257

Bibliography 258

9 Developing Regions 263

9.1 Introduction 264

9.2 Applications and studies 264

9.2.1 VoiKiosk 265

9.2.2 HealthLine 267

9.2.3 The spoken web 268

9.2.4 TapBack 271

9.3 Systems 275

9.4 Challenges 278

Bibliography 278

Index 281