Speech

The TwinCAT Speech Extension offers a connection to TwinCAT Speech systems. This allows a TwinCAT HMI client/browser to be used as an audio input and output device. For the extension, the .NET Core Runtime must be installed.

Note: As of 1.14, the package TwinCAT.HMI.Speech must be installed via TcPkg.

1. Install the TwinCAT Speech Extension using the NuGet Package Manager:
Speech 1:

The extension can now be used:
Speech 2:

The extension is now responsible for the exchange of data between TwinCAT Speech and the browser. Therefore, nothing needs to be configured on the extension itself.

The server must be announced first in TwinCAT Speech (see Documentation of TwinCAT Speech).

Basic configuration

By default, no Speech connection is established. For this purpose, a connection must be actively started by a TypeScript/JavaScript API call.

TcHmi.TcSpeech.openConnection({
   enableMicrophone: true, 
   enableSpeaker: true 
});

Further possible parameters of the call with the respective default value are:

defaultVolume: 1 This is the volume of the sound output when loading the application. This can be set from 0 to 1.

domain: 'TcHmiSpeech' Domain name under which the Speech Extension is reachable.

confidenceThreshold: 0.2 Each recognition event has a probability that the recognized text corresponds to the specified entry. This threshold value specifies the value from which this is treated as a genuine event. If, for example, 0.9 is specified here, an HMI event is only triggered in the case of a very precise correspondence. (see also the Documentation from Microsoft)

Use of recognized events in the project

Recognized texts call events in the Speech system. These are transferred to the browser and can be used there in the same way as control events (such as myButton.onClick). The Recognition Tag from the SRGS file is taken here as the name.

It can thus be used as an event in the Actions and Conditions Editor in the HMI. The name is entered in the Custom area for this purpose. If, for example, SpeechOnCommand<SET_ENGLISH> is entered here, then an action is started on recognition of the Recognition Tag SET_ENGLISH.

Use via CodeBehind is also possible. Sample code with TypeScript:

TcHmi.EventProvider.register('SpeechOnCommand<SET_ENGLISH>', (e, data: TcHmi.TcSpeech.IEventCallbackParameter) => {
    // Will output:
    // 'The user said a text which triggered the recoginition tag: SET_ENGLISH'
    console.log('The user said a text which triggered the recoginition tag: ' + data.Command);
    if (data.Confidence > 0.9) {
        console.log('The speech recognition was pretty sure about that.');
    }
    if (data.Parameter) {
        console.log('We got a parameter for this recognition: ', data.Parameter);
    }
});

	Available from version 1.12.744.0