Calculating the resistance value

The terminal calculates the value of the ADC (filtered, X_adc) with the manufacturer calibration offset (Offset_Vendor) and the manufacturer gain Gain (Gain_Vendor). The user offset (Offset_User) and user gain (Gain_User) are also taken into account.
The two calculations can be activated or deactivated via CoE object 0x80n0:01 and 0x80n0:02.

X_Vendor= (X_adc - Offset_Vendor) * Gain_Vendor / 0x10000 (object 0x80n0:01 activated)
X_Vendor= X_adc (object 0x80n0:01 deactivated)

PDO = (X_Vendor - Offset_User) * Gain_User / 0x10000 (object 0x80n0:02 activated)
PDO = X_Vendor (object 0x80n0:02 deactivated)

This is followed by scaling for the INT value and the REAL value.

Note: the unit of X, Offset, Gain depends on the resolution (see chapter Resolution).

Extended measuring range

All measuring ranges have an option to extend the range by 10% beyond the upper limit. This range is used to allow for the hysteresis of the Autorange function.
If the measured value is within extended measuring range, the Extended Range bit is set to TRUE.

	Left-aligned presentation The extended measuring range cannot be used in left-aligned mode. In this case the Autorange function operates without the 10% measuring range extension. This combination is not permitted.

INT value

The measured value is displayed based on the set presentation.

Presentation	Meaning	Example
0: left-aligned	31 bit+sign, end value:0x7FFFFF00 (extended measuring range cannot be used)	1,000,000,000_dec~4.7 kΩ (range 4)
1: right aligned	24 bit+sign, end value:0x7FFFFF	5,000,000_dec~6 kΩ (range 4)
2: scaled 1 bit / µΩ	Measured value in µΩ (upper limit at 2.147 kΩ)	1,000,000_dec=1 Ω
3: scaled 1 bit / mΩ	Measured value in mΩ (upper limit at 2.147 MΩ, not recommended for measuring ranges below 10 Ω)	1,000,000_dec=1 kΩ
4: scaled 1 bit / Ω	Measured value in Ω (not recommended for measuring ranges below 10 kΩ)	1,000,000_dec=1 MΩ
5: as REAL value	The calculated REAL value is entered instead of the INT value.	-

Table 3: Presentation

Only the REAL representation covers the whole measuring range including the measuring range with uniform accuracy.

REAL value

The following equation is used for calculating the REAL value from the value that has already been offset against the manufacturer and user values (F_scal the respective full scale value (100 mΩ, 1 Ω, 10 Ω, 100 Ω, 1 kΩ, 10 kΩ, 100 kΩ, 1 MΩ and 10 MΩ):

Y_real=Y_A * F_scal * 2^-23