KEVOS® MAKEOS · Quality

Linear Calibration Check (Straight-Line Fit, Calibrated Value, Calibration Interval)

I have measured a set of reference standards with my instrument: what straight-line calibration do they give, what is the true value for a new reading, and is the uncertainty of that calibrated value within my tolerance?

Straight-line calibration from up to eight reference standards: the least-squares intercept and slope with their standard errors, the residual standard deviation, t tests of zero intercept and unit slope, the calibrated value for a new instrument reading by inverting the line, and the approximate calibration interval for that value at your confidence level, against the half-width you can accept.

Open the calculatorRuns in your browser. No account needed.

How this tool works

Inputs, units and where to find them

InputUnitTypical source
Number of standards usedstandardsHow many standards below to use (at least 3).
Reference values and instrument readings
Standard 1 reference valueCertified value of reference standard 1.
Standard 2 reference valueCertified value of reference standard 2.
Standard 3 reference valueCertified value of reference standard 3.
Standard 4 reference valueCertified value of reference standard 4.
Standard 5 reference valueCertified value of reference standard 5.
Standard 6 reference valueCertified value of reference standard 6.
Standard 7 reference valueCertified value of reference standard 7.
Standard 8 reference valueCertified value of reference standard 8.
Standard 1 readingInstrument reading when measuring standard 1.
Standard 2 readingInstrument reading when measuring standard 2.
Standard 3 readingInstrument reading when measuring standard 3.
Standard 4 readingInstrument reading when measuring standard 4.
Standard 5 readingInstrument reading when measuring standard 5.
Standard 6 readingInstrument reading when measuring standard 6.
Standard 7 readingInstrument reading when measuring standard 7.
Standard 8 readingInstrument reading when measuring standard 8.
New measurement
New instrument readingInstrument reading to convert to a calibrated value.
Confidence levelchoiceConfidence level of the calibration interval.
Half-width acceptedLargest half-width of the calibration interval you accept, in reference units.

What each input means

Open an input to see what it is, its units, the worked example and the common mistakes, with a small diagram.

What this means: Number of standards used
Illustration: Number of standards usedA row of identical items being counted; Number of standards used is how many there are.nExample: 6 standardsNumber of standards used
What it is
How many of the calibration standards listed below are used in the fit, counting from the first.
Units and format
Enter a number in standards: standards. The form accepts 3 standards to 8 standards. Use a whole number. Required: the tool does not calculate until every input is valid.
Example
The form opens with 6 standards as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Standards beyond this count are ignored; more standards narrow the interval.
Get it right
Use at least four standards spread over the working range; three leave only one degree of freedom.

Reference values and instrument readings

What this means: Standard 1 reference value
Illustration: Standard 1 reference valueA number line with a marker at the value of Standard 1 reference value, between its lowest and highest accepted values.−10⁹10⁹X1Example: 0Standard 1 reference value
What it is
The certified (reference) value of calibration standard 1, the true value the instrument should read.
Units and format
Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
Example
The form opens with 0 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Sets where this standard sits on the calibration line; spreading the standards across the range narrows the interval.
Get it right
Take standard 1’s value from its calibration certificate, in the units you want the corrected readings in.
What this means: Standard 2 reference value
Illustration: Standard 2 reference valueA number line with a marker at the value of Standard 2 reference value, between its lowest and highest accepted values.−10⁹10⁹X2Example: 20Standard 2 reference value
What it is
The certified (reference) value of calibration standard 2, the true value the instrument should read.
Units and format
Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
Example
The form opens with 20 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Sets where this standard sits on the calibration line; spreading the standards across the range narrows the interval.
Get it right
Take standard 2’s value from its calibration certificate, in the units you want the corrected readings in.
What this means: Standard 3 reference value
Illustration: Standard 3 reference valueA number line with a marker at the value of Standard 3 reference value, between its lowest and highest accepted values.−10⁹10⁹X3Example: 40Standard 3 reference value
What it is
The certified (reference) value of calibration standard 3, the true value the instrument should read.
Units and format
Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
Example
The form opens with 40 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Sets where this standard sits on the calibration line; spreading the standards across the range narrows the interval.
Get it right
Take standard 3’s value from its calibration certificate, in the units you want the corrected readings in.
What this means: Standard 4 reference value
Illustration: Standard 4 reference valueA number line with a marker at the value of Standard 4 reference value, between its lowest and highest accepted values.−10⁹10⁹X4Example: 60Standard 4 reference value
What it is
The certified (reference) value of calibration standard 4, the true value the instrument should read.
Units and format
Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
Example
The form opens with 60 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Sets where this standard sits on the calibration line; spreading the standards across the range narrows the interval.
Get it right
Take standard 4’s value from its calibration certificate, in the units you want the corrected readings in.
What this means: Standard 5 reference value
Illustration: Standard 5 reference valueA number line with a marker at the value of Standard 5 reference value, between its lowest and highest accepted values.−10⁹10⁹X5Example: 80Standard 5 reference value
What it is
The certified (reference) value of calibration standard 5, the true value the instrument should read.
Units and format
Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
Example
The form opens with 80 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Sets where this standard sits on the calibration line; spreading the standards across the range narrows the interval.
Get it right
Take standard 5’s value from its calibration certificate, in the units you want the corrected readings in.
What this means: Standard 6 reference value
Illustration: Standard 6 reference valueA number line with a marker at the value of Standard 6 reference value, between its lowest and highest accepted values.−10⁹10⁹X6Example: 100Standard 6 reference value
What it is
The certified (reference) value of calibration standard 6, the true value the instrument should read.
Units and format
Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
Example
The form opens with 100 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Sets where this standard sits on the calibration line; spreading the standards across the range narrows the interval.
Get it right
Take standard 6’s value from its calibration certificate, in the units you want the corrected readings in.
What this means: Standard 7 reference value
Illustration: Standard 7 reference valueA number line with a marker at the value of Standard 7 reference value, between its lowest and highest accepted values.−10⁹10⁹X7Example: 50Standard 7 reference value
What it is
The certified (reference) value of calibration standard 7, the true value the instrument should read.
Units and format
Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
Example
The form opens with 50 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Sets where this standard sits on the calibration line; spreading the standards across the range narrows the interval.
Get it right
Take standard 7’s value from its calibration certificate, in the units you want the corrected readings in.
What this means: Standard 8 reference value
Illustration: Standard 8 reference valueA number line with a marker at the value of Standard 8 reference value, between its lowest and highest accepted values.−10⁹10⁹X8Example: 50Standard 8 reference value
What it is
The certified (reference) value of calibration standard 8, the true value the instrument should read.
Units and format
Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
Example
The form opens with 50 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Sets where this standard sits on the calibration line; spreading the standards across the range narrows the interval.
Get it right
Take standard 8’s value from its calibration certificate, in the units you want the corrected readings in.
What this means: Standard 1 reading
Illustration: Standard 1 readingA number line with a marker at the value of Standard 1 reading, between its lowest and highest accepted values.−10⁹10⁹Y1Example: 0.3Standard 1 reading
What it is
The instrument’s reading when measuring calibration standard 1.
Units and format
Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
Example
The form opens with 0.3 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Changes the fitted slope and intercept and the residual scatter, which sets the interval width.
Get it right
Let the instrument settle before reading standard 1; average repeat readings if the display fluctuates.
What this means: Standard 2 reading
Illustration: Standard 2 readingA number line with a marker at the value of Standard 2 reading, between its lowest and highest accepted values.−10⁹10⁹Y2Example: 20.6Standard 2 reading
What it is
The instrument’s reading when measuring calibration standard 2.
Units and format
Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
Example
The form opens with 20.6 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Changes the fitted slope and intercept and the residual scatter, which sets the interval width.
Get it right
Let the instrument settle before reading standard 2; average repeat readings if the display fluctuates.
What this means: Standard 3 reading
Illustration: Standard 3 readingA number line with a marker at the value of Standard 3 reading, between its lowest and highest accepted values.−10⁹10⁹Y3Example: 40.4Standard 3 reading
What it is
The instrument’s reading when measuring calibration standard 3.
Units and format
Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
Example
The form opens with 40.4 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Changes the fitted slope and intercept and the residual scatter, which sets the interval width.
Get it right
Let the instrument settle before reading standard 3; average repeat readings if the display fluctuates.
What this means: Standard 4 reading
Illustration: Standard 4 readingA number line with a marker at the value of Standard 4 reading, between its lowest and highest accepted values.−10⁹10⁹Y4Example: 61.1Standard 4 reading
What it is
The instrument’s reading when measuring calibration standard 4.
Units and format
Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
Example
The form opens with 61.1 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Changes the fitted slope and intercept and the residual scatter, which sets the interval width.
Get it right
Let the instrument settle before reading standard 4; average repeat readings if the display fluctuates.
What this means: Standard 5 reading
Illustration: Standard 5 readingA number line with a marker at the value of Standard 5 reading, between its lowest and highest accepted values.−10⁹10⁹Y5Example: 80.7Standard 5 reading
What it is
The instrument’s reading when measuring calibration standard 5.
Units and format
Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
Example
The form opens with 80.7 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Changes the fitted slope and intercept and the residual scatter, which sets the interval width.
Get it right
Let the instrument settle before reading standard 5; average repeat readings if the display fluctuates.
What this means: Standard 6 reading
Illustration: Standard 6 readingA number line with a marker at the value of Standard 6 reading, between its lowest and highest accepted values.−10⁹10⁹Y6Example: 101.2Standard 6 reading
What it is
The instrument’s reading when measuring calibration standard 6.
Units and format
Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
Example
The form opens with 101.2 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Changes the fitted slope and intercept and the residual scatter, which sets the interval width.
Get it right
Let the instrument settle before reading standard 6; average repeat readings if the display fluctuates.
What this means: Standard 7 reading
Illustration: Standard 7 readingA number line with a marker at the value of Standard 7 reading, between its lowest and highest accepted values.−10⁹10⁹Y7Example: 50.5Standard 7 reading
What it is
The instrument’s reading when measuring calibration standard 7.
Units and format
Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
Example
The form opens with 50.5 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Changes the fitted slope and intercept and the residual scatter, which sets the interval width.
Get it right
Let the instrument settle before reading standard 7; average repeat readings if the display fluctuates.
What this means: Standard 8 reading
Illustration: Standard 8 readingA number line with a marker at the value of Standard 8 reading, between its lowest and highest accepted values.−10⁹10⁹Y8Example: 50.5Standard 8 reading
What it is
The instrument’s reading when measuring calibration standard 8.
Units and format
Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
Example
The form opens with 50.5 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Changes the fitted slope and intercept and the residual scatter, which sets the interval width.
Get it right
Let the instrument settle before reading standard 8; average repeat readings if the display fluctuates.

New measurement

What this means: New instrument reading
Illustration: New instrument readingA number line with a marker at the value of New instrument reading, between its lowest and highest accepted values.−10⁹10⁹Y′Example: 50New instrument reading
What it is
The new instrument reading you want to convert to a calibrated (true) value using the fitted line.
Units and format
Enter a plain number with no unit. Negative values are accepted. Required: the tool does not calculate until every input is valid.
Example
The form opens with 50 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Moves the calibrated value; the interval is narrowest near the middle of the standards.
Get it right
Read it under the same conditions as the standards were measured.
What this means: Confidence level
Illustration: Confidence levelThe options for Confidence level, with the one currently selected highlighted.90%95%99%Example: 95%Confidence level
What it is
The confidence level of the calibration interval around the calibrated value.
Units and format
Choose one of 3 options from the list: 90%; 95%; 99%.
Example
The form opens with “95%” selected as a worked example. Choose the option that matches your case.
Effect on the result
A higher confidence widens the interval.
Get it right
Ninety-five per cent is usual for calibration certificates and uncertainty statements.
What this means: Half-width accepted
Illustration: Half-width acceptedA nominal size line with an upper and a lower limit around it; parts inside the band are acceptable. The figure illustrates Half-width accepted.upperlower±Example: 1Half-width accepted
What it is
The largest half-width of the calibration interval you can accept, in the units of the reference values.
Units and format
Enter a plain number with no unit. The form accepts 0.000001 or more. Required: the tool does not calculate until every input is valid.
Example
The form opens with 1 as a worked example. It is an illustration, not a recommended value: replace it with your own figure.
Effect on the result
Only moves the pass or fail line; the interval does not change.
Get it right
Set it from the tolerance of the measurement the instrument is used for, typically a quarter to a tenth of it.

Method and formulas

  1. Least-squares line Y = a + bX through the readings Y against the reference values X; residual standard deviation s with ν = n − 2 degrees of freedom; s_b = s ÷ √Sxx, s_a = s √(1 ÷ n + X̄² ÷ Sxx).
  2. Calibrated value X′ = (Y′ − a) ÷ b; tests of a = 0 and b = 1 compare |a| ÷ s_a and |b − 1| ÷ s_b with t₁₋α/₂,ν (NIST e-Handbook 2.3.6.6).
  3. Calibration interval: the values of X where Y′ − (a + bX) = ± t₁₋α/₂,ν σ_p(X), with σ_p(X) = s √(1 + 1 ÷ n + (X − X̄)² ÷ Sxx), the standard deviation of a new reading (NIST e-Handbook 4.5.1.2 and 4.5.2.1), solved exactly as a quadratic.
  4. Example: reading 50 calibrates to 49.289, interval 48.566 to 50.012 (± 0.72309).

Limitations

  • Straight line only, with the same scatter at every level; a curved or uneven response needs a quadratic or weighted fit.
  • Few standards give wide intervals; use standards that span the working range, with repeats.
  • Example values are illustrations. Replace them with values for your actual task.

Typical uses

  • Calibrate a load cell, pressure transducer or thermocouple against standards
  • Correct readings from an instrument with a gain or offset error
  • Check whether a calibration is good enough for a tolerance

Who it is for

  • Calibration and metrology technicians
  • Test and laboratory engineers
  • Quality engineers

What you get

  • Intercept, slope and residual standard deviation
  • Tests of zero intercept and unit slope
  • Calibrated value for the new reading
  • Calibration interval and its half-width

Interpreting the result

  • If the intercept test fails the instrument has a constant offset; if the slope test fails the error grows with size (NIST 2.3.6.6). If both pass, the instrument may not need correcting.

References

  • NIST/SEMATECH e-Handbook of Statistical Methods, 2.3.6.6 Calibration of future measurements — https://www.itl.nist.gov/div898/handbook/mpc/section3/mpc366.htm
  • NIST/SEMATECH e-Handbook of Statistical Methods, 4.5.2.1 Single-Use Calibration Intervals — https://www.itl.nist.gov/div898/handbook/pmd/section5/pmd521.htm
  • NIST/SEMATECH e-Handbook of Statistical Methods, 4.5.1.2 How can I predict the value and estimate the uncertainty of a single response? — https://www.itl.nist.gov/div898/handbook/pmd/section5/pmd512.htm
  • Neter J., Wasserman W. and Kutner M. H., Applied Linear Regression Models (Irwin, 1983): variance of the fitted value on a straight line.

Learn the Concepts Behind This Tool

These KEVOS® Library readings explain the method this tool uses, with worked examples.

KEVOS® tools are screening and decision-support utilities. Verify important financial, engineering, safety, legal or operational decisions using appropriate professional standards and your organisation’s approved data.

Open the Linear Calibration Check (Straight-Line Fit, Calibrated Value, Calibration Interval) calculator

Quality

Sine Bar Height

What height difference sets this angle?

About this tool →Quality

Process Capability from Summary Data

What Cp and Cpk follow from this mean and within-process spread?

About this tool →Quality

Independent Dimensional Uncertainty Budget

How do these independent measurement contributions combine?

About this tool →Quality

Dovetail Measurement over Rods

What should the measurement over (or between) rods read for this dovetail?

About this tool →Quality

V-Groove Measurement over a Pin

How far should a gauge pin stand above the surface in this V-groove?

About this tool →Quality

Gauge Block Stack Builder

Which gauge blocks make up this length with the fewest blocks?

About this tool →