Issues with long term unattended testing

Overview

The CX23-R system is designed to support fully unattended testing in remote locations. The primary functional requirement to support this is the following. Whenever a test run is interrupted for any of the following reasons, the CX23-R starts a new test run when it reboots.

  • Power supply is lost
  • Power is switched off (via the front panel push button switch, or a switch connected to the remote power control wires)
  • CX23-R resets on error
  • User initiates a CX23-R reboot using the web browser interface

Ideally, there should be no special system configuration options required to support this mode of testing. However, the CX23-R system is not currently working 100% ideally. The primary usage case where this is a factor is for long term unattended tests that are started/stopped by switching the power supply on/off.

Many improvements were made in the v1.18.0 firmware release to minimize the possibility of data losses. Additional improvements were made in the v1.18.1 release. At this time, there is no known issue that can result in “significant data losses.”

Known Issues

The CX23-R front panel push button switch should not be trusted at this time. We currently believe that the system can “lose track” of the system On/Off state that is changed each time this momentary contact switch is depressed. One instance of this was reported by a customer and it has also been seen once in QA testing. Because on this, there is a new option to completely disable the front panel switch, and also the remote power switch. See Hardware > Controlling a module (Reboot, Flash LED, Update firmware, calibration certificate, etc.) > Disable/Enable power switches for more information.

NOTE
It is also recommended that the System preference for “Max number of normal test restarts on error resets” be set to the maximum value of 10, even though the likelihood of this ever happening is minimal. See System preferences > System > Exception handling for more information about "Max number of normal test restarts on error resets."

Issues with MX modules: There are multiple issues that can come into play when MX modules are used with the CX23-R. These issues typically occur very rarely when only 1 or 2 MX modules are connected. However, as the number of connected MX modules increases and the EX23-R switch is used, the likelihood of these issues showing up increases significantly. Note that these issues come into play only when a test run is started after a power up or reboot.

  1. MX subscription time stamps out of phase tolerance: In brief, this means that one or more of the MX channels would be out of phase by more than the allowable limit of 10 µs if the test run was started. When this happens, the CX23-R reboots the MX module(s) and then reboots itself. If this happens, about 2.5 minutes of test data will be lost.
  2. MX modules never synchronize on power up or reboot: It always takes a little time for the MX modules to synchronize to the CX23-R system using PTPv2. Typically, the time required is a fraction of a minute. However, rarely the MX modules never achieve synchronization. If all MX modules are not synchronized within 5 minutes, the CX23-R reboots all MX modules and then reboots itself. If this happens, about 7.5 minutes of test data will be lost.

NOTE
There is a higher risk of this issue occurring if ports 5 and 6 on the EX23-R are used. Moreover, connecting a Camera to port 5 or 6 is not recommended.

  1. MX modules are not discovered on power up or reboot: The CX23-R discovers MX modules by monitoring the Ethernet connection traffic and waiting for the MX modules to provide an “announcement” message. Typically, this takes no more than a few seconds. However, sometimes no announcement messages are ever received for one or more MX modules. If the CX23-R does not restart a test within 5 minutes of power up or reboot, it will reboot all MX modules and then reboot itself. If this happens, about 7.5 minutes of test data will be lost.

NOTE
There is a higher risk of this issue occurring if the EX23-R is connected to the CX23-R ETH2 port. It is strongly recommended that the EX23-R always be connected to the CX23-R ETH1 port.

Summary

All of the known issues have workarounds that will limit data losses to relatively short periods (i.e., less than 10 minutes) when any of these issues comes into play on a system power up or reboot.