Lowering the Development Costs of Industrial Control Systems through Resource Partitioning (4)

By Kerry Johnson

Quantifying Development Savings

Often, task starvation results in intermittent, unexplained system behavior rather than in hard failures. Consequently, it is difficult to collect the appropriate data for follow-up troubleshooting. In most cases, the troubleshooting activities demand both a breadth and depth of system knowledge and therefore require a team to find and repair the problem. These activities can include the following:

1. A tester creates a problem report describing unexpected behavior that occurs during verification. For example, the HMI screen misses updates or occasionally becomes unresponsive. Because the problem isn't easily reproducible, the tester can't easily collect the right information to assist with problem resolution.

2. The developer makes several trial-and-error attempts to reproduce the problem. Eventually, the developer determines that the problem isn't the HMI, but another process that is consuming too much CPU and starving the HMI task.

3. Troubleshooting and resolution efforts now broaden to include a number of people. The resolution can consist of thread-priority adjustments or of changing a process's behavior.

4. Each affected person makes the necessary modifications and tests, then integrates their changes into the system software build.

5. The tester retests for the problem and closes the problem report. This assumes that no additional problems were introduced as part of the changes.

Based on these assumptions, Table 1 lists the conservative cost per incident.

Table 1 Conservative cost of resolving a task-starvation incident. A moderately sized control

system may experience several such incidents at integration time.

From this example, it is easy to see how task starvation can easily cause development delays; in this case, two to three calendar weeks. Even then, this example considers only four threads ??any industrial control systems have dozens of threads that interact in hundreds of ways as they compete for CPU time. As a result, it is common for several cases of task starvation to occur in even a moderately sized system. By preventing one subsystem from robbing another subsystem of CPU time, partitioning provides an efficient way of dealing with these task starvation issues.

Maximum CPU Utilization

Partitioning schedulers vary. Some strictly enforce CPU budgets at all times, so that each partition will consume its full budget even when it has no work to do. Other implementations can dynamically allocate these unused CPU cycles to other partitions, thereby maximizing overall CPU utilization and allowing the system to handle peak demands.