Introduction

This presently small document is intended to be a resource for questions that arise in the day to day operation of the MMT 6.5 meter telescope. At some point this may develop into an operators manual, and you may be reading a printed copy that is being passed off as such. Be aware that this document changes all the time. My intent is to add information to it as questions arise and are resolved so that this will be a resource for the future.

How many computers are involved in controlling the MMT

More than you think. Right now at least 5. First the sun workstation "ringo" is needed so that the VME computers can boot VxWorks. Secondly, "hacksaw" runs as the operator console and presents the various control and status displays to the operator. Finally, we have 3 VME boxes in scattered locations that run respectively the mount, the cell, and the hexapod (these machines are "mmtvme1", "mmtcell", and "hexapod".

Why so much interdependency?

First of all, the VxWorks machines have no local disk, so they have to boot over the network from some machine that does. At this point that machine is "ringo". We have been careful to make sure that all files needed to boot and run any VxWorks machine is on the mountain, so we can still run if our network link downtown is not available. We have also set up another machine "harrison" to have an identical set of files, so it can serve as a backup boot server in the event ringo should fail.

Why be dependent on the network at all?

The convenience is a greater benefit than any concern about reliability. Another view is that in a modern observatory, if the network has problems, nothing is going to get done anyway. If the network breaks, it is going to have to be fixed before anything else goes forward.

Isn't doing telescope control from a linux box over the network to a VxWorks-VME box risky?

It certainly could be if you were stupid about it. The important clarification here is that we don't actually control the telescope from the linux box. We run the user interface on the linux box, and it tells the real-time box how to control the telescope. A number of crucial design principles come to the fore here. First off, no real time control should be done over the network. Anything safety or time critical, such as limit switch handling or fault detection must be done entirely within the VxWorks crate. The network connection could be severed with an axe, ringo could tumble down the side of the mountain, and the mirror support cell would keep on running indefinitely performing all safety checks, and handling any error conditions (just as one example).

Tell me about troubleshooting the mirror support system.

A broad question, but here are some things to get you started.

The log file is kept on hacksaw at: /home/tom/mmt/celllogs/cell.log

What is the rcell program?

It is a program written in C that runs on a Sun or Linux workstation and communicates with the cell computer over the network. It has many options, some of them I can describe right here.

rcell with no options gets and prints a quick status from the cell computer. Among other things it is a good way to find out if the cell computer has been turned on and has booted properly. An example of the output is:

(~/mmt/celllogs) hacksaw $ rcell
Status: Running diagnostics
Mirror is in TEST mode.
boot: 11/17/99 16:06:29
cur:  11/17/99 16:24:32
Comparing the boot and current time is a good way to find out if the cell computer experienced an unexpected reboot, such as from a power failure or some such.

rcell -m gets a full snapshot of all the actuator forces, commanded and monitored. A good thing to do is to pipe this output into sort, so that you get the actuators in numerical order.

rcell -m | sort was run on 11-17-99 and gave the following output. This was at 5.29 degrees elevation angle and while we were experiencing a very large anomalous Z moment, so these forces are not entirely typical.

act    1 yellow     13.91      10.78      -0.00      -0.00
act    2   blue     -0.74      -3.61      -0.00      -0.00
act    3    red     79.20      76.84      -0.00      -0.00
act    4 orange     87.42      85.91      -0.00      -0.00
act    5 yellow    144.07     142.60      -0.00      -0.00
act    6   blue    139.50     137.42      -0.00      -0.00
act    7   blue     74.21      70.29      -0.00      -0.00
act    8    red    510.60     510.07    -599.55    -603.55
act    9   blue    443.36     440.42    -595.63    -595.12
act   10    red     52.24      48.46      -0.00      -0.00
act   11 yellow   -429.38    -433.64     601.52     593.36
act   12 orange   -420.82    -424.58     597.59     595.36
act   13 yellow   -392.32    -389.72     593.66     587.99
act   14 orange    127.46     123.96       8.29       5.65
act   15    red    -18.50     -23.22      -0.00      -0.00
act   16   blue   -480.12    -493.15     599.55     602.80
act   17    red   -399.12    -402.51     595.63     590.82
act   18   blue   -415.71    -417.92     591.70     587.27
act   19    red     94.65      89.00      -0.00      -0.00
act   20 orange    -15.92     -21.06      -0.00      -0.00
act   21 yellow   -474.64    -473.56     597.59     590.62
act   22 orange    444.66     439.42    -593.66    -597.46
act   23 yellow   -207.06    -200.86     347.58     334.62
act   24 orange    106.54     105.36      -0.00      -0.00
act   25    red     19.37      16.30      -0.00      -0.00
act   26   blue    440.83     438.21    -595.63    -593.49
act   27    red    457.19     451.34    -591.70    -594.21
act   28   blue    465.52     459.41    -587.77    -583.63
act   29 yellow     58.66      54.02      -0.00      -0.00
act   30 orange   -333.27    -333.79     597.59     592.35
act   31 yellow   -398.47    -402.50     593.66     589.45
act   32 orange   -414.85    -418.12     589.74     580.43
act   33 yellow    -66.41     -75.22      -0.00      -0.00
act   34    red     46.08      45.47      -0.00      -0.00
act   35   blue    519.88     515.21    -599.55    -603.18
act   36    red    439.27     438.33    -595.63    -600.16
act   37   blue    441.98     439.85    -591.70    -593.73
act   38    red    -66.10     -75.55      -0.00      -0.00
act   39 orange   -376.05    -380.11     601.52     611.39
act   40 yellow   -380.87    -383.74     597.59     602.43
act   41 orange   -392.46    -385.41     593.66     574.30
act   42 yellow    -76.54     -79.74     -12.12     -17.58
act   43   blue    -37.37     -42.37      -0.00      -0.00
act   44    red   -464.79    -461.49     599.55     587.51
act   45   blue   -394.66    -398.47     595.63     578.52
act   46    red    -27.46     -31.96      -0.00      -0.00
act   47 yellow    -34.77     -35.20      -0.00      -0.00
act   48 orange    -96.88     -97.97      -0.00      -0.00
act   49   blue   -121.96    -125.54      -0.00      -0.00
act   50 orange     -2.95      -5.96      -0.00      -0.00
act   51   blue     -4.55      -9.37      -0.00      -0.00
act   52    red    -72.18     -74.41      -0.00      -0.00
act  101 yellow     14.48      10.16      -0.00      -0.00
act  102   blue      0.61       0.08      -0.00      -0.00
act  103    red     80.72      78.39      -0.00      -0.00
act  104 orange     88.08      85.65      -0.00      -0.00
act  105 yellow    145.22     143.68      -0.00      -0.00
act  106   blue    141.93     142.30      -0.00      -0.00
act  107   blue     74.40      74.60      -0.00      -0.00
act  108    red    517.71     513.32    -608.72    -631.86
act  109   blue    456.62     458.44    -612.64    -631.47
act  110    red     55.04      53.34      -0.00      -0.00
act  111 yellow   -432.68    -435.39     606.75     610.20
act  112 orange   -429.40    -429.17     610.68     616.06
act  113 yellow   -405.39    -422.60     614.61     617.81
act  114 orange    142.99     146.63      -8.29     -14.90
act  115    red    -18.32     -21.65      -0.00      -0.00
act  116   blue   -486.17    -487.35     608.72     615.61
act  117    red   -410.56    -418.96     612.64     613.10
act  118   blue   -431.50    -435.44     616.57     592.07
act  119    red     98.02      96.01      -0.00      -0.00
act  120 orange    -16.09     -16.77      -0.00      -0.00
act  121 yellow   -484.02    -484.02     610.68     581.51
act  122 orange    459.95     452.41    -614.61    -615.74
act  123 yellow   -225.75    -245.12     376.37     395.51
act  124 orange    109.00     110.01      -0.00      -0.00
act  125    red     19.35      16.66      -0.00      -0.00
act  126   blue    453.90     462.85    -612.64    -631.31
act  127    red    476.63     477.59    -616.57    -628.79
act  128   blue    491.08     494.96    -620.50    -633.02
act  129 yellow     59.38      65.43      -0.00      -0.00
act  130 orange   -340.21    -343.04     610.68     615.44
act  131 yellow   -409.99    -408.55     614.61     558.16
act  132 orange   -431.92    -430.81     618.53     617.39
act  133 yellow    -63.19     -57.39      -0.00      -0.00
act  134    red     46.38      52.03      -0.00      -0.00
act  135   blue    528.12     533.37    -608.72    -597.93
act  136    red    454.33     476.01    -612.64    -614.86
act  137   blue    462.74     482.68    -616.57    -607.78
act  138    red    -63.10     -57.75      -0.00      -0.00
act  139 orange   -379.49    -372.11     606.75     627.38
act  140 yellow   -388.99    -393.25     610.68     630.84
act  141 orange      0.00       5.11       0.00       5.77
act  142 yellow    -91.98     -88.78      12.12      22.42
act  143   blue    -37.41     -36.33      -0.00      -0.00
act  144    red   -471.05    -475.71     608.72     620.43
act  145   blue   -405.80    -404.64     612.64     625.30
act  146    red    -26.46     -20.66      -0.00      -0.00
act  147 yellow    -34.50     -29.24      -0.00      -0.00
act  148 orange    -96.01     -90.39      -0.00      -0.00
act  149   blue   -120.98    -115.71      -0.00      -0.00
act  150 orange     -2.68       2.67      -0.00      -0.00
act  151   blue     -3.45       2.99      -0.00      -0.00
act  152    red    -70.95     -65.78      -0.00      -0.00

How long does it take to raise the mirror?

The bulk of the time is in getting the system ready to raise the mirror (turning on computers, compressors, opening the air valves, getting the control windows up, running the bump test). The bump test itself takes about 7 minutes. Once all this is ready, it takes 5 minutes and 35 seconds (give or take a few seconds) to go from the mirror down with air on to the mirror up and in the operating position. This breaks down as 3 minutes and 35 seconds doing the lift and 2 minutes moving the mirror to the operating position. These times can be extended if the mirror has been set down in some bizarre location (like by a panic).

Why does it take so long?

The obvious next question is can we raise it faster. (And the short answer is no.) The pacing item here is how fast we can extend the hardpoints. We have spent a fair bit of time finding out how fast we can pulse the stepper motors that drive the hardpoints and this is just the best we can do. The actuators have much more bandwidth and can in fact lift the mirror much faster than anyone or the mirror itself would like.

How does the bump test work?

The bump test is performed (or should be -- better be) every time the support system is turned on and before the mirror is lifted. It is quite important as it can detect severe problems that can endanger the mirror that can be detected in no other way. Specifically, a situation where a force monitor is broken (for example always reading zero), while an actuator is generating an out of control force will be detected by the bump test. In particular, the fact that the force monitor is not working will be detected, and the system should not be run with a faulty force monitor, since we will be unable to detect and respond to more serious problems. In most cases however, the bump test detects more mundane problems.

What the bump test does is to go through all the actuators in the support system one by one and verify that they will both push and pull a nominal amount. At this time the nominal amount is 50 pounds of force.

Each component of an actuator is tested as follows:

The desired force is commanded.
Wait 1 second.
If the monitor shows force within 10.0 pounds, accept +OK.
Wait 1 second.
If the monitor shows force within 10.0 pounds, accept OK.
If the monitor shows force within 16.0 pounds, accept ? OK.
Wait 1 second.
If the monitor shows force within 10.0 pounds, accept OK (slow).
If the monitor shows force within 16.0 pounds, accept ? OK (slow).
Otherwise the actuator fails the bump test.

The above procedure is performed first pushing (positive force), then pulling (negative force). Each axis of a dual actuator is tested independently. They are not tested in X and Y, but rather each "cylinder" and load cell is tested as a unit. System panic checks are performed independently of the bump test and are active while the bump test is running. These tests are described elsewhere.

The axial cylinder/load-cell is called the Main component, and the inclined assembly is called the Aux component. This is true for dual actuators, single actuators have no Aux component.

Exactly why does the support system panic?

First off and above all to protect the mirror, in fact this is the only reason the support system panics. Many things can cause a panic. The most common and important are erroneous forces generated by a single support actuator. At all times when the air is on in the system, the system diagnostics are running (they run twice a second). They cannot actually run significantly faster because the force monitors pass thru multiplexors which have settling times which pace how fast we can read out a full set of force monitor values.

The word "panic" has become a catch-all term for any sort of problem with any system involving a computer. What we specifically mean here by a support system panic is that the supply air valves are closed by the computer resulting in a very rapid setdown of the mirror onto the static supports. This happens in the span of just a few seconds and the rate is determined by the orifice size in the supply air valve assembly. We do not want to do this unless it is really necessary, but it is always viewed as a valid option in situations where the software (or operator), is not happy with continuing to support the mirror.

Why does a single actuator cause a panic?

Each actuator is expected to produce a force within some band around what it is commanded to produce. If it does not, we feel that the actuator is out of control and we call for a panic. The exact limits are as follows:

Panic with air off           -  88.0 pounds.
Warning messages in log file -  22.0 pounds.
Panic during bump test       - 110.0 pounds.
Panic during air on          - 155.0 pounds.
Panic at all other times     - 100.0 pounds.

These all deserve some explanation.

A panic with the air off is a bit of a misnomer, but it means that if an actuator shows a force significantly different than zero, we are unwilling to even consider turning the air on.

We issue warning messages for actuators that are out of tolerance more than we expect any well behaved actuator to be, expecting that these may be in need of attention even though they are not yet causing panics.

We need to open up the tolerance during the bump test since we are jumping the commanded force by 50 pounds and it doesn't respond right away (and we may compare its old response with its new command while it is busy slewing to a new value and panic needlessly).

The alarmingly large value during air on is allowed only for a brief period just after the air valves are opened. In part at least this allows for the -75 pound nominal integrator bias to be active for the initial turnon period. The actuators do produce significant transient forces when air is first enabled and this is necessary to get us up and running.

The value at "all other times" is also larger than anyone would like. Over time this has crept up to almost the bump test value, and I would like to narrow it to something on the order of 40 pounds or so, if our actuators would only cooperate.

Do I always need to lift the mirror before moving the telescope?

No, but it is preferable to lift the mirror before changing elevation angle. If the mirror cannot be lifted (or it is not desirable to wait the time necessary to turn on the support system and lift the mirror), go ahead and move the telescope. Lifting the mirror almost eliminates the problem we have had with sticky static support bolts releasing as we move in elevation angle. However, with the teflon sleeve inserts, this problem is not as severe as it once was.

Is it better to lift the mirror when Zenith pointing?

Not really. It is best to lift the mirror at whatever angle it was set down at. Again this is not an absolute rule, it is acceptable to lift the mirror at any orientation. It may be tricky to lift the mirror at horizon after a panic at horizon, but it is the right thing to do.