Configuring NTP in an OpenSSI Cluster
=====================================

This is a general document for NTP setup on an OpenSSI cluster. 
The commands and configuration mentioned in this document are for 
Fedora. Most of the configuration and commands used are the same 
across different distributions, except in a few cases.  Wherever 
it is required, distribution specific differences are discussed 
as "NOTE".

The README assumes that NTP was configured on the original
installation node and that OpenSSI has been installed.

NOTE:	On Debian, the equivalent commands for `service` and 
	`chkconfig` are `invoke-rc.d` and `update-rc.d` 
	respectively.  To start or stop a service, use command 
	line arguments "start" and "stop".

NOTE:	Once you perform the following procedures, the Red Hat
	`dateconfig` GUI cannot be used to change the NTP 
	settings.

NOTE:	All shell commands are for bash.

1. Make sure NTP is stopped if it is already running:
	# /sbin/service ntpd off
	
   NOTE: On Debian, the service name is ntp-server.

2.  Generate the NTP key file using command
	# /usr/sbin/ntp-genkeys -f

    NOTE: On Debian, the command is /usr/sbin/ntp-keygen.  You 
          may be required to create a file such as /root/.rnd, 
	  and have an entry in it.  See the standard NTP 
	  documentation.

3.  Each node will need an individual "drift" file. Edit your
    /etc/ntp.conf and make sure there is a driftfile line like
    the following:

        driftfile /var/lib/ntp/drift

    NOTE: On Debian, the default file name will be 
          /var/lib/ntp/ntp.drift.  See the NTP documentation.

4.  If the directory /var/lib/ntp already exists:

        # mv /var/lib/ntp /cluster/node$(where_pid 1)/var/lib

5.  Make /cluster/node<node>/var/lib/ntp directories for all the nodes
    in the cluster.
  	# ( cd /cluster; for d in node*; do \
	  mkdir -p $d/var/lib/ntp; chown ntp:ntp $d/var/lib/ntp; \
	  chmod 0755 $d/var/lib/ntp; done )

    NOTE: For Debian users, there might not be a ntp user or group.
 
6. Make a context-dependent symbolic link in /var/lib for the ntp directory:

	# ln -s /cluster/node\{nodenum\}/var/lib/ntp /var/lib/ntp


NTP Configuration #1: All nodes have access to an external timeserver
====================================================================

This is a very straightforward configuration. All nodes use the same
configuration files with the exception of the drift file we have
already set up. Please see the documentation for the NTP configuration 
file.

Edit your /etc/ntp.conf file:

Assuming your ICS network is 192.168.0.0/255.255.255.0, add the
following lines to /cluster/etc/ntp.conf (The key can be any
one of the keys generated in /etc/ntp/keys by ntp-genkeys.):

 	restrict 192.168.0.0 mask 255.255.255.0
	broadcast 192.168.0.255
	broadcastclient
	trustedkey 1
	requestkey 1

       NOTE: The configuration file has more comments.  Reading the
             comments could help you configure NTP to best meet your
             requirements.

This allows all the machines on the ICS network to use the
machine as a broadcast server and all nodes in the cluster to
use the other's idea of the time. The "requestkey" line is
needed so a script can modify the server configuration on the fly.

If the following lines are present in the ntp.conf, they should be
deleted or commented out:

	server  127.127.1.0     # local clock
	fudge   127.127.1.0 stratum 10

These lines allow ntpd to use the local clock as a time source
that NTP clients can synchronize to. We want only one node in
the cluster to have this setting at any given time: the
current CLMS master. A script named ssi-ntpsetrefclk will
be run from /etc/rc.d/rc and /etc/rc.d/rc.sysrecover to set
the local clock as a reference clock.

Add the ICS addresses for all the nodes to
/cluster/etc/ntp/step-tickers. (Any external servers should already be there.)

Start NTP server.

	/sbin/service ntpd start
        
	NOTE: On Debian, the service name is "ntp-server" and 
	      /etc/rc.nodeinfo should have an entry to start the server 
	      on the init node.

	Assuming your networking is setup correctly, you should see a series
	of messages similar to the following:

	(node 1)
	ntpd: Synchronizing with time server:                      [  OK  ]
	Starting ntpd:                                             [  OK  ]
	(node 2)
	ntpd: Synchronizing with time server:                      [  OK  ]
	Starting ntpd:                                             [  OK  ]


NTP Configuration #2: 
======================

A limited number of nodes have access to the external time server. 
The nodes with the external access should synchronize to the external 
clocks; all other nodes should synchronize with those nodes via the 
ICS network.

For the nodes in communication with the external network, you need one
ntp.conf file; for the nodes that are only on the internal network you
need another. This means we need another set of context-symlinks.

	mv /etc/ntp.conf /cluster/etc
	ln -s /cluster/node\{nodenum\}/etc/ntp.conf /etc/ntp.conf


Set up /cluster/etc/ntp.conf as for #1.  This ntp.conf file is used only by 
the nodes with access to the external time servers. (Note: all possible CLMS 
masters must have external access for things to work properly.)  For the 
step-tickers file, add the ICS addresses of only those nodes that can talk to 
the external servers.

NOTE:	Debian does not use step-tickers.  See below.  

Copy the /cluster/etc/ntp.conf to /cluster/etc/ntp.conf.dep and remove
the lines referring to external servers and the "broadcast" line and
the "requestkey" line. (The "broadcastclient" line is still required.)
This ntp.conf will be used by nodes without external access. (These
nodes are NTP clients only.)

For each node that has access to an external time server:

	ln -s /cluster/etc/ntp.conf /cluster/node<number>/etc/ntp.conf

For example node 1:

	ln -s /cluster/etc/ntp.conf /cluster/node1/etc/ntp.conf

For all nodes that do not have access to the external time servers:

	ln -s /cluster/etc/ntp.conf.dep /cluster/node<number>/etc/ntp.conf

Also, assuming that any node now added to the cluster will not have
access to the external network, make the following symlink as well.

	ln -s /cluster/etc/ntp.conf.dep /cluster/nodetemplate/etc/ntp.conf

All new nodes will get this symlink when they are added.

RedHat passes the servers in the step-tickers file to the ntpdate
command and uses it to set the time on boot. The ntp-server on the
CLMS master will take approximately 1 minute to stabilize. During this
window, nodes joining the cluster will get the following error:

	ntpd: Synchronizing with time server:			[FAILED]

The time on the node should have been set within a second by via the
ssi-timesync script and ntpd should begin to correct the time once the
server on the CLMS master node becomes ready.


NOTE:	For Debian, there is no step-tickers file.  Make sure the 
	ntpdate package is installed, and update the 
	/etc/default/ntp-servers with your favorite timeserver. 
	Add "ntpdate initnode Y" to /etc/rc.nodeinfo.

	A sample configuration file is shown here.  It should work 
	on all distributions:

##################################################
### begin Debian ntp.conf for Configuration #2 ###
##################################################
# /etc/ntp.conf, configuration for ntpd

broadcast 192.168.0.255 ttl 6
restrict 192.168.0.0 mask 255.255.255.0

# ntpd will use syslog() if logfile is not defined
logfile /var/log/ntpd

driftfile /var/lib/ntp/ntp.drift
#statsdir /var/log/ntpstats/

#statistics loopstats peerstats clockstats
#filegen loopstats file loopstats type day enable
#filegen peerstats file peerstats type day enable
#filegen clockstats file clockstats type day enable

### lines starting 'server' are auto generated,
### use dpkg-reconfigure to modify those lines.

server 127.127.1.0     # local clock
fudge  127.127.1.0 stratum 10

# Need a few external sources before your own server 
# believes in its own time serving capabilities
server time_server_01
server time_server_02
server time_server_03

authenticate yes
keys    /var/lib/ntp/ntp.keys
################################################
### end Debian ntp.conf for Configuration #2 ###
################################################

######################################################
### begin Debian ntp.conf.dep for Configuration #2 ###
######################################################
# /etc/ntp.conf, configuration for ntpd

broadcastclient

# ntpd will use syslog() if logfile is not defined
logfile /var/log/ntpd

driftfile /var/lib/ntp/ntp.drift
#statsdir /var/log/ntpstats/

#statistics loopstats peerstats clockstats
#filegen loopstats file loopstats type day enable
#filegen peerstats file peerstats type day enable
#filegen clockstats file clockstats type day enable

### lines starting 'server' are auto generated,
### use dpkg-reconfigure to modify those lines.

#server 127.127.1.0     # local clock
#fudge  127.127.1.0 stratum 10

authenticate yes
keys    /var/lib/ntp/ntp.keys
####################################################
### end Debian ntp.conf.dep for Configuration #2 ###
####################################################


Wait about five minutes, and check your initnode with ntpq:

# ntpq

ntpq> pe
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 192.168.0.255   0.0.0.0         16 -    -   64    0    0.000    0.000 4000.00
 LOCAL(0)        LOCAL(0)        10 l   39   64  377    0.000    0.000   0.008
+cudns.cit.corne gps1.tns.its.ps  2 u   98  256  377   46.723   -7.012   5.960
+louie.udel.edu  huey.udel.edu    2 u   27  256  377   23.131   -1.695   0.874
*ns1.usg.edu     ntp0.mcs.anl.go  2 u  229  256  377   35.252   -4.181   3.286

ntpq> as
ind assID status  conf reach auth condition  last_event cnt
===========================================================
  1   364  8000   yes   yes  none    reject
  2   365  9014   yes   yes  none    reject   reachable  1
  3   366  9414   yes   yes  none  candidat   reachable  1
  4   367  9414   yes   yes  none  candidat   reachable  1
  5   368  9614   yes   yes  none  sys.peer   reachable  1

ntpq> rv
status=0664 leap_none, sync_ntp, 6 events, event_peer/strat_chg,
version="ntpd 4.1.2a@1:4.1.2a-2 Tue Nov 11 11:33:28 UTC 2003 (2)",
processor="i686", system="Linux2.6.10", leap=00, stratum=3,
precision=-17, rootdelay=52.037, rootdispersion=29.729, peer=368,
refid=ns1.usg.edu,
reftime=c3df82e7.4c49cb25  Thu, Feb 19 2004 13:45:59.298, poll=8,
clock=c3df83d4.c7574f72  Thu, Feb 19 2004 13:49:56.778, state=4,
offset=-2.768, frequency=-185.529, jitter=4.524, stability=0.030
ntpq> 


Configuration #3: 
================

A cluster without access to any external NTP timeservers. All nodes 
should synchronize with the CLMS master via the ICS network.

As #2 above, except there are no external servers listed in the
ntp.conf or step-tickers files. The ntp.conf should be used by all the
potential master nodes in the cluster; the step-tickers file should
contain all the potential master nodes in ther cluster. The
ntp.conf.dep should be used by all non-master nodes.

Here is an example ntp.conf:

# Prohibit general access to this service.
restrict default ignore
# External servers
restrict 192.168.0.10 mask 255.255.255.255 nomodify notrap noquery
server 192.168.0.10	# linuxbuild1
restrict 192.168.0.11 mask 255.255.255.255 nomodify notrap noquery
server 192.168.0.11	# kahuna

restrict 192.168.0.0 mask 255.255.255.0
broadcast 192.168.0.255 key 1
broadcastclient
trustedkey 1
requestkey 1 

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1 


# -- CLIENT NETWORK -------
# Permit systems on this network to synchronize with this
# time service.  Do not permit those systems to modify the
# configuration of this service.  Also, do not use those
# systems as peers for synchronization.
# restrict 192.168.1.0 mask 255.255.255.0 notrust nomodify notrap


# --- OUR TIMESERVERS ----- 
# or remove the default restrict line 
# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.

# restrict mytrustedtimeserverip mask 255.255.255.255 nomodify notrap noquery
# server mytrustedtimeserverip


# --- NTP MULTICASTCLIENT ---
#multicastclient			# listen on default 224.0.1.1
# restrict 224.0.1.1 mask 255.255.255.255 notrust nomodify notrap
# restrict 192.168.1.0 mask 255.255.255.0 notrust nomodify notrap


# --- GENERAL CONFIGURATION ---
#
# Undisciplined Local Clock. This is a fake driver intended for backup
# and when no outside source of synchronized time is available. The
# default stratum is usually 3, but in this case we elect to use stratum
# 0. Since the server line does not have the prefer keyword, this driver
# is never used for synchronization, unless no other other
# synchronization source is available. In case the local host is
# controlled by some external source, such as an external oscillator or
# another protocol, the prefer keyword would cause the local host to
# disregard all other synchronization sources, unless the kernel
# modifications are in use and declare an unsynchronized condition.
#
#server  127.127.1.0 # local clock
#fudge	127.127.1.0 stratum 10	

#
# Drift file.  Put this in a directory which the daemon can write to.
# No symbolic links allowed, either, since the daemon updates the file
# by creating a temporary in the same directory and then rename()'ing
# it to the file.
#
driftfile /var/lib/ntp/drift
broadcastdelay	0.008

#
# Authentication delay.  If you use, or plan to use someday, the
# authentication facility you should make the programs in the auth_stuff
# directory and figure out what this number should be on your machine.
#
authenticate yes

#
# Keys file.  If you want to diddle your server at run time, make a
# keys file (mode 600 for sure) and define the key number to be
# used for making requests.
#
# PLEASE DO NOT USE THE DEFAULT VALUES HERE. Pick your own, or remote
# systems might be able to reset your clock at will. Note also that
# ntpd is started with a -A flag, disabling authentication, that
# will have to be removed as well.
#
keys		/etc/ntp/keys