Platform problems, rationale and choises

We did not define a list of requirements up front, or detailed it
during the project. However a number of casual requirements where
generally agreed on; and in order where

-	'Free' - either as in Free Beer or Free Speach
-	Stability
-	Managability
-	Low resource Footprint
-	Allow for the use of low cost x86 hardware and work
	with a reasonably wide range of WiFi hardware.

Although during the initial experiments Linux was used, a number of issues
caused us to experiment with FreeBSD: conflicts when using multiple cards
of the same type; shortage of IRQ with certain PCIC bridges, multitude of
distribution making fault finding hard and finally the splinting in the
communities accross 3 wireless stacks.

This move to FreeBSD brought us one release scheme, policy and
distribution, a single stable Prism/2 support stack and access to a more
mature configuration system more suitable for mass deployment. Another
benefit of the FreeBSD stack are the inclusion of several firmware cutoff
checks; which will restrict the functionality accessible from userland
when too old a version of the firmware is detected. 

It should be noted that each of the 3 wireless stack on Linux has at least
one or two elements which are vastly superior to the rather dated Prism/2
support in FreeBSD; but unfortunately each stack also exhibited
showstoppers; ranging from erroneous detection of cards to kernel 
panics.  And it is not unlikely that a maturing, or crystalizing, of one 
of the Wireless options of linux, will make Linux a viable choise in the 
near future. This, combined for example with a good AODV implementation, 
may again cause us to switch platforms

For the Compaq WL200 card we did have to fix a specific routing bug which
was introduced in between FreeBSD 4.6 and 4.7 as part of the
cardbus/newbus project
[http://wleiden.webweaving.org:8080/svn/node-config/master/patch.wl200]. This has
since been picked up by -current and will be in the next release of
FreeBSD. Prior to 5.0 release some issues where also found and reported
with regard to wicontrol and ifconfig settings not being propagated
properly. These have since been fixed. 

The fact that in FreeBSD most of the configuration is located in just two
locations; and in very few files (such as /etc/rc.conf) vastly simplifies
the setup and maintenance of nodes; and makes automation easy. 

Tools such as 'mergemaster' and the rather elaborate documentation make
upgrades in the field; even across versions reliable and predictable.
Likewise extensive use is made of the 'ports' collection to manage the
versioning and updating of third party packages such as the ISC-Bind,
Zebra and a few others. 

As FreeBSD has just a single version number and release path; the entire
'build' procedure of the master machine from which all nodes are
automatically build entails less than 5kbyte of script; just short of one
page. The 'definition' of a node, including kernel configuration, routing,
antenna frequencies and what not; i.e all that needs to be defined given a
specific version of FreeBSD currently runs at around 12kbyte of data. 

Another, unplanned, benefit of the BSD stack is the rather mature diskless
boot system which is part of the standard distribution; which is an
excelent starting point for automated installers and systems operating
from a read-only storage device. 

Node factory

The node factory consists of a central machine which contains the
distribution(s) to be installed on the node along with a diskless boot
environment. The latter is used during the inital boot as a staging ground
for the actual install. A new machine is connected to the central machine,
booted from floppy using 'etherboot' which subsequently launches into a
diskless FreeBSD install. The latter will detect all hardware; reformat
the hard disk and install the node software. 

stage zero

We build a server with two networkcards and a single wireless networkcard. A
intel based pentium 2 with 256Mb memory and 4Gb hardisk was used. 

The first network card is used to connect to the internet through a 
an adsl (512k/bit downstream). This is needed to
fethch source, binairies, configuration files.  The second
network card is connected to the node on which the software is to
be deployed. The wireless network card is used for the automated
tests during installation.

Version control. 
Most of the configuration files are under version control using a public
Subversion. Node specific files are generated by a separate system called 
Genesis. Subversion is a new breed of CTM; akin to SCCS or CVS - but more
suited to open source code management as it does not require extensive
unix account management and can be configured to use HTTP as its communication
protocol. Currently Subversion is hosted on a remote machine and is connected
to the internet through a DAV module and an apache 2.0 web server. Access
controls are intentionally light and commit access is virtually for the asking.
Genesis is a web based homebuild configuration culcualtor. It a CGI
perl script behind an apache web server and made by Jasper Koolhaas. 

Currently both the Subversion[http://wleiden.webweaving.org:8080/svn/node-config]
and Genesis[http://genesis.wleiden.net] installations are remote; and the
configuration master and nodes to rely on a live internet connnection during
installation and when deployed. However it should be noted that both systems
could be moved to a more 'local' environment close by; or could be replicated.
Sofar this has not been nessesary.

The second nic offers a NFS, an etherboot and a pxeboot enviorment. To enable
NFS normal FreeBSD rc.conf [http://wleiden.webweaving.org:8080/svn/node-config/master/etc/rc.conf]
and export [http://wleiden.webweaving.org:8080/svn/node-config/master/etc/export]
settings are used; etherboot floppys are created using the default verion in 
/usr/ports [http://www.freebsd.org/ports/net.html#etherboot-5.0.5] and the pxeboot environment relies
on the Internet Software Consortsiums DHCP deamon [http://www.freebsd.org/ports/net.html#isc-dhcp3-3.0.1.r9]
using a normal configuration [http://wleiden.webweaving.org:8080/svn/node-config/master/etc/dhcpd.conf]
from the handbook [http://www.freebsd.org/handbook/diskless.html].

In order to allow a newly installed node to access the internet routing
is enabled between the nics by a gateway_enable setting in rc.conf
[http://wleiden.webweaving.org:8080/svn/node-config/master/etc/rc.conf]

The wireless card is only used for testing the interfaces of the new
machine. 

The software was retrieved from the net by dowloading a FreeBSD image.
After installing this on the server to software was upgraded from 4.7 to
current (as of 12.12.02) using the instructions in the freebsd handbook.

The a new bootalbe tree was build by Dirk-Willem van Gulik. The next

XXX wack names - just add me to the ack's or authors.

Using the 'make world' mechanism with a differnt DESTDIR 
[http://wleiden.webweaving.org:8080/svn/node-config/node-config/make-mast.sh
a complete separate freebsd tree is build. Added to this tree at the root is 
a special diskless install kernel 
[http://wleiden.webweaving.org:8080/svn/node-config/master/kernconf/INSTALL] and, in the default
/boot/kernel location a stripped down run time kernel 
[http://wleiden.webweaving.org:8080/svn/node-config/master/kernconf/WLEIDEN]. Furthermore two
addition files are added in the root which will do the actual install; slice.sh
and calc.pl.  An extra
rc.local script http://wleiden.webweaving.org:8080/svn/node-config/etc/rc.local is
added which will run as the final step during the first diskless boot. The latter
causes slice.sh to run; thereby starting the reformatting and install.

The next step is making a bootfloppy for each of the network cards with the
right version of etherboot. On Soekris embedded systems the native PXE boot
is used instead. 

Preperation for building a new nodes means collecting a pentium (or
better) cpu some ram two or three wirelesslan cards and nic. A 500 Mb
hard (or more) will do fine. The bios is set to boot from floppy, or
in the case of the Soekris to boot PXE. Besides the Soekris we've not tried any BIOS-es
or ethernet cardss which supported PXE directly; but have relied on a few
etherboot floppies.

In the first stage the new machine boots from floppydisk to initiate
etherboot. DHCP will assign an IP address and provide the parameters which
allow the node to be to mount the NFS filesystem. After
loading the kernel a script prepares the harddisk by making it bootable, 
defines it partitions and formats filesystem. This is followed by copying
all the files from the server to its own harddisk. After a reboot is has it own 
system to run.

After the reboot is time install some packages as a dhcp-server, a
nameserver, snmp and more. 

The installing is managed by a set homebuild scripts. The reason to do so after a reboot; rather than during the diskless
boot is that the during the initial diskless boot there insuficient memory due
to the diskless system relying on an memory based /var and /tmp overlay.

The scripts are invoked by a state engine at the end of the boot procedure. 
[http://wleiden.webweaving.org:8080/svn/node-config/nodeconfig/etc/wireless.states/]

At the end the new machine will show a list of hostnames. After entering
one at the command line the machine will lookup its configuration in
Genesis. The old file will be backed up and the new ones will be put in
the right places. 

Then is new machine is in operational state en ready to located at
site.



Design rationale
->	simple to set up
->	simple to maintain
->	simple to turn out nodes
->	...
	easy to update, track current
	version control
	cheap