Mpumalanga Mesh:Project Chapter
FMFI Peebles Valley Mesh Network, White River, Mpumalanga Province, South Africa
| Researchers: | Dwayne Bailey (dwayne[at]translate.org.za) |
|---|---|
| David Johnson (djohnson[at]csir.co.za) |
Background
ACTS Clinic, a non-government HIV/AIDS clinic in the Masoyi Village near the Kruger National Park in South Africa, is also at the hub of an information and communication project supported by the IDRC. The clinic (green roof in pictures above) has an expensive VSAT connection but through developing a least-cost wifi mesh-network this signal is being shared enabling communication with their hospice nearby (middle picture), the village school, the home of one of the care givers and surrounding farmers.
Not only has the clinic brought hope to many, who have literally risen from their death beds (like Benjamin in the right hand corner) thanks to the ART drugs, but Portia Ndluli, 19 years old, has a new life as well because with access to the internet at her home, this school leaver has developed a new interest in life. She would otherwise be unemployed in this rural enclave. But through connectivity she has become self taught and now has a computer club of 6 youths, who come to her house to learn about the internet and online research and communication.
The tin can “cantennae” used to cover the wireless antennae erected onto of the Ndluli’s house is rusted, but the connection still works very well and Portia, together with the school, hospice, farmers and others are sharing the network at least cost with the clinic. The hospice now has a voice over internet phone connection to the clinic nearby, and will save the staff and clinic thousands of Rands in cellular phone bills.
This Clinic, which is mainly funded by a pharmaceutical company social grant, has brought hope to over 700 patients in the area. The clinic hopes to extend its influence to 32 other public clinics in the area so that the more than 5000 patients can have access to ART. One way to scale up is through improving communications between the clinics.
Strategies for influence
The project team put the infrastructure in place to cover the Peebles Valley with connectivity, with the clinic as the central hub for the initiative. The focus is therefore on the ACTS clinic and the strategies to enhance the use of the network in this situation, before dealing with the expansion to other clinics and the community at large. The initial technical part of the project saw a lot of difficulty and experimenting before the optimum configuration was arrived at and some stability was achieved in the network. The planned components of the project have been enhanced by some unexpected results to create a situation where they now had a far more robust true mesh network with built-in redundancy. The scalability and replicability is apparent. This created the framework for the possibility of organic growth of the network and a range of new users connecting.
The support and maintenances issues then had be dealt with, the main emphasis being that the clinic and its staff should be persuaded to take ownership of the network, maintain it and even improve its functionality. This was combined with promoting the development of a business model for utilisation in which the clinic would be able to recover some of the connectivity costs, by charging selected users, invoicing them and collecting the money. The onus was put on the clinic manager to start selling connectivity into the community, specifically targeting those they were able to pay, like the farmers in the valley.
To make the connectivity component of the network sustainable, attempts were made to build awareness of connectivity costs, to influence the users to be frugal in their use of the resource, but also to post a warning that somewhere down the line, users will be paying for access. Some users may have responded to this information, but usage monitoring was done and a bandwidth and download management system had to be put in place to limit some users to try and remain within the capped download of the satellite connection.
The research objectives of the project was to use the results as the basis for writing case study which supports policy changes with a particular focus on package the experiences for use as a use case study for DoC.
Main contributions
A community-owned mesh network with built-in redundancy has been created in a deep rural setting with a number of topographical constraints. It has been possible to spread the applications over a range of users, from the clinic to a hospice, to framers, to the school and individual users.
The demonstration of the benefits of using mesh networking caught the attention of a local computer shop owner in White River. He now plans to build a similar mesh network to connect outlying farms around White River. This mesh network will eventually connect to the Peebles mesh network and connectivity will then be supplied from a cheaper ADSL link in White River. He will also be responsible for maintaining and growing the mesh network in the area and will run the network as a business with some bandwidth concessions for places like clinics and schools.
Main challenges
The main challenge is the exit strategy that needs to be put in place for the researchers to be able to withdraw and leave behind a workable and viable ICT network in its current setting. There are however currently strong indications that a local computer shop owner in White River will be taking over the maintenance and running of the mesh network.
Boundary Partners
Five Boundary Partners are identified:
- School,
- Farmer,
- AIDS Care Training and Support (ACTS) clinic,
- CSIR, and
- Department of Communications (DoC)
Outcome Challenges and Progress Markers
ACTS Clinic
The Outcome Challenges for different types of personnel in the ACTS clinic are given below.
| ACTS clinic | |
|---|---|
| Doctors | that they actively use the network |
| Nurses | that they integrate it into their daily work and education |
| Clerical staff | that they use it for some of their clerical activities |
The Progress Markers linked to these Outcome Challenges are:
| Expect to See |
|---|
| Training staff on the use of applications/Internet that relate to the network |
| They use the tools that are given to them |
| Budgeting for the maintenance of the network |
| When the mesh is extended to their homes that it is used for research |
| Like to see |
| They use the Internet to research other medical related applications for their network |
| They designate staff to take responsibility for the technical aspects of the network |
| The clinic staff see inter-clinic connectivity as useful |
| Their home connectivity is used by neighbours |
| Love to see |
| A decrease in cellphone usage (increase of VOlP) |
| Find an innovative way to share knowledge that uses their network |
| They discover their own new uses for the network |
| Initiate projects to see other clinics connected |
| Staff at the clinic present use cases of connectivity at conferences |
| That other clinics through ownership of the network can see how other forms of ownership are beneficial e.g. community ownership of the clinic as apposed to it being an external government service |
ACTS Clinic - Harry Munnings
The document lists a second Outcome Challenge and a separate set of Progress Markers for Harry Munnings.
“Harry takes control and responsibility for the running of the Peebles Valley mesh network and internet connectivity.”
| Expect to See |
|---|
| Facilitates training of staff on the use of applications/Internet that relate to the network |
| Sells concept to farmers |
| Budgeting for the maintenance of the network |
| Manages bandwidth and sustainable cost model |
| Like to see |
| Work at continuing relationships with mesh experts/research |
| Profile of Internet usage shows signs of awareness of bandwidth costs due to bandwidth limiting of individual users |
| Set up Internet Cafe used by volunteers and patients |
| Skills training to ARV patients on computers and Internet |
| Invoicing customers |
| Love to see |
| Network is built into their clinic replication mode |
| Processes, skills and service in place to allow network to grow |
| ACTS clinic and the network help build bridges between communities |
| Initiate projects to see other clinics connected |
| Staff at the clinic present use cases of connectivity at conferences |
| General Comment |
| This is a well-structured set of graduated Progress Markers for the operationalisation of the ICT activities |
Strategy maps
The strategy map given focuses on general strategies for the project.
| Causal | Persuasive | Supportive |
|---|---|---|
| I-1 | I-2 | I-3 |
| What will be done to produce our immediate output? | What will be done to build capacity? | How will sustained support guidance or supportive monitoring be provided? |
| Install nodes to link hospice | Train installation person | Helpdesk |
| Firewall clinic from mesh | Place computer in ACTS foyer | Can access network from outside |
| Lightning protection | Sell idea at conservancy meetings | Usage monitoring |
| Equipment backup | ||
| E-1 | E-2 | E-3 |
| What will be done to change the physical or policy environment? | How will you use the media or publications? | What networks/relationship will be established or utilized? |
| Install a demo node at a farm | Help write case study which supports policy changes | Conservancy will support the network |
| Install tools for reselling with environment ??? | Present concept at the conservancy meetings | Package experience as a use case for DoC |
| Document positive effects of community |
Technical
The Peebles Valley mesh network makes use of a commodity low cost WiFi based router, the Linksys WRT54G, running an OpenWRT based operating system called Freifunk. Freifunk makes us of Optimized Link State Routing (OLSR) for it's routing, which is a well known pro-active mesh protocol being used across large parts of Berlin with network sizes up to 400 nodes. The Linksys WRT54G's have been repackaged into waterproof housing so that the entire unit can be mounted on a mast outdoors. A combination of various antennas ranging from 8dBi omni's to 24dBi grids to approximately 12dBi cantennas has been employed based on the distance between nodes and the topology.
The network is able to self-create itself with optimal routes being chosen based on path with the lowest probability of dropping packets. If a gateway to the internet is connected into the mesh, it is also able to automatically advertise this to all the nodes in the network. In our case the gateway is a VSAT link with a 256kb/s downlink and 64kb/s uplink.
A firewall is installed at the clinic to partition the clinic LAN and Internet connectivity from the Peebles valley mesh network. The firewall currently has the capability controlling the amount of bandwidth available to each user or groups of users in the mesh. It can also sub-cap each user to a specific number of Megabytes. Other functions of the firewall are to filter out virus traffic and other unwanted such as windows netbios traffic. We have given 50 Megabytes free to each user in the mesh and will increase this once a payment model is installed.
A server has been installed which contains local content such as Wikipedia, a music server, a local proxy-server, a samba file server, DNS and Asterisk VoIP server. With this server the network is still useful even without the internet connected.
Network Layout
The network for the project currently looks like this:
This network diagram shows the current connectivity between all the nodes in the mesh network and some of the servers and devices that have been installed. As can be seen from the above diagram, there is a fair amount of redundancy in the mesh network, for example, if Sakhile school goes down, network can still reach Agnus' house via the Kok farm. There is still a large dependence on US AID for all connectivity reaching the ACTS server and internet. This is something that should be improved in the future. The pictographs which are red show some future work that will be completed in 2007. The ACTS clinic has been nominated as a site for the installation of a digital doorway, which is a ruggerdized computer in a metal case which is placed in an accessible area so that there is 24 hour access to computer resources for the community.
The link overlay on the satellite diagram gives an idea of the distances between nodes and their positions in Peebles Valley. From the satellite picture you can clearly see the difference between the more affluent area which is used as farm land and the Masoyi tribal area where many people live in poor conditions in basic bungalow housing. The yellow lines show new links that where established, late in 2006 due to antennas shifting and possible vegetative changes.
Challenges
Mesh protocol choice
The mesh was initially deployed using the WDS protocol. From all reports this works successfully in other countries. In fact the simple choice of this protocol almost scuttled the project. We experienced terrible delays due to hidden nodes, but the system would at other time work very reliably. We couldn't make connections over a mere 1 km, where others claimed connections over 2km. All the equipment was deployed and the project lead was simply frustrated. It had a terrible impact on the deployment of the mesh as 3 month full-time on-site had been scheduled but all of it was used trying to solve the unreliability of the network. The farmers who where seen as critical sustainability adopters where never able to see the mesh working reliably. On switching the OLSR protocol the mesh quite literally came to life. All existing nodes started operating reliably as soon as they where upgraded and set-up to use OLSR.
Lightning
We have had surprisingly little damage from lightning. This area has a high prevalence of lightning strikes and loss of equipment due to lightning induced energy spikes. All our equipment has to be constantly connected to power to keep the mesh connected, this places the equipment at a greater risk to power spikes. But we have had no problems from power spikes from the grid and believe that this might be because step-down transformers act as sinks to regulate lightning induced power surges. While equipment connected to telephone lines is easily damaged as there are no similar sinks in the telephone infrastructure. When we did lose equipment it was through lightning at the clinic, but it was not through the electrical grid. The clinic had recently connected all clinic buildings with Ethernet, our equipment was also connected to this Ethernet. The wave front from a nearby lightning strike created a potential difference between the building as it moved over the clinic. This potential difference was enough to blow various routers and equipment. The Ethernet has been replaced by fibre and we've had no such experiences since.
Electricity
Electricity has not been a major issue. The valley does experience some severe outages, but on such occasions all electricity is out so computers and other devices are non-operational. Therefore, no adverse impacts from power loss where experienced and no provision for power redundancy was made.
Internet cut-off
There were two critical issues to Internet cut-off. The first the daily provider cut-off and the second the result of bandwidth capping occurring. The VSAT is cut-off every night by the provider. This is in part due to the capping mindset of broadband provision in South Africa. Many disconnect the user to do packet accounting and then decide if the user may reconnect. The result is that the connection is terminated and user on the mesh are presented with the clinics sign on page. The second related to bandwidth management. New users are often not aware of the cost of the Internet that they use, and are quick to download music. We had to implement bandwidth management which seems tragic as the idea was to provide access yet we spent much time limiting peoples access. The result for the clinic if bandwidth limits are reached is quite severe, the connection is cut-off until the following month, there is no option of a top-up. Both of these where frustrating and indicate the chasm between the objective of legislators and the results in the field. All indicate the truly limited nature of broadband in South Africa and the frustrating waste of resource spent managing usage and keeping usage down.
Managing the bandwidth
The bandwidth was managed using firewall rules installed on the Linksys WRT54G mesh router on the clinic roof. Soft maximum capacity rules were set up for each user. This means that only, for example, HTTP usage is blocked after the user's maximum capacity is reached. This allows other low bandwidth services like instant messaging to carry on functioning. Managing a firewall, however, is a very dynamic process which constantly needs monitoring as users usage behaviour changes and these rules will be adapted as time goes on.
In the usage graph below, you will notice that Agnus had a 100M limit, Sakhile school had a 40M limit and the rest had a 60M limit. The cumulative usage line flattens out as they reach their maximum capacity, but some traffic still creeps through due to, for example, Skype and instant messaging conversations.
Using this system users can still negotiate to have their maximum capacities increased by using instant messaging to speak to the network administrator. This was the case with Agnus' graph. You will notice she was originally limited to 60M and after some negotiating she was granted an extra 40M
Users can now understand how to use the Internet sparingly. By viewing a web page, which shows how much bandwidth they have used, they will start to understand usage behaviour which consume precious bandwidth.
Exciting Technology
Cantenna
We installed a cantenna on the house of one of the clinic councillors. The cantenna was singularly badly built:
- It was the wrong dimension
- The co-axial cable was not soldered but simply Prestick-ed in place
Yet is has continued to work extremely reliably. In later days if turned the wrong way around and instead of facing the school about 500m away it faced the clinic over 2km away. The router then established new connections between itself, the clinic and an even more distant farmer. The fact that it has worked reliably and yet was badly made and poorly orientated give us hope that home made antennas can be relied on for community networks.
New routes for all
The mesh has done as planned in that it has connected with other nodes adding its own redundancy. Apart from the node with the cantenna we also saw the node on the hospice alter orientation in very high winds and suddenly establish new connections with our high point near the clinic. The hospice was a difficult site as we could not clearly establish line of site. We used a high gain antenna and pointed it towards the farmer we new was the highest site. With the change in direction of the antenna it now faced more directly towards the clinic high site. So it now had connections with both the clinic high site and the farm, establishing a web of redundancy.
Long living mesh box
Our key installation on the high point above the clinic is located under the eves of the farm house. This was placed just above 2 DECT telephone boxes that had both been lost to lightning. The owners told us of how lightning has at times arced across the metal railing of the balcony about 5m from our box.
This node has operated for almost 2 years now. It remains on all the time, has had 1 hardware lock-up and has never been damaged by lightning. Either we have not had a bad strike yet or the location under the eves has kept it safe.
Community
Sustainability and cost
The network connects to the Internet using VSAT. The VSAT is paid for by ACTS clinic. Their main use for it was data capturing of patient data. They now have a local server for that as the web-based Internet tool proved to be frustratingly slow. Thus they have no daily demand for the connection. However, they do have a number of foreign volunteers and for these volunteers it is very important that they stay in contact with loved ones. Thus for the clinic this is also valuable.
The VSAT costs around R3 000 per month which is fully covered by the clinic.
Long-term sustainability
The clinic has always wanted cheaper connectivity, but its remote location precludes the supply of ADSL and other connectivity such as leased lines would be too expensive. The clinics IT support company has just begun working on a community mesh in the town of White River, about 20km from the clinic. This has a number of benefits. It creates local skills in mesh networking thus eliminating the need for the project staff to maintain the system Creates the opportunity to link White River to Peebles and thus share low-cost ADSL with the clinic and mesh, probably at no cost.
Clinics commitment
The clinic has refurbished and given their older computers to clinic staff. There is a strong desire to equip their staff members and give them computer skills. Thus we see the continued operation of the mesh secured as the clinic is creating its own demand and has a relationship with technically skilled individuals who are bringing mesh technologies to their own communities.
Hardware costs
None of the hardware was paid for by the community. This does pose an issue for long term viability of the mesh since it is much easier to obtain hardware for 3G on contracts and thus pay similarly priced or even more expensive equipment over 24 months. The equipment is however equivalent to the cost of equipment, installation and first months rental of an ADSL line, so it is not our of reach of most users willing to pay the upfront costs.
Users
There are many user stories we've given a selection here:
Agnes' printer
Agnes received a computer through the project, her other colleagues are now getting similar computers through the clinic. She has now acquired a printer. We are not sure if she purchased it or if it was given to her by the clinic. Agnes now insists that her house must be locked by her children, in case someone steals her printer. We hope this reflects more on a sense of ownership which to us is very encouraging, rather then a sense that technology has isolated members of the community.
Skype-a-holic and local computer club chairperson
Agnes' daughter Portia is one of our most avid users. She has amongst other things learnt how to:
- Email using Google mail
- Play music
- Search the Internet
- Chat using Skype
- Started a local computer club
Portia seems to show us that age does matter. This unemployed school leaver has the time and inclination to experiment. After being taught the basics of surfing and chatting she quickly got up and running. She runs a computer club and shows people things she has learnt on the computer in the last week. When the network is down she plays solitaire and music CDs.
3G for the wealthy
The farmers are now mostly using 3G. It was frustrating that the mesh took so long to overcome their technical hurdles. 3G is still however still a good option for the gateway from the mesh. We were always aware that this might be a problem, thus we added resources such as Wikipedia and music management software onto the network to make the network more valuable.
Linking people
On the site of one of the farmers we discovered a domestic worker who inherited part ownership of a security company. And a young university electrical engineering student, and gardener. While the farmers themselves might be moving to 3G the various staff and tenants on the farms are not. So this is an exciting possibility for the future of the mesh.
School sadness
One of our key nodes is one of the local high schools. We, quite late in the project, discovered that the head of the computer lab could not type on the computer. He clearly had very limited experience. While we discovered this a young teacher walked in, saw the Internet was up and began browsing... for a new job.
Games can take up bandwidth
With our bandwidth management we had placed each group: farmers, school, community, clinic in a separate bucket. Thus we could control and switch of each if they exceeded their allotted bandwidth. The clinic was left with unlimited access, since they pay for the connection. What we did not consider was that when we placed the connection in the hospice, which is 5km from the clinic, that someone would burn all the bandwidth. The son of the head nurse was quick to use the computer, downloading all sorts of games. Many flash games where playable, but Windows based games and demos where clearly not usable on Linux. But not to a new computer user. He burnt through all the available bandwidth without us knowing. We had to adjust the rules on account of this. But since the project team grew up on ZX Spectrum games we where overjoyed, there are budding computer geeks everywhere.
Visitors from afar
The clinic receives many visitors. Some for a few days, but many for months at a time. Many have been frustrated seeing wireless access but having no way of knowing how to connect. The clinic in its wisdom said that they needed a password. Which only fired up their desire.
Sustainability
Ray Smith from a local computer shop in White River, called Compuright, has agreed to take over the support and maintenance of the mesh network. His grand plan is to create a mesh network of his own based on the same concept that we have demonstrated to him and connect farmers in the outlying areas back to White River. He also plans to put up a mesh node on a high site above “Petra college” which is located on one of the largest dolomite domes in the area. This site has line of sight to the ACTS clinic and a water tower and school in White river. Once this is up, Ray will be able to sell his same internet services to ACTs clinic and all the farmers in Peebles Valley. The expensive VSAT link will become redundant and a much cheaper mechanism which shares an uncapped ADSL line throughout the mesh will be used. All left over mesh equipment from the project has been placed in safekeeping at the clinic with a combination lock. Ray will use this equipment, either to repair damaged nodes or to install new nodes in the area.
For places such as schools and people who cannot afford internet access, Ray is looking at the idea of supplying the first 20 to 30 Megabytes free per month and thereafter pay on a per Megabyte basis. The following phases are envisaged for the establishment of this business which will help the mesh to become self-sustaining:
The basic evolution of the network now is something like this:
Stage I: For next 3-6 months: Mesh is maintained by Ray via R20000 kitty established out of left over funds which weren't used to buy equipment.. Ray invoices ACTS and ACTS pays out of kitty. 20-30Meg free cap per month per user is given out to current users. (This is a total of about 240Meg per month out of the 2Gig available on the VSAT for the clinic)
Stage 2: High site at Petra is installed and White River mesh is bridged to Peebles Valley. Ray then sells a service to the current mesh farmers in the area and grows the network in Peebles area. Some users in the Masoyi area are again given a 20-30Meg free per month option but can buy up extra megabytes on a pay per use model. ACTS VSAT becomes redundant.
Stage3: Ray becomes a licensed wireless operator in the area with a VANS converted to infrastructure license. Another encouraging fact is that Ray has employed Lassey who we used originally to help us install many of out mesh nodes in the area. Although many of the farmers in the area have 3G, in reality they are only getting GPRS speeds and they will be very keen to connect to a much higher speed wireless internet service. So now we watch in anticipation as Ray begins sell the mesh network to a wide audience in the Peebles Valley area and area surrounding White River. We placed a node within the accommodation block to provide Internet to all visitors. The clinic loves this and so do all the visitors who can now do email in the bush.
Conclusion
It is easy to get bogged down in the technical parts of mesh network. Our aim was always to make them less technical. We feel that the technology does indeed make things easier, simply because the work is now about setting up antenna and less about managing boxes. The issues of gateway access and the expense of the access, both in cost and the resource and time required to manage the scarce resource still remain a problem. But the users have invigorated us. From gamers chewing up bandwidth, to new Skype users wanting to know how they can get dates. All of this encourages us when we realise that the only limitation is access, the people are willing and ready. The limit to access is not the technology but clearly the custodians of access to technology. Be they the school teacher who limits access to the computer lab, or the telco's and regulators who set structures in place to limit access. Mesh networks we have seen do work. We have not proven that they can remain sustainable but we have shown that there are people, and we hope enough people and enough needs, that such a network could remain operational.