Mpumalanga Mesh:Project Progress
Progress Reports
3 April, 2007 Tuesday All mesh nodes working and support and maintenance handed over to Compuright
Woke up wondering how it was going to be possible to get through all the sites I had to visit and possibly do some major repair work at some of them. So I had a double bowl of Pronutro to prepare me for the big day. First stop was US Aid, I found and that Jan and Alan were on holiday but their house sitter, a lovely old welsh lady by the name of Diane, helped me out. I was expecting the worst because the node was not responsive but simply rebooting it again proved to do the trick and I got it up and running within a few minutes – another one of those to flag for Sven Ola. I quickly hooked up her PC to the network and she said it was much faster than the slow telkom line she uses at home. I made contact with Mr Sehlabela from Sikhile high and he agreed to meet me at the school even though it was school holidays. When we got to the school lab, I noticed that it had changed again, they had put all the computers in small cubicles which makes absolutely no sense when you think that learning is all about sharing and group work. I also saw that all the computers had been given static IP's which again makes no sense if you want to have flexibility, are they scared of DHCP? Anyway I didn't want to meddle in the affairs of the computer lab so I quickly had a look at what was going on with the node and found it unplugged and the Ethernet cables were swapped around on the “Power over Ethernet” injector ... mmm more education on how to keep a mesh node running needed. I got Mr Sehlabela's skype up and running again – he had forgotten his password – seems there is a basic law that if someone doesn't use something for more than a few weeks, their password will be quickly forgotten. He wanted me to come and fix his home computer and had a list of things that he wanted help with – I could have helped, but this was not the time and something had to change in his attitude about why we do all of this for him. I still think he feels the rich city people must help him when computers break, they must teach his Java class, they must fix all his networking problems, fix his computer at home. Where is the “I want to turn this situation around and make this lab a dynamic fun learning environment and start putting myself into the problems and start using my own ideas”?. I was shocked that he was teaching his pupils Java, and he hardly knows how to fill in information in a web browser, he also admitted that he knew nothing about what he was teaching them. The sad tale is that this is probably a very common story in thousands of rural schools around the country.
Next stop Kok farm, This was a simple problem, the Linksys was unplugged. The unplugging behaviour, seems to follow a simple law of nature. If this thing doesn't work for a week, then there is no point keeping it plugged in but there is a useful two prong plug adapter here we can use for our hair-dryer when we can't find one. I showed Henry, Kobus' business partner, how the network works and explained the whole concept of mesh to him. He had an interesting question about being payed site rental for someone putting an ugly antenna on your roof – I realised that there is a very entrenched operator based thinking still in peoples minds when it comes to putting up infrastructure, where money is payed to hire site space ... the community ownership model will take a while to sink in. But he was very convinced that selling internet through the mesh will work well amongst farmers because they are unhappy with their slow GPRS. He knew Ray Smith well and was happy that he was taking over the running of the mesh.
I then had to rush back to the accommodation block for lunch with Ray. I handed over all the spare equipment from the project to him and explained what was there. I told him that it would be kept in a locked up trommel with a combination lock at the ACTS clinic for him to use whenever something breaks. I also handed over a manual which explains how the network works and has contact details for all the sights. We discussed the business model for quite a while. He had a good solution to connect the Peebles valley mesh back to white river using a repeater above Petra. This Repeater would have line of site to a school and a water tower in White river which would connect to the new White river and farmer mesh network. He will basically be reselling bulk ADSL at less than half of what they are paying for 3G. He agreed that low income earners across the valley and schools could be given an option of the first 20 or 30 Meg free per month and thereafter having the option of buying up more bandwidth on a pay as you go basis.
Samuel and Luke were getting really restless so I went into White river and took them to a nice place with farm animals, I left Nula and the boys there while I went back to tackle my final site, Legogote hospice.
I got to Legogote and found the mesh router, pole and antenna all lying on the patio. It had been taken down due to some building alterations that were being done at the hospice. It took me about 20 minutes to found all the bits and pieces but after finding everything I found another able bodied male nurse to help me install the node again. We got the bracket and mast up, clamped on the mesh router and re-layed a new Ethernet cable down to the room where they kept the VoIP phone. By now it was getting towards 5 and I had to rush back to pick up the kids, I brought the whole family back to the hospice to finish the job. I took out the old PC that was there which wasn't working, it wasn't really needed anyway because the big shiny new blue 3 terminal digital doorway on the Stoep was far more than this measly computer could ever hope to be. I taped down all the cables, connected the VoIP phone and to my extreme joy I managed to make a phone call back to the Clinic.
By know my body was really aching from the Cold I had caught the day before, so we headed back to the accommodation block, packed up our stuff and I tried to complete my final task of getting the Reverse ssh tunnel working from my linux server. After struggling for about 2 hours with public keys and authentication problems, I eventually hacked together a solution that would work only until the next reboot or until the VSAT reset itself, I gave up trying to get it to work through a reset and hoped that it would still be running by the time I got back to Pretoria to finish the task. By now Nula was not happy with me, it was 8PM and we were going to drive back to Pretoria in the dark and in the rain and arrive back at midnight. So we packed the car and headed back in terrible weather, the rain was so thick that I sometimes had drive at about 40 km/h because my wipers couldn't get enough water off the windscreen. We arrived back in Pretoria at about 1:00AM having achieved the impossible.
2 April, 2007 Monday Lightening strikes the firewall again
Arrived with my car and trailer and family at the ACTS clinic at about 11:30AM. My first set of tasks was to tackle all the problems at the ACTS clinic itself. Harry had moved the firewall and server down to the main clinic building and had no luck getting the firewall up and running. Harry had done a good job, everything was connected up correctly but the firewall was not responding at all. The next thought was to reach the firewall via the serial port, no serial port cross over cables were lying around, so my next task was to hack together a cross over from bits and pieces of modem serial cables I could find. This proved a long and arduous task with my primitive soldering iron – but after about an hour and a half I finally connected up the firewall with my shiny new cross over to the firewall. I poked around in the firewall and found that all the Ethernet ports were down, I later discovered that lightening was the culprit again.
I quickly setup a backup plan and installed a bridge, put the mesh server on the ACTS network subnet, and changed ACTS_CLINIC node IP to a routeable address on the ACTS network. The Linksys would now have to act as a firewall, but I didn't have time to slap together any clever iptables scripts to do what our FreeBSD firewall was doing. The accommodation block linksys node had disappeared, so I put up one of our spares and configured it to dish out DHCP addresses on the wireless network. This worked nicely and I demonstrated a google page loading to my suitably impressed wife in deep rural Africa. I noticed that the ACTS_FLAT node was dead so my next task was to figure this out. I climbed up on the roof and opened the box – all the lights were blinking away but I couldn't get any connectivity on the Ethernet or wireless side. I reset the box and everything worked again – this was another thing to flag – sometimes these nodes get stuck and need to be reset. So I need to send an another email to to Sven Ola, Freifunk's architect, about this. I then hooked up Avery's computer to the network and she was gleaming with joy, now that she could email all her relatives from her flat again. I noticed that the switch that our toasted firewall was plugged into was completely dead, which explains why our poor firewall didn't survive the strike. So now all the mesh networking at ACTS was up and running, it was time for a break and some Battlestar Galactica movies on my laptop. After the movie I decided to try and get reverse port forwarding working from my Linux server so that I can continue supporting the network from home, but had no luck, got tired and went to sleep
13 October, 2006 Friday Heloise visits our sites and finds new people who can use the mesh
Started the day with ACTS staff in their morning prayer and singing time. We hadn't planned to visit Alan and Jan of USAid but it was a very good thing we did. Their node was off, lightning? Well none of that, a house sitter had moved the plug and inadvertantly switched off the node. We got it up and running again. Showed Alan some things and got them onto Skype. Heloise spoke to Jan at length and got the low down on why they didn't want to use the network:
- Goes down at night and weekends
- Unreliable
- 3G allows them to use it when travelling
But what Heloise did coax from Jan was that her housekeeper was doing a computer course. And we then discovered that one of the staff staying on site was studying at the TUT satellite campus in Nelspruit. Studying Engineering! Victoria, the housekeeper had inherited a share in a security business from her brother. Not content to take a passive role she has insisted on getting involved in the business. Firstly getting a driving license and car and now teaching herself about computers. David and Dwayne chatted and it would be quite easy to setup a node on their houses which are slightly away from the main house, but then decided that giving a simple kit might be a better approach. David later got most of the kit together and we dropped it off then we visited Gundella. The site give us a panoramic view of the whole valley. A yagi should give good coverage and its back lobe should pick up the node on USAid. A very good site and also a nice social experiment to get buy in from some real users. Who as Heloise said – have no choice – they cannot afford 3G or other solutions. Off to Hospice to fix the network card and install SIP phone. Heloise and Dwayne left David there and proceeded to the school. Realised that the reason we had a good connection between the Hospice and USAid was because the antennae had shifted about 20 degree off alignment with Kok farm and was now pointing to USAid. It seems that you can in fact go through mountains if you signal is suitably focussed. David decided to reinstall ubuntu with the new network card installed as the ubuntu had been severly disembowelled. After reinstalling the built in ethernet and the new PCI ethernet card were detected and worked perfectly so something went wrong with the drivers in the previous Ubuntu setup.
Our visit to the high school was very frustrating. Mr Sehlabela has been a key person in the school and we got the lay of the land in terms of schools from him. We also got a node up at the school because of him. The school has a lab that was originally insatlled by netday and has been upgraded by Eskom. All key indicators to us that they were active. However, on the day we visited Mr S he asked when the Internet would be up. Dwayne simply switched on the main computer and it was there. It had been there for over 2 months and just needed someone to surf. David had had telephone conversations with Mr S to get him going but he seemed insistant that we do it personally. We got the Internet up and then proceeded to explore. Unfortunately it was in this time we realised that Mr S was not himself computer literate even though he is head of department for the whole computer lab. This was quite shocking and we need to review his role in the mesh. On the upside, at the same time as he was telling us that no one uses the Internet we looked across and saw a young man doing just that – surfing! Dwayne and Heloise spoke to him later. He is a teacher at the school who just happened to walk in while we demoed Google to Mr S. He thought oh the internet is up, turned on a computer and began reading his Yahoo mail.... and looking for a new job :(
Our visit at the school took rather long so David felt a bit abandoned at the Hospice. But he got the Ethernet card in. We met Duminsai's mom (Ainnah Nyundu) who works as a sister at the Hospice. Later feedback from Margie was that Sister Shabs (main Hospice sister) didn't want the computer as her staff weren't doing their jobs! Duminsani's brother, Godfrey who is only 13, was there and David showed him how to install various games. Later in the day we observed him trying to install a Java plugin so that he could play various Java games. See games are the way to build a future computer generation! David also did an interview with Ainnah. He started by asking if she had used a computer, to which the answer was No. At this point all further prepared questions were thrown out and the interview focused on how she managed to raise such bright young children. She has 6 children, 2 are at university, 1 has an IT job, 1 is a chef and the other 2 are still at school, 1 of which is Godfrey our geek in the making. Dumisani, her second son learnt to use a computer while living in Kriel at the Kriel computer centre and at the youth centre. He also did a 6 month microsoft course in Johannesburg at the Standford Business College (Not to be confused with THE Standford!) He then worked at ACTS as a data capturer for a while and is presently doing computer based graphic design for a company in Whiteriver but is very interested in starting his own internet cafe in peebles and possibly being contracted to support the growth of the mesh network.
Back to the clinic for lunch. We had planned to meet with Harry and Margie but since the clinic was quiet for the day we decided to meet later after the clinic closed early. Ray Smith, who does IT support for the clinic, had managed to contact David through the CSIR. Very funny since he didn't know how good his contacts where. We met with him to discuss mesh networks as he wants to set one up in White River. This turned out to be very exciting and Ray had all the best people round the table. Heloise gave some very good input on social and regulatory issues. David on tech and Dwayne – ate sandwiches :) The exciting part is that Ray is seriously looking at linking the clinic to White River and has all the right contacts. White River is ideal as its well healed yet most people are unable to get ADSL. Ray has subsequently told us that even the municipality is prepared to allow them to use their water towers as mesh realy points. One of these water towers can see the mountain behind the clinic and can also reach Margie and Harry's house in white river.
We installed a SIP phone at the clinic reception and then returned to the Hospice to get the other SIP phone installed.
Then off to Portia to get Skype and Gmail up and running. Got it all installed, took some nice pictures of Portia, Agnes and family outside the house with rusty, wrong pointing antenna. Dwayne took some short clips of Portia using the computer and talking about what she does on it. She has certainly searched for every talk show host and their websites! She plays cards, as every computer user seems to, when the Internet is down. She has written a CV. We talked a bit about her helping Mr S at the school to teach computers.
Heloise managed to convince us (and Gundela – a volunteer nurse at the clinic) that we should get internet working for her. This seemed like a good idea actually as Gundella has access to the clinic and could address issues such as the Telkom 24hr Internet switch off. Proceed to her house and to drop of the mesh kit for Alan. Without a USB wifi adapter we decided to us a Linksys and connect the laptop over Ethernet to the Linksys. All was well until we discovered that the laptop has no Ethernet port
Tired we drove home much later then usual.
12 October, 2006 Thursday David swaps the lightening damaged firewall for a shiny new one
Managed to get a car with large boot and no aircon. Heloise from IDRC is coming with us so had some entertaining and thoughtful talks on the way up. Left late as usual.
We arrived at acts at about 9:30PM and David noticed that Avery's light was still on and decided to quickly swap out the toasted firewall. It was literally an plug and play experience – within 30 seconds the network was up and running again. Noticed that US Aid was down Most of the rest of our planned activities for the night like setting up all the SIP phones were held off till the morning as it was getting late.
25 august, 2006 Friday Even an Ethernet connection needs love and care
David was determined to find out what strange things were happening with the firewall and he phoned John, our trusty FreeBSD guru, back at the CSIR to get some advice. It seemed that the Ethernet connection between the firewall and the Linksys box was faulty. Eventually John worked out that it was some low level Ethernet timing problem in the driver and he made the fix – retarted everything and we were back to a free flowing 100Mb/s Ethernet pipe.
We left the final Debian box installing (a trip over a power cable rendered the first partially completed install null and void). This box will be installed at the hospice.
Rushed home so that Dwayne could attend DST's Open Source launch.
24 August, 2006 Thursday Shiny new server with 240Gig harddrive installed
Chris was arriving around lunch time with the shiny server. We decided to go from place to place to get our network up. First US Aid. All seemed OK, power was on. We had to open the box, it seemed to have gone into some non operational state. We reset it and upgraded the firmware. The box was now alive and well. More positive was that the rest of the network looked healthy. The schools was still up, our lesson about leaving it on clearly had worked. Plus Agnus' node was also up.
Next stop Kok Farm. Met with Kobus, he has just got a 3G card which he was struggling with. We plugged the box in, upgraded and checked. The network was up. We promised Kobus that we would be back later in the day.
Now our last node – the Hospice. Kok farm has enough height to, we hope jump over the ridge, to get to the hospice. We were never able to predict line of sight. But today we can try. We arrived and our antennae was now pointing in the oposite direction. David realigned, we upgraded the software again and viola. A connection from the hospice to kok farm. We now have a fully functional network.
Chris met us at the hospice and we showed him around. Back for lunch and a meeting with Chris and Harry while David and Dare install the server. Chris quized Harry about the reason why he was interested in the project, what the clinic wanted and various other things. Chris wrote notes, we must get those :)
Dwayne and Chris visited Agnus' house while David continued with the server. But first stop the school. Being after 13:00 there was no hope that anyone would be around. Luckily the lab was open. Dwayne booted up a computer and checked the net connectivity. All working. Chris and Dwayne talked a bit about the school as a partner and their expectiations, the computer lab, etc
Agnus was at home casting a new front porch. We upgraded the computer and talked to the twins. When asked what they did on the computer they said that they had tried to play music and did a lot of solitaire (like computer users everywhere!). The upgrade went well we even got a h323 session between David and us. David was hacking things on the server.
We did another round of visits the participating nodes, mostly to demonstrate the connection to the new shiny server. The server by the way has: Wikipedia, MediaWiki, a music streamer and ZX Spectrum games. Firstly we visited Alan. We managed to stream music to his computer. It seemed however that Microsoft Media player was not as good as the Linux equivalents at displaying playlists so it was not as impressive. We met Jan on the way out, she has just returned from Jhb. She was singing the praises of 3G. It seems that in this environment it is a real win. It provides enough speed and much much improved reliability, even in this setting. Community networks should probably be using it as their connection to the Internet. Unfortuntely we feel for many people it may negate even the usefullness of a community network, which would be a shame. But with a server with many useful tools it seems a community network can still be useful.
Last stop Kobus at Kok Farm. As always we were running late, but this was going to be a quick one. At our first visit that morning we realised that Kobus would not be able to connect as the node was in another room and not close to his office. We setup a Linksys on his office desk. But couldn't connect, something was dragging our firewall down. We tried various things, even calling Alan to see if he was steaming audio. Eventually David reset the firewall and was able to get some remote desktop control software off the local server and install it on Kobus' laptop in case he needed some remote help. We had to leave it there to get to Harry are Margie for supper. We told him to try the network later and let us know how it worked.
Stopped of at the clinic and saw that many pretty women had left notes for David. Well actually had left notes for the Wifi guys but stuck it on David's door :) Seems the people most interested in this network are visiting clinic volunteers! Promised to look into things.
Finished the day off with supper with Harry and Margie from the clinic.
Returned home and David setup a box which we left in the acomodation block. We decided that we'd need to create a group for volunteers who could use this node in the acomodation block. All people in the block have net connectivity and where rather happy!
23 August, 2006 Wednesday David forgets to bring shiny new server - Chris Morris comes to the rescue
Got car with ... large car. Bounced from a B class to a C class. Must try fathom how David got this right! Needless to say we got a boot. Bought 2 P3 computers for deployment, given up on P2 as they simply can't run Linux. We were delayed as they hadn't quite arrived in time. Arrived at the clinic at around 16:00. Lots of visitors again (Dutch, Brits, Auzies). Did some looking around. David realised that even though we had a large boot he would leave the all too valuable server behind. But Chris came to our rescue. Not much that we were able to do that evening. Installed Agnus' computer instead. Quick analysis showed that US Aid was down so the network did not exist.
13 August, 2006 The power of remote control
USAID had been down for about a week, my fears were that USAID may have been damaged. So I contacted logged into the firewall at ACTS and started a ping session to USIAD. I then phoned Jan at USIAD and asked her to check the power on the node. She wiggled the plug a bit and Viola! the ping came up and I could confirm that the node was indeed alive again - hooray for remote management!
7 August, 2006 Wikipedia PC down
Looks like the power supply is bust, Dumisani checked the PC and couldn't power it up - OK we need to place on our next trip. Again this was due to remote management - noticed the PC wasn't responsive
5 August, 2006 Things going wrong in the mesh
Agnes computer is not connecting to the mesh - Dumisani discovered that the driver for the ethernet interface is not working. Hopefully we can move Dumisani to an advanced level of Linux support so that he can fix driver problems soon. I suggest an advanced Linux course of jim - I sure he will drink it up and enjoy the challenge.
1 August, 2006 We have a local expert at ACTS
I got a phone call from Dumisani at the ACTS clinic, he was sent to fix Agnes' computer. I took him re-installing Ubuntu, I was expecting the usual "Do you see the picture of the world - click on the picture - left, button etc." instead he just said, so what must the IP address be and what username and password should I give Agnes, that's it and he did the rest. Wow! this is truly good news for sustainability of the mesh
July 22, 2006 I can support this network now from my own lounge
I arrive at the clinic at 8:30AM and my good night sleep helps me realise that the link between the firewall and the server should be a cross over cable (this is a PC-to-PC link) luckily, I have one of these in my armoury of equipment. Then I realise that the internet port on the Linksys wireless router was set to 10.3.1.2 and not 10.2.1.2. I change the script on the firewall and reconfigure the interface to 10.3.1.1 – now the link to the wireless router is working. So all the networking stuff is working fine. I login to the wikipedia server and I can reach the outside world via the clinic VSAT. I phone John and ask him to check if the reverse port forwarding is working from Pretoria – he checks and all is well.
This is a big step forward – now we can remotely: Upgrade firmware, Check usage of the network, Change firewall rules, Remotely login to other computers in the mesh and the list goes on.
I neaten up all the cables glue by glueing them onto the wall and skirting, do one last test with my laptop. I open up wikipedia and mediwiki wirelessly from my laptop and login to the firewall and ssh to Pretoria and back again to the clinic – Yipppee! success.
I noticed that the school node is down and on Monday I phone Mr Sehlabela who tells me that all the computers, the server and the multi plug that the wireless router is plugged into are turned off after school every day. ARRGGGHH! I realise that there is still a lot of learning needed on how a wireless mesh network depends on other nodes having reliable power sources (people controlled and power utility controlled) to be viable. I ask him to make sure that the wireless router stays on in the future as it will help us to support his computers remotely and keep the rest of the mesh alive.
Hopefully, in the future, it will be the farmers who depend on the school link making the phone call, asking the teacher to keep the node up. Then we will truly see the symbiotic vision of mesh networking becoming a reality.
July 21, 2006 Firewall and Wikipedia installed
Finally David gets an Avis car with a large boot (Nissan Almera). Leave Pretoria with my wife and kids, Nula, Samuel and Luke. Arrive in White river at about 3:00PM. Nula drops me at clinic and heads back to B&B.
I manage to catch Harry before he heads home and chat about the current state of the ACTS network. Harry explains the layout of the fibre network linking the main ACTS building with the dispensary and the flats. He explains that it is working very well and that the speed of the network makes you almost think a remote PC is like a local hard drive. With the wireless node being on top of the flats and no dedicated fibre for the connecting the wireless node on the flats with the clinic firewall, I realise that the best place to put my firewall and server is in Avery's flat.
I explain the firewall rules that have been set up to save the precious 2Gig cap on the VSAT and Harry agrees that 250Meg for the farmers and 250 Meg for the school should be OK. He is also happy with the dynamic bandwidth allocation rules which look as follows:
The Peebles Valley mesh network in Mpumalanga consists of the following players
ACTS clinic infrastructure Clinic Staff Commercial farmers Schools General community members Currently the VSAT link at the clinic is the only internet access in the Mpumalanga wireless network. A firewall will be setup with three interfaces First interface: 192.168.0.4 to VSAT gateway Second interface 10.4.1.1 to local servers eg. wikipedia and mail Third interface 10.2.1.1 to wireless network The VSAT is a Telkom subscription with a 2 Gig cap 256K down 64K up link. The requirement for the firewall is the following Byte counting rules Schools should receive a total of 250Meg per month Commercial farmers should receive a total of 250Meg per month General community members will have no internet access - only have access to local wikipedia Clinic staff and clinic infrastructure will be uncapped Traffic shaping rules Clinic infrastructure and clinic staff should receive a guaranteed 128K download speed when there is other traffic on the VSAT. Clinic infrastructure and clinic staff should have access to the full 128K when there is no other traffic Schools and farmers will be able to use what ever bandwidth is left once after the clinic and clinic staff have received there guaranteed share. If both schools and farmers are accessing the internet – they should be distributed at a ratio of 2:1 Farmers:School Access control For now access control will be based on IP addresses (see diagram for current IP allocations – a better grouping of IP allocations for different users types might be useful in the future)
I take all my loot up to Avery's flat with the help of some volunteer workers. As I start setting up all the gear in her study, I realise that the Ethernet cable to the wireless box has been cut by the fibre installation guys. They probably thought that this was for some old Ethernet network that is not needed any more (lesson in informing people about what is out there or labelling equipment and cables!!). Harry fetches a crimping tool and some connectors and I crimp a connector back on to my Ethernet cable.
I set up the firewall and the server in the corner of the room – fortunately it doesn't take up too much space and the installation goes of quite smoothly. Then I notice that the firewall interface does not communicate with the server PC – but the link to the switch and the wireless router are active. I also can't make contact with the wireless router on the roof. With that I call it a day realising that I will have to carry on on Saturday.
June 09, 2006 Mini macs in schools and Hospice connected again
Started our day at about 7:30AM - well at least I did – maybe Dwayne and Dare secretly set up a few more mesh nodes without me knowing at the crack of dawn. Noticed that wikipedia hadn't got very far – Dwayne started a parallel process to tar all the files and hopfully copy across some time later in the day – left the two machines whirring away.
First went to the school to turn on the wireless node which was usually unplugged. Met Mr Sehlabela and talked about what had been happening in the school over the last few months. He spoke about the donation from Eskom of a wireless server and wireless cards fot all the PC's in the lab. The contact person is at Eskom is Lucky: 0843223491, the box has www.magwireless.co.za written on it – they are a company that have set up a learning management system for school teachers. The system worked for a day until there was a power failure and then never worked again. <Dwayne – what else did he say?> I remember he just spoke about needing more help in the computer lab
We checked the box and found a Apple minimac with an external USB harddrive connected to a Netgear access point all powered from a UPS. Firstly the Ups wasn;t set to autostart when there is a power outage and the minimac is also not set to auto start when it get power from the UPS. It is very clear that there are no skills in the school to get things fixed when something goes wrong. The other concern is how Eskom could install a system which won't auto start after a power failure.
After fiddling around with the magwire system and thinking about all the better things we could do with a mac mini rather that use it as a server, we made our way to the hospice to get the replacement mesh node installed.
I did a check on the ground to make sure that the wireless and LAN interface was still working properly and discovered that one of the ports (the one that had power over ethernet activated) had been damaged during the fall at the begnning of the year. The only solution was to go and buy a soldering iron at the local hardware.
After a trip to the local bouhandelaar and our stomachs full of local laeveld greasy sausage, we returned to the hospice to finish our job. We re-jumpered the power over ethernet hack to another port, sealed up the box, aligned the grid antenna to Kok farm and finally installed everything onto the pole.
we were hoping to return to the school and demonstrate the new wikipedia on the mesh to Mr Sehlabela, but Wikipedia was still whirring away back at the clinic, so we left Dare to finish the cabling into the hospice and went to the school to at least get all the school lab Pc's linked up to the mesh.
We arrived at the school at about 12:50 and everyone had gone home except the headmaster and the caretaker -Mr Sehlabela had said he would be there until 1:30 but he was nowhere to be seen. The headmaster said we could go ahead and fix up the lab. We went in and hacked a lan card out of an old PC, stuck in in the main server, used windows to bridge the to LAN cards to that the mesh was bridged onto the school network and Voila! it worked. All the lab PC's got a DNS address from the linksys on the roof and could see the clinic – we couldn;t test wikipedia yet as it wasn;t finished copying, but at least we knew it would work in principle.
Now we were on a high – Dwayne went back to the clinic to try and finish setting up wikipedia and I picked up Agnes' refurb PC and went to pick up Dare. When I gopt to the hospice Dare had done a sterling job of the cabling and we proceeded to Agnes' house stocked with our gear. After some 4x4 driving with a Hyundai Getz – which Dare assured me was tame compared to Nigeria's roads we arrived at Agnes' house and started installing her new refurb (oxymoron there?) Everything went well until we noticed the Compaq refurn didn't have a standard 220V monitor plug at the back for the monitor – no problem I hacked a plug off my soldering iron a remade the monitor power cable to fit.
Switched on the PC, connected it to the mesh node and sadly – it didn't want start up the LAN interface using the built in LAN port. And the PS/2 mouse was very sticky and was almost unusable. So Dare and I zoomed backed to the clinic to find Dwayne still clunking away at the Wikipedia server – the sun was sinking low by now and so were our spirits – we decided to pack all our stuff into the Imperial's definition of a car with a large boot (a hatchback Getz) and go past Agnes, finish off the PC setup and head back to Pretoria. I grabbed a spare serial mouse and new network card and we made our way to the 4x4 track.
We got to Agnes' house at about 4:50 put the network card in and it still didn;t see the network. In desperation we decided to reinstall ubuntu, 40 minutes and throughts of angry wives later we had ubuntu running and it could see the network, but now for some reason the clinic node was down – probably the power supply getting bumbed out of the socked – well it was a “2 pronged plug from the dark side of the Universe power supply so it's not suprising”. And the serial mouse didn't work, and Agnes' login was incorreclty spelt Agnus and we were exhausted. We tried to show Agnes how to use the computer but the sticky mouse just made us angry and we decided to call it a day.
June 08, 2006 Imperialist Imperial car rental
David gets the Imperial car with a large boot at 4:40PM – 1 hour late. This large boot car is none other than a Hyundai Getz ... “Step away from the car mam – LAPD” type
Dwayne tried to copy the wikipedia files from his PC to the wikipedia server at home – but the PC ran out of hard drive space. After buying a new hard drive and starting the copy process on Thursday afternoon – it was taking too long and we had to take both machines with hoping to let it run over night at the clinic to be ready the next morning. We arrived at the clinic at about 10:00 PM and set up the machines at the accomodation block of the clinic and started the copy process again. David tried to fix Dwayne's weird linksys box but gave up and went to bed.
May 26, 2006 Wikipedia and scrapy old computers
Began work on setup of Wikipedia and setup of computer for Agnus. Very frustrating, ended up swapping drive, finding DVD ROMS not working, too little memory to unpack Wikipedia, etc. Lost most of the morning through this. We had decided that we really didn't want to visit people if we couldn't demo anything so tried to stick it out. Eventually gave up.
Skipped the school and Kok farm as we didn't have anything to show. Went straight to the Hospice. Got antennae aligned and box plugged in. No joy. Eventually took box down. This one was definitely toast. Took a long time to try get it down and diagnosed.
Left it at that for the day.
Returned to the clinic. Spoke briefly to Harry about connecting one of the staff who lives near the hospice. Seems close enough but probably no line-of-site. Also spoke about putting computer in the waiting room.
May 25, 2006 Meshy time again in Mpumalanga
Setup a temporary node on the clinic roof as the fibre has not yet been complete (fibre has been run but not yet terminated). Talked to Harry about the fibre and discussed when he expected it complete. Their IT guy is good but they don't bank it being complete on time.
Checked on the node on the clinic flats. Unplugged from our previous visit. Equipment functioning perfectly. We had to do a bit of thinking here as this node is terminated as a gateway node while the temporary one on the clinic is a normal node. Decided that our objective was to check the network so Internet connectivity was not a priority for this visit. Talked to Avery about her need for Internet – she spoke about the difficulty keeping contact with her children and grandchildren in Australia. The Internet is a lifeline for her to get pictures of family to not feel so far removed from them.
USAID node up! This one has been up for over a year, in the worst lightning area with no ill effects!
Visited the school, the teacher was not present but we made plans to see him the following day. Gained access to the school labs. They have installed wireless on each of the machines with what looks like a wireless hub. Our guess is that this is an intervention for lightning, but need to confirm that with the teacher. The link was disconnected but otherwise running perfectly. Outside people where repairing the satellite phones, we assume they were Telkom, unsure if this relates to the lab.
Visited the hospice. Antennae was pointing in the wrong direction. Our plug point has been coopted for something else. Searched for plug point. Spoke briefly to sister Shabalala.
Stopped as Agnus' house. Rains have made the road terrible. Took us about 15 minutes to travel 200m. The cantenna is pointing in the wrong direction and badly rusted. Checked on the equipment. Was half unplugged ie it had fallen from the socket but had essentially been left plugged in. It seemed like it did interfere with the booster antenna for the TV set.
Returned to clinic and met the fibre technicians. Spoke to Ray, the IT guy on the phone. Located the Wikipedia server – Ray had it. Spoke about what we were doing. Ray is still interested in supporting the network. Was very interested in what we had found out about static wikipedia as he supports 2 schools and would like to deploy it in those schools. He also gave some hints as to products he has used for managing bandwidth. Was very interested in what David wanted to do with BW management.
Decided to stop visits for the day and to test the network. KOK_FARM is up! We didn't manage to get to it today, but this means that they have not unplugged and that the equipment has survived. AGNUS is also up! A rusty cantenna pointed the wrong way works! We lost these a little while later as someone must have turned off the school.
Went into town and bought equipment so that we could ensure the school doesn't get switched off.
September 22, 2005, WiFi in the bush or the plot
Duane visited the IDRC funded workshop for WiFi in Africa. Sorry don't know what the real project name is. Sat in on the session on regulation, the Department of Communications delegates decided that this would be a good time to leave. Nice when you know you can't actually stand the heat.
Had a look at their cantenna based access-point liming over about 5km. With a Linksys strapped to a tree with... red tape... man we can't get away from red tape in this industry.
Most surprising was plot.net a local WISP that delivers Internet to about 100 customers and nobody new that they existed before booking the venue. They do long haul links to ADSL connections in Pretoria. Quite an innovative solution to high cost access. I want to explore that possibility in Peebles Valley.
October 06, 2005 Wikipedia on the mesh
David and I visited the Peebles Mesh this week for two days and here is a quick synopsis of events:
- We installed a firewall. This has software which should allow us to control and cap bandwidth and it separates us from the clinic. - We installed a local copy of Wikipedia. Still some bugs but very fast - We can now access the site remotely
Big SURPRISE
Our cantennae was still operating! We did a data transfer from the wireless box to ourselves. 3 hops I think at 192kB/s
First farmer online
We connected the chairman of the local farmers group to the Internet. It was lovely we just watched and took notes. The farmers younger brother had returned from the UK with a new laptop. We got that connected. The first thing he asked is if he could download music - ho hum :)
The sister of the farmer was also visiting and started giving her mother a sales pitch about how they could talk to each other using skype for free.
Was lovely to watch and we had boere kos but arrived a bit late for lunch. We will ensure that doesn't happen again.
July 06, 2005, Realisation of a mesh vision
Today we had a film crew filming the installation of a mesh node. It was great fun but they did waste our time a bit. Technically this is what we got done:
* Installed the node on Agnus' house permanently. Lassy did all this work
* Put a 3m pole at the Hospice to see if we can get over the ridge
* Upgraded AGNUS, HOSPICE and SIKILE to OpenWrt
* The clinic is cabled to various buildings so we will be able to recover
one node and just run a cable from the FLATS node to the VSAT.
We had a 7ms delay between AGNUS and ACTS_VSAT which is a route that passes through all present nodes. VoIP is now truely achievable. Wonderful performance compared to WDS mode. Throw out WDS wherever you see it its just not worth the effort. It is honestly more simple to setup OLSR. Glad we've learnt that and can move on.
We still have the following technical hurdles to overcome. The default route keeps disappearing. We need to install a proper firewall. Once that is done we can move onto the community adoption.
The day was full of some lovely experiences of the potential of what we are doing.
Once we had AGNUS up we setup Dave's laptop and had Agnus' daughter and a neighbour sit at the computer. Dave opened Google. They didn't know what to do. We asked them what information they wanted to know. Stuck again. "Who is your favourite singer?" - Alicia Keys - "OK type that in the search box, then click on search" - Shouts of glee as the search results return. "Now click here". Off they went to Alecia Keys home site and disappeared for about 10 minutes. Does that count as training? The two of them were sitting inside quite a dark house, they sat together on a sofa and the laptop screen glowed in the semi-dark with their face highlighted. This exact image was the vision I had for this project. So that made my day.
I did a quick training course in the dirt. Explaining what the different aerials did, how it all fitted together. And of course how to use a high-gain antenna as a fish braai. Lassy provided Swati translation as soon as there was any confusion.
Now off to the school to film students working on computers. They wanted to interview the teacher, who was a standing in for Mr Sihlabela who we usualy work with. "What will you use the Internet for at your school", to which he replied, "I don't know". David did a quick exercise of taking the teacher, a history teacher, to Google. They Googled for "Shaka Zulu" and ended up at a very good site on the subject. "What will you use the Internet for at your school", and we got a 5 minute reply. Does that count as training?
What worked best about this trip was having a very focused objective - get the mesh network working. We didn't care about the firewall, the billing, shapping, etc. And we got it done. The next trips will be similarly technically focussed.
July 05,2005 Moving on to OLSR
I left Peebles Valley at the end of May and this is the first return since then. David and I are migrating all the Linksys boxes from Sveasoft to OpenWrt, specifically the builds produced by Freifunk which has a nice web interface. Lassy has joined us again and is helping us move the equipment around.
The task involves loading the Freifunk firmware onto the box. This requires local connectivity. We still had connection problem until we discovered the ghosts of a WDS past. Alot of WDS settings were still on each of the boxes. We've now had to go and remove those especially the Lazy WDS mode settings which keep establishing bridges. OLSR looks like the right solution. Its the same amount of work as WDS setup but you get real routes, ones that you see and can traceroute through so you can actually see exactly how packets are being routed on the network. I'm much much happier. Yesterday we got USAID, ACTS_VSAT and ACTS_FLATS converted. Today we are in the field with a film crew and will migrate: AGNUS house, SIKILE_HIGH and FARM_COBUS. If we can get traffic from either Agnus' house or Cobus' farm then we know that the protocol is good, both these points would have to route through. SIKILE, USAID, ACTS_FLATS then ACTS_VSAT. WDS could never do that and if we can get this working then the mesh is good and we can move onto the next challenging problem.
At the last conservancy meeting at which ACTS clinic was present they mentioned the Mesh but all the farmers were wondering when exactly anything would happen. Hopefully we have a working system before we leave.
We also spoke to Frans Benz whose property is below USAID and he is still interested.
Of the nodes that we have observed all were powered up except ACTS_VSAT this was good as it means that people are leaving the equipment to do its job. THe VSAT was down because the plug is just too close to a useful double adapter.
We have brought 300m of CAT5 cable which the clinic will use to run cable from the VSat to the Flat so we can eliminate one node and use it elsewhere. This clinic has arranged conduit and pulling of the cables.
February 19, 2005, Lessons learned from week long meshing
Lessons learnt
* Test test test: test devices in a known controlled environment. The box
couldn't connect and we weren't sure exactly why.
* Use your own stuff: don't use other peoples stuff. We tried to use the
alarm companies mast, it was rubbish and about to fall down. But it could
also endanger our equipment when the alarm company removes there antennae.
* Equipment: cover for your laptop so you can see the screen. Backpack/fishing
jacket also a head- mounted display wouldn't hurt (to monitor the wireless
network). Rifle-mounted antennae with sight. Tri-pod for temporary mounting
of equipment. Ladders are a problem you need a long extension ladder about 6m.
* Windows Sucks: we had major packet drop yet on Linux the network pings work
perfectly. Either Windows has a simple and incompetant network layer or it
puts so much rubbish on the network that it sucks up all bandwidth. Why do
people use this rubbish!
* Testing protocol: It would be usefull to have a set standard of how you tackle
problem. Many time you go down the wrong route. Simple idea: mount antennae
with laptop and get signal strengths right. Then connect box browse to the
box over the wire and check for links. Lastly connect to the box as a wireless
client.
Observations on the first stage
The object of this exercise was for David and I to learn how to setup the equipment before involving the community. This objective has been met: the mesh functions well, we can monitor it and we can use it from any node. Successes
We created a 4 node mesh that works and can connect to the Internet. Although we had issues these were just teething problem for the technicians and the hardware is really box-drop ready. This is really positive for a non-technical mesh community.
We met with the farmers and they now know what we are doing which give us easier access to their properties. The USAid couple were very helpful, very keen and are well known and liked in the community. We hope that this relationship and them being early users will help increase adoption within the farming community.
We validated that we can connect to the school through the USAid node. Although our contact at the school was on sick leave the principle made arrangements for another teacher to stay after school on Friday for almost 4 hours while we attempted to setup their node. We also noted the school now has a 15 seat computer lab.
The ACTS clinic staff were quite excited especially those that are volunteers from overseas. A number of ideas for the use of the network were put forward by the staff. These included using house bound AIDS patients to data capture for the clinic or simply to create new virtual relationships. Also suggested was using instant messaging as a form of anonymous counciling to allow school pupils to speak openly about AIDS. Issues noted
Support! We have already got our first support call from people who use operating system that know better then you what they want to do. This is a source of potential problems as people will phone the clinic and overbuden an already busy staff. We need to investigate how the community can provide that support or busget to pay for such support.
Lightning! This is going to be a serious problem, there are solutions such as mast spikes that we need to install.
Power downs. The clinic powers off most of their equipment at night thus taking the gateway down. We need to look at way to keep this up. The conection also require human intervention and we need to investigate automation of this taks.
Network separation. The mesh is currently on the clinic network. The clinic does deal with sensitive data but also for security and management issues we need to seperate the neworks from each other.
Usage monitoring, charging. We need to begin monitoring and controlling usage so that we can explore charging models in the future.
Windows... it puts junk on the network. We need to look at simple firewall rules on the nodes to drop NetBEUI junk from the network. Next steps
We will return to the site in a months time. At this time we wish to: separate the networks, install the school node, install a monitoring and access control system and announce a training course for the following month.
We will separate the networks by installing a new network card on the firewall. Most probably we will use NoCatAuth to control access to the network but need to investigate alternatives.
Other things to investigate are models for charging for network usage. Currently our strategy is to have farmers cross-subsidise the schools connectivity. A problem with this however is that other forms of connectivity are becoming relatively cheap (3G, etc) so this still needs to be reasonable.
February 19, 2005, Too much sun for a geek
Day 4: Had steak at the conservancy meeting. Talked a little about the mesh - nobody understood what I was tgalking about... But getting cheap Internet sounded cool to them. The farmers have serious problem with Telk^h^h^h^h connectivity, phones, lightning, macadamia thieves, water, etc.
Day 5: Dave J works for a big parastatal like company which means he always comes late cause someone wants him to do something (he tells me he was fighting Jhb traffic to get more Linksys boxes. Mainly spent the morning getting ntop (a great network monitoring tool working) and got EtherApe (nice graphical monitoring tool) working. These are quite good for seeing how badly Windows abuses network resources.
Spent the afternoon with David at Sikhile High. Was pretty fruitless we couldn't work out if it was the box, antennae, or sun that was messing us around.
Day 6: Installed the USAid node properly. Took a long time and got really sunburned. Note to self need a ladder, suntan lotion, umbrella and LOTS of water. Installed a 90deg patch panel. We had major problems getting links to the clinic which was frustrating.
February 17, 2005, More meshing
Day 3 cont...: Got USAid online, had been switched off not blown up by lightning. Phew. Installed a second mesh point which proved not to be needed. Worked out how to add parameters to the Kismet display, only worked out later that the signal strength bar graph is completely arbitrary. Will need to check all that again today. Installed an AP on the accomodation block, this seems a waste and we will probnably run cable from an AP installed on the staff flats instead.
Day 4: David J should be joining me today and we will be meeting with the farmers in the conservancy for a braai. I hope that we can raise the issue of what we are building and get people excited. Just before leaving yesterday I got everything up: I can ping all the other APs and can get out onto the net. I'll be checking this again remotely to see if I can get from USAid, to the clinic and out.
February 16, 2005, Meshing
I've had a busy few days here in Peables Valley:
Day 1: Setup a linksys on the roof of the clinic. Found that we really can't see Petra from here or anywhere actually, Petra is a nice high point but all buildings are hidden behind a large rock. We installed one at a house belonging to consultants for USAid. This is a good site as we can see the staff flats from there but none of the other buildings. The problem with this is we now need to find a good spot at the clinic so that we can see USAid without using two APs at the clinic.
I visited the school, unfortunately Mr Sihlabela is off ill until March. I met with the principal and two other teachers and have agreed to visit on Friday with David. The school can see USAid but we still need to confirm the signal strength.
In discussions with various people I realised that lightning is going to be a serious problem. With all the equipment on an AP costing about R2,000 it can cripple a network if they have a serious strike.
Day 2: A frustrating day trying to get Kismet working. This involved creating a patched kernel to allow the orinoco driver to go into Monitor mode. Finding, downloading and compiling takes a long time. I did manage to get onto the clinic roof and confirm that we couldn't see a signal from USAid. We did get a signal from the staff flats.
In discussing uses for the network we had some new ideas:
* Getting house ridden patients a computer to get them communicating and getting
outside virtually.
* Having the same patient do data capturing to bolster his income
* We also talked about using SMS and cellphones for compliance. Setting alarms for
medicine taking. SMS messages that require a reply to check compliance. Notifying
support group buddies about people not replying to SMS notification so that they
can check that they're OK.
Day 3: Kismet is working! Not all the information is that useful for what I need to do. Met the clinic staff today to try and explain what we're doing. Today I'm checking signal strength from the school, its seems however that USAid is offline.
February 11, 2005, Equipment Procured
We now have the equipment for the FMFI Wireless project in Mpumalanga. We have 10 Linksys wireless AP which we're firmware upgrading. The cards are in a nice heavy duty all weather box. A picture would be great :). All the boxes are being moded to do power over ethernet which makes hte whole installation clean and simple. Currently it looks like costs are around R2,000 a node.
I'm lugging the lot down to White River this weekend and will erect 3 of the nodes. Mainly to test equipment and get a working initial network. The other nodes will be setup by the community, following the Onno Purbo style of network deployment.