Thali, San Francisco and Emergency Response/Recovery

I went to SF to meet with folks from the city of SF to discuss how we could use Thali to help improve recovery after a major earthquake. The meeting was both productive and eye opening. Our next steps are to put together a proof of concept to hopefully use in a live trial later this year. Below is an explanation of the who, the what, the why and most interesting as a geek, the how. Technologies and emergencies don’t necessarily mix so we have to think hard about how to make things better, not worse.
On 6/27 I attended a meeting with representatives from the
  • City of San Francisco’s Neighborhood Empowerment Network,
  • SF’s Neighborhood Emergency Response Team (NERT),
  • Human Services agency of SF,
  • Red Cross and
  • Microsoft civic innovation team led in SF by Scott Mauvais.
We were there primarily to get briefed by Daniel Homsey from the Neighborhood Empowerment Network to discuss the Empowered Communities Program’s (ECP) Neighborhood HUB Program. I was there to discuss how Thali might help. Below I give a taste of the challenges SF faces, their plans for responding and the small role that Thali can play in helping to make things work better.

Dealing with the “big one”

When the next “big one” hits SF there are going to be enormous challenges. Most of the emergency response personnel (fire, police, medical, water, power, etc.) don’t live in SF. So when the big one hits many of those folks are probably not going to be in the city and are going to be focused on dealing with the disaster where they live. If the bay bridge in particular is damaged then many of those folks, who I believe tend to live in east bay, would have to come all the way down and around the bay to get to SF. In general that is a long drive but after an earthquake it may or may not be reasonably doable given the likely devastation of the road network.
So the bottom line is that the city is going to be largely on its own, potentially for quite some time, after a disaster. To make matters worse large parts of the city are subject to liquefaction (think: buildings sinking into the ground) and even in areas where this isn’t an issue and the buildings have been reinforced against seismic events, the streets haven’t. So it’s quite likely that in many areas the buildings will be standing but the streets won’t. But wait, it gets worse! SF has a very transient population. The average person apparently only calls SF home for about 5 years before moving on. This means that SF doesn’t have a lot of deep community ties. So after a disaster there really aren’t existing family and neighbor relationships to fall back on. And to complicate things further SF has a reasonably large elderly and medically challenged community that have been empowered by the city to remain in their homes. But they depend on a regular supply of drugs and medical equipment to remain healthy. When the big one hits, who is going to take care of them?
None of this is news to the city of SF and it is putting in place infrastructure to try and respond. One of those program is the HUB program.

ECP’s Neighborhood HUB Program

The HUB program breaks the city of SF into different areas and for each area identifies an “anchor institution” (think churches, community centers, etc.) which are seismically sound and socially connected to their surrounding community. The people in the anchor institution receive training and emergency supplies. Around the anchor institution are numerous hub member organizations These are doctor’s offices, grocery stores, etc. They affiliate with their local anchor institution and receive basic training. Finally there are the block champions. These are people who live in various neighborhoods who have also received some basic training as well as supplies. Groups of block champions are then affiliated with the anchor institution that covers their area of the city.
Once an emergency occurs then the anchor institutions turn themselves into Neighborhood Emergency Operation Centers (NEOCs) which have primary responsibility for collecting information about what is going on in their geographic area and routing supplies, people, etc. The hub member organizations turn into Service Providers (SPs) whom the NEOC can call upon to provide specific services ranging from food provision to medical care. Finally the block champions will form Care & Coordination Centers (CCC). This can literally be someone opening their garage or setting up in a designated park to provide a point of contact for the neighborhood. The CCCs have the most immediate responsibility to canvas their neighborhood and figure out which buildings are still standing, who is hurt, who needs food, who needs shelter, etc. The CCCs have their own supplies (tarps, basic medical supplies, etc.) but if the problem is bigger than they can handle then they need to kick the issue up to the NEOC who can use its supplies, facilities from the SPs as well as whatever emergency services (Fire, medical, etc.) that are still functional in the city to try and respond.

The Communication Problem

When an emergency hits the CCCs will kick into action. The volunteers will find other people in the neighborhood and start to walk their designated part of the city. They will figure out what is going on. This information is currently placed on paper which would then be collected in the local CCC to figure out what’s going on. Depending on the outcome more paper may be generated that would be forwarded to the NEOC. The NEOC would then generate more paper to send out requests to SPs or even help from other CCCs. Overlaying all of this is a voice based radio network to communicate with whatever emergency responders (Fire, Police, Medical, etc.) are in the city after the earthquake.
The good news about paper is that it’s incredibly resilient. When all else fails paper has a high probability of surviving.
The bad news about paper is that it’s slow to move, hard to handle and doesn’t update itself. It makes sharing information very challenging. For example, a CCC survey form has data that is likely to be interesting to Fire, Medical, Police, long term care, etc. But there is just one piece of paper. And moving and processing that paper in an emergency is tough.
Ideally we would just set up a cloud service to let CCCs submit data electronically and then route it as needed. But after an emergency we can’t rely on having any kind of communication infrastructure. Cell, twisted pair, coax and fiber are very likely to be down.
This is where Thali can come in.
Thali proposes using the phones in people’s pockets to let them collect and distribute information in an emergency. The city has also discussed putting phones into emergency supplies and they apparently are already putting in solar chargers. Using Thali’s peer to peer sync capabilities it becomes possible for CCCs to collect data onto phones and to use the local radios on those phones (BLE, Bluetooth, WiFi, etc.) to sync the data directly with other phones.
Using Thali it’s possible for all the data from all the volunteers in a single CCC to be put together on one device and get a global view of their immediate area. Similarly all of this data can then be sync’d up to the local NEOC who can now get a global view of its own area. Furthermore the information can be filtered and queried so it becomes possible for different experts, fire, medical, police, etc. to find the specific data they need while still having a unified data collection experience at the local level. And unlike paper, this is data that can update itself. So as new information comes in we can update the old information.
Using Thali also makes it much easier for the NEOCs to communicate with each other. Instead of handing over mountains of paper, a single device can be run over and both send and receive whatever data is available.
And best of all since Thali uses a completely dynamic ad-hoc synchronization model nothing has to be set up in advance. So the network and data can grow/shrink as the emergency progresses. And of course Thali is happy to work with the cloud so if the communication infrastructure does start working Thali can sync data up to the cloud and even relay data from parts of the city that don’t have access to those who do.
But of course, the devil is in the details.

Paper Matters - the role of OCR/OMR

It’s tempting to switch to a completely electronic data collection and dissemination system based on Thali. But that is probably a bad idea. It’s easy for phones to break in any number of awful ways as well as to run out of power. But the thing about an emergency is that it comes with problems but not IT support. So to the extent possible we really want to make sure that we can keep information flowing no matter what. So the current thinking is that our basic data collection mechanism would still be paper based. We are (as will be discussed below) exploring other ways of collecting data but we always want to be able to fall back to paper. It has a robustness that is very hard to match.
So the thinking is that we would start with paper based forms and then make it easy to put those forms into the phones using some form of Optical Character Recognition (OCR) and Optical Mark Recognition (OMR). This would allow us to maintain the paper based forms but quickly move them into the electronic infrastructure using phone cameras to scan them.

30 Feet is nice but we can do much better - the role of White Space and maybe 900 Mhz WiFi

The radios built into phones are typically good up to around 30 feet. That is a very practical distance for local activities but what happens when a NEOC wants to talk to a SP or when NEOCs want to talk to each other? If all we have are local radios then we will need runners who can move phone (or paper) between places. But we can easily do better. 802.11af, also known as White Space WiFi, lets one deploy WiFi access points using UHF and VHF frequencies that even in cities can easily cover a mile or more. And if we can get a FCC waiver for emergencies (assuming one doesn’t already exist) then we can extend that range to 10 or 11 miles. And SF, the hilly city, is a perfect place for White Space WiFi as we can stick an access point and antenna on the top of a hill and get even better coverage.
I would also like to see if we can do anything with 802.11ah which runs on 900 Mhz. 900 Mhz can run up to a mile and uses a smaller antenna and battery draw. We already see devices using 900 MHz that provide extended range data services to phones. Using something like that we could deploy the 900 Mhz devices with the phones in emergency stores and get longer distances for less power and infrastructure. It just adds another layer of resiliency.

A better way to collect information - Guided Forms

While paper should always be the base line we nevertheless want to make it easier to get better information, especially from the CCCs. The problem is that the more information we want the longer the forms get and the harder they are to fill out and process. An obvious way around all of this is to have electronic forms that use guided questions. In other words when someone walks up to a building the first thing the eform will ask is “Is the building on fire” or “Has the building collapsed” or whatever the high priority questions are. Depending on the answers different questions will start to come up. The point is to pack a lot of questions into a form but only require the volunteer to provide answers for ones that are relevant. So, for example, there may be a lot of questions about people’s health but those questions will only show up if prompted. In other words, if the form asks “are there babies in the building?” then further questions on supplies of food, diapers, formula, etc. will only come up if the first answer is yes.
The guided forms issue is actually more important than it might initially appear. The forms currently used by the HUB program are fairly small because they really only focus on issues relevant to the Fire Department whose focus is on critical triage. If it isn’t burning, falling or dying then it isn’t a Fire Department issue. But in an emergency like a major earthquake we have to focus on longer term issues. Do people have their medicines? If someone needs oxygen to breath do they have power and oxygen canisters? If an elderly person is an apartment by themselves, is there debris that is going to make them fall? If people have a days worth of food, how do we get them more? To handle these issues data has to be collected, triaged and acted on. All of which results in longer forms.
So if we are going to get the data we want without overloading volunteers we are almost certainly going to end up in a two tier world. There will be a stripped down set of basic questions on the paper forms. Those will be the baseline. But for those who have the phones they will have guided forms that will allow them to collect richer sets of information and so provide better emergency response and recovery services.

Accepting help - Side loading to volunteers

If Thali is adopted as part of HUB then my understanding is that the city would probably put phones into the various emergency supply stashes. Those phones would then be pre-provisioned with the Thali based app. But in an emergency all help is welcome so ideally we would enable volunteers who walk up with their own phones to get a copy of the app and start helping out. Now in a perfect world SF would have some kind of city app that helps its residents with normal day to day business with the city and in the back of that app would be the emergency app. But we can’t rely on that happening. So instead we want a way for people who already have the app to give it to folks who don’t.
In the case of Android this is pretty reasonably doable. The person who wants to get the app will need to change a security setting to allow them to side load. Then the person with the app would need to make their phone into a hotspot (this is actually something we can fully automate in the Thali app itself) and then the person who wants the app would need to connect, download the APK and then install it. Most of this can be automated and on screen instructions can handle the rest. This means we can deploy the apps to new devices in the field.
iOS, as far as I understand, is a much harder story. There is a way to side load on iOS but as far as I know it requires having a mac, a cable to the iPhone, a developer account and xCode. This all seems a bit much to me so I’m guessing we won’t worry about side loading to iOS devices.

Deputizing People - Security and privacy

During the meeting one of the issues that was brought up was the ability to share information. This is tricky because of privacy concerns, even after a disaster. The Red Cross, for example, was very worried about this issue. They work hard to earn the trust of the people they deal with and don’t want private medical information able to go where it shouldn’t. In general Thali can deal with these issues. Thali provides full authentication and authorization support. But the problem is that in the middle of an emergency we need to be able to deputize people to help, people who aren’t in the system, people who aren’t authorized.
There are two ways to potentially deal with this. One way, which Thali isn’t working on quite yet, is to allow for insecure relay. The idea is that person A can sync information to person B’s phone and then person B can sync with person C. The trick is that this can be done in a way that person B can’t access the data. This isn’t very difficult to do but it’s fragile and uses a lot of bandwidth because all the data has to be moved even if the receiver already has most of it.
One suspects we’ll need to take a different approach. Specifically, we need to allow authorized people to deputize other people. This means not just side loading the app but generating an identity and giving some level of trust to that identity. Thali already can generate the public keys and already has support for identity exchange to allow two people to securely exchange identities. What’s needed then is a way to give access rights. The idea being that those rights would only apply during the emergency and would be withdrawn when the emergency is over.
This is all doable. We have the math. But it’s going to take a bit of work to figure out all the requirements and implement them correctly.

Bigger screens and more power - the role of the laptop

My understanding is that plans have either been executed or soon will be to put laptops into the emergency supplies as well. Especially at the NEOC level there is just too much data to stick to paper unless one has no choice. I also suspect that visualization is a big part of the motivation. Phone screens are still pretty small. Although I do wonder if SF might not be better off just putting in a pico projector and using phones. But let’s assume they need those laptops.
Right now Thali focuses on mobile phones but there is no voodoo in Thali. At its heart Thali is node and that runs as well on a desktop as it down on a phone. In terms of talking with the phones, setting up an ad-hoc wifi network is easy with laptops. Just setup a hotspot and the phones can connect and sync both to the laptop and to each other (Thali supports local WiFi already). The bigger challenge will be packaging up the app to run on desktops. Right now we use Cordova and it doesn’t really have a desktop story outside of the Universal Windows Platform. But I doubt it would take much to translate a simple (e.g. no fancy plugins like, you know, Thali =) Cordova app and turn it into an a simple Node.js app that is interacted with via the browser. Ideally we would use something like Electron but we need PSK support and it isn’t in mainline node.js yet, although a PR has been submitted. But JXcore (which we currently use for Node.js on Android and iO) runs just fine on desktop too.

Pre-provisioning data - the always up to date directory

One area that didn’t get discussed in the meeting was how NEOCs know who the SPs know who the CCCs are. Presumably there is a directory somewhere that gets sent around. But this is the sort of thing that is going to get out of date fast. And ideally this information could also be updated during the disaster (e.g. SPs might be destroyed, CCC leaders might be absent, etc.). To make that possible we need volunteers to have the Thali app on their phones and to get updates (yes, Thali can do that of course) from the cloud on changes. That way when the earth quake hits a bunch of the data will already be pushed out and the information can flow via local radios.

It actually is all about the cloud - the years of recovery

It’s tempting to just think about say the scope of a NEOC and a bunch of phones and think we have things covered. But while certain organizations, like the Red Cross or even the Fire Department, have a fairly short window of responsibility, the city has to deal with the consequences of a major disaster for many years to come. Residents needs new housing, medical care, etc. These processes will take many years. Which means all the data collected during the disaster needs to live on and feed in to the city’s health and human services infrastructure so it can be sure all of its residents are properly helped. This means the data needs to get off the phones and laptops and into multiple different central stores that the city uses to run its daily business. So we can’t afford to forget about the cloud. We have to always be thinking how the data is going to be cleaned up and moved into the cloud.

Microsoft and October

The place where it seems like we are going to start is with a simple neighborhood level live drill in SF, probably somewhere around October (ish). The scenario is:
Step 1 - NERT Data Collection - Volunteers from NERT in a particular neighborhood (or two) will take Thali enabled disaster apps into their actual neighborhood and go to real locations with posted papers describing the contents of the building. The volunteers will ideally record that data onto paper forms that will then be read into the phones using cameras and OCR/OMR.
Step 2 - CCC - The volunteers will then go to a simulated CCC where they will sync the data on their phones with a laptop over WiFi.
Step 3 - Health & Human Services/Red Cross - At this point folks from either SF Health & Human Services and/or the Red Cross will join in. They will use the data from the NERT folks to send out their own folks in the field to collect more detailed data relevant to them (e.g. the NERT folks might identify that somebody needs medical care but the Red Cross folks would go visit the person to get the details). These folks would then go do their own data collection exercise like step 1 followed by their own sync as in step 2.
Obviously this barely touches on what Thali can do. But that’s o.k. This really and truly isn’t about Thali. We are just a small sprocket in a much larger and more important machine. This is really about showing that the computerized data entry and processing approach can work. Even more importantly, it’s meant to show that we can share data across different groups (in this case NERT and HHS/Red Cross) in order to build a clearer picture of what is going on.
There also was a lot of interest in side loading so we will probably demo that but we’ll handle it all manually just to show what can be done. We can create the automated version later.
I’m actually hoping to also bring along Ranveer Chandra from Microsoft Research. He is an expert on White Space Wi-Fi and I would love for him to do a demo of what White Space Wi-Fi can do during the drill.
Of course someone has to write the demo Thali app that will be used to make this all happen. I have zero cycles to work on the actual app, but that is where Dan’l Lewin’s civic engagement group at Microsoft comes in. The SF team, led by Scott Mauvais, is on the hook to bring in the resources to get the demo app coded up.


Sitting through the meeting was scary. I was familiar at a high level with the challenges SF faces (and of course the whole time I’m thinking of Seattle which faces much worse challenges as our earthquake threat is higher and we have to deal with Volcanoes) but seeing the details was disquieting. Being able to play even a small part in helping people to recover from this threat is certainly high on the Tikun Olam meter. So my hope is that we’ll be able to show what Thali can do in the October (ish) trial and then figure out how to go from a proof of concept to a useful app.

Leave a Reply

Your email address will not be published. Required fields are marked *