Thread by @benskuhn

Happy @threadapalooza! 100 (tweet-sized chunks of) stories from trying to build mobile money in Ethiopia, Ghana, Nigeria, Senegal and Cote d’Ivoire.

(Minus the ones I’m not supposed to talk about :P)

It all starts in ~Sep 2015 with me pretending to be an accountant.

“Huh?” It turns out doing the accounting for an international money transfer business is hard. There is a normal way of doing this, but instead of trying to find out what that was, we were like, “this seems like it should be 100% automated! We’ll just do that.”

LOL

(This was part of our general take that hiring was for chumps, instead we’d scale ourselves by building internal tools)

So that’s how I ended up closing Wave’s 2015 books by spelunking through a homegrown database for 3mo with handwritten SQL and a half-baked Flask-Admin UI.

That wasn’t the worst part—that was spending ~50% of waking hours in Jan / Feb on the phone, explaining our poor life choices to our auditors

We ultimately passed the audit fine! Although IIRC we did get a footnote being like “you guys are idiots, do it the normal way next year”

Anyway, for reasons that I’m sure were completely unrelated, around the same time I broke up with my then-girlfriend, who happened to be engineer #1 (I was #2)

So I did what any reasonable person would and moved 10 timezones away, to work on our mobile money pilot in Ethiopia

IIRC the actual reasoning was that if I lived in Ethiopia, I’d have enough comparative advantage at product work that it couldn’t possibly make sense for me to work on accounting in 2016

(LOL)

Anyway, at that point we were mostly doing international money transfer (US -> Kenya)

Mobile money was a tiny pilot project the founders were running because they realized that international money transfer was limited by market size, because Kenya was the only country with a really good mobile money system.

OK, challenge accepted—we’ll just build the mobile money system ourselves so that international transfers will work better!

(LOL, “just”)

Anyway, I moved into the house the founders were renting in Addis Ababa and started working on that.

[ok, meetings starting, more later!]

Tbh I don’t remember much about my first trip to Addis. Our product ROYALLY sucked because:
• it was an Android app (penetration was low)
• built assuming good cell network (Ethiopia had ~the worst mobile data in the world)
So we mostly just flailed around trying to get users.

Mostly I remember the founders deciding they needed me to manage the eng team so they could focus on finding product market fit with mobile money.

Timezones made that hard (also, my incompetence at management) so I moved back to the US.

By the next time I moved back we’d completely overhauled the product to use USSD, an old-school protocol where you dial a bunch of digits and symbols, and get a pop-up message in response. This was awesome because it didn’t need mobile data, or a smartphone:

It was non-awesome in that it required a server in Addis that was VPN peered with the govt monopoly telecom, which went down ALL. THE. TIME.

We ended up building a “backup data center” in Jijiga, the capital of the Somali region, where we lived bc most of our users were there:

The “data center” is a closet just outside the frame :)

As you’d guess, it went down a lot too. “@.channel USSD is down” became the call of doom.

Our users would angrily call support demanding their $, so we built Slack tools for the support team to enter “manual transactions.”

Obviously we’d get a lot more support calls than normal so we’d effectively stop the entire company and get everyone on support. Here’s a manual transaction party during (+ caused by) a power outage.

(phone flashlight + water bottle = lantern—great power outage lifehack!)

The other fun thing we started doing at this time was letting people buy phone airtime with their Wave balance. This got tons of users bc it was the only way of topping up electronically.

You might think the telco would give us some kind of API for this.

HA HA HA

Actually, it was done by another computer in our “data center” (read: closet)...

...plugged into a bunch of USB cellular modems...

...imitating a person (or 10) sending airtime to their friends with their phone.

“How did you get the airtime onto the USB modems though?”

Glad you asked! There was no way to buy it in bulk electronically—only via scratch cards. So we’d buy huge batches of scratch cards and have a team of people manually scratching them off and topping up our airtime SIMs.

Every couple days, a SIM would trip the telco fraud rules (understandably) and get blocked with $X000 of airtime on it, so iirc it also ended up being a ~full time job navigating telco’s call center phone tree to get them unblocked.

[break for more meetings!]

Jigjiga, where the founders + local team lived, was a city of ~300k in the middle of a desert.

The best part was the people. Somali culture had a strong tradition of money transfer (that’s how we ended up there) so people REALLY loved Wave. Like, “sacrifice a goat” level loved.

Our local team was incredibly smart and dedicated. The image I always remember was our head of support, Osob, dual-wielding phones to do manual transactions faster. Or our “super agent”, Ahmed Sharif, taking his cash-laden motorcycle across the river to settle a faraway agent.

The first time I met Ahmed, he told me our customer support didn’t use enough data and suggested a bunch of (very smart) dashboard improvements.

He and the others we hired were amazing coworkers—which made it sobering to think about how underutilized many were in previous jobs.

This turned out to be a common theme with other local teams—lots of smart, energetic people with crappy jobs because their environment sucked.

There was not a lot of opportunity in Jigjiga. IIRC we guesstimated regional GDP/capita at $600. The regional president was an autocrat.

Outside the cities there was an early curfew because the region had problems with separatist rebels. We were told not to go hiking because there were paramilitary police camps atop many hills.

(Jigjiga proper was very safe—female colleagues said they felt safe alone at night)

Anyway, aside from the people, it wasn’t exactly a fun place. Food was particularly challenging, at least if you wanted vegetables. I remember coworkers ecstatic about getting ahold of some lettuce, then drying it by swinging a pillowcase because they couldn’t buy a salad spinner

Finally I discovered Meskel, the one Amharic restaurant in town, which had UNBELIEVABLY DELICIOUS shiro wot (ground chickpea stew). I’d go there once every few days with some massive tupperware, ask them to fill ‘er up, and reheat for 3 meals a day. Never got tired of it.

Anyway, we were growing really fast in Ethiopia, so we decided to try launching the same thing in Ghana. Exciting! Of course we were too cool for Accra, instead we went to where we thought people had the biggest money transfer problem: Tamale, a desert town in the north.

I decided to visit to help do the local integrations we needed for launch.

Temale sucked in exciting, different ways from Jigjiga. It was a *hot* desert climate and I visited in the hot season: average high ~100f / 37c. It also had THE WORST electrical grid.

Our house had ACs, but the grid usually ran at 170 volts. We got voltage regulators, which dropped like flies. Our household was 3 people:
• CEO, talking to users / running the company
• Me, full time programming in my underwear
• Ops coworker, full time fixing the AC/plumbing

Unfortunately we realized that we weren’t prepared to compete with MTN, Ghana’s biggest telco (they had their own mobile money and could effectively shut us down if we ever got big). So we cancelled our Ghana plans. Shortly after, we also had to stop in Ethiopia (long story).

That really, really sucked—things were going incredibly well and suddenly we had to rebuild from ~zero.

The first place we tried was Nigeria. As in Ghana, we skipped the capital and went for the northern region with more unbanked people—in this case Kano.

[break, more later!]

(Photo by @capnfabs, who I am not tagging in the main thread because I’m worried it’ll notify him every time I update. Sorry if I got the twitter etiquette wrong!)

Kano (through no fault of its own) was definitely the low point of my mobile money adventure

1. We were all depressed from having to leave Ethiopia
2. Getting traction with mobile money was hard

It turned out that banks worked better in Nigeria than in most other places—

the central bank had mandated the implementation of instant inter-bank payments, so people used bank accounts kind of like they’d use mobile money elsewhere.

It still sucked because your closest branch might be hours away, but Wave wasn’t the same 10x improvement as in Ethiopia

But the real reason Kano sucked was the hotel.

We were so pumped about it! It was going to be so much better than Jigjiga! It had an Indian restaurant! With vegetables!

After a few months of 3 meals a day of Indian food and product not working though, it was... soul crushing.

My partner was on summer break staying with her parents in Warsaw, and I ended up bailing out of Nigeria earlier than planned to join her. I think her parents thought I was some kind of criminal at this point based on how I couldn’t predict which country I’d be in in 2 weeks

Anyway, soon after that Wave gave up on Nigeria and started working on Senegal instead. And things worked a lot better!

We ended up being popular with fish traders, since that’s a big industry in Senegal. The only disadvantage was it turns out cash carries scents really well...

and we were storing all the fish cash in a safe in our house. And that was also the cash we used for expenses (since banks sucked). Soon all our cash, and ourselves, smelled like fish.

It was worth it though because we finally had users again!!

[pausing again for a bit!]

OK, I realized I missed some silly things that happened in Ethiopia so I’m going to break chronology for a bit. Let’s see...

One was our antics trying to make our app work on bad Internet.

Even after we switched to USSD for customers, our agents used a smartphone app.

Unfortunately the cell network in Ethiopia was ludicrously bad, largely because the autocratic govt owned the single monopoly telco and they did. not. care.

When we first came to Jigjiga, internet only worked between ~11pm-7am. Too congested otherwise.

So during the day, we had meetings by using Skype to dial the founders’ cells long distance, paying ~$0.30/min.

It got better over time... in Jigjiga. In rural areas our app was near-unusable—think multi-minute loading spinners. Agents ended up with massive queues in their shops

You know what still worked though? SMS!!

Challenge accepted.

Some coworkers built a custom network layer that took our requests, jammed them into an itty-bitty SMS (~100 bytes), encrypted them, and sent them to our local SMS shortcode.

This was so ludicrously janky I still can’t believe it actually worked.

I remember we cut a corner and didn’t implement this protocol for login because it was more complex and “people don’t do it that often, it’ll be fine!”

HA HA HA

The next month I heard a story about a new agent who accidentally hit the logout button and had to drive 2h to the nearest town with mobile data so he could log back in 😭

We’d immediately compensated for the better product by opening agents in places with even worse internet

SMS fallback fixed our worst-case latency, but it was still really slow, because nobody foresaw that some unhinged software engineers would one day use it as a replacement for the internet. We needed more.

Then we hit national i̶n̶t̶e̶r̶n̶e̶t̶ ̶s̶h̶u̶t̶d̶o̶w̶n̶ college exam week

“What?” Yeah so about the govt-owned telco...

• Every year Ethiopia runs a national college entrance exam
• ~Every year the answer key leaks and people start sharing it
• What’s a proportionate and reasonable way to prevent that?

SHUT DOWN THE WHOLE INTERNET!

(every year!)

Not the whole internet though, it seemed like Ethiopian sites still worked. Which meant DNS was still working, which meant... we checked...

🎊 All UDP port 53 traffic was allowed 🎊

Coworkers quickly hacked up another custom UDP transport layer, disguising our traffic as DNS.

(To be clear, this “disguise” was “three dogs stacked up in a trenchcoat” quality. iirc the internet filtering got more effective in future years & this no longer works.)

The new transport was also better because it cut out 2 packet roundtrips by optimizing the cryptography...

Basically, it was an incredibly shitty half-baked homegrown QUIC (en.wikipedia.org/wiki/QUIC).

The shittiest part was that we decided requests would never be over 1kb, and implementing sequencing seemed hard, so we just didn’t do it 😂 Fit your junk into one packet or die trying.

Second shittiest part was retry behavior. Theoretically, UDP packets can be lost, so we should have done something with acknowledgements and retransmissions.

In practice, that sounded hard so instead we just re-sent every request and response four times 😂

We’ve kept using that UDP protocol ~unchanged since then! We’re just now killing it (it got hard to keep responses under 1kb, and data in SN is way better than ET).

[Wow long digression on networking! It’s near bedtime so I’ll have to write up the other fun bits tomorrow.]

[PS we’re far from done doing stuff like this—if you want stories like these of your own, we’re hiring all kinds of engineers + many other roles! DMs open 🙂]

OK right, more stories from Ethiopia. Let’s see...

One fun thing was just getting to Jigjiga. If you bought the fastest ticket, you’d end up with a 1h connection in the Addis airport that your checked bags would 100% likely miss. This was the origin of my checked-luggage trauma.

Another fun part is that the way the terminal was arranged, you didn’t actually *have* to go through passport control to get on the domestic flight. A coworker once did this (by accident) and ended up in Jigjiga without a visa. Some interesting hijinks went into fixing that one.

Then of course there was customs. The most exciting thing to import was our duffel bags of penguin stickers, which we used in ludicrous quantity because people loved them so much. Customs agents kept thinking they were hiding drugs or something (why else would we need so many?)

Some of my best memories were from company retreats where the entire team met up in Addis or Jigjiga. After one retreat, some of us visited Lalibela, a ~12th-century religious site in the amazingly beautiful highlands, with rock-hewn churches carved directly out of the mountain:

Still probably my favorite place I’ve ever been! Especially the nearby Scottish/Ethiopian fusion restaurant (no, really) Ben Abeba, a building that looked like it should be a wizard’s cottage, where you could eat suspended over the mountains:

OK, some random other fun memories from going back through photos:
1. retreat activity: attempting to learn a traditional dance from a Somali colleague
2. wave ad on a taxi (Ghana)
3. wandering around Jigjiga in a penguin suit

[meetings again, more later!]

[I don’t want to number all the tweets but I think this is tweet 57]

OK, so when I left off we’d relaunched in Senegal, and things were going great except we smelled like fish. Which I guess is the one time a penguin mascot has been appropriate for this Africa-focused business.

In 2018, my partner had a semester of grad school with no classes, and I convinced her to work remotely from Senegal with me. We both moved to Dakar.

Dakar is a super nice place to live. (If I spoke French well, I might still be there!) Lots of beaches, temperate all year...

...a French grocery chain (Auchan) that sold fancy cheeses... there was even a climbing wall (!! although it was tiny and they did not reset the holds in the few months I went there.) Definitely the nicest place I’ve lived in Africa.

In Senegal we couldn’t use our USSD-based app for fear of Orange, the biggest telco, messing with us. Instead, inspired by the breakout success of Wave stickers in Ethiopia, we built a thing where there were Wave stickers with NFC chips that you stuck on your phone.

Except of course, getting the custom-printed NFC stickers was a nightmare, so in classic Wave fashion, we hacked around it by buying a bunch of normal NFC stickers, non-NFC Wave stickers, and setting up a room of people sticking one to the other. Ugly as sin, but it worked!

Eventually we gave up on NFC because the stickers on people’s phones got abraded too quickly, and the readers we gave to agents were dropping like flies. Instead we switched to using cards with QR codes printed on them.

We didn’t do this originally because QR codes can be copied—that meant we needed to send the customer an SMS token to verify it was them. But, it would allow us to make a smartphone app that worked the same way (app generates a QR code, agent scans), for simpler UX overall.

We put the app on hold though, because we were excited about something else: using Wave for in-person “merchant” payments.

We decided to start in Mbour, a smaller city where we thought it would be easier to build a network effect. So we moved most of the company there.

For some reason we had absurdly bad luck with housing in Mbour.
• House 1: somehow had ~0 windows
• House 2: guard employed by landlord started harassing my partner
• House 3: landlady moved in one day, without notice. While we were still there.

Finally we and some other expats landed in a nice place where we stayed until leaving Mbour. When we left we asked if some coworkers wanted to take it over. They told us they’d rather not live above a brothel.

Oops?

That explained why the soundproofing was so good 🙃

We learned that a resort-town suburb of Mbour, Saly, was apparently the sex tourism capital of Senegal. This also explained why, when some friends of a coworker came to visit and rented a house there, the walls were absolutely covered in paintings of, uh, butts.

Actually, my partner wrote a blog post about this that is way better than this thread, you should probably just go read that: medium.com/pulpmag/sex-in…

[bedtime, more later!]