Alright I’m gonna bite on this one from the Square POS perspective because why not…
How many languages does your product support? Have you made sure that your features work in all these languages in a reasonable way? Who's testing all those features in all those languages?
Language is one thing, but what about actual differences by country? What cultural norms require you to change your product or build entire sets of functionality for given countries? Japan requires a specific type of receipt for expenses, for example.
Ok after that, are there different technologies for a given country? For payments, Japan has eMoney, Canada has Interac, Australia has single-message payments instead of the US standard auth + capture. All of this is eng months of work, and will change how your products function.
How many teams ship features in your app? How do you make sure the architectures and APIs they're designing are sane and consistent? How do you make sure their code is sane? Backwards compatible? You've got to write down and review all of these designs, and that takes time.
After the designs are approved, you've got to review the code, too. This is definitely a couple hours a day per engineer.
What about design and UI? If you've got 20 to 50 people working on an app, you probably want to have a set of primitives to make it easy to build these standard things and provide advice for how to build new ones. We call this Design Systems, and that’s at least a few people.
Design, though? We can move lower: CI and local builds? With 50 engineers, that's 50-100 PRs per day landing, which is an entire CI farm to run those tests, and another set of engineers ensuring that local builds stay fast. Every minute you save here is super important.
A bit about CI: About 4 years ago we had to build a custom test sharding solution because we had a combined 4 hours of unit and integration tests, which is obviously a shit an unacceptable amount of time for a PR. Sharding got this down to about 30 minutes, but was much effort.
Now tests are running, but what if they’re flaky? You got 50 engineers writing code all the time, code is state, therefore, you’re probably going to have tests flaking sometimes. Gotta build tooling to detect and disable flaky tests to make contributing not annoying as fuck.
This gets us to actually building incremental new features. Back to multiple teams – when you build a new feature, you've got to think about how it affects client, server, web (dashboard), including older versions of those native apps. That's a lot to consider!
What if your code is used by multiple apps? Will you break them?
Once that's done, you can actually begin writing your feature. This is the easy part that “just takes a few people”.
Once your feature ships, you've got to maintain it, improve it, and be ONCALL for it. You probably have an average of one engineer ONCALL for every 10 to monitor for issues, fix them as they arise, work with support to determine if issues are occuring or not, etc.
Ah shit, GDPR just happened. Time to spend a couple months ensuring your products are compliant to those deletion requests. Figuring out what we have to scrub for GDPR was fun, since the law is so ambiguous...
Double oh shit, Google and Apple just released new compiler toolchains and OS versions. Better get your 1+ million lines of code compiling on all those new tools and running smoothly on new OS versions, across all device and OS versions.
Don’t forget you also have to continue QAing and accounting for all old OS and device combinations, too! For us, that’s 4 iOS versions and like 10 devices.
Wow, this sure is a lot of work. Better go try to hire some more people – this is probably 2-3 hours per week of interviewing per engineer, not including coming up with a good interview question and learning how to calibrate your interviews.
Finally, what if your product is super popular now? Issues that only affect 0.1% of people could now be affecting 1000, 10,000, etc. You’ve gotta fix em now even if they weren’t an issue before. Especially if they deal with money.
Alongside popularity and scaling stability of your mobile apps, you’ve got to consider the server: You have to invest weeks or months into sharding services, improving data storage efficiency, etc. If you don’t, your company can literally fall over.
Some other random stuff that I can’t fit into the narrative here…
1) What if you hire a bad engineer? Statistically it’s going to happen eventually, leads and engineers have to deal with that and it drains time. I assume 1/20 engineers is not pulling their weight at any given time.
2) Commitments across teams? You have to plan those, well in advance of work.
3) Your team temporarily needs more people than it has? Gotta plan for that, probably borrowing.
3) Your team temporarily needs more people than it has? Gotta plan for that, probably borrowing.
4) New engineers have to come up to speed on a 7+ year old, 1 million+ line code base? That’s gonna take a while.
5) Want to let people move around the company to new things they find interesting? Gotta account for that in hiring and assumed attrition. Hire ahead of attrition.
5) Want to let people move around the company to new things they find interesting? Gotta account for that in hiring and assumed attrition. Hire ahead of attrition.
6) Did you account for sick days, paternity, medical issues, time off in your hiring? 10% of my team is out on pat leave this year, which is great! But gotta hire around it!
7) Got some 5 year old code that did the job but now 10x more people are relying on it and contributing to it? Gotta redo it to make everyone more efficient.
8) A module that was 50 source files is now 400 and it’s super hard to reason about? Gotta figure out how to break it apart into a sane dependency tree without rewriting everything or stopping the world for months.
Oh finally, double all these eng numbers because you have both iOS and Android apps, and then add some padding since you want to make sure both those apps stay in sync. Which also explains why something like Reactive Native is so attractive to management (don’t do it tho).
Finally finally, I wrote this in a Google Doc because bigco™.
Jk, one last thing: After posting this, we're having a SEV which crosses iOS, Android, and Server with no clear answer. Glad we have ONCALL engineers to help diagnose who are otherwise absolved from normal work.
