Thread by @mipsytipsy

I get this a lot: "Do you support on prem? I'd really like to use you guys for observability, but I can't convince the security team to let me." (forlorn look)

So listen up, because I'm about to give you my answer as well as something much better: answers from *actual experts*.

"No, we don't support on prem. Sorry, we have a very small team and more importantly

👋it's 2019👋

Also, observability is something you should affirmatively WANT to use a third-party service in the cloud for. Let's think this through...

"What happens if you run your systems and your o11y tooling on the same hardware facilities, network, etc as your production services? What happens to your o11y when prod goes down?

Now you have n^2 problems. *And* you're stumbling around in a closet blind drunk."

"In theory you could sequester your observability away from prod wholly & completely. I've never seen anyone successfully do that, but you /could/. But you cannot sequester the humans, so they will break things.

And who monitors your monitoring systems?

... omg wtf just happened, where did that chain of tweets come from and it became untweetable until I hit "tweet all" ... 😱

wait, so now nobody will know how long it takes me to compose shit? i can edit? THIS FEELS LIKE CHEATING

(anyhow ... back to the plot now that i've found the tweet button :P)

"in order to protect against human error, infra bugs, and availability zone outages, in order to preserve your ability to debug during outages; the best solution is to outsource your o11y. Use a service."

I consider all the other benefits of outsourcing to be nice-to-haves compared to this one. But they include things like nice features, cutting-edge visualizations, ease of use, oh and NOT HAVING TO HIRE A TEAM TO DO IT

whatever the vendor may cost, it won't compare with that.

"But what about my security?"

Well first of all, it's important to distinguish operational data from auditable data. I wrote a whole long-ass thread about this a few months ago, so read that.

https://twitter.com/mipsytipsy/status/1032146885375475713

The two types of data have radically different characteristics, footprints, use cases, etc. You should separate the streams as early as possible. No PII/PHI should make it into your operational data. Practice good data hygiene, and it should be easy to stream offsite.

More on operational data and what to gather, how to structure it, etc here:

https://twitter.com/mipsytipsy/status/1042817542648082432

or you can just use the honeycomb beelines, they do everything automagically. For extra scrubbage, one-way hash column values before you flush to disk: retains analytic value.

Ask your vendor if they have the ability to scan periodically from cron for PII/PHI, and if they can automatically shut down your ingestion upon detection until someone has okayed it. This isn't hard.

Good security teams are like good lawyers. They don't exist to tell you no; they exist to tell you *how to get to yes*. Any modern security team should be capable of telling you how to outsource your metrics and observability.

But now we've reached the edge of my knowledge. Which is why I reached out to some actual experts for help.

I just posted a three-piece series of guest posts on my blog to answer this question: "how can I get security to let me use a vendor for my observability?"

First: how to choose a third-party vendor and successfully champion them to your security team, by @georgechamales. charity.wtf/2019/02/13/524…

Has a super handy worksheet for you to fill out before approaching your security team. (Useful for vendors to fill out beforehand, too!)

@georgechamales Second: how to practice seeing security's paranoid point of view, and why it matters; and how to maintain your relationship with security over the long run. *great* stuff on empathy from @attacus_au.

charity.wtf/2019/02/13/out…