ADEPT-LEVEL IT TROUBLESHOOTING:
In this series, I will lay concepts and processes for ascertaining technical causes of IT failures and outages.
🎖I am a Microsoft MVP in Windows management, worked 10 years in Helpdesk and System Engineering, and now work as an F500 Security IC.
⭐️LESSON 1: EXECUTION CONTEXT
You initiate a process that should work. But it seems like it can't access what it needs. For example, you make a machine login script on a network share, but logs access denied. Or, you launch a process, and the target you have access to can't open.
A critical, advanced IT troubleshooting concept is understanding _execution context_.
You see a machine with everything on one screen. Diagnostic tool or another tool should have same experience as other apps, right? If you can access file, everything else should? No.
Explained:
All modern general-purpose operating systems are designed as multi-user systems. That means the system has certain rights, administrators have certain rights, users have certain rights, process have certain rights, etc. Your computer runs things in many different, limited roles.
This is inexact, and not a reference, but I will separate the ability to access resources into many different layers of entitlement checks. We will start at the furthest-out, which is your ability to identity to file servers (if applicable), and who that identity shows as.
When talking to servers, there can be different security requirements depending on execution context. A process running as SYSTEM on Win8+ cannot initiate SMB connections to DNS CNAMEs if there isn't a correct Kerberos SPN registered for that host name. This is to prevent EoP.
If communicating via web protocols, each user can also have different proxy settings. User, system, service account - can all be different. There's also complexity in WPAD/PAC usage. If you run a process as you it might work, but a SYSTEM process might not inherit that setup.
If the process is running as a service, there are user principals that don't get any network access, as a hardening measure. If you make your program run as LOCALSERVICE, it will be blocked from network communication. NETWORKSERVICE can, luckily without SYSTEM's root permissions.
Additionally, Windows Firewall can apply per-user Outbound block rules. I think IPSec can also mess with this, but I'm doing this all from memory. These tweets are free go pay somebody else for training.
Additionally, each Windows user has their own Certificates store for personal certificates and Credential Vault for saved passwords that get pulled in during supported authentication attempts by programs. This can cause problems in some proxy situations. All really complicated.
Each user also have their own Environment Variables, which can be used to store credentials or other access directions. There's also the merging of Classes from HKCU that override HKLM, but I don't know if we need to get into that right now.
Okay, now we get into dirty stuff...
REMOTE SERVER~ Next, SMB File Shares have an umbrella ACL that defines the maximum permissions network-source clients get, regardless of their local rights. For example, if you share C:\Test to Administrators but only give them share Read permissions, they can't write remotely.
Windows has an enormously stupid feature forced by Microsoft customers (probably DoD) named "Access Based Enumeration."
If you access a share, it only shows you the files/folders you have read access to. This is terrible overhead and confusing and Microsoft hates it don't use it.
Oh! I forgot! When you authenticate to a SMB server it enumerates your group memberships and caches that for an entirely indeterminate amount of time, refreshed under conditions nobody understands. So if you get added to an access group, it's probably best to just reboot.
Correction: Steve did a deep-dive into this awhile ago in the Windows code for a definitive response, and I forgot he found the answer. SMB NTFS group memberships update around every 45min-1hr. Thanks Steve you a real one. Always got our backs.
I'd like to just take a breather here, we're not half into the layers of access checks required to open a file, even leaving tons of boring and network stuff out, and acknowledge information systems are fractal complexities extremely difficult to correctly architect later.
What I am explaining here is not laid out in any chapter in any book. It is the accumulated complexity of extremely mature system behaviors and featuresets. Make no mistake – the simple act of accumulated time in exposure to them to discover this is its own enormous career skill.
Back to troubleshooting: I think this is system-wide, but I'll mention it anyway: Client-Side-Caching (C:\Windows\CSC of network files through either Offline Files or Transparent Cache) can cause access problems if this datastore is corrupted. This will be obvious in ProcMon.
Okay, now regardless of Allow rights to read files, Deny overrides that. So you need to check for those ACLs. Explicit Deny usage is pretty rare in the real-world, which is why it can trip people up. Its hard to troubleshoot for most admins so avoid using them unless unavoidable.
NTFS/NFS permissions are _extremely_ complex at the fringes when you involve inheritance. The use of the built-in "Effective Access" tool to validate entitlement to open specific areas can be a critical troubleshooting step.
When a Windows program accesses a remote file, it can start a session in the context of the user account, the mapped drive user account, or the computer's unique account in AD (if running as SYSTEM or NETWORKSERVICE). Computer accts are users so Everyone/DomainUsers includes them
Imagine you have 30 Win machines, all with account "user" and password "user."
It's a local admin. That means you can remotely admin all other machines, right? Actually, _local_ user accounts in the Administrators group, when authenticated remotely, are stripped of admin rights.
The exception to this stripping of rights on remote seasons is with the default local Administrator account (RID -500), which is disabled by default anyway.
This is a manifestation of UAC Token Filtering. It does NOT apply to any DOMAIN users in the local Administrators group.
Separately, by default, local accounts WITHOUT passwords CANNOT be logged into remotely. (There's a Group Policy to disable this protection if you're a moron.) There's nothing wrong with using blank passwords on single-user machines you're not worried about physical security of.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
I have to be careful with my platform, but the latest experimental GLP-1 "Retatrutide" has seemingly addressed an underlying deficit that is resolving multiple physical and mental health issues now at increased dose, completely unrelated to weight loss of previous GLP-1's...
It seems not unheard of: It's better even when I don't take them, but over the past five years I've had increasing issues with my ADHD medication having a little effect. Eventually switching to one of the most powerful at a high dose with essentially no impact. That's... fixed.
I don't know if I should be posting this you cannot legally or provably safely get this outside of prescription trials right now. But. It's like changing my life and I don't know what to do with that information.
I'm going to make an analogy as to what Taylor Swift is allowed to do with her seniority and autonomy. others would likely shy from.
And I want to make an analogy to my own experience watching and performing professionally in IT.
Taylor Swift is allowed to write about random characters in Shakespeare, and not care you have no idea who they are. She's allowed to write indulgent songs of purposely cringe lyrics about getting ravaged by soulmate.
But permission is something ENTIRELY different from audacity.
What Taylor Swift has is not permission. She has audacity.
The fact that she has transcended permission is sort of immaterial to my point.
So much of life is not about asking permission. It is having the audacity to simply fucking do it.
USCSB ‼️⚠️ @chemsafetyboard proposed to lose all funding under Trump budget. A rare agency with true bipartisan and industry support, recognized for how cheap it is in prevention analysis and education materials. grist.org/energy/trump-q…
From 2020 when Trump admin tried to kill @chemsafetyboard for reasons literally nobody could understand even Congress.
@chemsafetyboard ~$14m/yr for teams that work on stopping incidents that maim/kill and cost hundreds of billions of a year in capital alone, not to mention downstream societal impact. They are broad advisors, tell industry and regulators what went wrong and how to prevent it.
The thing about Active Directory, is you can't understand any of it unless you begin from the past before it. You cannot examine it from the future. You will get only nonsensicals.
And that's really where most commentators fail. They don't know why. Because there is a reason.
The reasons Active Directory fails is deeper than technology. It is from inception, to ironically be more open than you conceive. It is the sourcing of philosophy in staff whose only job was one portion. Whose users, absolute experts. Whose salary paid one. This... didn't happen.
Active Directory is truly beautiful. But it's a beauty you can only experience in the world it was envisioned for. Outside, it is a horror of hacks trying to address things you can only ascribe hate. Decades later. But trust me, it is beautiful. I wish you could see it, how I do.
I live on a secluded area of my street with little traffic but I purposefully make it evident my surveillance and you know what every dog walker picks up their poop.
👏Always👏be👏engineering👏perception👏
Even on gate I don't lock I have a fake one that makes it appear always padlocked. I have spike strips that are just plastic on areas you could boost over my fence.
I do the same thing in enterprise security. We appear to have three different top-tier antivirus, running on a malware analysis VM, with debug tools running, and more traces like that.
This is your playground they're in and stop denying yourself the freedom to fake it.