What I Learned from 15 Years in the Trenches
Security lessons learned the hard way.
Early involvement in design
Security must be involved early in the design process, not only near approval, as a rubber stamp. Late engagement creates findings, weakens the design, and turns security into a reactive checkpoint; it becomes an afterthought rather than a partner. Effective design reviews require first principles thinking to build security into systems from scratch.
Risk-based over compliance-driven
Risk-based security is more effective than purely compliance-driven enforcement. Sound security decisions focus on business impact, threat scenarios, and proportionate controls, rather than applying every control uniformly regardless of risk. Moving away from checklist compliance is essential for conducting evidence-based cybersecurity maturity assessments.
Cross-functional stakeholder engagement
Security outcomes depend on cross-functional stakeholder engagement as much as technical analysis.
Threat modelling boundaries
Security threat modelling does not determine impact or likelihood. Impact is determined by the business unit or functional department based on system criticality, information classification level, financial impact, reputational impact, regulatory impact, and operational and technology impact, without considering any controls. Likelihood is determined by the business unit or functional department based on the availability of mitigating and compensating controls.
AI and vendor governance
AI adoption increases governance and data leakage risk. Guardrails must cover data classification, DLP, data flows, and service boundaries. Similarly, outsourcing services to external partners requires maintaining accountability in vendor and third-party contracts.
Scaling through reusable standards
Reviews scale through reusable standards and patterns, such as control baselines, reference architectures, and repeatable evidence expectations.
Security is not absolute protection
Security is a trade-off between value, risk, cost, and usability. The goal is not perfect defense, but enough protection to make abuse difficult, detectable, and recoverable. A common example is MFA. Requiring MFA for every action may look secure, but it can create user friction and operational noise. A better approach is to apply stronger authentication where the risk is higher, such as privileged access, production systems, payment approvals, customer data, and cloud consoles. Security is not about making every action equally hard. It is about protecting the most valuable actions with proportionate controls, while keeping normal work usable, monitored, and recoverable.View Practical Example
If security is hard to use, it will fail
Controls that are difficult to configure or operate will be bypassed, misused, or ignored. Simplicity is not cosmetic; it is a security requirement. A classic example is password rotation. Many enterprises used to force password changes every 90 days, and some stricter environments used 30 days. This looked secure on paper, but often led users to create predictable passwords, reuse patterns, or store passwords unsafely. A better approach is to require strong passphrases, block known compromised passwords, use MFA, monitor suspicious logins, and reset passwords when compromise is suspected. A control that creates bad user behavior is not strong security.View Practical Example
A control is not enough without assurance
Security has three parts: what the rule is, how it is enforced, and whether it actually works. Many reviews fail because they check the control but not the proof. Verification is key to establishing true maturity. A common example is MFA for admin access. The policy may say all privileged access requires MFA, and the system may show that MFA is enabled. Verification means checking the evidence: review the MFA policy scope, confirm all admin groups are included, check for excluded accounts, test a privileged login, and review sign-in logs to prove MFA was enforced. A control only becomes mature when it can be proven to work.View Practical Example
Know which security property matters most
Not every system needs the same balance of confidentiality, integrity, availability, and accountability. The priority must follow the business purpose of the system. An online payment system is a good example. Confidentiality matters, but integrity matters more. The highest risk is someone changing the payee, amount, approval status, or transaction record. So the strongest controls should focus on approval workflow, segregation of duty, transaction validation, audit logs, reconciliation, and fraud monitoring. The security priority must follow what the system is meant to protect.View Practical Example
Access control must enable work, not only block abuse
Too little access creates operational friction. Too much access creates silent risk. Good access design supports both safety and legitimate work. Production support access is a good example. If access is too restricted, engineers cannot investigate incidents quickly. If access is too broad, too many people hold standing production privileges. A better design gives read-only access by default, requires approval for privileged actions, grants temporary access only when needed, and logs all activity. Access control should let people do their work safely, not force a choice between speed and security.View Practical Example
Authentication, authorization, and audit must work together
Knowing who someone is does not prove what they are allowed to do. Every sensitive action should show who acted, what permitted it, and where the evidence is recorded. This forms the foundation of robust authentication and authorization. Deleting a production database backup is a clear example. SSO and MFA prove who the engineer is, but they do not prove the engineer should be allowed to delete a critical backup. The system must check authorization through role, environment, approval, and policy. It must also record the action in audit logs, including who acted, what allowed it, what was deleted, and when it happened.View Practical Example
Configuration is security
Bad setup can defeat good architecture. Permissions, groups, keys, exposed services, and admin settings must be treated as security-critical assets. Cloud storage is a common example. A system may have good architecture, encryption, SSO, and logging. But if a storage bucket is accidentally made public, the configuration defeats the design. Permissions, keys, network rules, and admin settings must be reviewed like security controls. Bad configuration can turn a secure design into an exposed system.View Practical Example
Defense in depth helps, but it is not magic
Multiple layers reduce the chance that one failure becomes a breach. They only work when the layers are independent, tested, and monitored. Ransomware protection is a good example. A company may have email filtering, endpoint protection, MFA, network segmentation, privileged access control, and backups. But defense in depth fails if the backups use the same admin credentials, sit on the same network, and are never tested. Each layer must be independent, monitored, and verified. Otherwise, one compromise can still take down the whole environment.View Practical Example
Prevention will fail
Security programs that rely only on blocking attacks are incomplete. Detection, response, backup, recovery, and accountability are part of the design. A compromised user account is a clear example. Email filtering, MFA, and endpoint protection reduce risk, but they will not stop every phishing attempt or credential theft. Good security assumes a control may fail. The design must detect unusual logins, revoke sessions, reset credentials, review access logs, contain the account, and confirm what data was accessed. Prevention reduces the chance of compromise; response limits the damage after compromise.View Practical Example
Trust must be explicit
Do not accept vague statements like “this system is trusted.” State who trusts whom, for what action, under what scope, and based on what evidence. A third-party API integration is a good example. It is not enough to say the vendor system is trusted. The design must define exactly which vendor application is trusted, which internal system accepts that trust, what API actions are allowed, and what scope limits apply. The evidence must also be clear: approved OAuth scopes, certificates, source restrictions, logs, and ownership records. Trust that cannot be explained is not trust; it is an assumption.View Practical Example
Access decisions must be traceable end to end
Modern systems cross teams, vendors, clouds, APIs, and identity providers. The access path must be explainable from requester to resource. Cloud production access is a good example. An engineer may log in through SSO, get approval in a PAM tool, assume a temporary cloud admin role, and then change a production resource. The access path must be traceable from the engineer, to the approval ticket, to the PAM session, to the cloud role, to the exact API call, and finally to the resource changed. If the cloud log only shows “AdminRole” or “pipeline-service-account,” the trace is incomplete. You know what acted, but not clearly who requested it, who approved it, and why it was allowed.View Practical Example
Audit logs should explain why access was allowed
A log that says “user accessed file” is useful but incomplete. A better log shows the identity, role, policy, delegation, approval, and reason access was granted. See secure logging foundations for details. Privileged access to a production server is a good example. A weak audit log may only say, “User accessed server.” That shows activity, but it does not explain why the access was allowed. A better log records the user identity, privileged role, approval ticket, access policy, session ID, source device, time, command activity, and reason for access. The log should allow a reviewer to answer: who accessed production, who approved it, what permission allowed it, what they did, and whether it matched the approved purpose.View Practical Example
Exceptions need visibility
Every environment has exceptions. The risk comes from exceptions that are undocumented, permanent, excessive, or invisible to reviewers. A firewall exception is a strong example. A vendor needs temporary access to support a production issue, so the team allows access from the vendor IP to an admin interface for one week. The real risk starts when the exception has no owner, no expiry date, no ticket reference, and no review process. Months later, the rule is still active, the vendor no longer needs it, and reviewers only see a “working” firewall rule, not the original risk decision. Exceptions are not the problem. Invisible exceptions are the problem. Every exception should have a business reason, owner, approval, scope, expiry date, and review evidence.View Practical Example
Separate trusted and untrusted code
Untrusted code should not run with full user or system authority. Use isolation, sandboxing, code signing, and controlled data transfer. Browser-based SaaS access is a clear modern example. A user may open email, documents, and internal systems in the same browser where they also visit external websites. If malicious web content can run with the same authority as trusted business apps, one bad page can steal sessions, read data, or trigger actions as the user. A safer design separates trusted and untrusted code. Use browser isolation, sandboxing, device posture checks, app control, code signing, and controlled file transfer between environments. Untrusted code should never run with the same access as trusted business applications.View Practical Example
Design for verification
A secure system should be easy to reason about. If reviewers cannot understand how the control works, operators probably cannot manage it safely. Cloud production deployment is a good example. A team may say, “Only approved code can be deployed to production,” but that control is weak if approval happens in email, deployment is done manually, and evidence is scattered across different tools. A better design makes verification simple. The CI/CD pipeline should show who approved the change, which code was deployed, which tests passed, which security checks ran, which identity performed the deployment, and what production resource changed. If the control cannot be easily checked, it will be hard to operate safely.View Practical Example
Diversity does not remove risk by itself
Using different platforms may reduce mass exploitation, but it does not protect critical data if every platform can still reach it. Isolation matters more than variety. Using more than one edge security provider can reduce vendor concentration risk. For example, an organization may use Akamai and Imperva to avoid relying on a single provider for CDN, WAF, DDoS, bot, or API protection. But diversity does not protect the application by itself. If the origin is still reachable directly, or if policies, logs, failover, and ownership are inconsistent, attackers may bypass the edge or exploit the weakest setup. Provider diversity only helps when the architecture enforces isolation, blocks origin bypass, aligns security policies, and tests failover in practice.View Practical Example
Temporary access must expire cleanly
Tokens, sessions, temporary credentials, and signed links are powerful because they carry authority. They need short lifetimes, clear scope, and reliable revocation. Temporary cloud admin access is a good example. An engineer may be granted elevated access to troubleshoot a production issue. That access should be limited to the required role, approved ticket, target account, and short time window. The risk starts when temporary access becomes permanent. Tokens stay valid, sessions remain active, emergency roles are not removed, or signed links continue working after the task is complete. Temporary access must expire automatically, support revocation, and leave audit evidence showing who used it, why, and when it ended.View Practical Example
High-risk actions need more than one person
Separation of duty reduces insider risk and accidental misuse. Critical actions should require independent approval, especially for money movement, production change, and privileged access. Privileged cloud access is a practical modern example. A single engineer should not be able to approve their own request, assume a production admin role, disable security logging, or change critical cloud resources alone. A safer design requires one person to request access, a separate approver to approve it, and the platform to grant a short-lived role only after approval. The system should log the requester, approver, ticket, role assumed, session duration, API actions, and resources changed. High-risk cloud actions should require independent approval, not just technical permission.View Practical Example
References and Acknowledgements
This post was shaped by past mentors, colleagues I worked with, and leaders I reported to across current and past companies.
It also draws on studies from my MSc in Cybersecurity, including but not limited to:
- Butler W. Lampson’s Computer Security in the Real World paper
- Ross Anderson’s Security Engineering (3rd edition)
- Adam Shostack’s Threat Modeling: Designing for Security