The bus driver who can't see the traffic

This is somewhat of a real story.

The setup

A growing company calls for a focus on security. A head of security engineering is hired, who in turn hires a group of security engineers to do the work.

Among other things, the security engineers scan every dependency, machine and image against the current CVE catalog (there are plenty of good tools for that). They issue an alert to rotate all keys across the organization. A big list of CVEs is created and handed over to the rest of the company.

Management tries to find owners, asks for volunteers, and in some cases “volun-tells” engineers to work on them. After all, it's security, so it's always more critical. Dependency upgrades are usually nested and can be really tricky, and most people don't work on them often, which makes them even more painful. Product development and delivery practically stop.

The fun part

Less than 5% of the CVEs were on an exploitable path. More than 50% could have been fixed centrally, in one place, instead of every team struggling to find its own solution.

But there was no one in a role with experience across the infrastructure, the application and the security risks. The hands-on architects who used to operate at that level are extinct, for now, in most places.

It was a similar story with key rotation. It introduced more chaos and outages, because no one fully knew how everything was connected. Someone looking closely could have ruled out rotation for some keys that weren't good candidates for it.

How the roles evolved

It's interesting to line up how roles have evolved as the software industry has progressed. The most recent one I find amazing is the explicit security engineer role. Security used to be the default non-functional part of every engineer's job description.

The focus shifted in two directions at once: QA responsibility moved to the product engineer, while security responsibility moved to a new role. From what I understand:

1. People want to keep up with trends, assuming someone else has already done the due diligence.
2. The rise in security incidents shifted priority from quality to security.

Penny-wise, pound-foolish

The fun part is that it's rarely effective. In a lot of ways it's penny-wise and pound-foolish.

A security engineer is like an excellent bus driver who has the maps and knows the bus's location, but can't actually see the traffic. Meanwhile, the product engineer has been told to sit back and relax, because we already have the best driver.