Know-What-Good-Looks-Like Calibration
Triangulate unfamiliar work through credible evidence, observation, and peers
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 99%
Siemiatkowski recommends building a reference for good work from three sources. First, filter advisers by whether they personally contributed to the kind of success relevant to the decision, rather than assuming a senior title or famous employer proves judgment. Second, sit beside the specialist while they perform a real task and ask them to narrate the process; watching a CTO fix a bug gave him a practical feel for the work and its timing. Third, compare how several credible peers reason about the same concerns. Klarna met CTOs from five large Swedish companies, and the differences in their answers, philosophy, and optimism clarified what its own role required. The leader then evaluates fit against the company's needed outcome and scale. A mismatch does not mean the incumbent is bad; it means the role needs something different.
Origin
Unable to code, Siemiatkowski struggled to judge Klarna's engineering leadership. He learned by watching his CTO fix a bug and by discussing the same concerns with five other CTOs. Those comparisons helped him conclude that a capable programmer was not the right person to build Klarna's engineering organization.
Core principles
- 01Seniority is not the same as relevant evidence of success
- 02Direct observation makes unfamiliar work more legible
- 03Several credible peers provide a stronger benchmark than one adviser
- 04A capable person can still be wrong for the scale or mission
- 05Technical excellence and role fit are separate questions
How to run it
- 1
Define required success
Describe the outcomes, scale, pace, and cross-functional contribution the role must produce. Separate those requirements from a generic picture of competence.
Pro tip Ask what this person must make possible for the company in its next stage.
Watch out A vague requirement such as 'excellent engineer' cannot distinguish technical craft from leadership fit.
- 2
Filter advice by evidence
Examine whether each adviser actually helped create the kind of success being discussed and what their contribution was.
Pro tip Distinguish building a successful system from merely holding a role inside one.
Watch out Prestige, age, and corporate seniority can create confidence without relevant judgment.
- 3
Observe real work
Sit with the specialist while they complete an ordinary task and ask them to explain choices, sequence, tests, and tradeoffs.
Pro tip Choose a real task such as diagnosing and fixing a bug rather than a polished demonstration.
Watch out One observed task reveals the work but does not establish overall performance.
- 4
Build a peer benchmark
Discuss the same concrete concerns with several people who have performed the role successfully in relevant environments.
Pro tip Compare how they frame the problem, what they consider possible, and which tradeoffs they prioritize.
Watch out A peer's context may differ, so do not copy a single answer as a universal standard.
- 5
Judge fit, not worth
Compare the incumbent's approach with the role's requirements and the evidence gathered. Decide whether coaching, a role redesign, or a personnel change is needed.
Pro tip State the mismatch precisely so a role decision does not become a claim that the person is bad at everything.
Watch out Do not postpone a necessary change merely because the person is capable in a narrower role.
In the wild
Siemiatkowski asked his CTO to show him how he fixed a bug. He watched the CTO search the code, write the fix, and create a test case. Siemiatkowski says even that short period of observation materially improved his understanding of the work and how long it might take.
→ A previously opaque specialist process became concrete enough for a more informed management conversation.
Klarna met CTOs from five large Swedish companies, including Ericsson, a gambling company, and gaming company DICE. Siemiatkowski raised the concerns he had been discussing internally and compared the other CTOs' answers, philosophies, and optimism with those of Klarna's CTO.
→ The comparison helped Siemiatkowski conclude that Klarna needed different engineering leadership for its ambitions.
Common mistakes
Borrowing confidence from prestige
An adviser can sound authoritative because of age, title, or employer without having built the outcome the leader needs.
Managing only through abstractions
Technical language remains intimidating when the leader never watches how ordinary work is actually done.
Calling a mismatch incompetence
Siemiatkowski stresses that Klarna's CTO was capable, but not suited to the scale and organizational leadership Klarna required.
Is it for you?
Best for
It is best for leaders hiring or managing technical and specialist roles outside their own expertise.
Not ideal for
It is not ideal as a substitute for independent technical diligence where safety, security, regulation, or major capital is at stake.
From the transcript
“Have they really contributed to success? Have they really built success?”
“can you show me how you fix a bug?”
“we booked meetings with five other CTOs in five large Swedish companies”
From the episode
Klarna Founder: From $0 to $46 Billion: Sebastian Siemiatkowski