The Theory of Everything – Physics' Greatest Unfinished Quest

Drop your keys and they fall to the floor. Look at the phone in your hand and its screen glows because electrons inside it obey rules worked out a century ago. Both everyday events are explained, with astonishing precision, by modern physics. What is less well known is that they are explained by two different theories, and those two theories do not agree with each other.

Physics can sound supremely confident, so this may come as a surprise. Its two greatest achievements, each tested to extraordinary accuracy and never found wanting in its own territory, contradict one another at a deep level. Building a single framework that covers everything, from galaxies to grains of sand to subatomic particles, is often called the search for a theory of everything. It has occupied brilliant minds for the better part of a century, and it remains genuinely unsolved.

The Theory of the Very Large

The first pillar is Albert Einstein's general theory of relativity, published in 1915. It describes gravity not as a pull between objects but as the bending of space and time by mass and energy. A planet orbits the Sun because it is following the curves the Sun's mass creates in spacetime.

General relativity governs the big things: planets, stars, galaxies, black holes and the expansion of the universe itself. It predicted that light bends around the Sun, that clocks run slower in strong gravity, and that colliding black holes would send ripples through spacetime. Those gravitational waves were finally detected in 2015, with Australian physicists among the international team. Every test thrown at the theory so far, it has passed. Its ideas are explored in Einstein's Relativity – Time Is Not What You Think.

The Theory of the Very Small

The second pillar is quantum mechanics, the theory of atoms and the particles inside them. Down there the rules are strange. Particles behave like waves, exist in combinations of possibilities at once, and jump and fluctuate in ways that are fundamentally about probability rather than certainty.

Strange or not, quantum mechanics is spectacularly successful. It underpins chemistry, lasers and every computer chip, and some of its predictions match experiments to better than one part in a billion. Combined with special relativity, it became the framework behind the Standard Model, which describes the electromagnetic force and the strong and weak nuclear forces. For more on its oddities, see Quantum Mechanics – The Rules That Make No Sense.

Two Perfect Maps That Will Not Overlap

So where is the problem? The two theories picture reality in incompatible ways. General relativity describes a smooth, continuous spacetime that curves gracefully. Quantum mechanics describes a world that, at the smallest scales, is jittery and restless, never quite still.

Try to apply quantum rules to gravity itself, asking what spacetime does at the tiniest scales, and the mathematics breaks down. It produces infinities that cannot be tamed by the tricks that work for the other forces. It is like having two flawless maps of the same country, drawn by incompatible rules, with no way to lay one over the other.

There is a pattern in physics that makes people hopeful. In the 1800s, James Clerk Maxwell showed that electricity and magnetism are two faces of one force. In the twentieth century, electromagnetism and the weak nuclear force were joined into a single electroweak theory. Gravity is the one force that has stubbornly refused to join in.

Where the Clash Really Bites

For most of physics, the conflict does not matter. Things are either big or small, and you pick the right theory. But there are a few places where something is both enormously massive and incredibly tiny at the same time, and there you need both theories at once.

• The centre of a black hole, where general relativity predicts matter crushed to a point of infinite density. Most physicists read that infinity as a sign the theory has reached its limits, not as a real feature of nature.
• The first instant of the Big Bang, when the whole observable universe was squeezed into something unimaginably small and dense.

Understanding either requires a theory of quantum gravity, one that unites gravity with the quantum world. No such theory has yet been shown to work, which is why the birth of the universe and the hearts of black holes remain among the deepest mysteries in science.

The Leading Candidates

Physicists have proposed bold ways to bridge the gap. The best known is string theory, which suggests that the most basic ingredients of reality are not points but tiny vibrating strings. Different vibrations would show up as different particles, much as one guitar string can play different notes. String theory is mathematically rich and, in principle, could include gravity alongside everything else. It also requires extra dimensions of space beyond the three we know, presumably curled up too small to notice.

Other approaches attack the problem differently. Loop quantum gravity, for instance, starts from gravity and suggests that space itself may be made of tiny discrete units rather than being perfectly smooth. There are further ideas still, each with enthusiasts and critics.

Here is the honest heart of the matter: none of these has been confirmed. The differences between these theories and ordinary physics are expected to show up only at fantastically small scales and enormous energies, roughly a million billion times beyond what the Large Hadron Collider can reach. Nobody yet knows how to test them directly, and an idea, however elegant, only becomes established science once experiment can check it. For now they are serious candidates, not answers.

Why an Open Question Is Good News

It is worth being clear about what this is not. Physicists are not quietly sitting on the answer. Some of the smartest people alive have worked on unification for decades without cracking it. It is quite possible the solution will come from a direction nobody has yet thought of, or will demand ideas as revolutionary as relativity and quantum mechanics were in their day. Some physicists even question whether a single final theory exists at all, and that too is an open question.

Far from being embarrassing, this is one of the most exciting places in science to stand. Researchers look for indirect clues in the behaviour of black holes, the afterglow of the Big Bang and ever more precise experiments, hoping one of them will hint at what lies underneath.

A Quest Still Underway

The theory of everything is less something physics has than something it is reaching for: one description of reality from the largest galaxy to the smallest particle, with no contradictions at the seams. We have two of the greatest theories ever devised and the humbling fact that they cannot both be the final word. Somewhere beyond them may lie a deeper truth that contains them both. Finding it would rank among the greatest achievements in human thought, and for now it remains gloriously unfinished.