We are now in a position where we must trust the oracle, but we are forbidden from looking behind the curtain. Historically, enlightenment thinkers believed that explanation preceded trust . We believed the sun would rise because Newton explained gravity. We believed a surgeon was competent because we saw their diploma.
The machine was brilliant. It was also dangerously stupid. And no one could see the error until a person almost died. This opacity is crashing headlong into Western jurisprudence. The 14th Amendment guarantees "equal protection under the law." But what happens when a judge uses a black box algorithm to set bail? If the algorithm is biased against a zip code, how can a defense attorney cross-examine it? blackbox
Then came the neural network. Unlike classical software, where a human writes IF X THEN Y , a neural network learns by itself. You feed it millions of cat photos. It adjusts millions of internal "neurons" (weights and biases) until it recognizes a cat. But here is the horror: The final model is a soup of 100 million floating-point numbers. No human, not even the programmer who trained it, can look at that soup and tell you why it decided a particular image was a cat. We are now in a position where we