Artificial intelligence systems may be good at generating text, recognizing images, and even solving basic math problems—but when it comes to advanced mathematical reasoning, they are hitting a wall.

What if the next new mathematical discovery didn’t come from a human mind, but from an AI? Imagine a machine not just crunching numbers but proposing original solutions to problems that have baffled ...