Google shows off AI tool for reading handwritten text by rewriting it digitally

Google Research is showing off a new way to use AI to read handwriting that might radically change how machines convert what you put on paper into digital letters. The InkSight system transforms photos of handwritten words into digital text by leveraging AI without the need for any devices as intermediaries.

The idea is to replace the sometimes fallible optical character recognition (OCR) with AI that can emulate how humans actually learn to read, specifically by rewriting existing text to learn what whole words look like and mean. Doing so required the researchers to tutor the AI in both recognizing and mimicking handwriting by humans.

“Digital note-taking is gaining popularity, offering a durable, editable, and easily indexable way of storing notes in the vectorized form, known as digital ink. However, a substantial gap remains between this way of note-taking and traditional pen-and-paper note-taking, a practice still favored by a vast majority,” the researchers explain in their paper. “Our approach combines reading and writing priors, allowing training a model in the absence of large amounts of paired samples, which are difficult to obtain. To our knowledge, this is the first work that effectively derenders handwritten text in arbitrary photos with diverse visual characteristics and backgrounds.”

InkSight is more than just an alternative technique. It makes for more accurate results in circumstances that aren’t ideal. For instance, if the photo is taken in dim light, has partially obscured text, or is on a confusing background when examined with OCR. The researchers found that humans could read 87% of the InkSight-made tracings of text. Two-thirds were good enough…