Software that can translate hand-drawn diagrams into electronic files is being developed at Auckland University.
Computer science PhD student Rachel Blagojevic is creating software that recognises the elements used in most diagrams, such as flow charts, and converts them into a format that can be used in electronic files. Diagrams can be drawn on tablet PCs using a digital pen, but the software could also be adapted for other surfaces, says Blagojevic.
So far, Blagojevic has mainly worked on the divider between writing and drawing, she says. With hand-drawn diagrams there are both shapes and writing that need to be recognised by different recognition engines, she says. The first step is to work out which strokes made by the user constitute writing and which drawing.
Most diagrams have many different elements that are difficult for computers to recognise, she says. For example, a line could be a capital “I”, a lower case “l” or a connector in a chart, she says.
The software analyses the attributes of particular elements, such as timing and length of pen strokes, to decide what each is and place them in the diagram, says Blagojevic. A simple algorithm then comes up with a probability of whether an element is a shape or writing, she says. The software also takes context into consideration, looking at strokes around each element to figure out the nature of a particular stroke.
The background for the project is human-computer interaction. “We are trying to improve designs,” says Blagojevic.
People usually make designs using pen and paper, and then transfer the design to a computer. Or, because this is a cumbersome process, they might just start using the computer from scratch, she says. However, designs drawn straight onto the computer aren’t as good as those done using pen and paper, she says, citing literature that proves the point.
“If we could use the same kind of interaction that we have with pen and paper, but have this done straight onto the computer, we would not lose the improvement in design,” she says.
The software also helps save time. Transferring pen and paper designs to a computer is often tedious, she says. It hard to get things in the right place, and it can be really frustrating, says Blagojevic.
She thinks the software could be particularly useful in the education sector, but also in her own area of computer science — especially for designing user interfaces, she says.
The fact that the software may be used, and appreciated, by real people is one of the main reasons Blagojevic finds the area so interesting, she says.
The project is funded by Microsoft.