Microsoft’s Sketch2Code Makes Code from Your Poor Sketches

Earlier this year, Microsoft issued a program named “Ink to Code” that translated quick drawings of a user interface (UI) into working code. It was pretty cool, and obviously held a lot of promise: Even if you didn’t have a lot of time (or coding knowledge), you could still create a prototype of a great idea.

Now Microsoft has taken the next step: Sketch2Code, which allows users to upload a photo of a website or UI and have it translated into working code. The web-based program leverages Microsoft’s recent artificial intelligence (A.I.) research: For example, a computer-vision algorithm scans the photo and determines the placement of HTML elements, while a text-recognition module reads any text (for example, the “I agree to terms and conditions” statement within a field.).

Sketch2Code is a bit more advanced and flexible than Ink to Code. For example, Ink to Code required that users use a special Windows 10 app to sketch their design (with a stylus), and draw symbols to represent text fields and other specific elements. Sketch2Code, by contrast, just needs a handwritten image uploaded to the website—no special symbols necessary—and the A.I. tries to take things from there.

For anyone interested, the Sketch2Code code is also available via GitHub. And here’s a nifty graphic of its architecture:

It will be interesting to see if Sketch2Code’s HTML markup truly aligns with folks’ original designs. In the meantime, Microsoft seems increasingly dedicated to generating no-code and low-code tools for people to build apps and websites.

For example, Microsoft is currently pushing PowerApps, a low-code building platform that allows “citizen coders” to construct apps simply by dragging elements around. In a recent meeting with Dice, two Microsoft executives detailed how a security guard at Heathrow Airport used PowerApps to build an app that made security operations more efficient; he did it so quickly, in fact, that it took his management some time to realize that the airport’s security guards were using new, customized software for their job duties.

Nor is Microsoft the only company playing in the no-and-low-code space. In addition to a variety of startups with similar products, Google App Maker offers a low-code building environment for custom business apps. In theory, such platforms will free up corporate tech pros to focus on truly difficult problems, instead of trying to field every single employee request for a bug tweak or a new feature. But there’s also the possibility that it could kill some jobs, as companies forgo hiring full-time developers in favor of having their employees build as much as they can with these toolkits.

Related