What is Apple ARKit?
ARKit is Apple’s framework for augmented reality apps and games. These turn your living room into the environment, or world, of the app. It’s part of iOS 11.
And it’s not just for the iPhone 8 family.
ARKit is a developer tool. It’s not something we iPhone users have to worry about too much. However, a look into what it’s about can tell us a lot about what the future looks like for augmented reality on iPhone.
How does it work?
ARKit uses the iPhone’s cameras to map out an environment. A key part is recognising where the walls and floor are, establishing the basic geometry of the space.
It’s the Apple alternative to Google Project Tango, but puts fewer demands on the hardware. Project Tango requires a special camera array with dual cameras and an IR transmitter. This lets it make a rough 3D model of a room on the fly, and track motion with almost unnerving accuracy.
ARKit doesn’t require this. Instead, it recognises planes, like your floor. Then it uses the phone’s camera and its motion detectors to track movement as the iPhone is moved and tilted.
Motion co-processors like the iPhone 7 Plus’s M10 can efficiently track data from the gyroscope, accelerometer and magnetometer. This lets it monitor movement in a 3D space without excessive hardware demands. Google has actually announced something similar too, called ARCore. It’s Project Tango for phones without the flashy camera tech
The core of ARKit is about letting you drop objects into an environment, and manipulate them using the touchscreen. Other than recognising real-world objects, the main aim is to keep track of objects as the iPhone is moved. As soon as it loses track of this, the illusion is shattered.
One of the most obvious uses for this is interior design. You could place a chair in the corner of the room, and see how it looks as you turn the phone for a different view. It is like a more advanced version of some online glasses retailers’ virtual try-on feature.
We tend to think of far-fetched, futuristic real-world uses when we start thinking about AR and VR. But some of the best applications of ARKit are actually fairly mundane.
What apps will use ARKit?
It’s early days for ARKit. However, there are already lots of apps showing off what it can do (if only as trailer videos). Here are a few of the best demos.
The Machines is a multi-player RTS game that lets several people play in the same AR environment. You control a troop of soldiers, in the vein of Command Conquer.
It features advanced elements like audio that gets louder as you get closer to the action, and close to console-grade graphics.
Skyguide is a very neat astronomy app that pastes constellations onto the sky. We’re not talking made-up ones either. Using the compass and your location, it can determine the actual star charts for your position.
ModiFace 3D lets you see how you’d look with a different hair colour. It comes across a more advanced take on what some Snapchat filters do.
This is an interesting demo because it’s only really concerned with object recognition and tracking in a real space, not the room mapping so many associate with AR.
ARZombie is a classic example of an AR game. It puts you in a zombie home invasion scenario, 3D-modelled undead breaking in and attacking.
You fend them off, and build virtual defences in doors and windows to keep them out.
Rainforest Garage is a delightfully silly AR demo. You see an outline on a wall, and a blowtorch on the floor. Pick up the blowtorch and you can use it to cut out the outlined part of the wall.
It disappears, opening up to show a cartoony rainforest, like a low-rent Narnia.
AR Tape Measure is one of the ARKit apps we can really imagine using. The name explains it all.
You tap and drag out a virtual tape measure, which shows you the distance between objects. A similar tool is also one of the key uses for Project Tango. I’d probably still get the tape measure out for home decoration, but it seems pretty accurate already.
How does ARKit measure distance with “one eye”?
Some Apple phones have the dual cameras needed for traditional mapping of spaces. The iPhone 7 Plus, iPhone 8 Plus and iPhone X do. However, ARKit works with phones other than these too (see below). So how does it work?
No magic is required. An iPhone’s motion sensors let it approximate with surprising accuracy how much the device is moved or tilted.
ARKit may not be able to judge the layout of a room with a single, perfectly still view of a scene. But it can instantly re-analyse that same scene when your hand movements cause the phone to tilt by, say, two degrees.
This communication between the motion sensors and the camera more-or-less completely eradicates the issue of having one camera. I said this wasn’t magic, but it’s pretty close, right?
ARKit can use this progressive analysis of a space to build as accurate a portrait as Project Tango. In theory.
What are the limitations?
As with any form of AR, ARKit is fairly processor intensive. This means it’ll drain the battery quickly, and the phone/iPad may get warm or hot.
ARKit is not designed to cope with very fast motion either. Apple goes as far to explain to developers ways that they should politely advise users to slow down on the Apple Developer website.
Can I use it in any room?
How much space you’ll need depends on the kind of app being used. The hair changer Modiface 3D won’t need any at all.
However, if you’re hoping to play a full ARKit game, it’s not going to work that well if your living space consists of a bedroom that only just fits in a double bed. As the environment mapping relies on identifying planes, like the floor, a cluttered and/or cramped space probably won’t do the trick.
Which phones support ARKit?
ARKit isn’t just for the new iPhone 8, iPhone 8 Plus and iPhone X. It’s designed for iPhones and iPad that use the A9, A10 and A11 processors.
That includes the following devices:
- iPhone SE
- iPhone 6S
- iPhone 6S Plus
- iPad (2017)
- iPhone 7
- iPhone 7 Plus
- iPhone 8
- iPhone 8 Plus
- iPhone X
Have one of these? You just need iOS to try it out, once ARKit apps start appearing on the App Store.