Here at Placenote, we believe that we're witnessing a rapid change in the way people interact with computers. We've entered the age of spatial computing; we no longer have to settle for applications confined to the boundaries of your phone screen, and we can now build programs that interact with the world around us.

Developing these spatial apps has a bit of a learning curve, especially when it comes to working with 3D content. Although we take care of providing accurate camera poses (position and orientation) and a map of the space, it's still up to you, the amazing developer, to properly place your content in your scene.

In today's post, we'll teach you some of the background knowledge behind 3D applications, which will make the process of building your spatial application a little bit smoother!

TLDR

  • Creating spatial 3D apps can be a little daunting due to the new tools and terminology.
  • 3D coordinate frames are the foundation of these applications. Read the article to find out more!
  • Different tools have different coordinate frame conventions. Read the article to find out what they are!
  • The world, camera, and local content coordinate frames are the 3 key frames you need to know about when building your app. Read the article to find out why!
  • If I could fit this entire article into a TLDR, I wouldn't have written an article.

Why Should I Care???

Making the leap from developing a 2D application to 3D can be a little daunting, especially if it means starting to use some new tools like Unity or SceneKit. These programs have a ton of functionality and the documentation typically dives right into some of the technical terms. We've found that there's a significant amount of trial and error involved when you first get started with building a spatial app, especially when it comes to properly placing content and setting up your scene. Although coordinate frames can be boring (sorry in advance), they really are the foundation for any AR application. Having a strong understanding of the relationship between coordinate frames will save you a ton of time as you build your AR app.

Defining 3D Space

First, we have to define the world around us. We live in three-dimensional space, which means that three values are required to fully define the position of any point in the world. These three values can be expressed by the x, y, and z-axes of the coordinate frame shown below. Each of these 3 axes are orthogonal to each other, meaning that they all intersect with each other at a 90-degree angle.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/18514099-5118-4869-b1ac-c5ff6b3bf2b6/frame.png
Example 3D Coordinate Frame.

The intersection point of the 3 axes (the black dot) is the origin, and its position expressed in [x, y, z] coordinates is [0, 0, 0]. The coordinate frame is the foundation for defining content in 3D space. Without a reference coordinate frame, the definition of a point is meaningless; we always have to specify the reference frame to provide context.

For example, let's say we want to define a point p that is positioned at 3 metres in the x-axis, 2 metres in the y-axis, and 5 metres in the z-axis relative to this coordinate frame. This point in [x, y, z] coordinates is expressed as [3, 2, 5], and it would look as follows:

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/c6976150-6a46-493a-bcfc-b98d1e25682d/point.png
Example showing the point [3, 2, 5] in the reference coordinate frame.

The position is also called the translation of the point with respect to the reference frame. Typically in our AR application, we define a coordinate frame called World, and this frame is the fixed reference for all of our content.

You might be asking, "so what does this have to do with AR? I just want to put a 3D horse on my coffee table". You're right, most of the time we are working with objects instead of points. Objects in 3D have 6 Degrees of Freedom (DoF); to fully define an object in 3D space, we have to also define its orientation as well as its position. This requires 3 more parameters (hence 6 DoF - 3 to define translation, 3 to define rotation). These 3 parameters are the rotation around the x-axis (roll), y-axis (pitch), and z-axis (yaw).

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/553e8257-9134-4d5e-bd76-8832f9a742b9/rotations.png
The rotation parameters in 3D.

All this information brought together is called an object's pose (position and orientation). This fully defines the 3D transformation of the object with respect to the reference coordinate frame.

To build on the previous example, let's say we take our 3D horse and placed it at [3, 2, 5] as before.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/b810d5a0-4da8-40ed-baab-8bc998006493/horse.png
Object placed in 3D.

We can now visualize the effects of a 90 degree roll, pitch, and yaw when separately applied to the model.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/26667cea-f44b-4099-9228-07a0f5036921/horse90roll.png
90 degree roll applied to the object.
https://s3-us-west-2.amazonaws.com/secure.notion-static.com/d3402ec3-ebe1-4a26-8586-12dd956d2272/horse90pitch.png
90 degree pitch applied to the object.
https://s3-us-west-2.amazonaws.com/secure.notion-static.com/f5e21f1c-6fd7-45a2-a105-05d22282adc8/horse90yaw.png
90 degree yaw applied to the object.

Or of course, any combination of roll, pitch, and yaw can be applied to Li'l Sebastian here.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/0cdf524a-688a-4de5-96d2-5a5ee9c004be/horserandomrotation.png
Combination of rotations applied to the object.

In all cases, the position of Li'l Sebastian is unchanged, but since we are rotating the model, the horse's orientation, and therefore pose, are being modified.

Left-Handed versus Right-Handed Coordinate Frames

Based on the application you're using, coordinate frames can be defined either using the right-hand or left-hand convention. The right and left-hand rules are an easy way to remember the directions of the x, y, and z-axes. While it would be nice for every application to use the same convention, this is not the case, because all standards were created to be ignored.

All of the frames shown above in this article are right-handed coordinate frames. If you use your right hand (thumb for the x-axis, index finger for the y-axis, and middle finger for the z-axis), you can align your fingers into the 3D coordinate frame axes. If you use your left hand, the positive z-axis points downwards!

Really Important Note: Actually try to create the coordinate frame using your fingers. You'll look absolutely ridiculous. It's a rite of passage in the computer vision world.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/1636c034-c18d-489e-a5c1-0ca8d2b7d3bb/lhrrhr.png
Left and right-handed coordinate frame conventions. Retrieved from https://www.oreilly.com/library/view/learn-arcore-/9781788830409/assets/a465e4c5-b6ca-4006-a40e-1aa9ad2ebc5d.png

Placenote SDK is compatible with both Unity (left-handed) and Swift/SceneKit (right-handed), so it's always a good idea to know your application's coordinate frame convention before you get started. Below are examples of coordinate frames as seen in Unity and Swift.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/2d902034-1c09-45e8-8318-b64754e52ebf/Screen_Shot_2020-02-28_at_16.05.40.png
Left-handed coordinate frame as seen in Unity.

https://docs-assets.developer.apple.com/published/a9d3c84003/03f7f401-5c21-4ac9-9c8c-e7640ec11d81.png
Right-handed coordinate frame as seen in SceneKit. Retrieved from: https://docs-assets.developer.apple.com/published/a9d3c84003/03f7f401-5c21-4ac9-9c8c-e7640ec11d81.png

Left and Right-Handed Rotations

When switching between a left and right-handed coordinate system, the direction of a positive rotation angle also changes.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/ea577fbd-6523-4303-b70f-15773c229271/lefthandedrot.png
Left-handed coordinate frame, with arrows pointing in direction of positive rotation.
https://s3-us-west-2.amazonaws.com/secure.notion-static.com/bf2124d9-1186-4a42-8d54-350e145fe503/rotations.png
Right-handed coordinate frame, with arrows pointing in direction of positive rotation.

We see that for all the rotation angles, the arrow points in the opposite direction when compared to the other coordinate frame convention. However, it's easy to determine the proper direction by using your hand! Point your thumb (either left or right hand, depending on the convention) in the direction of the axis arrow, and then wrap your fingers closed. Your fingers will curl in the direction of a positive rotation.

https://i.stack.imgur.com/n6k7r.png
Right-hand rule for rotations. Image retrieved from https://i.stack.imgur.com/n6k7r.png

Important Coordinate Frames

For an AR application, there are several key coordinate frames that we have to be aware of.

World Coordinate Frame

The first frame is the world coordinate frame. This is a fixed reference frame that remains static for the duration of the application. When you are using an ARKit app, the pose of the world frame is determined as soon as the AR Session is started. When the session starts, the world frame is placed, and remains, at the initial position of your camera and is oriented as seen below, with the positive y-axis pointing in the opposite direction to gravity.

https://docs-assets.developer.apple.com/published/ffb3831f78/d937c9e9-05fd-46e3-b72d-eb353b79bfbd.png
Right-handed world coordinate frame as seen in SceneKit. Retrieved from: https://docs-assets.developer.apple.com/published/ffb3831f78/d937c9e9-05fd-46e3-b72d-eb353b79bfbd.png

In Unity, the world frame is at the same position and orientation, however as it is a left-handed frame, the positive Z-axis points away from the user. This frame is extremely important, as it is the global reference for all 3D objects in the scene.

Camera Coordinate Frame

The second important frame is the camera coordinate frame. This coordinate frame is placed at the center of the phone's camera and is used specifically to denote the pose of the phone at any given time. As per ARKit's documentation, the camera frame is a right-handed coordinate frame, with the x-axis pointing down the long side of the device, and the z-axis pointing towards the user. Using the right-hand rule, we can determine the direction of the y-axis.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/c0a3c7d3-2c35-43be-9821-4e4ec9db60be/arkitphonecoords.png
Camera coordinate frame as seen in ARKit.

Similar to the world frame, once we bring this into Unity, this becomes a left-handed coordinate frame. The x-axis still points down the long side of the device, but now the z-axis points away from the user, in the direction of the camera.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/2bb904ef-7fb9-494d-bdd9-2dfc6765e0a9/unityphonecoords.png
Camera coordinate frame as seen in Unity.

Unlike the world coordinate frame, which is always fixed, the camera coordinate frame moves as the user walks around. The only aspect that is "fixed" about the camera coordinate frame is that it is always placed at the center of the camera. Therefore, at each timestep, which is every image provided by ARKit, the pose of the camera will have changed with respect to the world frame. Tracking the camera poses as you walk through a scene might look like this:

https://s3.us-west-2.amazonaws.com/secure.notion-static.com/d04bae9c-d579-4e10-9ec9-efffa1988d48/Untitled.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAT73L2G45KKGMMXXN%2F20200304%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20200304T230125Z&X-Amz-Expires=86400&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEIv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJIMEYCIQDsC3vmKzmTA3liCZvQcAocJX33NNmXSwUhlV2HurN%2B7wIhAKgr3Nx8yBUuujKFbZTd9emsRUV%2FzpFsZw89N76Lvmm4KrQDCGQQABoMMjc0NTY3MTQ5MzcwIgzpRx6pxN1QSjn2YaoqkQMHYXqkGLSK00vRpIhbhrbtlR9qi6kVu%2FJ2jcczurbnLIUlz1jxLOqRjak3IdrG0D%2B%2BL99TTpgWh2DpvMJ0J9O56xJIzov4MAR1hfg56BrkYYxze4waZEtJR2BVd%2BHKIHx01IW5ugM58QxhVu2mPhxgFe72f1thMwKaH6OnS%2BklyULfGdnz1GnkK%2FzCpBcWnZkmLNRtBemjUekiOaeegwadAtZtXj4dJoRvcNf%2BDBb3xhmIM3UcS92TyH7Vvwj%2FFGMgrdu9sbdmKbpzk2kngYXDyWpQerAXiOhcMsa7PhpbY%2BxnPMAm0k11xwv7Lv56DHXnSS0ED7%2FA8r2g7W5cCG9Y%2B7Z1213pqJchMCvIRcNqnHvrJjfepoCXwRs%2Fcs0LGG5fUTCXg09yms8rZRIIaOtk4XkzFxPHQ0YsA92%2FDEQd4JDTJc4Pe5puAYftTN9NI1MC62%2FPZO4SLY6wwsu8%2BkdDcJMF%2FUojcKXH4ZH%2BQIfq0BegjH%2BwkOS8diVsOR%2FA1VVvG7Y4m34mAnSnjcwLczFDhzCa7%2F%2FyBTrqAcaettaHknIdeyzAVLmwCglO%2BB35cFub54pdA%2Bwapr8qSM6c2QkPXL43%2BuVXArEXc5XP2DujFvblZdgkGkhMz8b4BnBpUtCSOlSSn3pP0MC%2Fn5DhnhOuIPIMmXyLdLi7EI5%2Fz8XV8fVgj77G2VB%2FBqXGcwX5GRkjNp3NWQxVI7BD0E5gdDafFFABdK2W60r8iSTJKtIyRViuuZZfdpPZCm12VBT51vGv6%2FNXQTNE4QXft55iBCiVwDtrcz2vo9c%2Br26iv4IcrN8d1MEdM6SvONo%2FL1Uk13VBOyEBedYU1yNWRtfnabXYANO43w%3D%3D&X-Amz-Signature=c4ca0ab93c23d5d296c2d40958a5af0dc2e9710a05583aab23a41a2f8d0c0c35&X-Amz-SignedHeaders=host&response-content-disposition=filename %3D"Untitled.png"
Tracking of the local camera coordinate frame as it moves through a space in Unity. Blue indicates the z-axis, which is the direction the camera is pointing.

The camera coordinate frame is important because it's the user's interface with the rest of the world and your AR content. The user's AR experience is entirely dependent on the accuracy of the camera pose (which we're constantly working to improve!), and as we'll discuss in Part 2 of this blog series, Unity and SceneKit both provide functions that are especially convenient when applied to the camera transformation with respect to the world frame.

Object Coordinate Frame

The third important frame to know is the coordinate frame for any AR objects in the world. Whenever we place any content in the world, we are defining the pose of the object with respect to a reference frame.

We can actually represent each object as having its own coordinate frame, usually placed at the object's center. A coordinate frame is created for every single object placed in the scene. Following up with our horse example, we see that its local coordinate frame would appear as such. The w subscript represents the world frame, while the h subscript represents the horse frame.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/5d734d66-2bb9-44a2-9c62-447b611dedc0/localcoord.png
Example of a local coordinate frame, centered at the object.

This coordinate frame is always fixed at the center of the object, but as the horse moves with respect to the world frame, the local coordinate frame will also move.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/8e0778c8-07bb-4c95-b94b-939a9df0ec88/localrotated.png
Example of a coordinate frame, centered at the translated and rotated object.

Tying it Together

To wrap up this article, we will look at a scene in Unity (left-handed convention!) with a camera and a single object. The world frame is positioned at [0, 0, 0]. The camera frame is positioned at [-3, 3, 5], and is rotated by 20 degrees in pitch, and 120 degrees in roll. The horse is positioned at [5, 1, 3], and is rotated -60 degrees in pitch.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/d88c1999-884d-4e73-8e6e-1e1a9779c06f/Screen_Shot_2020-03-05_at_13.43.04.png
Example scene in Unity, with a world coordinate frame, camera, and horse. Horse model retrieved from https://poly.google.com/view/5ocnVSh_ZF-

Viewing this scene from the camera shows us what the user would experience.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/24cb685f-997e-4839-a4b3-f3e0c7365231/Screen_Shot_2020-03-05_at_13.30.57.png
Example scene seen from the camera viewpoint.
https://s3-us-west-2.amazonaws.com/secure.notion-static.com/63102a92-b243-4d67-ac2b-dc31067a5662/Screen_Shot_2020-03-05_at_14.21.57.png
Example seen from the camera's viewpoint.

We've set the horse's pose via the inspector.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/7bbb1406-82e3-4bb2-ad5e-8f8ae869ec12/Screen_Shot_2020-03-05_at_14.12.13.png
Horse pose in Unity.

One important thing to note is that the rotation is always expressed in the local coordinate frame, and not the world frame! For example, let's say we apply 40 degrees in roll. The horse rotates about the content x-axis, and not the world x-axis. As we can see in the right hand corner, the world x-axis (red) is parallel with the z-axis (blue) of the horse.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/5ca0c8ac-c50f-47ee-86a6-f41d0fc24100/Screen_Shot_2020-03-05_at_14.16.35.png
40 degree roll applied to the horse. Note that the rotation is about the LOCAL coordinate frame, and not the world frame.

The updated inspector pose is now:

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/7b37a624-28c4-4b24-ac15-e80a22400a81/Screen_Shot_2020-03-05_at_14.17.53.png

Now, we can move the horse closer to the camera. If we set the world position to [2, 1, 3], this is the new updated scene and camera view.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/a0e95719-5bdb-4185-bf6c-85a6e7c22cad/Screen_Shot_2020-03-05_at_14.24.04.png
Updated scene with horse at [2, 1, 3], rolled 40 degrees, and pitched -60 degrees.
https://s3-us-west-2.amazonaws.com/secure.notion-static.com/b780f6ba-ae80-4f51-963e-e6e5de1a1030/Screen_Shot_2020-03-05_at_14.24.50.png
Updated scene from the view of the camera.

If you have any questions or comments, feel free to tweet me (@navganti), and join us on our Slack developer channel (placenotedevs.slack.com).

We are hiring - check out our job openings! If you or someone you know wants to help build the future of computing, reach out to us :).