[Unity] Create your own interactive AR Android program

1 Create an AR scene

Following the creation of an AR facial special effect Android program in the previous article , this time create an interactive AR program of your own. Also use the previous project, use Ctrl+N to create a new Scene (or File→New Scene) under the Scene folder of Assets, select the AR scene, and click Create to create:

2 Configure AR Camera as the front camera

Since the face is also recognized here, it is also necessary to use the front camera of the mobile device

1. Find the AR Session Origin object in the Hierarchy, expand its collapsed object and find the AR Camera object.
2. Find the AR Camera Manager component in the Inspector panel of the AR Camera, and set the Facing Direction property to User:
   

3 Configure the packaging scene

Select File → Build Settings from the menu bar, add the newly created scene to the packaging list, and hide or delete the previous scene:

4 Download the official InteractiveFaceFilterAssets resource

The zip download location is here , click to download.
After downloading and decompressing, you can import InteractiveFaceFilter.unitypackage into the Assets folder of the project.

Import method:

In the folder where the zip file is decompressed, find the unitypackage in it, drag it to the Assets folder of Unity, and then a window for importing resources will pop up, click Import in the lower right corner to import.

Imported resources:

• Meshes: There are five glasses models with different styles.
• Prefabs: AR_Canvas prefab, which is the UI for interaction.
• Resources: This contains UI templates in PSD format for creating our own.
• Sprites: Contains the UI for creating user interfaces.

5 Configure AR Face Manager

In this step, we will use AR Face Manager to tell AR Session Origin what special effects will be displayed on the face when it is detected.

1. Delete the Cube object in the Hierarchy, it is just for testing.

2. Select the AR Session Origin object, search for and add the AR Face Manager component in its Inspector window, which is used to detect faces:
   

6 Create the glasses prefab

In this step, create a prefab for all available glasses, called "GlassesGroup", for the user to select glasses.

1. Find _InteractiveFaceFilter → Meshes in the Assets folder, drag the three pairs of glasses you want to install on the mobile device to the Hierarchy window, and set their positions to (0,0,0) in the Inspector panel.
2. With the three pairs of glasses selected, right-click Create Empty Parent to create an empty parent object for the three pairs of glasses, and name the empty object "GlassesGroup".
   

7 Set up AR face tracking

In order to make the glasses appear exactly where the eyes are, we need to make the GlassesGroup into an AR Face prefab.

1. Add the AR Face component to the GlassesGroup in the Inspector panel for the GlassesGroup to track the movement of the face:
   

2. Drag the GlassesGroup in the Hierarchy to Assets → _InteractiveFaceFilter → Prefabs to make a prefab:
   

3. Drag the prepared prefab to the Face Prefab of the AR Face Manager component of the AR Session Origin object:
   

8 Test results

8.1 Testing in Unity

First make sure that the Game view is adapted to the 16:9 resolution of the mobile phone, then return to the Scene window, select the GlassesGroup object in the Hierarchy window, and change its position on the Z axis to 0.4 so that it can be displayed in the center of the Game window:

Similarly, in order to make these glasses appear in the center of the field of view of the mobile device, the PositionZ axis of the GlassesGroup prefab needs to be modified to 0.4:

8.2 Testing on an Android device

If you are testing in Unity, use the GlassesGroup prefab in the Hierarchy; and if you want to test on an Android mobile device, you need to hide the GlassesGroup prefab in the Hierarchy.

1. Hide the GlassesGroup prefab in the Hierarchy.
2. Refer to Section 3.2 of this article for project configuration .
3. Ctrl+S to save the scene, menu bar File → Build Settings, select Buid to package apk, export to Android device for testing.
4. After the program opens and detects a face, all three glasses will be displayed on the face.

9 Add your own ideas based on this AR scene

9.1 Changing the size of the glasses model

The size of the glasses model in the project is based on the general human face. We can modify the size ratio of the glasses or rotate them to make them more interesting.

9.2 Import a new model from the Internet

There are many models of glasses on websites such as Sketchfab , which can be downloaded and imported into the project. In terms of model selection, there are three general considerations:

• model format. Make sure that the format of the model is supported by Unity, such as FBX format.
• Model complexity: It is best not to find a model that is too complex and has too many faces, which is not friendly to mobile optimization.
• Licensing of the model: Note that the model will have a CC license, which will specify what the artist is allowed to use in it. See here for details on the CC license .
  Use this steampunk glasses resource as in , by Kisielev Mikhail, licensed under Creative Commons Attribution .

9.3 Make your own model

Here are a few things to consider when making your own glasses models:

• Only the front of the glasses is included : We always see all parts of the glasses model, so when the user turns their head, the temples may overlap the ears, breaking the immersion. We could just model to where the glasses would go around the ears, but if the user's face was a little longer, the model might not touch their ears.
• Keep the face count of the model in a low range : Since the final project will be used on mobile devices, the model should be as optimized as possible.
• Limit the model to one material : Since we will implement the ability to change the color of the glasses later, we need to limit them to one material.
• The origin and direction of the model : The origin of the glasses model we created needs to be at the origin of the system, and the orientation of the glasses needs to be aligned with the AR face. The Face_Reference_Mesh model can be imported as a reference.

10 Configure the UI interface

10.1 Add AR_Canvas prefab

The project uses the UGUI system, that is, all UIs are managed using Canvas components, which are all sub-objects of Canvas.

1. Find the AR_Canvas prefab under _InteractiveFaceFilter → Prefabs in Assets, and drag it to the Hierarchy window.
2. You can view the UI display effect in the Game window:
   
   AR_Canvas is rendered with Screen Space as Overlay, which means that it is always rendered on top of other objects and can be accurately displayed in a fixed position in the Game window.
3. AR_Canvas consists of 3 sub-objects: Glasses Buttons and Materials Buttons are used to control the switching of glasses and glasses materials respectively. The EventSystem is used to detect user input.
   

10.2 Custom glasses button

Now the glasses switching buttons are represented in the form of circles. In order to allow users to more intuitively distinguish which button switches to which pair of glasses, the circle needs to be replaced with a specific glasses icon.

Unity uses a sprite sheet to store an atlas, which can be sliced ​​so that the images can be used in Unity alone:

1. Find the UI_Icon_Glasses atlas under _InteractiveFaceFilter → Sprites in Assets, click ">" next to it to expand it, and you can observe the sliced ​​Sprites.
   
2. Map the glasses icon to the glasses button. Use the drag and drop sprite to the Source Image under the Image component of the button to achieve:
   

10.3 Custom Material Buttons

In order to make the glasses more beautiful, different material (color) buttons need to be defined here:

1. Expand the Materials Buttons in the Hierarchy window, find the three sub-objects under it, and modify the Color under the Image component respectively.
2. The final effect is as follows:
   

10.4 Other Attempts

We can also try some of the following:

• Move the button to other positions in the user interface such as the top: modify the Rect Transform component of the icon component or directly use the shortcut key W to move to a suitable position.
  

• Make your own glasses icon: If you are using your own glasses model, you can make your own glasses icon by modifying the PSD file UI_Icon_Glasses_Template

11 Interaction with Visual Scripting

11.1 Introduction to Visual Scripting

We can create logical functions for our projects using Visual Scripting, which is a way of visualizing logic.

The following left and right diagrams are the same logical expression:

Visual Scripting also provides non-coders with access to the entire Unity Scripting application programming interface (API). This API contains a complete set of definitions of classes, events, methods (behaviors) and properties (settings) that can be manipulated in Unity scripts.

11.2 Creating a graph

Visual scripts are stored in graphs, which can be applied to any GameObject in the form of components.
1. Create an empty object in Create Empty in the Hierarchy, make sure its coordinates are at (0,0,0), and name it "Visual Scripts".
2. Add the "Script Machine" component to the Inspector window of "Visual Scripts".

The "Script Machine" component is a tool for running visual scripts.

3. Select "New" to create a new Graph, and name it "MeshChanger" and save it under the newly created "Visual Scripts" folder.
4. It can be observed that the newly created "MeshChanger" graph appears in the Graph parameter of the "Script Machine" component, and two new parameters appear: Title and Summary. These two parameters can be used to distinguish different "Script Machines".
   
5. Modify the Title and Summary as follows to indicate the purpose of the visual script:
   

11.3 Edit Visualization Script

11.3.1 Click the button console output information

1. Click Edit Graph in the Script Machine component to open the visual scripting workspace. If it is the first time to open, a dialog box will pop up, just select Change now.
   

   And a new GameObject called VisualScripting SceneVariables will be automatically generated in the Hierarchy.

2. Nodes are the components used to build logic in the graph, and there are several different types of nodes. Whenever a new graph is created, it will load the On Start and On Update nodes.
   
   On the right side of the Script Graph window is the Graph Editor, where the graphics logic is established; on the left side of the window is the Graph Inspector, which will show us the details of the selected object; under the Graph Inspector window is the Blackboard, where variables are created.

3. Operations in Graph Editor:

   • Press and hold the middle mouse button to drag the view to control the movement;
   • Hold down the Ctrl key and scroll the mouse wheel to zoom the view.
   • Right-click to create a new node.

4. Select the On Start node, and the Inspector window on the left lists that the node has an Output called "Trigger". The inputs (and outputs) of the flow (Flow) determine the logical flow of the graph.
   

5. The functions we need to implement here do not need to use the On Start and On Update nodes. Therefore, after selecting them, right-click and select Delete to delete or directly press the Delete key to delete.

6. Right-click to create an On Button Click node or click Events > GUI > On Button Click to create:
   
   Unlike the previous On Start and On Update nodes, On Button Click has an Inputs section.

   There are two types of inputs: control inputs and data inputs.

   • The control input controls the logic flow of the graph.
   • Data inputs receive values ​​of a specific type from other nodes that the receiving node needs to perform its task.
    

   The On Button Click node only executes the event when the button is pressed, so it needs to know which button it should monitor.

   In the Graph Inspector window, the input of the On Button Click node is looking for a GameObject target, and this GameObject is the input button that the On Button Click node will monitor.

   By default, the On Button Click node has a parameter labeled This, which is where the target input will be connected. When no input node is connected, the node has a safeguard: it will look for the target input on the GameObject it is assigned to.

   In this case, if the parameter is left blank, the On Button Click node will look for a Button component on the Visual Scripts GameObject attached to the Script Machine we created. This is fine if the script image is attached to a button in the AR_Canvas;

   But if the Graph is on different GameObjects, then you need to use variables to control it.

   Variables are containers for different kinds of data. These data can be numbers, GameObjects, sounds, or anything else. Variables can be generated in the graph itself, or as references to other objects in the application. In this example, we will create a variable that will reference a button on the user interface.

7. Create a variable. Variables are created in Blackboard, and there are some tabs at the top of Blackboard as follows:
   

   • Graph
   • Object
   • Scene
   • App
   • Saved
    

   These labels are different categorical variables that we can create. Since we need to create a variable that references the GameObject in the scene, we will create an Object variable.

8. First, select the Object label, enter "glasses1Button" and click the "+" sign or press "Enter" to add a variable; then, set the type of the variable (Type) to Button, and select Glasses1Button in the Hierarchy for its Value parameter; finally, set Drag the "glasses1Button" variable created in Blackboard to the Graph to become a Get Variable node.
   
   
   The Get Variable node of Glasses1Button also has a This parameter, which, like On Button Click, is a backup option when the return value of Get Variable is not filled. But since this node is defined as glasses1Button, there is no need to have a return value.

9. Connect the Get Variable node and the On Button Click node, so as to realize the detection of the event that the Glasses1Button button is pressed.
   
   Observe that the This value on the On Button Click node disappears because it now accepts the glasses1Button variable.
   After this connection, click Play to run the project and you will find that when you click the first glasses icon, the On Button Click node is flashing, which means that it has received the button click.

10. To output a message in the Console window after clicking the button, you can add a Debug Log (Message) node and a String Literal node in the Graph to output a text message.
    
    Among them, the function of the Debug Log (Message) node is to output a message to the Unity console. This node has a Flow and Object input, and the content of the Object input is a string or object converted into a string representation, which is convenient for display. So you need to create a String Literal (string) node.
    Connect the Flow output on the right side of the On Button Click node to the Flow input on the left side of the Debug Log node, then connect the output of the String Literal node that says "Button 1 pressed!" to the Message input on the left side of the Debug Log node.
    
    After running, you can find that the console outputs "Button 1 pressed!" after detecting that the button is pressed

In addition, you can try to use the Get Name, String, and Concat nodes to get the GameObject name and splicing the strings to output:

11.3.2 Click the button to trigger the display model event

1. In order to achieve the effect of the glasses model appearing when the button is clicked, the Mesh Renderer component of the glasses model needs to be disabled first. This component is used to manage the visibility of a GameObject.
   

2. In Unity, there are several ways to quickly locate a specified GameObject in a script. One of the methods is to assign a unique tag, the label, to the GameObject. Here, a unique label is added to each of the three glasses to better identify and distinguish them.
   

3. In the Hierarchy window, select the GlassesGroup object, select the drop-down menu to the right of Overrides in its Inspector window and click Apply All to save the label settings to the prefab.
   

4. Update the script graph, in order to identify the game object through the tag, you need to use the Find With Tag node.
   
   You can see that this node returns an active GameObject in the scene. Since we just disabled the mesh renderers of the glasses model before, they are still in the active state, so they can be found using Find With Tag. This node requires a data (Flow) input, which is the name of the tag to look for.

5. Connect the Data (Flow) output of the On Button Click node to the Data (Flow) input of the Find with Tag node.
   

6. Use the Mesh Renderer: Set Enabled node to realize the function of the user clicking the button to display the glasses model.
   
   The details of this node are as follows:
   

   It can be observed that two pieces of data are required to use this node:

   • Target: the specific mesh renderer that the node will ring
   • Value: The mesh renderer is enabled or disabled. If the Bool value is set to true (enabled), then the specified Mesh Renderer will be enabled. If the Bool value is set to false (disabled), then the Mesh Renderer will be disabled.

7. Connect the data (Flow) output of the Find with Tag node to the input of the Set Enabled node, connect the result output of the Find with Tag node to the input of the Set Enabled node, and tick the Bool value. This means that when the button is clicked, the glasses model for the specific tag will be displayed.
   
   When you click Play Test, you will find that you click the first button, and the glasses displayed on the button will also appear in the scene.

8. Copy two prepared script graphs, add two Object variables, drag the remaining two buttons to their Value, and modify the corresponding value on the Get Variable node. Then modify the Tag value of the Find with Tag node to the glasses-specific tag set before.
   
   
   When you click the Play test, you will find that the glasses appear in the scene one by one after clicking the button, but they are superimposed, because the function to hide them has not been added yet.

Then it can be packaged into apk and run the test on the Android device. Note that the GlassesGroup in the Hierarchy needs to be disabled.

11.3.3 Click the button to trigger the switching model event

Since the glasses model appeared superimposed in the previous section, it is necessary to use the Mesh Renderer: Set Enabled node to disable the visibility of the glasses model.

1. Copy the Find With Tag node and Set Enabled node of the three buttons in the script graph, and uncheck the Set Enabled node.
   

2. Modify the script graph so that when the user clicks any button, all glasses models are not displayed. Since Visual Script runs very fast, the hidden glasses that run sequentially appear to be hidden at the same time.
   

3. Next, you need to put the above connected nodes into the process indicated by the arrow in the figure below, because all the glasses models need to be hidden before the corresponding glasses models are displayed.
   
   But it can be seen that the sequence in the red box above is relatively long, so it needs to be converted into a custom event, so that we can use it as a single node to call the event.

4. Create a new Custom Event node, name it "DisableRenderers", and connect its right output stream to the left input stream of the above sequence.
   

   It can be observed from the graph Inspector panel that the input stream of the Custom Event node can be GameObject or String.
   
   Here a string "DisableRenderers" is used to name the custom event, which is then used to call the event.

5. Use the Trigger Custom Event node to call the above custom event. Create a new Trigger Custom Event node and duplicate two. Right-click on the node's output stream or input stream to disconnect and insert the node after the OnButtonClick node.
   
   We can think of the Trigger Custom Event node as a link (link), when this node is called, it will start the sequence of nodes in its associated custom event.
   

Click Play to run the test in the editor, and you can find that when you click the corresponding button, the corresponding model will appear, and it will not appear superimposed.

Then it can be packaged into apk and run the test on the Android device. Note that the GlassesGroup in the Hierarchy needs to be disabled.

11.3.4 Click the button to trigger the switch model color event

When using script graph to add logic functions to GameObject, it is best to implement a function with a graph, so that the graph will not be messy.

1. In the Hierarchy panel, select the Visual Scripts object, and click Add Component in its Inspector panel to add a "Script Machine" component to it. Then, click New, name the Visual script "Material Changer", and fill in "Material Changer" in its Title column, and fill in "Changes the material assigned to all of the glasses when the material buttons are selected .” indicates that the visual script is used to click the button to change the material of the glasses model.
   

2. Similarly, in this function, we don't need to use the On Start node and the On Update node, so select them and press the delete key to delete them. First, create a new On Button Click node, and set an Object variable for it in the blackboard to obtain the change material button.
   
   

3. Create a new folder in the _InteractiveFaceFilter folder under Assets and name it "Materials". Create three color materials corresponding to the button colors, and name them "Material1", "Material2" and "Material3".
   

4. In blackboard, create three variables of Object type to refer to three materials, their type is Material, and name them "Material1", "Material2" and "Material3".
   

5. To change the color of the glasses model, you need to get the glasses model, and then modify its material. Therefore, you need to use the Find With Tag node to obtain the corresponding glasses model with a specific tag.
   

6. Use the Mesh Renderer: Set Material node to change the material of the glasses model.
   

The Set Material node has two inputs:

• Target: A reference to the GameObject it is changing.
• Value: The material to apply.
  

Among them, Target uses the glasses model obtained from the Find With Tag node, and Value uses the variables of the corresponding material:

7. Duplicate two Find With Tag nodes and Set Material nodes, and modify the tags in them to the tags of the other two pairs of glasses. Connect as shown in the figure below, so that the first material can be applied to three pairs of glasses at the same time:
   
   Click Play to test, and you can find that the first material can be assigned to three pairs of glasses:
   

In the same way, the other two color material assignment functions can be added to the script graph. However, a repeated node sequence is actually used here, so the next section uses a custom event with parameters to further add the functions of the other two materials.

11.3.5 Create a custom event and call it with parameters

1. First, you need to separate all "Find With Tag" nodes and "Set Material" nodes into Custom Events, so that they can be called by calling events when each material modification button is pressed.
   

2. Arguments can be passed into events or functions to change the outcome of the event or the output of the function. Since we need to apply three different materials to the glasses model, it depends on which button was clicked. Add a data output for the Custom Event node, called a parameter, to pass different data to those Set Material nodes.
   

3. Calling the defined ChangeMaterial event requires the use of the Trigger Custom Event node, which also requires a parameter.
   
   At this time, click Play to test, and also assign the first material to the three pairs of glasses.

4. Duplicate the node in step 3 twice, and modify the buttons and materials in the two Get Variable nodes to the other two buttons and materials respectively.
   

   Hit the Play button to test in the editor, all the buttons work.
   

Then, you can disable the GlassesGroup object in the Hierarchy and package it on an Android device for testing. It can be observed that the glasses can follow the movement of the face and move.

11.3.6 Load the user's choice

In the test in the previous section, it can be found that once the user's head leaves the screen and then returns to view, the previous glasses will not reappear on the face. This is because we did not save the user's choice. After the user's face leaves the screen, AR Face Manager will delete the GlassesGroup component; and when the user's face returns to the screen again, AR Face Manager will re-instantiate a GlassesGroup prefab, which Only the default settings are configured, and the material selected by the previous user is not applied to the glasses model.

1. First, you need to ensure that the GlassesGroup object in the Hierarchy is enabled, then edit the MeshChanger visualization script, create a String type variable (Object) named "currentGlassesTag" in the blackboard to store the glasses tag currently selected by the user, and display it in the graph Find the Set Variable node for the "currentGlassesTag" variable by searching for "Set currentGlassesTag".
   
   

2. Since the value of currentGlassesTag needs to change according to the button selected by the user, we need to create a custom event called "SaveTag" so that it can be called by the three buttons, and it has a parameter for outputting the obtained tag. Add another node to get the object Tag, which is used to get the tag of the current model.
   

3. Use the Trigger Custom Even node to call the SaveTag event of the tag, change the number of parameters to 1, and make the following connection:
   

4. In the previous step, we got a reference to which pair of glasses the current user has selected. Therefore, in this step we will apply the glasses selected by the user to the GroupGlasses when the new GroupGlasses prefab is instantiated.
   Here, create another Graph named "FilterUpdater" on the Script Machine, and fill in "Filter Updater" in its Title column, and fill in "Reloads the last worn pair of glasses and selected material after a user's face is lost and rediscovered by AR Foundation", which means that the previously selected glasses and materials are reloaded after the user's face moves away from the camera.
   

5. First delete the On Start node and keep the On Update node, because we need to detect whether the user's face is within the detection range of the camera in real time. The On Update node will run continuously while the application is running. The function of the FilterUpdater graph is to obtain the glasses model currently selected by the user through the currentGlassesTag variable and enable its Mesh Renderer.
   

6. You can test the functionality of the above script in the editor in this way:

   • Click the Play button to run the program, and you will find that the Find With Tag node in the FilterUpdater graph turns red, because the application is looking for an object with a specific tag in real time, but the tag is still empty.
   • Therefore, in the Game window, select a glasses button arbitrarily, so that the currentGlassesTag variable will have a value.
   • Next, in the Hierarchy window, select the GlassesGroup prefab and delete it.
   • Finally, in the Project window, re-drag the GlassesGroup prefab to the Hierarchy window under Assets → _InteractiveFaceFilter → Prefabs, and you will find that the previously selected glasses model appears again.

7. In the same way, choose to edit the MaterialChanger graph, create a Material type variable (Object) named "currentMaterial" in blackboard to store the glasses material currently selected by the user, and find "currentMaterial" by searching "Set currentMaterial" in the graph Variable's Set Variable node.
   
   

8. Since the Cutom Event of ChangeMaterial is used to transfer the material selected by the user, it can be obtained directly by using the Set Variable node:
   

9. Finally, back to the FilterUpdater diagram, use the Mesh Renderer Set Material node to update in real time to assign the material selected by the user to the glasses model:
   

10. If you click Play to test at this time, you will find that when a pair of glasses is selected, the material on the glasses is missing. This is because the user did not select the material of the glasses at the beginning, resulting in the value of the currentMaterial variable being empty. Therefore, an initial value needs to be set for the currentMaterial variable.
    

11. Find the currentMaterial variable in blackboard, and set its Value to Lambert1, which is the gray material by default.
    

12. After clicking Play Test again, you will find that everything is displayed normally. Next, you can disable the GlassesGroup prefab in the Hierarchy and package it on an Android device for testing.