Linking to Other Games in App

Project Context

Did you know NYT Games has games other than the Daily Crossword? Currently, the NYT Crossword App only provides experience to three games: The Crossword, The Mini, and Spelling Bee. Due to tracking limitation, the Games Team doesn’t have insight into how users interact with expansion games outside of the app like Tiles, Vertex, Letter Boxed, and Sudoku. As a result, we wanted to understand how providing a clear path to more games will impact users’ engagement with the app and games. For an AB test, we implemented a row of expansion games (”More Games”) to see that impact. 

*The design implementation is not permanent within the app; it was used for an AB test on the actual app.

Timeline ⌛

7 weeks – Fall 2020

Design Tools ✏️

Illustrator
Figma

Role ✨

I was responsible for UI and UX for this project. I received feedback from lead designers and stakeholders from the team.




︎ PROBLEM

Games team doesn’t have much insight on how users interact with the game products like Tiles, Vertex, and Sudoku outside the app

Close to my summer internship being over, I joined Discovery Squad of the games team, which has a meaningful mission of creating an intutive experience for users to discover games they love. One of the projects I was assigned to was Linking to Other Games in Apps.

As you can see (below), the current crossword app doesn’t provide native app experiences to other games like Vertex, Tiles, Letter Boxed, or Sudoku. We wanted to determine how giving a clear path to these games, links to expansion games in web browser, will impact users’ engagement.




What’s the best design that fits with user’s mental model? / How does presenting more games impact user engagement?



︎ SOLUTION

More Games row with individual game cards was designed to test user engagement. 

For a 2 week AB test, a row of expnasion games like Spelling Bee, Tiles, Vetex, and Sudoku (More Games row) was implemented to find out the impact of providing more games. With the exception of Spelling Bee card which has a native app experience, the colorful individual game cards bring users to the web browser of the games.

*This design implementation is not permanent within the app; it was used for an AB test.  

Control (left) | Variant (right)

In this case, user clicks on the Tiles Card. It brings him/her into the web browser of the game.
iOS
iOS

More Games row was carried out on both android screens and ipad screens.


Android (left) and Tablet (right)


00. CONTEXT

Providing a link to our web games directly from the app, we will increase awareness of these games, allow easier access to find them, and will increase game play.

Is the statement above true? We wouldn’t know until we have a strong validation. Therefore, the goal of this project was to find out if this hypothesis is true/false through the design I had to create. In order to be on the same page with everyone, PM, engineers, and designers came together to go over the brief of the project and important touchpoints to keep in mind throughout the project.

00a. GOALS

There were three goals I wanted to accomplish through the design.

(1) How can we give users a clear path to new games?
(2) How can we keep new users interested in our expansion games?
(3) How can we help users have a clear understanding of our expansion games?
00b. IMPACT

How does this project impact the team on a holistic level?︎✨

Right now, the NYT crossword app is very crossword centric. Although there are more games outside of the app, it only includes The Crossword, The Mini, and Spelling Bee. In addition, the app is still called “New York Times Crossword”. This project will possibly inform an app strategy in the future. (i.e. Is it more effective to include all games in one app or create separate apps for each game? Is it most effective if everything stays the same?)
01. RESEARCH

1. How are NYT games presented outside of the Crossword app?

I looked into places like the marketing emails or nytimes.com/crossword (the Hub) in order to analyze what visual elements are needed to trigger users to play the games and understand what the games are.

2. In what use cases will users see this design implementation?

Although we are focusing on new users who are less familar with expansion games, there are still users who are habituated to the app to play the Mini (free game). I wanted to clarify these scenarios to make sure my designs consider different user groups. 

3. When do people feel okay about opening up a web browser?

The risk of the project is that we’re bringing users to the web browser. It is possible for users to get annoyed with this interaction, so I looked into other platforms where they take users out of the app into a web browser.

03. IDEATION/SKETCHES

How can links to web games be visually represented on the play screen of the app?

Based on the use cases from the research, I started to sketch out different ways in which more games could be displayed within the app. Asking the questions (below), I took four different directions

At this stage, I also thought about where these designs will take users to: Should the design take them to the Hub where all the game cards are or indivudal web browser of the games?

04. FEEDBACK

I had opportunities to present initial sketches and research to designers and stakeholders in games team. 

These are the three most important feedback/suggestions I received:

︎ Clarity/Prominence: Because this is a test to see the impact of giving clear path to more games, it would make sense to have clarity with the design and prominence with where the design is located. However, it is worth user testing if users who are habituated to the app are disrupted by the prominent placement of the design.

︎ Placement of the design: Take into consideration where these designs will be in the play screen. Will the design be above the fold so users can see it right away when they open the app? Will the design be on bottom of the screen?

︎ Overall: The team as a whole was most drawn to the designs using the cards and messaging. Based on the timeline of project, ephemeral design could take up too much time figuring out when the design will be displayed. The designs using the arrow is too integrated within the app, which will be difficult for users to notice the change. 
05. MID-FIDELITY

Based on the feedback, I chose three design directions that could be explored in different ways.

After the design review, my manager and I discussed on what design directions could be further explored; we finalized our directions with one card, individual cards, and messaging.

In order to get most out of user testing, it is important to think about usability concerns before testing.

For example, for V3: messaging that is recognizable when scrolling to the very bottom of the play screen, we could predict that people might not find the design. However, we still want to find out how users react to it and validate our hypothesis. At this stage, I had to call strong attention to what we are trying to learn for each mockup version for user testing. 

Below are the final mid-fidelity mockups prepared for user testing:
Click on the photo to view the mockups in detail

06. USER TESTING

Collaborating with a user researcher, I wanted to find out which design is the most effective and clear.

This was the most exciting part of the project! After communicating with the user researcher about what we want to learn from user testing, I formed a list of possible questions that could be asked during testing. She helped me narrow down the questions and created a test on usertesting.com.

We tested three variants with 15 people: 8 people familiar and 7 people unfamiliar with NYT Games. 

*Want to know more about how the test was run? Spare two minutes to go through these quick slides︎︎︎.
07. USER TESTING RESULTS

Did we validate the hypothesis?

Remember this hypothesis? Providing a link to our web games directly from the app, we will increase awareness of these games, allow easier access to find them, and will increase game play. Yes, we did validate the hypothesis. Seeing a breath of new games, users were interested in the new games and were willing to play them. 

Every user prefered design two. Why?

(1) Convenience: V2 directly launches individual games.

(2) Clarity: Users liked how there is a clear indication that they were going to be taken out of the app to the web browser.

(3) Color: Users were really drawn to the colors of the cards.

︎ Why not V1/V3? : There was less clarity on where users are going to be taken to. For V1 (one card), users thought they were going to be taken to the app store. V3 felt like an advertisement to them. Also, users could not naturally find V1 and V3 because of their location within the app. 

I finally decided to move on with V2: Individual Games!

08. AFTER USER TESTING: CHANGES

There are still improvements to make after user testing.

After checking with the brand designer with typographic treatment and spacing, I finalized the changes on both iOS and Android screens.

Collaboration with copy team

After user testing, I also had a chance to revisit and finalize the two word description with the help from copy writers in marketing team. I described the context of the project, what message we want to convey through the words, and character limits.

There were two goals I wanted for the two words descriptions:

(1) Have enough clarity for users to understand what each game is about

(2) Convey the characteristics of each game in a playful, dynamic, and delightful tone. It should approachable and inviting to encourage all users to try one of our games. 


Did you notice something different?
Letter Boxed has been taken out from the list. Why?


After discussing with the product manager, we decided to take out Letter Boxed from the New Games row.

(1) It raises questions to Apple guideline reviewers about taking users out to the browser to purchase Letter Boxed.

(2) We haven’t yet metered Letter Boxed for non-subs to experience the game.


09. FINAL MOCKUPS

These are the final mockups delivered to the engineers. Now, it’s ready for an AB test on the actual app!

Even after the final deliverables were sent, I constantly communicated with engineers to make sure there are no problems when the design was built along the way. Like mentioned before (Go to Top), AB test has a control variant, what the app looks like now, and a test variant with expansion game cards. Half of the users will be allocated to the variant with the control, and the other half will be allocated to the test variant.  
10. SUCCESS METRICS

Did we meet the goals?

The primary metric of this project is the number of new users who click an expansion game link as a percentage of total users. There are also secondary metrics to measure the success of AB test: incremental lift in WAU (weekly active users) of individual expansion games, referrers from the app to expansion games, the number of sessions that include a click on the expansion game cards. The countermetric to make sure we’re not overly optimizing is the WAU of the daily Crossword (puzzle) across all platforms. 

︎It’s being launched soon! We don’t have the data yet.
REFLECTION

Overall, I felt really grateful for getting the responsibility to work on this project. First, I learned how to cross-collaborate and communicate with stakeholders like PMs, user researchers, and engineers. It was critical to share my work throughout the process and stay on the same page with others so stakeholders could bring their point of view to reflect on my design. For example, a design with good usability I strongly support might not be effective in the perspective of a product manager who thinks about business metrics or an engineer who thinks about technical feasibility. I also learned the significance of system thinking. Thinking about different use cases and how my design will impact each user group helped me broaden my perspective and approach problem in various ways. Also, reflecting on how my design work impacts the larger ecosystem of NYT game users and the team as whole was meaningful.

Special Thanks to 🎉:

Product Manager: Cydney Mitchell
Designers: Caroline Oh (Manager), Deanna Destefano, Andrew Dore, Robert Vinluan 
Software Engineers: Danny Wu and Jason Mayer
Data Analyst: Dan Lim
Marketing: Loren Chidoni , Chelsea Athron, Joshua Levi
Brand: Jason Fujikuni