lucasmoreira.

The main metric that the Consumer Onboarding team within the Growth Tribe monitors is successful access, guiding the user from purchase to the Club and the first access to a lesson (First Access) within 7 days. A user who never accesses their first product generates R$50 (~$10) less for Hotmart over 12 months. This results in a projected loss of 4M per monthly cohort. At the time of writing this document, the rate stands at 76.07%, representing a 2.47% decrease from the baseline of 80.06%.

We have considerable traffic in our app, 3M MAU (monthly active users), and there is a good opportunity to highlight users' newly purchased items. After analyzing the data in this document, we identified that, on average, over the past 12 months, 1.15M users made purchases each month and already have more than two products in their accounts. These users potentially have difficulty finding their products in the app since the carousel is sorted by the most recent access and only fits two product cards in the "My Purchases" carousel before requiring horizontal scrolling. With the addition of the Nav/Tab Bar in the app, the "My Purchases" area gained new prominence, but if a user who has just made a purchase wants to access that content, they will need to search for the desired content, which creates access friction.

There are several important differences between in-app notifications and push notifications. Here's a helpful table that summarizes the differences:
Push notifications and in-app notifications are messaging tools used by mobile apps to communicate with users. However, they differ in their delivery method, timing, and purpose.

Push notifications are messages sent to users from a server, even when the app is not open. They appear on the user's lock screen or notification center, regardless of the app they are currently using. Push notifications are typically used to send urgent alerts, reminders, or marketing messages. When we mention mobile app messages, everyone assumes we are talking about push notifications. But in fact, push notifications are essentially the common messages that app users receive even without interacting with the app. They usually appear on the screen or notification tray and alert users about any ongoing activities.

In-app notifications, on the other hand, are messages displayed within the app itself. They are visible when the app is actively being used and are used to provide contextual information, guidance, announcements, news, and updates to the user. In-app notifications can be used to highlight new features, provide exclusive access to sales, or offer personalized product recommendations.

The current ways of sending notifications, such as modals and push notifications, may have high open rates but lower click rates. In-app notifications drive user engagement and retention by sending personalized messages in a centralized location. These rich media notifications are contextually rich and persistent in the app, meaning users can return to the notifications whenever they need to. User interest is maintained through engaging and relevant messages. Sending such notifications can result in a better user experience and higher CSAT, as users receive highly relevant and engaging content throughout their customer journey.

Some negative points about push notifications include the fact that users can choose whether to receive them or disable them, thus preventing their visibility. Additionally, users may see them but save them for later, postponing their reading, or they may get mixed up with other notifications on the phone screen.

1. We believe that a reduction in the average time to first access the product

  a. will be achieved if the user who has the app installed and has just made a purchase

     i. can easily find where to access their newly purchased product within the Hotmart app

     ii. with easy access on the app's home screen for a newly made purchase

Context
We know that our content access platforms are passive and do not facilitate the discoverability of newly acquired content. For example, the "My Purchases" section in the app is sorted by the most recently accessed product only. This means that the newly purchased product will never be prominently displayed, assuming that a user is interested in accessing the product they have just purchased as quickly as possible.

App Situation
We display a maximum of 2 cards in the initial view of the "My Purchases" carousel, and 10 if scrolled sideways.

Another option for accessing purchased products is through the "My Purchases" tab in the Nav/Tab Bar. On average, each user has access to Y products in their account. In the last 30 days, X users with more than 2 products made a purchase. These X users potentially had difficulty finding their newly purchased product when accessing the app.

In demand-side analysis, we extracted the average number of purchases per user in accounts created each year, which helps justify the argument, as accounts created before 2020 already purchase more than two products per year. The number of repeat buyers is higher than that of new buyers and has been on an upward trend for years.

‍

The app has approximately 3M MAU (monthly active users), according to Firebase. The app has 7.7 million users who have opted in to receive push notifications, according to our provider, OneSignal. Push notifications in the app have an average daily CTR (click-through rate) of 2.74%, based on information provided by our contact at OneSignal.

In the past 12 months, a monthly average of 1.15 million purchases were made by buyers who have more than two products in their accounts. In a universe of 2.26 million monthly purchases on average, this means that 51% of purchases would not be easily found in the app. On average, 929,000 users are potentially impacted monthly (48% of monthly buyers).

To better understand the impact, we need to know how many of these users who made purchases use the app to consume their products. A metric that would help us is the percentage of course access in the Club web versus the app: in recent months, 30% of course access occurred in the app.

‍

Proposed Solution
The access component for the purchased product will appear on the "Home" screen and in the "My Purchases" section of the app when access to the product is available. The idea is for it to appear simultaneously with the access email and push notification triggers. Clicking on the component should take the user directly to the content within the app. The component will remain persistent on the "Home" screen and in the "My Purchases" section of the app for a set period until the user accesses the purchased product and starts consuming it. After that, it will disappear from the "Home" screen and the "My Purchases" section of the app, making way for the "Continue Watching" feature (on the Home screen) related to that content. The component cannot be closed by the user.

Step-by-step:
● Define expected behavior for each product format and distribution channel (replicate Hub behavior);
● Implement backend logic for display;
● Register the component with the copy within the app's structure;
● Provide the necessary data related to the copy.

‍

● The display of the component is enabled and should disable the banner carousel due to the screen space occupied by the carousel;
     ○ Once the user interacts with the component and reaches the first lesson, we can re-display the carousel on the home screen;
● Component display time window: initially 48 hours;
● Redirection: after the click, direct the user to the product home;
● Display the recent purchase component when the user buys:
     ○ a new online course distributed via the New Club;
          - Always display the most recent purchase; if the user buys one course after another, the most recently purchased one should be displayed;
          - If the user has not accessed the previously purchased course, we can display this purchase in the component after interaction with the most recent purchase;
● Pause trigger for component display: reaching the first lesson (and not the product page);
● App notification: The recent purchase component triggers a notification event in the navbar (red dot);
     ○ If the user enters the app directly from the home screen, the notification needs to be in the navbar and disappears when the user interacts with the component, the home listings, or any other icon in the navbar;
     ○ If the user enters the app from a push notification or other communication and lands on a screen other than the home screen, the notification needs to be displayed in the navbar and disappears when the user interacts with the home icon in the navbar;
     ○ Removing the notification from the navbar should not interfere with the banner display; the functions are separate, and the banner display rules should follow what is described above;
● In some scenarios, the banner and the first product in the listing will be the same, and the display of the last purchase in the banner and home listings should happen normally; we will track through instrumentation which has more interaction to define future opportunities;
● When the user clicks back, we return the user either to the app home or to the My Purchases listing (evaluate possibilities and implementation effort).

‍

Experiments
We can conduct various experiments after development, all described in the document. Below, in order of execution, are the proposed tests.

Without Component vs. Fixed Component

Format: A/B test
Group A: without component
Group B: with component, keeping the banner
Objective: validate if the feature changes our rate of access to a first lesson within 7 days
Implementation:
Metrics:
● Adoption
● feature adoption rate (in percentage)
● total views vs. total clicks
● drop-off rate (in percentage) - number of users who used the feature vs. number of users who continued to use the feature
Conversion Funnel: click funnel from the feature to watching the first lesson
● feature view
● feature click
● product home view
● lesson click
● lesson view

Display Rules
1. Display the recent purchase component when the user buys a new online course distributed via the New Club;
2. Always display the most recent purchase; if the user buys one course after another, display the most recently purchased one;
3. Bundle: display purchased courses randomly, one after the other (the simplest way to display due to how we process access information);
4. Order bump: display purchased courses randomly, one after the other (the simplest way to display due to how we process access information);
5. Import: display the online course as soon as the consumer receives access via import;
6. Subscriptions: display the online course only the first time the consumer subscribes to the product; if they renew, do not consider it;
7. Redirection: after the click, direct the user to the product home;
  a. when the user clicks back, return them to the app home (from where they came after interacting with the recent purchase component);
8. Pause trigger for component display: reached the first lesson (not the product page);
  a. we have an event triggered when the user accesses the first lesson, implemented by the LEX team;
9. Component display time window: initially 48 hours (if the consumer does not reach the first lesson);
10. The recent purchase component and the first card in the "My Purchases" listing will display the same product (when online course/new club). We will track through instrumentation which one has more interaction to define future opportunities;
11. If it's a creator, display the recent purchase component in the same way for courses purchased by them, distributed via new clubs from other producers.
‍
Consolidation - Recent Purchase Component Test v1
Test: Optimizely
Test execution period (start/end): 03 to 09/04/2024
Results:
● 99% statistical significance for the recent purchase component
● 13.78% higher conversion rate from access to the first lesson compared to those who did not receive the component
● The flows for reaching the first lesson via my clubs and my purchases decreased compared to the group that did not receive the component because its function is precisely to highlight the user's most recent online course/new club purchase and prompt quicker interaction.

‍

Floating Component vs. Fixed Component
‍
Format: A/B test
Group A: fixed component
Group B: floating component
Objective: test if the component on the user's screen would bring more engagement and if reaching the first lesson would be easier

Display Rules
1. Display the recent purchase component when the user buys a new online course distributed via the New Club;
2. Always display the most recent purchase; if the user buys one course after another, display the most recently purchased one;
3. Bundle: display purchased courses randomly, one after the other (the simplest way to display due to how we process access information);
4. Order bump: display purchased courses randomly, one after the other (the simplest way to display due to how we process access information);
5. Import: display the online course as soon as the consumer receives access via import;
6. Subscriptions: display the online course only the first time the consumer subscribes to the product; if they renew, do not consider it
7. Redirection: after the click, direct the user to the product home;
  a. when the user clicks back, return them to the app home (from where they came after interacting with the recent purchase component);
8. Pause trigger for component display: reached the first lesson (not the product page);
  a. we have an event triggered when the user accesses the first lesson, implemented by the LEX team;
9. Component display time window: initially 48 hours (if the consumer does not reach the first lesson);
10. The recent purchase component and the first card in the "My Purchases" listing will display the same product (when online course/new club). We will track through instrumentation which one has more interaction to define future opportunities;
11. If it's a creator, display the recent purchase component in the same way for courses purchased by them, distributed via new clubs from other producers.

Consolidation - Recent Purchase Component Test v2
Test: Optimizely
Test execution period (start/end): 23/04 to 06/05/2024
Android: 23/04/2024
iOS: 29/03/2024
Results:
● 63% statistical significance for the floating recent purchase component, we did not achieve significance even with an extended testing period;
● We did not see an improvement in reaching the first lesson compared to the group that received the fixed version;
● The close button on the floating component had high interaction, with 40% of users who received the modal interacting with it;
● The flows for reaching the first lesson via "My Clubs" and "My Purchases" increased compared to the group that received the fixed component, which can be explained by users not interacting with the floating purchase component and closing the modal, thereby reaching the purchased product through traditional means of the app (listings and navbar).

‍

After analyzing the two tests conducted above, it became clear to us that the fixed recent purchase component performed better than the floating recent purchase component.

The data from the first test (v1) showed how having the recent purchase component on the home screen benefits the user in reaching the first lesson, positively impacting our main KR.

The data from the second test (v2) indicated that with the ability to close (x) the floating component, users did not complete the JTBD of reaching the first lesson.

Additionally, in the second test, we observed that accesses from the home screen to Club and Purchases, as well as via the navbar "My Purchases," increased. This is another indication that the floating component, being possible to "close," was not as effective for us, causing users to navigate to their product/lesson through other paths.

Therefore, we will permanently implement the fixed recent purchase component for all app users, on both platforms, for all New Club purchases.

‍

Product Vision, Project Management, UX Design, Research, Interaction Design, Prototyping, Usability Testing, Visual Design.

‍

‍