Get alert on abnormal trends in survey data

Revamping TikTok's COP to enhance monitoring and alerting capabilities

Timeline

Device

Collaboration

Responsibilities

Aug. - Oct. 2024
Web App
PM: Riley Jin
R&D: Chao Li
Research, User flow,  
Design iteration, Prototyping

Project Overview

COP is an internal tool for PM and operation to monitor in-app survey data, which provide one-time exploratory surveys, AA(Abusive Activity) monitoring and observation tools, data collection for algo modeling and optimization.

This project focuses on optimizing the data card modules of the survey platform to support high-frequency data analytics, real-time monitoring, and early warning systems for both individual and global surveys. The goal is to improve data processing efficiency, enhance visualization capabilities, and ensure timely detection of critical trends and anomalies.

What I design

Filtering fundamental data display

Allow users to monitor fundamental survey data with filtering capabilities to directly obtain survey data.

Adding customized data card for better understanding and monitoring

Creating customized data card to gain deep insight on survey

Setting up alert for abrupt data changes

COP will alert users on abrupt data change on workplace tool as they set up.

Design context

🤔 What is in-app survey and what is the purpose of survey?

When users swipe up on a TikTok video, they may see some surveys collecting feedback on the content they have viewed.

The surveys have three primary purposes:
1. Direct data understanding
2. AA(Abusive Activity) monitoring and observation
3. Data collection for algo modeling and optimization

PMs and TikTok operations highly depends on the internal tool - COP to spread out a great amount of surveys and collect survey data for analysis.

🤔 Why revamp the survey analyzer page?

Current survey analyzer can't meet all requirement of the three main purpose. It is crucial to rapidly collect results and analyze user option distribution to derive insights. Currently, obtaining customized data or data segment highly depend on data team support, leading to high costs and inefficiency. Additionally, users depend on the data team for alerts on abrupt data trend changes.

The project scope is

Revamp singular and global survey analyzer page to meet the high-frequency data analytics and early warning requirements.

🎯 Business goal

Reducing reliance on the data science team and boost productivity.

🎯 Users goal

1. Quickly and easily get survey data analyzed
2. Get warning on data abnormal trends

🏅Success Metrics

1. Success data collecting
2. Provide direct high-frequency data analytics
3. Be able to support the various monitor alert

🤔 What was the design question?

HMW statement

How might COP help product operation to directly get the most used survey data collected and analyzed, meanwhile be able to get informed on abnormal data trend without tracking manually.

Design Process

Understanding users and scenarios

Riley

Jessica

Chenyi

Required
data types

Survey understanding data

Monitoring data

Modeling data

Need

1. Quickly collect results after publishing a survey
2. Analysis data for deeper understanding

1. Quickly collect results after publishing a survey
2. Monitoring the change of the foundation data, presentations rates, and response rate
3. Identify the reason of data trend changes

1. Monitoring the change of the foundation data and other essential data
2. Identify the reason of data trend changes

Pain points

1. Current platform only has survey data for the past day
2. Unable to observe data based on regions

1. Unable to build customized data monitoring
2. Current trend chart only shows UV and submit rate trends

1. Unable to view data based on regions and time
2. The data change alert fully rely on human observation
3. The platform does not display the distribution of survey exposure frequency

User story

Reily launched a new survey on users' sense of control and aims to quickly collect responses to analyze overall submission rates and choice distribution.

Additionally, the goal is to determine whether there are significant differences in submission rates and choice distribution across different regions.

Jessica is monitoring user satisfaction on TikTok content and the trend of popular content. She wants to quickly track the weekly trend of the key satisfaction metric, user like rate.

Chenyi utilized survey samples for model training but has previously encountered anomalies in event reporting, causing a sudden drop in sample size and ultimately impacting the online model.

To prevent this in future, she seeks to implement real-time alerts when a significant decline is detected—specifically, when impressions decrease by over 80% compared to the previous hour or by more than 30% compared to the same hour last week.

Question 1
How might we help users quickly collect essential survey operational data and analyze those data easily?

Question 2
How might we help users easily monitor sudden operational data changes?

Question 1

How might we help users quickly collect essential survey data and analyze data easily?

1. Understand what data are required from users

After understanding user's need and pain points, I came up with that what data should be displayed on survey data analyzer page and assorted them into groups and sub-groups.

2. Explore layout for better data display

Given the data that are required for users, I began by exploring the layout of the analyzer page. To solve the design problem, I prioritized designing the essential data area and the trend chart area.

In the end, I chose to adopt the third layout which transparently display all core data on individual cards directly without any collapse. The layout is also scalable for adding more customized data card by just having an extra row.

3. The layout of essential data card

Since users monitor eight fundamental data points and customized metrics, I explored various layouts to optimize readability and data processing. I ultimately chose the third data overview design, which organizes information into three categories, making it easier for users to understand.

4. Filter data for deeper understanding

The filter helps users focus on a specific data segment for deeper insights, enabling precise analysis and informed decision-making. I ultimately chose option 2, which maintains the original spacing.

Question 2

How might we help users easily monitor sudden data changes?

1. Set up alert for sudden data change

Users can set up alerts for sudden data changes, which are automatically sent to their office software - Lark. I ultimately chose option 2, as the number of alert users is typically fewer than five, making it easier to view and manage all information within the alert window.

2. Notify users data change on data card

I also iterated the data card to include notifications for data changes. Explored multiple options for displaying fluctuations and, considering space constraints and the need to present all data, ultimately adopted the narrow card design.

Impact

Enhanced survey operations and data analysis capabilities.

This project empowered product operations to quickly access and analyze survey data independently, reducing reliance on the data team. It enhanced team efficiency and lowered operational costs.

Manage

Millions

Surveys

Improve

86%

Survey analysis efficiency

Reduce

23%

Tickets from survey operation

Takeaway

1. Fast-Paced collaboration

Given the short timeline and fast-paced nature of this project, the team adopted many methods from Lean UX. Collaboration among stakeholders played a crucial role in its success. I learned that establishing a shared understanding early on is essential. The PM, designers, R&D, and legal teams aligned their goals to ensure everyone was working toward the same product objective, enabling a streamlined development process.

Additionally, I learned the importance of building a frequent feedback loop through documentation and regular review meetings. Critiques on prototypes were diverse, with each piece of feedback contributing meaningfully to facilitating and refining the design process.