Content engineering and model architecting to create an AI support tool for integrity
Creating AI support to help advertisers fix ad rejections
Problems
Business Ad rejections cost $246M
User
No preventa
tive experiencesWe do not offer proactive guidance during ad creation, meaning advertisers make mistakes that then get rejected, setting them up to fail.
Lastly, a classic conundrum for content design - all of this had to be communicated in notices and dialogs; a stunningly small piece of real estate.
What I did
- Drove creation of Meta's first AI Transparency playbook
- Prompt creation and model training, including labelling
- Policy risk mapping
- Created scalable eval criteria aligned at leadership level
- AI response frameworks, including guidance on canned vs. freeform responses
- AI modality strategy
- Model architecture
Who I worked with
x2 PD x5 engineers x1 DS x1 PGA x1 PMM x1 UXR x1 PM X1 EM x1 Policy Manager x3 Ops partners x1 Lawyer
Constraints
Real-time abuse vector
AI editing tool could become a new abuse vector. If we allowed unlimited queries, bad actors could abuse this and map our rules to circumvent our policies.
PXFN review.
To avoid legal liability should the model hallucinate, we were unable to give advertisers instructions on how to fix, e.g. Say “Remove this image”. How can we frame guidance to be useful while managing transparency?
Surface owner alignment
We don’t own the components, or interactions we were proposing and needed to align with the surface-owner team, which was slow- especially as some of the things we were proposing were net-new.
Limited real estate
AI is known for it’s verbosity. We had the space of an error message to explain why a particular image/text/video was violating and needed to be succinct. How to overcome?
How I did it
⚽ Step 1. Kick off
Mapping E2E flows
Because the scope was so large and there was a lot of emphasis on the messaging being absolutely right, I tried to create a set of guiding principles that we could come back to and use as a rational benchmark for all our decision making when I communicated with leads and XFN.
These were:
🖊️Step 2: Defining a comms strategy
Understanding constraints and dependencies
Once, I’d established product principles, I started to pull in PMM and MarComms to understand timelines and broader company positioning of Meta accounts that we might need to pull into our own product comms. As it transpired, we were not the only product team who were launching Meta accounts for the first time. Our sister team in Horizon OS, who were driving the Meta account log in flows, were launching the same day as us. We’ll discuss more about how we converged and diverged with their comms strategy later.
⭕ Mapping dependencies
- Horizon OS team for the Meta account ‘sign up’ flows we were integrating into our onboarding
- Legal to review new sign up flows, and well, all and any UX related to Meta accounts
- Privacy to review technical architecture of new Meta account flows and ensure user data was being upheld
- As we were anticipating an uptick in overall users into Horizon Worlds, Integrity needed to review our policy around user reporting and bullying and harassment in immersive spaces, and Workrooms needed to onboard to those policies for the first time. This also meant we had to build out user reporting flows in our workrooms spaces.
- Content strategy team for our Help Center articles on new Meta account setup and troubleshooting
- Workrooms engineering to rebuild the new account container and asset migration flows for existing Workrooms users- no mean feat.
- To hit a public launch date of August 20th, we had precisely 1 month to be design complete
- Cadence/ “when”
Here I worked with PMM to devise a GTM strategy. Because we were migrating (and in some instances deleting) user data as part of the Meta account launch, we were legally obliged to inform users 90 days before. This was our first firm constraint.
We prepared a staggered comms plan with notification going live:
- 90 days before launch
- 1 month before launch
- 2 weeks before launch
- Day 0 (launch day)
In all, 4 sets of notices.
2. Audience/ “who”
We anticipated that once Meta’s PR team sent out initial comms, there’d be a lot of public scrutiny and we’d see additional traffic to the product. This could be new users, the press, or someone with ill-will towards the company who were looking to misrepresent us. We didn’t know who might come across our messaging, so we needed to strike a neutral balance between factual, but also excited. For legacy users, especially, the new Meta accounts came with some pretty neat benefits that we actually wanted to preach to about to balance out the negative implications of the forced account migration.
We decided to segment our users into groups and offer up a slightly different onboarding experience. These were:
- Legacy users
- Admins (These guys had stuff to do to support the account migration, so we really didn’t want them ignoring comms).
- Non-admins
- New users
3. Component/ the “what”
I wanted to find out more about engagement model to inform the methods we used to communicate outwards. Email is a notoriously poor way to contact users; statistically, only 10% of users actually received our emails (because of bounce-back or it landing in Junk) and only 3% actually opened them once received. This meant we couldn’t necessarily rely on email as our only means of communication.
Data told us of our 40,000 MAU, 75% were logging in once a month and only 30% were logging in daily. This meant that a large chunk of our user base might miss in-product comms.
In the end we hedged out bets. To ensure full coverage, we targeted all surfaces: web, email and VR, including an email campaign; dismissible dialogs and sticky live countdown in VR; and dismissible QPs on web.
Announcement email to admins and non-admin legacy users
We couldn’t anticipate where users would log in, so we covered both web and virtual reality.
How we announced Meta accounts in VR
Adding a persistent countdown to the top of the personal user interface (PUI) in your virtual personal office, so you never missed a beat.
4. Strategy/ the “how”
Now, it came to writing. I started off imagining how a user might be feeling in that moment when they received those initial notices about the account model change. Annoyed? Frustrated? Apprehensive? It felt wiser to err on the side of caution and assume worst case scenario. I avoided excited or celebratory tones, as we just couldn’t predict how a user might feel. And, exclamation marks (!) - forget it.
Instead, we focused on actionable guidance, answering these 3 questions:
- What is happening?
- When is it happening?
- What do I need to do and how do I do it?
Notification email sent prior to launch that pre-empted any user concerns and prepared them for launch day.
Email sent on Day 0
Once, we had core flows down, me and my PD pulled in UXR to test the E2E flows; value props and methods of communications we were employing.
We learnt that people were accepting of the new account, as long as we showed a clear rationale for why we were doing it and offered them a helping hand during the change state.
The sign up screen
What legacy users saw on Day 0
Impact
Business
- Mitigated risk: no mainstream backlash or negative user or press feedback
- Perceived as highly successful launch by company > received personal thank you from Zuck
Workflow
- PD and CD working relationship used an example in company for how product and content can work together
- Established product principles used for duration of product shelf-life
Product
- 75% increase in sign-ups for Workrooms
- 100% of all returning users migrated their assets into the new account to continue using Workrooms
User feedback
- Generated excitement for legacy users in community feedback groups
- Helped legacy users look forward to- rather than than dwell on- the onerous changes
- “This little message was reassuring”: No greater compliment to content design and demonstrates that a little goes a long way.
