Shiva Chandrashekher is head of product at Amazon and has launched and scaled several AI and GenAI consumer products at Amazon.
Personalization makes digital experiences smarter. Privacy keeps them trustworthy. For product leaders building with AI, these priorities often pull in opposite directions. The sharper the recommendation, the more data it likely draws on, and the more data in play, the more likely users are to question what’s being collected and why.
According to Adobe’s 2022 trust report, 72% of users, especially Gen-Z and millennials, said they trusted brands more when experiences feel relevant. Yet, 81% emphasized the importance of having control over how their data is used.
This is where many AI-driven products stumble. The issue isn’t personalization per se, as users still want smart, relevant experiences. But they’re asking for it on better terms, wanting to know how those systems arrive at their conclusions and whether they can do so with more transparency. When trust erodes, users opt out, delete apps or switch to platforms with clearer boundaries. It’s important to understand how to address this in your own platforms and products.
Where The Balance Breaks
Most personalization engines operate on long-term identity signals: profile data, behavioral history and implicit inference. Although these can produce accurate results, they often do so in a way that users find difficult to interpret, and so it’s easy to mistrust.
The underlying flaw is this: Many systems optimize for who the user is, rather than what they’re doing right now. This approach assumes continuity of intent, but context changes quickly. What’s helpful in one session might feel invasive in another. Users pick up on this intuitively.
Twilio found that 63% of users are comfortable with personalization when it’s based on data they’ve explicitly shared, but “only 40% trust brands to use their data responsibly.” The difference is largely perceptual. Users describe these systems as “creepy” when personalization feels too specific and too unearned. Trust breaks when users can’t tell why a recommendation appeared or they didn’t feel like they had a choice in the matter.
Four Principles For Privacy-Respectful Personalization
Having built AI systems across voice, vision and multimodal platforms, I’ve seen the most durable solutions take the stance that personalization is something you earn. Here’s what that looks like in practice.
1. Personalize by context, not identity.
The most effective personalization often doesn’t require a detailed user profile. Instead, it responds to what’s happening in the moment: current session behavior, device state, time of day and declared intent.
Contextual personalization often feels more relevant, more accurate and, paradoxically, less intrusive. Across industries, we’ve seen better engagement when recommendations are tied to the immediate task, not a profile history. Profile-driven solutions tend to “follow the user around,” which is when it starts to feel invasive and when privacy-sensitive audiences start opting out and disengaging.
2. Make the personalization transparent.
As discussed in my previous article about making AI transparent, if a user doesn’t know why something was suggested, they won’t trust it, even if it’s right. And in practice, the explanation doesn’t need to be technical. A short line like “Based on your recent searches” or “Because you liked X” is often enough.
What matters is how you inform the user. Expose the logic of the system enough for the user to feel in control. Provide clear preferences and explanations for choices and make them part of the product experience, instead of hiding them behind a privacy policy. When users feel informed, they feel respected and engage more readily.
3. Ask, don’t assume.
Many products still treat personalization as a default setting, but people increasingly prefer to be invited into tailored experiences.
Lightweight opt-in prompts can make a difference. “Would you like recommendations for this?” or “Should we remember your preference for next time?” builds trust without heavy user experience (UX) cost. Although this may have once been considered a potential friction point or intrusive to the flow, it’s becoming an important trust touchpoint and opportunity for collaboration. Assumptions damage trust. Invitations—and shared decisions—build it.
4. Build with a privacy-centric infrastructure.
How personalization is delivered matters just as much as what it does. Technologies like on-device inference, federated learning and differential privacy allow for personalization without exposing raw user data.
In its implementation of Gboard, Google found that using federated learning on users’ devices improved next-word predictions compared to models trained on centrally collected data. Apple’s on-device models for Siri and app suggestions follow a similar logic, using differential privacy to obscure individual signals while still learning across the population.
The takeaway: If a model gets you 80% of the way there using anonymized or local data, that last 20% may not be worth the cost in user trust. Effective personalization doesn’t have to stretch every data source to its limit.
A Better Approach To AI Recommendations
The tension between personalization and privacy isn’t going away, but it doesn’t have to be a zero-sum trade-off. To summarize, the best-performing systems I’ve seen in this space share four habits:
1. They scope personalization to tasks and context, instead of anchoring in user identity.
2. They explain system behavior clearly, so users understand what’s happening and why.
3. They make user agency visible and accessible, not buried in a settings screen or privacy policy.
4. They invest in privacy-preserving infrastructure early, before regulation enforces it.
Machine learning can power deeply personal user experiences but only when people have a clear sense of what it’s doing and why. Personalization without trust can be seen as surveillance, and trust without clarity can be fragile.
Even from a business technology perspective, adopting these privacy-first methods is becoming much more necessary. With the phase-out of third-party cookies and stricter regulations, many organizations are retooling their personalization tech stack.
As privacy expectations rise, the only personalization worth scaling is the kind users would choose for themselves.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
link