AI Horizons: Designing AI Agents People Will Actually Use

As AI systems become more capable, a new challenge is coming into focus: building intelligent technology is no longer enough. The real question is whether people will trust it enough to let it manage their inbox, organize their files, or even help make decisions.

That question anchored Penn Engineering’s AI Month webinar, How to Design AI Agents People Will Actually Use. Led by Stefano Puntoni and Thomas McKinlay, Founder of Science Says, the session explored what it takes to move AI from technical capability to real-world adoption.

Because in today’s AI landscape, success isn’t just about what systems can do. It’s about whether people are willing to use them.

The Psychology Behind AI Adoption

One of the most significant barriers to AI adoption isn’t technical—it’s psychological.

As discussed during the session, users experience multiple forms of psychological friction when interacting with AI agents. Among the most prominent: trust, control, and uncertainty about how these systems should fit into their workflows.

People don’t just evaluate whether AI is accurate—they evaluate whether it feels safe, understandable, and aligned with their needs.

Research shared during the webinar revealed a powerful insight: people are more likely to trust AI agents when they understand their limitations. Rather than diminishing confidence, transparency strengthens it.

And how AI communicates matters just as much as what it communicates. As Puntoni emphasized when discussing system outputs, “Give them precision if you are able to give it to them.” Specific, quantifiable responses signal confidence and credibility, helping users feel more comfortable relying on the system.

The Delegation Dilemma: How Much Control Is Too Much?

Beyond trust, another key friction point is delegation.

AI agents are designed to act on behalf of users, but that raises a fundamental question: how much should we actually hand over?

Puntoni described this as a central challenge in AI design: “It is difficult for the user to decide how much to delegate.”

Too little autonomy, and the system becomes inefficient—requiring constant oversight and defeating its purpose. Too much autonomy, and it can feel risky or even threatening.

The solution lies in balance.

Effective AI agents operate as co-deciders—handling tasks independently when appropriate, but checking in when decisions carry greater weight. This approach preserves efficiency while maintaining user confidence and control.

McKinlay reinforced this idea by emphasizing that perception matters as much as functionality. “Agents feel less threatening when they feel less powerful,” he explained. When users feel they are still “in the driving seat,” they are more willing to engage with AI as a collaborator rather than a replacement.

Even small design choices can reinforce that sense of control. For example, allowing users to personalize or name their AI agent can increase comfort and adoption, because it subtly shifts the dynamic from external system to user-directed tool.

The Risk of Over-Reliance

As AI systems become more accurate, a new challenge emerges: complacency.

Research shared during the session, including insights from Hamsa Bastani, highlights a paradox. When AI performs correctly the vast majority of the time, users become less motivated to verify its outputs.

As Bastani noted, if AI is right 99% of the time, the incentive for human oversight drops significantly, making it difficult to maintain vigilance for the rare but critical error.

This dynamic underscores the importance of thoughtful system design. Without mechanisms that encourage continued engagement, even highly accurate AI can introduce risk through overreliance.

What It Takes for AI to Work in Organizations

These challenges become even more complex at the organizational level.

For AI agents to be successfully adopted in the workplace, the session outlined three essential conditions:

• Perceived Confidence: Do employees understand what the AI can and cannot do? Do they see it as a collaborator—or a threat?
• Trust: Do users believe the system produces reliable outputs, while remaining appropriately cautious?
• Delegation of Control: Are there clear guidelines for when to rely on AI, and when human oversight is required?

As McKinlay emphasized, these elements are not optional. “You need these things in place for AI to be sustainable in the workplace.”

In other words, successful AI adoption is not just a technical rollout, it’s a behavioral and cultural shift.

Building AI That People Choose to Use

What emerged from AI Horizons is a clear takeaway: the future of AI will not be defined by capability alone.

It will be defined by design—specifically, how well AI systems align with human psychology, decision-making, and trust. This represents a shift in what it means to work in AI.

The next generation of practitioners won’t just build models. They will design systems that people feel comfortable using—systems that balance autonomy with oversight, precision with transparency, and efficiency with control.

At Penn Engineering, that mindset is already shaping how AI is taught and explored.

Programs like the Online Master of Science in Engineering in Artificial Intelligence prepare students to engage with AI as both a technical and human-centered discipline—equipping them to build systems that don’t just work, but are actually adopted.

Because the most advanced AI system means little if no one trusts it.

And the professionals who understand how to bridge that gap won’t just build AI agents. They will define how AI fits into the future of work.