Are AI tools actually helping people at work?

This episode of Out-Of-Pocket is brought to you by...

Champ AI automates what APIs can't reach. They identify, build, and run multimodal workflows end-to-end — it's a single platform that reliably works across browsers, portals, unstructured documents, and phone calls, covering everything an operations team touches in a given day.

Their custom approach helps organizations close the "last mile" gaps that often block AI adoption. Learn more about how healthcare companies like Pair Team and Float Health use Champ AI to automate operations.

→ See how it works

—

We're planning our content calendar for next year, if you wanna partner we should chat.

"Working hard? Or hardly working?" - I say to my agents

We've been setting up more AI tools for the Out-Of-Pocket team. Plus I've been chatting with a lot of other teams about how they're using AI at work in advance of our Ship it and Knowledgefest conferences where we're going to do some workshops around it.

It's pretty interesting to see how large the delta is between how people talk about using AI at work and what they actually say in private conversations.

A few notes, just my experience.

People aren't yet excited to use AI at work because they use the wrong tools

I get a ton of pitches around companies trying to sell AI to enterprises. But IMO all of them rely on the basic idea that people at the company are going to be down with using AI tools.

My anecdotal experience is that a very tiny group of people are excited about AI. Even fewer are excited enough that they'll be willing to learn new workflows using it. Some things I've noticed:

• Most people still don't pay for AI tools, which means their exposure to it is either the free version of tools or the enterprise versions like Microsoft Copilot. These are both pretty bad and generally not time saving. For most people in healthcare this is their only experience. I think whoever is in charge of Copilot used to run escape rooms or something.

• A lot of people believe the level of correctness needs to be 100% before using it, which is simply not feasible. One incorrect answer will turn people off using it completely. Many people tried using AI in 2024 when it was hallucinating and this is their view on how it still works today.
• Some people think using AI is actively training it to replace them. Good thing AI can't have an existential crisis about their career choices every 2.6 years, or I'd be worried.
• Most people at the company are not economically benefitting from AI use. They don't own enough equity in the company to benefit from productivity gains, they aren't getting bonuses for AI usage, and they're still expected to do their own jobs while training something that management is implying might cause them to be laid off.

All of these things will lead to people not using it. Companies trying to sell larger enterprises on employees training on their AI tool are going to have a tough time because the issue is deeply cultural.

In fact I think the main advantage so called "AI-native" companies have is that they self-select for employees that are excited about using AI tools in their workflow and automation is looked upon positively from peers.

Quick aside - OOP stuff! Fun stuff!

Since I have you...a few quick updates:

We're really cooking on Knowledgefest, we have some great topics for workshops this year (they'll be announced soon).

If you're in SF and you're building a healthcare company, you're gonna want to be there. We have a lot of ballers coming. Why not apply just to see :)

And a reminder that we have some great free/paid courses. If you can't make it to the free courses, you'll still get the slides and recordings if you sign up.

• FHIR 101 starts next week, which we're doing with the Redox squad. Learn what the technical standard is, even if you yourself are not technical (though with LLMs now...are we not all engineers...)
• Value-based care 301 with Navina (9/15 - 9/17). Lots of new CMMI models, lots of new tools like AI, same old problems in value-based care. Free course, we're getting into the details.
• Healthcare 101 Crash Course (9/21 - 10/2) - Professor Krishnan will teach you what you need to know about how US healthcare works. If you have a team that's new to the space (LOOKING AT YOU YC COMPANIES AND HORIZONTAL AI COMPANIES), I'll teach you.

You can see the full course library here.

AI is forcing interesting changes at organizations

We have our two conferences coming up, Ship It for software engineers and Knowledgefest for healthcare ops. In our applications for them we ask people how they're using AI at their orgs, pros/cons of rolling it out, etc. Some interesting notes comparing them:

For both engineers and ops people their job has gone from doing the work to checking AI output. In some ways people like this if the automated work is drudgery. But in many cases people think checking the work is taking more time esp if the output is slop. It can also be harder to catch issues if you're not the one creating the output in the first place, increasing the chance of missing things.

However issues arise when human checking now becomes the bottleneck + orgs are reluctant to hire because their people are supposed to be "more productive" thanks to AI.

People struggle with using AI in areas where they themselves haven't actually written out the rules. Things like scheduling logic, which patients are high risk, etc. requires a lot of judgment calls that kinda live in people's heads.

"Concepts humans understand intuitively – like 'healthy' or 'don't send duplicates' – had to be clearly defined and supported with structure data"
- Knowledgefest applicant

But when rules ARE clear the productivity gains seem to be large. Scheduling, claims document review, filling out forms for things like credentialing, parsing out 270/271 responses, etc.

Non-engineers love being able to do technical things while engineers are working around that. The ops people are like "wow I vibe coded a feature for our frontline teams" and then engineers are like "oh my god no they know what production means now". There seems to be a lot more engineering work spent now so that non-engineers can build themselves.

Managerial dynamics are changing. For ops in particular it was interesting to read about how companies are building more individual contributor tracks for technical ops people and now needing to think more about what upskilling means with AI tools. But also a lot of people (eng/ops) have built tools to help them be better managers - gathering data on direct reports, coaching them, etc.

It also introduces more surveillance-y dynamics - e.g. teams can now run AI against 100% of nurse-patient interactions and transcribed conversations with patients. We talked about how this was going to change compensation in a previous post.

There are power users and non users - Orgs seem to split around people using it all the time or basically not at all. This creates an interesting dynamic where companies are trying to figure out if they need to allocate things like token budgets to people, or just let their power users use as much as they want.

The second order effect of that is whether token budgets become a part of compensation, especially if that ends up being directly connected to your work productivity. But then should you be dinged for getting less done if you didn't negotiate a token package?

AI makes people want more determinism - Funny enough both sides basically said "AI makes it easier to build deterministic workflows". Ops people said they can now build these themselves for smaller tasks they didn't have engineering resources for. Engineers talk about using AI to understand all the different edge cases that come up in a process and then creating deterministic rules to handle 80% of them and human escalation paths for the rest.

No one knows how to quantify if AI is working. There are two dimensions to this.

The first is figuring out if vendors work when they all claim to do the same thing. Everyone *gestures to evals inconclusively* (this is part of why we have an evals workshop at Ship it from someone at Anthropic).

The second is whether the spend and investment of time into these is actually generating a return on that investment. But how do you measure that, dollars the org got? "Doing more" with less people, but were they even the right tasks? Basically everyone seems to be asking themselves this question and want to hear how peers are thinking about it.

"It's more expensive than many people assume once you account for inference, review, and operational oversight"
- Ship It applicant

This episode of Out-Of-Pocket is also supported by...

A few things on my mind re: AI at work

There are a few things I'm particularly interested in watching unfold re: AI in the workplace in the next few years.

User Experience > Model Capabilities - The marginal benefits to the models getting better no longer matters to companies that are not working on some cutting edge tech. The question is how do they get companies to get their employees excited and using AI at all.

There are a few things I think would really help:

• Setting up integrations for employees to use out of the box
• Embedded in interfaces they already use (e.g. chat, text, email, Drive)
• Better passive memory that makes it extremely easy for the AI to learn a workflow and copy it in the future
• Proactive completion of tasks without needing a prompt to start it

I've noticed that if you can own 1 or 2 tasks end-to-end that take up a lot of time with a "push to start" button for people, even skeptics start becoming interested.

How do you create internal cultures that encourage AI use, but not overuse? Have you or a family member been subject to claudeslop?

It's interesting to watch companies both WANT their employees to use AI, but also not rely on it for everything. For example, Clay now has policies around when to use AI for writing and I'm curious if other companies will end up making their own versions.

"3. More time should be spent writing a document than consuming it

If you generate a document from a short prompt then ask your readers to go through the longer output, you are disrespecting their time. They can talk to ChatGPT themselves if they want to."

It's very easy once you start using these tools to just rely on it to do everything. But obviously that is also wrong. I'm curious how companies are going to navigate this - it feels near impossible to outright stop people from using it. I have a feeling that Pangram or Claude Watermarks will end up playing a role in monitoring how frequently internal employees are using AI to do their work for them.

Cybersecurity is an incredibly underrated part of this - Another part of working with AI tools is that you can start sussing out when OTHER people are using them. For example, I can spot most AI writing and design quite quickly. But it's clear that some of the aunties and uncles cannot do this, and I suspect it's because they don't use the tools every day.

The area I think this will end up mattering most is cybersecurity. In many of the recent security tests on agents, they have gotten much more sophisticated when asked to do malicious things. This incident report from the AI security institute shows how far the agents would go to try and complete their task, which includes social engineering.

"2. Attempts to deceive and target real people. As part of the same effort, the agent tried to contact real people directly, sending messages and files through an online file-transfer service to persuade them, or their own AI coding tools, to run malicious code. Some messages carried harmful payloads, and some were attempts at social engineering; targeted at real people – something we've never previously observed. "

I think one of the reasons to get employees familiar with AI tools is to also start training them to look at how AI agents try to communicate with them and build their own spidey senses. This is especially going to matter in healthcare, where the cybersecurity systems are already bad and the consequences are much higher.

Thinkboi out,

Nikhil aka. "LLM controlled meatsack" aka. "AI Nativity Scene"