MC.42: Focus on What Actually Works in AI

Hi everyone,

The 3-day event AI Engineer World's Fair is taking place right now in San Francisco.

There's a lot happening, so expect more highlights in the next newsletters.

Message of the day: we need to focus on what actually works in AI.

Quinn Slack from Sourcegraph said only about 5% of professional developers use AI coding tools. This is low because many developers don't know about these tools, had bad experiences, or don't see their value. There are also concerns about costs and legal issues.

His advice for builders? Keep it real. If your team isn't using the AI tool daily, why would anyone else? And forget the flashy features - focus on making everyday tasks easier. Can’t disagree, I follow the exact same playbook with Alter, an AI built for macOS.

Beyang Liu, also from Sourcegraph, discussed measuring AI's return on investment. He shared these frameworks:

1. Role Eliminated: Can AI replace jobs?

2. A/B Testing: Compare teams with and without AI tools

3. Time Saved: Measure time saved by AI actions

4. Impact on Key Initiatives: Link AI use to company goals

5. Survey-Based: Ask developers for their opinions

The key is to set clear success criteria through evaluation frameworks. It must be a continuous process.

Aman Sanger from Cursor shared their work on predicting what coders will do next. They're not just guessing the next character, but the next action - like jumping to a new file. It's pretty wild stuff, but they're keeping it practical.

Their goal is to empower developers: AI in coding should make daily work easier and faster, not replace humans or do magic.

Links:

• Schedules: https://www.ai.engineer/worldsfair/2024/schedule#2024-06-27

• Live stream: https://www.youtube.com/@aiDotEngineer/streams

Enjoy,

Sam

Enjoyed this newsletter? Forward it to a friend and have them signup here.

Until next Thursday 🎉