1021: We got addicted to an AI model we can't talk about

Dax Raad explains his team's setup of renting bare metal servers and slicing them into virtual machines for remote development. This setup offers high performance, easy hardware upgrades, and multi-device access, making remote dev environments accessible and practical. They use beefy servers hosted in data centers across the globe (Europe, US, Singapore) to minimize latency, and the setup includes tools like tmux for session management.

Dax discusses the development of OpenCode 2.0, emphasizing meticulous design and over-engineering of APIs to ensure robustness and flexibility. The new version aims to run as a service, sync seamlessly with desktop and web, and allow customization via scripting and APIs. The team dedicates much resources to primitive building blocks, prioritizing performance and cross-platform compatibility, which are foundational for their AI integrations.

The conversation covers the team's intense interest and activity with cutting-edge AI models, noting a recent explosion in token usage due to access to unreleased, high-performance models. Dax discusses model routing strategies, the cost implications of using various models, and the shift toward orchestrated AI agent setups where cheaper models are managed by more expensive, primary models. They highlight the rapid improvements in open-source models like GLM and the decreasing gap with proprietary offerings.

Dax explains the advantages of hosting models in the cloud, such as cost efficiency and simplicity, versus local hosting, which is often motivated by privacy. He notes current cloud inference margins are extremely high (around 90%), making cloud hosting more economically viable. He also discusses the community's interest in local hosting solutions, emphasizing that cost reductions in cloud computing are leading to affordable, scalable options, and that local hosting remains a privacy choice rather than a cost-saving one.

The episode covers the opaque nature of access to new AI models, variations in company policies on API use, and the implications for developers and enterprises. Dax points out that major providers like OpenAI favor consumer-facing, API-based access for revenue and growth, which can limit integration flexibility. He warns about potential future restrictions, model safety concerns, and the strategic motivations of big tech labs to control model deployment within their ecosystems.

Dax advocates for richer interfaces for AI assistance, such as producing visual artifacts and supporting dynamic UI components through standards like MCP. Voice prompting is highlighted as an increasingly popular interaction method, with innovative hardware (foot pedals, rings) enhancing productivity. He emphasizes the importance of designing tools that support both technical and non-technical users, aiming for seamless, intuitive experiences.