Get the full experience! Sign up to access transcripts, personalized summaries, and more features.
Yesterday, I shrunk the size of my production database from four terabytes to just under one terabyte.
Something interesting happened last weekend that made me realize I needed to change how I think about scale.
This episode is sponsored by Paddle.com — if you're looking for a payment platform that works for you so you can focus on what matters, check them out.
The blog post: https://thebootstrappedfounder.com/running-lean-at-scale/
The podcast episode: https://tbf.fm/episodes/352-running-lean-at-scale
Check out Podscan to get alerts when you're mentioned on podcasts: https://podscan.fmSend me a voicemail on Podline: https://podline.fm/arvid
You'll find my weekly article on my blog: https://thebootstrappedfounder.com
Podcast: https://thebootstrappedfounder.com/podcast
Newsletter: https://thebootstrappedfounder.com/newsletter
My book Zero to Sold: https://zerotosold.com/
My book The Embedded Entrepreneur: https://embeddedentrepreneur.com/
My course Find Your Following: https://findyourfollowing.com
Here are a few tools I use. Using my affiliate links will support my work at no additional cost to you.- Notion (which I use to organize, write, coordinate, and archive my podcast + newsletter): https://affiliate.notion.so/465mv1536drx- Riverside.fm (that's what I recorded this episode with): https://riverside.fm/?via=arvid- TweetHunter (for speedy scheduling and writing Tweets): http://tweethunter.io/?via=arvid- HypeFury (for massive Twitter analytics and scheduling): https://hypefury.com/?via=arvid60- AudioPen (for taking voice notes and getting amazing summaries): https://audiopen.ai/?aff=PXErZ- Descript (for word-based video editing, subtitles, and clips): https://www.descript.com/?lmref=3cf39Q- ConvertKit (for email lists, newsletters, even finding sponsors): https://convertkit.com?lmref=bN9CZw
Arvid discusses a significant database optimization where he reduced his production database size from four terabytes to just under one terabyte. He reflects on his recent observations about PodScan's data ingestion patterns, noting that the platform processes significantly more episodes during the weekdays than on the weekends. This realization prompts him to reassess whether immediate processing of every podcast episode is necessary, leading to a more strategic approach to growth.
The episode highlights the importance of smart database management, emphasizing the need to optimize queries rather than rely on costly hardware. Arvid shares his approach to dynamic chunking of database operations, adjusting the size of each operation based on performance metrics. This method, which he refers to as back pressure, allows for effective resource utilization without compromising performance.
A key advancement discussed is the implementation of automatic data compression for large text fields in the database. Arvid explains how using gzip compression for transcripts and JSON data led to a dramatic reduction in storage size by around 85%, while maintaining transparency for users. This optimization not only decreases storage costs but also improves speed and bandwidth, showcasing a significant achievement in managing large datasets.
Arvid stresses the importance of questioning long-held assumptions about his business model, focusing on delivering the right value to users rather than an overwhelming amount of content. He considers shifting from an immediate processing model to a more intelligent one that prioritizes content based on user needs. This pivot is designed to enhance user experience by ensuring that relevant content is delivered efficiently.
Join other podcast enthusiasts who are getting podcast summaries.
Sign Up Free