Scaling Web Video / Services

Scaling is not my thing, but Greg Linden continues to be a great source to poach from. His post on Facebook’s database architecture included a link back to a video presentation of YouTube’s approach to scaling during their buildup.

There’s lots of good info here, but I was struck by a couple of things:

• Small teams of talented people can accomplish amazing things in short amounts of time. In my mind the challenge for software development companies is to put together these types of teams and then figure out how to get out of their way. The amount of easily accessible info provided by the Internet has turned the world upside down in a way that mainstream software development hasn’t really figured out (information access = broader and deeper depth of knowledge for cross disciplinary teams).
• Heat and pressure make diamonds. The YouTube team didn’t know what to expect or have all the answers in their pockets, but were willing to aggressively tackle the challenges. There’s a point where people fracture under pressure, but if you never put yourself in challenging positions you miss out on what you’re really capable of as an individual and a team.

After watching, I was rifling through some of Greg’s past thoughts on scaling and found the nugget above about how often you should push code on the web. We’ve being playing around with an *agile* SCRUM development process at TechSmith, but there is a ton of discussion / disagreement about how frequently our web dev teams should be pushing code. Do you push during the course of the sprint, at the end of the sprint or quarterly?

Lots of questions arise if you’re pushing frequently. How do you ensure quality? Is it worth the risk? How do you market changes and upgrades to the site when change is constant? I’ve waffled a lot on this topic. My web dev roots tell me fast and furious updates, but after 2 1/2 years in the commercial software world I’ve gotten cautious and felt like a web product should only really make major pushes quarterly. It’s hard for the organization’s other pieces (support, marketing, sales) to keep pace when change is so frequent. The question is, have I become tainted by the long cycle feature slogs of desktop development and lost my way?