Web 2.0 grows up

Leslie Knudsen meets Stewart Butterfield, founder and CEO of web phenomenon Flickr, and Jay Adelson, CEO of revolutionary internet news site Digg, to find out the reasons behind their success and why other companies should take note.

Originally coined by O’Reilly Media in 2004 and reported to be the most cited Wikipedia entry of 2006, ‘Web 2.0’ alludes to a web culture founded upon the principles of mass participation and collective intelligence. In essence, Web 2.0 appeals to social networks that thrive upon sharing and social interaction, representing everything from blogging to folksonomies to photo and video sharing, and everything in between.

Flickr, the social photo-sharing site that offers one of the largest real-time search indexes in the world and joined forces with Yahoo in 2005, and Digg, the social newsgathering site that has attempted to revolutionize the way we create and consume news, have, along with the likes of MySpace, YouTube and Facebook, been particularly successful. Their lasting power comes from core principles of innovation – human-powered tagging with real-time search and a collective social consciousness hell-bent on reinventing the consumption of content – and has generated a loyal following that accounts for hundreds of thousands of users and is still growing.

But just what is it that makes this new web revolution so interesting? And how can more traditional businesses capitalize on the success of Web 2.0?

The story behind Digg

Launched in December of 2004, Digg garnered more than 100,000 registered users within 11 months. Now with 800,000 registered users, the social bookmarking site has blossomed into one of the most popular sites on the web, while providing an alternative to traditional news models in which content is generated by a single editorial vision.

At Digg, users vote upon submissions such as news stories, podcasts or videos, and then submissions are promoted and ranked based on the number of ‘Diggs’ they receive. Users also have the option to ‘bury’ a story to demote it in the rankings, and thus what appears on the homepage is based upon what the collective intelligence of the masses decide is interesting or relevant. Giving users the final say on what deserves to be top content can result in anything from software releases to politics to celebrity sightings appearing on the homepage. Despite how wise or unwise the masses prove to be, Digg’s wild popularity attests to the fact that audiences are eager to have a say in the perpetuation of news and other media across the web.

In fact, there have been a number of instances where Digg has broken a news story before the major networks or traditional news outlets. Even so, Jay Adelson, CEO of Digg, still views the company as a complement to traditional news media rather than a challenge. “Digg was never meant to compete with traditional news outlets, and its relationship with online news that is gathered and fact-checked by traditional editorial staff is totally symbiotic. We drive traffic back to the original content source, after a collaborative filter (our registered user base) votes to advance a story. Because of this, the CNETs and CNN.coms of the world see Digg as a positive tool. What makes Digg truly unique is that it levels the playing field, because everything – whether it is a personal blog post, or a New York Times story – carries the same submission weight.”

Traditional media outlets can’t help but acknowledge the larger implications that the success of Digg hints at – both the cracks in a single-minded editorial process for deciding and disseminating news, as well as the fact that these new media vehicles require little to no resources to convey information to large audiences.

“Digg’s objective is to create a truly level playing field, allowing any media object to access equal size markets,” Adelson remarks. “What publishers in general are seeing more and more of is that you don’t need to have a printing press any longer, or a large publishing network, to get the same idea or media object (blogs, videos, news articles, etc.) to millions of people. That Digg can surface a major news story more rapidly than some traditional media organizations is really a testament to our community. Digg is the platform for the discovery of news, but it takes a large and passionate community to bubble it to the top – in some cases, more quickly than a group of editors or a search engine spider can.”

The Flickr phenomenon

Utilizing advanced tagging technology and real-time search capabilities, Flickr capitalizes on the social nature of photography and the rise of the digital camera to promote sharing photographs online. Since being acquired by Yahoo! in March 2005, Flickr has shot up from around 400,000 registered users to more than 6.5 million and today the popular photo-sharing site attracts about 20 million unique visitors per month.

The Flickr and Yahoo marriage has proven to be fruitful for both, and the greatest adjustment they had to make as a result of the acquisition was coordinating the logistics of moving their nine-person team from Vancouver to California. “At the time we were acquired, we were growing at about 70 percent a month; we had a small team of nine people who all moved down at the same time, so at first it was pretty disruptive,” admits co-founder and CEO Stewart Butterfield. “But it was a good environment for us. Another reason we went with Yahoo! versus other companies was because it was a really good philosophic fit.”

Butterfield notes that the alignment of philosophies has been a key factor in forming a successful union and retaining the original Flickr team. “Yahoo! has a more human-focused approach, which includes a lot of existing social products and technologies. With hundreds of millions of accounts, Yahoo has a good understanding of disseminating to lots of people since dealing with large audiences is sort of their core competence. One of the key things with an acquisition is making sure that the vision that the start-up people – the original founders and team – have for where they want to go is something that the acquiring company can really get behind.”

Not only were they able to match their philosophies, but Flickr was also able to leverage the brainpower of thousands of engineers alongside Yahoo!’s search technology. “One of the things we were lacking when we were acquired was good search technology and we’ve actually been able to take advantage of not just homegrown data stuff but a lot of really good internal search technology,” Butterfield acknowledges. “One platform we use is called Vespa and it’s been fantastic. It lets us do real-time search and today Flickr is probably the largest real-time search index anywhere. Flickr has more than 360 million photos that are all searchable in real-time, including geographic search, which is pretty cool.”

So, what can Yahoo! say it’s acquired from Flickr? “We were a company with three or four engineers coming into a company with maybe 4000-5000 engineers, so there were some innovative things we had done but it’s not like there was a measurable impact on the overall engineering talent together,” says Butterfield. “At the same time, we have a pretty kick-ass engineering team, particularly in the area of tagging, which is one of the motivations for being acquired in the first place. We’re still by far the largest repository of tags – there’s around a billion tags in the system – and even getting rid of misspellings and other outliers, we have about eight million unique tags. That was a homegrown Flickr technology that was interesting to Yahoo!”

The experimental element of innovation along with the rapidly growing audience Flickr has brought to Yahoo! has proved the acquisition to be a smart move for the media giant. “At the time we were acquired, our audience was maybe a million and a half unique visitors per month, but I think the acquisition was more about the trajectory because Flickr was growing so fast and it’s still growing very quickly without any advertising or promotion – just through word of mouth. I think we offer a small contribution but I think it’s an area where there’s significant innovation and traction among users. Therefore it’s a good thing to bet on even if the bet is based on a lot of experimental or as yet unproven strategies. 20 million unique visitors a month is a lot, so having an audience wasn’t a motivation to buy; however, the potential given the trajectory and the growth rate was, and that’s actually how it played out.”

Mass intelligence or mass chaos?

Both companies operate on the foundational principle of giving control to the masses. This can be a risky strategy, though formulaic processes and algorithms have been designed to keep things in check.

At Digg, they view the users as both the propellants and the deterrents of any cracks in the system. “There are challenges to any open system, but fundamentally I do not see any downside to completely democratizing media,” Adelson says. “In many ways, it’s a massive experiment and we’re learning every day how to harness the collective wisdom of our users. In the case of inaccurate stories and spam, Digg’s first line of defense is the users themselves, and they are incredibly efficient at burying these items before they have a chance to get promoted to the homepage.”

Similarly, Flickr lets the users decide what deserves to be in the most popular rankings but uses intricate calculations to determine the criteria for advancing a photo to the top 500 list. “We do several hundred billion calculations to determine those 500 photos out of the million that get uploaded every day, taking into account all kinds of things – simple things like the number of views that a photo gets or the number of times people add it to their favorites list, but also more complicated things like where people are clicking through from, and the relationship between the viewer and the person who uploaded the photograph,” Butterfield explains.

For both sites, however, the masses are only getting more massive. One of the greatest challenges faced by both companies is retaining the same user experience in light of this exponential growth. “Taking something that was popular initially because it was a really innovative social application and scaling that is hard to do, because when there’s a small group of people it’s much easier both to manage and participate in,” Butterfield acknowledges. “There’s sort of a core community that has its own internal structure that has to be preserved.”

Flickr has experienced rampant growth since its acquisition, but goes to great lengths to ensure the user experience is the same. “The biggest challenge over the last 18 months or so has been really preserving what people like about Flickr in the face of it getting about 20 times larger than it was when we were first acquired, he continues. “You have people who signed up last week who have the same kind of experience that people had when they first created their accounts two years ago. That’s something that is underappreciated and isn’t really recognized as an area of innovation, but there’s a lot that goes into that – community management, customer care, creating and managing groups, implementing social networking features and privacy controls – and making those work at both a product level and technological level.”

The next revolution of the web

Looking to the future of participatory media, Adelson feels continuing to improve information sharing and remaining at the cusp of innovation should be key strategies for all Web 2.0 companies. “The trick for sites in the so-called Web 2.0 space is in innovation. You see plenty of features being implemented too fast on a lot of services that don’t seem to feed a broader strategy. At Digg, we don’t want to rush and add a functionality as a ‘me too’ feature. Instead, you’ll see us continue to experiment with new features that are consistent with our vision (Swarm and Stack are two examples) as well as take the time to listen to the community and develop features driven by them (such as podcasting, which was widely requested by our community).”

While acquisition rumors abound for Digg, the company is focused on improving its core model in 2007. “We are headed into different areas where a collaborative human filter will help people discover content that’s relevant to them,” Adelson says. “Our recent rollout of podcasting is one example of an additional form of content to which we’ve applied our model. The possibilities are endless, and to date we have only implemented a fraction of our vision for Digg. In the new year, expect to see improvements in Digg when it comes to sharing information between people within the site.”

Butterfield echoes the focus on information sharing, in particular, improving analysis amid the onslaught of data. “Very few things have reached the scale of something like Flickr or YouTube where the number of participants measures in the tens of millions. When that happens, the amount of data gets really enormous – billions and billions of discrete facts about the system – that includes not just photos but tags, comments and notes. So the next thing will be doing really good analysis and helping people find what they’re looking for, not just purely with search technology but using a combination of search, ranking, relevancy and personalization.”

With user participation only expected to flourish, Butterfield sees a definite window in the area of personalization. “I think there’s still a lot of room left in personalization. It’s a term that’s been around in the internet space at least since the late 1990s, but still hasn’t really been exploited to its full potential. This is not meant to be a slight at all, but when I go to Digg, I don’t see a lot that’s interesting to me in particular. I went there the day that North Korea tested their nuclear weapons and the first story was about a new version of Ubuntu Linux being released, which is something I couldn’t care less about. The same sort of criticism can be leveled to Flickr, where we do a lot of automated tools for displaying photos that are really popular or hot based on a metric we use called ‘Interestingness’. Around a million photos are uploaded every day and you can find 500 just based on purely algorithmic means that people find really interesting. Of course, it may not be to everyone’s taste so taking that data and really presenting it in a personalized way is something that there is a lot of room left for – it’s potentially a great source of innovation.”

While the foundational principles of Web 2.0 – including user participation and community sharing – are expected to flourish, the tired reaction to the phrase suggests a readiness for the next revolution. “It’s hard to think about the next revolution of the web since we’re really just scratching the surface of the user-driven web,” Adelson reflects. “One idea that we think a lot about is more sophisticated ways of grafting human interactions and patterns onto purely technology-driven systems in order to surface information more efficiently than either model could on its own. It feels like there would be a ton of applications that could benefit, not the least of which is search.”

Even if the next revolution of the web isn’t around the corner, it’s clear that Butterfield is at least ready for a new term. “Web 2.0 has already been around now for over three years and I think it’s just become a tired term. I’m not sure if something in particular will replace it, but I think people will eventually get tired of using that just because it’s old and has lost its descriptive power. As for the actual trend or innovations that will cause people to realize that and come up with a new and refreshing term – I have no idea. I’m not sure if the Web 2.0 phenomenon will ever truly be ‘replaced’ by the next web revolution, because I think in essence Web 2.0 is really a return to the roots of the web and what attracted people to the web in the first place. The ability for people to connect with people they care about – like friends and family – but also with people that share common interests and interact with people who they never would otherwise is really the interesting part of Web 2.0 to me.”

The evolution of the internet that’s brought on the advent of the Web 2.0 movement can most accurately be reflected by how society’s perception of web participation has changed. “There was a time when the internet had become popular around 1996 or 1997 as an item of technological curiosity, but where participating online was still considered really weird,” Butterfield recalls. “If you think back to editorial cartoons from that era, the internet user was always like a fat man in his underwear using the computer. It was kind of creepy and deviant; people who would use online dating sites were assumed to be desperate and message board users were freaks. But now it’s totally different. Most people know someone who’s in a dating service and has actually built a relationship out of it. Grandparents and grandchildren are instant messaging back and forth and none of this is weird or creepy or deviant anymore. It’s part of the natural course of things and to me, this is really the legacy of Web 2. 0. I think there will continue to be technological innovation, but I think the sociological phenomenon is really the core of it and that’s not going to change.”