MySpace.com is one of the fastest growing site on the Internet with 65 million subscribers and 260,000 new users registering each day. Often criticized for poor performance, MySpace has had to tackle scalability issues few other sites have faced. How did they do it?

Platform : Microsoft

• .Net: The Microsoft .NET Framework is a software component that can be added to or is included with the Microsoft Windows operating system. It provides a large body of pre-coded solutions to common program requirements, and manages the execution of programs written specifically for the framework. The .NET Framework is a key Microsoft offering, and is intended to be used by most new applications created for the Windows platform. http://en.wikipedia.org/wiki/.NET_Framework”>ASP.NET 2.0
• Windows
• IIS: Microsoft Internet Information Services (IIS; formerly called Server) is a set of Internet-based services for servers using Microsoft Windows. It is the world’s second most popular web server in terms of overall websites. As of May 2007 it served 31% of all websites according to Netcraft.[1] The servers currently include FTP, SMTP, NNTP and HTTP/HTTPS. http://en.wikipedia.org/wiki/Internet_Information_Services”>IIS
• SQL Server: Microsoft SQL Server is a relational database management system (RDBMS) produced by Microsoft. Its primary query language is Transact-SQL, an implementation of the ANSI/ISO standard Structured Query Language (SQL) used by both Microsoft and Sybase. http://en.wikipedia.org/wiki/Microsoft_SQL_Server”>SQL Server

What’s Inside?

 

• MySpace processes 1.5 Billion page views per day and handles 2.3 million concurrent users during the day
• Membership Milestones:
  - 500,000 Users: A Simple Architecture Stumbles
  - 1 Million Users:Vertical Partitioning Solves Scalability: Scalability is the ability to keep solving a problem as the size of the problem increases. Scale is measured relative to your requirements. As long as you can scale enough to solve your problem then you have scale. If you can handle the number of objects and events required for your application then you can scale. It doesn’t really matter what the numbers are. Scaling often creates a difference in kind for potential solutions. The solution you need to handle a small problem is not the same as you need to handle a large problem. If you incrementally try to evolve one into the other you can be in for a rude surprise, because it won’t work as you pass through different points of discontinuity. Scale is not language or framework specific. It is a matter of approach and design. Scalability Woes
  - 3 Million Users: Scale-Out Wins Over Scale-Up
  - 9 Million Users: Site Migrates to ASP.NET, Adds Virtual Storage
  - 26 Million Users: MySpace Embraces 64-Bit Technology
• 500,000 accounts was too much load for two web servers and a single database.
• At 1-2 Million Accounts
  - They used a database architecture built around the concept of vertical partitioning, with separate databases for parts of the website that served different functions such as the log-in screen, user profiles and blogs.
  - The vertical partitioning scheme helped divide up the workload for database reads and writes alike, and when users demanded a new feature, MySpace would put a new database online to support it.
  - MySpace switched from using storage devices directly attached to its database servers to a storage area network (SAN), in which a pool of disk storage devices are tied together by a high-speed, specialized network, and the databases connect to the SAN. The change to a SAN boosted performance, uptime and reliability.
• At 3 Million Accounts
  - the vertical partitioning solution didn’t last because they replicated some horizontal information like user accounts across all vertical slices. With so many replications one would fail and slow down the system.
  - individual applications like blogs on sub-sections of the Web site would grow too large for a single database server
  - Reorganized all the core data to be logically organized into one database
  - split its user base into chunks of 1 million accounts and put all the data keyed to those accounts in a separate instance of SQL Server
• 9 Million–17 Million Accounts
  - Moved to ASP.NET which used less resources than their previous architecture. 150 servers running the new code were able to do the same work that had previously required 246.
  - Saw storage bottlenecks again. Implementing a SAN had solved some early performance problems, but now the Web site’s demands were starting to periodically overwhelm the SAN’s I/O capacity—the speed with which it could read and write data to and from disk storage.
  - Hit limits with the 1 million-accounts-per-database division approach as these limits were exceeded.
  - Moved to a virtualized storage architecture where the entire SAN is treated as one big pool of storage capacity, without requiring that specific disks be dedicated to serving specific applications. MySpace now standardized on equipment from a relatively new SAN vendor, 3PARdata
• Added a caching tier—a layer of servers placed between the Web servers and the database servers whose sole job was to capture copies of frequently accessed data objects in memory and serve them to the Web application without the need for a database lookup.
• 26 Million Accounts
  - Moved to 64-bit SQL server to work around their memory bottleneck issues. Their standard database server configuration uses 64 GB of RAM.

 

MySpace Insiders

Rupert Murdoch
Chairman, News Corp.
As the creator of a media empire that includes 20th Century Fox, the Fox television stations, the New York Post and many other news, broadcast and music properties, Murdoch championed the purchase of MySpace.com as a way of significantly expanding Fox Interactive Media’s presence on the Web.

Chris Dewolfe
CEO, MySpace
DeWolfe, who is also a co-founder of MySpace.com, led its creation while employed by Intermix Media and continues to manage it today as a unit of News Corp.’s Fox Interactive Media. Previously, he was CEO of the e-mail marketing firm ResponseBase, which Intermix bought in 2002.

Tom Anderson
President, MySpace
A co-founder of MySpace, Anderson is best known as “Tom,” the first person who appears on the “friends list” of new MySpace.com members and who acts as the public face of the Web site’s support organization. He and DeWolfe met at Xdrive, the Web file storage company where both worked prior to starting ResponseBase.

Aber Whitcomb
Chief Technology Officer, MySpace
Whitcomb is a co-founder of MySpace, where he is responsible for engineering and technical operations. He speaks frequently on the issues of large-scale computing, networking and storage.

Jim Benedetto
Vice President of Technology, MySpace
Benedetto joined MySpace about a month after it launched, in late 2003. On his own MySpace profile page, he describes himself as a 27-year-old 2001 graduate of the University of Southern California whose trip to Australia last year included diving in a shark tank. Just out of school in 2001, he joined Quack.com, a voice portal startup that was acquired by America Online. Today, Benedetto says he is “working triple overtime to take MySpace international.”

Jason Feffer
Former vice president of operations, MySpace
Starting with MySpace’s launch in late 2003, Feffer was responsible for MySpace’s advertising and support operations. He also worked with DoubleClick, the Web site advertising vendor, to ensure that its software met MySpace’s scalability requirements and visitor targeting goals. Since leaving MySpace last summer, he has been working on a startup called SodaHead.com, which promises to offer a new twist on social networking when it launches later this year.

Duc Chau
Founder and CEO, Flukiest
Chau, as an employee of Intermix, led the creation of a pilot version of the MySpace Web site, which employed Perl and a MySQL database, but left Intermix shortly after the production Web site went live. He went on to work for StrongMail, a vendor of e-mail management appliances. Chau now runs Flukiest, a social networking and file-sharing Web site that is also selling its software for use within other Web sites.

Share This

If you're new here, you may want to subscribe to my RSS feed. Thanks for visiting!