Unlimited Storage with WP Engine’s Proprietary LargeFS

Recently, we snuck something new to our pricing page and many of you began to take notice on Twitter and in emails.  What we snuck in is a pretty sweet feature called LargeFS^TM Storage. LargeFS is a new feature that we developed to store and transfer huge amounts of data, and integrate it into the standard WordPress experience.

We built LargeFS using Amazon S3, the industry-standard mechanism for storing and serving any number of files of any size, from 10 million icons to a million movies.  It’s inexpensive and bills per-use.  And because S3 is already redundant, it also solves the issue of backup.  S3 really is the perfect tool for storing and serving large files or large quantities of files.

Basically, LargeFS is a way to expand your server storage from your normal server onto S3, where storage is practically infinite.

We had to spend some time coding away to solve the issue of integrating WordPress with S3.  We wanted to get everything just right.

Historically it was challenging to cleanly integrate the standard WordPress UI with the scale of S3. For example, there were often problems uploading images through the media uploader and also retaining support for other plugins and themes which depended on those files being on the local filesystem.

LargeFS retains the advantages of S3 while mitigating the downsides, in many respects simply because it’s running at WP Engine.  Just like the rest of our infrastructure,  LargeFS is fully managed by WP Engine — all you have to do is create an S3 account and set us up with credentials to read and write in it.

How LargeFS works:

1. We transparently move content off the server’s filesystem onto S3.  This happens during low-traffic periods and is fully automated.
2. We move files only once they’re sufficiently old, so it doesn’t interfere with other server operations and plugins.
3. Our fast, front-end caching load balancers understand which paths are controlled by LargeFS, and automatically serve content either from the server’s filesystem or from S3, depending on where the file is.
4. Uploading new files or modifying files are available instantly; there’s no lag and no caching delay.
5. Larger files are 301’ed directly to S3 for better throughput and scalability under traffic loads, and also those URLs in HTML are automatically rewritten to go straight to S3, so under normal conditions this is as fast as possible for the end user.
6. Smaller files which are accessed frequently are automatically cached by our caching load-balancers and will also be placed on our CDN automatically, so popular content is as fast as possible and doesn’t have any of the speed issues with S3.
7. All of the above is tunable for specific customers, and even for specific paths within an installation.

Unlimited Means Unlimited Again

What this means for you is that any hosting plan at WP Engine literally has unlimited space.  “Unlimited” has become a hosting buzzword that doesn’t mean anything because in many cases, and on many hosts, “unlimited” simply wasn’t.  There was a file limit to your plan and you hoped you never ran into it.  With LargeFS, we’re “re-claiming” the word unlimited, and adding yet one more feature to our already industry-leading Managed WordPress Hosting Platform.