On the fly image resize with Amazon S3 storage

Serving images with on-the-fly resize would give the store owner the maximum flexibility when it comes to the final design. We chose this approach for this reason also we didn't want to limit the designer to only a few variants of the product image (eg. small, medium, large).

What this means is that the theme designer has full control over the size of the image from the store template through the use of with/height url parameters. There are some extra parameters for adaptive resize and image alignment but I shall go through all of them in a later post.

<img src="{{ product.images.first }}?width=100&height=100" />

This will use the first image of the product and resize it to 100x100 pixels. When doing this we can't pre-process the images to store the thumbnail since we don't know what the width and height will be. We could parse the template and extract them but that would create a lot of overhead taking into account that the values could change at any time.

So we solved this issue another way. We've moved the image thumbnail generation into a separate asynchronous process. This allowed us to free the existing codebase from the extra business logic of creating thumbnails and the initial functionality remained the same: when there was a request for a smaller image we just resized it on the fly - but with a twist.

Every few hours a cron script will grep the API access log for image patterns and it will create an index containing the url of the image and the access count. We select the images with the access count greater than a preset threshold, we make another request with that url to get that particular version of the image and we store that thumbnail in S3.

After saving the smaller image in Amazon S3 we also add an entry in memcache. The key of the entry will be the url from the access log (eg. /myimage.jpg?width=100&height=100) and the value will be the url from Amazon S3 where we just stored the actual 100x100 version of the image.

This was the twist I've mentioned earlier: when we get a request to resize a particular image we look for that exact url in memcache and if we find anything we immediately redirect the user to that location.

You might think that a memcache restart will make the server start resizing images again because the redirect map is lost which is absolutely true and that you might have to re-process the access logs which would be one way of doing it albeit not the best one. To overcome this issue we did one extra action when we stored the images in S3 which was to add the initial url from the access log as the image meta header (you have an option of adding custom headers when saving objects in S3). This means that all the required information for this whole process to work is stored in S3, freeing us from any business logic on the application side (except for the small redirect part).

So now when we need to restart memcache (which we don't do very often anyway) we have a script that will list the items stored in S3, get the headers and rebuild the memcache index.

This process gives us a lot of freedom when handling images and it allows us to make full use of the distributed nature of Amazon S3.

comments powered by Disqus