DSpace Gains Cloud Storage Layer
Submitted by on Fri, 2016-01-22 10:52
From Scott Siddall, Longsight Longsight has developed enhancements to the DSpace storage system, providing a system for cloud storage of assets, that has now been officially merged into the DSpace 6 code base. Sites using DSpace can now more easily scale their DSpace instances to the cloud through the addition of a storage connector that uses Amazon Web Services Simple Storage Service (AWS S3). Using S3 storage provides institutions a low price, extremely durable, highly available, and infinitely scalable storage facility provided by S3.
One of the growing pains that large DSpace installations often encounter as they increase scale is making multiple storage systems appear as one, and to manage the durability of the objects they each store. With the addition of S3 to DSpace, a site can migrate all of their DSpace assets to S3, vastly simplifying the management of stored objects. For sites that don't plan on using S3, the improved storage system in DSpace still offers them the feature of a new assetstore migration tool. Sites could be in a situation where to grow their DSpace storage, they add an additional hard drive, and store new files there. The migration tool, once you have a combined larger disk, would allow you to properly migrate bitstreams off of one storage system, and over to another, thus allowing sites to defragment their storage from multiple storage systems.
"One of our challenges has been the 1 TB upper limit on EBS stores without spinning up another EC2 instance just to do RAID. We're only using 20% of the EBS right now, but expect to grow as we continue laying out ingestion infrastructure. Not to mention we have to pay the full cost for the EBS. It's not that expensive, but S3 is pay as you consume and has no space limit.So all that is to say an S3 asset store would be welcome."