When hosting your data in the Azure cloud, there is a wide variety of storage services you can choose from. Often, you need to combine multiple of these storage services to build a fully functional cloud system. In this article, you will learn about the main Azure storage services, including tips for optimizing and securing big data in the Azure cloud.
File Storage (AFS)
File storage is a service you can use to store file shares. It uses the Server Message Block (SMB) protocol to allow distributed access to files. AFS is useful for lift and shift migration of file shares, centralization of development assets, and creating a central resource for configuration or log files.
Queue Storage
Queue Storage is a service you can use to ensure reliable, persistent, and asynchronous messaging between your various services. It supports strings or arrays in a variety of formats, including XML and CSV. Stored messages are held for up to a week and each message can be up to 64KB. You can store a total of 200TB worth of messages.
Azure Storage Tips
Once you’ve selected the storage services you want to use, there are multiple ways you can increase your ROI. Below are some good methods to start with.
Carefully Select Storage Account Type
When you create storage accounts you need to specify the type you are using. In general, this type is General-purpose v2. This type works for Blob, File, Queue, Table, Disk, and gen2 Data Lake storage.
You also have the option to use either Block Blob Storage or File Storage for premium accounts. These account types are recommended for enterprise and high-performance applications.
Vary Your Storage Access Tiers
Depending on the access requirements for your data, you can optimize costs and performance with various access tiers. You can either set these tiers by file or for the account as a whole. Tiers include hot, cold, and archive storage. As you move from hot to archive your storage costs decrease but your access costs and latency increase. Access tiers are available in Blob and File services.
Optimize Storage Accounts
You can have multiple storage accounts in Azure to expand your storage limitations beyond that of a single account. The total number of accounts you can have depends on the service you’re using and the region of use. If you find that you’re hitting the maximum number of accounts and still need more resources, consider the following:
-
If using unmanaged disks with virtual machines try switching to managed disks. These disks scale automatically and eliminate the need to create or manage additional accounts
-
If using storage accounts to isolate data, try switching to individual blob containers. You can achieve the same isolation with containers by applying individual role-based access controls (RBAC) to each container.
Enable Parallel Uploads
When you need to upload large volumes of data to blob storage you can save significant time by using parallel uploads. Enabling this feature can help you reduce upload times by up to 200%. When setting parallel uploads, you can get the greatest performance gain by setting your ParallelOperationThreadCount to eight times the number of available cores.
Leverage Snapshots
You can use snapshots (point-in-time backups) to increase the availability of your data. For example, you can create multiple snapshots of a single blob and distribute snapshots to your customers. This gives them access to the data without overloading your access limits. You can also use snapshots for your read operations and reserve storage operations for writing, provided you don’t need real-time data syncing.
Encrypt Your Data
Regardless of what data you store in Azure, you need to be sure to encrypt it both in transit and at rest. Most storage services include built-in encryption that you only need to activate.
With this encryption, you can either have Azure manage encryption keys for you or you can use an add-on service to manage keys independently. If services do not include built-in encryption, ensure that your data is being transferred over secure channels, such as a virtual private network.
Optimize Blob Bandwidth and Operations
Each of your blobs can handle up to 500 requests per second. However, if you have multiple clients accessing the same blob, this may not be enough. If this is the case, consider using the BlockBlobStorage account mentioned above. This account type grants access to a higher number of I/O operations per second (IOPS).
Alternatively, you can use a content delivery network (CDN) to distribute your operations. CDNs cache content on distributed servers and serve content to users based on whichever server is closest.
Conclusion
Azure provides a wide range of enterprise-level big data, typically hosted on a number of storage services, such as blob storage, table storage, file storage, and queue storage. If you’re using more than one service, be sure to use the Azure Total Cost of Ownership (TCO) Calculator to estimate and optimize your storage costs. Be sure to leverage snapshots to reduce costs, and always encrypt your data to keep it secure.
