Many applications and systems emit logging information that can be used for performance or diagnostics analysis.
This type of data is typically a combination of structured and semi-structured data – but not always. This log data can be sent out in many different ways and from numerous sources, such as applications, VM operating systems, and even the Azure platform itself. To make sense of the data, you need to collate it and then be able to visualize it.
Again, we could use Azure Blob or even Azure File Storage. Then, an external logging viewer could be used to make sense of the data, or you could opt for Azure’s logging analytics solution – Azure Log Analytics, which uses the Kusto query language and enables data to be easily formatted and visualized.
We will cover Log Analytics and Application Insights, which is an application-specific monitoring and logging tool, in more detail in Chapter 15, Designing for Logging and Monitoring:
Figure 9.4 – An example logging solution
The next use case we will consider is a programming pattern for decoupling application components from each other.
Queuing mechanisms
Sometimes, data is transient and only used by the application to keep track of state or process information. This type of data still needs to be persisted but only temporarily – and often, a mechanism is required to ensure the data is removed once the process or task has been completed so that it isn’t duplicated.
A typical application development pattern is message queuing, which does just that – a task that needs to be performed is recorded, along with the relevant parameters, by one part of a solution, to be read and actioned by another component.
You can build your application to manage this queuing process, or you can use a technology that manages it for you.
Azure Storage accounts contain tables and queues. Tables could store temporary messages, but the application would need to manage the process entirely. Storage queues are a better option, as they are built specifically for this task and provide messaging-specific capabilities.
We will cover queuing and messaging from a technology perspective in more detail in Chapter 11, Comparing Application Components; however, the following diagram shows a high-level view of how this might work:
Figure 9.5 – A messaging architecture example
Many Azure services are built on top of these native storage mechanisms, such as Azure SQL, Azure Cosmos DB and Azure Storage accounts. However, another use case is specific to Azure VMs. VMs use virtual hard disks for the operating system, and as data disks, we will examine these next.
Archiving
Sometimes, data needs to be stored for long periods for compliance and legal reasons, or simply just as a long-term record. Savings can be made by separating this data from live data.
Azure Storage accounts support differing access tiers – hot, cold, and archive, which we looked at in detail at the beginning of this chapter.
By considering how often data will be accessed, and with a well-thought-out data life cycle management plan, you can achieve the best cost to performance ratio. So far, we have looked at these requirements from a pure data perspective. Next, we will examine a different use case for storage – that is, VM disk storage.
Leave a Reply