This use case was provided by AppFirst.
Engine Yard is a leading Platform-as-a-Service (PaaS) provider from San Francisco who partners with their customers to plan, build, deploy, manage and scale applications in the cloud. Engine Yard delivers a trusted, complete platform solution that enables customers to focus on their core business of creating applications (SaaS) rather than on operations.
Customers count on Engine Yard to ensure the platform supporting their applications is running optimally and to offer enterprise class operations support and professional services. Their goal is to establish best-in-class insight, to gain competitive services and support advantage. They began scoping a new project that would provide increased operational insight into its own support and IT operations teams and provide improved operational vision to its customers. Engine Yard and its customers would have visibility into the runtime attributes of servers and processes. This deep visibility is critical to guiding effective operations and planning future development.
Requirements include the ability to diagnose system bottlenecks, improve key business performance indicators, increase application performance and decrease support response times. The solution must collect some phenomenally Big Data to make this happen; metric, log, alert and security information from every server within the Engine Yard PaaS product, as well as from each server within its other products. The data then needs to be stored in a central database and then finally moved to a data warehouse for analytics and reporting. This Big Data project demanded information to be collected for 11 separate categories of metrics.
Engine Yard considered writing an in-house solution or patching, extending and linking several open source solutions, but both were quickly deemed too time-consuming. Instead they searched for a vendor that could handle the maximum number of categories possible and decided on AppFirst, an APM and operational intelligence company based in New York City. AppFirst was able to handle 10 of the 11 required categories, as opposed to other solutions that could handle three at most.
AppFirst helps Engine Yard collect in excess of 100 different health, operations, application and business metrics on a server-wide or per-process level across all of their products; including CPU, memory, disk and network statistics. They also collect and monitor system logs, Nagios checks and enhanced StatsD metrics. StatsD helps to provide both Application as well as Business process metrics. Using this data in real-time, users can quickly troubleshoot problems, stay on top of system status and communicate issues quickly with the rest of the organization.
Beyond that, retention of these metrics allows for correlation and analysis of data on application, server and enterprise levels. Being able to acquire system state at a point in time is a powerful tool for planning upgrades and deciding what features to add to an application.
Armed with this knowledge, Engine Yard support staff can now quickly resolve performance issues without impacting customers application performance. Shortly after Engine Yard went live with this feature, peak loads of approximately 1,000 servers were monitored. The feature proved so popular with customers that Engine Yard soon added monitoring for an additional 9,000 servers.
Partnerships for integrated solutions are required to meet the full scope of customers requirements. The key is to find companies that have the same passion as you do for solving customers problems, the flexibility to adjust their engineering roadmap when needed and the drive to keep accelerating their own product solutions.
