Resource Data

Pushing Analytics Upstream: The Value of Data Processing at the Edge

Ozan Unlu is the founder and CEO of Seattle On-board delta, an edge observability platform. Previously, Unlu served as a senior solutions architect at Sumo Logic, a software development manager and program manager at Microsoft, and a data engineer at Boeing. He holds a BS in nanotechnology from the University of Washington.

For years, organizations have leveraged analytics to transform data into information and then into action. Traditionally, many have relied on an approach known as “centralize and crawl,” where they bring all of their application, service, and system health data into a central repository for indexing and processing.

In recent years, this approach has become increasingly problematic in several respects, including the difficulty of coping with the explosion of data volumes and, therefore, of monitoring costs. As teams strive to leverage all of their data to optimize the overall health of the department, they find themselves forced to make painful decisions about which data to analyze and which to neglect – a very risky proposition given the temperamental nature of performance issues.

As a result of this decision, teams often don’t have the data they need to anticipate or resolve issues quickly. This shows that despite advances in technology and strong industry investment in resilience, outages persist, with the number of outages lasting for more than 24 hours. increasing substantially.

Here, we’ll explore how a new approach to application monitoring solves this problem. Rather than compressing and shipping huge volumes of data to compute downstream resources, this new approach turns traditional monitoring on its head. Now it is possible to push your calculation to your datasets. Pushing data analytics upstream — or processing data at the edge — can help organizations overcome certain challenges and maximize the value of their data and analytics.

Analyze all application and system health data at source

Geographically bringing compute resources closer to users reduces latency and helps organizations deliver significantly better user performance as well as the ability to monitor new services without creating bottlenecks in downstream systems and on-premises data centers . Simply put, teams no longer need to predict in advance which datasets are valuable and worth analyzing in order to troubleshoot issues that may impact the customer/user experience.

Pushing analysis upstream to the periphery can help organizations avoid such dilemmas by addressing everything application, service, and system health data at different points on the edge, simultaneously and broken down into smaller chunks. This allows organizations to have an effective eye on everything their data, without having to overlook even a single dataset.

Protect and boost conversions

For transaction-intensive online services — e-commerce businesses and travel booking sites, for example — high-performance applications and systems are the lifeblood of business. When these apps stop – or even slow down, by just a few milliseconds – the result is a noticeable impact on conversion rates. According statistics, the highest e-commerce conversion rates occur on sites with web page load times between 0 and 2 seconds, and with each additional second of load time, website conversion rates drop an average of 4.42%. These statistics also note that a site which loads in one second has a conversion rate three times higher than a site that loads in five seconds.

In this context, the requirements for Mean Time to Detection (MTTD) and Mean Time to Response (MTTR) are extremely slim, essentially zero. As noted above, pushing analytics upstream allows teams to identify and address anomalies more proactively, while intuitively pinpointing the exact location of growing hotspots or a particular infrastructure or application that is failing. performs there. Teams can resolve issues much faster, ideally before user performance is impacted in the first place – which is perhaps the most important step in backup conversions.

But when it actually comes down increasing conversions, applications, and system state data aren’t the only types of data that can benefit from more advanced analytics. Today, nearly three out of four dollars spent on online purchases are made via a mobile device. A matter of nanoseconds can be the difference between capitalizing on a site visitor’s fleeting attention span – or not. When customer behavioral data is processed at the edge, bypassing long-distance communication streams to the cloud, an organization can become much more agile and instantaneous in delivering highly personalized, high-velocity marketing that fuels conversions.

Control monitoring costs

The old “centralize and analyze” approach involved routing all application and system health data to warm, searchable, and relatively expensive retention tiers. Many organizations experience sticker shock when they run into data usage limits and, in many cases, unknowingly exceed them. An alternative is to pre-purchase more capacity than might actually be needed, but small businesses in particular cannot afford to spend money on capacity that they end up not using. Another downside is that the more data there is in a repository, the longer the expected lookup time tends to be.

Against the backdrop of these challenges – and as edge processing grows – data stores must follow suit. Gartner estimates that by 2025, 70% of organizations will shift their analytical approaches from “big” to “small and large”, and a key factor is that the edge provides enormous flexibility and creates space for more real-time data analysis on larger data volumes. When analytics are pushed upstream, organizations process their data immediately, directly at the source. From there, teams can move their data to a more cost-effective storage option in the cloud, where it remains searchable.

Whether driven by compliance requirements, the desire to extract historical data for further analysis, or something else entirely, there will be occasions when teams need access to all of their data. In those cases, it will be there, easily and easily accessible to anyone who needs it, without exhausting or going over budgets in the process.

Conclusion

As data volumes grow exponentially, processing data at the edge becomes the most practical way to profitably and comprehensively leverage an organization’s rich data. Now teams can realize the potential of analyzing all their data to ensure high performance, availability, and robust user experiences.