Data Management & Analytics: Legacy Challenges and Technological Trends

17/03/20
ALL articles
By Alexey Utkin
Principal Solution Consultant, Finance Practice
Share
Data Management & Analytics: Legacy Challenges and Technological Trends

In the previous article, we outlined a few business imperatives that affect the climate in data management and analytics today: 

  • The volume of data grows; data becomes more granular; businesses drown in terabytes of real-time and alternative data. 
  • Advanced analytics thrives on artificial intelligence and machine learning. 
  • More companies acknowledge that data-driven insights are a source of competitive advantage. Accordingly, they design and implement data analytics and self-service BI tools in collaboration with and to satisfy the needs of internal business stakeholders, in addition to data-driven applications, products, and services for external clients.

Though topical today, these imperatives may become irrelevant tomorrow in the world where change is the only constant. Business agility comes to the fore in all realms. To transform the threats of the new era into opportunities, businesses must remain flexible and quickly adapt to change. However, legacy data landscapes and archaic architectures in analytics systems still hinder business agility in many companies. 

Major Issues in Archaic Data Analytics Platforms

Most legacy data architectures are inflexible (bound to specific use cases, like batch processing and reporting), unable to link different data sets and cope with the growing volume, variety, and velocity of data. Plus, they are expensive. 

Legacy data analytics platforms and techniques make businesses less  competitive

Like the majority of software designed and developed for large organizations, data analytics platforms were historically built on top of monolithic, on-premise architectures -- slow, clunky, fragile, and difficult to secure. Before the Big Data era brought about low cost and elastic storage and compute resources, reporting and analytics workflows were structured for the sole purpose of data processing within the required time. As technology and computing evolved, such workflows became legacy. Batch processing and pre-calculations can now be replaced by ad-hoc and just-in-time analysis, which allows to simplify data modelling and transformation and quickly accommodate changes. 

Most legacy platforms, often scaled vertically, and related infrastructures fail to meet the high standards for data storage, management, and governance now in place in many industries, like healthcare and finance. Such platforms also tend to have high licensing and maintenance costs.

Old data analytics platforms fail to accommodate large volumes and new sources of data

Companies are creating data at an unprecedented rate. Master and reference data, semantic data, knowledge graphs, real-time streaming data, cloud-enabled data, Big Data, IoT and smart device data, wearable data, social media data, and behavioral and lifecycle data are just a few types of data currently being generated by people and machines. However, most archaic analytics platforms fail to process the massive data volume and new data types. With the growing variety of data, striving to have 100% of it well structured and curated is the wrong goal.

graphics

Modern businesses have a growing appetite for absorbing even “alternative” or “dark” data -- that is, unstructured, untagged, and untapped data accumulated by computer network operations, which is generally not used for analytics purposes. Although challenging to analyze, dark data might yield truly valuable insights into customer preferences and behavior and business performance in general. Unstructured data, including Big Data, calls for a brand new technology and toolsets to collect, store, and analyze it on a company-wide level.

Forrester described an approach they call BI Governance: on top of the well-structured and curated data, businesses enable users to quickly evaluate and utilise new data sources. Once certain data is proven valuable and relevant for a wide group of users, they turn it into a well- structured pipeline. A common benchmark for this approach is that there is much more unstructured data than structured. In other words, businesses unlock huge potential in the use of certain data before an EDW team integrates and models new datasets and implements corresponding use cases.

Technological Trends in Data Management and Analytics

To address common issues and other drawbacks of archaic analytics setup, there is an increasing trend to modernize and optimize existing data analytics platforms or build entirely new, advanced, and future-proof ones. More companies will do so in the coming years.

1. Cloudification of data analytics platforms

Omnipresent Big Data and cloud technologies underpin the modernization of data analytics platforms in most industries. Ponderous or inert systems that are difficult to deploy, nearly impossible to change, and expensive to maintain are being replaced by lightweight open source technologies. More and more companies are either rebuilding their legacy on-premise data platforms, or completely abandoning them for more flexible cloud environments, such as AWS, Microsoft Azure, IBM Cloud, Google Cloud Platform.

2. Flexible modular architectures supersede clumsy old-fashioned ones

In the past, implementing a data analytics solution with data warehouse architecture could take years. The process was bulky and complex; the requirements kept changing over time, and change requests took ages to implement. Such systems were very high in associated costs for both development and maintenance. Today, analytics applications are deployed using an easier, adaptable iterative methodology.

Modern architectures of data analytics platforms require flexibility -- to evolve with the rapidly changing demands, and business agility – to update and replace technologies and accommodate future data sources and on-demand data sets. These architectures are often decoupled and/or modular, and they rely extensively on microservices and APIs.

3. Popular technologies for advanced analytics

Modern technologies, like machine learning powered AI or Big Data, enabled through data analytics, are a core piece of advanced analytics, and they take center stage in business, tech, and data strategies. These technologies accommodate different data sources, including social media, and can be used for rapid on-demand analysis. Businesses often need to test an analytical hypothesis on a custom, on-demand data set. Technologically, doing so is relatively easy through some proofs-of-concept with an ML model. If the analytical pattern proves helpful, it enhances all of the business’s pre-existing analytical data sets. 

image

At DataArt, we can help your business unlock advanced analytical capabilities with a future-proof data analytics platform. Our teams have proven experience designing and building such platforms from scratch, as well as modernizing existing systems for organizations of any size. We consult on data integration and management, addressing three dimensions: technology & architecture, operational modelling, and analytical use cases and scenarios.

Sign Up for Updates!

Subscribe now to receive industry-related articles and updates

Choose industries of interest
Thank You for Joining!

You will receive regular updates based on your interests. No spam guaranteed

Add another email address
Read more
Sign Up for Updates!
Choose industries of interest
Thank You for Joining!

You will receive regular updates based on your interests. No spam guaranteed

Add another email address
Google_Ads_1200_628_logo.jpg
REGISTER FOR A WEBINAR WEBINAR: The Role of Data Lakes in Modern Data Platforms
WHAT YOU WILL LEARN

Envision practical steps to modernize your data architecture with data lakes. Learn more about cloud data analytics platforms and data lakes.
Register now!


WHEN: April 9, 2020, 12.00-13.00 EDT

Welcome
We are glad you found us
Please explore our services and find out how we can support your business goals.
Let's Talk