The typical big data infrastructure can be likened to a Frankenstein's monster of legacy hardware, cloud connections, and storage environments. The data resides in different silos in every conceivable location. This makes data accessibility difficult, hampering analysts' productivity. Analysts shouldn't have to worry about where their data is, where it's moved, or whether their company decides to move to the cloud, they just want quick access.

As big data piles continue to evolve and data sources change, how can data users keep spinning the wheels of the system despite IT outages? The answer to this question lies in the concept that we call 'consumption layer' (also known as abstraction layer, semantic layer, query structure, etc.).

‍

What is the consumption layer?

Simply put, the consumption layer is a tool between your data users and data sources. This layer takes an SQL query as input (from a BI tool, CLI, or ODBC/JDBC, etc.) and executes that query as quickly as possible, querying the necessary data sources, and even combining data between sources as needed. Ideally, this layer should be highly scalable and have an MPP design.

Thanks to this design, your analysts can access data from anywhere, without any ETL or data movement. Even better, they don't even need to know if the data is in Teradata, Hadoop, or S3. They only get the results, which is all they really care about.

‍

Benefits of the consumption layer

A consumption layer brings many benefits to the institution. First, it isolates users from any data migration and eliminates many of the risks inherent in data movements. This is becoming increasingly important as data strategies evolve and organizations update their databases or move to the cloud.

Instead of embarking on a massive data migration effort (an extremely painful process that can take months or even years to complete), database administrators can move (or extract data from these sources) to any of the data sources/new data sources at any time, without disrupting the operations of end users.

A consumption layer also frees network architects from much of the complexity involved in building and maintaining a solution stack. Instead of having to plan years later, architects have the power to add and subtract data sources as they see fit, while also taking advantage of existing infrastructure that requires a lot of time and money to build.

As a result, a consumption layer can keep front-end users informed about back-end operations because it can access data from any source at any location. Starburst Enterprise was built with this capability in mind. Neither Starburst Enterprise cares about the format and location of the analyst data that uses it for SQL queries — both just want to calculate large numbers and get results quickly.Given that migrations to the cloud will continue to increase, Starburst Enterprise is ideal for ensuring end users can stay productive.

‍

Get rid of the limitations of the technology provider

Unfortunately, most of us are very familiar with this story... Database providers require you to store much, if not all, of your data in their own data stores, usually in a proprietary data format. They also desire to bind you for several three-year cycles, and in the meantime sharply limit your agility and freedom. As your data grows, so do your fees. This is one of the biggest risks you face when building your big data platform.

Using Starburst Enterprise as your consumption layer immediately solves this dilemma. Since Starburst Enterprise can connect to almost any data source, you effectively commoditize your storage and choose solutions that are right for your business without fear of the limitations of the technology provider.

‍

Quickly access data with your existing investments

With an appropriate consumption tier like Starburst Enterprise, organizations can continue to leverage the infrastructure they have today without worrying about the problems that the technology provider brings.

IT teams can identify and deploy the optimal combination of technologies that deliver the best value and return on investment to leverage the different strengths across each platform. It can also properly prepare and execute migrations to the cloud over time.

Starburst Data is neither a database supplier nor a storage company. As a result, Starburst Enterprise does not care about where the data is held or its format. It only helps you make your analysis as quick and easy as possible.

‍