The Case for Data Gateways

 Microservices influence the two-dimensional data layer. First, it requires a separate database per microservice. From a practical implementation standpoint, this can range from a stand-alone DB instance to separate schemas and logical groupings of tables. The main rule here is that only one microservice owns and touches a set of data. And all data is accessible through proprietary microservice APIs or events. The second way that a microservices architecture has influenced the data layer is the proliferation of the data warehouse. Likewise, by allowing microservices to be written in different languages, this architecture allows every microservice-based system to have a polyglot persistence layer. With this freedom, one microservice can use a relational database, another can use a document database, and the third microservice uses an in-memory key-value store.


While microservices give you all this freedom, it still comes at a cost. It turns out that operating a large number of data warehouses comes at a cost for which existing tools and practices were not prepared. In the modern digital world, storing data reliably is not enough. Data is useful when it becomes knowledge and for that it must be accessible in a manner controlled by many. AI / ML experts, data scientists, and business analysts all want to dig deep into data, but application-centric microservices and their data access models are not designed for these data-hungry demands.


API and Data Gateways Offering Similar Capabilities at Different Layers


This is where data gateways can help. A data gateway is like an API gateway, but it understands and acts on the physical data layer rather than the network layer. Here are some areas where data gateways differ from API gateways.


Abstraction


An API gateway can hide deployment endpoints and help update and restore services without affecting service consumers. Likewise, a data gateway can help extract a physical data source, its details, and help modify, migrate, and dismantle without affecting data consumers.


security


An API manager protects endpoints of resources based on HTTP methods. A mesh of services is secured based on network connections. But none of them can understand and secure the data and the way it travels through it. A data gateway, on the other hand, understands and acts on the different data sources and the data model. You can apply RBAC per row and column of data, filter, hide, and clean up individual data items as needed. This is a more detailed security model than networking or API-level security of API gateways.



Climbing


API Gateway Services can perform service discovery, load balancing, and help scale services through an orchestrator such as Kubernetes. But they can't scale the data. Data can only be scaled by replication and caching. Some datastores can replicate in cloud native environments, but not all. Specially designed tools like Debezium can capture changed data from data warehouse transaction logs and enable data replication for scale and other use cases.


A data gateway, on the other hand, can speed up access to all kinds of data sources by caching the data and providing materialized views. You can understand queries, optimize them for the capabilities of the data source, and produce the most efficient execution plan. The combination of materialized views and the flow nature of change data capture would be the latest technique for scaling data, but cloud native implementations are not yet known.


Federation


In API management, composing responses is a common technique for aggregating data from several different systems. In the data space, the same technique is called federating heterogeneous data. Heterogeneity is the degree of differentiation in various data sources, such as network protocols, query languages, query capabilities, data models, error handling, transaction semantics, etc. A data gateway can support all of these differences as a transparent and transparent data federation layer. .


Diagram first


API gateways enable the development of first-contract clients and services with specifications such as OpenAPI. Data gateways allow consumption of first schema data based on the SQL standard. An SQL schema for data modeling is the OpenAPI equivalent of APIs.

Comments

Post a Comment

Popular posts from this blog

Outsourcing

  Small businesses increasingly rely on information technology (IT) to catapult productivity, improve efficiency and offer them a competitive advantage. But in the event of computer problems, small businesses suffer. In fact, most small businesses cannot afford to have a computer scientist on their payroll, and hiring a computer can take hours or even days. This is bad news.   The good news is that it is becoming increasingly convenient for small businesses to outsource their IT help and level the playing field with big business owners. We will explain the essential differences between the Outsourced Help Desk and the maintenance of an IT guru in the staff and we will let you know the advantages of each option.   Advantages of Outsourcing Your IT Help Desk    One of the main advantages of IT help desk outsourcing is the potential for saving money. IT helpdesk outsourcing saves money on payroll because you won't have to pay the salary or benefits of a computer scientist. Also, if e
Read more

What is Managed Wi-Fi and Why do You Need It?

  The days of spotty connectivity and internet cables are long gone; Today's people expect instant connectivity, anywhere, anytime. And when it comes to a business environment, fast, easily accessible Wi-Fi connectivity isn't just about making business easier, it has tangible benefits like productivity, efficiency, and hard work. ”Improved equipment. Over the past decade, the enterprise WLAN landscape has changed dramatically; From managing a few access points (APs) to providing wireless access points for a small number of employees, office Wi-Fi is now expected to provide comprehensive company-wide coverage for hundreds and thousands of users. Managed WLAN Defined To manage this new scale of demand for wireless connectivity and ensure that many employees, guests and users enjoy instant and secure connectivity, companies must hire professional service providers to manage their wireless LAN. Cloud-based Wi-Fi services are the way to go; The service provider remotely manages the
Read more

Three Key Responsibilities of the IT Help Desk

  For many companies, the IT support service acts as the facade or front for the IT organization. Manages employee problems and service requests, managing most, if not all, of the communications between end-users and the IT organization. The helpdesk is a key player in providing IT services to the company and has a number of responsibilities that must be taken to succeed.   In this blog post, I explain three of the main IT Help Desk Services responsibilities and offer suggestions on how to best meet them.   Deliver Great Customer Service   First of all, at the moment employees are experiencing a superior customer experience in their personal life (and they carry the same expectations in the workplace), the number one responsibility for any IT Help Desk Services should be to meet the needs of end-users while providing excellent customer service.   That being said, it is easy for IT organizations to overcome this obstacle because they are overly focused within, the help desk policies a
Read more