IDC's Worldwide Data Integration and Integrity Software Taxonomy, 2017

Loading ...
Techceler...
Business & Economics
Integration
<p>April 2017, IDC #US41715317
TAXONOMY
IDC's Worldwide Data Integration and Integrity Software 
Taxonomy, 2017
Stewart Bond
IDC'S WORLDWIDE DATA INTEGRATION AND INTEGRITY SOFTWARE TAXONOMY
FIGURE 1
Data Integration and Integrity Software Primary Segments
Source: IDC, 2017
©2017 IDC
#US41715317
2
DATA INTEGRATION AND INTEGRITY SOFTWARE TAXONOMY CHANGES FOR 2017
Three primary changes have occurred since the last published taxonomy for the data integration and 
access (DIA) software functional market (see IDC's Worldwide Data Integration and Access Software 
Taxonomy, 2015, IDC #255856, June 2015):
 Name change from data integration and access software to data integration and integrity 
software
 Consolidation of the general data quality and domain-based matching and cleansing market 
segments into one data quality software segment
 Addition of the self-service data preparation software segment
Table 1 maps these changes as they were made in 2016, and how they relate to the 2015 market 
definition.
TABLE 1
Summary of Data Integration and Integrity Software Functional Market Changes 
for 2016
2015 Market
2016 Market
Comments
Application development 
and deployment
Data integration and 
access software
Data integration and 
integrity software
Renamed and expanded the scope of 
this market (Self-service data 
preparation is added as a submarket. 
Overall market name changed [to data 
integration and integrity] to emphasize 
the growing importance of data quality.)
Data integration and 
integrity software
General data quality 
segment consolidated 
with domain-based 
matching and cleansing 
segment
Data quality segment
Market consolidation – software vendors 
with general data quality software are 
investing in domain-based matching and 
cleansing and vice versa
Self-service data 
preparation segment
New segment describing data access 
and preparation tools suitable for 
business users
Source: IDC, 2017
Data is core to digital transformation, and data without integrity will not be able to support digital 
transformation initiatives. Data with integrity is data that is trusted, available, secure, and compliant 
across the enterprise IT environment, regardless of whether data is persisted on-premises or in the 
cloud. Data integration software vendors have the tools, intellectual property, and products that can 
improve the integrity of data. Integrity has become the secondary focus of solutions in this market, and 
for this reason, IDC changed the name of the software market from data integration and access to data 
integration and integrity (DII).
©2017 IDC
#US41715317
3
Until 2016, IDC tracked data quality software spend within the data integration and access software 
functional market as two subsegments: general data quality, and domain-based matching and 
cleansing software. These two segments are consolidating as software vendors in the general data 
quality segment add domain-based matching and cleansing software, and conversely, domain-based 
matching and cleansing software vendors are adding general data quality software to their portfolio. 
While in some cases the products continue to be separate, it is becoming increasingly difficult to track 
spend as the lines are blurring. As the market consolidates, so must the segments that IDC is tracking.
A lot happened on the way to big data insights, including a need to put data preparation into the hands 
of business users. 2015 saw a surge of activity in a self-service data preparation software category, 
and therefore, IDC started tracking spend in this category in 2016. In many cases, the purpose of 
software in this segment is to prepare data for analysis, and because functionality is similar to extract, 
transform, and load (ETL), IDC decided to record and track revenue within the DII software market 
instead of the business analytics market. 
TAXONOMY OVERVIEW
Data integration and integrity (DII) software enables the access, blending, movement, and integrity of 
data among multiple data sources. The purpose of data integration and integrity software is to ensure 
the consistency of information where there is a logical overlap of the information contents of two or 
more discrete systems. Data integration is increasingly being used to capture, prepare, and curate 
data for analytics. It is also the conduit through which new data types, structures, and content 
transformation can occur in modern IT environments that are inclusive of relational, nonrelational, and 
semistructured data repositories. 
DII software employs a wide range of technologies, including, but not limited to, extract, transform, and 
load; change data capture (CDC); federated access; format and semantic mediation; data quality and 
profiling; and associated metadata management. Data access is increasingly becoming open with 
standards-based APIs; however, there is still a segment of this market focused on data connectivity 
software, which includes data connectors and connectivity drivers for legacy technologies and 
orchestration of API calls. 
DII software may be used in a wide variety of functions. The most common is that of building and 
maintaining a data warehouse, but other uses include enterprise information integration, data 
migration, database consolidation, master data management (MDM), reference data management, 
metadata management, and database synchronization, to name a few. Data integration may be 
deployed and executed as batch processes, typical for data warehouses, or in near-real-time modes 
for data synchronization or dedicated operational data stores. A data migration can also be considered 
a one-time data integration. Although most applications of DII software are centered on structured data 
in databases, they may also include integrating data from disparate, distributed unstructured or 
semistructured data sources, including flat files, XML files, JSON, and legacy applications, big data 
persistence technologies, and the proprietary data sources associated with commercial applications 
from vendors such as SAP and Oracle. More recently, an interactive form of data integration has 
emerged in the application of self-service data preparation.
The data integration and integrity software market includes the submarkets as illustrated in Figure 2
and discussed in the Definitions section that follows.
©2017 IDC
#US41715317
4
FIGURE 2
IDC Data Integration and Integrity Software Taxonomy Model
Source: IDC, 2017
DEFINITIONS
Bulk Data Movement Software
This software, commonly referred to as extract, transform, and load software, selectively draws data 
from source databases, transforms it into a common format, merges it according to rules governing 
possible collisions, and loads it into a target. A derivative of this process, extract, load, and transform
(ELT), can be considered synonymous with ETL for the purpose of market definition and analysis. ELT 
is an alternative commonly used by database software vendors to leverage the inherent data 
processing performance of database platforms. ETL and/or ELT software normally runs in batch but 
may also be invoked dynamically by command. It could be characterized as providing the ability to 
move many things (data records) in one process.
©2017 IDC
#US41715317
5
The following are representative companies in this submarket:

IBM (InfoSphere DataStage)

Informatica (PowerCenter)
 Oracle (Oracle Data Integrator)
 SAP (SAP Data Integrator, SAP Data Services, HANA Smart Data Integration)
 SAS (SAS Data Management)
 Talend (Open Studio for Data Integration)
Dynamic Data Movement Software
Vendors often refer to this functionality as providing "real time" data integration; however, it can only 
ever be "near" real time because of the latency inherent in sensing and responding to a data change 
event occurring dynamically inside a database. Data changes are either captured through a stored 
procedure associated with a data table or field trigger or a change data capture (CDC) facility that is 
either log or agent based. 
These solutions actively move the data associated with the change among correspondent databases, 
driven by metadata that defines interrelationships among the data managed by those databases. The 
software is able to perform transformation and routing of the data, inserting or updating the target,
depending on the rules associated with the event that triggered the change. It normally either features 
a runtime environment or operates by generating the program code that does the extracting, 
transforming, and routing of the data and updating of the target. It could be characterized as software 
that moves one thing (data record or change), many times. 
The following are representative companies in this segment:

IBM (InfoSphere Data Replication)

Informatica (PowerCenter Real Time Edition)
 Oracle (Oracle GoldenGate) 
 SAP (SAP Replication Server, SAP Data Services)
Data Quality Software
This submarket includes products used to identify errors or inconsistencies in data, normalize data 
formats, infer update rules from changes in data, validate against data catalog and schema definitions, 
and match data entries with known values. Data quality activities are normally associated with data 
integration tasks such as match/merge and federated joins but may also be used to monitor the quality 
of data in the database, either in near real time or on a scheduled basis.
The data quality software submarket also includes purpose-built software that can interpret, 
deduplicate, and correct data in specific domains such as customers, locations, and products. Vendors 
with domain-specific capabilities typically offer products that manage mailing lists and feed data into 
customer relationship management and marketing systems. The following are representative 
companies in this submarket:
 Experian Data Quality

IBM (InfoSphere Discovery, BigInsights BigQuality)

Informatica (Informatica Data Quality and Data-as-a-Service)
©2017 IDC
#US41715317
6
 Melissa Data
 Pitney Bowes (Code-1 Plus and Finalist)
 SAP (SAP Data Services, SAP HANA Smart Data Quality)
 SAS Data Quality
 Talend (Talend Open Studio for Data Quality)
 Syncsort (acquired Trillium Software)
Data Access Infrastructure
This software is used to establish connections between users or applications and data sources without 
requiring a direct connection to the data source's API or hard-coded database interface. It includes 
cross-platform connectivity software. It also includes ODBC and JDBC drivers and application and 
database adapters. The following are representative companies in this submarket:
 Micro Focus (Attachmate Databridge)

Information Builders (iWay Integration Solutions)
 Progress Software (DataDirect Connect and DataDirect Cloud)
 Rocket Software (Rocket Data)
Composite Data Framework
This market segment includes data federation and data virtualization software, enabling multiple 
clients (usually applications, application components, or databases running on the same or different 
servers in a network) to share data in a way that reduces latency. Usually involving a cache, this 
software sometimes also provides transaction support, including maintaining a recovery log. 
Data federation permits the access to multiple databases as if they were one database. Most are read 
only, but some provide update capabilities. Data virtualization products are similar but offer full schema 
management coordinated with the source database schemas to create a complete database 
environment that sits atop multiple disparate physical databases. Originally created to provide the data 
services layer of a service-oriented architecture (SOA), data virtualization has been applied to multiple 
use cases including data abstraction, rapid prototyping of data marts and reports, and providing 
abilities for data distribution without replication.
The following are representative companies in this submarket:
 Cisco Data Virtualization Platform
 Denodo

IBM (InfoSphere Federation Server)

Informatica Data Virtualization
 Red Hat (JBoss Data Virtualization)
 SAS (SAS Federation Server)
Master Data Definition and Control Software
The master data definition and control software market segment includes products that help 
organizations define and maintain master data, which is of significance to the enterprise and multiple 
systems. Master data is usually categorized by one of four domains: party (people, organizations), 
©2017 IDC
#US41715317
7
product (including services), financial assets (accounts, investments), and location (physical property 
assets).
Master data definition and control software manages metadata regarding entity relationships, 
attributes, hierarchies, and processing rules to ensure the integrity of master data for where it is to be 
used. Key functionalities includes master data modeling, import and export, and the definition and 
configuration of rules for master data handling such as versioning, access, synchronization, and 
reconciliation. 
This market segment also includes reference data management software. Reference data is used to 
categorize or classify other data and is typically limited to a finite set of allowable values aligned with a 
value domain. Domains can be public such as time zones, countries and subdivisions, currencies, 
units of measurement, or proprietary codes used within an organization. Domains can also be industry 
specific such as medical codes, SWIFT BIC codes, and ACORD ICD codes. As with master data, 
reference data is referenced in multiple places within business systems, and in some cases, can 
deviate across disparate systems. Reference data management software could be considered a 
subset of master data management (MDM) software and, in many cases, may share the same code 
base as master data management software sold by the same vendor.
Master data definition and control products can include operational orchestration facilities to coordinate 
master data management processes across multiple information sources in either a batch, a service, 
or an event-based context. Master data definition and control software provides capabilities to facilitate 
single or multiple master data entity domain definition and processing and usually serves as the core 
technical component to a broader master data management solution (a competitive market in the IDC 
taxonomy).
Representative vendors and products include:

IBM (IBM InfoSphere Master Data Management)

Informatica (Multi-Domain MDM, PIM)
 Oracle (Oracle Product Hub, Oracle Customer Hub, Oracle Site Hub, Oracle Higher Education 
Constituent Hub, and Oracle Data Relationship Management)
 SAP (SAP Master Data Governance)
 TIBCO (Master Data Management Platform)
Metadata Management Software
Metadata is a data about data. It is commonly associated with unstructured data, as a structure in 
which to describe the content. However, it is increasingly being used in the structured data world, as 
the data becomes more complex in its definition, usage, and distribution across an enterprise. At a 
basic level, metadata includes definitions of the data, when, how, and by whom the data was created 
and last modified. More advanced metadata adds context to the data, traces lineage of the data, cross-
references where and how the data is used, and improves interoperability of the data. Metadata has 
become critical in highly regulated industries as a useful source of compliance information.
The metadata management submarket has grown out of the database management, data integration,
and data modeling markets. Metadata management solutions provide the functionality to define 
metadata schema, automated and/or manual population of the metadata values associated with 
structured data, and basic analytic capabilities for quick reporting. These solutions also offer 
©2017 IDC
#US41715317
8
application program interfaces (APIs) for programmatic access to metadata for data integration and
preparation for analytics.
Representative vendors and products include:
 ASG (Enterprise Data Intelligence)

IBM (InfoSphere Information Governance Catalog)

Informatica (Business Glossary, Enterprise Information Catalog, Live Data Map)
 Oracle (Enterprise Metadata Management)
 SAP (SAP Information Steward Metadata Management and Metapedia)
Self-Service Data Preparation Software
Self-service data preparation is an emerging segment in the DII software functional market. Demand is 
coming from today's tech-savvy business users wanting access to their data in increasingly complex 
environments, without IT getting in the way. IT has also been looking for technology that can put data 
preparation capabilities into the hands of business users, where the requirements and desires of 
analytic outcomes are best understood. 
These solutions are targeted at business analysts, but some offer a range of user interfaces by 
persona. Some require deeper levels of data analytics knowledge, and others require deeper levels of 
data integration knowledge. Functionality ranges from cataloging data sets within repositories, 
providing interactive capabilities for cleansing and standardizing data sets, joining disparate data sets, 
performing calculations, and varying degrees of analytics for validation prior to visualization.
Software in this segment represents vendors that explicitly sell self-service data preparation 
components. Business intelligence and analytics vendors that offer self-service data preparation 
capability within a suite as part of the analytics process they support are excluded from this segment. 
Representative vendors and products include:
 Alteryx (Designer)
 Datawatch (Monarch)
 Paxata
 SAP (Agile Data Preparation)
 Talend (Data Preparation)
 Trifacta (Wrangler)
 UniFi
RELATED MARKETS
Table 2 provides data integration and integrity software related markets.
©2017 IDC
#US41715317
9
TABLE 2
Data Integration and Integrity Software Related Markets
Competitive Market
Relationship
Master data management 
(MDM) software
The master data definition and control segment of the DII market, combined with portions of 
data access, movement, quality and metadata segments make up much of the MDM platform 
segment of the MDM competitive market.
Business analytics 
software
A portion of the data integration and integrity software market is used to derive the business 
analytics market, specifically within the data warehouse generation segment.
Source: IDC, 2017
LEARN MORE
Appendix: Patterns of Use
A pattern of use is a commonly observed assembly of software components, meeting the needs of 
commonly observed business and technical requirements, within specific organizational and technical 
constraints. The DII software market segments are aligned with data integration patterns of use. Some 
patterns that leverage components from across the segments, and conversely some patterns support 
solutions in other segments and competitive markets (e.g., master data management).
Vendors could consider patterns in the packaging and licensing of software products. Vendors are in a 
unique position to help identify new patterns that are emerging in the install base and respond with 
new products, packaging, and licensing options to gain a competitive advantage.
The patterns discussed in the sections that follow are typical of many (but not all) data integration 
solution implementations, applied to meet business requirements within technical and organizational 
constraints. They are presented in a way that vendors could help customers identify the pattern and 
justify motivation for, and implications of, using the pattern.
Scheduled Bulk Data Integration
Scheduled bulk data integration is used to move large sets of data from a source to a destination on a 
frequent schedule. This pattern primarily leverages software in the bulk data movement segment of the 
market, namely ETL or the derivative ELT. Figure 3 illustrates the two types of implementation within 
this pattern.
©2017 IDC
#US41715317
10
FIGURE 3
Scheduled Bulk Data Integration Patterns
Source: IDC, 2017
The extract process happens on a scheduled basis, depending on the data latency requirements in the 
target. This pattern is often used to populate data warehouses, data marts, or operational data stores 
(ODSs) on a regularly scheduled basis. It can also be used as a one-time data migration.
This pattern is efficient for processing large data sets, as bulk data movement technology is built to 
handle large volumes of data. Some data transformation requirements can only be achieved with 
ETL/ELT, such as match/merge between multiple source data sets, complex de-normalization, and 
summarizing and sorting of transactional data.
Use of this pattern introduces latency between the source and target and is constrained by the 
physical data stores and processing capabilities of the underlying infrastructure.
Although the bulk data movement market segment primarily serves this pattern, vendors could bundle 
components from the data access segment into solutions for this pattern. Vendors could also consider 
metadata management add-on components for traceability of data lineage through the transformation 
process, and data quality components for cleansing the data.
Near-Real-Time Data Integration
Near-real-time data integration is used to move smaller sets of data more frequently. This pattern 
primarily leverages components in the dynamic data movement market segment. It is most often used 
when changes in source data need to be captured and distributed to other systems. Figure 4 illustrates 
the pattern and the multiple methods used to capture changes.
©2017 IDC
#US41715317
11
FIGURE 4
Near-Real-Time Data Integration Pattern
Source: IDC, 2017
Changes in source data can be captured using a stored procedure or database trigger. The trigger 
may populate a change log inside the source database or send to an external persistence service. 
CDC is a technology that runs outside of the source database and monitors database log files for 
changes. Once a change is recognized, "before" and "after" images of the data in scope are sent to a 
persistence service. A micro-batch is another alternative for picking up changes. These run every few 
minutes and pull only the data that has changed since the last time the micro-batch was executed. It 
may pull data out of a change log or directly out of source tables. The persistence service could be a 
message queue, file system, external data store, or other types of containers. Mediation services pick 
up the changed data from the persistence service and perform the necessary transformation, 
formatting, and distribution to the target system(s) that need a copy of the change.
This pattern is efficient for processing data changes when latency between source and target needs to 
be minimized. Change data capture is an efficient, noninvasive method for recognizing changes in 
application databases when API-level access is not available, such as packaged applications with 
proprietary APIs, or legacy systems that are costly to modify. Database triggers, change tables, or 
other source data schema modifications are invasive and can nullify application vendor support 
agreements.
Dynamic data movement is the primary market segment that serves this pattern. Vendors could also 
bundle data access and metadata management components for solutions in this pattern. Vendors 
could also bundle this pattern as the underlying middleware in master data management solutions, 
distributing changes to master data from the systems of entry to systems of record and reference.
Real-Time Data Integration
Real-time data integration provides blending and transformation of data across multiple disparate 
sources in a virtualized view. Also known as data virtualization, it could be considered data federation 
without persistence. Figure 5 illustrates this pattern assuming two disparate data sources.
©2017 IDC
#US41715317
12
FIGURE 5
Real-Time Data Integration Pattern
Source: IDC, 2017
In this pattern, the composite data framework accesses the disparate data sources, querying for the 
data in scope from both, joins the results in memory and provides a virtual view of the integrated data, 
isolating the data access and integration logic from the consumer of the view. The virtual view is made 
available to consumers via APIs.
This pattern is efficient when zero latency is required between the source data and consumers. It is 
also useful in situations where flexibility and agility of reporting schemas are important. A virtual view 
can be changed without any underlying impacts.
With real-time access comes impact. Querying against live systems can impact performance of the 
source databases and in turn can impact the response time of the virtual views. Availability of sufficient 
memory on the infrastructure running the composite data framework software, in addition to sufficient 
network bandwidth, is important when considering this pattern. It is also important that all underlying 
data security policies are replicated in the virtual views, to prevent unauthorized access.
Packaging of composite data framework, metadata management, and data access software 
components is required for end-to-end solutions in this pattern. Vendors that have composite data 
frameworks, but not data quality solutions, could encourage the application of this pattern to identify 
quality inconsistencies across disparate data sources.
Data Federation
Data federation pulls data from multiple disparate sources, blends and transforms it into one data set, 
and persists the results into another physical data store. It could be used to populate an operational 
data store, reporting database, or data mart that needs a federated view of disparate data. Figure 6
illustrates the data federation pattern.
©2017 IDC
#US41715317
13
FIGURE 6
Data Federation Pattern
Source: IDC, 2017
Data federation can be used when latency is allowed between source systems and the federated 
views. Latency can be reduced with near-real-time population of the target data store. It is also 
appropriate when memory and network bandwidth constraints preclude data virtualization.
Federated data should be used in read-only mode for reporting and analysis purposes. Allowing write 
access to the federated repository can cause a data governance issue as it creates an alternate 
version of the source data, leading to inconsistencies.
The data federation use case leverages software in the composite data framework segment and can 
use any combination of dynamic and bulk data movement components to move data from source to 
target. Vendors could consider bundling these components together and adding metadata 
management components for a comprehensive solution that will also provide context and lineage of 
data in the federated data store.
Data in Motion Pattern
Data in motion represents the application of data integration technology against streaming data. Many 
3rd Platform technologies constantly emit data, available to organizations that have an interest or need 
for the data. Data is constantly in motion, coming from social networks, the Internet of Things, stock 
market tickers, and so forth. Figure 7 illustrates an application of data integration technology to data in 
motion.
FIGURE 7
Streaming Integration Pattern
Source: IDC, 2017
©2017 IDC
#US41715317
14
This is an emerging pattern and is applicable to those organizations that are working with, or have a 
desire to leverage, streaming data. Any foray into the use of data streams needs to be made taking 
into consideration the abilities of the organization to capture and respond to business-relevant events 
in the stream. Available system resources also need to be understood in the context of the velocity and 
volume of the streaming data.
This pattern leverages several components in the data integration software market, and vendors are 
encouraged to consider bundling alternatives to meet the requirements of this pattern, as applicable to 
customer requirements. Data access software provides the capability to interface with social media, 
Internet of Things, and other data stream sources. Mediation capabilities available in bulk and dynamic 
data movement components assist with identifying and filtering data in the streams that are relevant to 
the organization. Data quality components can be used to validate the data prior to allowing into the 
persistent storage of the organization.
Related Research

IDC's Worldwide Services Taxonomy, 2017 (IDC #US42356617, March 2017)
 Market Analysis Perspective: Worldwide Data Integration and Integrity Software, 2016 (IDC 
#US40799016, September 2016)
 Worldwide Data Integration and Integrity Software Market Shares, 2015: Year of Big Data and 
the Emergence of Self-Service (IDC #US40696216, June 2016)
 Worldwide Data Integration and Integrity Software Forecast, 2016–2020 (IDC #US40696116, 
June 2016)
Synopsis
This IDC study provides taxonomical definitions for the data integration and integrity software market. 
The study segments the market into data access, movement, integration, and governance functional 
categories. Market segments include bulk data movement, dynamic data movement, composite data 
frameworks, data quality, master data definition and control, metadata management, and self-service 
data preparation.
"Data is core to digital transformation, and data without integrity will not be able to support digital 
transformation initiatives," says Stewart Bond, research director for Data Integration and Integrity 
Software at IDC. "Data integration software provides organizations with the ability to catalog, move, 
transform, match, cleanse, and prepare data improving the level of trust and integrity of data, 
improving the outcomes of digital transformation initiatives on the 3rd Platform."
About IDC
International Data Corporation (IDC) is the premier global provider of market intelligence, advisory 
services, and events for the information technology, telecommunications and consumer technology 
markets. IDC helps IT professionals, business executives, and the investment community make fact-
based decisions on technology purchases and business strategy. More than 1,100 IDC analysts 
provide global, regional, and local expertise on technology and industry opportunities and trends in 
over 110 countries worldwide. For 50 years, IDC has provided strategic insights to help our clients 
achieve their key business objectives. IDC is a subsidiary of IDG, the world's leading technology 
media, research, and events company.
Global Headquarters
5 Speen Street
Framingham, MA 01701 
USA
508.872.8200
Twitter: @IDC
idc-community.com
www.idc.com
Copyright Notice
This IDC research document was published as part of an IDC continuous intelligence service, providing written 
research, analyst interactions, telebriefings, and conferences. Visit www.idc.com to learn more about IDC 
subscription and consulting services. To view a list of IDC offices worldwide, visit www.idc.com/offices. Please 
contact the IDC Hotline at 800.343.4952, ext. 7988 (or +1.508.988.7988) or sales@idc.com for information on 
applying the price of this document toward the purchase of an IDC service or for information on additional copies 
or web rights. 
Copyright 2017 IDC. Reproduction is forbidden unless authorized. All rights reserved.
</p>