Data integration is the practice of consolidating data from disparate sources into a single dataset with the ultimate goal of providing users with consistent access and delivery of data across the spectrum of subjects and structure types, and to meet the information needs of all applications and business processes. The data integration process is one of the main components in the overall data management process, employed with increasing frequency as big data integration and the need to share existing data continues to grow.
Data integration architects develop data integration software programs and data integration platforms that facilitate an automated data integration process for connecting and routing data from source systems to target systems. This can be achieved through a variety of data integration techniques, including:
- Extract, Transform and Load (ETL): copies of datasets from disparate sources are gathered together, harmonized, and loaded into a data warehouse or database.
- Extract, Load and Transform (ELT): data is loaded as is into a big data system and transformed at a later time for particular analytics uses.
- Change Data Capture (CDC): identifies data changes in databases in real-time and applies them to a data warehouse or other repositories.
- Data Replication: data in one database is replicated to other databases to keep the information synchronized to operational uses and for backup.
- Data Virtualization: data from different systems are virtually combined to create a unified view rather than loading data into a new repository.
- Streaming Data Integration: a real-time data integration method in which different streams of data are continuously integrated and fed into analytics systems and data stores.
Data integration is defined as the system of merging data from various resources and convert it into valuable information to provide a unified view of the data to the user; it also allows tools to generate effective business intelligence and actions as the basic operation involved in the data integration is that the client sends a request to master server in order to access the data and as a return, the master server fetches the data and send it to the client, due to these features it is significantly used in a variety of situations like commercial and scientific domain.
Types of Data Integration
1. Data Consolidation
Data consolidation substantially gets data together from several individual systems establishing a single data store. Data consolidation aims to achieve a reduced number of data storage locations, which is supported by ETL (Extract, Transform, Load) technology. ETL fetches the data from repositories, transfers it to the readable format, and then transports it to another data warehouse.
2. Data Propagation
It uses the application to duplicate the data from one location to another. It can be made possible in a dual way between source and client. Data propagation is supported by Enterprise Data Replication (EDR) and Enterprise Application Integration (EAI).
- EAI manages application system sharing messages and is mostly executed in a real-time scenario.
- EDR transmits a huge amount of data between databases that are used to fetch and distribute data sharing between the resource and servers.
3. Data Virtualization
Virtualization manages an interface to offer present unique data from separate sources with varied data models. Data virtualization interprets and extracts the data from any pool without any single point of contact.
4. Data Federation
It is a theoretical form of data virtualization and utilizes virtual databases and builds a general data model for hybrid data from different systems. Data is gathered from various sources and accessible as a single view.
Data abstraction is to provide a discrete view of data from a hybrid source by Enterprise Information Integration (EII). The data can be analyzed in a trending way via many applications. Data consolidation is expensive because of its advanced security features and compliance.
5. Data Warehousing
Warehousing is included as the last step because of its large repositories of data. Data warehousing implements data storage, reformatting, and cleaning similar to data injection.
Data Integration Rules
Data integration rules are provided by a data service to control how the service updates your Salesforce records. Rules tell the data service how to find records to update and which updates to make. Rules also control how updates affect other features, such as triggers and workflow rules.
The flow of technographic data into Salesforce is controlled by Data Integration Rules. There are six data integration rules:
- Two that match and bring in the correct company record for your Account or Lead.
- Two integration rules that append Technographics to your Accounts and Leads.
- Two that activate HG Spend Intelligence.
You will not need to activate the Spend Integration rules if you have not licensed this information from HG Insights.
HG for Salesforce does not modify account or lead data and only appends HG Data Technologies records.
There are several rule settings that can be configured:
- Update all records (recommended): This allows HG Data to enrich all account records, and activates continuous enrichment.
- Bypass triggers: If enabled, will skip triggers for HG Data Technologies when updated. (Recommended: leave unchecked).
- Bypass workflow rules: If enabled, will skip workflow rules for HG Data Technologies when updated. (Recommended: leave unchecked).
- Leave last-modified information unchanged: If enabled, will not update the last-modified date when HG Data Technologies records are updated.
Challenges to Data Integration
Taking several data sources and turning them into a unified whole within a single structure is a technical challenge unto itself. As more businesses build out data integration solutions, they are tasked with creating pre-built processes for consistently moving data where it needs to go. While this provides time and cost savings in the short term, implementation can be hindered by numerous obstacles.
Here are some common challenges that organizations face in building their integration systems:
- How to get to the finish line: Companies typically know what they want from data integration — the solution to a specific challenge. What they often don’t think about is the route it will take to get there. Anyone implementing data integration must understand what types of data need to be collected and analyzed, where that data comes from, the systems that will use the data, what types of analysis will be conducted, and how frequently data and reports will need to be updated.
- Data from legacy systems: Integration efforts may need to include data stored in legacy systems. That data, however, is often missing markers such as times and dates for activities, which more modern systems commonly include.
- Data from newer business demands: New systems today are generating different types of data (such as unstructured or real-time) from all sorts of sources such as videos, IoT devices, sensors, and cloud. Figuring out how to quickly adapt your data integration infrastructure to meet the demands of integrating all these data becomes critical for your business to win, yet extremely difficult as the volume, speed, and new formats of data all pose new challenges.
- External data: Data taken in from external sources may not be provided at the same level of detail as internal sources, making it difficult to examine with the same rigor. Also, contracts in place with external vendors may make it difficult to share data across the organization.
- Keeping up: Once an integration system is up and running, the task isn’t done. It becomes incumbent upon the data team to keep data integration efforts on par with best practices, as well as the latest demands from the organization and regulatory agencies.
Activate Data Integration Rules
- If your users use Salesforce Classic, let them view data and update records manually by adding the Data Integration Rules related list to the page layouts for accounts, contacts, and leads.
- Assign object permissions to users. The object permissions required depend on the rule. For details, contact the data service provider.
- In Setup, use the Quick Find box to find Data Integration Rules.
When a rule is activated, the following happens:
- When records are created, the rule looks for matches in the data service.
- On existing records, when users change the value of fields that are used in matching, the records are updated.
- Except for geocode rules, data integration rules never overwrite your data — data is added only to blank fields.
- Users can view rule status and update a specific record at any time.
- When the Update all records option is selected, a rule doesn’t necessarily run immediately after you edit the rule’s field mapping or confidence score. To run the rule immediately, deactivate the rule, change the settings, and reactivate the rule.
Additional conditions:
- After a rule is activated, it runs on records that are added or edited.
- After a rule (except a geocode rule) is activated, it runs on all records periodically at the frequency determined by the data service provider.
- When the following options are selected, they’re applied whether records are updated by the rule or by users manually:
➯Bypass triggers: Triggers aren’t activated.
➯Bypass workflows: Workflow rules and workflows created via Process Builder are bypassed.
➯Leave last-modified information unchanged: The values of the Last Modified by Id and Last Modified Date fields on records aren’t updated. The System Modstamp field is always updated, regardless of this setting.
- Review the field mapping for the rule.
- Activate the rule.
