Data integration projects do not always have the best reputation. They are said to be cumbersome, complex and sometimes risky.
Nowadays, companies have invested heavily in systems to collect data in order to better serve their clients. This data is priceless and many are worried about putting their data at risk when attempting to connect these systems.
The benefits of system integration are numerous. Better data quality, ability to provide superior customer service, and reduced inefficiencies and manual processes are but a few of these benefits.
However, some IT leaders are still reluctant to take the next step towards fully integrated systems. A simple approach to data integration project planning can secure a successful system integration project.
What is data integration?
Data integration simply refers to connecting data between two systems. Data integration aims at securing that data flows seamlessly between different systems. This could be be between an ERP system and a CRM system.
When you integrate ERP and CRM, each system handles data in a specific way according to the business processes they support. Each system has a specific built-in business logic.
When connecting CRM and ERP systems, a data integration system typically transfers operational data from one system to the other. The most common integration end-points are: Customers, Contacts, Items, Quotes, Orders, Invoices or Sales History. With an integration, the data stored in one system is synced with the data corresponding data in the other system.
The scope of data integration can be a lot broader that the above-mentioned transfers. It could cover traditional master data or any table or field that you would wish to sync and align between the system.
There are no limits to which data is relevant for data integration. It depends on your system landscape and business processes.
How do you setup a data integration project?
There are different types of data integration approaches. You can take a vertical integration approach or a horizontal integration approach.
Horizontal integration combines data within the same data source. Vertical data integration creates connections between entities of different types. Another type is star integration where systems are connected point to point.
A few years ago, many data integration projects were done using batch interfaces where data was extracted to a file. That file was then sent to the target system and imported also via a batch job.
This requires a lot of work upfront - in the programming area and the testing area. This method had a number of pitfalls:
#1 - poor data quality
When you use a batch transfer approach, you need to know how to handle poor quality and incorrect data
#2 - keep the sync going
Once the batch transfers are done, you must have a plan for keeping the integrated systems in sync moving forward. If the synced systems are not modified, the setup could run for years.
If you have agile systems like ERP or CRM systems, it can become cumbersome and expensive to keep the systems in sync.
In such a case, it is more cost-efficient and a lot more robust to use a data integration platform.
Data integration platforms like RapidiOnline provide with most of the transfers out-of-the-box. Then your work is reduced to a simple configuration and mapping of data.
CREATING A DATA INTEGRATION PROJECT PLAN
As for any project, the project planning phase of a data integration is a critical step towards success. There are many approaches to project planning. In reality, how to create a project plan is not much different in the data integration sphere.
As in any project, when you write a project plan, you will need to consider:
- A project team including all direct or indirect team members
- A project schedule
- A project timeline
- A project management plan
- A resource plan
- A risk analysis
- A detailed output description
To achieve a smooth implementation of data integration solutions, you should consider the following easy steps:
STEP 1: SPECIFY BUSINESS PROCESSES AND DATA INTEGRATION PROCESSES
Most companies do data integration for the same purpose. They want to optimize business processes and gain efficiency.
Look at your current business processes and think about how you could improve workflows. Companies often have to convert a CRM lead to a customer in ERP. If this happens manually, it is an inefficient error-prone process.
It would be better to have a customer automatically created in the ERP system at a triggered event in the CRM system, say the first time they order something.
Before the data integration project starts, you should investigate your processes with critical eyes. Analyze potential data transfers entity by entity for example customers, contacts, items, etc...
STEP2: DEFINE DATA INTEGRATION SCOPE
When you have your full data transfers wish list, you investigate further which transfers you need and which are nice to have.
Based on a thorough analysis of your business processes, you will decide which transfers should be one direction and which will be bi-directional. Often people assume that they need bi-directional transfers, but in many cases one direction is enough.
STEP 3: CONNECT THE SYSTEMS
Connecting the 2 systems happens via the data integration platform. The system designs (meaning tables, fields, and other relevant information for the integration) are read and stored in order to do the mappings correctly and with the right information.
The data has then to be sequenced the right way in the data transfers. If you are transferring an order, it makes sense to transfer items first because there might be a new item on the order. Without that item there would be a broken reference.
In this phase, you decide technically which system has priority over the other (s) for bi-directional tables in case a piece of data has been updated in both systems. This is what we call the definition of the master and slave logic.
When the data transfer is set up, you start testing. Testing must start with a few records, until everything is ready. This is often when you realize that your data quality is not as good you expected. You need data cleaning.
If the data has been created manually in both systems (for example, customers without a clear reference), you could have some duplicates. This, of course, is not what you want.
To solve the duplicates issue, you will either have to make a batch job or a query that can update this automatically. Alternatively, you could manually look them up and add an identifier that clearly links the two records together. It is a simple procedure, but if you have a lot of customers it can be time consuming to fix.
STEP 4: SYNC THE SYSTEMS
Once everything is tested and the initial data transfer is made, you have to bring the two systems in sync. You don’t have to all do this at one time. It is possible to take one logical area at a time.
In the same way you can take them live one at a time. This way, you will show your users the benefits of the system integration. This is usually a boost for user adoption.
IS DATA INTEGRATION SUPPORT A PAIN?
When everything is completed, you can start relaxing and switch to support mode. If you are using a data integration platform environment like RapidiOnline, it is easy.
WIth theRapidiOnline integration platform, if anything goes wrong with the data transfers, the people in charge will receive an email notification. This email contains the relevant information and a direct link to the data transfer that has an error, for investigation and resolving.
This task is smooth and simple when you use data integration platforms (some even with pre-configured to pre-defined end-points) to ease your data integration project. Many of our customers are using the RapidiOnline integration platform to quickly and easily connect their Salesforce and Microsoft Dynamics systems.
