Most of the IT executives that are looking for a data integration system want to have real time integration. While it is often what they want, it is rarely what they need.
Real time data integration is somewhat the Nirvana of data integration. Why not instantly synchronize your data sources when you can do it? Why should you wait for information to be transferred later when you can have it right now?
It is easy to understand their desire to streamline and super optimize the business processes in their company. However, the question remains as to whether or not they need instant transfer of information. In many cases, periodic batch processing of data is enough to cover their needs.
Having timely information is important for any business. That does not mean that all businesses would benefit from a truly real time integration of their data. Some might be better off with a batch processing type of solution. Other businesses might decide to use a mix of both approaches.
Solutions like RapidiOnline offer a range of packages to cover all data integration needs, real time or not.
What is real time data integration?
Let us start by defining data integration. The definition of data integration is very simple and straight-forward. Data integration consists in gathering and combining information (or data) from different sources. Businesses use data integration to bring together data from various data sources into one consolidated and unified view.
Data integration is done using different technologies and methods and following different architectures. For example, services might access and gather data from one or several sources before loading them into a target system.
The definition of real time integration is somewhat relative. Real time data integration is the processing and transfer of data from one system to the other in a very restricted time constraint. True real time should be an instant transfer of data. If not instant, then as close to instant as possible.
Because truly instant integration can be hard to achieve, people often talk about "near" or "soft" (as opposed to hard) real time integration. There are indeed various real time data integration techniques that allow for "almost" instant synchronization.
Companies that want to do real time analytics require data warehouses populated with 100% up-to-date information. They want full control of their supply chain processes and instant integration of their data sources. Most often, even if such organizations need to access data in real time, a hard real time integration is not what they need.
Many businesses can run their operations with system integration based on batch processing methods. When batch processing is used for data integration, the solution collects data, stores it and processes it in groups rather than one at a time. Batch processing is perfect if your work processes do not require immediate data transfers.
According to the real time integration definition, each piece of data is processed and transferred as soon as it is created. Because such integration systems can add significant load to the source systems, you should consider it carefully.
You might want to learn from the best practices in this domain. Choosing between real time integration, near real time integration, batch processing, or other triggered alternatives based on web services can be tricky. The right choice depends very much on your business processes and requirements.
What are the limitations of real time data integration?
If real time integration seems like the holy grail of data integration, it is not entirely flawless. Even in 2020, implementing real time data integration can be challenging because:
- it might require that businesses upgrade their IT infrastructure, which could be very costly.
- poor data quality and generally poor data integrity will make real time data integration more difficult.
- the amounts of data collected can be huge, especially in the time of big data. This calls for advanced data management, which very few companies are able to handle at this point.
While the value of real time data integration is easy to acknowledge, it is not yet that simple to implement. Today, data flows are so important that people talk about streaming data.
Streaming data refers to the continuous flow of data from a growing amount of data sources. Such developments in data collection do not make data collection, data processing or data integration any easier.
New applications show up every day, bringing additional potentially valuable data for businesses. The challenge is not only to deal with the increasing amounts of data. It is important to know which data is relevant, and of course how to process it so it can truly become valuable .
Why has the need for real time integration increased?
In the past decades, the number of applications and external systems used by companies to collect data has increased exponentially. Today, we are referring to the huge data quantities as big data and data warehouse management has become a critical aspect of any company.
As mentioned above, companies gather data from many different solutions and applications. As a result, it can be difficult to have one unified view of the data. Unless these data sources are integrated, companies end up with silos of data that do not bring much value to their processes.
Data management has become and will remain a vital function in a company. In this context, many organizations turn towards a data integrator to help them connect these systems.
Customer information is a central piece in this data collection exercise, and many businesses use Salesforce as their preferred Customer Relationship Management (CRM) solution. CRM is not the only source of customer information. Most companies also use an Enterprise Resource Planning system (ERP) to gather and process order and invoicing information about their customers.
Some companies use Supply Chain Management systems to manage their operations. Likewise, Service Management systems can also collect relevant customer information.
Customer information ends up being stored in separate systems. Therefore, companies can only truly get a 360 degree view of their customers by integrating Salesforce with their ERP. To solve this challenge, there are various data integration platforms.
Many integration platforms are cloud based, using web services. Some offer batch processing of data while others promote real time processing. The question remains as to whether real time is truly necessary.
Most of the companies that wish to integrate Salesforce with an ERP solution rarely need real time integration. This is true for any CRM solution. It is also the case for any Microsoft Dynamics ERP solution: AX, NAV, GP, 365 Business Central or 365 for Finance & Operations.
Salesforce real time integration scenarios
It is interesting to look at the needs behind real time data integration projects. Businesses that start a data integration implementation to connect their ERP and CRM systems, typically want to:
- Remove the need for double data entries.
- Eliminate the risk of manual errors.
- Optimize and streamline processes so that staff will not have to look for information in several systems.
- Increase productivity because their staff will have data they can trust.
Most of these benefits can easily be achieved without real time data integration. Unless your business works in an environment where seconds matter, you probably don't need instant synchronization of your data.
The need for actual real time transfers can occur. For example, you would need it if your operations are closely linked to stock exchange transactions. In that case, true real time integration would be necessary. If your data transfers are not that time sensitive, chances are you can live without real time integration.
Why people think they need real time Salesforce integration
Most companies that integrate Salesforce CRM with an ERP solution such as Microsoft Dynamics primarily want to integrate:
- customer information (account and contact)
- product or item information
- pricing information
- quote and order information
Sometimes, they also want to have sales history information transferred from their ERP to their CRM as well as payment information. These are the most common needs of the large majorities of companies. The need for integration is there without a doubt. It is, however, not a need for instant transfers.
Say that you create a quote in Salesforce but will need the calculation of the price to be done in the ERP solution. How often will you need this information to be done instantly? Most people will be able to wait a couple of minutes to get the information back from the ERP.
Quote generation, order creation and customer data update are not time critical processes. They can easily way a couple of minutes to be transferred in the other system. When you update the address of a customer in Salesforce, what are the chances that someone needs this information at the exact same time, without being able to wait a couple of seconds?
Often, the reason behind a request for instant transfers has nothing to do with actual needs for real time processes. Implementing the highest frequency possible of transfers will always cater for all needs. It is a “better safe than sorry” approach.However, it can be an expensive approach.
Generally, this approach has significant limitations, including high cost and load on source systems.
Salesforce real time data integration tools
It is rare to meet companies that truly require real time Salesforce data integration. Even in the time of business intelligence, big data and multiple data sources. Therefore, many integration solutions offer different levels of transfer frequency.
Most top Salesforce integration tools will claim that they can provide real time Salesforce integration. These claims are likely to be valid. Mulesoft, Jitterbit, Boomi and even Salesforce themselves have a real time integration offering for Salesforce CRM.
Typically, the time performance of the integration will highly depend on the data integration method used by these providers. Finding the most optimal integration tool also means finding the solution that is designed to perform with minimal impact on the integrated systems.
Real time data integration can put significant load on the source system and the destination system. This is especially the case if you transfer all data from one system to the other every time. It is a good idea to select a data integration tool that only transfers data that is new or updated.
RapidiOnline gives a clever solution to this challenge. It is a cloud based data integration platform that gives businesses the possibility to decide exactly how often they want transfers to run.
With RapidiOnline, it is possible to have transfers done in groups, every 5 minutes or 10 minutes according to a defined schedule. Integration can even be scheduled to run once an hour if that is what you need. The cloud data integration solution also also supports more frequent system integration requirements.
RapidiOnline gives you the possibility to decide that some more time sensitive processes will trigger an immediate transfer. This could be the case every time you create a quote for example. When such an "event" occurs, the information is immediately transferred to the queue for processing. This makes sure that this information is sent directly to the queue of transfers..
What if your business process is much more time sensitive and cannot wait to be placed in the queue of transfers? What if you want it to be processed immediately? In this case, RapidiOnline offers the possibility to add a web service that will place your critical transfer in front of the queue.This solution gives you an alternative that is as close to real time as possible. But it does not add unnecessary load to your systems. It is a highly flexible solution, and it is not as expensive as other true real time Salesforce integration.
Is real time data integration what you need?
Real time is not always the best data integration approach for a business. It is usually expensive and sometimes, it is not even faster than a "near" real time integration as described above. Real time will never be faster than the system is at processing the data.
If time is the essence for your decision, it is important to know that 100% real time data integration can be an illusion. As soon as the data needs to be transferred, it also needs to be processed and this can take time. Even if the transfer is immediate, you may not experience it as instant.
Most businesses can satisfy their requirements with a combination of triggered transfers and near real time integration solutions. This kind of combination can satisfy by far most of the business processes optimizations of companies of all sizes.
