There’s a great discussion of the reasons for ERP implementations failing to deliver expected benefits going on here, prompted by the 2010 ERP Report by Panorama Consulting.

But…what’s missing from this very incisive post?

A deeper drill-down into the ongoing, chronic and ever-present root of ERP failure—a failure to adequately, quickly and robustly integrate ERP systems with other systems.

We see this issue in our integration practice, and many of our partners focus on this specific area.

Case in point: climbing out of the integration tar-pit, one firm realized an 80% reduction in maintenance time and a 20% reduction in customer on-ramp time gains in productivity from their ERP system. Yes, we are a little integration-centric, but with potential impact like this, it’s no wonder.

But how many organizations actually achieve such a lofty goal? And how do they do it?   

It's a great opportunity to gain insight into achieving meaningful progress towards this worthwhile goal.

Data Integration – It’s the value, stupid!

Written in Gartner’s typically diplomatic language, the latest Gartner Magic Quadrant on Data Integration reveals just how much pressure is being put on traditionally complex and expensive products in the data integration space – “During 2009, vendors in the data integration tools market experienced increased scrutiny from buyers with regard to pricing, cost models, time to implementation and the quality of service and support”.   

It’s interesting that it’s taken unprecedented economic pressures to get companies to really start getting to grips with the horrendous amounts of money and time they traditionally spend on integration. The good news is that there are documented approaches to doing this methodically (see the Bloor Report).

As a thought leader in this space, I suspect Gartner will start reflecting value as one of the key criteria in its magic quadrant rankings. Hopefully a reflection of value will come in time for next years’ holiday season, building on the integration-related things to be thankful for noted by Loraine Lawson.

For the past 18 months, the Pervasive DataSolutions team has been building innovative cloud-based integration solutions (Pervasive DataSynch for QuickBooks and Salesforce as a flagship example).  Mainly the solutions have come in the form of a value-add integration service, and the Pervasive DataSynch has become one of the most proliferated integration add-on applications for salesforce.com and QuickBooks - with 150 customers and counting and the highest number  of reviews for a QuickBooks integration on salesforce.com’s AppExchange site. 

 These cloud-based integration solutions are deployed on a platform called the Pervasive DataCloud.  The Pervasive DataCloud was created as a by-product of the DataSolutions team needing to build their own products more rapidly and efficiently, but they were the only ones building them. 

That is until the Pervasive DataRush group asked DataSolutions to build a new product for them.  DataSolutions didn’t have the bandwidth to squeeze in another product, but realized if they could expose the web services, the DataRush group could build the product themselves.  The Pervasive DataRush team, and one of their partners, Aha! Software, recently released a beta of this new service, deployed on the Pervasive DataCloud. The service, called Strato-Studio, is a MonteCarlo simulation that does predictive analysis.  What does that mean?  Well, you feed Strato-Studio some predictive modeling data, it crunches it many times, really fast, and then gives you pretty good idea where things are headed.  That’s the quick version, but you get the idea. 

So where does Pervasive DataCloud fit?

David says in his post that data integration is critical to delivering cloud-based applications, but that most organizations approach it as an afterthought.   He adds that implementing integration before joining the cloud is the optimal approach.   I entirely agree.   It is all too easy for your data to end up just being isolated in the cloud and all the other expected benefits disappear into thin air.    However, I do not believe that this is a before vs after kind of discussion.    You can start earlier or later, but rather than worry about when to integrate, the main issue will be the proliferation of apps and interfaces on multiple clouds as well as on-premise – and you will need a way to handle these challenges within tight budgetary limits.   That’s why we will be talking more about how you can tackle this challenge without breaking the bank.

There has been a lot of talk about delivering integration in the form of appliances. I recently came across a blog posting by Loraine Lawson of ITBusinessEdge (August 19^th) titled A Tale of Two Data-Integration Technologies that touches on this topic.   Integration appliances are supposed to remove complexity for users, but there is no hard evidence to support this proposition.   More likely, delivering software bundled with hardware (aka appliances) is a distraction.   In reality, integration success depends on the quality of the software in terms of connectivity, the ability to handle high data volumes and to make changes quickly without specially trained staff.

• integrated customers of all sizes, from the very large (like Honeywell) to the very small 
• done salesforce integrations that involve a few thousand records, and others that involved many, many millions of records (like Chevy Chase Bank)
• implemented simple synchronizations, simple bi-directional data movement, and sophisticated systems involving many different endpoints (like Schroders PLC, who had over 160 different data formats)
• built integrations of all types: SaaS to SaaS , SaaS to on-premises - and multiple SaaS applications to multiple on-premises systems
• been embedded inside OTHER applications (SaaS as well as on-premises) that wanted to include Salesforce CRM integration as a key built-in feature

In order for any organization to leverage the operational benefits of business intelligence, to comply with governmental or industry regulations, or simply to make well-informed decisions, they must have a solid data foundation.  The point of all this is that your business intelligence platform is only as good as the data it operates on.  And the data it operates on is typically brought to the BI platform through some sort of data integration layer.   Simply speaking, it’s critical to get the data integration “right” if you’re embarking on a BI initiative.

Data comes from many different places, is in many different formats – Cloud-based data, legacy data, data in spreadsheets, data in different databases, data inside of operational applications, data inside SaaS applications, etc.  Just “getting at” data can be a challenge.  Then the real fun begins, as data can also be famously difficult to reconcile – dealing with semantics and metadata issues, cleansing and remediating data, matching and merging data.

Although many organizations start off with a homegrown approach to data integration, which may initially appear to be cost effective, this quick and dirty approach rarely leads to a happy ending.  Even if the homegrown solution is able to deliver the same “quality” of data, and even if there may appear to be initial savings (which may or may not be the case), these hard-coded solutions are ineffective at dealing with the two key issues facing any data integration scenario – increasing variety and increasing rate of change.  As time progresses, the disadvantages of a hand-built solution increase dramatically.  

It’s not unusual for organizations to spend anywhere between hundreds of thousands to tens of millions of dollars on BI implementations, and then use those tools to make decisions that impact the future of their organization.  With all that at stake, a solid data integration platform isn’t a luxury, it’s a necessity.

Follow Pervasive Integration on Twitter and keep up with all the latest news!

http://www.twitter.com/PVSWintegration

I love the software industry, and I love working for independent software vendors.  There is something very energizing about a group of people who have a vision about how something can be done better or differently, who then create a software product to make that happen.

At the Metamorphosis 9 event,  I had the chance to speak with an entertaining and informed guy by the name of Matthew Bather, who is a product manager working for Exact Software.  Exact Software is a highly successful vendor of ERP software – over 100,000 companies make use of its products and services, which are provided by 2500 employees spread across 4 continents.

So it’s interesting that a company whose focus is on making the best possible ERP software for its customers would place such a high value on offering an integration platform.  Many ISVs focus their efforts on enhancing their core software product – be it ERP, content management, sales force automation, project management or whatever, and to offer a comprehensive integration layer would be considered a distraction.

Joe McKendrick over on ZDNet had some good points about why SOA sometimes gets bad press, despite it's advantages.  Good things to keep in mind before trying to implement one in your business.

http://blogs.zdnet.com/service-oriented/?p=2056 

 Paige

A Shortcut for Dealing with Complex Parent-Child CRM Data

by Jacob Hughes, Integration Technical Support Engineer, Pervasive Software, Inc.

Stuck on a complex integration problem such as parent child relationships for CRM targets? Try staging your child data in a delimited ASCII file, and performing lookups for the necessary relationship fields before uploading them to CRM applications like Salesforce or MS CRM. This follows the KISS principle.

I recently used this technique on a consulting project that I helped out on, and it resolved problems we were having with duplication in Salesforce due to the complexity of the event handlers needed to handle parent and child data cleanly all in one map.

The ASCII connector is one of the fastest, if not the fastest connector Pervasive has, so the sorting and filtering happened at lightning speed. By storing child data in an ASCII file, it makes your maps much simpler and doesn't add much wait time to the overall process. Staging data in ASCII files can actually speed up processing by Pervasive Data Profiler.

by S. Patton, Technical Writer, Pervasive Software, Inc.

Salesforce offers outbound messaging so that you can monitor changes in your online CRM data. Once the SOAP message goes out, however, you're on your own. Salesforce leaves it up to you to handle the message and channel it into something useful. Pervasive has done over 500 Salesforce implementations. Our customers wanted an easy way to handle those messages and turn them into emails or rows in a database, alter the format of the messages, pass them on to an ESB or otherwise turn them into useful data.

The new Salesforce Outbound Message Service is an add-on for version 9.2 of Pervasive Data Integrator that listens for and processes SOAP outbound messages from salesforce.com. When a message arrives from Salesforce, the service accepts the message and spawns a Pervasive Integration Engine instance. The message is then processed based on the specification file that you create with Map or Process Designer. Since Salesforce controls how messages are aggregated and sent, a single SOAP message may contain several notifications, all of which can be handled appropriately by a single specification file.

To use the Salesforce Outbound Message Service, you need to complete four steps:

1. Set up outbound messaging and associated workflow rules in salesforce.com.
2. Create a specification file to handle the incoming messages.
3. Start the service container.
4. Configure and deploy the Web service.

1. Set up outbound messaging and associated workflow rules in Salesforce.
Determine which fields and tables you wish to monitor in Salesforce. Specify workflow rules for when messages are triggered. For example, a message can be triggered each time a new lead is added, or when a specific contact name is changed. You'll need to configure at least one workflow rule for each outbound message type. Information on how to set up workflow rules and outbound messages is located in the Salesforce online documentation.

2. Create a specification file to handle the incoming messages.
The specification file tells the service how to handle incoming messages. For example, you may want to save the message in its original format, format it to a specific standard, email it to others, etc. Your specification file can either be a simple process, or a .djar file. Any Pervasive process that does not contain any maps is fully specified in one XML file, and it can simply be uploaded to Salesforce as is. For processes with maps, .djar files can be created that contain all of the XML files for a full process compressed into one file. Since only one file can be uploaded to salesforce.com for each service you configure, upload a .djar file for processes with maps.

3. Start the service container.
Before you can access the Web service application, you need to start the service. On a Windows machine, Start/All Programs/Pervasive/SFDC Outbound Messaging Service/Start Service will get the service running. (Check the Pervasive documentation for Linux instructions.) To stop the service, select the same path to Stop Service.

4. Configure and deploy the Web service.
Log into the Web service application, configure and deploy your application, and you are ready to receive outbound messages. The Service Management Console is on the same Start path as the Start and Stop Service commands.

That's really all there is to it, but here are a few helpful tips:

To see if a message is undelivered, log into salesforce.com, and click "View Message Delivery Status."

To view the service log file, use the Start/All Programs/Pervasive/SFDC Outbound Messaging Service/View Service Container Log command on Windows, or open it directly with a text editor in the Pervasive installation directory\SFDC-OM\servicemix-3.2.1 on your server.