Data Integration Blog

Rethinking: Data Integration – the foundation for successful Business Intelligence

hollis_pvsw — Mon, 22 Feb 2010 17:47:00 GMT

Given the importance of Business Intelligence across all aspects of today's business environment, and the ever increasing number of blog postings and published articles on the topic, I felt that my July blog posting was ready for a refresh and expansion.

The value of business intelligence (BI) is well-proven. In any case, I’ll leave it up to the slew of BI vendors to make that point. This discussion is how data integration acts as the foundation for enabling all the goodness that BI brings to the enterprise.

In order for any organization to leverage the operational benefits of business intelligence, to comply with governmental or industry regulations, or simply to make well-informed decisions, they must have a solid data foundation. The point of all this is that your business intelligence platform is only as good as the data it operates on. And the data it operates on is typically brought to the BI platform through some sort of data integration layer. Simply speaking, it’s critical to get the data integration “right” if you’re embarking on a BI initiative.

Data comes from many different places, is in many different formats – Cloud-based data, legacy data, data in spreadsheets, data in different databases, data inside of operational applications, data inside SaaS applications, etc. Just “getting at” data can be a challenge. Then the real fun begins, as data can also be famously difficult to reconcile – dealing with semantics and metadata issues, cleansing and remediating data, matching and merging data.

Although many organizations start off with a homegrown approach to data integration, which may initially appear to be cost effective, this quick and dirty approach rarely leads to a happy ending. Even if the homegrown solution is able to deliver the same “quality” of data, and even if there may appear to be initial savings (which may or may not be the case), these hard-coded solutions are ineffective at dealing with the two key issues facing any data integration scenario – increasing variety and increasing rate of change. As time progresses, the disadvantages of a hand-built solution increase dramatically.

It’s not unusual for organizations to spend anywhere between hundreds of thousands to tens of millions of dollars on BI implementations, and then use those tools to make decisions that impact the future of their organization. With all that at stake, a solid data integration platform isn’t a luxury, it’s a necessity.

ERP failure…or success? A look at some chronic issues.

hollis_pvsw — Thu, 04 Feb 2010 20:45:00 GMT

There’s a great discussion of the reasons for ERP implementations failing to deliver expected benefits going on here, prompted by the 2010 ERP Report by Panorama Consulting.

But…what’s missing from this very incisive post?

A deeper drill-down into the ongoing, chronic and ever-present root of ERP failure—a failure to adequately, quickly and robustly integrate ERP systems with other systems.

We see this issue in our integration practice, and many of our partners focus on this specific area.

Case in point: climbing out of the integration tar-pit, one firm realized an 80% reduction in maintenance time and a 20% reduction in customer on-ramp time gains in productivity from their ERP system. Yes, we are a little integration-centric, but with potential impact like this, it’s no wonder.

Trying to Understand Cloud Integration?

david.inbar — Thu, 28 Jan 2010 19:37:00 GMT

Integration with the cloud, in the cloud, through the cloud or even despite the cloud? It all seems like pretty dense fog. As usual, it all depends on your vantage point and where you need to get to.

Traditional ISVs are mostly focused on ways to reach the cloud from their on-premise apps (assuming that they have already conquered all the issues around integration with other on-premise systems). SaaS vendors have to somehow reach from their own app-in-the-sky to other clouds and on-premise systems. And IT organizations are trying to figure out if they can off-load some tasks to public or private clouds.

The cloudy weather ahead poses lots of challenges and lots of opportunities. If you’re trying to figure it out, join the conversation at Cloud Integration .

The 360 Degree View of the Customer – Attainable Holy Grail or Mirage?

hollis_pvsw — Fri, 22 Jan 2010 19:53:00 GMT

For years and years, I’ve been reading and hearing about the “360˚ view of the customer” from pundits, analysts, vendors, and systems integrators alike. I’ve been bombarded with seminar, webinar, symposia, and white paper download invitations. I might even admit to having written such an invitation. The goals are worthy, and the benefits compelling: better understanding your customers in order to more effectively acquire and retain business. By creating a 360˚ view of their customers, Best-in-Class companies increase customer satisfaction and retention. And of course, satisfied customers translates to increasing their share of the "customer's wallet," new account acquisition and improvement of bottom-line results.

But how many organizations actually achieve such a lofty goal? And how do they do it?

The Aberdeen Group has a pretty valuable offer out there – by participating in a brief survey by the Aberdeen Group you will be able to see how your experiences in providing a "360 degree view of the customer" compare with those of your peers, benchmark your performance, and see how you can achieve Best-in-Class results.

It's a great opportunity to gain insight into achieving meaningful progress towards this worthwhile goal.

Data Integration – It’s the value, stupid!

hollis_pvsw — Fri, 11 Dec 2009 17:04:00 GMT

Data Integration – It’s the value, stupid!

Written in Gartner’s typically diplomatic language, the latest Gartner Magic Quadrant on Data Integration reveals just how much pressure is being put on traditionally complex and expensive products in the data integration space – “During 2009, vendors in the data integration tools market experienced increased scrutiny from buyers with regard to pricing, cost models, time to implementation and the quality of service and support”.

It’s interesting that it’s taken unprecedented economic pressures to get companies to really start getting to grips with the horrendous amounts of money and time they traditionally spend on integration. The good news is that there are documented approaches to doing this methodically (see the Bloor Report).

As a thought leader in this space, I suspect Gartner will start reflecting value as one of the key criteria in its magic quadrant rankings. Hopefully a reflection of value will come in time for next years’ holiday season, building on the integration-related things to be thankful for noted by Loraine Lawson.

If you think your data is a mess now…just wait.

hollis_pvsw — Thu, 03 Dec 2009 21:50:00 GMT

If you think your data is a mess now…just wait. It will get worse, if you’re not careful.

That loud rumbling sound you hear isn’t thunder, or an earthquake—it’s the noise from an avalanche of exploding volumes of data, in places and formats that never existed before.

This explosion is making a difficult situation even more challenging, and what’s worse, it has a direct impact on business productivity. Only 21% of CEOs have the comprehensive information they need about their customers to make strategic decisions, according to the recently released 2009 PriceWaterhouseCoopers CEO report (http://www.pwc.com/gx/en/ceo-survey/index.jhtml).

The avalanche of disparate data, formats and applications has exacerbated a classic organizational challenge: critical data about customers stored in a myriad of different places and formats inside the enterprise, ranging from spreadsheets to databases to various enterprise systems (such as CRM or ERP) as well as various types of structured and unstructured data files.

Now it’s in all those same places AND in a host of other places outside the control of the enterprise. SaaS deployment models are here to stay, and enterprises are adopting SaaS applications—especially those for managing all the various aspects of customer relationships—at breakneck speed. And Cloud computing platforms such as Amazon EC2 are increasingly being leveraged by organizations as well.

With data in all the traditional places, in a myriad of different SaaS applications as well as on various Cloud platforms, your problem has grown by several dimensions. The explosion has fractured corporate information into a shrapnel of even more diverse, dispersed and complicated messiness.

What had been merely a difficult problem- for example, how to reconcile your customer data from your sales system with the billing information in your accounting system with the support data in your customer support system—now includes the added complexity of data hosted and managed in multiple places outside the corporate walls, as well as data located on-premises.

How can you cushion yourself and your organization from the impact of the explosion of data, applications, and associated headaches?

Advance planning will provide what’s needed: a capable and flexible architecture for data integration in place before you start deploying strategic assets in new and unfamiliar places.

You can survive the avalanche of exploding volumes and variety with architecture based on data integration components that have leading-edge connectivity to all SaaS applications, Cloud platforms and to all of your key existing on-premises systems. This architecture needs to simultaneously support data integration inside the firewall and among various SaaS and Cloud systems. It also needs to bridge on-premises to the SaaS/Cloud world. This will require a comprehensive connectivity architecture, and may demand the ability to deploy integration components on the Cloud as well as on-premises.

As with so many other data challenges (such as data quality), the best approach to managing exploding volumes of data—data in a multiplicity of new places and formats—is planning. It will be a lot easier and a lot less expensive to contain the avalanche now than to be buried by it later.

Look Here for Cloud Innovation

hollis_pvsw — Fri, 09 Oct 2009 15:01:00 GMT

For the past 18 months, the Pervasive DataSolutions team has been building innovative cloud-based integration solutions (Pervasive DataSynch for QuickBooks and Salesforce as a flagship example). Mainly the solutions have come in the form of a value-add integration service, and the Pervasive DataSynch has become one of the most proliferated integration add-on applications for salesforce.com and QuickBooks - with 150 customers and counting and the highest number of reviews for a QuickBooks integration on salesforce.com’s AppExchange site.

These cloud-based integration solutions are deployed on a platform called the Pervasive DataCloud. The Pervasive DataCloud was created as a by-product of the DataSolutions team needing to build their own products more rapidly and efficiently, but they were the only ones building them.

That is until the Pervasive DataRush group asked DataSolutions to build a new product for them. DataSolutions didn’t have the bandwidth to squeeze in another product, but realized if they could expose the web services, the DataRush group could build the product themselves. The Pervasive DataRush team, and one of their partners, Aha! Software, recently released a beta of this new service, deployed on the Pervasive DataCloud. The service, called Strato-Studio, is a MonteCarlo simulation that does predictive analysis. What does that mean? Well, you feed Strato-Studio some predictive modeling data, it crunches it many times, really fast, and then gives you pretty good idea where things are headed. That’s the quick version, but you get the idea.

So where does Pervasive DataCloud fit?

The DataSolutions platform (Pervasive DataCloud) is implemented on Amazon’s elastic cloud services layer. Services can run on just about any hardware and software combination and are billed by usage. The value for Strato-Studio is that Monte Carlo Simulations are very machine intensive, and the bigger the machine the better the analysis. It also requires some parallel processing power that is rare, but is available in the Pervasive DataRush engine. These requirements present a challenge for businesses that do not have (machine or programming) resources readily available. By empowering the Pervasive DataRush team and Aha! to build their simulations on the Pervasive DataCloud, they are able to leverage both in a very repeatable and scalable model. Consumers of their service can enjoy the benefits of a powerful computer plus sophisticated software and only pay for what they use!

This is exciting news. By creating an API for the existing DataCloud infrastructure, not only do we enable other divisions within Pervasive the ability to develop services on the DataCloud, but anyone and everyone who wants to publish their own services can do so! This marks a new era in the availability of data services and the power that can be delivered. But most importantly is how quickly the services can be delivered. Pervasive has always been able to help innovators implement data services, with its multiple data processing engines (transformation, process flow, parallel process flow, profiling, etc.), and now those same services can be exposed in the form of consumable web services. Platforms that never before had direct access, or limited access and required dedicated hosted hardware can access data services on demand. Developers on force.com, Intuit Partner Platform, Amazon AWS, Microsoft Azure, e-commerce sites, or any process that can call a web service (SOAP or REST), can leverage the power of a data service running on the Pervasive DataCloud.

Over the course of the next few months, Pervasive will be rolling out documentation on how to design, deploy and manage the services from a developer’s perspective. And we will also be posting examples of how to consume the services already running on the Pervasive DataCloud. We’ll be unveiling more information in the coming weeks ahead so stay tuned.

Cloud Isolation?

david.inbar — Thu, 24 Sep 2009 22:12:00 GMT

As you may have guessed, Pervasive Software has strong ideas about where data integration is heading. After reading a recent blog post from David Linthicum on “Why Data Integration is Critical to Cloud Computing” on ebizQ, I thought I should weigh in on the discussion.

David says in his post that data integration is critical to delivering cloud-based applications, but that most organizations approach it as an afterthought. He adds that implementing integration before joining the cloud is the optimal approach. I entirely agree. It is all too easy for your data to end up just being isolated in the cloud and all the other expected benefits disappear into thin air. However, I do not believe that this is a before vs after kind of discussion. You can start earlier or later, but rather than worry about when to integrate, the main issue will be the proliferation of apps and interfaces on multiple clouds as well as on-premise – and you will need a way to handle these challenges within tight budgetary limits. That’s why we will be talking more about how you can tackle this challenge without breaking the bank.

Do Appliances Make Sense for Data Integration?

david.inbar — Thu, 03 Sep 2009 15:00:00 GMT

There has been a lot of talk about delivering integration in the form of appliances. I recently came across a blog posting by Loraine Lawson of ITBusinessEdge (August 19^th) titled A Tale of Two Data-Integration Technologies that touches on this topic. Integration appliances are supposed to remove complexity for users, but there is no hard evidence to support this proposition. More likely, delivering software bundled with hardware (aka appliances) is a distraction. In reality, integration success depends on the quality of the software in terms of connectivity, the ability to handle high data volumes and to make changes quickly without specially trained staff.

In the end, there may be a place for appliances, particularly where they incorporate non-standard but very high performance hardware. However, in a world of powerful, cheap standard hardware, virtualization and the mix of SaaS and on-premise applications, integration appliances probably add complexity to the mix.

Reflections on 1,000 SaaS integrations

hollis_pvsw — Wed, 12 Aug 2009 16:43:00 GMT

Pervasive Integration recently celebrated its one thousandth SaaS integration. Pervasive has been integrating Salesforce CRM longer than any other vendor (since 2001), and has more successful integration projects (more than all the other integration vendors combined).During the past 8 years, we've learned a lot about Salesforce integration, as well as SaaS and Cloud integration. We've:

integrated customers of all sizes, from the very large (like Honeywell) to the very small
done salesforce integrations that involve a few thousand records, and others that involved many, many millions of records (like Chevy Chase Bank)
implemented simple synchronizations, simple bi-directional data movement, and sophisticated systems involving many different endpoints (like Schroders PLC, who had over 160 different data formats)
built integrations of all types: SaaS to SaaS , SaaS to on-premises - and multiple SaaS applications to multiple on-premises systems
been embedded inside OTHER applications (SaaS as well as on-premises) that wanted to include Salesforce CRM integration as a key built-in feature

Our goal has always been rapid time-to-value for Salesforce customers. That's a key tenet of the SaaS world. I don't know the details of every Salesforce integration ever done by Pervasive, but Falcon Storage was up and running (in production) in 60 minutes. Even the more complex environment of Schroders PLC (a global asset management company with £113.3 billion (EUR 133 billion / $186.5 billion) under management as at 30 June 2009) had an implementation timeframe of only 48 hours.Pervasive Integration for Salesforce - it's powerful, easy to implement, offers fast time-to-value, easy to manage, and affordable. That's the nature of the SaaS world, and that's why we've been successful at it.

Data Integration – the foundation for successful Business Intelligence

hollis_pvsw — Fri, 17 Jul 2009 19:42:00 GMT

Pervasive Integration is now Tweeting!

hollis_pvsw — Thu, 09 Jul 2009 19:21:00 GMT

Follow Pervasive Integration on Twitter and keep up with all the latest news!

http://www.twitter.com/PVSWintegration

Why should software vendors care about integration?

hollis_pvsw — Mon, 06 Jul 2009 17:29:00 GMT

I love the software industry, and I love working for independent software vendors. There is something very energizing about a group of people who have a vision about how something can be done better or differently, who then create a software product to make that happen.

At the Metamorphosis 9 event, I had the chance to speak with an entertaining and informed guy by the name of Matthew Bather, who is a product manager working for Exact Software. Exact Software is a highly successful vendor of ERP software – over 100,000 companies make use of its products and services, which are provided by 2500 employees spread across 4 continents.

So it’s interesting that a company whose focus is on making the best possible ERP software for its customers would place such a high value on offering an integration platform. Many ISVs focus their efforts on enhancing their core software product – be it ERP, content management, sales force automation, project management or whatever, and to offer a comprehensive integration layer would be considered a distraction.

Matthew gave a rationalization along the following lines, “technological and architectural upgrades are prerequisites for the successful realizations of the ambitious business plans we have.” By offering a value creation and value delivery process that integrates information - including multiple Exact products as well as third-party back office applications and data, Exact was creating more value for their existing customers and a much larger targetable market for their product.So, in my interpretation of this statement, integration IS a core part of ERP, and it’s strategic to the successful realization of an ERP vendor’s strategy. Or a CRM vendor’s strategy. Or…or…or. We live in a world full of hundreds of thousands of different kinds of data sources. Businesses run on this data. If an ISV wants to create a product that adds maximum value to its customers, and wants to make it easy to integrate into existing business operations – and wants to address the largest possible market, a solid integration strategy is not only pragmatic, it’s necessary. And that’s a good reason for ISV’s to care about integration.

Trust SOA?

Paige Roberts — Tue, 02 Jun 2009 17:51:00 GMT

Joe McKendrick over on ZDNet had some good points about why SOA sometimes gets bad press, despite it's advantages. Good things to keep in mind before trying to implement one in your business.

http://blogs.zdnet.com/service-oriented/?p=2056

Paige

Dev Hack: Make Complex Maps Simple

claviolette — Fri, 15 May 2009 17:21:00 GMT

A Shortcut for Dealing with Complex Parent-Child CRM Data

by Jacob Hughes, Integration Technical Support Engineer, Pervasive Software, Inc.

Stuck on a complex integration problem such as parent child relationships for CRM targets? Try staging your child data in a delimited ASCII file, and performing lookups for the necessary relationship fields before uploading them to CRM applications like Salesforce or MS CRM. This follows the KISS principle.

I recently used this technique on a consulting project that I helped out on, and it resolved problems we were having with duplication in Salesforce due to the complexity of the event handlers needed to handle parent and child data cleanly all in one map.

The ASCII connector is one of the fastest, if not the fastest connector Pervasive has, so the sorting and filtering happened at lightning speed. By storing child data in an ASCII file, it makes your maps much simpler and doesn't add much wait time to the overall process. Staging data in ASCII files can actually speed up processing by Pervasive Data Profiler.