Software Integration - EAI - Enterprise Application Integration
Integration is a matter of life and death to most businesses. As businesses keep acquiring software packages, they come to realisation that those packages alone can't deliver competitive advantage.
Competitive advantage is about INNOVATION, and there is no software package for that!
Mixing and matching different technologies is what gives the edge to many businesses, and that is impossible without glue holding all the parts together. Software Integration solutions is that glue.
We do realize that there are plenty of Software Integration solutions on the market.
Our people have had experience with many of them and our solution was developed not because we had nothing better to do, but because we found better and more efficient ways of doing things.
Just a few features of our Integration Software:
- Slim and Fast.
- Self-recovery from network and database connectivity problems.
- Can run on very low spec antic computers, but is also scalable to very large systems.
- Performs Dry Runs right in Production. No need in Dev, Test environments and SDLC. No unnecessary deployments, hardware and software costs. Avoid compliance costs for Dev and Test data regulations. Test in real production conditions.
- Provides separate notification channels for Application and Data problems. That allows for automatic redirection of data problems to those who supplied the data and minimisation of own support personnel and costs.
- Acts differently on transient and permanent errors.
- Out of box policies enforcement on incoming data.
- Exact row and column location reporting on data that violates configured policy.
- Subscription to alerts for different Interchange events.
- Functional acknowledgements returned to partner's computer systems.
- Safeguards against multiple submissions of the same file or data record.
- Monitors its own hardware footprint and other vital parameters.
- Monitors its own alerts, won't flood your mailbox with the same message every second.
- Ability to set resource usage priority for each individual Interchange.
- Works with very large files (including XML) without taking over RAM.
- Resistant to SQL injection attacks.
- Allows for multiple database transactions within the same interchange.
- Robust system with self-diagnostics.
- Multiple failover levels for diagnostics.
- Configurable throttling for each Interchange to give other applications a window.
- Flexible design with easily added Extractors and Loaders.
- Flexible configuration. Keep your installation slim and pay only for what you use.
- Interchanges configurations can be easily copied, version-controlled and compared side-by-side in text editing tools.
- Flexible scheduling functionality.
- Does not require any RDBMS for own functioning. That significantly reduces number of points of failure, cost and simplifies installation.
- Interchanges separated into different Accounts. Management permissions can be configured to entire Account or individual Interchange.
- Third, most advanced incarnation built from ground-up, with previous incarnations built for The ECN Group (now b2be.com) and Eventchain Systems, serving Toll in2store, TNT, Qantas, CIBA Vision, The Dempsey Group, AutoNexus, etc.
Licensing fees are on server/desktop basis. There are no transactional fees when you run the software on your servers.
Unlike many other Integration Software products, for the same fee you can run as many instances as you want on the same machine, and because the software is slim, you can fit many instances into the same box.
That is handy should you still want to have separate QA, UAT and Production environments in the same box, as well as creating independent instances for different Business Partners Networks.
For small tasks you can license only Extractors and Loaders you need and buy other components later when you need them.
Give us a call or drop a line. No challenge is too big or too small.
Business Considerations for EAI and EDI options
When a business need arises (typically first time) to establish integration between applications, often the first thing that comes to mind is to ask a technical guy to prove his worth and script it.
When a software developer is available, that business requirement can end up as in-house software development project.
While all that looks like a reasonable course of action, there are many risks that come along with that:
- EAI and EDI require extensive experience in the field, as well as awareness of the best practices for any given use case.
- While the developer may deliver on a functional requirement to move data from A to B, in Integration domain it is just some 10% of the job. Most often overlooked non-functional requirements are:
- SDLC. If a small change needs to be made, does it require hiring the specialist again and him going through entire development-testing-deployment cycle?
- Maintenance of Development and Testing environments to enable improvements and fixes.
- Application self-recovery from connection interruptions, services and systems restarts, bad data, etc.
- Prevention of data loss or corruption caused by sudden systems failures, like power or network outage.
- Interchange roll-back in different ways depending on whether it was the App or Data at fault.
- Making clear distinction between Application exceptions and Data exceptions and having mechanisms of dealing with them accordingly.
- Audit functionality.
- Notifications redundancy.
- Functional acknowledgements.
- Extensive diagnostics to be able to pinpoint problems in timely manner.
- Security. Can you afford your data being stolen or altered the way you become liable for millions in damages?
- Extensibility. Can new features be easily added without rewriting significant portions of the application?
- Typical ad-hock solutions are not products that achieve desired outcome through configuration.
Instead, they achieve the outcome through writing code. While an EAI product had been tested with hundreds of implementations and millions of data records,
home-grown code has years to mature and reapplying it to another circumstance later on will likely require more bug fixes and more code maintenance.
The next step up is use of tools designed to do Extract-Transform-Load operations. The most notorious would be Microsoft SSIS which keeps changing brand names, but not so much its nature. A short Google search will show how much pain software developers have to go through to accomplish even most common tasks, and how much you should be prepared to pay them for their pain.
Downsides of using SSIS are many. Apart from the complaints software developers have, here are just a few business implications:
- Often takes a long time to find the cause of a problem, some problems do not cause exceptions being thrown. If you are contemplating use of SSIS where operations cannot be stopped for longer than an hour, you are suicidal.
- Neither can you edit SSIS package in production nor can you perform a Dry Run. Every small change or urgent fix has to go through SDLC. It is also expensive to maintain those Development and Test environments in synch with production.
There are specific cybersecurity regulations for data residing in Dev and Test environments, but very few companies are aware, leave along following those regulations.
Either you are going against those regulations or your data in not what you will encounter in production.
Often you also need to emulate environments you are integrating with and expect surprises when you start dealing with the real thing.
- Slight change in database table structure, and SSIS package will refuse to work.
- Running even a dozen of SSIS packages at the same time may result in RAM and database connections exhaustion as well as insane number of threads created by SSIS.
- At least 2008R2 version has memory leak, prepare to constant need of restarting your production server.
- Transactions management is very limited. It is difficult for a package to record a failure in a database and then roll everything back while leaving that recording committed.
- The designer is buggy. Often you can't figure out what the problem is and it is faster to start over from scratch.
- To have the full picture of a package one has to drill-down inside every object.
- Copy-and-paste, find-and-replace can't be used to create a new package from similar one.
- You can't compare packages side-by-side in diff tools; version control is not of much use.
- Even for common tasks you still need to write code, which returns us back to the problems described in previous chapter. That is aggravated by the fact that to use new functionality in latest .NET framework you need to upgrade the SSIS, which will cost you dearly.
- Working with complex XML files is unnecessarily labour-intensive.
- If your XML files have reference to DTD, you are screwed, SSIS will refuse processing them.
- SSIS is mostly limited to ETL; using it for EDI is a royal pain.
- Due to limitations of SSIS, it is often used as ETL engine inside applications developed in-house, hence the drawbacks described in previous chapter also apply.
- SSIS developers who are familiar with that minefield are very expensive.
The best way of doing Software Integration is to use specialised middle-tier applications as glue.
This approach capitalises on the fact that those applications have been already tested by many clients under all sorts of conditions and unlike one-off programming code, evolved to be robust, secure and efficient.
There are many EAI products out there, but be careful not to be fooled by the marketing.
Apart from plain vapourware, there are other gotchas to look for. Most vendors will make their software look like it will magically solve your problems as soon as you buy and install it.
Nothing can be further from the truth. EAI is a very complex discipline. Implementation of EAI requires experience!
Most integration products come to the market with promise to make the implementation done fast. Most often they have designer tools that promise accomplishing tasks by drag-and-drop in minutes.
Those features are designed to impress the decision-makers and procurement department into belief that development of EAI from now on will be a walk in the park.
The truth is that the technical people who have to do the actual work will discover that the designer tools are not really helping to get the job done faster, they are rather an obstruction.
UI in EAI software is notorious for corrupting existing solutions and making copying and version-controlling a risky or impossible business.
Every SSIS developer will testify how much pain may be involved in making a package's copy work. Quite often the only way to fix the corrupt package is to edit the dtsx file directly.
Vendors of EAI products have another trick up their sleeve you should be aware of. They complement their product with training offer (typically at extra cost as another stream of revenue).
That also serves as guarantee that the software can be handled by the end user.
In reality, in most cases it ends up with the vendor getting contract to build the integration and maintaining it. The client's staff will be reluctant
touching anything because it may cause untold damages and the vendor will charge premium to fix results of what seemed like a reasonable tweak.
Surely, part of the problem is that the one building interchange must be experienced in the field, which is quite complex and not for uninitiated (though developers who never did it think it is easy).
Another part is that the vendor has vested interest in not fixing "unexpected behaviours" in their product and only their representatives know how not to step on those landmines.
So, what is the value of the designer tool if as an end user you never touch it?
The best way to go about EAI is to find a service provider who will deliver your software integration within your budget and steer away from those pushing other services you will never need.
Just make super-sure that the Application they use won't require you to invest in new hardware, won't be breaking every day, consuming entire RAM of your computer and won't create security holes.
On the other hand, if you think that your integration task is indeed trivial, why not to get a copy of our software, follow the manual and do it yourself?