db Systems

We are always working to best leverage our experience and tools with potential partners. Our latest partnership was to propose, and seed the staff of, a team to design, develop, deploy and operate a geo-diverse Integration Platform as a Service (IPaaS - Wikipedia) for a large wireless provider. The primary goal of the team is to provide a solution to the business, in spite of being told some goal can't be achieved by existing processes.

Many companies today use an outsourcing strategy to focus on core competencies and reduce risk. Our customer's outsourcing strategy was more extreme than most, requiring a core competency in managing Enterprise Application Integration (EAI - Wikipedia). We leveraged this competency to develop an equally extreme tactical integration team.

In legacy telecom networks, EAI is primarily implemented using point-to-point connections between vendors' components. This works reasonably well because most interfaces are standardized, allowing vendors to establish interoperability in integration labs. However, current wireless providers' networks are far from fully standardized. Many new over-the-top technologies are invading the space causing operators to scramble for consumer value and market share.

To deal with emerging services and rapidly evolving interfaces the, EAI preferred, hub-and-spoke topology is used over point-to-point. (A bus topology is a specialization of hub-and-spoke.) The most obvious reason is point-to-point connections between N partners can require up to (N(N-1))/2 connections. Whereas hub-and-spoke requires only N connections. Most importantly, the hub provides a place to host custom integration software. (This feature of hub-and-spoke is exploited to achieve rapid time-to-market, below.)

Having to implement only N interfaces is considerably less work than (N(N-1))/2. Conventional wisdom says, if the N interfaces are clones of a single set of general interfaces, it is even less work. In addition, since there are N partners working on N interfaces in parallel, all N partners can do the work in the same amount of time as 1 can. (See the Wikipedia EAI page for a more expanded version of this argument.) But take note, it also says this fails 70% of the time. There is a fundamental problem with a process that has less than a 1 in 3 chance of succeeding. What is it?

What is great about software being non-physical, is the ability to draw analogy from other non-physical disciplines. In this case, consider the field of international negotiations, specifically language and cultural barriers. What result would one expect if a common language and culture (e.g. Esperanto) first be developed and learned before negotiations begin? (Ignoring the fact the ability to negotiate a common language and culture begs the question.) The burden of translation is born by each party. To have a chance of working, all parties must be equally motivated to negotiate.

However, in our case we have a single party driving the need for negotiations (our customer). Which tactic will provide a greater chance for success? The single party:

• Develops a common language first then forces all parties to learn and use it.
• Forces all other parties to use the single party's native language (e.g. English).
• Provides translation services to each other parties' native language?

Even ignoring the considerable advantage controlling language and cultural translation has, the third tactic always carries the day. By far, the worse tactic is the first (and perhaps why Esperanto has been replaced by English as the de facto language of international negotiation, the second tactic).

The conventional wisdom is wrong in the EIA case. Creating general interfaces to cover all planned (and future) use cases is a daunting task for all but the most trivial businesses. The information required to design common interfaces requires considering all partners unique needs (it begs the question). Consider all the latency in communications; document reviews, meetings, calls and emails. Time zones, priorities and understandings don't align. Then, after the interfaces are released, the communication roles are reversed with the same challenges. The smallest interface change requires all the partners to agree and schedule releases. It can't be a surprise that time-to-market is destroyed.

Our team uses the third tactic. We address time-to-market directly - minimizing or eliminating communication latency:

• We use hub-and-spoke too, but the spokes are each partner's native or off-the-self interface (we provide translation services).
• There is no common information model (or interfaces) to be created or communicated.
• Internally, the topology is point-to-point. The integration team accepts the burden of integration complexity and implementation. Partners' changes and releases don't have to be coordinated with other partners.
• By using several key strategies (see below), the size of the team required is much smaller than would be expected to deal with partners' information models.
• Partners don't have to change anything to integrate. The requirements flow is from the business to partners to integration team. Partners speak in their terms. Interfaces are usually partners' existing APIs.

The strategies used to create a smaller team are:

• Deliver services not software - Almost nothing requires more intense communications than an inter-partner SDLC. By keeping software as an integration team internal artifact, partner integrations are limited to verification and use of their unique point-to-point interfaces.
• Decrease implementation complexity - The KISS principle says it best. A small time to repair is easier to achieve than a large mean time between failure. Since either delivers high availability, do it the easier way.
• Out source the easy stuff - hardware, space, power, cooling, network and Internet access.
• Eliminate "refactoring" as a response to, even radically, changing requirements by eliminating structure in software. Go "back to the future" - before structured programming.
• Eliminate software specialization - Use a few computationally efficient orthogonal languages - C, SQL and rules.
• Very few library dependencies - operating system (FreeBSD), SQL library (SQLite), messaging library (Spread Toolkit) and rules library (CLIPS and DTC).
• Fewer people are required for a 24/7 team since there is only one fundamental role, a programmer/operator (devops).
• Don't duplicate existing functions - All other roles are filled by the company's existing out sourced partner support teams. Requirements generation, project management, test plan generation, testing execution, vendor management and NOC functions don't need to be duplicated.

The result? Over four years, with dozens of partners added and hundreds of releases per year, not a single delivery date was missed. To provide a sense of the scope of the project:

• Integration Partners: Just under 40.
• Volume: Thousands of transactions per second (TPS) during busy hours.
• Availability: 4 9s.
• Sites: Three - New York, Los Angeles and Dallas.
• Nodes per site: Four consumer grade "commodity" dedicated servers.
• Staff: Three.
• Operational budget (not including staff): Just under $700 per month per site.

References are available upon request.