The shift towards cloud apps is growing fast with most organizations shifting significant portions of their application portfolios to the cloud. The overall enterprise application architecture thus inevitably exists of inhouse apps and a variety of cloud apps that all need to talk to each other. In parallel, the SOA wave of a few years ago is still catching up in products with most vendors exposing atleast rudimentary services for connectivity with their apps. While Integrating with cloud apps one of the key issues is putting these together is to ensure transaction consistency. In particular atomicity i.e. that transactions either get done completely or get rolled back to their original state.
How is calling a series of service operations different from calling a series of native API calls?
- The cost of calling is not insignificant. Network latency (the time taken to send the message back and forth from the service) constitutes a large portion of the overall processing time.
- Messages to and From services are thus defined to be bulkier taking as much in the network journey as possible.
- Proxies, Gateways and Applications servers can introduce additional delays, latency, sequencing and throttling issues
- Method Level Transaction Atomicity: Its possible for services to respond to a request (considering it done) and for callers to not receive these responses, generally due to network issues. This makes callers retry and cause issues like duplication etc.
- Atomicity across a series of method calls: Most real life scenarios can require more than one call to make them complete successfully. You could for example need to make a call to create an Invoice, then another one to Create an Address and then another one to associate them together.
One of the most common patterns here is to pass in a transaction ID with the request. Services are required to build in a transaction check in that if the same transaction is passed in twice an error is thrown or the second one is ignored. Instead of an explicit transaction ID, a natural key can also be identified based on the data being passed in, e.g. a InvoiceNumer-DateModified combination can identify an Invoice Update Uniquely.
Methods are designed so that if they are called twice that has no impact. This can be tricky but possible in most scenarios and makes execution very efficient in most scenarios. Most large ERPs though dont allow for complex coarse grained service design like this. Nevertheless if possible a Create-Or-Update method is usually a lot more resilient than a Create and Update method separately.
Data persistence and Manual Rollbacks
The above three methods mostly address method level atomicity approaches. In scenarios with multiple method calls however, the problem of overall atomicity will still remain an issue. Assuming that an overall transaction coordinator is not regulating the interactions (which is the case in most scenarios), there is only one option left with the consuming client: To manually simulate the behavior of a transaction. Basic fine grained CRUD transactions usually can enable undoing anything. e.g. for a delete for a create, a create for a delete and an older update for an update call. This is a loose approximation of a transaction but probably the best we can do given the lack of proper transaction management on the services. The approach also requires complex data persistence expectations and are extremely hard to test out in their entirety.
Another common pattern here is to control sequencing of tasks so as to minimize the number of things that require rollback.
A related question here is whether you really need your services to be atomic. Design changes depending on whether you NEED your services to be atomic (i.e. for the data to be back in its original state in case of failure) or whether you would like it to just be resumable. E.g. if you had a service of three method calls A, B and C to complete a transaction, and a failure occurred at B. Do you really need to roll back A or just to be able to resume from B in a subsequent call.
Newer WS-I protocols make special consideration for managing distributed service transactions and to ensure reliable delivery. These provisions exist both at the method level and across multiple calls. The adoption of WS-I with vendors is still very limited so we will not be discussing it in detail but suffice to say, if your service vendor supports these, they are definitely worth considering in the implementation.