WADL in a bottle eating noodles

In my last entry about REST web services I talked about its biggest weakness for me: the lack of description model of REST services.

The idea of hitting an HTTP endpoint as a shot in the dark is for me quite a leap of faith, and very likely an invitation to spend hours troubleshooting.

But despair no more, enters WADL! If it sounds like WSDL, it’s because it’s essentially the same acronym:

Web Service Definition Language -> WSDL

Web Application Description Language -> WADL

So WADL aims to be the WSDL of REST.

But… it was submitted the W3C in December 2009 by Sun Microsystem… one month before it was acquired by Oracle. Since them, it hasn’t budged… coincidence?

No other parties seem to have backed it, so it seems deemed to join the junkyard of unilateral attempt at standardizing global assets!

You can look up at an example on Wikipedia.

Maybe we’ll have another standard one day. Or maybe it’s a non-issue and I’m the only one to worry about it.

REST style with Hypermedia APIs

Once upon a time there was SOAP. SOAP really was a multi-vendor response to CORBA. It even share the same type of acronym, derived from object. Objects are so 90′s dude… The S in SOAP stands for Simple by the way. Have a go at a bare WSDL and try to repeat in your head that it is simple…

Then REST came along. I remember reading about REST back in 2002. It was a little after Roy Fielding‘s seminal article (actually his PhD thesis). Then there were a few articles about how SOAP bastardized the web and how XML RPC was so much better. But like the VHS vs Betamax battle before, the winner wasn’t going to be chosen on technical prowess. At least not at the beginning.

Then I stopped hearing about REST in 2003 and started seeing SOAP everywhere. We implemented it like COM+ interfaces really. A classic in the .NET community was to through Datasets on the wire via SOAP services. That really was a great way to misuse a technology… Ah… the youth… (a tear).

Microsoft tried to correct the trajectory by introducing WCF which enforced, or at least strongly suggest, a more SOA approach with a stronger focus on contracts and making boundaries more explicit. But somehow it was too late… something else was brewing beneath the SOA world…

In 2007, REST came back into fashion but now it was mainstream, i.e. people didn’t understand it, misquote it and threw it everywhere. Basically, it was: cool man, no more bloody contracts, I just send you an XML document, it’s so much simpler! Which of course works awesomely for 2-3 operations, then you start to get lost without a service repository because there are no explicit documentation!

If you see a parallel with the No-SQL movement (cool man, no more bloody schema, I just throw data in a can without ceremony, it’s so much simpler), I got no idea what you are talking about.

Anyway, if it wasn’t obvious, I’m not at all convinced that REST services solve that many issues by themselves. Ok, they don’t require a SOAP stack which make them appealing for a broader reach (read browser & mobile). But without the proverbial Word document next to you to know which service to call and to do something with, they aren’t that easy to use.

Then, finally, came Hypermedia API… I’ve a few articles about those, including the very good Designing and Implementing Hypermedia APIs by Mike Amundsen. I found in Hypermedia APIs the same magic I found when looking at HTML the first time: simple, intuitive & useful.

Hypermedia APIs are basically REST Web Services where you have one (or few) entry doors operations and from which you can find links to other operations. For instance, a list operation would return a list of items and each item would contain a URL pointing to the detail of the item. Sounds familiar? That’s how a portal (or dashboard) work in HTML.

Actually, you already know the best Hypermedia API there is: OData. With OData, you group many entities under a service. The root operation returns you a list of entities with a URL to an operation listing the instances of those entities.

The magic with Hypermedia APIs is that you just need to know your entry points and then the service becomes self-documented. It replaces a meta data entry (a la WSDL) with the service content itself.

The difference between now and the 2000′s when SOAP was developed is that now we really do need Services. We need them to integrate different systems within and across companies.

SOAP failed to deliver because of its complexity but mostly because it’s a nightmare to interoperate (ever tried to get a System.DateTime .NET type into a Java system? Sounds trivial, doesn’t it?).

REST seems easier on the surface because it’s just XML (or JSON). But you do lose a lot. The meta-data but also the WS-* protocols. Ok it was nearly impossible to interoperate with those but at least there was a willingness, a push, to standardise on things such as security & transactions. With REST, you’re on your own. You want atomicity between many operations? No worries, I’ll bake that into my services! It won’t look like any else you’ve ever seen or are likely to see though.

Mostly, you lose the map. You lose the ability to say ‘Add Web Reference’ and have your favorite IDE pump the metadata and generate nice strongly type proxies that will show up in intellisense as you interact with the proxy. Sounds like a gadget but how much is Intellisense responsible for the discovery of APIs for you? For me, it must be above %80.

Hypermedia API won’t give you Intellisense, but it will guide you in how to use the API. If you use it in your designs, you’ll also quickly find out that it will drive you to standardise on representations.

Applied SOA: Part 9–Service Versioning

This is part of a series of blog post about Applied SOA. The past blog entries are:

In this article, I’ll cover Service Versioning.

In SOA a Service usually has many consumers & it should be able to evolve independently of its consumers.  By that I mean that when a Service evolves, consumers shouldn’t break.  Otherwise we would have a hard time to evolve services.

We can do quite a bit of changes to a Service without breaking existing consumers logic. For instance:

  • Changing Service implementation details, i.e. not its behaviour, e.g. tap to another Database, add logging, route information to different systems, etc.
  • Adding a new Service Operation to a Service
  • Adding optional data-items in message payload

What needs to remain stable for consumers to continue to operate normally is called the Service contract:  Service behaviours and information exchange patterns, i.e. schemas & bindings.  This definition of Service contract is broader than WCF Service Contracts.

Service Contract can be backward compatible, i.e. a new Service Contract can support the same consumer base (for instance when we add a Service Operation without altering existing ones).

We need to version Service Contracts in order to accommodate for breaking changes.  When an evolution of a Service requires to introduce breaking change in its Contract, we need to support the current version until all consumers can be moved to the new version.

We want to keep track of two types of Service Versions:  breaking ones and non-breaking changes.  Breaking change versions require multiple versions to co-exist in order to support different consumers while non-breaking change versions are useful to determine the capacity of a Service.

A typical versioning scheme is to track breaking change versions with major version number while non-breaking versions are tracked with minor version number.  This is the scheme used by Windows Azure REST services for instance.

Concurrent Service Versions

Maintaining more than one Service Contract version at the same time is notoriously hard.  Typically you introduce breaking changes because either a business process or a back-end system imposes that change on you.  Those changes are often hard to burry under a legacy Service Contract version.

That problem isn’t unlike its equivalent in the component world.  How do you evolve a component?  You version its interface and try to keep old interface backward compatible as long as you can before deprecating them.

Service Behaviours

Just a word about Service Behaviours:  those are encapsulated in Service Contract along operation signatures.  Behaviours are any logic surfaced to the consumer, as opposed to implementation details which aren’t surfaced.

For example, if a Service encapsulates access to a Database, a consumer won’t know which Database the Service read / writes from.  The exact Database is therefore an implementation detail.

On the other hand, if a Service operation takes an integer in input and validates the input as being greater than zero by raising a fault if that pre-condition isn’t met, that becomes a behaviour.  A consumer depends on that validation:  if a change requires the input to be greater than 5, consumer sending values between 1 and 5 will break.

Service Behaviours are hard to document with tools.  For instance, WCF Service Contracts do not capture pre & post conditions.  This typically require documentation on the side.

Applied SOA: Part 8–Security

This is part of a series of blog post about Applied SOA. The past blog entries are:

In this article, I’ll cover security.  Security is a very broad topic and by no mean unique to SOA.  I’ll focus on what is specific about service security.

I once had a colleague whose mantra around security was the triple ‘A’:

  • Authentication
  • Authorization
  • Audit

I actually find that triple ‘A’ a very good summary of security concerns in most systems:  who are you, what can you do with the system and keeping traces of what you did.

Authentication

When a service is invoked, it is done by something (e.g. a browser, another service, etc.) on behalf of somebody (e.g. an end user, a system user, an anonymous agent).

Some concerns jump in:

  • How do we identify users in your system, i.e. what is the identity model in your system?  Is the identity a username?  A SID?  A set of claims?
  • What is the authentication process, what are the credentials involved?  User name / password?  Certificate?
  • Where and when is the authentication done?  Once and a proof of that authentication (e.g. token, cookie) is transported thereafter or is it done at each tier boundary?  Service boundary?
  • How does it flow through service composition?  Does the end user identity flows all the way or does the identity of services take over somewhere in the pipeline?  How do you address the case where a service composition is done through asynchronous means, i.e. can the invoker identity be stored with the asynchronous message?

In the Microsoft realm of technology, some classical approach exist.  Using Windows Authentication is often used within a corporate network as a secure way to authenticate user with little friction (user is already logged in on his / her workstation).  Identity can flow via impersonnification or caller context.  The database is often considered a trusted subsystem, i.e. that it doesn’t require the end-user identity, trusting the calling services.  In some cases, you might want to flow the identity right through the database though, in order to strictly enforce security policy.

Outside corporate firewalls identity is often managed using custom means, i.e. username / password and a combination of cookie & custom token.  Claims based security model is rising and becoming more common but is still a young technology.  Interoperability is a challenge with Claims based as very little products support it hence you would have problem to flow the identity everywhere.  This is often managed by having a security policy of controlling the user identity at the Enterprise boundary and then flowing the service identity.

Authorization

Now that you know the identity of the agent invoking your service, how do you control access?  Many models are possible given the following variables:

  • Are you going to be context insensitive (e.g. this operation requires role X to execute) or context sensitive (e.g. this operation requires role X and role Y.<the name of the department you are trying to access> to execute)?
  • Are you going to control the access at the operation border only?  If so, a user can either invoke your service operation or not.  You might also need to control what the user can see and perform security trimming even once the user has been given access to the service operation.
  • Are you going to grant access based on role (RBAC), to a set of claims or to a set of permissions, itself determined from role, claims and other data about the user identity?
  • Where are you going to control access?  Everywhere?  At an entry point only?

Typically, you’ll have a mix of those variables in your system.  Just being aware of them and which compromise you are making in your architecture will go a great way of ensuring the clarity of the security model, how well it will be understood by different stakeholder and hence the strength of its eventual implementation.

Audit

Often forgotten in security is the audit.  Indeed, we can secure a system with access control (authorization), but another big requirements for a secure system is to be able to trace back what a given user did or who did what in a system.

Audit is the ability of tracing back actions.  This is typically enabled by logging actions to a store.

Now the first question you need to ask yourself is if you’ll need to audit easily, i.e. via some UI with search facilities, etc. in which case you want the logging information to be easily mined or is audit going to come as a consequence of some rare legal problems, in which case you just want the information to be available so that some professional could go and mine the information on a need-to base.

Logging can be done at different level:

  • At the service boundary you can log information about an operation, payload in / out, time of request, identity of the user, etc.  .  In an SOA architecture, this should be sufficient to reconstruct user actions.
  • You can continue to log within a service, at different layer, even going as far as logging DB access.  This is useful if your service implementation changes often or if their logic is strongly influenced by some configurable sources (e.g. business rule engine in BizTalk, custom application configuration, etc.) in which case service operation treatment might vary over time and it could be hard to reconstruct actions just by looking at service log.

You can log actions within your system, e.g. by stamping changes with a user name, by keeping all changes, etc.  .  This is done by systems such as TFS which never forget anything.  It is easier to do in such systems because they have generic entity management system.  On system managing many type of entities (e.g. many tables in a DB), it can become difficult to keep all changes in the system without cluttering information schema.  In those cases, you might want to perform logging on the side, e.g. in log files.

Regardless of where you log, it might be challenging to reconstruct user actions from log.  So it’s better to think in advance at what type of reconstruction you would need to do to make sure it will be feasible.  Often the reconstruction scenarios are simple, e.g. who changed the salary of X?  But sometimes, it can be harder, for instance, if you need to reconstruct a long running business process where different people can have played a different role.

Again, you’ll often end up with a mix of the options presented here, i.e. logging on the side + keeping last changes in store which you can then expose at the application level.

Golden rule

Architect with security in mind day 1!

We repeat that everywhere all the time but more often than not, security comes second.  This is actually quite natural since as an architect you typically sit down with business people to specs out an application.  Those stakeholders often have very remote concerns about security.  A good way to get around this tendency is to involve security-aware people (e.g. legal, your Enterprise Security guru) early in the process.

Applied SOA: Part 7–System Consistency

This is part of a series of blog post about Applied SOA. The past blog entries are:

In this article, I’ll cover system consistency.  In every distributed solution, a major concern should be consistency.  How to you ensure that different moving parts in your solution create a consistent state?

Many patterns and solutions exist to address that problem.  We’ll look at a few and see how it impacts other attributes of a SOA solution.

Distributed ACID Transaction

First solution that pops to mind is ACID transactions.  ACID transaction is a very elegant and very mature technology allowing distributed parts of a system to remain consistent.  I won’t go into details here and just assume you know the basics of ACID transactions.

ACID transactions can actually solve the consistency problem in a SOA solution as well.  We would need to use distributed transactions (through WS-Atomic protocol for instance) where transaction would span through the calling tree of a service.  This way, everything in the calling tree would be part of the same transaction and would either commit or rollback as a whole.  It would conflict with solution attributes of SOA though.

The main attribute it would conflict with is service autonomy / isolation.  A distributed transaction implies that a calling service holds lock in a composed service for a certain duration.  This means the composed service can’t guarantee to other callers it can process their requests.  That control has been partially externalized by the transaction.  It breaks the trust boundaries between services:  a service now needs to trust services invoking those services will hold locks on its resources.  Locks also limit scalability of services.

ACID transactions are excellent at ensuring data consistency but they come at a high cost.  This cost is deemed unacceptable in Cloud Computing because of scalability.  The same goes for services invoked across trust boundaries (e.g. B2B services).

Compensation

The typical fall-back pattern when transactions aren’t appropriate is Workflow + Compensation.  With this pattern, each service expose a rollback, or compensation, logic so that a calling service can call an operation and later on call another operation in order to rollback the effect of the previous transaction.  This typically assume that some sort of workflow engine is at the root of all those calls, orchestrating the service call chain.

This patterns also work and solve the consistency problem.  It introduces different compromises.  For instance, services must trust the workflow engine to take care of the consistency.  Mostly, it is a more complicated solution.  Instead of relying on a system (the transaction coordinator) to take care of compensation, you basically must implement it yourself.

Above the technical difficulty, it might be difficult to compensate after a certain lapse of time.  This would be a business problem.  A typical example is A puts $100 in B’s account, B pays a $90 bill, A wants to rollback its original transaction, but the money is already spent, what can we do?  This is where the orchestrating workflow will get more and more complicated.

Conclusion

We’ve seen the two basic patterns for consistency.  Typically we mix them together.  For instance, a service implementation will use ACID transaction internally to ensure consistency internally.  For read-only operations, composition without distributed transaction can be ok in many system where changes aren’t happening too quickly.  This typically leaves you with only a small set of services requiring state-altering composition.  Those often are business processes and are best handled in a Workflow engine.

Applied SOA: Part 6–Service Composition

This is part of a series of blog post about Applied SOA. The past blog entries are:

In this article, I’ll quickly cover service composition.

Service Composition is often stated as a goal of SOA.  I think it’s rather a consequence of a Service Oriented Architecture.  You first deploy building block-type of services.  Once those are running, you can start deploying higher order services reusing those building blocks.

As we’ve seen in the Service Taxonomy blog entry, a good service taxonomy helps you to structure service composition.  The one I presented, from soapatterns.org, has composition built-in.  Typically, a business processes compose many services.  An activity within a process might also compose a few more agnostic services.

Composition is good, it’s incarnated reuse, so we should want it everywhere, shouldn’t we?  Well, like with everything sweet, there’s a tension related to its usage.  In the case of composition, this tension is with isolation.  As we’ve seen in different SOA definitions, isolation is important.  Composition compromises isolation in many ways:

  • It couples SLA:  the composing service is just as strong as its weakest link at any moment in time.
  • It couples services time-wise:  composition increases latency but also require the composed service to be up-and-running.
  • It also often couples service contracts if the composing service expose data from the composed services.  One way or the other, changes in the composed service will affect the composing service.

This tension is the reason why you won’t want to layer your services in many layers.  Doing so would result in weak SLA, long response time and high-maintenance.  If you do not layer at all, you’ll end up with duplicated logic.  As usual, patterns and judgement should help you balance your solution!

Integration in the Cloud

I ran across a series of good blog posts I wanted to share with you, dear readers.

From Richard Seroter, the series is about integration patterns in the cloud, looking at cloud-to-cloud & cloud-to-on-premise scenarios.

Richard looks at three main patterns:

For each pattern, Richard describe the pattern, the general challenges, the challenges specifically related to the cloud and finally he demonstrates the pattern using different cloud provider (e.g. Google, Amazon, Azure).

His text is very thorough yet straight to the point.  He puts his fingers on the right issues.  Great work Richard!

Applied SOA: Part 5 – Interoperability

This is part of a series of blog post about Applied SOA. The past blog entries are:

In this article, I’ll cover the Interoperability.  SOA aims is to expose capabilities through services in order to let different agents leverage those capabilities.  A common theme is therefore interoperability.  Given the universal reach of services (as opposed to objects or components for instance), the interoperability bar tends to be quite high.

Technical Interoperability

Now the first thing that pops into technologists’ mind when interoperability is mentioned is technical interoperability:  the ability for two systems to communicate with each other despite their underlying technological stack differences.  This is no small challenges.

Indeed, we would think that with SOAP & WS-*, all those interoperability problems are behind us, all taken care of by standardisation bodies, right?  Well, not quite.  Although those standards enable basic interoperability in an unprecedented way, they still do not guarantee it.  The center of the problem is the way SOAP specs can be interpreted and how differently it was interpreted by different vendors.

As a result, it isn’t trivial to achieve interoperability between heterogeneous systems.  For instance, Microsoft published four months ago a package of bindings to use with different vendors.

Two opposing forces are at play when we consider technical interoperability:

  • The richness of the protocol used in the services
  • How interoperable we want it to be

For instance, consider the following list of Services protocol:

  • WS-* (e.g. WS-Atomic, WS-Security, etc.)
  • WS-I Basic Profile
  • REST / OData
  • SQL Views
  • File share

As we go down the list, protocols get simpler and simpler.  The first ones allow rich scenarios (e.g. distributing a transaction, sharing a claims based security context, etc.) while the later ones do not.  The amount of potential clients grow as we go down though:  who  can’t drop a file on a file share?

Now once you’ve decided on how much interoperability you can afford given the protocol your solution needs, you have many tactics to make it happen.  For web services, this typically involves getting closer protocol standards and not letting your development platform do as much as it otherwise would.

Indeed, many web services platform offer a great story to quickly author web services by eliminating a lot of the gory details of SOAP.  Different development platform (e.g. .NET vs Java) takes different decisions on those details which make them incompatible though.  Pesky details such as field ordering, native types to SOAP types mapping, how list of objects are mapped to XML schema, etc. can all bring incompatibilities between a service and its consumers if written in different platforms.

All that being said, those problems are known and great guidelines exist.  For the .NET platform, an excellent tool for authoring interoperable web services is WSCF (Web Service Contract First) by thinktecture.  Those tools accelerate protocol-centric development instead of marking details.

The key here is to test with the different target platforms.  Quite similar to multi-web browser development really.

Business Interoperability

Another aspect of interoperability is business interoperability.  Let’s say we solved the technical interoperability problem and we have a great recipe to build web services interoperable with our target consumers but that we do not speak the same business language, how really interoperable are we?

An example I like to give concerns two hypothetical services within a consulting company (yeah, I love consulting company examples!).  The first service returns a list of employees who didn’t fill up their weekly timesheets while a second service sends email to an employee.  We could easily combine those two services in order to warn offenders they haven’t filled their timesheet yet, couldn’t we?  What if the first service signature looks something like this:

EmployeeData[] GetOffenders()

where the EmployeeData data contract has the following shape:

EmployeeData :
{
  EmployeeNumber : integer,
  FirstName : string,

  LastName : string
}

and the second service looks like this:

SendEmail(emailAddress : string, mailBody : string)

Well…  we got ourselves a typical data mismatch problem.  The first service identify employees by employee number while the second one expect their email address.

A simplistic example, but quite common one in large enterprises, especially ones where acquisitions created a patchwork of different semantics (e.g. here a project is something with a billing code and people staffed on it, there it’s an aggregation of teams with no budget semantics).

Again, this is a problem with solid existing solutions, but it is still something to be mindful of when elaborating a Service Oriented Architecture.  Two typical solutions to this problem are:

  • Schema rationalisation (or Canonical Data Model, CDM) where services expose a canonical model where different entities only have one meaning and representation.
  • Entity Aggregation (e.g. Master Data Management) where different semantics and representations are allowed to exist but a central authority performs reconciliation.

In conclusion, it is important to define what type of interoperability (both technical & business) we target and to take the necessary steps to achieve it given the challenges of the solution’s context.

Applied SOA: Part 4 – Service Taxonomy

This is part of a series of blog post about Applied SOA. The past blog entries are:

In this article, I’ll cover the Service Taxonomy.  On an SOA project, we’re going to have services.  How should we classify them?

First why would we want to categorise them?

First, to give a structure to the solution.  Without structure, it’s hard to govern an important number of elements, especially if those elements are services.

Second, at the architecture level, we can give rules for categories as opposed to individual services.  Again, this is structure, we can manage fewer pieces.

Third, at the design level the taxonomy will help us clarifying the boundaries of a service.  At the end of the day, there are two extremes:  either you have one service with all the operation of your company in it or a lot of services with one operation each.  A service is just a container and unless you have some clear taxonomy, which operation belongs to which service can be a matter of taste and individual preferences, which seldom yields strong design decisions.

Finally, it gives us the ability to reason about the structure at a higher level without getting cut in the intricacy of specific services.

There are many ways to categorise services.  A popular one was defined by Tomas Erl in his book SOA Design Patterns.  That taxonomy is based on a few patterns exposed in that book.  Each pattern invokes separation of concerns in order to define a service layer.  Two axis are used for the separation of concern:

  • Agnosticism to business context
  • Business Logic

A component is said to be agnostic to the business context if it can be invoked from any business process and behave identically.  A component is said to be business logic specific if it contains business logic.

Nevertheless, this taxonomy is useful.  It defines three layers:

  • Utility Layer, agnostic of both business context and business logic.  On this layer exist services manipulating information without applying business logic on it or varying depending on the business context.  Typically those services offer communication capability:  emails, pub-sub, mapping, etc.  .
  • Entity Layer, agnostic of the business context but implementing business logic.  Services in this layer are typically data centric.
  • Task Layer, business context sensitive and implementing business logic.  Services at this layer typically participate in the activities of a specific business process.

Reusability is maximal at the Utility Layer and minimal at the Task Layer.  At the Task Layer, a service is specific to a business process (context) and can’t be reused elsewhere.

Business Logic increases as we go to the Task Layer.

Composition is done for the Task Layer to the Entity and to the Utility layer.  By composition I mean that a service from one layer can call a service from another one.  This is probably the strongest property of this taxonomy:  the composition rule is deduced from the definition of the taxonomy as opposed to imposed as another principle.

For example, a service at the Entity Layer, by definition, can be called from any business process.  It cannot, therefore, call a service at the Task Layer, being specific to only one business process.

I’ve used this taxonomy in the architecture of a sizable SOA project.  It worked well in terms of given structure and was quite resilient to changes along the project (e.g. security requirements, multiple consumers, etc.).

Something we did to enrich the taxonomy was to further subdivide each layer per domain.  A domain being an information / process breakdown.  Given those subdivisions, each service could only be within one layer and serving one domain within that layer.  That gave a very strong & clear service boundary definition.

Now the weakness of the taxonomy, we’ve found, is the vagueness of the definition of agnosticism to business context & business logic.  It sounds great on paper, but my service designer team, composed of brilliant and seasoned designers, spent the project having philosophical discussions about what is and what isn’t business context agnostic.  It boiled down to the fact that an operation being or not aware of the business context isn’t always clear.

On smaller project I’ve seen a taxonomy based only on data domains, hence having only an entity layer.  This simplifies the clustering of services as data domains are typically quite clear.  It does mix the concerns of process centric & data centric capabilities though.