How to improve Azure: Can you keep a secret?

In this blog series I explore some of the shortcomings of the Windows Azure platform (as of this date, March 2014) and discuss ways it could be improved. This isn’t a rant against the platform: I’ve been using and promoting the platform for more than four (4) years now and I’m very passionate about it. Here I am pointing at problems and suggesting solutions. Feel free to jump in the discussion in the comments section!

   
 

What is a secret in the context of a Cloud Application?

A secret is any credentials giving access to something. Do I mean a password? Well, I mean a password, a username, an encryption key, a Share Access Signature (SAS), whatever gives access to resources.

A typical Cloud application interacting with a few services accumulates a few of those. As an example:

  • User name / password to authenticate against the Azure Access Control Service (ACS) related to an Azure Service Bus (you access more than one Service Bus namespace? You’ll have as many credentials as namespaces you are interacting with)
  • SAS to access a blob container
  • Storage Account Access key to access a table in a Storage Account (yes you could do it with SAS now, but I’m striking for diversity in this example ;) )

All those secrets are used as input to some Azure SDK libraries during the runtime of the application. For instance, in order to create a MessagingFactory for the Azure Service Bus, you’ll need to call a CreateAsync method with the credentials of the account you wish to use.

This means your application requires to know about the credentials: a weakness right there!

Compare this with a typical way you configure an application on Windows Server. For instance, you want an IIS process to run under a given Service account? You asked your favorite sys-admin to punch in the Service Account name & password into the IIS console at configuration time (i.e. not at runtime). The process will then run under that account and never the app will need to know the password.

This might look like a convenience but it’s actually a big deal. If your app is compromised in the Windows Server scenario, there is no way it can reveal the user credentials. In the case of your Azure app, well, it could reveal it. Once a malicious party has access to account credentials, it gives it more freedom to attack you than just having access to an app running under that account.

But it doesn’t stop there…

Where do you store your secret on your Azure app? %99 of the time, in the web.config. That makes it especially easy for a malicious party to access your secrets.

Remember, an application deployed in Azure is accessible by anyone. The only thing protecting it is authentication. If you take an application living in your firewall and port it to the cloud, you just made it much more accessible (which is often an advantage because partners or even your employees, from an hotel room, have access to it without going through the hoops of VPN connections) but are also forced to store credentials in a less secure way!

On top of that, in terms of management, it’s a bit awkward because it mixes application parameters with secrets. Once a developer deploys or creates a deployment package to pass it to the sys-admin (or whoever plays that role, it might be a dev-ops developer, but typically, not everyone in the dev group will know about production credentials), it must specifies some arbitrary config keys the sys-admin must override.

So in summary, we have the following issues:

  • Application knows secrets
  • Secrets are stored in an unsecure way in the web.config
  • Secrets are stored with other configuration parameters and do not have a standard naming (you need to come up with one)

 

Ok. How do we fix it?

This one isn’t easy. Basically, my answer is: in the long run we could but cloud platforms haven’t reached a mature enough level to implement that today. But we can establish a roadmap and get there one day with intermediary steps easing the pain along the way.

Basically, the current situation is:


That is, the app gets credentials from an unsecure secret store (typically web.config) then request an access token from an identity / token provider. It then uses that token to access resource. The credentials aren’t used anymore.

So a nice target solution would be:


Here the application requests the token from Windows Azure (we’ll discuss how) and Azure reads the secrets and fetch the token on behalf of the application. Here the application never knows about the secrets. If the application is compromised, it might still be able to get tokens, but not the credentials. This is a situation comparable to the Windows Server scenario we talked above.

Nice. Now how would that really work?

Well, it would require a component in Azure, let’s call it the secret gateway, to have the following characteristics:

  • Have access to your secrets
  • Knows how to fetch tokens using the secrets (credentials)
  • Have a way to authenticate the application so that only the application can access it

That sounds like a job for an API. Here the danger is to design a .NET specific solution. Remember that Azure isn’t only targeting .NET. It is able to host PHP, Ruby, Python, node.js, etc. . On the other hand, if we move it to something super accessible (e.g. Web Service), we’ll have the same problem to authenticate the calls (i.e. requirement #3) than how we started.

I do not aim at a final solution here so let’s just say that the API would need to be accessible by any runtime. It could be a local web service for instance. The ‘authentication’ could then be a simple network rule. This isn’t trivial in the case of a free Web Site where a single VM is shared (multi-tenant) between other customers. Well, I’m sure there’s a way!

The first requirement is relatively easy. It would require Azure to define a vault and only the secret gateway to have access to it. No rocket science here, just basic encryption, maybe a certificate deployed with your application without your knowledge…

The second requirement is where the maturity of the cloud platform becomes a curse. Whatever you’ll design today, e.g. oauth-2 authentication with SWT or JWT, is guaranteed to be obsolete within 2-3 years. The favorite token type seems to be changing every year (SAML, SWT, JWT, etc.), so is the authentication protocol (WS-Federation, OAuth, OAuth-2, XAuth, etc.).

Nevertheless it could be done. It might be full of legacy stuff after 2 years, but it can keep evolving.

I see the secret gateway being configured in two parts:

  • You specify a bunch of key / values (e.g. BUS_SVC_IDENTITY : “svc.my.identity”)
  • You specify token mechanism and their parameter (e.g. Azure Storage SAS using STORAGE_ACCOUNT & STORAGE_ACCOUNT_ACCESS_KEY)

You could even have a trivial mechanism simply providing you with a secret. The secret gateway would then act as a vault…

We could actually build it today as a separate service if it wasn’t from the third requirement.

 

Do you think this solution would be able to fly? Do you think the problem is worth Microsoft putting resources behind it (for any solution)?

Hope you enjoyed the ride!

How to improve Azure

I’m very passionate about Windows Azure. I’ve been using and promoting the platform for more than four (4) years now.

So I’ve been working with the technology for a while but in the recent month I’ve been involved on an intensive architecture project where we pushed the envelope of the platform. As a consequence we did hit quite a few limitation of the platform.

I also had the pleasure of working directly with Microsoft to resolve some of those issues.

In this blog series I will address what still remain to this date (March 2014) limitations of the platform. Instead of winning about it, I will suggest ways Azure could be improve to address those shortcomings. That will be more constructive and will generate some discussion. Feel free to jump in the discussion in the comments section!

Azure ACS fading away

ACS is on life support for quite a while now.  It was never never fully integrated to the Azure Portal, keeping the UI it had in its Azure Labs day (circa 2010, for those who were born back then).

In an article last summer, Azure Active Directory is the future of ACS, Vittorio Bertocci announces the roadmap:  the demise of ACS as Windows Azure Active Directory (WAAD) beefs up its feature set.

In a more recent article about Active Directory Authentication Library (ADAL), it is explained that ACS didn’t get feature parity with WAAD on Refresh Token capabilities.  So it has started.

For me, the big question is Azure Service Bus.  The Service Bus uses ACS as its Access Control mechanism.  As I explained in a past blog, the Service Bus has a quite elegant and granular way of securing its different entities through ACS.

Now, what is going to happened to that when ACS goes down?  It is anyone’s guess.

Hopefully the same mechanisms will be transposed to WAAD.

Full Outer Join with LINQ to objects

Quite a few times it happened to me to be looking for a way to perform a full outer join using LINQ to objects.

To give a general enough example of where it is useful, I would say ‘sync’. If you want to synchronize two collections (e.g. two collections of employees), then an outer join gives you a nice collection to work with.

Basically, a full outer join returns you a collection of pairs. Every time you have both items in the pair, you are facing an update: i.e. the item was present in both collections so you need to update it to synchronize. If only the first item of the pair is available, you have a creation while if only the second item is you have a delete (I’m saying first and second, but it actually really depends on how you formulated the query but you get the meaning).

Whatever the reason (a sync is the best example I could find), here is the best way I found to do it. It is largely inspired on an answer I found on stack overflow.

public static IEnumerable<TResult> FullOuterJoin<TOuter, TInner, TKey, TResult>(

this IEnumerable<TOuter> outer,

IEnumerable<TInner> inner,

Func<TOuter, TKey> outerKeySelector,

Func<TInner, TKey> innerKeySelector,

Func<TOuter, TInner, TResult> resultSelector,

IEqualityComparer<TKey> comparer)

{

if (outer == null)
{

throw new ArgumentNullException("outer");

}

if (inner == null)

{

throw new ArgumentNullException("inner");

}

if (outerKeySelector == null)

{

throw new ArgumentNullException("outerKeySelector");

}

if (innerKeySelector == null)

{

throw new ArgumentNullException("innerKeySelector");

}

if (resultSelector == null)

{

throw new ArgumentNullException("resultSelector");

}

if (comparer == null)

{

throw new ArgumentNullException("comparer");

}

var innerLookup = inner.ToLookup(innerKeySelector);

var outerLookup = outer.ToLookup(outerKeySelector);

var allKeys = (from i in innerLookup select i.Key).Union(

from o in outerLookup select o.Key,

comparer);

var result = from key in allKeys

from innerElement in innerLookup[key].DefaultIfEmpty()

from outerElement in outerLookup[key].DefaultIfEmpty()

select resultSelector(outerElement, innerElement);

return result;

}

So here it is and it works.

You can easily optimize the signature by specializing for special cases (e.g. bumping the comparer, considering two collections of the same type hence requiring only one key selector, etc.).

For performance, I didn’t bother… but I wonder if creating those two lookups isn’t actually slower than doing a cross product (double loop) over both collection items and checking for key equality. My gut feeling is that it’s probably wasteful for small collections, worth it for big ones, hence if you optimize it, you do it for small collection which do not have performance problem anyway.

Enjoy!

Copy blob using SAS

I have been trying for a couple of days to find an easy way (read:  using tools) to copy blobs in Windows Azure Storage, not by using management keys but using Shared Access Signature (SAS).

Sounds simple enough.  I remembered the AzCopy tool.  I looked around and found a blog post explaining how to use it with SAS, using the DestSAS switch.

I spent hours and I could never make it work.  For starter, AzCopy is designed to copy folders instead of individual files.  But also, I could never get the SAS to work.

After those lost hours, I turned around and look at the Storage REST API.  Turns out you simply need to do an HTTP PUT in order to write a blob into a container.  If the blob doesn’t exist, it creates it, if it exists, it updates it.  Simple?

In PowerShell:

$wc = new-object System.Net.WebClient

$wc.UploadFile(<Blob Address>, "PUT", <Local File Path>)

The Blob Address needs to be the URI containing a SAS.

Enjoy!

Securing Azure Messaging Service Bus access

I am currently working on a very exciting project involving systems integration across the Azure Messaging Service Bus. I thought I would share some of the painfully acquired knowledge nuggets with you.

About %90 of examples you’ll find on Internet uses Azure Bus SDK with ‘owner’. That is basically ‘admin’ privilege because owner has read/write AND manage on an entire Service Bus namespace.

Although that is ok nice to get use to the SDK, but isn’t a very secure setting for a production environment. Indeed, if the owner credentials get compromise, it would compromise the entire namespace. To top it, Microsoft recommends not to change the password & symmetric key of the owner account!

So what is it we can do?

I’ll give you a few guidelines here but you can read in length on this excellent blog post or watch Clemens Vasters’s video.

Entities in Service Bus (i.e. Queues, Topics & Subscriptions) are modelled as relying parties in a special Azure Access Control Service (ACS): the Service Bus trust that its buddy-ACS, i.e. the one having the same name with a -sb happened to it, as a token Issuer. So access control is going to happened in that ACS.

You do not have access to that ACS directly, you must pass by the Service Bus page:


Once on that ACS, you can find the Service Identities tab:


And there, you’ll find our friend the owner:


So owner is actually a Service Identity in the buddy-ACS of the Service Bus.

Now, let’s look at the relying parties:


As I said, Relying parties represents Service Bus’ entities. Basically, any topic is the realm:

http://<namespace&gt;.servicebus.windows/net/<topic name>

while any subscription is

http://<namespace&gt;.servicebus.windows/net/<topic name>/Subscriptions/<subscription name>

But there is a twist: if you do not define a relying party corresponding exactly to you entity, ACS will look at the other relying parties, basically chopping off the right hand side of the realm until it finds a matching realm. In this case here, since I haven’t define anything, the root of my namespace is the fallback realm.

If we click on Service Bus, we see the configuration of the Service Identity and at the end:


The permissions are encoded in the rules. A rule is basically an if-then statement: if that user authenticates against this relying party, emit that claim. For Service Bus, the only interesting claim type is net.windows.servicebus.action:


So here you have it. Service bus performs access control with the following steps:

  1. Check ACS for a relying party corresponding to the entity it’s looking at
  2. If that relying party can’t be found, strip url parts until finding one
  3. ACS runs the rules of the relying party with the Service Identity of the consumer
  4. ACS returns a SWT token with claims in it
  5. Service Bus looks for the claim corresponding to the action it requires to do: Listen (receiving messages), Send & Manage.

So… if you want to give access by a specific agent (e.g. web role) to send messages on a topic, you create a Service Identity for the agent and create a relying party corresponding to the topic. You then enter a rule that emits a Send action and you should be all set.

This requires you to store secrets about to (Service) entity in the agents.

 

Hope this very quick overview gives you some ideas. As mentioned at the beginning, I recommend you read this excellent blog post or watch Clemens Vasters’s video.