Portico/Z v0.1.0

Credentials + Secrets

Here's how we store credentials and secrets in Z. It's simple by design, and an extension of the behaviour that we see in a default Phoenix project.

  1. All modules and libraries that reference API keys and similar secrets will store those references into config variables eg. Application.get_env(:z, MyModule).secret_api_key. This keeps us from hand-rolling config lookups into a million different modules and consolidates into the config files where that stuff belongs.
  2. In production, all secrets are stored in environment variables and accessed with System.get_env() in the file config/runtime.exs.
  3. We're leveraging Elixir's new "runtime" configuration setup - more on how that works in the Config docs as well as this video by Jose Valim explaining the feature when it was launched.
    1. Within config/runtime.exs, an error should be raised if the environment variable isn't present, because that will cause a lot of bugs. We have created a simple env.(key, message) function to make this dead simple.
  4. In test and development, all secrets are stored in config/test.secret.exs and config/dev.secret.exs and loaded on a per-environment basis. Those keys are hard-coded into those files for dev/test, which is why we don't store those files in version control. Similar to Rails' "master key", you'll have to get this from somebody who has it :) You will receive bright red warnings if you lack these files when doing anything, so it'll be hard to forget.

Here's an example API configuration from config/runtime.exs. Note that a missing HIVE_API_URL doesn't raise an error, since the default is fine for all intents + purposes.

config :hive_service,
  hive_api_token: env.("HIVE_API_TOKEN", "")

Here's how that same example might look in config/dev.secret.exs:

    config :hive_service, [
        hive_api_token: "yer token here"
    ]

In this case, the same code might be used for config/test.secret.exs or we might decide to not use the HIVE API in testing, in which case we wouldn't need to put anything there.

Caveats

Secret Drift for Development

Cleary, this setup could cause a situation where developers' personal config/*secret.exs files get out of sync. So we keep the latest version of this file in our shared password app.

Private Dependency Management

Another caveat to this system is the credential storage for internal EXPLO libraries - hive_monitor, hive_service, and explo_comm for example. Since Gigalixir (our host) doesn't have the capability of sharing SSH public key from the deployment server, we have to put a username/password into our source, so that the deployment server can clone the relevant git repositories. We used the suggested method (look towards the end of the linked docs section) of encoding the username/password as an environment variable in mix.exs. The problem with this approach, as the document points out, is that the password is encoded in mix.lock.

So, to mitigate the damage that finding that username/password combo in mix.lock could cause, we've ensured that none of those libraries are storing credentials in source code any more... Then we made them public on GitHub:

API Tokens

Design

This is good: https://dockyard.com/blog/2020/01/14/tips-for-tokens , and it's generally the advice we should use if we ever want to store user tokens in Z. See also https://hexdocs.pm/phoenix/Phoenix.Token.html which is an excellent built-in resource.

However, for API purposes we have different needs:

  • It's good if the token is associated with the user of the token i.e. can be paired with a provenance.
  • The token needs to be stored as an environment variable for security, not in the database.

So the solution is a simple text-based token language that looks like this:

    # name:token pairs, separated by commas, no whitespace.
    API_TOKENS="portico:qed456,lynton:abc123"

Requests

The ZWeb.Plugs.APIAuthentication plug will handle API authentication by checking the request header for an {"authentication", "bearer #{TOKEN}"} header, where TOKEN is the actual token. The request will be logged against the user contained before the ":" in the matching environment token.

We have a similar ZWeb.Plugs.QueryTokenAuthentication plug to handle endpoints that need to be hit by humans, via URL, with auto-login (ie query token auth).

Research, Notes, and Further Reading

The primary resources for how configuration and environments work in Elixir are in the module docs:

Those are all pretty much required reading ☝ . In addition, this blog post gives a really helpful overview of the above: https://blog.nytsoi.net/2020/05/05/elixir-time-for-configuration .

This post is really helpful, but outdated since newer versions of Elixir have different configuration settings: https://blog.plataformatec.com.br/2016/05/how-to-config-environment-variables-with-elixir-and-exrm/

Looking at solutions like https://github.com/findmypast/vaultex , it strikes me that we'd still have to store some sort of configuration variable (authentication method) in an environment variable OR we'd have to deal with managing SSH credentials or something. This approach seems like the worst of both worlds at our scale.

I did a branch where I implemented this: https://github.com/kieraneglin/encrypted_secrets_ex , but I didn't like that it keeps secrets in memory while the app is running. I also read through the documentation for https://github.com/Nebo15/confex which solves a lot of the problems of using System.get_env() AND/OR Application.get_env() with fallbacks, but is overkill for our needs.