Vault: self-hosted secrets system, for personal password DB and more
Posted: Fri Aug 04, 2023 4:39 pm
I want to spin up Hashicorp Vault as a backing store for Minio keys. I'll first focus on a simpler use case of personal password storage to kick the tires.
First, find the docker image and start it up with appropriate settings.
https://hub.docker.com/r/hashicorp/vault
I found this tutorial and the only differences from default nginx + certbot config are:
I'm trying it out with the defaults and I should look into what those settings are.
I navigated to the location and was presented with a question about "Key Shares" and found a tutorial, clicked into "Web UI:"
https://developer.hashicorp.com/vault/t ... started-ui.
I had a good idea of what those settings mean, but not what I should set them to. Literally just look up anything you aren't 100% sure about. I intuitively knew what the key split configuration was, but I searched for it anyway. It's always an investment to do that. The mental web gets so dense. Forget your fear of wasting time.
This is cool because the server will never be unable to decrypt the content from a disk dump. N of M keys are required to start the server, and the memory is pinned meaning it will never be put into the page cache.
Success logging in with root user and creating a harlanji user and group (do in that order, add user to group).
I see metadata and policies, seems like a hybrid of LDAP and IAM. I noticed menu items for OIDC (I guess OpenID connect, sweet; indeed, I can create apps within), and Multi-factor Auth providers. Is this the hidden self-hosted auth method? All I am looking to do is create a password store. I guess there is an API that apps can use to manager their users transparently.
In the getting started tutorial the KV storage engine is mentioned.
I figured out how to enable the KV storage engine and added some secrets within a namespace, the same way I did for Minio/S3.
Next I need to figure out access controls for the user/entity and groups. I have 2+ of my own brands that I can use as a mental model when thinking through use cases/personas.
At first glance in the policy doc, it looks like what I expect. Seems I need to create an auth method. Username & password looks like a good first choice, and JWT, LDAP, and TLS certificates are things I've used as well and can mentally map out some expansion paths to my existing apps.
I am unsure what token type is, so I found the doc for the Userpass auth method. No mention. What's there makes sense though, I'll need to create a policy to attach to each group and optionally user like with Minio. Clicking into the API doc is often a good choice when the user docs don't hit on a setting. The UI doesn't expose "default" so I have to make a choice... it says "service" is the default choice. Again I have some intuition about this but still want to resolve it before I move forward: https://developer.hashicorp.com/vault/t ... tch-tokens. I'll go with service tokens to begin with; default-service it is.
In creating a user it appears to have no connection to the entity I created. We'll want to create the policy before we create tokens since we need to attach it to generated tokens. I found a default policy, good way to learn about what the tool does out of the box. There is a little default policy that is interesting and dynamic:
I guess I can use this a template to grant access to a namespace within the KV store exactly like I did with Minio buckets, where the identity name is the prefix.
I'll want a policy that looks something like this, called harlanji-group-kv:
Since userpass doesn't appear to be automatically mapped to the entity we created, we'll need to do that. I found the aliases documentation and created a userpass alias for the harlanji entity, and attached the policy above to the harlanji group which the entity is in. I think I can test by logging into the web UI with the harlanji user I just created and seeing if the gears all turn as expected; can I view the secrets I created? No dice out of the gate, but it does appear that this 'cubbyhole' per-user store works. So we can use that policy and engine configuration as a guide.
I like using groups so that I can create sub-users with different policies, but it's possible we can just do this at the token level. I figured out that the policy was working at some level because when I removed it I wasn't even able to access the main screen for the kv store within the web UI. After some reading and pattern matching, I figured out I had two mistakes. First, I don't need to use a policy template in this simple case. Second, the policy needs to include data/ in the namespace after kv/. The revised policy that works:
Once I made this the policy I was able to access the secrets I created earlier. I still can't list secrets for navigation like I can with the root user. I feel like it's a success at this point since I can just record the secret path in plain text and access it after I log in. I don't think there's an inherent risk in stopping here and moving forward with integration, calling this software worth using.
I could create a template for the harlanji group that I created and use its name, but that's kinda redundant because if I rename the group I'll have th move the data anyway.
Hashicorp Vault goes way above and beyond what I expected in a good way. One of the initial motivations for taking the step and looking into it is as a key store for Minio to allow encrypted storage. That'll be a future post since I've proven it out with a simpler use case. That can build on this exploration and focus on integrating Minio KMS specifically.
It seems like it will scale with an organization, from simple password manager that required business administrators to start up to login system for end user apps. It's like the hidden sauce to "building your own auth" for those of us bullish on self-hosting for off-grid/intranet scenarios.
Update: listing secrets
I found a way to list secrets via the CLI but still no dice via the web UI. I tried logging out and back in to refresh any cached policy or similar.
I added another snippet to the harlanji-group-kv policy based on the documentation linked above, making the full policy:
Update: Listing via web UI
I'm able to list secrets from the web UI if I navigate to the harlanji/ namespace specifically. Might be a small policy tweak to make this possible to list the root namespace and only show allowed keys, but more likely might require a tweak to the UI to check for exact matches in the data namespace.
First, find the docker image and start it up with appropriate settings.
https://hub.docker.com/r/hashicorp/vault
Code: Select all
docker volume create vault-data_vault.harlanji.com
docker run -d --name vault_vault.harlanji.com \
--cap-add=IPC_LOCK \
-v vault-data_vault.harlanji.com:/vault/file \
-e 'VAULT_LOCAL_CONFIG={"storage": {"file": {"path": "/vault/file"}}, "listener": [{"tcp": { "address": "0.0.0.0:8200", "tls_disable": true}}],
"default_lease_ttl": "168h", "max_lease_ttl": "720h", "ui": true}' \
-p 127.0.0.1:8200:8200 \
hashicorp/vault:1.14.1 \
server
Code: Select all
ssl_session_cache shared:le_nginx_SSL:10m; # vs. :SS:
ssl_session_tickets off; # vs. on
I navigated to the location and was presented with a question about "Key Shares" and found a tutorial, clicked into "Web UI:"
https://developer.hashicorp.com/vault/t ... started-ui.
I had a good idea of what those settings mean, but not what I should set them to. Literally just look up anything you aren't 100% sure about. I intuitively knew what the key split configuration was, but I searched for it anyway. It's always an investment to do that. The mental web gets so dense. Forget your fear of wasting time.
This is cool because the server will never be unable to decrypt the content from a disk dump. N of M keys are required to start the server, and the memory is pinned meaning it will never be put into the page cache.
Success logging in with root user and creating a harlanji user and group (do in that order, add user to group).
I see metadata and policies, seems like a hybrid of LDAP and IAM. I noticed menu items for OIDC (I guess OpenID connect, sweet; indeed, I can create apps within), and Multi-factor Auth providers. Is this the hidden self-hosted auth method? All I am looking to do is create a password store. I guess there is an API that apps can use to manager their users transparently.
In the getting started tutorial the KV storage engine is mentioned.
I figured out how to enable the KV storage engine and added some secrets within a namespace, the same way I did for Minio/S3.
Next I need to figure out access controls for the user/entity and groups. I have 2+ of my own brands that I can use as a mental model when thinking through use cases/personas.
At first glance in the policy doc, it looks like what I expect. Seems I need to create an auth method. Username & password looks like a good first choice, and JWT, LDAP, and TLS certificates are things I've used as well and can mentally map out some expansion paths to my existing apps.
I am unsure what token type is, so I found the doc for the Userpass auth method. No mention. What's there makes sense though, I'll need to create a policy to attach to each group and optionally user like with Minio. Clicking into the API doc is often a good choice when the user docs don't hit on a setting. The UI doesn't expose "default" so I have to make a choice... it says "service" is the default choice. Again I have some intuition about this but still want to resolve it before I move forward: https://developer.hashicorp.com/vault/t ... tch-tokens. I'll go with service tokens to begin with; default-service it is.
In creating a user it appears to have no connection to the entity I created. We'll want to create the policy before we create tokens since we need to attach it to generated tokens. I found a default policy, good way to learn about what the tool does out of the box. There is a little default policy that is interesting and dynamic:
Code: Select all
# Allow a token to look up its own entity by id or name
path "identity/entity/id/{{identity.entity.id}}" {
capabilities = ["read"]
}
path "identity/entity/name/{{identity.entity.name}}" {
capabilities = ["read"]
}
I'll want a policy that looks something like this, called harlanji-group-kv:
Code: Select all
path "kv/{{ identity.groups.names.harlanji.id }}/*" {
capabilities = ["create", "read", "update", "patch", "delete", "list"]
}
I like using groups so that I can create sub-users with different policies, but it's possible we can just do this at the token level. I figured out that the policy was working at some level because when I removed it I wasn't even able to access the main screen for the kv store within the web UI. After some reading and pattern matching, I figured out I had two mistakes. First, I don't need to use a policy template in this simple case. Second, the policy needs to include data/ in the namespace after kv/. The revised policy that works:
Code: Select all
path "kv/data/harlanji/*" {
capabilities = ["create", "read", "update", "patch", "delete", "list"]
}
I could create a template for the harlanji group that I created and use its name, but that's kinda redundant because if I rename the group I'll have th move the data anyway.
Hashicorp Vault goes way above and beyond what I expected in a good way. One of the initial motivations for taking the step and looking into it is as a key store for Minio to allow encrypted storage. That'll be a future post since I've proven it out with a simpler use case. That can build on this exploration and focus on integrating Minio KMS specifically.
It seems like it will scale with an organization, from simple password manager that required business administrators to start up to login system for end user apps. It's like the hidden sauce to "building your own auth" for those of us bullish on self-hosting for off-grid/intranet scenarios.
Update: listing secrets
I found a way to list secrets via the CLI but still no dice via the web UI. I tried logging out and back in to refresh any cached policy or similar.
I added another snippet to the harlanji-group-kv policy based on the documentation linked above, making the full policy:
Code: Select all
path "kv/data/harlanji/*" {
capabilities = ["create", "read", "update", "patch", "delete", "list"]
}
path "kv/metadata/harlanji/*" {
capabilities = ["list"]
}
Update: Listing via web UI
I'm able to list secrets from the web UI if I navigate to the harlanji/ namespace specifically. Might be a small policy tweak to make this possible to list the root namespace and only show allowed keys, but more likely might require a tweak to the UI to check for exact matches in the data namespace.