Why you should not load secrets from .env vars ?

Introduction

Secret handling is one of those tasks where you have to be very careful.

If you have been developing application there is a good chance that you came access the method of storing secrets in a .env file. Something like this

$ cat .env

USERNAME=XXXXX
PASSWORD=XXXXX
MONGO_URI=XXXX

These files are called a dotenv file and generally used while passing secrets to your application.

Why are people storing secrets in file ?

According to Twelve-Factor app which mentions that config should be should be separated from the application. This is completely understandable. Its good when configuration is separated from your application. This makes the application highly tunable and manageable. This was the story but later people started passing secrets along with the file and along with this “security by obsecurity” mentality took over.

Myth and reality around .env

There are several myths about .env some of which are .env is the secure way of passing secrets to your program as it is not accidentally commited to git. Since .env is not listed at part of ls command, it is secure as it is hidden.

But in reality .env is not different to any other files. Git treats all your files including .env same.

If you forget to put it as part of your .gitignore file then it will be commited and pushed to the remote repository just like any other files in your repository.

Another thing is, when values are loaded from .env into your running program which is a process, all the values are set as environment variables and can be accessed using command similar to os.Getenv("KEY") in Go or process.env.S3_BUCKET in node.

Passing secrets as environment variables has been there for a long time, but there will always be a risk of process leaking environment variables to its child process or via Remote Code Execution ( RCE ). As we all know that no application is fully secure and there is are chance someone quite competent might be able to run arbitary code via your application and pull in the environment variables.

Your application might be secure, but the library with which you spawn another subprocess or child process might be leaking the environment variables.

This sounds like a hoax but it has actually happened. Here are few reads

https://gitlab.com/gitlab-org/gitlab/-/issues/337601

There are various dangers of using these approach: https://www.trendmicro.com/en_us/research/22/h/analyzing-hidden-danger-of-environment-variables-for-keeping-secrets.html

https://towardsdatascience.com/leaking-secrets-in-web-applications-46357831b8ed

I could go on but you get the gist.

Alternative approach

I understand, it’s easy to use dotenv and be done with this, but if we have any secrets in our application, then we are responsible to handle them correctly. Here are few alternative approach that could be used to best manage your secrets

Using external secret storage

The best approach is to use external secret storage or secret manager like Hhashicorp vault or AWS Secret manager. These services are designed to protect your secret. Benefit of using external secret storage it the ease of managing the secrets. You can easily rotate, update, delete secrets without breaking the application. External secrets also provide secrets encryption at rest, and in transit.

Using file such as yaml or json

The above solution is good for corporate world, but in real life its hard to get hand on such services without spending some cash, so another approach to do this better would be to store the secret in a file with tight permission and load them into your application just like any other configuration file.

So you might ask how it is different from storing in .env?

This is same as the reason explained above, with values are loaded from .env into your running program which is a process, all the values are set as environment variables. There will always be a risk of process leaking environment variables to its child process.

But if you are using configuration using file such as yaml or json, then the secrets are never stored in environment variables. Now those variables will only be accessible only through some interface in your application and not by all the methods or functions. This minimizes the risk of your secrets being exposed. Except if your server gets pwned where secrets are stored then well ¯\(ツ)/¯.

Note: You still need to add these yaml or json into your .gitignore files

Using platform provided secret management feature

Different providers provide features for secret management i.e docker secrets from docker, k8s secrets from k8s etc. Instead of adding secrets in your repository you could create those secrets in the platform and pull in from there. At least this way, your secrets are not pushed into the repo.

Note: Secrets provided by these are still not secure as you think. They just encode them into base64 so keep these in mind and also careful when using these

Demo

Here is an example how you can pass secrets as config file to your program without loading them as environment variables. I am using Go, but you can use any language

First of all define your yaml configurations

environment: "dev"

server:
  host: "localhost"
  port: 9090
db:
    password: "XXXXXX"

Note the password there. You might be thinking, “oh why is there a password there “?. Don’t worry, we will encrypt the whole file later before we push it into code.

So, let’s continue. Now load this file into the program natively using go or programming language of choice. Here i am usin go and viper to load the configuration variable. Here i am using golang tool called viper to do that for me.

type Configuration struct {
	Environment string

	Server struct {
		Host string
		Port int
	}
	Db struct {
		Password string
	}
}
func LoadConfiguration(configPath, configName, ConfigType string) (*Configuration, error) {
	var config *Configuration
	viper.AddConfigPath(configPath)
	viper.SetConfigName(configName)
	viper.SetConfigType(ConfigType)
	viper.AutomaticEnv()
	viper.SetEnvKeyReplacer(strings.NewReplacer(`.`, `_`))
	err := viper.ReadInConfig()
	if err != nil {
		return nil, fmt.Errorf("could not read the config file: %v", err)
	}
	err = viper.Unmarshal(&config)
	if err != nil {
		return nil, fmt.Errorf("could not unmarshal: %v", err)
	}
	return config, nil
}

Once configuration are loaded into the program, you can use it into your code wherever as necesssary as shown beloe

config, err := config.LoadConfiguration(configPath, configName, ConfigType)
	if err != nil {
		log.Fatalf("could not load configuration file: %v", err)
	}
	fmt.Println(config.Environment)  <==== Accessing configuration

Note: configPath, configName and configType are passed in as an constant to the application.

Here is the link to the git repository containing the working code: https://github.com/pgaijin66/go-config-yaml

Encrypting at rest

Apart from this, our work is not done here. As we discussed the secrets should be encrypted at rest, we will be also encrypting them before pushing them into the repository. Once config file is added into the repository, i like to encrypt the file using ansible vault as shown below but you can do however

ansible-vault --encrypt --vault-id app-prod@prompt config.yaml

New vault password (project):
Confirm new vault password (project):
Encryption successful

Now when we open this file we will see the file being encrypted and not in plain text like this

$ cat config.yaml

$ANSIBLE_VAULT;1.2;AES256;project
31363961366639393931396663313938333338383063623934353463323633636139366339373764
3762623663316163333163366136613336636265613534340a343266313338343864396233373033
31306239613735346565643364353866363739333431663562356464303031383136636337623063
3537303536653139320a356430306233343337343965366630386164643536343263323136356232
65636663333535326434663733346465633530343231306436663339663432363762323836633463
39336237666666623031346163613865356331346434346531303565323263376638376636356531
38353039353064396231666465326236363435383766343632666165323633643734663562623365
30333939333033646333

Now, as part of the CI/CD step, we will pass this file to the desired location and with proper file permission and access control and load it into the application when its needed.

Conclusion

In conclusion, whatever method you use, make sure you do not ignore the ignore files such as dockerignore or gitignore. Make sure you add these files containing secrets into those ignore files. Since .env and yaml files both are stored in plain text if one has access to the server, and are readable by anyone, i would put my best bet on files with tight filesystem permission and tight acl and inject those var into the project as a normal variables rather than environment variable.