PyConfig: Our configparser alternative
Configuration comes in all shapes and sizes. Sometimes configuration parameters are too technical for ordinary users but interesting for very advanced users or even for programmers during the development of the app. Other times configuration parameters are secrets (such as passwords) that you do not want to type-in all the time. In general, configuration parameters are things that might differ from user to user but rarely change after they are chosen.
Python has a standard configuration library called “configparser”, which is very convenient and powerful, but does not address many common problems. So we set out to write our own library to make it easier to follow the best practices in app configuration. We will present a few of these common issues here and explain how PyConfig solves them.
Disclaimer: There are many interesting packages for app configuration out there and we value every contribution to the open-source community. Some people might prefer other packages to ours depending on their programming style, their needs and mindset. Of course, PyConfig is our baby and we might be biased, but we will try to stay as objective as possible.
Best practices by default
If you really know what you are doing, if you are disciplined and careful, you can do a good job just using configparser. But let your guard down for a minute and there are several mistakes anyone could make in this area. PyConfig is meant to make the “right way” the “only way”. Where that is not possible, we try to make the “right way” at least the “natural way”. Let us start the break down!
Verifying the configuration early
The apps on your phone usually have a menu screen dedicated to changing “settings”. However, in apps made by programmers for other programmers or for data-analysts or as internal tools within a company, we often do not invest the time to make such a screen, and in fact these apps often don’t have any screens at all but instead just run on a text-based terminal. Therefore, configuration files are just edited by the users directly as text documents, and as developers, we have to be ready for anything!
In our simple example with the timer app, for example, the “vibrate” parameter should be either “yes” or “no”. But since users are just writing their own configuration files, nothing is stopping them from typing the word “maybe” in there!
In case the configuration is invalid, you always want to know about it as soon as possible. If you’re deploying an important, long-running web service and there’s a password missing in the configuration file, it’s usually okay to fail immediately, with a helpful and clear error message saying which password is missing, and then fix the configuration and launch the service again, within minutes. The danger is when the service runs despite having an invalid configuration. It could run for days without needing that missing password and no one would notice, until it finally needs the password for the first time and crashes horribly. True to Murphy's Law, this always happens in the middle of the night or when everyone who could solve the problem is on vacation. The next morning you wake up to an inbox full of angry e-mails from your clients, and a few from your boss.
If you use our “timer” above to bake a cake and the “sound file” parameter is missing, the app will first wait the 25 minutes until your cake is ready and only then try to access the missing parameter and crash. You’ll instead be notified much later by the smoke detector. At Next Kraftwerke, on the other hand, this kind of undetected crash in a long-running service could make us miss a lot of important data, prevent us from playing our role in stabilizing the electricity grid, or just cost us a lot of money on the trading floor.
PyConfig avoids this scenario by automatically validating the app’s configuration as soon as it is loaded. All of it. It will check that every parameter has a value (or at least a default value) and that every value is of the right “data type” (that is, a number, a date, text, a URL, yes/no etc.). Additionally, developers can define more complex checks to be performed automatically. If there is a problem, it will be noticed as soon as the app starts up.
Keeping secrets with PyConfig
Another common mistake is unintentionally leaking sensitive configuration parameters (such as passwords). During development, it is common to include configuration parameters in error messages, so we can better understand the context of an error. One developer logs an innocent error message that includes the app’s configuration, another developer separately adds a new configuration parameter that’s supposed to be secret... and suddenly you have a serious security issue!
PyConfig offers a custom data type called “SecretString” made especially for this situation. Every sensitive parameter should be marked with this special type. When you print the configuration in an error message or any other log, PyConfig automatically masks secrets as “*****” instead. It not only saves you the trouble of checking whether there are any secrets that need masking, but also ensures that new secrets added later will also be masked appropriately.
The library will also forbid you from giving secret parameters a default value (a trick that is unfortunately still common for convenience during development). It just will not run until you remove that “future embarrassment” from your code.