Now that containers have become widely accepted in many Enterprise organizations, additional Enterprise security and operational problems arise, particularly when running Docker containers on premise. One of these issues is Internet access through a proxy. C2 Labs has worked with our clients to build out full Continuous Integration (CI)/Continuous Deployment (CD) pipelines, and proxy access has been a challenge at every step of the way. Particularly when dealing with Linux servers/containers in an enterprise environment, there are gaps that many organizations do not adequately plan for or address. The following are a few lessons we have learned and how to successfully overcome these issues.
What is a Proxy and an SSL Intercept Proxy?
First, let’s make sure we understand WHAT a proxy and an SSL intercept proxy are doing. When using a proxy, all your internet traffic is going through the proxy server. This allows organizations to filter out sites based on prohibited categories or other means. This approach helps protect an organization against threats from known bad sites, as well as legal issues from sites categorized as adult, gambling, illegal, and others. A proxy also examines the internet traffic going through it to make sure viruses, malware, and other items cannot easily pass.
In recent years, nearly all sites on the internet are now using HTTPS, which means they are encrypting traffic from the client’s browser to the remote servers. This is a good thing, preventing someone looking at the traffic, such as your bank records or other private information. However, this encryption presents a problem for organizations that want to be able to examine this traffic. Thus, the introduction of SSL intercept proxies. They are able to actually decrypt the traffic from the client to the server (basically serving as a “man in the middle”). The exact mechanics of this are beyond the scope of this post and can be found elsewhere. However, the basic mechanics require the organization to issue a certificate to the proxy that all the clients TRUST. Most organizations distribute this certificate through Active Directory to Windows servers/clients. Other mechanisms can be used to distribute the certificate to Linux, Mac, or other devices, but it must be done for those devices to access HTTPS through the proxy. Now that we understand the basics, we can talk about the mechanics of what this means for containers.
Specifying a Proxy for Containers
From a security-perspective, you want to minimize network access to/from containers, particularly access to/from the Internet. However, there are times when your container does need access to the internet to download updates or other items. In order to do this in a Linux container, you will need to set at least one of the following environment variables. All may be required.
HTTP_PROXY – The proxy server for accessing any HTTP websites.
HTTPS_PROXY – The proxy server for accessing any HTTPS websites
NO_PROXY – Sites where the above values should be ignored (normally for internal sites)
For the HTTP_PROXY and HTTPS_PROXY, the format is the following (assuming the proxy is using https and is running on port 1234):
https://username:password@proxyurl.domain.com:1234
These can be set as environment variables within your Dockerfile or within a script that runs as part of your container. Please be aware if you set it as part of a script, it is only available for that script or scripts called from that script. For instance, consider the following example of chaining three scripts and the various effects it has on the proxy:
script1.sh – calls script2.sh and then script3.sh
script2.sh – sets the HTTPS_PROXY
script3.sh – The proxy environment variable is not present since it was not called by script 2 and would need to be set in either script 1 or 3. NOTE: If the proxy was set in script1.sh, it would be available to both script2.sh and script3.sh.
If instead you set it as an environment variable in your Dockerfile or docker-compose file, it will be available the entire time the container is running, but it will also be exposed to anyone who has privileges to examine the container’s environment variables (such as via Docker Universal Control Plane (UCP) in Docker Enterprise Edition (EE)).
Trusting SSL Intercepting Proxies
So, now you have the proxy specified and life is good, right? Well, maybe. If your organization is simply using an HTTP proxy and/or not doing SSL intercept of HTTPS sites, you are likely good to go. However, if your organization is using a proxy using SSL intercept, there is still work to be done. As we discussed above, the organizational certificates must be distributed to all servers in the environment to access the internet through a proxy. So, how do we do this in a container? Actually, it is pretty straightforward once you realize what needs to be done. For this example, I will be using a basic Alpine image (alpine:latest). Other images will have a similar process, but the exact commands may be different.
Obtain Certificates – The first step is to obtain the certificate chain of the proxy server. You will need this within your container. You can reach out to your proxy team or use the following steps:
If you have a Windows computer that has access through the same proxy server as the Linux container, these steps will work
Open your web browser (Google Chrome in this example) and navigate to an HTTPS site, such as https://www.docker.com
Click the lock icon to show the SSL information (screenshot here)
Click the Certificate to view details
Click on Certification Path (screenshot here)
Click on the Root (top CA), and then click View Certificate (screenshot here)
Click Copy to File… (screenshot here)
Choose Base-64 encoded X.509. Then save the file
After you save, rename the file from a .cer to a .pem extension (for instance, ca.pem)
Repeat this for any other root or intermediate certificates in the chain (in this example Root CA and Proxy-SSL-Scanning-CA)
Copy files into the container build directory – Now that you have obtained the certificates that your container needs to trust, you need to prepare them to be copied into the container.
I find this easiest if within your build directory, you create a `certs` folder. This helps with the clutter. Additionally, it allows you to add additional certificates without having to modify your Dockerfile
Copy all the certificates from above into this folder (All should be in *.pem format, as Base-64 encoded certificates)
Install the CA Certificates module within the container and update the certificates in the container – Now that the files are within your working directory, you need to update your Dockerfile to use them. Here is my basic Dockerfile:
Let's examine the key pieces of this Dockerfile, as it relates to this process:
Lines 6-8 are adding the following packages: `bash`, `coreutils`, `curl`, and `ca-certificates`. The key one here for certificates is `ca-certificates`. That is the package that allows you to add and update the root certificates of the container
Line 10 copies all the *.pem files from the certs directory (created and populated in step 3 above) into the correct directory in the container
Line 11 updates the certificates within the container to add your certs to the certificates store
Line 13 can be whatever you need to run the program in your container, but at this point, your certs are trusted by the container and ready to be used/tested
Build the updated image – Now that you have your Dockerfile updated, you are ready to build it. If you are building this container on a server that needs to access the internet through a proxy, you will need to pass the proxy info as a build argument. The good thing is that APK update uses only HTTP, so you do not have to worry about your runner or build machine needing to trust SSL. NOTE: proxycontainer:latest is the container image name and can be whatever you want for your container (this is all on one line, ensure you scroll to see it all):
**NOTE: You may see the following warning. This can safely be ignored: WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
Run the container – Now that your proxy container is built, run it, passing HTTP_PROXY and HTTPS_PROXY as environment variables:
Test it out – The container is now running, and you are in the container (NOTE: the -it flag creates an interactive shell within the container; similar to SSH into a remote machine). Run a simple `curl` command to test it:
If this works properly, you should get the following along with an HTML dump of the webpage: HTTP/1.1 200 Connection established
If this does NOT work, you will see something similar to the following and you should retrace the steps above to correct:
Congratulations! You now have a container working through an SSL intercept proxy with your organization’s certificates. You can easily extend this to add other certificates that your container may need to trust.
At C2 Labs, we love solving challenging, real-world problems that are faced by enterprise clients. We serve our clients as a Digital Transformation partner, ensuring no-fail projects are successful from inception to operations and maintenance handoff. We would love to talk to you more about the challenges your organization is facing and how we can help you fundamentally transform IT to move at the speed of business and Take Back Control. Please contact us to learn more.