CheckStat

Cyber Resistance

Information Security Blog

PfSense: Filtering HTTPS Traffic with Squid

August 31st, 2014 – this tutorial was updated for the new release of Diladele Web Safety 3.4 and pfSense 2.1.5. If you are updating from Diladele Web Safety 3.2 or 3.3, be sure to follow the upgrade instructions. Please *NOTE*: after uninstall of Diladele Web Safety 3.2/3.3, remove qlproxy user and group, reboot your pfSense box and add qlproxy user and group again!!!. Please also note, all scripts mentioned in this article can be downloaded here.

This article will tell you how to install and configure Squid proxy capable of filtering encrypted HTTPS connections using Diladele Web Safety ICAP content filtering server running on pfSense Firewall 2.1.5 (amd64). Being able to look into HTTPS contents greatly increases your ability to control what is allowed and accepted within your network while keeping inappropriate contents away.

Why We Need to Filter HTTPS

HTTPS protocol was designed to provide secure means of communications between internet browser and remote web servers. In order to achieve this goal HTTPS protocol encrypts data passing through established connections so that it cannot be decrypted in reasonable amount of time thus preventing anyone from sniffing the contents interchanged over this connection. This protocol was primarily invented to enable safe and secure communication between the user and financial sites or government institutions over the insecure medium such as the Internet.

Recently more and more web sites started to use HTTPS encrypted communications to increase online privacy of users. Google who as first enabled HTTPS for all its searches by default probably initiated this trend. Although there are no doubts that HTTPS encryption is a good thing for safety on the wire we must take into account that it also creates several problems for controlled networks typically found at home or offices. The main problem here is the essence of the HTTPS protocol itself – no one except the browser and the web server is able to see and thus filter transferred data. This may not always be desired. Contents that are usually blocked suddenly become immediately accessible by anyone. As an example imagine a school network where minors can see questionable content by just mistyping a search term in Google. Moreover the law often forces administrators in educational institutions to block access to such content (e.g. CIPA for educational environments) and encrypted access to web sites makes it nearly impossible to fulfill such an obligation.

In order to overcome these limitations it is advised to setup HTTPS filtering of web contents with help of SSL bump feature of Squid proxy server and Diladele Web Safety web filter.

How It Works

In order to filter web requests user’s browser needs to be explicitly directed to use the proxy that is deployed in the same network. It is also possible to set the transparent proxy but we are not going to explain how this is done in this tutorial because steps involved may be quite different from explicit proxy setup.

When a user tries to navigate to a web site, browser sends the request to proxy server, asking it to get the requested page on his behalf. The proxy establishes a new connection to the remote site and returns the response to browser. If normal HTTP is used then proxy is able to see the original contents of the response and filter it. In case of HTTPS the flow of data is a little different. Browser asks the proxy to establish a virtual tunnel between itself and remote server and then sends encrypted data through the proxy. Domain name to which a virtual tunnel is being established is usually known, so proxy is able to block this virtual tunnel when it finds out that domain name belongs to a prohibited category. Unfortunately this is not a complete solution as there are a lot of sites on the Internet which are general in nature (like Google or YouTube) but allow you to easily navigate to something undesired.

To improve the quality of web filtering and get access to contents in encrypted connections, browsers in the network may be setup to trust proxy to act on their behalf for establishing HTTPS connections, filtering them and passing the allowed data to clients while blocking everything that is not allowed. Although this assumption is too strict to be implemented in public networks, it is easily doable in controlled home, educational or corporate environments where administrators act as sole owners of network devices and may force any trusting rules. After established trust browser is able to ask proxy to connect to a remote site in a safe manner with HTTPS, proxy is able to decrypt the traffic, filter it, encrypt it again and pass it to browser. As browser trusts the proxy it continues working with filtered HTTS without any errors or warnings.

Assumptions

I assume you have already installed pfSense 2.1.4 amd64 with two NIC (LAN and WAN). Lan IP address is 192.168.1.1. I also assume you have already done the initial login to the Web UI of pfSense and completed the initial setup wizard and successfully rebooted the pfSense box at least once.

image0011

 

Step 1 – Install Squid built with SSL decryption support.

Login to Web UI and select System -> Packages -> Available Packages. Find and install package squid3-dev.

image0031

 

Wait until squid3-dev package is installed and after installation, reboot your pfSense box.

image0051

 

Login to Web UI, select Status -> Services. The green dot on the right side will indicate Squid service is up and running.

image0071

 

Now adjust your browser configuration to point to pfSense box and try browsing the web.

image0091

 

The response from Squid indicates that we do not have access to it.

image0111

 

To get the access we will add our subnet to the allowed ACL. Select Services -> Proxy server and click the ACL tab. Add 192.168.1.0/24 to the “Allowed subnets” field. Scroll all the way down and click Save. Reload Squid service in Status -> Services. Try browsing web again and see that Squid is now working.

image0131

 

image0151

 

Step 2 – Install Diladele Web Safety for Squid Proxy

SSL Bumping feature alone is not enough to block questionable web content. We also need the filtering server that could be paired with Squid. We will use Diladele Web Safety (called qlproxy) for the filtering and blocking part. It is an ICAP daemon capable of integrating existing Squid proxy and providing rich content filtering functionality out of the box. It may be used to block illegal or potentially malicious file downloads, remove annoying advertisements, prevent access to various categories of the web sites and block resources with explicit content.

We will use the stable release version 3.3 of qlproxy. It was designed specifically with HTTPS filtering in mind and contains rich web administrator console to perform routine tasks right from the browser.

By default, qlproxy comes with four polices preinstalled. Strict policy contains web filter settings put on maximum level and is supposed to protect minors and K12 students from inappropriate contents on the Internet. Relaxed policy blocks only excessive advertisements and was supposed to be used by network administrators, teachers and all those who do not need filtered access to web but would like to evade most ads. Third policy is tailored to white list only browsing and the last group contains less restrictive web filtering settings suitable for normal web browsing without explicitly adult contents shown.

In order to install Diladele Web Safety for Squid Proxy, download package for FreeBSD 8 64-bit Diladele B.V. web site at http://www.quintolabs.com/download_freebsd.php using browser and upload the TBZ binary using WinSCP to you pfSense box.

Diladele Web Safety uses qlproxy user and group to run. Normally it creates those upon installation but for some reason they are not saved during reboots so we must create required users manually. Go to System -> User Manager select Groups Tab and add a new group qlproxy.

image0171

 

image0191

 

Click Save and then select the Users tab to add a new user qlproxy. Do not forget to make in a member of qlproxy group. Enter some arbitrary password.

image0211

 

Again click Save.

image0231

 

In order to install all required libraries and programs we will use 3 scripts. First script installs Python and all needed libraries for the Web UI of Diladele Web Safety, second installs Apache web server that runs the Web UI and third installs Diladele Web Safety itself.

The first script is named 01_django.sh and looks like the following. It should be run from the pfSense command like (e.g. using Putty) as sh 01_django.sh. Please watch out for possible errors!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# setup some configuration variables
ARCH=`uname -m`
# bail out on any error
set -e
# prepare the environment to get some packages from FreeBSD 8
PACKAGESITE=http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/$ARCH/packages-8.3-release/Latest/
export PACKAGESITE
# add python (not sure about py27-ldap)
pkg_add -r python27 py27-sqlite3 py27-pip
# the following command does not work as cc compiler is not present in pfSense, so no LDAP browsing from Web UI will work!
# someone knows how to overcome this? may be install openldap libraries from FreeBSD somehow???
#pkg_add -r py27-ldap2
# add django 1.5
/usr/local/bin/pip install Django==1.5
# and report
echo "01_django.sh script has finished successfully, please run script 02_apache.sh!"

Now we need to install Apache server, copy the following to 02_apache.sh script and run it in the console of pfSense box as sh 02_apache.sh too. This script install all prerequisites for Apache web server and configures it to serve Diladele Web Safety’s Web UI on port 8080 (as standard port 80 is already taken by pfSense Web UI).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# setup some configuration variables
ARCH=`uname -m`
# bail out on any error
set -e
# prepare the environment to get some packages from FreeBSD 8
PACKAGESITE=http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/$ARCH/packages-8.3-release/Latest/
export PACKAGESITE
# add apache
pkg_add -r apache22 ap22-mod_wsgi
# in order to correctly start up apache at boot time init script needs to be renamed
cp /usr/local/etc/rc.d/apache22 /usr/local/etc/rc.d/apache22.sh
# make apache autostart
sed -i '' 's/apache22_enable=\"NO\"/apache22_enable=\"YES\"/' /usr/local/etc/rc.d/apache22.sh
# make apache listen on 8080 port
sed -i '' 's/Listen 80/Listen 8080/' /usr/local/etc/apache22/httpd.conf
# and include the virtual hosts
sed -i '' 's/\#Include etc\/apache22\/extra\/httpd-vhosts.conf/Include etc\/apache22\/extra\/httpd-vhosts.conf/' /usr/local/etc/apache22/httpd.conf
# and report
echo "02_apache.sh script has finished successfully, please run script 03_diladele.sh!"

Finally run the 03_diladele.sh script by typing sh 03_diladele.sh in pfSense console. It will download latest stable build of Diladele Web Safety and adjusts Apache configuration for the Web UI.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
# setup some configuration variables
ARCH=`uname -m`
DDWS_VERSION=3.4.0
DDWS_BUILD=9307
# see if qlproxy group exists
echo "Searching for group qlproxy..."
getent group qlproxy >/dev/null
if [ $? -ne 0 ] ; then
    echo "Group qlproxy is not found, please add it through pfSense Web UI."
    exit 1
else
    echo "Group qlproxy already exists."
fi
# see if qlproxy user exists
echo "Searching for user qlproxy..."
getent passwd qlproxy >/dev/null
if [ $? -ne 0 ] ; then
    echo "User qlproxy is not found, please add it through pfSense Web UI."
    exit 2
else
    echo "User qlproxy already exists."
fi
# how to check user qlproxy is in qlproxy group???
# get latest version of diladele icap server
fetch http://updates.diladele.com/qlproxy/binaries/$DDWS_VERSION.$DDWS_BUILD/$ARCH/release/freebsd8/qlproxy-$DDWS_VERSION-$ARCH.tbz
# and install it
pkg_add qlproxy-$DDWS_VERSION-$ARCH.tbz
# now copy default apache virtual hosts file
if [ -f /usr/local/etc/apache22/extra/httpd-vhosts.conf.default ]; then
    echo "Not saving default vhosts file"
else
    cp /usr/local/etc/apache22/extra/httpd-vhosts.conf /usr/local/etc/apache22/extra/httpd-vhosts.conf.default
    echo "default vhosts file is backed up"
fi
# virtual hosts file needs to contaion only diladele virtual host
echo "NameVirtualHost *:8080" > /usr/local/etc/apache22/extra/httpd-vhosts.conf
echo "Include /usr/local/etc/apache22/extra/qlproxy_virtual_host" >> /usr/local/etc/apache22/extra/httpd-vhosts.conf
# restart apache
/usr/local/etc/rc.d/apache22.sh restart
# and report
echo "03_diladele.sh script has finished successfully!"
echo "Please, reboot your pfSense box and login to Diladele Web Safety's Web UI at http://192.169.1.1:8080!"

Now reboot your pfSense box and login to http://192.169.1.1:8080 using root and [email protected] credentials to finally see the Web UI of Diladele Web Safety.

image0291

 

Step 3 – Integrate Squid Proxy and Diladele Web Safety

To integrate qlproxy and squid, go to Services / Proxy Server, scroll all the way down and in Custom ACLs (Before AUTH) field type:

icap_enable on
icap_preview_enable on
icap_preview_size 4096
icap_persistent_connections on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_header X-Client-Username
icap_service qlproxy1 reqmod_precache bypass=0 icap://127.0.0.1:1344/reqmod
icap_service qlproxy2 respmod_precache bypass=0 icap://127.0.0.1:1344/respmod

acl qlproxy_icap_edomains dstdomain "/opt/qlproxy/etc/squid/icap_exclusions_domains.conf"
acl qlproxy_icap_etypes rep_mime_type "/opt/qlproxy/etc/squid/icap_exclusions_contenttypes.conf"

adaptation_access qlproxy1 deny qlproxy_icap_edomains
adaptation_access qlproxy2 deny qlproxy_icap_edomains
adaptation_access qlproxy2 deny qlproxy_icap_etypes
adaptation_access qlproxy1 allow all
adaptation_access qlproxy2 allow all

Click Save and restart Squid proxy. Try to browse to some adult site and see that HTTP filtering works correctly.

image0311

 

image0331

 

image0351

 

Now we need to enable SSL Filtering to make Diladele Web Safety filter the HTTPS requests too. Create a certification authority in System / Cert Manager.

image0371

 

Click Save.

image0391

 

This certificate will be used to bump the HTTPS connections. Go to Services / Proxy Server, scroll to SSL Man-in-the-Middle filtering and fill the fields as indicated on the following screenshot. Note we are not filling the port settings as we are not doing transparent HTTPS filtering for now. Save and restart Squid service.

image0411

 

If you navigate to google.com you may clearly see the HTTPS connection was NOT bumped. The reason for this is a missing directive “ssl_bump server-first all“ in Squid. Add it before the “icap_enable on” directive added earlier. Save and restart Squid. Now your SSL connection to Google will be bumped.

image043

 

image045

 

To get rid of this warning, we need to install the root CA certificate from pfSense box as trusted in your browser(s). Download the certificates from pfSense using WinSCP for example and import it into trusted certificates as indicated on the following screenshot (instructions for Google Chrome and Internet Explorer may be different).

image047

 

image049

 

image051

 

image053

 

After importing of certificate, reopen your browser, navigate to Google and make sure the certificate warning is away. If you click on the lock icon in the internet address box then it clearly indicates the google.com was signed by proxy’s certificate and not by original certificate by google.

image055

 

If you try to search Google with some adult only terms (e.g. NSFW) Diladele Web Safety blocks the access to explicit contents showing its denied page.

image0331

 

Setup Automatic Updates and Reporting

By default periodic package that runs automatic updates of definition files and report generation scripts is not installed on pfSense. We will use cron functionality to run the scripts manually. Open System / Packages in pfSense Web UI and install the cron package. After that open Services / Cron and add two cron entries:

  1. Run the command to update definition files every 59 minutes as root user – /bin/sh /usr/local/etc/periodic/daily/511.qlproxy_update
  2. Run the command to generate reports of browsing activities every day at 01:00 as root user – /bin/sh /usr/local/etc/periodic/daily/510.qlproxy_report

image0611

 

Resume

Now we have HTTPS web filtering up and running and our network environment become a little safer for those who need protection at most. Next steps would be direct all client browsers to use Squid proxy, correctly setup authentication and authorization to get user specific reports in Diladele Web Safety and optionally setup transparent HTTPS filtering. It is also advisable to enable caching DNS server on pfSense proxy to further increase speed of connections.

If you happen to be gladly using this product be sure to purchase the home, educational or business license that will make the work required to have it up and running in native pfSense format possible.

Source: http://sichent.wordpress.com/2014/02/22/filtering-https-traffic-with-squid-on-pfsense-2-1/

1 Comment

Add a Comment
  1. That’s a sensible answer to a chlleanging question

Leave a Reply

Your email address will not be published. Required fields are marked *

Cyber Resistance © 2017 Frontier Theme