Thanks to the Author.
This is an excellent guide on how to do adblock using Ubiquiti USG and backup config on controller.
Copied this from source for future references.
There are lots of good ways posted on how to use the hosts file, or pi-hole, to keep ads and unwanted content off of your network via DNS blacklisting / blocking.
One very good approach was posted here at:
https://community.ui.com/questions/b8b05010-ccd1-488d-ade4-37782a7e376b
I started using it and liked it immediately as it did exactly what I wanted (ad blocking via DNS blacklisting) and it exists on the USG itself which is something I wanted.
Along the way I made some changes to the script and am continuing to make changes based on the feedback of others.
This post is basically my branch of that script where I will keep the script source and change notes updated as it moves forward.
One of the main differences in this script vs others is that I do not use /etc/hosts to power the blacklist, I use files in /etc/dnsmasq.d. I went this route to keep the hosts file small, containing only the hosts that are added via dhcp and so that I can exclude this blacklist from being used by other dnsmasq instances that I run on my USG and they still have access to the hosts file itself. Also by using the files in /etc/dnsmasq.d, the entries are treated as domains, so blocking of entire domains can be accomplished without having to specify individual hosts. This is useful if a given domain utilizes multiple subdomains to serve their content.
You can have the script updating the blacklist nightly, pulling in info from 12 sources including the pi-hole lists (thank you @Smat ) and the shallalist.de list. Currently it is blacklisting 129441 hosts / domains (this number will be reduced in your log due to the automatic removal of subdomains of existing parent domains) and can email you a summary if you like in addition to the running count it can log for historical tracking.
Note - This script in installed on the USG in /config/scripts. The /config/scripts directory is carried over between provisions / firmware upgrades so you will not need to re-install this script each time you update your firmware.
Note to note - While the script will carry over from firmware upgrades, the dnsmasq configuration files the script generates will not. Be sure to manually run the script after firmware upgrades to re-configure dnsmasq.
Note to ER-X users - This script will run on your hardware. However the ER-X is using busybox to provide 'grep' functionality. This script requires the functionality of the full grep package. Here are the steps to install full grep on your system. (tested on ERX v1.10.0)
A huge thanks to @bubiquit for working with me on the development/testing of this.
ER-X only info- skip this entire box if you are a USG user. - first, ensure you are using busybox grep (note the busybox reference below) root@ubnt:~# grep --version grep: unrecognized option '--version' BusyBox v1.19.0 (2018-01-25 08:48:03 UTC) multi-call binary. - follow steps 1,2,3 in this article: https://help.ubnt.com/hc/en-us/articles/205202560-EdgeRouter-Add-other-Debian-packages-to-EdgeOS - stop after step 3, do not do step 4, it will fail due to the existing links to the busybox binary. - download the grep package: sudo apt-get download grep - force the install of grep using this command, it will overwrite the links to the busybox grep. (use the same filename that just downloaded, below it is grep_2.12-2_mips.deb) sudo dpkg -i --force-all grep_2.12-2_mips.deb - verify that grep is now being provided by the native grep package. root@ubnt:/bin# grep --version grep (GNU grep) 2.12 Copyright (C) 2012 Free Software Foundation, Inc. The grep package you just installed will not survive firmware upgrades on your system, however you can do some scripting if you want it to automatically re-install on upgrades: https://help.ubnt.com/hc/en-us/articles/204961814-EdgeMAX-Are-my-changes-lost-when-I-upgrade-the-firmware-image-
(if you are upgrading, start here)
Due to the character limit of the posts, the full changelog file is attached. The two most recent versions will be noted here.
V8.6 Changes:
- Maintenance release
- There was a missing blank line after the smtp header in the email being sent (if enabled) which was causing the email layout to be garbled for some users. Thank you to @msavazzi for pointing it out and to @kkb18 for helping diagnose the cause and test the fix.
- The URL sources in the .conf file can now be a zip file. The file will be downloaded, unzipped and processed. This is to support the previously existing malware-domains source which changed its data to a zip file.
- URLs for malwaredomains.com and hosts-file.net were updated - existing .conf file entries will be updated automatically.
Here is how to install / upgrade / configure:
- ssh into your USG
- If you are replacing an existing version that did not utilize the .conf file (pre V3), be sure to make note of any configuration changes you made in the existing getBlacklistHosts.sh file before you replace it.
- If you made any configuration changes to the script itself, be sure to either make note of the changes or make a backup copy of your exising script.
- New elements in the .conf file that V4 introduces will be added to your existing .conf file if you have one from a previous version. Your existing personalization of your .conf file will be preserved.
- change to root, cd to /config/scripts and get the file: (note we are saving as just.sh, not .txt. I add the .txt to post as some antivirus programs block the download of .sh if it is downloaded to a PC via the browser)
sudo -i cd /config/scripts curl 'https://raw.githubusercontent.com/unifiMynet/dnsmasqAdBlock/master/getBlacklistHosts.V8.6.zip' > getBlacklistHosts.zip
- unzip the file (you will be prompted to overwrite your old script if it exists)
unzip getBlacklistHosts.zip
- remove the zip file
rm getBlacklistHosts.zip
- make the file executable via chmod +x:
chmod +x getBlacklistHosts.sh
- Manually run the script to generate the getBlacklistHosts.conf file.
- This will import and delete the getBlacklistHosts.currentcount and getBlacklistHosts.oldcount files.
./getBlacklistHosts.sh
- The script will alert you that it has created the getBlacklistHosts.conf file, and not updated the blacklist itself.
- If you are upgrading from a version that utilized the .conf file and the new version introduced new elements to the existing.conf file, the script will alert you that it has updated the getBlacklistHosts.conf file, and not updated the blacklist itself.
- (if desired) update the getBlacklistHosts.conf to personalize your configuration. If you want to use the default settings (no emails, no history) you do not need to exit the file.
- Once you have updated the getBlacklistHosts.conf file (if desired), run the script again to perform the inital update.
./getBlacklistHosts.sh
- The script will run, updating your blacklist and display its progress.
- At this point your DNS is using the blacklist.
- I've added a test domain to the blacklist so you can test. If you do a nslookup from a client that is using the USG as its DNS server (windows example below), looking up testMyLocalUSGDns.com should return 0.0.0.0
C:\Users\me>nslookup Default Server: MainRouter Address: 192.168.0.1 > testMyLocalUSGDns.com Server: MainRouter Address: 192.168.0.1 Name: testMyLocalUSGDns.com Address: 0.0.0.0
- if you want to schedule the script to run nightly to update your blacklist (in this case 4:30 AM nightly) add this to your config.gateway.json
- This can be directly used as a config.gateway.json file with no modification if you are not already using a config.gateway.json
- Note - When a system is provisioned with a json file that includes a task referencing an executable file, the system double checks that the executable file both exists and is executable. If the file does not exist or is not executable, the provisioning will fail. This is something to keep in mind if you remove the script but do not remove the task from your json file, or if you attempt to provision a system that does not have the script in place but has the task included in the json file.
{
"system": {
"task-scheduler":{
"task":{
"hostblacklist":{"executable":{"path":"/config/scripts/getBlacklistHosts.sh"},"crontab-spec":"30 4 * * *"}
}
}
}
}
Additional information regarding the Unifi config.gateway.json file:
- The Unifi USG uses the config.gateway.json file to allow customizations that will persist across provisions / firmware updates.
- The file is placed on the Unifi controller, not the USG itself, in the folder <unifi_base>/data/sites/<site_name> - the default <site_name> is default.
- This article will help you determine where your <unifi_base> directory is depending on your controller platform: https://help.ubnt.com/hc/en-us/articles/115004872967
- For example, the path to the folder where the config.gateway.json file exists on the Cloudkey is /usr/lib/unifi/data/sites/default
- Once you have your config.gateway.json file on the controller, when you provision your USG via the controller, the controller takes the GUI configuration, combines it with the config.gateway.json and sends it to the USG.
- Further information on the config.gateway.json file: https://help.ubnt.com/hc/en-us/articles/215458888-UniFi-How-to-further-customize-USG-configuration-with-config-gateway-json
- The controller will not read the file in automatically once it is created. It will either provision the USG if a change it made in the GUI that requires a provision, or you can force a provision immediately via the Unifi controller by performing: controller - devices (on left) - click on USG - 'config' tab on the properties pane that opens - 'manage device' + - 'provision' button.
- Once a provision is forced, the 'status' of the USG in the controller will change to provisioning.
- Once provisioning is complete you can check to make sure the task is set in the USG by SSH'in into it, entering 'configure' mode and doing a 'show system task-scheduler'. Note that the 'task hostblacklist' exists. Enter an 'exit' to leave configure mode:
@MainRouter:~$ configure[edit]
@MainRouter# show system task-scheduler
task hostblacklist { crontab-spec "30 4 * * *" executable { path /config/scripts/getBlacklistHosts.sh } }[edit]
@MainRouter# exit
exit
@MainRouter:~$
At this point you are done, it will be running and will update itself nightly.
Adding a whitelist:
If you find there is/are some domains/hosts that you want to exclude from the blacklist:
- Create the file 'dnswhitelist' (exactly this, no extension) in your USG in the folder /config/scripts/
- include one host/domain per line that will be excluded from the blacklist
- include a * prefix to whitelist the entire domain (See below for examples).
- If the file does not exist it will not be used.
Examples below show the whitelist results on these blacklist entries:
somedomain.com
api.somedomain.com
cdn.somedomain.com
events.somedomain.com
no dnswhitelist entry:
entire somedomain.com is blocked due to 'somedomain.com' being included in the blacklist data
dnswhitelist entry: *somedomain.com (note no dot between * and domain name)
resulting blacklist entries:none - entire domain whitelisted
dnswhitelist entry: somedomain.com
resulting blacklist entries:
address=/api.somedomain.com/0.0.0.0
address=/cdn.somedomain.com/0.0.0.0
address=/events.somedomain.com/0.0.0.0
dnswhitelist entry: api.somedomain.com - this one subdomain will be whitelisted
resulting blacklist entries:
address=/somedomain.com/0.0.0.0
address=/cdn.somedomain.com/0.0.0.0
address=/events.somedomain.com/0.0.0.0
Adding a user-defined blacklist:
If you find there is/are some domains/hosts that you want to include in your blacklist:
- Create the file 'dnsblacklist' (exactly this, no extension) in your USG in the folder /config/scripts/
- include one host/domain per line that will be included in your blacklist based on a partial match.
- If a domain is listed the entire domain and all subdomains will be blocked.
- If a subdomain or specific host is listed, only that will be blocked.
- This does not use the * to denote a domain as the whitelist does.
- If the file does not exist it will not be used.
Examples:somedomain.com - will blacklist all DNS entries including subdomains, so it will blacklist:
somedomain.com, sub1.somedomain.com, sub2.somedomain.com, etc.
sub1.somedomain.com would only blacklist sub1.somedomain.com, keeping:
somedomain.com and sub2.somedomain.com as non-blacklisted DNS entries.
To add a user-defined source URL:
- Please see the getBlacklistHosts.conf file for information on how to add your own download sources.
To retain debugging files:
- Please see the getBlacklistHosts.conf file for information on how to enable debugging files.
To have the script email you a summary when it runs:
- configure sSMTP on your USG to send mail: https://community.ui.com/questions/4166e72d-d210-4010-9a03-23748dcdcc8d
Edit your 'getBlackListHosts.conf' file - change:
- 'sendEmails=false' to 'sendEmails=true'
- set your 'to' email address on the line containing: emailtoaddr="youremail@somedomain.com"
- set your 'from' email address on the line containing: emailfromaddr="youremail@somedomain.com"
To have the script run another file when it completes:
- Create an executible file in the same directory as the script named 'getBlacklistPostRun.sh'. This file will be run on completion.
- To set a different file name, edit the getBlacklistHosts.conf file, 'postRun' section.
To remove this script from the USG entirely:
Remove the task from the task-scheduler. If you do remove the script, be sure to remove the task from your json file. (See the note above regarding provisioning a system with a task that references a file that does not exist) You can either remove it from your config.gateway.json and reprovision, or you can remove it by hand if you do not have it in your config.gateway.json:
sudo -i configure delete system task-scheduler task hostblacklist commit save exit
remove the blacklists from the dnsmasq configuration and restart dnsmasq:
sudo -i rm /etc/dnsmasq.d/blackhost* /etc/init.d/dnsmasq force-reload exit
clean up some of the remaining log files, the whitelist (if you used one), and delete the script itself (some of these files may not exist in your environment)
sudo -i rm /var/log/getBlackListHosts.log rm /config/scripts/dnswhitelist rm /config/scripts/dnsblacklist rm /config/scripts/getBlacklistHosts.oldcount rm /config/scripts/getBlacklistHosts.currentcount rm /config/scripts/BlacklistHistoryCount.txt rm /config/scripts/getBlacklistHosts.sh rm /config/scripts/getBlacklistHosts.conf rm /config/scripts/nonFilteredHosts rm /config/scripts/filteredHosts rm /config/scripts/finalHosts rm /tmp/tmp.gzippedhosts.tar.gz rm /etc/dnsmasq.d/getBlacklistOptions.conf rm /config/scripts/getBlacklistPostRun.sh exit
No comments:
Post a Comment