Release ipfilter-v4.5

Changes: - Added support for downloading blocklists from I-BlockList as registered user (by supplying username and PIN via configuration items IBL_USER and IBL_PIN). Fixes #8 - Obfuscate I-BlockList username and PIN in console/log output (to prevent leaking sensitive information via logs and/or when posting script output on GitHub)
fonic · Aug 18, 2024 · 84f4c80 · 84f4c80
1 parent e22359f
commit 84f4c80
Show file tree

Hide file tree

Showing 4 changed files with 65 additions and 16 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,13 @@
+## Changelog for v4.5 release
+
+Changes:
+- Added support for downloading blocklists from I-BlockList as registered
+  user (by supplying username and PIN via configuration items `IBL_USER`
+  and `IBL_PIN`). Fixes https://github.com/fonic/ipfilter/issues/8
+- Obfuscate I-BlockList username and PIN in console/log output (to prevent
+  leaking sensitive information via logs and/or when posting script output
+  on GitHub)
+
 ## Changelog for v4.4 release
 
 Changes:
@@ -46,7 +56,7 @@ Changes:
 
 GitHub-only changes (not part of release):
 - Added test suite script
-- Added I-Blocklist scraper script
+- Added I-BlockList scraper script
 - Added hints document for Windows runtime environments
 - Updated `README.md`
 
@@ -89,4 +99,4 @@ Initial release.
 
 ##
 
-_Last updated: 04/26/24_
+_Last updated: 08/18/24_
diff --git a/README.md b/README.md
@@ -1,5 +1,5 @@
 # IP Filter Updater &amp; Generator (ipfilter)
-Generates PeerGuardian (`.p2p`) blocklist from [I-Blocklist](https://www.iblocklist.com/) blocklists and [GeoLite2](https://dev.maxmind.com/geoip) country blocks. Suitable to run as a [system service](#set-up-as-system-service) on always-on devices (e.g. NAS, Raspberry Pi).
+Generates PeerGuardian (`.p2p`) blocklist from [I-BlockList](https://www.iblocklist.com/) blocklists and [GeoLite2](https://dev.maxmind.com/geoip) country blocks. Suitable to run as a [system service](#set-up-as-system-service) on always-on devices (e.g. NAS, Raspberry Pi).
 
 ## Donations
 I'm striving to become a full-time developer of [Free and open-source software (FOSS)](https://en.wikipedia.org/wiki/Free_and_open-source_software). Donations help me achieve that goal and are highly appreciated.
@@ -27,7 +27,7 @@ Refer to the [releases](https://github.com/fonic/ipfilter/releases) section for
 Open `ipfilter.conf` in your favorite text editor and adjust the settings to your liking. Refer to embedded comments for details. Note that before changing any settings, it is recommended to run _ipfilter_ with *default settings* first to make sure it works as expected. Refer to [this section](#configuration-options--defaults) for a listing of all configuration options and current defaults.
 
 ## Subscriptions
-Using the [I-Blocklist](https://www.iblocklist.com/) feature does not require a subscription, as most of the provided lists are free to download. There are a few lists that are only available to subscribers, though. Non-free lists are currently unsupported - please open an [Issue](https://github.com/fonic/ipfilter/issues) if you have a subscription and want to help adding support for these lists.
+Using the [I-BlockList](https://www.iblocklist.com/) feature requires a username and a PIN which can be obtained free of charge after [signing up](https://www.iblocklist.com/register). Note that while most lists are free to download, there are also a few lists that are only available to subscribers. Those subscription-only lists are currently unsupported - please open an [Issue](https://github.com/fonic/ipfilter/issues) if you have a subscription and want to help with adding support.
 
 Using the [GeoLite2](https://dev.maxmind.com/geoip) feature requires an account ID and a license key which can be obtained free of charge after [signing up](https://www.maxmind.com/en/geolite2/signup) (refer to [this blog post](https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/) for details).
 
@@ -117,7 +117,7 @@ Configuration options and current defaults:
 #  IP Filter Updater & Generator (ipfilter)                                    -
 #                                                                              -
 #  Created by Fonic (https://github.com/fonic)                                 -
-#  Date: 04/15/19 - 04/26/24                                                   -
+#  Date: 04/15/19 - 08/18/24                                                   -
 #                                                                              -
 # ------------------------------------------------------------------------------
 
@@ -161,7 +161,24 @@ CURL_OPTS=("--fail" "--location" "--silent" "--show-error" "--retry" "2" "--conn
 # Default: WGET_OPTS=("--no-verbose" "--tries=3" "--timeout=60")
 WGET_OPTS=("--no-verbose" "--tries=3" "--timeout=60")
 
-# List of blocklists to download from I-Blocklist (https://www.iblocklist.com)
+# Username to use to download blocklists from I-BlockList (https://www.iblocklist.com)
+# NOTE:    A valid username is required to use the I-BlockList feature; this is the same username
+#          that you use to sign in to your I-BlockList account (https://www.iblocklist.com/login)
+# Format:  String
+# Example: IBL_USER="user"
+# Default: IBL_USER=""
+IBL_USER=""
+
+# PIN to use to download blocklists from I-BlockList (https://www.iblocklist.com)
+# NOTE:    A valid PIN is required to use the I-BlockList feature; to obtain your PIN, sign in to
+#          your I-BlockList account and click on the 'Member' button (or navigate manually to URL
+#          https://www.iblocklist.com/member)
+# Format:  String
+# Example: IBL_PIN="123456"
+# Default: IBL_PIN=""
+IBL_PIN=""
+
+# List of blocklists to download from I-BlockList (https://www.iblocklist.com)
 # NOTE:    To identify valid ids, inspect hyperlink targets or column 'Update URL' on this web page: https://www.iblocklist.com/lists
 #          e.g. 'level1' -> 'https://www.iblocklist.com/list?list=ydxerpxkpcfqjaybcssw' -> id is 'ydxerpxkpcfqjaybcssw' -> ["level1"]="ydxerpxkpcfqjaybcssw"
 # Format:  Array of name-id-pairs (i.e. string-string-pairs)
@@ -218,4 +235,4 @@ COMP_TYPE="none"
 
 ##
 
-_Last updated: 04/26/24_
+_Last updated: 08/18/24_
diff --git a/ipfilter.conf b/ipfilter.conf
@@ -5,7 +5,7 @@
 #  IP Filter Updater & Generator (ipfilter)                                    -
 #                                                                              -
 #  Created by Fonic (https://github.com/fonic)                                 -
-#  Date: 04/15/19 - 04/26/24                                                   -
+#  Date: 04/15/19 - 08/18/24                                                   -
 #                                                                              -
 # ------------------------------------------------------------------------------
 
@@ -49,7 +49,24 @@ CURL_OPTS=("--fail" "--location" "--silent" "--show-error" "--retry" "2" "--conn
 # Default: WGET_OPTS=("--no-verbose" "--tries=3" "--timeout=60")
 WGET_OPTS=("--no-verbose" "--tries=3" "--timeout=60")
 
-# List of blocklists to download from I-Blocklist (https://www.iblocklist.com)
+# Username to use to download blocklists from I-BlockList (https://www.iblocklist.com)
+# NOTE:    A valid username is required to use the I-BlockList feature; this is the same username
+#          that you use to sign in to your I-BlockList account (https://www.iblocklist.com/login)
+# Format:  String
+# Example: IBL_USER="user"
+# Default: IBL_USER=""
+IBL_USER=""
+
+# PIN to use to download blocklists from I-BlockList (https://www.iblocklist.com)
+# NOTE:    A valid PIN is required to use the I-BlockList feature; to obtain your PIN, sign in to
+#          your I-BlockList account and click on the 'Member' button (or navigate manually to URL
+#          https://www.iblocklist.com/member)
+# Format:  String
+# Example: IBL_PIN="123456"
+# Default: IBL_PIN=""
+IBL_PIN=""
+
+# List of blocklists to download from I-BlockList (https://www.iblocklist.com)
 # NOTE:    To identify valid ids, inspect hyperlink targets or column 'Update URL' on this web page: https://www.iblocklist.com/lists
 #          e.g. 'level1' -> 'https://www.iblocklist.com/list?list=ydxerpxkpcfqjaybcssw' -> id is 'ydxerpxkpcfqjaybcssw' -> ["level1"]="ydxerpxkpcfqjaybcssw"
 # Format:  Array of name-id-pairs (i.e. string-string-pairs)

diff --git a/ipfilter.sh b/ipfilter.sh
@@ -5,7 +5,7 @@
 #  IP Filter Updater & Generator (ipfilter)                                    -
 #                                                                              -
 #  Created by Fonic (https://github.com/fonic)                                 -
-#  Date: 04/15/19 - 04/26/24                                                   -
+#  Date: 04/15/19 - 08/18/24                                                   -
 #                                                                              -
 # ------------------------------------------------------------------------------
 
@@ -47,7 +47,7 @@ esac
 # Windows; same for INSTALL_DST below; macOS has no 'realpath' and there's
 # no simple workaround for that, so just don't support/use it)
 SCRIPT_TITLE="IP Filter Updater & Generator (ipfilter)"
-SCRIPT_VERSION="4.4 (04/26/24)"
+SCRIPT_VERSION="4.5 (08/18/24)"
 if [[ "${PLATFORM}" == "macos" ]]; then
 	SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 else
@@ -85,10 +85,12 @@ CURL_OPTS=("--fail" "--location" "--silent" "--show-error" "--retry" "2" "--conn
 WGET_OPTS=("--no-verbose" "--tries=3" "--timeout=60")
 
 # I-BlockList settings (https://www.iblocklist.com/lists)
-IBL_URL="https://list.iblocklist.com/?list=%s&fileformat=p2p&archiveformat=gz"
+IBL_URL="https://list.iblocklist.com/?list=%s&fileformat=p2p&archiveformat=gz&username=%s&pin=%s"
 IBL_FIN1="iblocklist-%s.p2p.gz"
 IBL_FIN2="iblocklist-%s.p2p"
 IBL_FOUT="iblocklist-merged.p2p"
+IBL_USER=""
+IBL_PIN=""
 declare -A IBL_LISTS=(["level1"]="ydxerpxkpcfqjaybcssw" ["level2"]="gyisgnzbhppbvsphucsw" ["level3"]="uwnukjqktoggdknzrhgh")
 
 # GeoLite2 settings (https://dev.maxmind.com/geoip/geoip2/geolite2)
@@ -547,6 +549,7 @@ fi
 # use; cleanup is handled by exit_trap)
 print_hilite "Creating temporary folder..."
 TEMP_DIR="$(mktemp -d "/tmp/${SCRIPT_NAME}.XXXXXXXXXX")"
+[[ "${KEEP_TEMP}" == "true" ]] && print_normal "Temporary folder: ${TEMP_DIR}"
 
 # Set up logging (NOTE: log is written to temporary dir/file first and then
 # processed/finalized on exit by exit_trap; backups of stdout/stderr allow
@@ -565,15 +568,17 @@ fi
 # --------------------------------------
 
 > "${TEMP_DIR}/${IBL_FOUT}"
-if (( ${#IBL_LISTS[@]} > 0 )); then
+if (( ${#IBL_LISTS[@]} > 0 )) && [[ "${IBL_USER}" != "" && "${IBL_PIN}" != "" ]]; then
 
-	# Download blocklists
+	# Download blocklists (NOTE: using 'src2' to obfuscate username and PIN
+	# in console/log output)
 	print_hilite "Downloading I-BlockList blocklists..."
 	for list in "${!IBL_LISTS[@]}"; do
 		print_normal "Downloading blocklist '${list}'..."
-		printf -v src "${IBL_URL}" "${IBL_LISTS["${list}"]}"
+		printf -v src "${IBL_URL}" "${IBL_LISTS["${list}"]}" "${IBL_USER}" "${IBL_PIN}"
+		printf -v src2 "${IBL_URL}" "${IBL_LISTS["${list}"]}" "xxx" "xxx"
 		printf -v dst "${TEMP_DIR}/${IBL_FIN1}" "${list}"
-		vprint_transfer "${dst}" "${src}"
+		vprint_transfer "${dst}" "${src2}"
 		download_file "${src}" "${dst}"
 	done