essexld is a simple, high performance service, written in C using the sxe event driven programming library, for serving information about domains and URLs over HTTP from Linux. It was built and tested on Mint/debian Linux. It doesn’t support multiple categories, but could easily be extended to do so.
The source code is under an MIT like license, allowing you to do almost anything with it. It can be downloaded from my GitHub repository:
The remainder of this article borrows heavily from the README (see the repo for all the details).
essexeld requires the latest version of sxe, which is available from my GitHub account. essexeld can be built using the latest version of my experimental baker build tool, also available from my GitHub account.
The unit tests now use tables data/domains and data/urls. These tables were generated from the MESD blacklist’s porn domains and urls lists (available here: http://www.squidguard.org/blacklists.html) using the utility program util/target/essexeld_blacklist. The generated tables are sorted lists of MD5 checksums. To regenerate the tables, run the following commands:
The script test/benchmark_essexeld.py throws the 165719 URLs in the MESD blacklist porn urls list at the service. Running the benchmark on a single system (via the loopback address, 127.0.0.1):
|SXE Build||Total Time||URLs/s|
Deploying and Restarting the Server
The server can be deployed with ansible using the provided
playbook.yml file. See the README for further details.
To restart the server,
ssh in to each host, become the root user, and run:
nohup /opt/essexeld/bin/essexld -p 80 &
The protocol for looking up the block list is simple. Issue an HTTP get request with the path string /urlinfo/1/host.name[:port][/query/string]. If the URL is not found, the server will respond with status code 404 (Not Found). If found, the server will respond with 200 (OK). Note: the body will contain the hard-coded string “porn”.