kirsle.net/www/wiki/Self-Hosting.md

12 KiB

Self Hosting

These are my notes from an experiment with running a bunch of self-hosted cloud services, and to see how well Android works in 2018 without Google services.

My device is a Nexus 7 (2013) w/ LTE (Project Fi) running Lineage OS 14.1 (Android 7) without Google Play Services installed. My server is running Debian 9 (Stretch).

Later on when my Nexus 7's screen took some physical damage and stopped working, I re-ran this experiment on a Pixel (1st gen) with Lineage OS 16 (Android 9), again without Google Play Services installed.


Summary of Solutions

My Android without Google tablet has the following features now:

  • Self-hosted e-mail account.
  • Contacts and Calendar sync from self-hosted WebDAV.
  • File sync for photo backups, password vault, etc.
  • Fennec browser which is just rebranded Firefox for Android with Firefox Sync, uBlock Origin and other familiar features.

Links to software used:

  • E-mail Hosting:
  • Calendar and Contact Sync:
    • Run standard WebDAV services (CalDAV and CardDAV)!
    • Android: DAVdroid from F-Droid.
    • Desktop (Thunderbird): CardBook for Contacts and Lightning for Calendar, then just add remote CalDAV sources to each.
    • Server: Radicale
  • Password Manager:
    • KeePass for a complete self-hosted solution.
    • Desktop: KeePass XC for Windows, Mac and Linux.
    • Android: KeePass DX
    • I sync my password vault with Syncthing.
  • Files and Password Vault Sync:
    • Syncthing - runs everywhere, works very well, no web access! My preferred pick.
    • Nextcloud - PHP, if you want web access like Dropbox, but that's not for me.
  • Open Source Android Apps (Without Google):
    • App Stores
      • F-Droid - my preferred pick, only fully open source software.
      • Amazon App Store - for a market that competes with the Play Store but without Google apps.
    • Fennec F-Droid is upstream Firefox for Android under a different brand. Supports Firefox Sync.
    • Chromium: Auto Updater for Chromium
      • Calendar and Contact Sync: DAVdroid from F-Droid.
      • KeePass: KeePass DX from F-Droid.
      • File sync: Syncthing or Nextcloud from F-Droid.
      • Maps & Navigation: OsmAnd+ seems to be the best contender but is a very clunky app. Will take my tablet on adventures just to see how it does.
      • Messaging:
        • Signal is not available on F-Droid but you can download the .apk directly from their site, and it will self-update.
        • Riot.im on F-Droid is a client for any Matrix server.

Play Store Apps

Most of this page talks about using only open-source software (F-Droid) with no Google Play Services or Play Store apps involved. Some things can be found on Amazon's App Store but most of the popular apps (Netflix, Hulu, etc) are only on Play Store.

A lot of Play Store apps rely on Google Play Services at runtime and might not work on a device without Google services installed.

To install apps from the Play Store, I used Aurora Store from F-Droid. Yalp Store is another open source client for the Play Store.

Some notes on testing how well certain apps work once installed (with no Google services on the phone):

  • Netflix
    • Works well for local playback! I was able to log in and stream shows on my Google-free phone.
    • The Chromecast button identified my SHARP ROKU TV with built-in Netflix app, but it did not see any Google Chromecast devices on my network.
  • Hulu
    • I was able to log in to the app, but after that it crashes often. My guess is it crashes trying to look for Chromecast devices. If I'm fast I can get it to play back content but haven't tested for extended periods.
  • Venmo: was usable, crashes randomly though.

Other apps I use that worked fine on my Google-free device:

  • Sync for Reddit
  • Firefox
  • Slack
  • Twitter
  • Snapchat? (didn't log in as I forgot my password but the app didn't crash)
  • Fly Delta? (doesn't crash but I wasn't flying anywhere so haven't fully tested all the app functionality)

Apps that strongly required the Google Play Services and pop up an error message right away and won't work:

  • YouTube
  • Postmates

Server Software

Email: IMAP & SMTP

I used postfix for the SMTP server and Dovecot for the IMAP server.

Using a tutorial like: https://www.tecmint.com/install-postfix-mail-server-with-webmail-in-debian/

I highly recommend this being the FIRST thing you set up and verify working. E-mail is so easy to fuck up.

Webmail

I installed Roundcube from the official Debian apt repo (apt install roundcube) and configured it in nginx.

Install MariaDB-server first and get it up and running; Debian's roundcube asks questions about the database immediately. Use dpkg-reconfigure roundcube to reconfigure it later.

I used nginx instead of Apache to host Roundcube. I needed to apt install php7-fpm and use this config:

# /etc/nginx/sites-enabled/mail.caskir.net
server {
	server_name mail.caskir.net;
	listen 443 ssl;
	listen [::]:443 ssl;

	index index.cgi index.php index.html index.htm;

	access_log /var/log/nginx/mail-access.log;
	error_log /var/log/nginx/mail-error.log;

	ssl on;
	ssl_certificate /etc/letsencrypt/live/caskir.net/fullchain.pem;
	ssl_certificate_key /etc/letsencrypt/live/caskir.net/privkey.pem;
	include ssl_params;

	root /var/lib/roundcube;

	# legacy CGI scripts
	# https://wiki.debian.org/nginx/FastCGI
	location ~ \.php$ {
		try_files $uri =404;
		fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
		fastcgi_index index.php;
		fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
		include fastcgi_params;
	}
}

server {
	server_name mail.caskir.net;
	listen 80;
	listen [::]:80;
	return 301 https://mail.caskir.net$request_uri;
}

Calendars and Contacts

I installed Radicale into a Python3 virtual environment as my normal user account.

% export WORKON_HOME=~/.virtualenvs
% mkvirtualenv -p /usr/bin/python3 radicale
Installing into ~/.virtualenvs/radicale/bin/python...
(radicale)% pip install radicale
(radicale)% which radicale
/home/kirsle/.virtualenvs/radicale/bin/radicale

I put my Radicale config at ~/radicale/config with these contents:

# /home/kirsle/radicale/config
[server]
hosts = 127.0.0.1:5232

[auth]
type = http_x_remote_user

[storage]
filesystem_folder = ~/.var/lib/radicale/collections

The Radicale service is managed by supervisor which runs it as a low-privileged account:

# /etc/supervisor/conf.d/radicale.conf
[program:radicale]
command = /home/kirsle/.virtualenvs/radicale/bin/radicale --config /home/kirsle/radicale/config
user = kirsle
directory = /home/kirsle/radicale

And I put an nginx proxy in front so I can terminate SSL there (using Let's Encrypt for free automated SSL certs).

# /etc/nginx/sites-enabled/caskir.net
server {
	server_name www.caskir.net;
	listen 443 ssl;
	listen [::]:443 ssl;

	index index.html index.htm;

	access_log /var/log/nginx/caskir-access.log;
	error_log /var/log/nginx/caskir-error.log;

	ssl on;
	ssl_certificate /etc/letsencrypt/live/caskir.net/fullchain.pem;
	ssl_certificate_key /etc/letsencrypt/live/caskir.net/privkey.pem;
	include ssl_params;

	root /home/kirsle/www;

	location /dav/ { # The trailing / is important!
		proxy_pass http://localhost:5232/; # The / is important!
		proxy_set_header     X-Script-Name /dav;
		proxy_set_header     X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header     X-Remote-User $remote_user;
		auth_basic           "Radicale - Password Required";
		auth_basic_user_file /etc/nginx/htpasswd;
	}
}

server {
	server_name caskir.net;
	listen 443 ssl;
	listen [::]:443 ssl;

	ssl on;
	ssl_certificate /etc/letsencrypt/live/caskir.net/fullchain.pem;
	ssl_certificate_key /etc/letsencrypt/live/caskir.net/privkey.pem;
	include ssl_params;

	return 301 https://www.caskir.net$request_uri;
}

server {
	server_name www.caskir.net caskir.net;
	listen 80;
	listen [::]:80;
	return 301 https://www.caskir.net$request_uri;
}

HTTP Basic Auth refresher:

# To get the htpasswd commands.
$ apt install apache2-utils

# Create the password database.
$ htpasswd -c /etc/nginx/htpasswd kirsle
Password:
Verify:

# Add another user.
$ htpasswd /etc/nginx/htpasswd alice

Radicale has a very minimal web interface so you'll need a WebDAV client to actually import your data. I used CardBook in Thunderbird and just imported my contacts file from Google Takeout.

Thunderbird has CardBook and Lightning add-ons that can sync with the WebDAV service. GNOME Calendar works, too, but depends on a full GNOME desktop environment (installing it by itself on Xfce leaves it in a broken state as it can't interact with GNOME's Online Accounts system).

For Android, DAVdroid is available on F-Droid and will sync contacts and calendars to your device.


File Sync

I chose Syncthing over Nextcloud because it fit my needs better. Nextcloud is a PHP application that has a web interface, like Dropbox, to log in and access your files. Nextcloud also syncs contacts and address books (so you don't need Radicale).

I don't require web access to my files, as I'll always have either my phone or one of my computers with me, and I really only use Syncthing to sync my password database. Having a complicated web app written in PHP would present quite a surface area for random drive-by attacks.

I sync between my desktop PC, offsite web server, two laptops and two Android devices.

Download Linux packages at Syncthing.net; they have Debian/Ubuntu APT repositories to keep it updated.

To access its web interface securely, tunnel it through SSH like:

% ssh -L 8384:localhost:8384 user@hostname

And then accessing http://localhost:8384/ on your desktop should access the web interface on the server.

For Android, Syncthing is available on F-Droid.

Messaging

I found out you're not allowed to sync Signal between two different mobile devices (like my phone and my Nexus 7 tablet), only between my phone and desktop PCs. However, it does seem Signal can be downloaded directly from their website and would probably work on a normal phone. I couldn't activate it with my tablet's "phone number" because it doesn't receive SMS.

Riot.im is available on F-Droid and can get you on to the Matrix federated protocol.