 | From The Admin’s Desk
|
This is about https://hear-me.social. It’s about what I’ve done to set it up and to maintain it. I am not an expert in many of the things I had to do. My work may not be optimal for the requirements, but I’m presenting the information here in case it does help someone else.
- Current Costs (~$170/month)
• $22/month for Cloudflare services
• $48/month for cloud VPS (main)
• $12/month for cloud VPS (Elasticsearch)
• $64/month for PostgreSQL RDS
• About $17/month for Wasabi S3 bucket storage
• Some Cloudflare caching costs.
• Backup and snapshot costs
• $2.50/month for domain renewal fees - Where To Donate
Buy me a coffee: https://ko-fi.com/JerryL
Patreon: https://www.patreon.com/JerryFediverse
Liberapay: https://liberapay.com/JerryFediverse
Bitcoin: bc1qqgjydw28athy9jare342eh703jagmwecmmgemm
ETH: 0x07aD621C7661f75123b0667407C007B4A3acb5c6
Aug 19, 2024 – Upgraded from Mastodon 4.2.11 to 4.2.12
Updated Mastodon from version 4.2.11 to 4.2.12 using essentially the same steps used on Sept 24, 2023
Aug 16, 2024 – Upgraded from Mastodon 4.2.10 to 4.2.11
Updated Mastodon from version 4.2.10 to 4.2.11 using essentially the same steps used on Sept 24, 2023
Jul 13, 2024 – Doubled the power of the DB cluster. It now has 4 GB RAM and 2 CPUs. Also increased the number of background workers from 40 to 90 by adding a 3rd sidekiq process, since the number of available DB connections increased from 47 to 97. Now each of the 3 processes manages 30 workers. See the entry for Apr 21, 2024 for information on how I set this up.
By adding Flipboard relaying, to the already large number of relays, the number of messages coming in in the middle of the day would cause the “pull” queue to back up too much because posts were waiting for link resolutions. 40 workers were not enough. This should no longer be an issue.
Also, when a suspended spam account gets deleted, the over 16K messages that suddenly enter the “pull” queue to notify every ActivityPub connected instance to delete the account, will now clear out more quickly. On some days, there can be over 160K jobs suddenly placed into the “pull” queue in one moment as a bunch of spam accounts are marked for deletion at the same time.
The “pull” queue also handles post deletions, post delete and repost, resolving links in posts. A lot more than just account deletion. When there are a lot of spam account deletions, these other jobs sit in the “pull” queue waiting their turn and can be delayed too long. This is why I became interested in adding more workers, which I couldn’t do without increasing the DB cluster CPUs and RAM.
Jul 4, 2024 – Upgraded from Mastodon 4.2.9 to 4.2.10
Updated Mastodon from version 4.2.9 to 4.2.10 using essentially the same steps used on Sept 24, 2023
Jun 1, 2024 – Upgraded Ubuntu from 23.10 to 24.04 LTS. No issues.
May 30, 2024 – Upgraded from Mastodon 4.2.8 to 4.2.9
Updated Mastodon from version 4.2.8 to 4.2.9 using essentially the same steps used on Sept 24, 2023
Apr 21, 2024 – Added a 2nd sidekiq process
I’ve been noticing that all the queues, except the pull queue, have been at 0 entries. The pull queue, the lowest priority queue, has been backing up since the past 2 weeks. This is a new situation. It may be because of Threads federating (although there are only roughly 1.3K Threads accounts federating), and because I added a relay to Flipboard.com (I’m assuming this is the main reason).
Although it’s a low-priority queue, I don’t like to see it backing up more than 5 minutes. Today it backed up 56 minutes.
The database is on a separate server, and uses pgBouncer for connection pooling. I use 40 connections for 40 sidekiq tasks (each task needs its own connection). I had one process running, doling out the work to these 40 tasks (/etc/systemd/system/mastodon-sideqik.service). But I noticed that hardly ever were more than 30 tasks ever active at the same time. Clearly, the single process was unable to handle 40 workers.
Today, I created a second process, so each process now manages 20 tasks. This quickly cleared up the pull queue. While it was cleaning up, the CPU went from its usual 38% to about 68% and now is down to the normal 38%.
I used the instructions here to create the second process: https://github.com/mastodon/mastodon/discussions/19797
Feb 23, 2024 – Upgraded from Mastodon 4.2.7 to 4.2.8
Updated Mastodon from version 4.2.7 to 4.2.8 using essentially the same steps used on Sept 24, 2023
Feb 16, 2024 – Upgraded from Mastodon 4.2.6 to 4.2.7
Updated Mastodon from version 4.2.6 to 4.2.7 using essentially the same steps used on Sept 24, 2023
Feb 15, 2024 – Upgraded from Mastodon 4.2.5 to 4.2.6
Updated Mastodon from version 4.2.5 to 4.2.6 using essentially the same steps used on Sept 24, 2023
Feb 14, 2024 – Upgraded from Mastodon 4.2.4 to 4.2.5
Updated Mastodon from version 4.2.4 to 4.2.5 using essentially the same steps used on Sept 24, 2023
Jan 27, 2024 – Upgraded from Mastodon 4.2.3 to 4.2.4
Updated Mastodon from version 4.2.3 to 4.2.4 using essentially the same steps used on Sept 24, 2023, except the update was to Ruby 3.2.3.
Dec 05, 2023 – Upgraded from Mastodon 4.2.2 to 4.2.3
Updated Mastodon from version 4.2.2 to 4.2.3 using essentially the same steps used with the previous upgrade (see below).
Dec 04, 2023 – Upgraded from Mastodon 4.2.1 to 4.2.2
Updated Mastodon from version 4.2.1 to 4.2.2 using essentially the same steps used with the previous upgrade (see below).
Dec 03, 2023 – Upgraded the O/S from Ubuntu from 23.04 to 23.10
All went smoothly, not like the last upgrade.
Oct 10, 2023 – Upgraded from Mastodon 4.2.0 to 4.2.1
Updated Mastodon from version 4.2.0 to 4.2.1 using essentially the same steps used with the previous upgrade (see below).
Oct 9, 2023 – Increased concurrent sidekiq queue workers from 25 to 35 to speed up queue processing.
To speed up queue processing, I updated the number of concurrent sidekiq queue workers from 25 to 35
- I used PGBouncer associated with the Digital Ocean managed database and increased the number of connections in the pool to 43.
- Edited the /etc/systemd/system/mastodon-sidekiq.service file, after making a copy, just to be safe.
- Set the DB Pool size to 35
- Environment=”DB_POOL=35″
- Set the EXECSTART to start 35 concurrent workers. Note that each worker needs 1 database pool connection
- ExecStart=/home/mastodon/.rbenv/shims/bundle exec sidekiq -c 35
- systemctl daemon-reload
- systemctl stop mastodon-sidekiq
- systemctl reload mastodon-sidekiq
- Set the DB Pool size to 35
Sept 24, 2023 – Upgraded from Mastodon 4.1.9 to 4.2
Updated Mastodon from version 4.1.9 to 4.2. Some notes:
- Dependencies in the documentation are misleading. Need to install Ruby 3.2.2, not 3.2 as it indicated at the time of this writing.
- The instructions for how to update Ruby are incomplete and won’t work if taken literally. Update to 3.2.2 and make sure it’s the default version.
Sept 20, 2023 – Upgraded from Mastodon 4.1.8 to 4.1.9
Updated Mastodon from version 4.1.8 to 4.1.9 using essentially the same steps used with the previous upgrade (see below). This was necessary because 4.1.8 broke translation.
Sept 19, 2023 – Upgraded from Mastodon 4.1.7 to 4.1.8
Updated Mastodon from version 4.1.7 to 4.1.8 using essentially the same steps used with the previous upgrade (see below).
Sept 8, 2023 – Upgraded from Mastodon 4.1.6 to 4.1.7
Updated Mastodon from version 4.1.6 to 4.1.7 using essentially the same steps used with the previous upgrade (see below).
July 31, 2023 – Upgraded from Mastodon 4.1.5 to 4.1.6
Updated Mastodon from version 4.1.5 to 4.1.6 using essentially the same steps used with the previous upgrade (see below).
July 29, 2023 – Upgraded the O/S from Ubuntu 22.04 to 23.04
Did not go smoothly.
- First, there was a bad yarn key in the repository. Fixed it with:
- sudo apt-key del 23E7166788B63E1E
- curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add –
- sudo apt update
- Then got, “error No such file or directory – /home/mastodon/live/vendor/bundle/ruby/3.0.0/gems/charlock_holmes-0.7.7/lib/charlock_holmes/charlock_holmes.so”
- Had to rebuild Mastodon but instead of using “bundle install”, had to use “bundle install –redownload” in the build procedure.
July 22, 2023 – Upgraded from Mastodon 4.1.4 to 4.1.5
Updated Mastodon from version 4.1.4 to 4.1.5 using essentially the same steps used with the previous upgrade (see below).
July 8, 2023 – Upgraded from Mastodon 4.1.3 to 4.1.4
Updated Mastodon from version 4.1.3 to 4.1.4 using essentially the same steps used with the previous upgrade (see below).
July 6, 2023 – Upgraded from Mastodon 4.1.2 to 4.1.3
Updated Mastodon from version 4.1.2 to 4.1.3 using essentially the same steps used with the previous upgrade (see below).
April 8, 2023 – Upgraded from Mastodon 4.1.1 to 4.1.2
Updated Mastodon from version 4.1.1 to 4.1.2 using essentially the same steps used with the previous upgrade (see below). However this update also required upgrading to a higher Ruby version.
March 18, 2023 – Upgraded from Mastodon 4.1.0 to 4.1.1
Updated Mastodon from version 4.1.0 to 4.1.1 using essentially the same steps used with the previous upgrade (see below)
February 26, 2023 – Enabled translations
Added ability to translate posts, warts and all.
Mastodon’s new version provides two ways to incorporate post translations. One is using LibreTranslate, the other using DeepL. LibreTranslate is completely free if you set it up on your own rather beefy server, so not so free in terms of cost and effort. They have a paid option that allows using their hosted API. DeepL is only a hosted API, and it’s free should you translate 500,000 characters or fewer per month.
I went with DeepL because of the low effort, and for this instance, low cost. It was easy. I created an account at https://www.deepl.com, got my API key and then added two simple lines to the .env.production file:
DEEPL_API_KEY=the api key here
DEEPL_PLAN=free
If you need the paid plan, it would be DEEPL_PLAN=pro
And this all. Reboot. And it should start working, but you may need to clear the browser cache and maybe even the Cloudflare cache or equivalent.
You can run this curl command to confirm it’s enabled (of course, change the instance domain name)
curl -s https://hear-me.social/api/v2/instance | jq .configuration.translation
You should see:
{
“enabled”: true
}
Some things don’t work:
- If the poster’s preferred language matches yours, then no translation option will be offered, even if they indicate the language to not be their preferred language before sending the message. This looks like a defect.
- If the language used, or the one you want to translate to, doesn’t exist in DeepL, you get a 503 error. Not classy.
- Poll questions are translated, but not the choices.
February 11, 2023 – Upgraded from Mastodon 4.0.2 to 4.1, increased post size to 1,500 characters, updated cron jobs (see cron job below)
Updated Mastodon from version 4.0.2 to 4.1, using these steps:
- Took the instance down and made a final version 4.0.2 snapshot, just in case
- In the console, became user mastodon
- In /home/mastodon/live, ran the following to restore the files that were changed to allow 1,200 character posts, else the git fetch would fail
- git restore app/javascript/mastodon/features/compose/components/compose_form.js
- git restore app/serializers/rest/instance_serializer.rb
- git fetch && git checkout v4.1.0
- bundle install
- yarn install
- RAILS_ENV=production bundle exec rails db:migrate
- RAILS_ENV=production bundle exec rails assets:precompile
- Exited back to root
- systemctl restart mastodon-sidekiq
- systemctl reload mastodon-web
- systemctl restart mastodon-streaming
- Followed instructions at https://www.draklyckan.se/2021/11/how-to-increase-the-character-limit-for-toots-in-mastodon/ to change posting length to 1,500 characters
January 14, 2023 – Added outbound email service to the server
While the Mastodon server has built-in outbound email for sending emails to users, the server itself does not have any. Because I’m running cron jobs, and cron likes to email output and error messages, I added outbound email today using the information in https://gist.github.com/movd/7a9e3db63d076f85d16c7dcde62fe401
Speaking of cron, the cron jobs weren’t running, and it was hard to know the problem without email capability. I believe the cron jobs will now work and will also send emails. See below for the cron job setup.
December 31, 2022 – Put elasticsearch into its own server
After installing elasticsearch, I discovered that it eats most of the memory on the instance. As this instance grows, it needs its memory or things are going to get slow and may start failing. So, today I put elasticsearch onto its own $12/month VPS. It has 1 virtual CPU, 2 GB memory, and 40 GB disk space. I think it should be enough.
If you want to do this on your instance, keep in mind that elasticsearch is set up to expect all connections to be from localhost, not from a different server. To make it work:
- In the .env.production file, change WS_HOST to the local IP address (or host name) of the server that now has elasticsearch
- In the /etc/elasticsearch/elasticsearch.yml file, make these changes:
- network.host: <ip of elasticsearch instance or host name> (Don’t include the brackets)
- http.port: 9200
- discovery.type: single-node
- Specifying the port, although it defaults to 9200, is important. I found it trying to use a different port for some reason. This makes sure it stays 9200
- If the discovery type is not specified, it assumes multiple hosts are going to connect, and it will insist, therefore, that you provide more than one host connection IP and it won’t start
December 30, 2022 – Added automated cleanup (updated this on 11-Feb-2023 to include V 4.1 options)
I leave both the Media Cache and Content Cache retention as undefined in the Admin because, while it seemed like a good idea that they added it in V4, it has the unfortunate side effect of removing people’s bookmarked and favorited posts. So, I can’t use it. But I also don’t want stuff piling up. Today I added 5 cron jobs to do clean ups weekly. The media orphan removal may not be a good idea if you pay S3 API costs. It may be cheaper to leave the media. But, with Wasabi S3, I don’t currently pay API fees. I added these to the cron job for user mastodon.
[email protected]
[email protected]
SHELL=/bin/bash
PATH=/home/mastodon/.rbenv/shims:/home/mastodon/.rbenv/bin:/usr/local/bin:/usr/bin:/bin
RAILS_ENV=production
# Run every Monday at 12 AM ET
0 0 * * 1 cd /home/mastodon/live && bin/tootctl media remove –remove-headers –include-follows
# Run every Monday at 1 AM ET
0 1 * * 1 cd /home/mastodon/live && bin/tootctl media remove –prune-profiles –include-follows
#Run every Tuesday at 1 AM ET
0 1 * * 2 cd /home/mastodon/live && bin/tootctl media remove-orphans
#Run every Tuesday at 3 AM ET
0 3 * * 2 cd /home/mastodon/live && bin/tootctl preview_cards remove
#Run every Wednesday at 1 AM ET
0 1 * * 3 cd /home/mastodon/live && bin/tootctl accounts cull
December 29, 2022 – Getting relay feed from infosec.exchange
Because there aren’t many users here yet, the Federation timeline is not the most interesting timeline because this instance hasn’t seen many posts yet from other servers. The Admin at infosec.exchange graciously allows this instance to share his local feed. Since his instance has 20K users, it’s good stuff and makes for an interesting Federation feed. So, when people come to take a look, they will get a good idea of what is being posted on Mastodon.
December 25, 2022 – Added text searching
Added ElasticSearch to the instance so people can do text searches. I used https://docs.joinmastodon.org/admin/optional/elasticsearch/ as instructions and it went smoothly. Here is the official description of how searching works:
Mastodon’s full-text search allows logged-in users to find results from their own statuses, their mentions, their favorites, what they boosted, and their bookmarks.
It deliberately does not allow searching for arbitrary strings in the entire database.
December 19, 2022 – Recreated the instance with new domain name hear-me.social, replacing hear-ye.com
Because Kanye West changed his name to “Ye”, the name hear-ye.com suddenly became unpalatable. Since Mastodon instances are permanently tied to their domain names, the only way to change the instance to hear-me.social was to entirely rebuild everything from scratch. Today I did this. Luckily, only 4 users signed up so I will contact them and ask them to create new accounts on the new server. Once this happens I will permanently redirect hear-ye.com to hear-me.social, and eventually release hear-ye.com .
December 11, 2022 – Increased maximum message size from the default 500 characters to 1,200 characters
I increased the maximum post size from the default 500-characters to 1,200 characters. A 280-character limit for Twitter is good because there are so many bot and troll postings. An enforced short posting is great over there. But I don’t see crap postings on Mastodon, and I think people can be trusted with a 1,200-character limit.
I used these instructions: https://www.draklyckan.se/2021/11/how-to-increase-the-character-limit-for-toots-in-mastodon/
If anyone wants to do the same on their instance, keep in mind that the changes will need to be reapplied with every upgrade, unless changing the limit is added to the Administrator UI. I don’t think you can do this if you are using Docker or if you are using a pre-created image. It can only be done if you can recompile the application.
When modifying any of these files, make a copy first, just in case you need to revert.
The last step in the article is to add a construct to the end of the “instance_serializer.rb” file. Just to point out, it goes before “private”, and the value does not include the brackets. Mine looks like this:
def max_toot_chars
1200
end
December 10, 2022 – Enabled S3 bucket caching via Cloudflare
We are using Wasabi S3 buckets for media storage because they provide 11 9’s of reliability, are relatively inexpensive, and Wasabi does not charge for egress, although they reserve the right to kick you off if you download too much, especially for a website. Before this cache change, every request for a cacheable file was going back to Wasabi directly without any caching other than the browser cache. Types of cacheable files can be found here: https://developers.cloudflare.com/cache/about/default-cache-behavior/
Cloudflare provides S3 bucket caching which means that when a file is downloaded from Wasabi, they will retain it in their cache for 1 hour (on the pro plan, 2 hours on the free plan).
To enable this, all requests for cacheable files must go through Cloudflare. This is done by setting up a Cloudflare proxied CNAME that matches the name of the Wasabi bucket. The CNAME points to, in our case, s3.wasabi.sys. This way all requests for files from the Wasabi bucket will first go through the Cloudflare proxy. Cloudflare will not send the request, however, to Wasabi, but instead direct the call into the Cloudflare cache process for fulfillment.
To enable this, the following was done:
- Set up a proxied CNAME in Cloudflare: CNAME “files.hear-me.social” was set up to point to “s3.wasabi.com”
- Then, in the “.env.production” file:
- changed S3_ALIAS_HOST=s3.wasabisys.com/files.hear-me.social, to S3_ALIAS_HOST=files.hear-me.social
- Luckily, the original bucket name, files.hear-me.social, was named, so I could simply set up a CNAME that exactly matched it. If this weren’t the case, I would have had to first rename the existing bucket to match the CNAME I wanted.
- I am not using the Cloudflare “Cache Reserve” because I don’t pay egress fees with Wasabi. If you are paying egress fees, then Cache Reserve may be a cheaper option than paying your S3 egress fees.
December 9, 2022 – Uploading images to Mastodon gives a 403 – Not Authorized error
This type of unauthorized error is not related to authentication. It means that something is not allowed by the server or infrastructure, in this case uploading an image file. Uploading an image shouldn’t be blocked. Turns out that enabling the OWASP firewall in Cloudflare (“Pro” feature) violates some 26 to 29 OWASP rules, mostly illegal and suspicious character input rules. It seems to me that Mastodon is not properly uploading image files if all the bytes are being tested by the firewall. This must be a significant vulnerability. I tried making the rules as forgiving as possible, but it didn’t help. I had to change the action from “block” to “log” to get image uploading to work. I’ll test this again in the next version.
When I registered a new domain with Cloudflare Professional, the OWASP setup screen was entirely different and I didn’t have this problem. It’s possible they are testing more than one interface.
December 3, 2022 – Enabled Digital Ocean Connection Pools on the managed PostreSQL database
Because of everything I’ve heard about sidekiq and all the database connections it uses, I decided to enable connection pooling on the PostgreSQL instance. I couldn’t find any documentation, and it took experimenting to make it work.
The first thing to note is that the connection pool (which uses PGBouncer) requires an SSL connection, which Mastodon does not do by default. Here is everything that must be done:
- Create a pool named “mastodon_production_pool”
- Select the “mastodon_production” database from the drop down
- Select user “doadmin” from the drop down
- Make sure you select “Transaction” mode
- I used a pool size of 12 because it says to use about half of the available connections to start
- Under connection details, under the connection pool, click “Download CA Certificate”. Then create the folder, (owned by user mastodon), /home/mastodon/.postgresql
- Copy the downloaded certificate to this folder and rename it to “root.crt”
- In the .env.production file (make a copy first), add SSLMODE=verify-full, PREPARED_STATEMENTS=false, DB_NAME=”mastodon_production_pool”, and change DB_PORT to the port of the connection pool. It’s probably 1 higher than the database port
- Save it and reboot
November 26, 2022 – Site opened to the public
Hosted in Digital Ocean consisting of:
- One VPS (Virtual Private Server) 8 GB RAM / 4 CPU / 160GB SSD and 5TB transfer
- Ubuntu 22.04 LTS
- Hosted Services: Web, Streamer, Redis, Sidekiq
- Separate PostgreSQL Digital Ocean managed instance (RDS) (1 GB RAM / 10 GB disk) so always backed up and maintained
- Technical Note: All Digital Ocean documentation and posts say only SSL connections are allowed with Digital Ocean managed databases. However, in the default configuration, Mastodon uses “prefer” connection request mode and does not do an SSL connection. There’s always the chance that Digital Ocean will change their configuration to require SSL. If this happens, see the notes above relating to Connection Pools on how to enable SSL connections. The Connection Pool does require SSL.
- Media files are stored in Wasabi S3 buckets for reliability (eleven 9’s reliability)
- Domain email services provided by Dreamhost.com
- Cloudflare Professional services for an abundance of protection and performance enhancements
Technical Note: The firewall is set to only allow traffic from Cloudflare proxy servers and so the server is both hidden by Cloudflare and protected from hackers since they must go through Cloudflare to interact with it.
Technical Notes when building:
- Add some swap space. Compiling needs a lot of memory
- When creating the mastodon user, suggest initially creating it with a password and giving it sudo access. Came across issues relating to the nodejs versions. For example, Mastodon code must be compiled using nodejs V16, but I somehow had V18. Needed to install nvm (node version manager) to overcome this. Also had to install a higher version of yarn so “yarn install –pure-lockfile” would work. It was handy being able to fix these things. Remove user mastodon from sudo and remove the password at the end
- Do “sudo gpasswd -a www-data mastodon” to make sure mastodon has the right access to all the web pages. Sometimes it ends up not having it
- Must compile with Ruby 3.0.4. Some documentation has 3.0.3
- If using a cloud VPS, take a snapshot before adding Mastodon code. This way if things go bad, you have your Ubuntu image ready to try it again and you can use it as a base for additional servers if you break out some of the services later or want to set up a test instance
- If you are using Cloudflare caching, DO NOT USE AUTO-MINIFY. If you do, the static elements will not appear in the browser because after Cloudflare minifies them, the calculated hash of the element will no longer match the hash for the element on the Mastodon instance and the browser will refuse to render it. This becomes difficult to notice because if you happen to have the elements in the browser cache from before turning on the auto-minify, they are in your browser cache in their original form and so will render. What happens is that when others try to load the web pages for the first time, they will only see a couple of elements render. You’ll see the hash calculation error in the browser’s console window. After turning off the auto-minify, if you had it on, you will have to manually flush the Cloudflare cache, or it will retain the minified versions and keep giving it to the browsers until they go stale.