Hi all. I just setup a new install from scratch on FreeBSD (write up to come once I work out all the issues) but I can’t seem to subscribe to remote communities.
I’m running the backend and UI versions 0.17.4 and you can see the instance at https://discuss.petersanchez.com
Anytime I try to search for any community (ie, !lemmy_support@lemmy.ml) it says not found. Logs show errors about values, etc. I try via full URL or via the local instance url (`/c/lemmy_support@lemmy.ml) and it’s the same. I was able to subscribe to 1 remote community but about 10 others fail to be found.
Here’s an example of an error from the logs:
2023-06-30T00:10:30.187507Z ERROR HTTP request{http.method=GET http.scheme="http" http.host=discuss.petersanchez.com http.target=/api/v3/ws otel.kind="server" request_id=bfe8111b-8a78-4bc8-8e04-d4a91ff08837 http.status_code=101 otel.status_code="OK"}: lemmy_server::api_routes_websocket: couldnt_find_object: error decoding response body: missing field `properties` at line 1 column 158
0: lemmy_apub::fetcher::search::search_query_to_object_id
at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lemmy_apub-0.17.4/src/fetcher/search.rs:17
1: lemmy_apub::api::resolve_object::perform
with self=ResolveObject { q: "!lemmy_support@lemmy.ml", auth: Some(Sensitive) }
at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lemmy_apub-0.17.4/src/api/resolve_object.rs:21
2: lemmy_server::root_span_builder::HTTP request
with http.method=GET http.scheme="http" http.host=discuss.petersanchez.com http.target=/api/v3/ws otel.kind="server" request_id=bfe8111b-8a78-4bc8-8e04-d4a91ff08837 http.status_code=101 otel.status_code="OK"
at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lemmy_server-0.17.4/src/root_span_builder.rs:16
I can access my users json details with curl -H 'Accept: application/activity+json' https://discuss.petersanchez.com/u/peter
just fine.
Here is my nginx config:
https://paste.sr.ht/~petersanchez/2bd686389f7d0bdd5e8645fcedbae65717344885
I’m thinking this may be the issue because I pieced this together based on past ansible-lemmy changes as the newer version seems to be geared towards 0.18.0 and doesn’t appear to pass into the UI (port 1234) any more.
Any ideas? Thanks in advance!
Apologies - This is a repost. Seems my last question got lost in all the turbulence over the last 4 days but the issue continues here.
Not sure how much it helps, but that log is of your server getting bad JSON from a remote server, so your nginx config is unlikely to be the culprit there, and FWIW your nginx is quite similar to mine and I am still on 0.17.4.
I often have to search multiple times or wait a very long time before I get results, and I have to unsub/resub multiple times before it goes through on massively overloaded instances (.world, .ml, beehaw, etc). For the sake of debugging you could try searching for a community on a smaller host so you can be sure it isn’t the remote’s fault.
!admin_blog@lemmy.nrd.li
if you want to use one on my instance.edit: Just looking at this post on lemmy.ml indicates that it is still massively overloaded as I am missing all of the other comments for this thread on my instance.
edit2: A pending subscribe basically just means that your local isntance knows you want content from that community, not that the remote has accepted your subscription and not that anything will actually be federated. All of the Kbin magazines I have tried to subscribe to have never gotten out of “pending” and have never federated anything to me despite retrying multiple times.
Thank you. Yea, I’ve compared with quite a few nginx.conf’s now and I’m not quite sure what the issue could be with my nginx config. Trying the community on your instance, I get this error:
couldnt_find_community: Failed to resolve actor for admin_blog .nrd.li
I assume this is not DNS related because I can resolve
lemmy.nrd.li
without issue from the server where lemmy is running.Yeah, that (probably) isn’t a DNS resolution problem. “Failed to resolve actor” basically means it was trying to look up the actor for the community (a community is just a Group in ActivityPub parlance, so it does a WebFinger request to
https://lemmy.nrd.li/.well-known/webfinger?resource=acct%3Aadmin_blog%40lemmy.nrd.li
in this case to find that Group), and that request is failing. It could be some sort of network issue, I think I’ve seen someone with a similar issue due to missing their ca certs.Make sure you are doing all of your debugging from inside the
lemmy
container not just the server if you’re using containers.edit: Also, I had to individually load your comment with a search so I could reply, so .ml is running pretty far behind in terms of federation…
Thanks for confirming. I am running lemmy from Supervisord and not using any containers. I’m debugging via it’s log output. This is so strange as I can’t really see anything else wrong or missing from nginx config.
Are you running 0.18.0? I ask because I notice in lemmy-ansible now the config is different and there seems to be no proxy-ing for the UI, just the backend. Not quite sure how that is supposed to work now. I think I read that 0.18.0 has some changes from web socket to API calls but not sure if that means the UI is totally bypassed now?
My server is on 0.17.4 for the moment as I am waiting for them to add back captcha support with 0.18.1 before I upgrade. AFAIK the only thing you might change_ as far as nginx would be you don’t need to do the connection upgrade stuff any more since the websockets got ripped out, all the endpoints and stuff are still the same and the routing of frontend vs backend is still basically “certain routes, POSTs, and anything with
Accept
of the apub content type(s) go to backend, everything else goes to frontend”. You can see the current version of their nginx config file here: https://github.com/LemmyNet/lemmy/blob/main/docker/nginx.conf
Getting my instance to federate was an absolute ball ache and I’m running docker. I don’t envy your situation, I hope it resolves quickly :)
I’m just having a gander of your logs and instance, I’ll let you know if I find anything. I’m not familiar with FreeBSD so hopefully someone, in the future, will be your saving grace but I’ll try anyway.
The fact that you can subscribe to one community suggests that your lemmy setup works correctly and the issue may be external, maybe DNS. Are you able to provide more logs? What does your lemmy.hjson look like? That one community you’ve subscribed to, does it show a green subscribed button and load its posts, votes and comments?
Oh and really dumb question, have you enabled federation in the server settings?
Right, I’ve moved to my PC and your post makes more sense now.
It seems like the problem section of your logs is
couldnt_find_object: error decoding response body: missing field properties at line 1 column 158
. I’m thinking it may have something to do with your ngnix.conf because it might be implying that it can’t see anything at all, rather than just a missing field property.I’ve shared my nginx.conf below in the hope that, by comparison you can get your instance to federate. (This is from my instance that is federating fully and correctly).
worker_processes auto; events { worker_connections 1024; } http { upstream lemmy { # this needs to map to the lemmy (server) docker service hostname server "lemmy:8536"; } upstream lemmy-ui { # this needs to map to the lemmy-ui docker service hostname server "lemmy-ui:1234"; } server { # this is the port inside docker, not the public one yet listen 1236; listen 8536; # change if needed, this is facing the public web server_name localhost; server_tokens off; gzip on; gzip_types text/css application/javascript image/svg+xml; gzip_vary on; # Upload limit, relevant for pictrs client_max_body_size 20M; add_header X-Frame-Options SAMEORIGIN; add_header X-Content-Type-Options nosniff; add_header X-XSS-Protection "1; mode=block"; # frontend general requests location / { # distinguish between ui requests and backend # don't change lemmy-ui or lemmy here, they refer to the upstream definitions on top set $proxpass "http://lemmy-ui"; if ($http_accept = "application/activity+json") { set $proxpass "http://lemmy"; } if ($http_accept = "application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\"") { set $proxpass "http://lemmy"; } if ($request_method = POST) { set $proxpass "http://lemmy"; } proxy_pass $proxpass; rewrite ^(.+)/+$ $1 permanent; # Send actual client IP upstream proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } # backend location ~ ^/(api|pictrs|feeds|nodeinfo|.well-known) { proxy_pass "http://lemmy"; # proxy common stuff proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; # Send actual client IP upstream proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } } }
Thank you for sharing your config. Mine was nearly exactly identical. I updated some of the headers (yours were slightly different than mine) but that doesn’t seem to have made any difference. I will try enabling federation debug and see what the logs show
I’ve had a look at your instance and the community you’ve said you’re subscribed to… you’re not, there’s no posts or anything. It may show subscribed but it’s lying to you.
The UI version is showing as unknown version which is strange. I also can’t sign up which is doubly strange. So, there’s definitely an issue with your setup. I’d also guess that there’s an issue with network connectivity between your instance and the www. I can’t say anymore without further info. I’m sorry I couldn’t be of more help. I’m happy to help out more though, from one instance owner to another, drop me a message over at https://iam.dru5k1.com/mastodon. You know the saying two heads are better than one.
Thank you for the help so far! Another user successfully sent me a message so it seems some things are working. I do have federation enabled in settings (see image)
Do I need to specifically list all the instances I want to federate with? That seems counter intuitive as by default I’d like to federate with all instances and just block the ones that give me a headache.
My lemmy.hjson looks like this:
{ database: { password: "NOT REAL PASSWD" } # replace with your domain hostname: discuss.petersanchez.com bind: "127.0.0.1" federation: { enabled: true } # remove this block if you don't require image hosting pictrs: { url: "http://localhost:8080/" } }
I don’t even know what other options are available for this file. I can’t seem to find any docs for it. Let me know if something should be added/removed/changed.
DNS seems OK on the system when trying to resolve all the given communities from a server shell. I know ActivityPub isn’t for the faint of heart but in my toying around with it in the past, it was straight forward enough, just different software use different fields for different purposes. So that can create confusion in federation but in this case I’m dealing with Lemmy to Lemmy so I assume it should work heh.
Oh and the 1 community I was able to actually find is on kbin.social and it says subscribe pending:
Thanks again for the help so far!
signup hangs for me. https://discuss.petersanchez.com/signup
If you are doing a new install, I’d suggest go with latest main as opposed to 0.17.4
I see you did get one kbin community on your system.
that server is really the most problem in my testing. Maybe try https://lemmy.ca/c/nostupidquestions ?
Thanks for the answer! I have registration closed (it’s just a personal instance).
I was able to get the kbin subscription to status pending. Trying the lemmy.ca community gets this in the logs
2023-07-03T19:17:43.357158Z WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: couldnt_find_community: error decoding response body: missing field `properties` at line 1 column 166 0: lemmy_apub::fetcher::resolve_actor_identifier at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lemmy_apub-0.17.4/src/fetcher/mod.rs:16 1: lemmy_apub::api::read_community::perform with self=GetCommunity { id: None, name: Some("nostupidquestions@lemmy.ca"), auth: Some(Sensitive) } at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lemmy_apub-0.17.4/src/api/read_community.rs:30 2: lemmy_server::root_span_builder::HTTP request with http.method=GET http.scheme="http" http.host=discuss.petersanchez.com http.target=/api/v3/community otel.kind="server" request_id=8cef1a93-33c3-4197-89b9-4d709b0e9b10 http.status_code=400 otel.status_code="OK" at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lemmy_server-0.17.4/src/root_span_builder.rs:16 LemmyError { message: Some("couldnt_find_community"), inner: error decoding response body: missing field `properties` at line 1 column 166 Caused by: missing field `properties` at line 1 column 166, context: "SpanTrace" }
I think I’ll try updating to latest in main and see what happens.
So if you search for !lemmy_support@lemmy.ml and wait about 1-2 minutes, nothing happens? Because that was the way to connect my own instance. I still cannot subscribe (‘subscription pending’), but I assume that’s a software bug.
Well all. Thanks for the help. I manually built 0.18.0 and so far, it seems to be working as expected. I can find communities and subscribe to them OK, etc. Thanks for the help!