Hi all. I just setup a new install from scratch on FreeBSD (write up to come once I work out all the issues) but I can’t seem to subscribe to remote communities.

I’m running the backend and UI versions 0.17.4 and you can see the instance at https://discuss.petersanchez.com

Anytime I try to search for any community (ie, !lemmy_support@lemmy.ml) it says not found. Logs show errors about values, etc. I try via full URL or via the local instance url (`/c/lemmy_support@lemmy.ml) and it’s the same. I was able to subscribe to 1 remote community but about 10 others fail to be found.

Here’s an example of an error from the logs:

2023-06-30T00:10:30.187507Z ERROR HTTP request{http.method=GET http.scheme="http" http.host=discuss.petersanchez.com http.target=/api/v3/ws otel.kind="server" request_id=bfe8111b-8a78-4bc8-8e04-d4a91ff08837 http.status_code=101 otel.status_code="OK"}: lemmy_server::api_routes_websocket: couldnt_find_object: error decoding response body: missing field `properties` at line 1 column 158
   0: lemmy_apub::fetcher::search::search_query_to_object_id
             at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lemmy_apub-0.17.4/src/fetcher/search.rs:17
   1: lemmy_apub::api::resolve_object::perform
           with self=ResolveObject { q: "!lemmy_support@lemmy.ml", auth: Some(Sensitive) }
             at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lemmy_apub-0.17.4/src/api/resolve_object.rs:21
   2: lemmy_server::root_span_builder::HTTP request
           with http.method=GET http.scheme="http" http.host=discuss.petersanchez.com http.target=/api/v3/ws otel.kind="server" request_id=bfe8111b-8a78-4bc8-8e04-d4a91ff08837 http.status_code=101 otel.status_code="OK"
             at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lemmy_server-0.17.4/src/root_span_builder.rs:16

I can access my users json details with curl -H 'Accept: application/activity+json' https://discuss.petersanchez.com/u/peter just fine.

Here is my nginx config:

https://paste.sr.ht/~petersanchez/2bd686389f7d0bdd5e8645fcedbae65717344885

I’m thinking this may be the issue because I pieced this together based on past ansible-lemmy changes as the newer version seems to be geared towards 0.18.0 and doesn’t appear to pass into the UI (port 1234) any more.

Any ideas? Thanks in advance!

Apologies - This is a repost. Seems my last question got lost in all the turbulence over the last 4 days but the issue continues here.

  • terribleplan@lemmy.nrd.li
    link
    fedilink
    English
    arrow-up
    2
    ·
    edit-2
    1 year ago

    Not sure how much it helps, but that log is of your server getting bad JSON from a remote server, so your nginx config is unlikely to be the culprit there, and FWIW your nginx is quite similar to mine and I am still on 0.17.4.

    I often have to search multiple times or wait a very long time before I get results, and I have to unsub/resub multiple times before it goes through on massively overloaded instances (.world, .ml, beehaw, etc). For the sake of debugging you could try searching for a community on a smaller host so you can be sure it isn’t the remote’s fault. !admin_blog@lemmy.nrd.li if you want to use one on my instance.

    edit: Just looking at this post on lemmy.ml indicates that it is still massively overloaded as I am missing all of the other comments for this thread on my instance.

    edit2: A pending subscribe basically just means that your local isntance knows you want content from that community, not that the remote has accepted your subscription and not that anything will actually be federated. All of the Kbin magazines I have tried to subscribe to have never gotten out of “pending” and have never federated anything to me despite retrying multiple times.

    • lemmy_newbieOP
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      Thank you. Yea, I’ve compared with quite a few nginx.conf’s now and I’m not quite sure what the issue could be with my nginx config. Trying the community on your instance, I get this error:

      couldnt_find_community: Failed to resolve actor for admin_blog@lemmy.nrd.li

      I assume this is not DNS related because I can resolve lemmy.nrd.li without issue from the server where lemmy is running.

      • terribleplan@lemmy.nrd.li
        link
        fedilink
        English
        arrow-up
        1
        ·
        edit-2
        1 year ago

        Yeah, that (probably) isn’t a DNS resolution problem. “Failed to resolve actor” basically means it was trying to look up the actor for the community (a community is just a Group in ActivityPub parlance, so it does a WebFinger request to https://lemmy.nrd.li/.well-known/webfinger?resource=acct%3Aadmin_blog%40lemmy.nrd.li in this case to find that Group), and that request is failing. It could be some sort of network issue, I think I’ve seen someone with a similar issue due to missing their ca certs.

        Make sure you are doing all of your debugging from inside the lemmy container not just the server if you’re using containers.

        edit: Also, I had to individually load your comment with a search so I could reply, so .ml is running pretty far behind in terms of federation…

        • lemmy_newbieOP
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 year ago

          Thanks for confirming. I am running lemmy from Supervisord and not using any containers. I’m debugging via it’s log output. This is so strange as I can’t really see anything else wrong or missing from nginx config.

          Are you running 0.18.0? I ask because I notice in lemmy-ansible now the config is different and there seems to be no proxy-ing for the UI, just the backend. Not quite sure how that is supposed to work now. I think I read that 0.18.0 has some changes from web socket to API calls but not sure if that means the UI is totally bypassed now?

          • terribleplan@lemmy.nrd.li
            link
            fedilink
            English
            arrow-up
            1
            ·
            1 year ago

            My server is on 0.17.4 for the moment as I am waiting for them to add back captcha support with 0.18.1 before I upgrade. AFAIK the only thing you might change_ as far as nginx would be you don’t need to do the connection upgrade stuff any more since the websockets got ripped out, all the endpoints and stuff are still the same and the routing of frontend vs backend is still basically “certain routes, POSTs, and anything with Accept of the apub content type(s) go to backend, everything else goes to frontend”. You can see the current version of their nginx config file here: https://github.com/LemmyNet/lemmy/blob/main/docker/nginx.conf

  • Dru5k1@lem.dru5k1.com
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 year ago

    Getting my instance to federate was an absolute ball ache and I’m running docker. I don’t envy your situation, I hope it resolves quickly :)

    I’m just having a gander of your logs and instance, I’ll let you know if I find anything. I’m not familiar with FreeBSD so hopefully someone, in the future, will be your saving grace but I’ll try anyway.

    • Dru5k1@lem.dru5k1.com
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      The fact that you can subscribe to one community suggests that your lemmy setup works correctly and the issue may be external, maybe DNS. Are you able to provide more logs? What does your lemmy.hjson look like? That one community you’ve subscribed to, does it show a green subscribed button and load its posts, votes and comments?

      Oh and really dumb question, have you enabled federation in the server settings?

      • Dru5k1@lem.dru5k1.com
        link
        fedilink
        English
        arrow-up
        1
        ·
        edit-2
        1 year ago

        Right, I’ve moved to my PC and your post makes more sense now.

        It seems like the problem section of your logs is couldnt_find_object: error decoding response body: missing field properties at line 1 column 158. I’m thinking it may have something to do with your ngnix.conf because it might be implying that it can’t see anything at all, rather than just a missing field property.

        I’ve shared my nginx.conf below in the hope that, by comparison you can get your instance to federate. (This is from my instance that is federating fully and correctly).

        worker_processes auto;
        events {
            worker_connections 1024;
        }
        http {
            upstream lemmy {
                # this needs to map to the lemmy (server) docker service hostname
                server "lemmy:8536";
            }
            upstream lemmy-ui {
                # this needs to map to the lemmy-ui docker service hostname
                server "lemmy-ui:1234";
            }
        
            server {
                # this is the port inside docker, not the public one yet
                listen 1236;
                listen 8536;
                # change if needed, this is facing the public web
                server_name localhost;
                server_tokens off;
        
                gzip on;
                gzip_types text/css application/javascript image/svg+xml;
                gzip_vary on;
        
                # Upload limit, relevant for pictrs
                client_max_body_size 20M;
        
                add_header X-Frame-Options SAMEORIGIN;
                add_header X-Content-Type-Options nosniff;
                add_header X-XSS-Protection "1; mode=block";
        
                # frontend general requests
                location / {
                    # distinguish between ui requests and backend
                    # don't change lemmy-ui or lemmy here, they refer to the upstream definitions on top
                    set $proxpass "http://lemmy-ui";
        
                    if ($http_accept = "application/activity+json") {
                      set $proxpass "http://lemmy";
                    }
                    if ($http_accept = "application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\"") {
                      set $proxpass "http://lemmy";
                    }
                    if ($request_method = POST) {
                      set $proxpass "http://lemmy";
                    }
                    proxy_pass $proxpass;
        
                    rewrite ^(.+)/+$ $1 permanent;
                    # Send actual client IP upstream
                    proxy_set_header X-Real-IP $remote_addr;
                    proxy_set_header Host $host;
                    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                }
        
                # backend
                location ~ ^/(api|pictrs|feeds|nodeinfo|.well-known) {
                    proxy_pass "http://lemmy";
                    # proxy common stuff
                    proxy_http_version 1.1;
                    proxy_set_header Upgrade $http_upgrade;
                    proxy_set_header Connection "upgrade";
        
                    # Send actual client IP upstream
                    proxy_set_header X-Real-IP $remote_addr;
                    proxy_set_header Host $host;
                    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                }
            }
        }
        
        • lemmy_newbieOP
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 year ago

          Thank you for sharing your config. Mine was nearly exactly identical. I updated some of the headers (yours were slightly different than mine) but that doesn’t seem to have made any difference. I will try enabling federation debug and see what the logs show

        • Dru5k1@lem.dru5k1.com
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 year ago

          I’ve had a look at your instance and the community you’ve said you’re subscribed to… you’re not, there’s no posts or anything. It may show subscribed but it’s lying to you.

          The UI version is showing as unknown version which is strange. I also can’t sign up which is doubly strange. So, there’s definitely an issue with your setup. I’d also guess that there’s an issue with network connectivity between your instance and the www. I can’t say anymore without further info. I’m sorry I couldn’t be of more help. I’m happy to help out more though, from one instance owner to another, drop me a message over at https://iam.dru5k1.com/mastodon. You know the saying two heads are better than one.

      • lemmy_newbieOP
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        Thank you for the help so far! Another user successfully sent me a message so it seems some things are working. I do have federation enabled in settings (see image)

        Do I need to specifically list all the instances I want to federate with? That seems counter intuitive as by default I’d like to federate with all instances and just block the ones that give me a headache.

        My lemmy.hjson looks like this:

        {
          database: {
            password: "NOT REAL PASSWD"
          }
          # replace with your domain
          hostname: discuss.petersanchez.com
          bind: "127.0.0.1"
          federation: {
            enabled: true
          }
          # remove this block if you don't require image hosting
          pictrs: {
            url: "http://localhost:8080/"
          }
        }
        

        I don’t even know what other options are available for this file. I can’t seem to find any docs for it. Let me know if something should be added/removed/changed.

        DNS seems OK on the system when trying to resolve all the given communities from a server shell. I know ActivityPub isn’t for the faint of heart but in my toying around with it in the past, it was straight forward enough, just different software use different fields for different purposes. So that can create confusion in federation but in this case I’m dealing with Lemmy to Lemmy so I assume it should work heh.

        Oh and the 1 community I was able to actually find is on kbin.social and it says subscribe pending:

        Thanks again for the help so far!

    • lemmy_newbieOP
      link
      fedilink
      English
      arrow-up
      2
      ·
      1 year ago

      Thanks for the answer! I have registration closed (it’s just a personal instance).

      I was able to get the kbin subscription to status pending. Trying the lemmy.ca community gets this in the logs

      2023-07-03T19:17:43.357158Z  WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: couldnt_find_community: error decoding response body: missing field `properties` at line 1 column 166
         0: lemmy_apub::fetcher::resolve_actor_identifier
                   at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lemmy_apub-0.17.4/src/fetcher/mod.rs:16
         1: lemmy_apub::api::read_community::perform
                 with self=GetCommunity { id: None, name: Some("nostupidquestions@lemmy.ca"), auth: Some(Sensitive) }
                   at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lemmy_apub-0.17.4/src/api/read_community.rs:30
         2: lemmy_server::root_span_builder::HTTP request
                 with http.method=GET http.scheme="http" http.host=discuss.petersanchez.com http.target=/api/v3/community otel.kind="server" request_id=8cef1a93-33c3-4197-89b9-4d709b0e9b10 http.status_code=400 otel.status_code="OK"
                   at /home/lemmy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lemmy_server-0.17.4/src/root_span_builder.rs:16
      LemmyError { message: Some("couldnt_find_community"), inner: error decoding response body: missing field `properties` at line 1 column 166
      
      Caused by:
          missing field `properties` at line 1 column 166, context: "SpanTrace" }
      

      I think I’ll try updating to latest in main and see what happens.

  • olizet@lemmy.works
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    So if you search for !lemmy_support@lemmy.ml and wait about 1-2 minutes, nothing happens? Because that was the way to connect my own instance. I still cannot subscribe (‘subscription pending’), but I assume that’s a software bug.

  • lemmy_newbieOP
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    Well all. Thanks for the help. I manually built 0.18.0 and so far, it seems to be working as expected. I can find communities and subscribe to them OK, etc. Thanks for the help!