I’ve noticed that whenever I post from my instance (midwest.social) to lemmy.ml that the upvotes don’t match that of what’s on lemmy.ml and the comments don’t show up for a while.

  • @nutomicA
    4•2 years ago

    It could be this issue, but hard to say. Do you notice any errors in your Lemmy logs?

    • seahorseOP
      3•2 years ago

      Sorry, which command do I need to run to get those?

      • @nutomicA
        4•2 years ago

        sudo docker-compose logs -f --tail=100 lemmy

        You might want to change the log level for Lemmy to warn or error. I also checked the logs on our side just now, but there were no errors for midwest.social.

          • @nutomicA
            4•2 years ago

            That one is probably harmless, it just means that the activity was already received and processed before. It is strange though that Lemmy on your instance would send the same activity twice.

            Earlier I changed lemmy.ml logs to warn in docker-compose.yml and ran docker-compose up -d. Since then it has collected some better logs:

            $ sudo docker-compose logs lemmy | grep midwest.social

|2022-01-12T22:17:30.845533ZWARNlemmy_apub_lib::activity_queue:Sendhttps://lemmy.ml/activities/announce/c3c3ff96-a8c4-453d-96c1-e02a7e8dcc41tohttps://midwest.social/inboxfailedwithstatus400BadRequest:error:0909006C:PEMroutines:get_name:nostartline:crypto/pem/pem_lib.c:745:Expecting:PUBLICKEY
|2022-01-12T22:18:53.522306ZWARNlemmy_apub_lib::activity_queue:Sendhttps://lemmy.ml/activities/announce/a1c956e6-9b5b-4df0-a245-cefae75a39fbtohttps://midwest.social/inboxfailedwithstatus400BadRequest:error:0909006C:PEMroutines:get_name:nostartline:crypto/pem/pem_lib.c:745:Expecting:PUBLICKEY
|2022-01-12T22:23:41.824212ZWARNlemmy_apub_lib::activity_queue:Sendhttps://lemmy.ml/activities/announce/0850d562-72c7-4df7-aa70-b0a4817e7564tohttps://midwest.social/inboxfailedwithstatus400BadRequest:Headerisexpired
|2022-01-12T22:27:16.411288ZWARNlemmy_apub_lib::activity_queue:Sendhttps://lemmy.ml/activities/announce/a98ef44b-49c2-447a-b385-f0c90d7b9287tohttps://midwest.social/inboxfailedwithstatus504GatewayTimeout:<html>
|2022-01-12T22:28:21.648136ZWARNlemmy_apub_lib::activity_queue:Sendhttps://lemmy.ml/activities/announce/ed0aa97b-2b6a-4e81-9941-bd736b2178e1tohttps://midwest.social/inboxfailedwithstatus408RequestTimeout:
|2022-01-12T22:28:43.533383ZWARNlemmy_apub_lib::activity_queue:Sendhttps://lemmy.ml/activities/announce/136c948f-fdc7-4a4e-949d-cdbc6bb21e21tohttps://midwest.social/inboxfailedwithstatus408RequestTimeout:
|2022-01-12T22:28:45.404798ZWARNlemmy_apub_lib::activity_queue:Sendhttps://lemmy.ml/activities/announce/626542e4-23b8-4b8e-981f-b5b14a821ac0tohttps://midwest.social/inboxfailedwithstatus408RequestTimeout:
|2022-01-12T22:29:09.170348ZWARNlemmy_apub_lib::activity_queue:Sendhttps://lemmy.ml/activities/announce/36b21997-8f4d-43e5-8d8b-9491bb9d6d2ftohttps://midwest.social/inboxfailedwithstatus400BadRequest:Headerisexpired

            I suggest you follow the same steps for logging (with grep lemmy.ml), and see more useful log entries. In this case it looks like midwest.social is sometimes unreachable. In theory Lemmy will retry a few times, but that isnt really tested. And there are some weird public key errors, which should have more information in your logs.

        • seahorseOP
          2•2 years ago
          Attaching to lemmy_lemmy_1
lemmy_1     | 2022-01-12T21:14:42.074410Z  INFO lemmy_websocket::handlers: 93.43.251.95 joined
lemmy_1     | 2022-01-12T21:14:44.929992Z  INFO lemmy_server::scheduled_tasks: Done.
lemmy_1     | 2022-01-12T21:14:44.930232Z  INFO lemmy_server::scheduled_tasks: Reindexing table concurrently comment_aggregates ...
lemmy_1     | 2022-01-12T21:14:46.286495Z  INFO lemmy_server::scheduled_tasks: Done.
lemmy_1     | 2022-01-12T21:14:46.286744Z  INFO lemmy_server::scheduled_tasks: Reindexing table concurrently community_aggregates ...
lemmy_1     | 2022-01-12T21:14:46.305327Z  INFO lemmy_server::scheduled_tasks: Done.
lemmy_1     | 2022-01-12T21:14:46.306563Z  INFO lemmy_server::scheduled_tasks: Clearing old activities...
lemmy_1     | 2022-01-12T21:14:47.128085Z  INFO lemmy_server::scheduled_tasks: Done.
lemmy_1     | 2022-01-12T21:14:50.284418Z  INFO lemmy_websocket::handlers: 143.59.248.133 joined
lemmy_1     | 2022-01-12T21:14:56.247886Z  INFO Websocket Request{trace_id=00000000000000000000000000000000}:perform{self=CreateCommentLike { comment_id: CommentId(26276), score: 1, auth: Sensitive }}:send:send_activity_in_community:send_lemmy_activity: lemmy_apub::activities: Sending activity https://midwest.social/activities/like/4a8855b6-3d46-4a8e-8be3-edff0d15001c
lemmy_1     | 2022-01-12T21:14:56.253978Z  INFO lemmy_apub_lib::activity_queue: Sending https://midwest.social/activities/like/4a8855b6-3d46-4a8e-8be3-edff0d15001c to https://lemmy.ml/inbox
lemmy_1     | 2022-01-12T21:16:01.872202Z  INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=dc91242d-6bf2-4478-b518-a2a5ee6f5873}:shared_inbox: lemmy_apub::http: Received shared inbox activity {"@context":["https://www.w3.org/ns/activitystreams",{"stickied":"as:stickied","pt":"https://join-lemmy.org#","sc":"http://schema.org#","matrixUserId":{"type":"sc:Text","id":"as:alsoKnownAs"},"sensitive":"as:sensitive","comments_enabled":{"type":"sc:Boolean","id":"pt:commentsEnabled"},"moderators":"as:moderators"},"https://w3id.org/security/v1"],"actor":"https://lemmygrad.ml/c/leftsthetics","to":["https://www.w3.org/ns/activitystreams#Public"],"object":{"actor":"https://lemmygrad.ml/u/nour","to":["https://www.w3.org/ns/activitystreams#Public"],"object":"https://lemmygrad.ml/comment/100510","cc":["https://lemmygrad.ml/c/leftsthetics"],"type":"Like","id":"https://lemmygrad.ml/activities/like/3bfcb836-e1c0-488b-9411-784c1db45095"},"cc":["https://lemmygrad.ml/c/leftsthetics/followers"],"type":"Announce","id":"https://lemmygrad.ml/activities/announce/640ff365-07c2-4eed-8e13-8790178cb880"}
lemmy_1     | 2022-01-12T21:16:01.882757Z  INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=dc91242d-6bf2-4478-b518-a2a5ee6f5873}:shared_inbox:receive_activity: lemmy_apub::http: Verifying activity https://lemmygrad.ml/activities/announce/640ff365-07c2-4eed-8e13-8790178cb880
lemmy_1     | 2022-01-12T21:16:01.899481Z  INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=dc91242d-6bf2-4478-b518-a2a5ee6f5873}:shared_inbox:receive_activity: lemmy_apub::http: Receiving activity https://lemmygrad.ml/activities/announce/640ff365-07c2-4eed-8e13-8790178cb880
lemmy_1     | 2022-01-12T21:16:03.079024Z  INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=5c276d65-900c-4cf1-9ec3-f21b7e374e94}:shared_inbox: lemmy_apub::http: Received shared inbox activity {"@context":["https://www.w3.org/ns/activitystreams",{"stickied":"as:stickied","pt":"https://join-lemmy.org#","sc":"http://schema.org#","matrixUserId":{"type":"sc:Text","id":"as:alsoKnownAs"},"sensitive":"as:sensitive","comments_enabled":{"type":"sc:Boolean","id":"pt:commentsEnabled"},"moderators":"as:moderators"},"https://w3id.org/security/v1"],"actor":"https://lemmygrad.ml/c/communism","to":["https://www.w3.org/ns/activitystreams#Public"],"object":{"actor":"https://lemmygrad.ml/u/Rylasasin","to":["https://www.w3.org/ns/activitystreams#Public"],"object":"https://lemmygrad.ml/comment/97958","cc":["https://lemmygrad.ml/c/communism"],"type":"Dislike","id":"https://lemmygrad.ml/activities/dislike/323009e1-31f9-49db-8131-61545e615742"},"cc":["https://lemmygrad.ml/c/communism/followers"],"type":"Announce","id":"https://lemmygrad.ml/activities/announce/c919f208-a23a-47ae-a393-cd3fa6cdb88b"}
lemmy_1     | 2022-01-12T21:16:03.084235Z  INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=5c276d65-900c-4cf1-9ec3-f21b7e374e94}:shared_inbox:receive_activity: lemmy_apub::http: Verifying activity https://lemmygrad.ml/activities/announce/c919f208-a23a-47ae-a393-cd3fa6cdb88b
lemmy_1     | 2022-01-12T21:16:03.090818Z  INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=5c276d65-900c-4cf1-9ec3-f21b7e374e94}:shared_inbox:receive_activity:verify:verify:verify_person_in_community: lemmy_apub_lib::object_id: Fetching remote object https://lemmygrad.ml/u/Rylasasin
lemmy_1     | 2022-01-12T21:16:03.205027Z  INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=5c276d65-900c-4cf1-9ec3-f21b7e374e94}:shared_inbox:receive_activity: lemmy_apub::http: Receiving activity https://lemmygrad.ml/activities/announce/c919f208-a23a-47ae-a393-cd3fa6cdb88b
lemmy_1     | 2022-01-12T21:16:04.732109Z  INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=48268bff-7a4d-4fb4-8a6d-fd9ad5b36278}:shared_inbox: lemmy_apub::http: Received shared inbox activity {"@context":["https://www.w3.org/ns/activitystreams",{"stickied":"as:stickied","pt":"https://join-lemmy.org#","sc":"http://schema.org#","matrixUserId":{"type":"sc:Text","id":"as:alsoKnownAs"},"sensitive":"as:sensitive","comments_enabled":{"type":"sc:Boolean","id":"pt:commentsEnabled"},"moderators":"as:moderators"},"https://w3id.org/security/v1"],"actor":"https://lemmygrad.ml/c/communism","to":["https://www.w3.org/ns/activitystreams#Public"],"object":{"actor":"https://lemmygrad.ml/u/Rylasasin","to":["https://www.w3.org/ns/activitystreams#Public"],"object":"https://lemmygrad.ml/comment/97934","cc":["https://lemmygrad.ml/c/communism"],"type":"Dislike","id":"https://lemmygrad.ml/activities/dislike/a50ea1fe-31c5-4536-833d-af91172eb2df"},"cc":["https://lemmygrad.ml/c/communism/followers"],"type":"Announce","id":"https://lemmygrad.ml/activities/announce/297a68c6-102d-46da-b0be-8174fe42c57a"}
lemmy_1     | 2022-01-12T21:16:04.736151Z  INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=48268bff-7a4d-4fb4-8a6d-fd9ad5b36278}:shared_inbox:receive_activity: lemmy_apub::http: Verifying activity https://lemmygrad.ml/activities/announce/297a68c6-102d-46da-b0be-8174fe42c57a
lemmy_1     | 2022-01-12T21:16:04.743282Z  INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=48268bff-7a4d-4fb4-8a6d-fd9ad5b36278}:shared_inbox:receive_activity: lemmy_apub::http: Receiving activity https://lemmygrad.ml/activities/announce/297a68c6-102d-46da-b0be-8174fe42c57a
lemmy_1     | 2022-01-12T21:19:19.390390Z  INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=ca0ed8b2-2e7c-443f-a624-df2df6015b67}:shared_inbox: lemmy_apub::http: Received shared inbox activity {"@context":["https://www.w3.org/ns/activitystreams",{"stickied":"as:stickied","pt":"https://join-lemmy.org#","sc":"http://schema.org#","matrixUserId":{"type":"sc:Text","id":"as:alsoKnownAs"},"sensitive":"as:sensitive","comments_enabled":{"type":"sc:Boolean","id":"pt:commentsEnabled"},"moderators":"as:moderators"},"https://w3id.org/security/v1"],"actor":"https://lemmygrad.ml/c/communism","to":["https://www.w3.org/ns/activitystreams#Public"],"object":{"actor":"https://lemmygrad.ml/u/Rylasasin","to":["https://www.w3.org/ns/activitystreams#Public"],"object":"https://lemmygrad.ml/post/83464","cc":["https://lemmygrad.ml/c/communism"],"type":"Like","id":"https://lemmygrad.ml/activities/like/16204a7e-1883-49bd-9094-e755d6b4d911"},"cc":["https://lemmygrad.ml/c/communism/followers"],"type":"Announce","id":"https://lemmygrad.ml/activities/announce/231157fc-98d9-49bd-bc8a-906e67076d6d"}
lemmy_1     | 2022-01-12T21:19:19.395819Z  INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=ca0ed8b2-2e7c-443f-a624-df2df6015b67}:shared_inbox:receive_activity: lemmy_apub::http: Verifying activity https://lemmygrad.ml/activities/announce/231157fc-98d9-49bd-bc8a-906e67076d6d
lemmy_1     | 2022-01-12T21:19:19.404319Z  INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=ca0ed8b2-2e7c-443f-a624-df2df6015b67}:shared_inbox:receive_activity: lemmy_apub::http: Receiving activity https://lemmygrad.ml/activities/announce/231157fc-98d9-49bd-bc8a-906e67076d6d
lemmy_1     | 2022-01-12T21:19:20.534398Z  INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=b0fc1656-571e-4bd4-80f4-c2088934dab0}:shared_inbox: lemmy_apub::http: Received shared inbox activity {"@context":["https://www.w3.org/ns/activitystreams",{"stickied":"as:stickied","pt":"https://join-lemmy.org#","sc":"http://schema.org#","matrixUserId":{"type":"sc:Text","id":"as:alsoKnownAs"},"sensitive":"as:sensitive","comments_enabled":{"type":"sc:Boolean","id":"pt:commentsEnabled"},"moderators":"as:moderators"},"https://w3id.org/security/v1"],"actor