Hi everyone,

My router went from IPv4 to IPv6 after an update from my ISP back in April, and so I decided to try and get my selfhosted Raspberry Pi server to work with it. It’s been less trivial than I hoped it would be, though. It worked and was reachable when it still used IPv4, but it’s been out of the air since April.

I’m running Arch Linux ARM on the device and use networkd to connect it to the internet. I use https://now-dns.com to get a dynamic DNS and have connected it to my server using their Linux script.

This is my Caddyfile:

{
	debug
	
}

# Jellyfin:
https://myserver.now-dns.net:26347,
https://myserver.now-dns.net:443,
[(my IPv6 address here)]:26347 {
	header / {
		# Enable cross-site filter (XSS) 
		# and tell browser to block detected attacks    
		X-Frame-Options "Deny"
		Content-Security-Policy "
	            default-src 'self' data: blob:;
	            style-src 'self' 'unsafe-inline' bootstrapcdn.com *.bootstrapcdn.com https://ctalvio.github.io/Monochromic/default_style.css https://ctalvio.github.io/Monochromic/jfblue_style.css https://ctalvio.github.io/Monochromic/jfpurple_style.css https://ctalvio.github.io/Monochromic/bottom-progress_style.css https://ctalvio.github.io/Monochromic/customcolor-advanced_style.css https://ctalvio.github.io/Monochromic/improve-performance_style.css https://fonts.googleapis.com/css2;
	            script-src 'self' 'unsafe-inline' bootstrapcdn.com *.bootstrapcdn.com googleapis.com *.googleapis.com https://www.gstatic.com/cv/js/sender/v1/cast_sender.js worker-src 'self' blob:;
	            font-src 'self' bootstrapcdn.com *.bootstrapcdn.com;
	            img-src data: 'self' imgur.com *.imgur.com;
	            form-action 'self';
	            connect-src 'self' pokeapi.co;
	            frame-ancestors 'self';
	            report-uri {$CSP_REPORT_URI}
	        "
	}
	reverse_proxy 127.0.0.1:8093
	#reverse_proxy localhost:8093
}

# Nextcloud:
#https://192.168.1.96:65002,
https://myserver.now-dns.net:65001 {
	root * /usr/share/webapps/nextcloud
	file_server
	#        log {
	#                output file     /var/log/caddy/myserver.now-dns.net.log
	#                format single_field common_log
	#        }

	#php_fastcgi 127.0.0.1:9000
	#php_fastcgi unix//run/php-fpm/php-fpm.sock # veranderd naar correcte adres uit /etc/php/php-fpm.d/www.conf
	php_fastcgi unix//run/nextcloud/nextcloud.sock # veranderd naar nieuwe correcte adres uit /etc/php/php-fpm.d/nextcloud.conf

	header {
		# enable HSTS
		Strict-Transport-Security max-age=31536000;
	}

	redir /.well-known/carddav /remote.php/dav 301
	redir /.well-known/caldav /remote.php/dav 301

	# .htaccess / data / config / ... shouldn't be accessible from outside
	@forbidden {
		path /.htaccess
		path /data/*
		path /config/*
		path /db_structure
		path /.xml
		path /README
		path /3rdparty/*
		path /lib/*
		path /templates/*
		path /occ
		path /console.php
	}

	respond @forbidden 404
}

(myserver.now-dns.net is not actually my server name, I changed it to stay a bit more anonymous. Maybe this is unnecessarily cautious, let me know if I should change this to my actual address to aid your help.)

This is a journalctl log from fresh after a Caddy restart:

Aug 01 14:36:12 baspi2 systemd[1]: Starting Caddy web server...
Aug 01 14:36:13 baspi2 caddy[23895]: {"level":"info","ts":1690893373.0834036,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Aug 01 14:36:13 baspi2 caddy[23895]: {"level":"warn","ts":1690893373.0915132,"msg":"No files matching import glob pattern","pattern":"/etc/caddy/conf.d/*"}
Aug 01 14:36:13 baspi2 caddy[23895]: {"level":"info","ts":1690893373.1047359,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4394a00"}
Aug 01 14:36:13 baspi2 caddy[23895]: {"level":"info","ts":1690893373.1278725,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Aug 01 14:36:13 baspi2 caddy[23895]: {"level":"info","ts":1690893373.1279871,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv1","https_port":443}
Aug 01 14:36:13 baspi2 caddy[23895]: {"level":"info","ts":1690893373.1280322,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv1"}
Aug 01 14:36:13 baspi2 caddy[23895]: {"level":"info","ts":1690893373.128112,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv2"}
Aug 01 14:36:13 baspi2 caddy[23895]: {"level":"info","ts":1690893373.1328619,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x4394a00"}
Aug 01 14:36:13 baspi2 caddy[23895]: Valid configuration
Aug 01 14:36:13 baspi2 caddy[23904]: caddy.HomeDir=/var/lib/caddy
Aug 01 14:36:13 baspi2 caddy[23904]: caddy.AppDataDir=/var/lib/caddy
Aug 01 14:36:13 baspi2 caddy[23904]: caddy.AppConfigDir=/etc/caddy
Aug 01 14:36:13 baspi2 caddy[23904]: caddy.ConfigAutosavePath=/var/lib/caddy/autosave.json
Aug 01 14:36:13 baspi2 caddy[23904]: caddy.Version=v2.6.4
Aug 01 14:36:13 baspi2 caddy[23904]: runtime.GOOS=linux
Aug 01 14:36:13 baspi2 caddy[23904]: runtime.GOARCH=arm
Aug 01 14:36:13 baspi2 caddy[23904]: runtime.Compiler=gc
Aug 01 14:36:13 baspi2 caddy[23904]: runtime.NumCPU=4
Aug 01 14:36:13 baspi2 caddy[23904]: runtime.GOMAXPROCS=4
Aug 01 14:36:13 baspi2 caddy[23904]: runtime.Version=go1.20.1
Aug 01 14:36:13 baspi2 caddy[23904]: os.Getwd=/
Aug 01 14:36:13 baspi2 caddy[23904]: LANG=C
Aug 01 14:36:13 baspi2 caddy[23904]: PATH=/usr/local/sbin:/usr/local/bin:/usr/bin
Aug 01 14:36:13 baspi2 caddy[23904]: NOTIFY_SOCKET=/run/systemd/notify
Aug 01 14:36:13 baspi2 caddy[23904]: HOME=/var/lib/caddy
Aug 01 14:36:13 baspi2 caddy[23904]: LOGNAME=caddy
Aug 01 14:36:13 baspi2 caddy[23904]: USER=caddy
Aug 01 14:36:13 baspi2 caddy[23904]: INVOCATION_ID=131202f1b6e3472bab7e6fc48933c731
Aug 01 14:36:13 baspi2 caddy[23904]: JOURNAL_STREAM=8:2593614
Aug 01 14:36:13 baspi2 caddy[23904]: SYSTEMD_EXEC_PID=23904
Aug 01 14:36:13 baspi2 caddy[23904]: XDG_DATA_HOME=/var/lib
Aug 01 14:36:13 baspi2 caddy[23904]: XDG_CONFIG_HOME=/etc
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.4270308,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"warn","ts":1690893373.4276912,"msg":"No files matching import glob pattern","pattern":"/etc/caddy/conf.d/*"}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.4616253,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.4650905,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4e32000"}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.4871185,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.4872386,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv1","https_port":443}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.4872835,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv1"}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.4874046,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv2"}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.9077604,"logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.9084256,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.909473,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy"}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"debug","ts":1690893373.9139633,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.9140959,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.9144514,"logger":"http","msg":"enabling HTTP/3 listener","addr":":65001"}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"debug","ts":1690893373.91526,"logger":"http","msg":"starting server loop","address":"[::]:65001","tls":true,"http3":true}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.9154122,"logger":"http.log","msg":"server running","name":"srv2","protocols":["h1","h2","h3"]}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"debug","ts":1690893373.9156892,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.9158008,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.9160817,"logger":"http","msg":"enabling HTTP/3 listener","addr":":26347"}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"debug","ts":1690893373.9165256,"logger":"http","msg":"starting server loop","address":"[::]:26347","tls":true,"http3":true}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.9165914,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.916624,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["(my IPv6 address here)","myserver.now-dns.net"]}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"info","ts":1690893373.917206,"logger":"tls","msg":"finished cleaning storage units"}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"warn","ts":1690893373.920347,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [(my IPv6 address here)]: no OCSP server specified in certificate","identifiers":["(my IPv6 address here)"]}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"debug","ts":1690893373.920421,"logger":"tls.cache","msg":"added certificate to cache","subjects":["(my IPv6 address here)"],"expiration":1690917213,"managed":true,"issuer_key":"local","hash":"8aa98ab4d6a397ee8784859f4ba69d8df96d6d978247a3436a20cc8373cf9a8a","cache_size":1,"cache_capacity":10000}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"debug","ts":1690893373.920493,"logger":"events","msg":"event","name":"cached_managed_cert","id":"2420e703-5823-4962-ad5b-05a084aafacb","origin":"tls","data":{"sans":["(my IPv6 address here)"]}}
Aug 01 14:36:13 baspi2 caddy[23904]: {"level":"debug","ts":1690893373.923109,"logger":"tls","msg":"loading managed certificate","domain":"myserver.now-dns.net","expiration":1697974414,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/var/lib/caddy"}
Aug 01 14:36:14 baspi2 caddy[23904]: {"level":"debug","ts":1690893374.1269143,"logger":"tls.cache","msg":"added certificate to cache","subjects":["myserver.now-dns.net"],"expiration":1697974414,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"7db3c32211ccb2942c5d329650e92ddd63cd9a17670eba2ce29476f3c3e3a741","cache_size":2,"cache_capacity":10000}
Aug 01 14:36:14 baspi2 caddy[23904]: {"level":"debug","ts":1690893374.1271243,"logger":"events","msg":"event","name":"cached_managed_cert","id":"fc000be0-ac06-4ca2-aa53-c14c6fb3ae27","origin":"tls","data":{"sans":["myserver.now-dns.net"]}}
Aug 01 14:36:14 baspi2 caddy[23904]: {"level":"info","ts":1690893374.1345215,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/autosave.json"}
Aug 01 14:36:14 baspi2 systemd[1]: Started Caddy web server.
Aug 01 14:36:14 baspi2 caddy[23904]: {"level":"info","ts":1690893374.137206,"msg":"serving initial configuration"}
Aug 01 14:36:14 baspi2 sudo[23887]: pam_unix(sudo:session): session closed for user root
  • I “opened” the necessary ports on my router (my router calls it that, I realise it’s probably more like I unblocked these ports in its IPv6 firewall)
  • I scanned the open ports with an IPv6 port scanner (this one) and it shows ports 80 and 443 to be open, as well as ports 65001 and 26347
  • I used my public IPv6 address to scan, as well as the myserver.now-dns.net address (which is actually another server name)
  • I used a smart phone unconnected to wifi to test, but the site cannot load there, either. I tested from other networks while away from home, too, which also didn’t work.

Unfortunately, I still cannot connect to the server with a browser. https://myserver.now-dns.net:26347/ gives an ¨Unable to connect" error on Firefox.

I have been whittling away at this issue on and off since April and haven’t really made any big breakthroughs. What would be your first steps in troubleshooting this issue?

When I scan one of the open ports with an online tool, a message like this pops up in the journalctl log:

Aug 01 14:45:49 baspi2 caddy[23904]: {"level":"debug","ts":1690893949.6947021,"logger":"http.stdlib","msg":"http: TLS handshake error from [2a01:4f8:1c1c:2d4e::1]:50079: EOF"}

  • gerdesj
    link
    fedilink
    English
    arrow-up
    5
    ·
    1 year ago

    There is a lot going on there so I suggest divide and conquer. Shut down Caddy and configure something like nginx or apache with a simple static index.html page with a single word in it. Does that work? Start really simple, so open port 80 as well as 443 and add ssl only once you have proven http works OK.

      • gerdesj
        link
        fedilink
        English
        arrow-up
        2
        ·
        1 year ago

        Best of luck. Let us know how you get on

        • toothpaste_sandwich@feddit.nlOP
          link
          fedilink
          arrow-up
          1
          ·
          edit-2
          1 year ago

          Well, I set up a basic nginx server and disabled Caddy. The nginx server only serves http for now, not https.

          I used the basic nginx.conf and added my IPv6 address like so:

          #user http;
          worker_processes  1;
          
          #error_log  logs/error.log;
          #error_log  logs/error.log  notice;
          #error_log  logs/error.log  info;
          
          #pid        logs/nginx.pid;
          
          
          events {
              worker_connections  1024;
          }
          
          
          http {
              include       mime.types;
              default_type  application/octet-stream;
          
              #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
              #                  '$status $body_bytes_sent "$http_referer" '
              #                  '"$http_user_agent" "$http_x_forwarded_for"';
          
              #access_log  logs/access.log  main;
          
              sendfile        on;
              #tcp_nopush     on;
          
              #keepalive_timeout  0;
              keepalive_timeout  65;
          
              #gzip  on;
          
              server {
                  listen       80;
                  listen [::]:80;
                  server_name  localhost;
          
                  #charset koi8-r;
          
                  #access_log  logs/host.access.log  main;
          
                  location / {
                      root   /usr/share/nginx/html;
                      index  index.html index.htm;
                  }
          
                  #error_page  404              /404.html;
          
                  # redirect server error pages to the static page /50x.html
                  #
                  error_page   500 502 503 504  /50x.html;
                  location = /50x.html {
                      root   /usr/share/nginx/html;
                  }
          
                  # proxy the PHP scripts to Apache listening on 127.0.0.1:80
                  #
                  #location ~ \.php$ {
                  #    proxy_pass   http://127.0.0.1;
                  #}
          
                  # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
                  #
                  #location ~ \.php$ {
                  #    root           html;
                  #    fastcgi_pass   127.0.0.1:9000;
                  #    fastcgi_index  index.php;
                  #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
                  #    include        fastcgi_params;
                  #}
          
                  # deny access to .htaccess files, if Apache's document root
                  # concurs with nginx's one
                  #
                  #location ~ /\.ht {
                  #    deny  all;
                  #}
              }
          
          
              # another virtual host using mix of IP-, name-, and port-based configuration
              #
              #server {
              #    listen       8000;
              #    listen       somename:8080;
              #    server_name  somename  alias  another.alias;
          
              #    location / {
              #        root   html;
              #        index  index.html index.htm;
              #    }
              #}
          
          
              # HTTPS server
              #
              #server {
              #    listen       443 ssl;
              #    server_name  localhost;
          
              #    ssl_certificate      cert.pem;
              #    ssl_certificate_key  cert.key;
          
              #    ssl_session_cache    shared:SSL:1m;
              #    ssl_session_timeout  5m;
          
              #    ssl_ciphers  HIGH:!aNULL:!MD5;
              #    ssl_prefer_server_ciphers  on;
          
              #    location / {
              #        root   html;
              #        index  index.html index.htm;
              #    }
              #}
          
          }
          

          I can reach the webpage through the local IP (v4) address, but not online using the IPv6 address. Testing with a port checking tool does show that port 80 is open. I tried testing with my smartphone with wifi disconnected, too, but still no success… Any ideas on what I could try? I’m searching for tutorials for setting up an IPv6 nginx server but so far I’m not seeing a big difference with what I’m doing.

          Yes, using now-dns.com is different from paying for a host name, but in theory entering the plain IPv6 address in square brackets in Firefox should also work, right?

          • gerdesj
            link
            fedilink
            English
            arrow-up
            1
            ·
            1 year ago

            I can see that you have bound nginx to port 80 on both IPv4 and 6 - the two Listen directives.

            • Can you get to it locally via IPv6 as well as IPv4?
            • Can you get to it via IPv4 externally?

            Let’s get right down to basics:

            I’m on GMT+1/BST/UTC+1 so its a bit late now. I’ll pick up tomorrow pm

            • toothpaste_sandwich@feddit.nlOP
              link
              fedilink
              arrow-up
              1
              ·
              edit-2
              1 year ago

              Thanks for all the pointers! Let’s see, I’ll take this one by one.

              • Can you get to it locally via IPv6 as well as IPv4?

              Well, turns out reaching a link-local address with a browser is not really easy to do, but tried with SSH port forwarding and that seems to work, at least…

              (I used this command on my PC:

              ssh -N -L '8082:127.0.0.1:80' fe80::dea6:32ff:fe54:67fb%eno1

              where fe80::dea6:32ff:fe54:67fb%eno1 is the link-local address of my server. Then I browsed to 127.0.0.1:8082 on my PC.)

              • Can you get to it via IPv4 externally?

              I hadn’t thought of testing this! Yes, I can. I also tested by navigating to the IPv4 address with my phone on data (so without wifi)

              • ping -6 google.com[1] - from the web server, does it work?

              It does, yes.

              • ping -6 google.com[2] - from your PC/laptop.phone, does it work?

              Likewise, yes, this works.

              This works, at least from my PC. I tried to reach it from the server using w3m for the heck of it but without Javascript that didn’t work. Alas.

              I really appreciate your help, I hope we can get to the bottom of this. Otherwise I think I’ll just revert to IPv4, as that will probably still work. But I can’t stand IPv6 not working!

              • gerdesj
                link
                fedilink
                English
                arrow-up
                2
                ·
                1 year ago

                As well as a link local address you should also have one or more globally routeable ones too. Hopefully you have at least one of those set up in DNS with a AAAA address. Therefore you should be able to put the address of your web server into your browser and off it goes. In theory IPv6 should be preferred by your browser, so even if both an A record and a AAAA record resolve for the name, IPv6 should kick in.

                A quick check would be:

                $ host mywebserver.example.co.uk
                

                That should return an IPv4 and an IPv6 address. The IPv6 address is the same for internal and external - there is no distinction, which can be surprising if you are used to IPv4 and NAT. The final bit of the equation is that your internet router needs to allow access “from all to globally routeable ipv6 address of the web server”.

                • toothpaste_sandwich@feddit.nlOP
                  link
                  fedilink
                  arrow-up
                  1
                  ·
                  edit-2
                  1 year ago

                  Hopefully you have at least one of those set up in DNS with a AAAA address.

                  I suspect that this is not the case, but also I’m not sure how I would set this up. Is that something I should configure on my internet router? This is what the DNS settings there look like at the moment:

                  A quick check would be:

                  $ host mywebserver.example.co.uk

                  Well, that gives me this:

                  host myserver.now-dns.net 
                  myserver.now-dns.net has address 192.168.1.96
                  myserver.now-dns.net has IPv6 address (my global IPv6 address here)
                  myserver.now-dns.net mail is handled by 1 myserver.now-dns.net.
                  

                  Entering my IPv6 address between square brackets in the browser still doesn’t load, though.

                  The final bit of the equation is that your internet router needs to allow access “from all to globally routeable ipv6 address of the web server”.

                  Is that the same as setting a DMZ for IPv6 to the web server? That’s an option I could find in the router settings, though enabling it didn’t seem to make any difference…

              • toothpaste_sandwich@feddit.nlOP
                link
                fedilink
                arrow-up
                1
                ·
                1 year ago

                By the way—don’t know how relevant this is, but there’s two ways for “port forwarding” on my router for IPv6:

                I can either use the MAC address of my server or use the IPv6 address.

                When I use the MAC address, scanning the opened port 80 works with online port scanning tools, but when I use the link-local address, the port appears closed. Not sure if that means anything, but I figured more information can’t hurt.

  • lemmylukas@lemmy.n8h.eu
    link
    fedilink
    arrow-up
    2
    ·
    1 year ago

    I also had problems for a long time after changing my provider to make my services available again. After a lot of back and forth, I found out that I now have a DS-lite connection (this is now the case with most providers in Germany). Do you know if you have a real ipv4 address or a DS Lite connection?