Hello all, I’m taking my first steps in the realm of self-hosting and am learning as I go. I have a VM running ubuntu and I got it connected to tailscale network to fend off unwanted visitors. I also have discovered Docker and am using it to deploy two web applications: FreshRSS and Podfetch. I can deploy them through Docker and they both have their own ports which I can access through ipadrress:portnumber URL in my webbrowser. But, the connection is unsecured over HTTP. I’d like to take it a step further in order to make the connections go over HTTPS.

I thought to use Caddy to make a reverse proxy as it is supposed to have good support with Tailscale but I’m not being particularly successful. I can connect to the individual applications (FreshRSS, PodFetch) by using the given tailscale DNS name (machine.domain.ts.net) and port directly in the browsers URL, but going to the machine.domain.ts.net does only yield in a connection error.

I’ve attached the stdout from running Caddy, my spidersense is telling it is something to do with getting a cert from letsencrypt. Over at tailscale admin, I’ve ensured I have a tailnet name, MagicDNS and HTTPS certificates enabled.

Here’s some relevant information, Caddy log file is at the end.

Thanks in advance

EDIT: solution to my problem at the end of this post.

sudo docker ps

CONTAINER ID   IMAGE                         COMMAND                  CREATED          STATUS          PORTS                                                                                         NAMES                                                                                                                 

86a72dbd2686   samuel19982/podfetch:latest   "./podfetch"             20 minutes ago   Up 18 minutes   0.0.0.0:8480->8000/tcp, :::8480->8000/tcp                                                     podfetch_podfetch_1                                                                                                   

a7dae64308f9   caddy:latest                  "caddy run --config …"   25 hours ago     Up 17 seconds   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 443/udp, 2019/tcp   caddy                                                                                                                 

141bbf69ad62   freshrss/freshrss             "./Docker/entrypoint…"   2 months ago     Up 2 months     0.0.0.0:8080->80/tcp, :::8080->80/tcp                                                         freshrss

Current Caddyfile:

machine.domain.ts.net

respond "hello"
file_server

docker-compose.yml for Caddy

version: "3"

services:
  caddy:
    image: caddy:latest
    container_name: caddy
    restart: always
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /home/ubuntu/caddy/caddy_data:/data
      - /home/ubuntu/caddy/caddy_config:/config
      - /home/ubuntu/caddy/Caddyfile:/etc/caddy/Caddyfile

log output from running sudo docker-compose up in the directory where docker-compose.yml is located

Starting caddy ... done                                                                                                                                    

Attaching to caddy                                                                                                                                         

caddy    | {"level":"info","ts":1691499456.0689287,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"} 

caddy    | {"level":"warn","ts":1691499456.0720005,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"

caddyfile","file":"/etc/caddy/Caddyfile","line":9}                                                                                                         

caddy    | {"level":"info","ts":1691499456.0762668,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origi

ns":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}                                                                                                

caddy    | {"level":"info","ts":1691499456.0775971,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}       

caddy    | {"level":"info","ts":1691499456.077673,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection po

licies; adding one to enable TLS","server_name":"srv1","https_port":443}                                                                                   

caddy    | {"level":"info","ts":1691499456.077703,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv1"}        

caddy    | {"level":"info","ts":1691499456.07822,"logger":"http","msg":"enabling HTTP/3 listener","addr":":2016"}                                          

caddy    | {"level":"info","ts":1691499456.0783753,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB

). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}                                                                             

caddy    | {"level":"info","ts":1691499456.0794368,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}                  

caddy    | {"level":"info","ts":1691499456.079528,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}                                          

caddy    | {"level":"info","ts":1691499456.079708,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}                   

caddy    | {"level":"info","ts":1691499456.0798655,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2

","h3"]}                                                                                                                                                   

caddy    | {"level":"info","ts":1691499456.0800827,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}                

caddy    | {"level":"info","ts":1691499456.0801237,"msg":"serving initial configuration"}                                                                  

caddy    | {"level":"info","ts":1691499456.0802798,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00032950

0"}                                                                                                                                                        

caddy    | {"level":"info","ts":1691499456.080402,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}                    

caddy    | {"level":"info","ts":1691499456.0843327,"logger":"tls","msg":"finished cleaning storage units"}                                                 

********************
***** Connection to caddy is made here                                             ********************                                                                                                      

caddy    | {"level":"warn","ts":1691499478.27926,"logger":"http","msg":"could not get status; will try to get certificate anyway","error":"Get \"http://loc

al-tailscaled.sock/localapi/v0/status\": dial unix /var/run/tailscale/tailscaled.sock: connect: no such file or directory"}                                

caddy    | {"level":"error","ts":1691499478.2793655,"logger":"tls.handshake","msg":"getting certificate from external certificate manager","remote_ip":"100

.125.48.40","remote_port":"60140","sni":"machine.domain.ts.net","cert_manager":0,"error":"Get \"http://local-tailscaled.sock/localapi/v0/cert/vaulty.tail

a5148.ts.net?type=pair\": dial unix /var/run/tailscale/tailscaled.sock: connect: no such file or directory"}                                               

caddy    | {"level":"info","ts":1691499478.2794874,"logger":"tls.on_demand","msg":"obtaining new certificate","remote_ip":"100.125.48.40","remote_port":"60

140","server_name":"machine.domain.ts.net"}                                                                                                              

caddy    | {"level":"info","ts":1691499478.2796874,"logger":"tls.obtain","msg":"acquiring lock","identifier":"machine.domain.ts.net"}                    

caddy    | {"level":"info","ts":1691499478.2826056,"logger":"tls.obtain","msg":"lock acquired","identifier":"machine.domain.ts.net"}                     

caddy    | {"level":"info","ts":1691499478.2827125,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"machine.domain.ts.net"}             

caddy    | {"level":"info","ts":1691499478.285254,"logger":"tls","msg":"waiting on internal rate limiter","identifiers":["machine.domain.ts.net"],"ca":"h

ttps://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}                                                                              

caddy    | {"level":"info","ts":1691499478.2852805,"logger":"tls","msg":"done waiting on internal rate limiter","identifiers":["machine.domain.ts.net"],"

ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}                                                                        

caddy    | {"level":"info","ts":1691499479.3021843,"logger":"tls.acme_client","msg":"trying to solve challenge","identifier":"machine.domain.ts.net","cha

llenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}                                                                          

caddy    | {"level":"error","ts":1691499479.867296,"logger":"tls.acme_client","msg":"challenge failed","identifier":"machine.domain.ts.net","challenge_ty

pe":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up A for machine.domain.ts.net - 

check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for machine.domain.ts.net - check that a DNS record exists for this

 domain","instance":"","subproblems":[]}}                                                                                                                  

caddy    | {"level":"error","ts":1691499479.867339,"logger":"tls.acme_client","msg":"validating authorization","identifier":"machine.domain.ts.net","prob

lem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up A for machine.domain.ts.net - check that a DNS record

 exists for this domain; DNS problem: NXDOMAIN looking up AAAA for machine.domain.ts.net - check that a DNS record exists for this domain","instance":"",

"subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1247308536/200246894916","attempt":1,"max_attempts":3}                          

caddy    | {"level":"info","ts":1691499481.1934462,"logger":"tls.acme_client","msg":"trying to solve challenge","identifier":"machine.domain.ts.net","cha

llenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}                                                                              

caddy    | {"level":"error","ts":1691499481.7219243,"logger":"tls.acme_client","msg":"challenge failed","identifier":"machine.domain.ts.net","challenge_t

ype":"http-01","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up A for machine.domain.ts.net - che

ck that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for machine.domain.ts.net - check that a DNS record exists for this do

main","instance":"","subproblems":[]}}                                                                                                                     

caddy    | {"level":"error","ts":1691499481.7219615,"logger":"tls.acme_client","msg":"validating authorization","identifier":"machine.domain.ts.net","pro

blem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up A for machine.domain.ts.net - check that a DNS recor

d exists for this domain; DNS problem: NXDOMAIN looking up AAAA for machine.domain.ts.net - check that a DNS record exists for this domain","instance":""

,"subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1247308536/200246898176","attempt":2,"max_attempts":3}

EDIT - SOLUTION: many weeks later, I’ve learn a few things. Running Caddy bare-metal removed the complexity of dealing with docker networks, but it wasn’t as robust as I expected (lets just say - I ran into a very edge-case issue that ruined my day).

The solution to my actual problem was to actually directing the requests to the URL to the actual IP adress of the docker container running the service I want to make avaible, and ensure that both docker and the service are on the same docker network. A very obvious solution in hindsight, and to be fair, I think I’ve had the misfortune to run into several issues before reaching this insight.

  • krashOP
    1·
    1 year ago

    Good find.

    I am running Caddy through docker (with sudo docker-compose up, yml is listed above). I know, sudo:ing docker isn’t best practice, but I’m learning the ropes in a non-production enviorment 🙃 Also, I verified that docker is running as root by ps -eo euser,ruser,suser,fuser,f,comm,label |grep caddy

    As for the docker version, I verified it by inspecting the image ID and saw that the image version is 2.7.2:

               "Labels": {
                "org.opencontainers.image.description": "a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go",
                "org.opencontainers.image.documentation": "https://caddyserver.com/docs",
                "org.opencontainers.image.licenses": "Apache-2.0",
                "org.opencontainers.image.source": "https://github.com/caddyserver/caddy-docker",
                "org.opencontainers.image.title": "Caddy",
                "org.opencontainers.image.url": "https://caddyserver.com",
                "org.opencontainers.image.vendor": "Light Code Labs",
                "org.opencontainers.image.version": "v2.7.2"
            }

    It seems that my next step is to look into the issue why dockerized-Caddy can’t communicate with Tailscale. Now I have a direction to investigate further into 🙂

    • AnonymousDeity@beehaw.org
      1·
      1 year ago

      ah, yeah, that’s why. You need to mount the unix socket into Caddy’s container as a volume. Docker uses overlayfs by default to create a layered filesystem, and then launches a distinct user, process, network, etc. namespace for the container’s process, which is why everything is isolated inside the container. You’ll need to make sure the unix socket is available to Caddy’s process inside the container, so you’ll have to mount it using -v or the volume key in the yaml.

      sudo is actually entirely unnecessary with Docker, because most containers will run as the container’s root. Part of containers having their own user and process namespace means their root user is not your root user (technically we can have a debate about semantics for overlayfs and mounted files), and almost all images will ship with the default user as their root. Therefore, almost all processes will be “run as root” from within their container by default, meaning sudo does nothing except elevate the perms for the user calling docker. It would really only get around an issue with your user account not having access to docker or the docker daemon (also via socket btw). That said, because of the user namespace thing, running sudo docker run or sudo docker compose up doesn’t actually guarantee the process in the container is run as root… just that the container was created as root with perms over the host’s system.

      The important part is that Caddy inside the container will be run by a user that has permissions over the mounted socket.