Initial deployment troubleshooting: docker services, user creation

[RRawlings (Rachel Rawlings)]

I’m doing a test deployment for one of our researchers and think I have all the components talking to one another, but I cannot create the first admin user.

I’ve deployed everything via docker compose and am using Apache rather than traefik to proxy connections, in order to have Shibboleth protection on the server.

When I go to https://(devhost)/api/researcher and log in with admin:admin, I get a “403.no-such-credentials” error, and no new password is generated in the log, as Testing the LAMP Platform | LAMP Platform says should happen. (/api is proxied to the server container on port 3000)

The logs only show

lamp-server-server-1         | GET /researcher 401 - 1.738 ms
lamp-server-server-1         | GET /researcher 403 - 11.403 ms

The test database is working, but if I look at the “mindlamp” database in mongo db, there is only a single collection called “delete_me”. running db.delete_me.drop() in mongosh and restarting the docker services did not cause the normal collections to magically appear.

Here are the containers I have running, from a single compose file:

CONTAINER ID   IMAGE                                                COMMAND                  CREATED      STATUS                       PORTS                                                          NAMES
86a619ecba16   mongo                                                "docker-entrypoint.s…"   5 days ago   Up About an hour             127.0.0.1:27017->27017/tcp                                     lamp-server-mongo-1
2a0d0438395d   mongo-express                                        "/sbin/tini -- /dock…"   5 days ago   Up About an hour             127.0.0.1:8081->8081/tcp                                       lamp-server-mongo-express-1
a0f5e4df07e8   ghcr.io/bidmcdigitalpsychiatry/lamp-server:2022      "docker-entrypoint.s…"   5 days ago   Up About an hour (healthy)   127.0.0.1:3000->3000/tcp                                       lamp-server-server-1
4b31293b4b43   ghcr.io/bidmcdigitalpsychiatry/lamp-dashboard:2023   "/docker-entrypoint.…"   5 days ago   Up About an hour (healthy)   127.0.0.1:8000->80/tcp                                         lamp-server-dashboard-1
f627d3076063   redis                                                "docker-entrypoint.s…"   5 days ago   Up About an hour             127.0.0.1:6379->6379/tcp                                       lamp-server-redis-1
f97f2c56746d   nats                                                 "/nats-server --conf…"   5 days ago   Up About an hour             127.0.0.1:4222->4222/tcp, 127.0.0.1:8222->8222/tcp, 6222/tcp   lamp-server-nats-1

• Am I missing a component here?
• How is the production database supposed to be generated?
• Could I create a new user in MongoDB instead of the web API interface?

Thanks,

Rachel Rawlings
Penn Medicine

[clangholm (Carsten Langholm)]

The admin password should be generated by the lamp-server service. In your post I can only see one line, it looks like you’ve only captured a portion of the service logs. Could you print multiple lines and check again?

Can you send a copy of your docker-compose file for the “server” and “dashboard” services? (passwords redacted)

[RRawlings (Rachel Rawlings)]

The two logs lines for “GET /researcher” are the only ones that appear in the log on login, and are repeated whether I use the Apache proxy, or connect directly to port 3000. Using ‘user:password@localhost:3000’ with curl skips the 401 error and just gives the 403, but at no point does a password generation happen in the log. I’ve stopped and rebuilt the container a few times and there’s no change in the logged data.

$ curl -k admin:admin@localhost:3000/researcher
{
  "error": "403.no-such-credentials"
}

And then in the container log:

2024-01-23T19:36:46.202960275Z  GET / 200 - 8.985 ms
2024-01-23T19:37:16.273677702Z  GET / 200 - 5.852 ms
2024-01-23T19:37:46.349817439Z  GET / 200 - 6.171 ms
2024-01-23T19:37:54.269642358Z  GET /researcher 403 - 9.921 ms
2024-01-23T19:38:16.413598503Z  GET / 200 - 6.072 ms
2024-01-23T19:38:46.489804242Z  GET / 200 - 6.045 ms
2024-01-23T19:39:16.564054945Z  GET / 200 - 6.099 ms

(The 200s are the Apache proxy’s heartbeat check.)

Here’s the docker-compose file. I tried unsuccessfully to use password files in a couple places rather than having to keep them in the config, so those have been commented out along with the traefik container options.

version: '3.7'

networks:
  public:
    name: public
    # external: true
    driver: overlay
    attachable: true
# traefik:
#   name: traefik
#   external: true
secrets:
  mongo_password:
    file: mongodb_password.txt 

services:
  mongo:
    image: mongo
    restart: no
    env_file:
      - .env
      - mongo.env
    environment:
      MONGO_INITDB_DATABASE: db
      MONGO_INITDB_ROOT_USERNAME: admin
      MONGO_INITDB_ROOT_PASSWORD: [[REDACTED]]
      # MONGO_INITDB_ROOT_PASSWORD_FILE: /run/secrets/mongo_password
    hostname: database
    networks:
      - public
#     - traefik
    ports:
      - 127.0.0.1:27017:27017
    volumes:
      - /data/db:/data/db
#   secrets:
#     - source: mongo_password
#       target: mongo_password
#       uid: '1002'
#       gid: '1002'
#       mode: 0440

  mongo-express:
    image: mongo-express
    restart: no
    hostname: express
    networks:
      - public
#     - traefik
    ports:
      - 127.0.0.1:8081:8081
    env_file:
      - .env
      - mongo.env
    environment:
      ME_CONFIG_MONGODB_SERVER: mongo
      ME_CONFIG_MONGODB_PORT: 27017
      ME_CONFIG_MONGODB_ADMINUSERNAME: admin
      ME_CONFIG_MONGODB_ADMINPASSWORD: [[REDACTED]]
      # ME_CONFIG_MONGODB_ADMINPASSWORD_FILE: /run/secrets/mongo_password
      ME_CONFIG_SITE_SSL_ENABLED: false
      ME_CONFIG_BASICAUTH_USERNAME: admin
      ME_CONFIG_BASICAUTH_PASSWORD: [[REDACTED]]
#   secrets:
#     - source: mongo_password
#       target: mongo_password
#       uid: '1002'
#       gid: '1002'
#       mode: 0440

  redis:
    image: redis
    volumes:
      - /data/redis:/data/redis
      - /home/mindlamp/redis.conf:/usr/local/etc/redis/redis.conf
    command: redis-server --include /usr/local/etc/redis/redis.conf
    restart: no
    hostname: redis
    networks:
      - public
    ports:
      - 127.0.0.1:6379:6379

  nats:
    image: nats
    restart: no
    hostname: nats
    networks:
      - public
    ports:
      - 127.0.0.1:4222:4222
      - 127.0.0.1:8222:8222

  server:
    image: ghcr.io/bidmcdigitalpsychiatry/lamp-server:2022
    healthcheck:
      test: wget --no-verbose --tries=1 --spider http://localhost:3000 || exit 1
    environment:
      HTTPS: 'off'
      ROOT_KEY: [[REDACTED]]
      DB: 'mongodb://admin:[[REDACTED]]@database:27017/'
      PUSH_API_GATEWAY: 'https://app-gateway.lamp.digital/'
      PUSH_API_KEY: 'YOUR_PUSH_KEY_HERE'
      DASHBOARD_URL: 'mindlamp-dev.pmacs.upenn.edu/api'
      REDIS_HOST: 'redis://redis:6379/0'
      NATS_SERVER: 'nats:4222'
    networks:
      - public
#     - traefik
    ports:
      - 127.0.0.1:3000:3000
    logging:
      options:
        max-size: "10m"
        max-file: "3"
    deploy:
      mode: replicated
      update_config:
        order: start-first
        failure_action: rollback
#     labels:
#       traefik.enable: 'true'
#       traefik.docker.network: 'traefik'
#       traefik.http.routers.lamp_server.entryPoints: 'websecure'
#       traefik.http.routers.lamp_server.rule: 'Host(`pumas-dev-web.pmacs.upenn.edu`)'
#       traefik.http.routers.lamp_server.tls.certresolver: 'default'
#       traefik.http.services.lamp_server.loadbalancer.server.port: 3000
#         placement:
#           constraints:
#             - node.role == manager

  dashboard:
    image: ghcr.io/bidmcdigitalpsychiatry/lamp-dashboard:2023
    logging:
        driver: "json-file"
        options:
          max-size: "50m"
    environment:
      REACT_APP_LAMP_RESEARCHER_ALIAS: 'Investigator'
    networks:
      - public
    ports:
      - 127.0.0.1:8000:80
    healthcheck:
      test: wget --no-verbose --tries=1 --spider http://localhost:80 || exit 1
    deploy:
      mode: replicated
      update_config:
        order: start-first
        failure_action: rollback
      labels:
        portainer.autodeploy: 'true'
        #traefik.enable: 'true'
        #traefik.http.routers.lamp_dashboard.entryPoints: 'websecure'
        #traefik.http.routers.lamp_dashboard.rule: 'Host(`dashboard.example.com`)'
        #traefik.http.routers.lamp_dashboard.tls.certresolver: 'default'
        #traefik.http.services.lamp_dashboard.loadbalancer.server.port: 80
      placement:
        constraints:
          - node.role == manager

[RRawlings (Rachel Rawlings)]

So after further testing, I have a an administrator account and password that works in the API, but not in the sign-in page, nor do I see how in Update a Researcher. | LAMP Platform, Add a Credential for a Researcher, Study, Participant, Activity, or Sensor. | LAMP Platform, and associated pages to create a password for a user to sign in. Where am I going wrong this time?

$ curl  -su admin http://localhost:3000/researcher/w48j2wg4m6ecarafv0d5
Enter host password for user 'admin':
{
  "data": [
    {
      "id": "w48j2wg4m6ecarafv0d5",
      "name": "Doctor Scratchmonkey"
    }
  ]
}

$ curl  -su admin  -L -X POST http://localhost:3000/type/w48j2wg4m6ecarafv0d5/credential -H 'Content-Type: application/json' -H 'Accept: application/json' 
Enter host password for user 'admin':
{
  "error": "400.malformed-credential-object"

(The 400 error also happened when I replaced “type” with “researcher”, just in case I was supposed to use “type” as a variable.)

[clangholm (Carsten Langholm)]

How did you update the admin password? The steps are to choose an admin password (ex: testpassword), encrypt it, then insert the encrypted string into the test.credential collection of the mongo database. Afterward, you would use “testpassword” to log-in (not the encrypted string) via the sign-in page.

[RRawlings (Rachel Rawlings)]

I haven’t actually updated the password. I’m using the initially-generated password as extracted and decrypted from the database. This works on API access, but not the dashboard.

[clangholm (Carsten Langholm)]

Something certainly went wrong here… what happens if create a fresh DB service mounted to a new volume?

[RRawlings (Rachel Rawlings)]

I stopped the docker services, completely wiped out the data volume, and started again. I got a new admin password from the server startup, was able to create a researcher in the API, but once again fail to log in at the dashboard.

However, with the console open this time, I’m noticing that I’m getting a lot of 404s from the dashboard service, notably /service-worker.js

lamp-server-dashboard-1      | 2024/03/05 16:15:52 [error] 22#22: *15 open() "/usr/share/nginx/html/locales/en-US/translation.json" failed (2: No such file or directory), client: 172.18.0.1, server: localhost, request: "GET /locales/en-US/translation.json HTTP/1.1", host: "localhost:8000", referrer: "https://mindlamp-dev.example.org/"
lamp-server-dashboard-1      | 172.18.0.1 - - [05/Mar/2024:16:15:52 +0000] "GET /locales/en-US/translation.json HTTP/1.1" 404 153 "https://mindlamp-dev.example.org/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:123.0) Gecko/20100101 Firefox/123.0" "192.168.123.45"
lamp-server-dashboard-1      | 172.18.0.1 - - [05/Mar/2024:16:15:52 +0000] "GET /locales/dev/translation.json HTTP/1.1" 404 153 "https://mindlamp-dev.example.org/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:123.0) Gecko/20100101 Firefox/123.0" "192.168.123.45"
lamp-server-dashboard-1      | 2024/03/05 16:15:52 [error] 22#22: *15 open() "/usr/share/nginx/html/locales/dev/translation.json" failed (2: No such file or directory), client: 172.18.0.1, server: localhost, request: "GET /locales/dev/translation.json HTTP/1.1", host: "localhost:8000", referrer: "https://mindlamp-dev.example.org/"
lamp-server-dashboard-1      | 172.18.0.1 - - [05/Mar/2024:16:15:52 +0000] "GET /logo.png HTTP/1.1" 200 19870 "https://mindlamp-dev.example.org/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:123.0) Gecko/20100101 Firefox/123.0" "192.168.123.45"
lamp-server-dashboard-1      | 172.18.0.1 - - [05/Mar/2024:16:15:52 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "https://mindlamp-dev.example.org/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:123.0) Gecko/20100101 Firefox/123.0" "192.168.123.45"
lamp-server-dashboard-1      | 2024/03/05 16:15:52 [error] 22#22: *15 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 172.18.0.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "localhost:8000", referrer: "https://mindlamp-dev.example.org/"
lamp-server-dashboard-1      | 2024/03/05 16:15:52 [error] 22#22: *15 open() "/usr/share/nginx/html/service-worker.js" failed (2: No such file or directory), client: 172.18.0.1, server: localhost, request: "GET /service-worker.js HTTP/1.1", host: "localhost:8000"
lamp-server-dashboard-1      | 172.18.0.1 - - [05/Mar/2024:16:15:52 +0000] "GET /service-worker.js HTTP/1.1" 404 153 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:123.0) Gecko/20100101 Firefox/123.0" "192.168.123.45"

Firefox is also complaining about cross-origin scripting in the browser

[RRawlings (Rachel Rawlings)]

I’m using Apache rather than traefik as the front end, and here’s the dashboard section from the compose file:

  dashboard:
    image: ghcr.io/bidmcdigitalpsychiatry/lamp-dashboard:2023
    logging:
        driver: "json-file"
        options:
          max-size: "50m"
    environment:
      REACT_APP_LAMP_RESEARCHER_ALIAS: 'Investigator'
    networks:
      - public
    ports:
      - 127.0.0.1:8000:80
    healthcheck:
      test: wget --no-verbose --tries=1 --spider http://localhost:80 || exit 1
    deploy:
      mode: replicated
      update_config:
        order: start-first
        failure_action: rollback
      labels:
        portainer.autodeploy: 'true'
        #traefik.enable: 'true'
        #traefik.http.routers.lamp_dashboard.entryPoints: 'websecure'
        #traefik.http.routers.lamp_dashboard.rule: 'Host(`dashboard.example.com`)'
        #traefik.http.routers.lamp_dashboard.tls.certresolver: 'default'
        #traefik.http.services.lamp_dashboard.loadbalancer.server.port: 80
      placement:
        constraints:
          - node.role == manager

[clangholm (Carsten Langholm)]

And what happens if you log-in using the auto-generated password (without doing any encryption/decryption, just copy paste where it says “Administrator Password”) into our dashboard. at dashboard.lamp.digital (i.e. dont deploy your own)?

[jtorous (John Torous)]

Can you please try it with traefik first and let us know if that works.

-John

PLEASE NOTE: This message is intended for the use of the person to whom it is addressed. It may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the intended recipient, your use of this message for any purpose is strictly prohibited. If you have received this communication in error, please delete the message and notify the sender so that we may correct our records. See our web page at http://www.bilh.org for a full directory of Beth Israel Lahey Health sites, staff, services and career opportunities.

[RRawlings (Rachel Rawlings)]

I’d already tried it with traefik and a complete reference implementation on another server and made no progress. I could go back to that well again.

[clangholm (Carsten Langholm)]

Does your dashboard have an SSL certificate?

What happens if you use our dashboard?

[RRawlings (Rachel Rawlings)]

There’s no cert on the node proxy and it doesn’t use https,

    # MindLamp API from the lamp-server container
    ProxyPass         /api/   http://localhost:3000/
    ProxyPassReverse  /api/   http://localhost:3000/

    # MindLamp Frontend from the lamp-server container
    ProxyPass         / http://localhost:8000/
    ProxyPassReverse  / http://localhost:8000/

Using your dashboard would fail because the dev server doesn’t have an opening in the university firewall or an external DNS entry.

[RRawlings (Rachel Rawlings)]

I added Header set Access-Control-Allow-Origin "api.lamp.digital" to the Apache config to take care of the CORS errors,

I then went into the container and created symbolic links to get rid of the 404s

$ docker exec -t -i 822e090dad57 /bin/sh
/ # find . -name translation.json
./usr/share/nginx/html/locales/da/translation.json
./usr/share/nginx/html/locales/de/translation.json
./usr/share/nginx/html/locales/en/translation.json
./usr/share/nginx/html/locales/es/translation.json
./usr/share/nginx/html/locales/fr/translation.json
./usr/share/nginx/html/locales/hi/translation.json
./usr/share/nginx/html/locales/it/translation.json
./usr/share/nginx/html/locales/ko/translation.json
./usr/share/nginx/html/locales/zh-CN/translation.json
./usr/share/nginx/html/locales/zh-HK/translation.json
/ # cd /usr/share/nginx/html/locales
/usr/share/nginx/html/locales # ln -s en en-US
/usr/share/nginx/html/locales # ln -s en dev

This only leaves the permission errors for api.lamp.digital

If I change the server address in the dashboard login screen to my dev host, I get 404s both via the proxy and direct to the container, where going to your server I get 403/401.

lamp-server-dashboard-1      | 172.18.0.1 - - [05/Mar/2024:20:45:33 +0000] "GET /type/me/parent HTTP/1.1" 404 153 "-" "curl/7.61.1" "-"
lamp-server-dashboard-1      | 2024/03/05 20:45:33 [error] 22#22: *594 open() "/usr/share/nginx/html/type/me/parent" failed (2: No such file or directory), client: 172.18.0.1, server: localhost, request: "GET /type/me/parent HTTP/1.1", host: "localhost:8000"
lamp-server-dashboard-1      | 2024/03/05 20:46:17 [error] 22#22: *596 open() "/usr/share/nginx/html/type/null/attachment/lamp.dashboard.admin_permissions" failed (2: No such file or directory), client: 172.18.0.1, server: localhost, request: "GET /type/null/attachment/lamp.dashboard.admin_permissions HTTP/1.1", host: "localhost:8000"
lamp-server-dashboard-1      | 172.18.0.1 - - [05/Mar/2024:20:46:17 +0000] "GET /type/null/attachment/lamp.dashboard.admin_permissions HTTP/1.1" 404 153 "-" "curl/7.61.1" "-"

[RRawlings (Rachel Rawlings)]

After creating a NAT and opening the firewall to lamp.digital, I am still getting the same errors.