mirrord container not working

[shaoye]

Bug Description

we got the following errors when running mirrord container -- docker run nginx with the latest version 3.125.2

  x ! mirrord container is currently an unstable feature
    ✓ preparing to launch process
      ✓ operator not found
      ✓ agent pod created
      ✓ pod is ready                                                                           Error:   × Command output indicates an error [docker logs -f
  │ 78227db451644e0faa3ee02fa711a3a3a16b6d8a173daa4ec98df05e81cbeef2]: Error:   × Main
  │ internal proxy logic failed: connecting with agent failed: could not
  │   │ create connection with tls: Connection reset by peer (os error 104)
  │   ├─▶ connecting with agent failed: could not create connection with tls:
  │   │   Connection reset by peer (os error 104)
  │   ├─▶ could not create connection with tls: Connection reset by peer (os error
  │   │   104)
  │   ╰─▶ could not create connection with tls: Connection reset by peer (os error
  │       104)
  │   help:
  │ 
  │         - If you're still stuck:
  │ 
  │         >> Please open a new bug report at https://github.com/metalbear-co/
  │         mirrord/issues/new/choose
  │ 
  │         >> Or join our Discord https://discord.gg/metalbear and request help
  │         in #mirrord-help
  │ 
  │         >> Or email us at [email protected]
  │ 
  │ 
  │ 
  │ 
  help: This is a bug. Please report it in our Discord or GitHub repository.

Not sure what connections it's trying to make but I noticed it created the mirrord pod successfully in our cluster and the pod exited immediately with the following logs:

agent ready - version 3.125.2
  2024-12-04T02:51:11.142940Z  WARN mirrord_agent::outgoing::udp: interceptor_task -> no messages left
    at mirrord/agent/src/outgoing/udp.rs:225 on ThreadId(5)

  2024-12-04T02:51:11.143069Z  INFO mirrord_agent::entrypoint: main -> mirrord-agent `start` exiting successfully.
    at mirrord/agent/src/entrypoint.rs:847 on ThreadId(1)
2024-12-04T02:51:11.143188990Z

We are using EKS (v1.30.6-eks-7f9249a) and kubeconfig on the host machine is generated with aws eks update-kubeconfig --name OUR_CLUSTER_NAME. Everything worked with mirrord exec

Steps to Reproduce

1. install the latest mirrord
2. connect to EKS v1.30
3. mirrord container -- docker run nginx

Backtrace

No response

mirrord layer logs

No response

mirrord intproxy logs

No response

mirrord agent logs

No response

mirrord config

No response

mirrord CLI version

No response

mirrord-agent version

No response

mirrord-operator version (if relevant)

No response

plugin kind and version (if relevant)

No response

Your operating system and version

ec2 ubuntu 22.04

Local process

using container

Local process version

No response

Additional Info

No response

[linear]

MBE-553 mirrord container not working

[shaoye]

We found there is no log in the local mirrord-cli container (44e0faa3ee02fa711a3a3a16b6d8a173daa4ec98df05e81cbeef2) and there is no nginx container...

$ docker ps -a
CONTAINER ID   IMAGE                                                                   COMMAND                   CREATED              STATUS                       PORTS                                     NAMES
78227db4516   ghcr.io/metalbear-co/mirrord-cli:3.127.0                                "mirrord intproxy"        About a minute ago   Up About a minute            0.0.0.0:5000->80/tcp, [::]:5000->80/tcp   stoic_kalam

so we think the issue might be mirrord not being able to start the nginx container on the host?

[DmitryDodzin]

Hi, this looks like the mirrord-cli container was unable to connect back to our process running on the host machine, is there any reason the spawned container should be unable to connect to host machine with using the host's network addr?

Just to check the communication by itself could you please run nc -l 8080 in one terminal and docker run --rm busybox nc <ip addr> 8080 -e "hostname" in another where you replace <ip addr> with your local addr (to get the ip addr we are trying to connect to linux: ip route get 1.1.1.1 | grep -oP 'src \K[0-9.]+' macos: ipconfig getifaddr $(route -n get default | grep interface | awk '{print $2}'))

If you see some hostname (can be just some hex string) appear on the nc -l 808 this means it should be able to communicate with extproxy and extproxy probably didn't start (there should be logs in /tmp folder for extproxy logs and those can help with figuring out the issue), if the request fails I would take a look at why? we do need to access the host from the container, some overlap between docker network addrs and local network addrs can cause this issue.

[shaoye]

Thank you. I tried the command you shared and nc -l 8080 does return with a hex 14e6b45e0a29 string. But there is no logs under /tmp folder

coder /tmp $ ip route get 1.1.1.1 | grep -oP 'src \K[0-9.]+'
172.31.114.236

coder /tmp $ nc -l 8080 & docker run --rm busybox nc 172.31.114.236 8080 -e "hostname"
[1] 7807
14e6b45e0a29
[1]+  Done                    nc -l 8080

coder /tmp $ ls -a
.  ..