First version

.gitignore

@@ -0,0 +1,55 @@
+bin
+obj
+csx
+.vs
+edge
+Publish
+
+*.user
+*.suo
+*.cscfg
+*.Cache
+project.lock.json
+
+/packages
+/TestResults
+
+/tools/NuGet.exe
+/App_Data
+/secrets
+/data
+.secrets
+appsettings.json
+local.settings.json
+
+node_modules
+dist
+
+# Local python packages
+.python_packages/
+
+# Python Environments
+.env*
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+*.tgz
+# Byte-compiled / optimized / DLL files
+__pycache__/
+**/pycache/*
+*.py[cod]
+*$py.class
+
+# Azurite artifacts
+__blobstorage__
+__queuestorage__
+__azurite_db*__.json
+
+
+.idea/
+.vscode
+
+playground

Dockerfile

@@ -0,0 +1,16 @@
+FROM python:3.12-slim
+
+WORKDIR /app
+
+COPY requirements.txt /app/
+
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY testbed /app/testbed/
+
+COPY main.py /app/
+COPY entrypoint.sh /app/
+
+EXPOSE 8000
+
+ENTRYPOINT ["/app/entrypoint.sh"]

README.md

@@ -0,0 +1,132 @@
+# Client-Testbed Architecture with Kubernetes and ZeroMQ
+
+## Overview
+
+This document describes the architecture of the Client-Testbed system, which utilizes Kubernetes for orchestration and ZeroMQ for communication. The system is designed to create isolated evaluation environments (testbeds) and facilitate communication between clients and these testbeds.
+
+## Key Components
+
+1. TestbedManager
+2. TestbedClient
+3. Server
+4. ZeroMQCommunicator
+5. Testbed
+
+## Kubernetes Integration
+
+The system leverages Kubernetes for managing and orchestrating testbed environments:
+
+- **TestbedManager**: Interacts with the Kubernetes API to create and manage testbed instances.
+- **Kubernetes Job**: Each testbed is created as a Kubernetes Job, ensuring the testbed runs to completion.
+- **Kubernetes Service**: A LoadBalancer service is created for each testbed to expose the ZeroMQ ports to external clients.
+- **Containers**:
+  - Testbed Container: Runs the actual evaluation code.
+  - Sidecar Container: Handles communication and manages the testbed lifecycle.
+
+## ZeroMQ Communication
+
+ZeroMQ is used for efficient, asynchronous communication between the client and the testbed. The system uses two patterns:
+
+1. **REQ-REP Pattern**: Used for ping-pong communication to check connectivity.
+   - **REQ (Client)**: Sends ping requests.
+   - **REP (Server)**: Responds with pong messages.
+
+2. **PUB-SUB Pattern**: Used for run_evaluation and result communication.
+   - **Publisher (PUB)**: The server (testbed) publishes evaluation results.
+   - **Subscriber (SUB)**: The client subscribes to messages from a specific testbed.
+
+## Workflow
+
+### 1. Testbed Creation
+
+```python
+def create_testbed(self, instance_id: str, user_id: str | None = None) -> CreateTestbedResponse:
+    # ... (code to create Kubernetes Job and Service)
+    return CreateTestbedResponse(testbed_id=testbed_id)
+```
+
+- The TestbedManager creates a Kubernetes Job and Service for a new testbed.
+- The Job runs two containers: the testbed and the sidecar.
+- The Service exposes the ZeroMQ ports (5555 for PUB, 5556 for SUB).
+
+### 2. Client Connection
+
+```python
+def create_client(self, testbed_id: str, timeout: float = 30) -> TestbedClient:
+    # ... (code to create TestbedClient)
+    return TestbedClient(testbed_id=testbed_id, pub_address=pub_address, sub_address=sub_address)
+```
+
+- The TestbedManager creates a TestbedClient with the external IP and ports of the testbed's Service.
+- The TestbedClient initializes a ZeroMQCommunicator to handle messaging.
+
+### 3. Communication Flow
+The client and server communicate using ZeroMQ's PUB-SUB pattern, with an additional ping-pong mechanism for checking connectivity.
+
+#### Client to Server (Ping):
+
+```python
+def ping(self, timeout=30):
+    self.communicator.send_message(message_type="ping", data={})
+    start_time = time.time()
+    while time.time() - start_time < timeout:
+        messages = self.communicator.receive_messages()
+        for message in messages:
+            if message.type == "pong":
+                return True
+        time.sleep(1)
+    return False
+```
+
+1. The client sends a "ping" message to the server.
+2. It then waits for a "pong" response, with a specified timeout.
+
+#### Server to Client (Pong):
+
+```python
+def process_message(self, message):
+    if message.type == "ping":
+        self.communicator.send_message(Message(type="pong", body={}))
+        logger.info("Sent pong response.")
+    # ... handle other message types
+``` 
+
+1. The server receives the "ping" message.
+2. It immediately responds with a "pong" message.
+
+This ping-pong mechanism allows the client to check if the server is responsive and the communication channel is working correctly.
+
+### ZeroMQ Communicator
+Both client and server use the same ZeroMQCommunicator class for sending and receiving messages:
+
+```python
+class ZeroMQCommunicator(Communicator):
+    def __init__(self, testbed_id: str, pub_address: str, sub_address: str):
+        # ... initialization code ...
+
+    def send_message(self, message_type: str, data: dict[str, Any]) -> None:
+        self.pub_socket.send_multipart([
+            self.testbed_id.encode(),
+            message_type.encode(),
+            json.dumps(data).encode()
+        ])
+
+    def receive_messages(self) -> Iterable[Message]:
+        messages = []
+        while self.sub_socket.poll(timeout=100):
+            testbed_id, message_type, data = self.sub_socket.recv_multipart()
+            if testbed_id.decode() == self.testbed_id:
+                messages.append(Message(
+                    message_type.decode(),
+                    json.loads(data.decode())
+                ))
+        return messages
+```
+
+This implementation allows for bidirectional communication:
+- The client's PUB socket connects to the server's SUB socket.
+- The server's PUB socket connects to the client's SUB socket.
+
+By using the same ZeroMQCommunicator class for both client and server, we ensure consistent message formatting and handling. The testbed_id is used to filter messages, ensuring that each client only receives messages intended for it.
+
+This approach demonstrates how the client and server can communicate effectively using ZeroMQ, with the ping-pong mechanism serving as a concrete example of the message exchange process.

deploy.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+IMAGE_NAME="testbed-sidecar"
+ACR_NAME="moatless"
+NEW_VERSION="latest"
+
+echo "Logging in to Azure Container Registry..."
+az acr login --name ${ACR_NAME}
+
+echo "Building Docker image..."
+docker build -t ${ACR_NAME}.azurecr.io/${IMAGE_NAME}:${NEW_VERSION} .
+docker push ${ACR_NAME}.azurecr.io/${IMAGE_NAME}:${NEW_VERSION}

entrypoint.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+set -e
+
+# Add any startup commands here
+
+# Start the main application
+exec python /app/main.py

infra/apply_configmaps.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+# Define the source and target namespaces
+SOURCE_NAMESPACE="testbeds"
+TARGET_NAMESPACE="testbed-dev"
+
+# Directory to store the ConfigMaps
+CONFIGMAP_DIR="configmaps"
+
+# Create a directory to store the ConfigMaps
+mkdir -p $CONFIGMAP_DIR
+
+# Apply each ConfigMap to the target namespace
+for file in $CONFIGMAP_DIR/*.yaml; do
+  # Modify the namespace in the YAML file before applying it
+  sed -i "s/namespace: $SOURCE_NAMESPACE/namespace: $TARGET_NAMESPACE/g" $file
+  kubectl apply -f $file -n $TARGET_NAMESPACE
+  echo "Applied $(basename $file) to namespace $TARGET_NAMESPACE"
+done
+
+echo "All ConfigMaps have been applied to the target namespace."

infra/configmaps/instance-astropy--astropy-12907-configmap.yaml

@@ -0,0 +1,27 @@
+apiVersion: v1
+data:
+  instance.json: |-
+    {
+      "repo": "astropy/astropy",
+      "instance_id": "astropy__astropy-12907",
+      "base_commit": "d16bfe05a744909de4b27f5875fe0d4ed41ce607",
+      "patch": "diff --git a/astropy/modeling/separable.py b/astropy/modeling/separable.py\n--- a/astropy/modeling/separable.py\n+++ b/astropy/modeling/separable.py\n@@ -242,7 +242,7 @@ def _cstack(left, right):\n         cright = _coord_matrix(right, 'right', noutp)\n     else:\n         cright = np.zeros((noutp, right.shape[1]))\n-        cright[-right.shape[0]:, -right.shape[1]:] = 1\n+        cright[-right.shape[0]:, -right.shape[1]:] = right\n \n     return np.hstack([cleft, cright])\n \n",
+      "test_patch": "diff --git a/astropy/modeling/tests/test_separable.py b/astropy/modeling/tests/test_separable.py\n--- a/astropy/modeling/tests/test_separable.py\n+++ b/astropy/modeling/tests/test_separable.py\n@@ -28,6 +28,13 @@\n p1 = models.Polynomial1D(1, name='p1')\n \n \n+cm_4d_expected = (np.array([False, False, True, True]),\n+                  np.array([[True,  True,  False, False],\n+                            [True,  True,  False, False],\n+                            [False, False, True,  False],\n+                            [False, False, False, True]]))\n+\n+\n compound_models = {\n     'cm1': (map3 & sh1 | rot & sh1 | sh1 & sh2 & sh1,\n             (np.array([False, False, True]),\n@@ -52,7 +59,17 @@\n     'cm7': (map2 | p2 & sh1,\n             (np.array([False, True]),\n              np.array([[True, False], [False, True]]))\n-            )\n+            ),\n+    'cm8': (rot & (sh1 & sh2), cm_4d_expected),\n+    'cm9': (rot & sh1 & sh2, cm_4d_expected),\n+    'cm10': ((rot & sh1) & sh2, cm_4d_expected),\n+    'cm11': (rot & sh1 & (scl1 & scl2),\n+             (np.array([False, False, True, True, True]),\n+              np.array([[True,  True,  False, False, False],\n+                        [True,  True,  False, False, False],\n+                        [False, False, True,  False, False],\n+                        [False, False, False, True,  False],\n+                        [False, False, False, False, True]]))),\n }\n \n \n",
+      "problem_statement": "Modeling's `separability_matrix` does not compute separability correctly for nested CompoundModels\nConsider the following model:\r\n\r\n```python\r\nfrom astropy.modeling import models as m\r\nfrom astropy.modeling.separable import separability_matrix\r\n\r\ncm = m.Linear1D(10) & m.Linear1D(5)\r\n```\r\n\r\nIt's separability matrix as you might expect is a diagonal:\r\n\r\n```python\r\n>>> separability_matrix(cm)\r\narray([[ True, False],\r\n       [False,  True]])\r\n```\r\n\r\nIf I make the model more complex:\r\n```python\r\n>>> separability_matrix(m.Pix2Sky_TAN() & m.Linear1D(10) & m.Linear1D(5))\r\narray([[ True,  True, False, False],\r\n       [ True,  True, False, False],\r\n       [False, False,  True, False],\r\n       [False, False, False,  True]])\r\n```\r\n\r\nThe output matrix is again, as expected, the outputs and inputs to the linear models are separable and independent of each other.\r\n\r\nIf however, I nest these compound models:\r\n```python\r\n>>> separability_matrix(m.Pix2Sky_TAN() & cm)\r\narray([[ True,  True, False, False],\r\n       [ True,  True, False, False],\r\n       [False, False,  True,  True],\r\n       [False, False,  True,  True]])\r\n```\r\nSuddenly the inputs and outputs are no longer separable?\r\n\r\nThis feels like a bug to me, but I might be missing something?\n",
+      "hints_text": "",
+      "created_at": "2022-03-03T15:14:54Z",
+      "version": "4.3",
+      "FAIL_TO_PASS": "[\"astropy/modeling/tests/test_separable.py::test_separable[compound_model6-result6]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model9-result9]\"]",
+      "PASS_TO_PASS": "[\"astropy/modeling/tests/test_separable.py::test_coord_matrix\", \"astropy/modeling/tests/test_separable.py::test_cdot\", \"astropy/modeling/tests/test_separable.py::test_cstack\", \"astropy/modeling/tests/test_separable.py::test_arith_oper\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model0-result0]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model1-result1]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model2-result2]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model3-result3]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model4-result4]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model5-result5]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model7-result7]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model8-result8]\", \"astropy/modeling/tests/test_separable.py::test_custom_model_separable\"]",
+      "environment_setup_commit": "298ccb478e6bf092953bca67a3d29dc6c35f6752"
+    }
+kind: ConfigMap
+metadata:
+  annotations:
+    kubectl.kubernetes.io/last-applied-configuration: |
+      {"apiVersion":"v1","data":{"instance.json":"{\n  \"repo\": \"astropy/astropy\",\n  \"instance_id\": \"astropy__astropy-12907\",\n  \"base_commit\": \"d16bfe05a744909de4b27f5875fe0d4ed41ce607\",\n  \"patch\": \"diff --git a/astropy/modeling/separable.py b/astropy/modeling/separable.py\\n--- a/astropy/modeling/separable.py\\n+++ b/astropy/modeling/separable.py\\n@@ -242,7 +242,7 @@ def _cstack(left, right):\\n         cright = _coord_matrix(right, 'right', noutp)\\n     else:\\n         cright = np.zeros((noutp, right.shape[1]))\\n-        cright[-right.shape[0]:, -right.shape[1]:] = 1\\n+        cright[-right.shape[0]:, -right.shape[1]:] = right\\n \\n     return np.hstack([cleft, cright])\\n \\n\",\n  \"test_patch\": \"diff --git a/astropy/modeling/tests/test_separable.py b/astropy/modeling/tests/test_separable.py\\n--- a/astropy/modeling/tests/test_separable.py\\n+++ b/astropy/modeling/tests/test_separable.py\\n@@ -28,6 +28,13 @@\\n p1 = models.Polynomial1D(1, name='p1')\\n \\n \\n+cm_4d_expected = (np.array([False, False, True, True]),\\n+                  np.array([[True,  True,  False, False],\\n+                            [True,  True,  False, False],\\n+                            [False, False, True,  False],\\n+                            [False, False, False, True]]))\\n+\\n+\\n compound_models = {\\n     'cm1': (map3 \u0026 sh1 | rot \u0026 sh1 | sh1 \u0026 sh2 \u0026 sh1,\\n             (np.array([False, False, True]),\\n@@ -52,7 +59,17 @@\\n     'cm7': (map2 | p2 \u0026 sh1,\\n             (np.array([False, True]),\\n              np.array([[True, False], [False, True]]))\\n-            )\\n+            ),\\n+    'cm8': (rot \u0026 (sh1 \u0026 sh2), cm_4d_expected),\\n+    'cm9': (rot \u0026 sh1 \u0026 sh2, cm_4d_expected),\\n+    'cm10': ((rot \u0026 sh1) \u0026 sh2, cm_4d_expected),\\n+    'cm11': (rot \u0026 sh1 \u0026 (scl1 \u0026 scl2),\\n+             (np.array([False, False, True, True, True]),\\n+              np.array([[True,  True,  False, False, False],\\n+                        [True,  True,  False, False, False],\\n+                        [False, False, True,  False, False],\\n+                        [False, False, False, True,  False],\\n+                        [False, False, False, False, True]]))),\\n }\\n \\n \\n\",\n  \"problem_statement\": \"Modeling's `separability_matrix` does not compute separability correctly for nested CompoundModels\\nConsider the following model:\\r\\n\\r\\n```python\\r\\nfrom astropy.modeling import models as m\\r\\nfrom astropy.modeling.separable import separability_matrix\\r\\n\\r\\ncm = m.Linear1D(10) \u0026 m.Linear1D(5)\\r\\n```\\r\\n\\r\\nIt's separability matrix as you might expect is a diagonal:\\r\\n\\r\\n```python\\r\\n\u003e\u003e\u003e separability_matrix(cm)\\r\\narray([[ True, False],\\r\\n       [False,  True]])\\r\\n```\\r\\n\\r\\nIf I make the model more complex:\\r\\n```python\\r\\n\u003e\u003e\u003e separability_matrix(m.Pix2Sky_TAN() \u0026 m.Linear1D(10) \u0026 m.Linear1D(5))\\r\\narray([[ True,  True, False, False],\\r\\n       [ True,  True, False, False],\\r\\n       [False, False,  True, False],\\r\\n       [False, False, False,  True]])\\r\\n```\\r\\n\\r\\nThe output matrix is again, as expected, the outputs and inputs to the linear models are separable and independent of each other.\\r\\n\\r\\nIf however, I nest these compound models:\\r\\n```python\\r\\n\u003e\u003e\u003e separability_matrix(m.Pix2Sky_TAN() \u0026 cm)\\r\\narray([[ True,  True, False, False],\\r\\n       [ True,  True, False, False],\\r\\n       [False, False,  True,  True],\\r\\n       [False, False,  True,  True]])\\r\\n```\\r\\nSuddenly the inputs and outputs are no longer separable?\\r\\n\\r\\nThis feels like a bug to me, but I might be missing something?\\n\",\n  \"hints_text\": \"\",\n  \"created_at\": \"2022-03-03T15:14:54Z\",\n  \"version\": \"4.3\",\n  \"FAIL_TO_PASS\": \"[\\\"astropy/modeling/tests/test_separable.py::test_separable[compound_model6-result6]\\\", \\\"astropy/modeling/tests/test_separable.py::test_separable[compound_model9-result9]\\\"]\",\n  \"PASS_TO_PASS\": \"[\\\"astropy/modeling/tests/test_separable.py::test_coord_matrix\\\", \\\"astropy/modeling/tests/test_separable.py::test_cdot\\\", \\\"astropy/modeling/tests/test_separable.py::test_cstack\\\", \\\"astropy/modeling/tests/test_separable.py::test_arith_oper\\\", \\\"astropy/modeling/tests/test_separable.py::test_separable[compound_model0-result0]\\\", \\\"astropy/modeling/tests/test_separable.py::test_separable[compound_model1-result1]\\\", \\\"astropy/modeling/tests/test_separable.py::test_separable[compound_model2-result2]\\\", \\\"astropy/modeling/tests/test_separable.py::test_separable[compound_model3-result3]\\\", \\\"astropy/modeling/tests/test_separable.py::test_separable[compound_model4-result4]\\\", \\\"astropy/modeling/tests/test_separable.py::test_separable[compound_model5-result5]\\\", \\\"astropy/modeling/tests/test_separable.py::test_separable[compound_model7-result7]\\\", \\\"astropy/modeling/tests/test_separable.py::test_separable[compound_model8-result8]\\\", \\\"astropy/modeling/tests/test_separable.py::test_custom_model_separable\\\"]\",\n  \"environment_setup_commit\": \"298ccb478e6bf092953bca67a3d29dc6c35f6752\"\n}"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"instance-astropy--astropy-12907-configmap","namespace":"testbeds"}}
+  creationTimestamp: "2024-07-19T14:30:09Z"
+  name: instance-astropy--astropy-12907-configmap
+  namespace: testbed-dev
+  resourceVersion: "5590689"
+  uid: dad78a2e-dffb-4198-ae30-ab7d70c696a3