Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURES] Support for Node-Specific Restrictions in Fluid for K8s 1.31 #4488

Open
cheyang opened this issue Feb 8, 2025 · 2 comments
Open
Labels
features features

Comments

@cheyang
Copy link
Collaborator

cheyang commented Feb 8, 2025

What feature you'd like to add:

Fluid should support Node-Specific Restrictions as outlined in Kubernetes Enhancement Proposal ENH-4193. This feature will enable Fluid components (e.g., Runtime, Controller, CSI Driver) to dynamically recognize and adhere to node-level restrictions (e.g., allowed Pod types, resource quotas) for Fluid CSI Plugin Daemonset.

Why is this feature needed:

@cheyang cheyang added the features features label Feb 8, 2025
@TrafalgarZZZ
Copy link
Member

TrafalgarZZZ commented Feb 11, 2025

/assign @TrafalgarZZZ

@xliuqq
Copy link
Collaborator

xliuqq commented Feb 13, 2025

@cheyang, I use minikube with k8s 1.30 for quick testing.

Step 1: modify service account fluid-csi ClusterRole with 'get,patch' node rbac.
Test 1: In csi-nodeplugin-fluid-XXX pod, we can use the projected token to add label in node named 'minikube' (causing security problem).

# patch succeed
curl -k -X PATCH \
-H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
-H "Content-Type: application/json-patch+json" \
-d '[
    {"op": "add", "path": "/metadata/labels/AAAA", "value": "BBBB"}
]' \
https://kubernetes.default.svc/api/v1/nodes/minikube

Step 2: add ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding

  • as there is only one node 'minikube', so set the goal: service account fluid-csi token in node named minikube can not modify node minikube
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: "test-node-policy"
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        # supported values: "*", "CONNECT", "CREATE", "DELETE", "UPDATE"
        operations: ["UPDATE"]
        resources: ["nodes"]
  matchConditions:
    - name: isRestrictedUser
      expression: request.userInfo.username == "system:serviceaccount:fluid-system:fluid-csi"
  variables:
    - name: userNodeName
      expression: >-
        request.userInfo.extra[?'authentication.kubernetes.io/node-name'][0].orValue('')
    - name: objectNodeName
      expression: >-
        object.?metadata.name.orValue('')
  validations:
    - expression: "variables.userNodeName != ''"
      message: "userNodeName is empty"
    - expression: "variables.objectNodeName != variables.userNodeName"
      messageExpression: >-
        "can not modify the node using the same node binding token"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: "binding-test"
spec:
  policyName: "test-node-policy"
  validationActions: [Deny]

Test2: do the same curl command as Test 1, the policy takes affect and the result is failed as below:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "nodes \"minikube\" is forbidden: ValidatingAdmissionPolicy 'test-node-policy' with binding 'binding-test' denied request: can not modify the node using the same node binding token",
  "reason": "Invalid",
  "details": {
    "name": "minikube",
    "kind": "nodes",
    "causes": [
      {
        "message": "ValidatingAdmissionPolicy 'test-node-policy' with binding 'binding-test' denied request: can not modify the node using the same node binding token"
      }
    ]
  },
  "code": 422
}

So I think the node binding token and ValidatingAdmissionPolicy can replace the kubelet config used in CSI Plugin DaemonSet.
If there are no other issues and requirements, I will develop the code later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
features features
Projects
None yet
Development

No branches or pull requests

3 participants