BUG: value too long for type character varying(100)

[tdruez]

While running a d2d pipeline on the following inputs:

https://github.com/tchiotludo/akhq/archive/refs/tags/0.25.1.zip#from
https://github.com/tchiotludo/akhq/releases/download/0.25.1/akhq-0.25.1-all.jar#to

Pipeline failure:

value too long for type character varying(100)

Traceback:
  File "/Volumes/nexB/repos/scancode.io/aboutcode/pipeline/__init__.py", line 199, in execute
    step(self)
  File "/Volumes/nexB/repos/scancode.io/scanpipe/pipelines/deploy_to_develop.py", line 145, in collect_and_create_codebase_resources
    pipes.collect_and_create_codebase_resources(self.project)
  File "/Volumes/nexB/repos/scancode.io/scanpipe/pipes/__init__.py", line 129, in collect_and_create_codebase_resources
    model_class.objects.bulk_create(batch, batch_size)
  File "/Volumes/nexB/repos/scancode.io/.venv/lib/python3.12/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/nexB/repos/scancode.io/.venv/lib/python3.12/site-packages/django/db/models/query.py", line 835, in bulk_create
    returned_columns = self._batched_insert(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/nexB/repos/scancode.io/.venv/lib/python3.12/site-packages/django/db/models/query.py", line 1875, in _batched_insert
    self._insert(
  File "/Volumes/nexB/repos/scancode.io/.venv/lib/python3.12/site-packages/django/db/models/query.py", line 1847, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/nexB/repos/scancode.io/.venv/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1836, in execute_sql
    cursor.execute(sql, params)
  File "/Volumes/nexB/repos/scancode.io/.venv/lib/python3.12/site-packages/debug_toolbar/panels/sql/tracking.py", line 235, in execute
    return self._record(super().execute, sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/nexB/repos/scancode.io/.venv/lib/python3.12/site-packages/debug_toolbar/panels/sql/tracking.py", line 160, in _record
    return method(sql, params)
           ^^^^^^^^^^^^^^^^^^^
  File "/Volumes/nexB/repos/scancode.io/.venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 122, in execute
    return super().execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/nexB/repos/scancode.io/.venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/nexB/repos/scancode.io/.venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/nexB/repos/scancode.io/.venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/nexB/repos/scancode.io/.venv/lib/python3.12/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/Volumes/nexB/repos/scancode.io/.venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/nexB/repos/scancode.io/.venv/lib/python3.12/site-packages/psycopg/cursor.py", line 97, in execute
    raise ex.with_traceback(None)

This is caused by the following file: to/META-INF/micronaut/io.micronaut.inject.BeanDefinitionReference/io.micronaut.http.client.$ServiceHttpClientConfiguration$ServiceSslClientConfiguration$DefaultTrustStoreConfiguration$Definition$Reference

The extension field value is provided as .$ServiceHttpClientConfiguration$ServiceSslClientConfiguration$DefaultTrustStoreConfiguration$Definition$Reference in the batch_create and fails since the extension field max length is 100.

from scanpipe.models import Project, CodebaseResource
project = Project.objects.all()[0]

relative_path = "to/META-INF/micronaut/io.micronaut.inject.BeanDefinitionReference/io.micronaut.http.client.$ServiceHttpClientConfiguration$ServiceSslClientConfiguration$DefaultTrustStoreConfiguration$Definition$Reference"

resource_data = {'type': CodebaseResource.Type.FILE, 'name': 'io.micronaut.http.client.$ServiceHttpClientConfiguration$ServiceSslClientConfiguration$DefaultTrustStoreConfiguration$Definition$Reference', 'extension': '.$ServiceHttpClientConfiguration$ServiceSslClientConfiguration$DefaultTrustStoreConfiguration$Definition$Reference', 'mime_type': 'inode/x-empty', 'file_type': 'empty', 'is_text': True, 'tag': 'to'}

codebase_resource = CodebaseResource(
    project=project,
    path=relative_path,
    **resource_data,
    )
codebase_resource.save()

[Avan2021]

Hi @tdruez, I want to work on this issue can you tell me how you would like this problem to be solved, do you want to extend the overall length or do you want to ensure that the extension value does not exceed 100 characters or we can also make it so that it only stores the last part since extension is meant to store file extensions or we can do both .

[mjherzog]

This problem also occurs with simpler SCIO pipelines like load_inventory - a recent example for this error message is:

id: None
name: core
type: npm
scope: dependencies
subpath:
version: 0.1.7::__archiveUrl=https%3A%2F%2Fnpm.pkg.github.com%2Fdownload%2F%40company%2Fcore%2F0.1.7%2Fefc49032392e34bae05e2aa8d661c2eaa3d5cec0
is_direct: True
is_pinned: True
namespace: @company
is_runtime: True
qualifiers:
is_optional: False
datasource_id: yarn_lock_v2
dependency_uid: pkg:npm/%40company/[email protected]::__archiveUrl%3Dhttps%253A%252F%252Fnpm.pkg.github.com%252Fdownload%252F%2540company%252Fcore%252F0.1.7%252Fefc49032392e34bae05e2aa8d661c2eaa3d5cec0?uuid=83a509ac-c3e6-4bfb-aaf2-e6791d5617ba
extracted_requirement: 0.1.7::__archiveUrl=https%3A%2F%2Fnpm.pkg.github.com%2Fdownload%2F%40company%2Fcore%2F0.1.7%2Fefc49032392e34bae05e2aa8d661c2eaa3d5cec0

"company" is a placeholder for a private repo

[Avan2021]

Hi @tdruez, following up on my earlier question about how you'd like this handled (extending the length, capping at 100 characters, storing only the last part, or a combination). Also, is bumping the length to 500 characters alright?