v0.110.0

[0.110.0] - 17-02-2025

Added

• [ALPHA] Added standardization of version and space for views in DMS data model under neat.prepare.data_model.standardize_space_and_version().
• The neat.to.excel(...) now supports passing a data model directly in the include_reference parameter.
• [ALPHA] Ability to subset data model to desired concepts (classes/views) via neat.subset.data_model endpoint
• Validation of DMS rules does not allow re-definition of any of the resources
• Validation of INFO rules does not allow re-definition of any of the resources
• Validation of DMS rules does not allow re-definition of any of the resources

Fixed

• In Model was not automatically set to True importing from spreadsheet missing value in this column
• Change the Instance source field on InformationProperty to be less restrictive. All URIs are now allowed.
• The neat.infer() maintains acronyms when renaming invalid properties/classes.
• neat.show.data_model() fixed for information rules

Changed

• Moved examples under neat.read.examples[nordic44, pump_example]
• [BREAKING] In Information rules, the Instance Source field is now a URIRef list instead of a RDFPath. In addition,
  the classes now also have an Instance Source column with the rdf:type of the class.

Improved

• The neat.read.excel(...) now gives more information about the location of the error in the Excel file.
• Automatic drop of rows in Excel rules if cells for critical columns are missing
• [BREAKING] multi value types are now serialized as a comma separated values (previous we used | as separator)
• The neat.infer() now is case-insensitive when inferring the data model. This is to match CDF's behavior.

[0.109.4] - 03-02-2025

Fixed

• The neat.create.enterprise_model() now sorts properties based on (view + property) in alphabetical order.

[0.109.3] - 03-02-2025

Added

• Support for standardizing of naming in the neat.prepare.data_model.standardize_naming() method.

Improved

• neat.inspect.issues() for errors in Metadata sheet and ill-formed views in te Views and Properties sheets
• neat.infer() now automatically makes the inferred classes and properties comply with the CDF naming conventions.

Fixed

• The neat.create + neat.to.excel(..., include_reference=True) now correctly includes the reference data model
  in the Excel file.

[0.109.2] - 31-01-2025

Added

• [ALPHA] Support for external modification of data model from NeatSession and its re-import
• Export of data model to Excel will now automatically hide the columns used for the internal neat processes.
• [ALPHA] when exporting data model to Excel one can specify to export only properties of views which are in the same space as the data model

Fixed

• The neat.create.data_product_model no longer includes properties pointing to views not in the model.

[0.109.1] - 28-01-2025

Fixed

• The neat.inspect.issue() now returns data model issues.
• Issue with setting new data model id for rules that have raw filter

[0.109.0] - 28-01-2025

Improved

• AML and DEXPI reader for neat session automatically perform extraction and transformation
• The get_cognite_client function no longer prints an irrelevant warning message when running outside a git
  repository.
• The neat.verify() no longer gives warnings about empty Cognite system containers.
• The neat.infer() is now much faster for large number of instances. (It now scales linearly with number of properties
  and not instances.). In addition, it supports inferring schema with an existing data model.

Changed

• [BREAKING] NeatSession.prepare.data_model.cdf_compliant_external_ids is moved under NeatSession.fix.data_model.cdf_compliant_external_ids
• [BREAKING] cdf_compliant_external_ids expects validated InformationRules as input instead of InformationInputRules
• [BREAKING] The neat.prepare.data_model.to_solution/to_enterprise/to_data_product methods are moved to
  neat.create.solution_model/enterprise_model/data_product_model. In addition, the methods been simplified with
  fewer parameters and better defaults.
• [BREAKING] NeatSession.prepare.data_model.prefix works only on verified Information and DMS rules, where in case of both rules are provided, it will use DMS rules
• [BREAKING] The neat.convert() no longer takes a target parameter. Only conversion from Information to DMS rules is
  supported.
• [BREAKING] The neat.prepare.data_model.drop(...) has been moved to neat.drop.data_model.views(...). In addition,
  the drop paramter has been split into view_external_id and group to differentiate between dropping individual
  views and groups of views.

Removed

• [BREAKING] NeatSession.prepare.dexpi and NeatSession.prepare.aml methods are removed. Use NeatSession.read.rdf.dexpi and NeatSession.read.rdf.aml instead.
• [BREAKING] NeatSession.prepare.data_model.cdf_compliant_external_ids

Fixed

• Fixed issue with not correctly set of max count when inferring properties which value type are multi type
• neat.read.cdf.classic.graph no longer requires read access to data sets in CDF.

Added

• Support for "on-disk" storage for oxigraph in NeatSession
• New method neat.inspect.views() to check the views in the data model.

[0.108.0] - 22-01-2025

Added

• Support RDF Datasets in NeatGraphStore enabling writing of sources triples to dedicated named graphs
• Support for classic graph using externalIDs neat.read.cdf.classic.graph(..., id="externalId")

Improved

• The neat.infer() no longer skips externalId/external_id properties when inferring the data model.
• Importing rules which were exported then modified externally then re-imported to the neat session
• The neat.read.cdf.classic.graph(...) gives an error is the root asset is not found in CDF.
• The neat.read.cdf.graph(...) method now shows progress bars for each view when loading large graphs.
• Initiating NeatSession will automatically select the best possible graph storage
• Form of internal neat data model
• The neat.read.cdf.classic.graph(...) now looks-up internal ids for external ids upon extraction instead of a
  separate step. This has a significant performance improvement for large graphs.

Changed

• The neat.read.cdf.graph(...) no longer extracts instances from cognite views by default. Instead, the parameter
  skip_cognite_views has been added to the method to allow for this.

[0.107.0] - 15-01-2025

Fixed

• The neat.prepare.instances.relationships_as_edges() no longer creates invalid identifiers for the edges.
• The neat.to.cdf.instances() can now create edges with properties.

Added

• Fast serialization of NeatGraphStore
• neat.to.session(...) and neat.from.session(...) methods to save and load a NeatSession object to and from a file.
• Internal support for remote oxigraph store
• Neat can now read a knowledge graph from DMS with the neat.read.cdf.graph(...) method.

Improved

• Better error message if NeatSession(..., storage="oxigraph") and the oxigraph package is not installed.
• NeatIDs are now human-readable and deterministic.
• The neat.convert() from DMS ot information now sets the container.usedFor to "node"/"edge"/"all" based on
  how the classes are implemented.

Changed

• [BREAKING] The Transformation column in the Information Data Model properties sheet has been renamed to
  Instnace Source.
• [BREAKING] The neat.convert() no longer has a property mode used to recognize classes that should be implemented
  as edges. Instead, information model to dms model conversion classes with both startNode and endNode properties
  are implemented as edges.

Removed

• [BREAKING] The neat.prepare.instances.classic_to_core() and neat.prepare.data_models.add_implents_to_classes()
  methods have been removed. These were only used for the classic to core flow. The simplified classic to core flow
  now solves this automatically.

[0.106.0] - 09-01-2025

Added

• Method for setting session client, neat.set.client(...).

Fixed

• neat.prepare.instances.make_connection_on_exact_match can now run multiple times.
• The mapping.data_model.classic_to_core method now produces a ..SourceSystem with a guid property that ensures
  that only instances written through this view are returned when querying it.
• The neat.read.cdf.classic.graph(...) now handles lack of access to CDF resources more gracefully.

Improved

• neat.infer(...) and neat.to.cdf.instances(...) now display a progress bar if the number of instances is large.

[0.105.2] - 08-01-2025

Added

• The get_cognite_client function uses the current working directory to store the environment file if neat
  is not run inside a git repository.

[0.105.1] - 08-01-2025

Added

• neat.to.cdf.instances() now automatically ignore relations above the DMS limit of 100. This is to ensure that
  the instances are successfully loaded into CDF. A warning is issued to the user if any relations are ignored.

Fixed

• The neat.prepare.instances.make_connection_on_exact_match no longer raises a SyntaxError when called.

[0.105.0] - 07-01-2025

Added

• All neat.read methods now supports reading from a URL.
• neat.prepare.instances.connection_to_data_type(...) added to convert connection to data type.
• neat.prepare.instances.classic_to_core() which bundles the preparation from classic to core.

Fixed

• neat.read.yaml(...) no longer raises a ValueError if a view is referencing an enum property in a container
  that is not part of the model.
• neat.read.yaml('....zip', format='toolkit') now correctly puts containers into a subfolder in the zip file.
• neat.to.cdf.data_model(...) now correctly handles list of json objects in the properties field of a view.

Improved

• neat.read.yaml(..., format="toolkit") no longer removes direct relations sources that points to views that are not
  part of the model.
• When using the neat.prepare.data_model.to_solution(...mode="read") and neat.prepare.data_model.to_data_product()
  methods, the newly created views inherits the filter from the source data model containers. This ensures that the new
  views will return the same instances as the source views.
• The neat.prepare.data_model.to_data_product(include="same-space") method no longer produces data models that uses
  values types that are not part of the data model. This ensures that the data model is self-contained. This is solved
  by dropping all properties that have a value type that is not part of the data model.
• The neat.to.cdf.instances(...) now correctly ignores read-only properties when creating instances in CDF.
• Reading sequences from CDF using read.cdf.classic.graph(...) now includes rows. In addition, the columns are
  now created as a list and not a blob.

Changed

• The neat.mapping.data_model.classic_to_core(...) now includes all connection properties from the view that
  it is implementing.

[0.104.0] - 20-12-2024

Improved

• When using a NeatSession object in a notebook. The return issues now has context for what actions they
  were caused by. In addition, the NeatSession object now ensures a linear provenance path for the
  data modeling.

Added

• Writing a model to Excel now includes the reference data model if the model was created from another model
  using the prepare.data_model.to_enterprise(...), neat.prepare.data_model.to_solution(...), or
  neat.prepare.data_model.to_data_product(...) methods.

[0.103.1] - 17-12-2024

Fixed

• Writing a model to Excel or YAML no longer skips specifying version and space of Views unless they both match.

[0.103.0] - 16-12-2024

Added

• Support for converting data types with neat.prepare.instances.convert_data_type(...).
• Support for converting data types to connection with neat.prepare.instances.property_to_type(...).

Fixed

• The neat.to.cdf.instances() now accounts for the dependencies between the views when
  creating the instances in CDF.
• Implementing a view with a reverse connection and not overwriting reverse connection from the parent,
  no longer returns a ReversedConnectionNotFeasibleError in neat.verify().
• Dumping any data model in neat now uses space and not prefix in the Metadata sheet.

Changed

• To the newest release of neatengine; v.2.0.3

[0.102.0] - 09-12-2024

Fixed

• Reading data models from CDF ensures unique names for enum collections.
• Edge creation

Improved

• AttachPropertyFromTargetToSource can now convert literal to URIRef
• Handling of undefined value type properties when loading instances to CDF

Added

• .prepare.instances.dexpi()
• .prepare.instances.aml()
• DEXPI onboarding tutorial
• AML onboarding tutorial

Changed

• The get_cognite_client now prioritize the env_file_name parameter over loading variables from the environment.

[0.101.0] - 06-12-2024

Added

• Support for reading a csv from any URL in neat.read.csv(...).

Fixed

• The neat.to.cdf.data_model() no longer fails to update space when existing='recreate' and drop_data=True.
  Instead, spaces are now updated, while containers are successfully deleted and recreated.

[0.100.1] - 05-12-2024

Fixed

• The neat.to.cdf.data_model() no longer raises a TypeError when the existing='recreate' parameter is used.

[0.100.0] - 04-12-2024

Added

• Support for converting relationships to edges.
• Support for setting limit when running neat.read.cdf.classic.graph(...).
• RegexViolation warning on Information rules that check if entity ids are DMS compliant
• Support for using parent property name in neat.mapping.classic_to_core(...).
• Classes that have properties but are not defined in Classes sheet will raise error
• Ability to make connection at granular level between Info and DMS rules when doing conversion
  form Info to DMS rules
• Ability to change names of properties and views / classes and still be able to load instances
• XMLReadAPI to extract triples from dexpi and aml files via NeatEngine in NeatSession

Changed

• NeatEngine version to the latest release that has a breaking change to the interface; 2.0.0
• NeatEngine version 2.0.1 with bugfix for getting catalog file from zipped folder
• [BREAKING ]The neat.to.cst.data_model(...) has been reworked. The new parameters drop_data and component
  ensures that containers and spaces with data are not accidentally deleted, while component allows
  for more granular control over what is deleted. The fallback_one_by_one parameter has been removed, and
  instead is now the default behavior.

Fixed

• Implementing a view with a reverse connection no longer raises a ReversedConnectionNotFeasibleError.

[0.99.1] - 28-11-2024

Changed

• Remove "make_compliant" from OWL and IMF importers
• Information rules post validation is now solely raising warnings
• Renamed neat.prepare.instances.relationships_as_connection to neat.prepare.instances.relationships_as_edges.
  Changed how the conversion is done. The edge is as a node between the start and end node.

Fixed

• neat.to.cdf.data_model(...) no longer tries to deploy Cognite Models.
• Wrong connection in neat.mapping.classic_to_core(...) is now fixed.

Improved

• Inference, instances are ordered by no of properties
• Inference, if Unknown value is present in multivalue property it is dropped

[0.99.0] - 26-11-2024

Changed

• We are now using RDF datasets (namedgraphs) instead of simple graphs, enabling support for multi space
  resolution of nodes and edges in CDF.

Added

• Support for dropping instances of a given type with neat.drop.instances(...)
• Helper methods neat.prepare.instances.relationships_as_connection and neat.prepare.data_model.prefix added
• The neat.verify(...) method now looks in CDF for the data model if it is not found in the NeatSession.

Fixed

• Running neat.to.cdf.data_model(..., dry_run=True) now counts created and deleted items correctly.
• Bug in inference of multi-value types
• Setting filter for a view in DMS sheet is no longer ignored.

Improved

• Bump dep on pyoxi to 0.4.3 and oxrdflib to 0.4.0
• Increase speed in loading RDF files in oxistore

Removed

• [BREAKING] The neat.read.cdf.classic.asset(...) is removed. Use the neat.read.cdf.classic.graph(...) instead.

[0.98.0] - 21-11-2024

Added

• Support for reading YAML
• Support for writing YAML in toolkit format.
• Added end-to-end test for CDM extension
• Support for dropping instances of a given type with neat.drop.instances(...)

Fixed

• Verification of reversed connections was done of property (linage) instead of view_property

Changed

• [BREAKING] Rules are simplified and normalized, especially metadata
• [BREAKING] removed reference, class (linage) and property (linage).
• [BREAKING] removed last, reference, auto-generated filters, data_model_type, schema_, extension

[0.97.3] - 16-11-2024

Improved

• ...make_connection_on_exact_match now takes strings instead of URIRefs

Fixed

• Smart filling in of container properties lead to bug (triggered by edge case)

[0.97.2] - 14-11-2024

Added

• Added parameter, model, to neat.to.excel(...) to allow to specify 'dms' or 'information' model.

Fixed

• neat.set.data_model_id() did not result in the verified data model being stored
• Due to issues with reverse connections neat.set.data_model_id() does not allow changing data model id-

Improved

• neat.to.cdf.instances() now auto creates space if one is not passed, also does not allow usage of data model space for instances storage
• neat.to.cdf.instances() no longer loads the data twice.
• neat.to.cdf.instances() now has a more informative output.
• neat.inspect.outcome.... reduced the amount of information shown in the output to make it more readable.

[0.97.1] - 14-11-2024

Changed

• neat.show.instances() now only works if NeatSession storage is set to oxigraph

Fixed

• neat.prepare.data_model.to_enterprise/to_solution was not picking up source entity

Changed

• lxml is now an optional dependency for DexpiExtractor. This is to support running in pyodide environment.

[0.97.0] - 14-11-2024

Added

• Added provenance on rules in NeatSession
• Option to move connections from reference to new model in DMS rules when generating Enterprise model
• Plotting of data model provenance
• Plotting of data model implements
• Support for loading NeatEngine.
• Support for inspecting outcome of neat.to.cdf.instances(...) with neat.inspect.outcome.instances(...).
• neat.prepare.instance.make_connection_on_exact_match added to enable adding connections
  between instances based on exact match of properties.
• Support for reading instances from csv neat.read.csv. Including reading csv from a public GitHub repository.

Improved

• Case-insensitive "direct" connection type in DMS Rules
• Validation over view types for connections in DMS Rules
• Validation of reverse connection feasibility in DMS Rules

Changed

• The neat.infer() now always infer integer and float as their 64-bit counterparts long and double. The motivation
  for this change is to have a more flexible data model that can handle 64-bit integers and floats.

[0.96.6] - 08-11-2024

Fixed

• neat.verify() no longer gives a PrincipleMatchingSpaceAndVersionWarning when you include views from
  the CogniteCore or CogniteProcessIndustry data models.
• In the DMSSheet you will now get a RowError if you try to set container or container property for
  an edge or reverse direct relation as these are not stored in containers.
• neat.read.excel(...) now correctly reads the Enum and Nodes sheets.
• In the DMSSheet, reverse relations no longer give a RowError if the reverse property is referencing
  a property in the reference sheets.

[0.96.5] - 07-11-2024

Fixed

• Serializing ResourceNotDefinedError class no longer raises a ValueError. This happens when a ResourceNotDefinedError
  is found, for example, when calling neat.verify().
• Setting neat.to.cdf.data_model(existing_handling='force) will now correctly delete and recreate views and containers
  if they already exist in CDF.

Improved

• When running neat.to.cdf.data_model() the entire response from CDF is now stored as an error message, not just the
  text.

Added

• neat.to.cdf.data_model() now has a fallback_one_by_one parameter. If set to True, the views/containers will
  be created one by one, if the batch creation fails.

[0.96.4] - 05-11-2024

Fixed

• neat.to.excel or neat.to.yaml now correctly writes ViewTypes and Edge that do not have the default
  value. For example, if the Connection was edge(direction=inwards) it would not be written to the Excel or
  YAML file as edge as direction=inwards was not the default value. This is now fixed.

[0.96.3] - 05-11-2024

Added

• Introduce neat.inspect.outcome(...) to check the outcome of cdf.to.data_model.

Fixed

• neat.to.cdf.data_model no longer outputs warnings when creating a new data model in CDF.

[0.96.2] - 05-11-2024

Added

• Can configure neat.to.cdf.data_model behavior for data model components that already exist in CDF

Changed

• When reading a data model from CDF, inwards edges are now treated as an edge with direction inwards and
  not the reverse edge.

[0.96.1] - 04-11-2024

Fixed

• naet.show working in a pyodide environment

[0.96.0] - 04-11-2024

Improved

• Handling of CDM extension
• Switched from Cytoscape to PyVis for data model and instances visualization

Added

• neat.prepare.reduce now support dropping individual views from a Cognite model.
• neat.set.data_model_id a convenience method to set the data model id in a NeatSession.
• neat.version returns the version of the neat package.
• neat.prepare.to_enterprise prepares template for creation of an enterprise model in Cognite Data Fusion.
• neat.prepare.to_solution prepares template for creation of a solution model in Cognite Data Fusion.

[0.95.0] - 29-10-2024

Fixed

• NeatSession subcommands no longer gives traceback for NeatSessionError exceptions, instead it
  gives a more user-friendly error message.

Improved

• Reduced matplotlib version to 3.5.2 due to PYOD compatibility issues
• Shorter and more concise summary of the data model in NeatSession

[0.94.0] - 29-10-2024

Added

• Support for information rules and instance plotting in NeatSession
• From Source to CDF tutorial

Improved

• Better plotting of rules for dms and information rules in NeatSession (accounts for subClassOf and implements)

[0.93.0] - 28-10-2024

Improved

• IODD extractor to also extract XML elements that should map to time series data type
• This includes extracting VariableCollection elements and ProcessDataIn elements
• Will probably need to revise how the tag/id of the time series is created
• Interfaces for InferenceImporter, IMFImporter and OWLImporter are leveraging BaseRDFImporter
• Renamed rules.examples to rules.catalog to start building catalog od data models
• Improved IMF rules that will handle IMF AttributeType onboarding
• Improved handling of unknown, multi-data, multi-objet and mixed value types in conversion from Information to DMS rules
• Reorg prefixes
• Added more detail regex testing of entities
• Transformation is now generated for every RDF based rules importer
• Improved session overview in UI

Added

• Added NeatSession
• Rules exporter that produces a spreadsheet template for instance creation based on definition of classes in the rules
• Rules transformer which converts information rules entities to be DMS compliant
• Rules transformer RuleMapping that maps rules from one data model to another
• Graph transformer SplitMultiValueProperty which splits multi-value properties into separate properties with single value
• Support for xsd:decimal which is now mapped to float64 in DMS rules
• Added RDF based readers for NeatSession
• NeatSession.read.examples.nordic44
• NeatSession.show.data_model show data model in UI

Removed

• State on DataType stored in _dms_loaded attribute

Changed

• Required env_file_name explicitly set in the get_cognite_client function. This is to avoid loading the wrong
  environment file by accident when running the function in a notebook.
• NeatIssue are no longer immutable. This is to comply with the expectation of Exceptions in Python.
• [BREAKING] All NEAT former public methods are now private. Only NeatSession is public.

[0.92.3] - 17-09-24

Fixed

• Prefixes not being imported or exported to Excel
• Trailing whitespace in Excel files causing issues with importing

[0.92.2] - 17-09-24

Added

• Method in InformationAnalysis which returns class URI based on class entity
• Method in InformationAnalysis which returns definition of property types for give class entity
• Allow different class entity in transformations then classes for which transformation are written(typical use case when we are renaming classes from source to target graph)

Improved

• Handling of namespace removal in Describe query (now only values which are of URIRef type or values of properties defined as object property get get namespace removed)

Removed

• logging from InformationAnalysis module

Fixed

• NEAT can now run in a minimal environment without raising a KeyError when using the default
  configuration in NEAT importers.
• NEAT now strips extra whitespace in all imported strings.

[0.92.1] - 12-09-24

Fixed

• The version of the released docker image was not updated correctly. This is now fixed.

[0.92.0] - 12-09-24

Added

• ClassicExtactor to extract all classic resource types from CDF from a given data set or root asset.

[0.91.0] - 11-09-24

Added

• IODDExtractor for IO-link standard: https://io-link.com/
• The extractor will parse XML files that follow the IO-link standard for an IODD device, and create rdf triples
  that will form the knowledge graph for the device.
• Improved XML utils method get_children to be able to extract nested children as well as direct children, and ignore
  namespace prefix of each tag element if the XML contains namespaces.