[ALS-6816] Data Dictionary Loader Job from HPDS #157
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This script creates necessary directories for Jenkins and log storage. It sets read/write permissions for these directories to facilitate development. The script will help streamline the configuration process for local environments.
Removed unnecessary '-i' flag from sed_inplace commands in update-jenkins.sh to streamline the syntax. Additionally, adjusted indentation within conditional blocks in the start-picsure.sh script for improved readability and consistency.
They get interpreted as literals
# Conflicts: # start-picsure.sh
…ALS-6816 # Conflicts: # start-picsure.sh
The configuration file had two identical JDK-21 entries, which was redundant. This change eliminates the duplicate entry, ensuring the config remains clean and efficient.
Specify the docker build platform as `linux/amd64` in relevant scripts to ensure consistent builds across different environments. Correct syntax errors related to exit codes by removing parentheses, ensuring proper script execution. Also, fix a typo in variable syntax within the `start-picsure.sh` script.
Introduce `generateEnv.sh` to automate `.env` file creation for the Data Dictionary, ensuring secure password generation. Also include an example `.env` file for reference and integrate the script into the dependency installation process.
Includes four new Jenkins jobs: Hydrate Data Dictionary Database, HPDS and Dictionary Data Loader, Build and Deploy Dictionary API, and Build Data Dictionary ETL. Each job automates key steps in the data pipeline, such as ETL, dataset loading, and API deployment.
Upgraded GitSCM plugin to v5.5.2 across jobs and updated workflow-cps and related plugins to newer versions. Enhanced build configurations by adding new parameters, stages, and deployment options, while standardizing submodule configurations. Minor text formatting and script adjustments were also implemented.
Gcolon021
commented
Dec 19, 2024
| @@ -76,17 +66,18 @@ docker run --name=wildfly --restart always --network=picsure -u root \ | |||
| -v $DOCKER_CONFIG_DIR/wildfly/passthru/:/opt/jboss/wildfly/standalone/configuration/passthru/ \ | |||
| -v $DOCKER_CONFIG_DIR/wildfly/aggregate-data-sharing/:/opt/jboss/wildfly/standalone/configuration/aggregate-data-sharing/ \ | |||
| -v $DOCKER_CONFIG_DIR/wildfly/visualization/:/opt/jboss/wildfly/standalone/configuration/visualization/ \ | |||
| -v $DOCKER_CONFIG_DIR/wildfly/deployments/:/opt/jboss/wildfly/standalone/deployments/ \ | |||
There was a problem hiding this comment.
We either need to mount this dir or copy in each of the wars.
Enhanced the Jenkins job to handle retries and failure scenarios when starting the DictionaryEtlApplication. Updated the initialization API call to include a JSON payload for improved flexibility. Added a script to optionally disable Jenkins security settings for easier development or testing scenarios.
This now uses a post request with the values in the request body.
Moved the .my.cnf configuration to occur after starting MySQL containers. If any files exist in the picsure-db directory before MySQL attempts to initialize it will not run its normal init scripts.
Updated the script to replace `export` with `local` for `rc_file` to improve scoping and avoid unintended side effects. Added `set_mysql_config_dir` calls with support for a second parameter to initialize MySQL configuration paths. These changes improve modularity and streamline configuration management.
Updated the `set_mysql_config_dir` function to include explicit checks for empty input values, ensuring fallback defaults are assigned appropriately. Added debug statements for better clarity during execution and commented out unused function calls at the script's end.
The `--platform=linux/amd64` argument was removed from Docker build commands to simplify configurations. Additionally, the Maven build step was removed from the Jenkins job, replaced with streamlined Docker build commands for HPDS and HPDS ETL. These changes improve maintainability and focus on the necessary build steps.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[ALS-6816] Data Dictionary Loader Job from HPDS
[ALS-5375] Configurable All-in-one to select public datasets of interest