General info

 What needs to be improved?

• Structure of documentation (Conflunce + Figma) for filters in CRM Andrii Kupriianov
• Focus on correct texts (+ translations) in documentation before work with it in the sprint Andrii Kupriianov
• Figure out issues with time tracker Andrii Kupriianov

This document outlines the disaster recovery plan for an Elasticsearch cluster with 3 nodes. It covers procedures to ensure high availability, data protection, and recovery from failure scenarios.

Cluster Architecture Overview

• Total Nodes: 3 master-eligible nodes.

• Quorum Requirement: A majority of nodes (2 out of 3) must be available for the cluster to remain functional.

• Snapshots:

  • Configured for regular backups to a remote location (DigitalOcean Spaces)

  • Configured for regular backups of the virtual machines (droplets)

Common Failure Scenarios and Recovery Steps

1. One Node Fails

Impact:

• The cluster remains operational.

• Quorum is maintained (2 out of 3 nodes).

Recovery Steps:

1. Identify the failed node using monitoring tools (Kibana, Elastic Monitoring API).

   curl -u "user:password" -X GET "http://localhost:9200/_cat/nodes?v&pretty"

2. Investigate the issue (droplet failure, network outage,or container crash).

3. Restore the failed node:

   • Restore Droplet from the snapshot in DigitalOcean (if needed)

   • Restart the Elasticsearch container.

   • Verify the node rejoins the cluster using GET _cat/nodes.

4. Monitor shard reallocation to ensure data is rebalanced across nodes.

2. Two Nodes Fail

Impact:

• The cluster becomes unavailable.

• Quorum is lost, and no master node can be elected.

• Writes and reads are not possible until quorum is restored.

Recovery Steps:

1. Determine which nodes are offline.

2. Check the logs for errors docker logs <container>

3. Restore at least one of the failed nodes:

   • Restore Droplet from the snapshot in DigitalOcean (if needed)

   • Restart the nodes using docker-compose up.

4. Once at least 2 nodes are operational, verify the cluster state:

   • Use curl -X GET "http://127.0.0.1:9200/_cluster/health?pretty".

   • Ensure the cluster status moves from red to yellow or green.

5. Begin the re-indexing process in Elasticsearch:

   • connect to the API container in docker swarm cluster

   • execute the following commands:

     php artisan scout:import -c 500 Modules\\Profile\\Models\\Profile
     php artisan scout:import -c 500 Modules\\Vacancy\\Models\\Vacancy
     php artisan scout:import -c 500 Modules\\Publication\\Models\\Publication

6. Investigate and fix the cause of failure to prevent recurrence.

3. Data Loss or Corruption

Impact:

• Data stored in the cluster is partially or fully lost.

Recovery Steps:

1. Verify the availability of the latest snapshot:

   • Get the list available snapshots

     curl -u "user:password" -X GET "http://localhost:9200/_cat/snapshots/?v&s=id&pretty"

2. Restore the data:

   • Use the https://cheitgroup.atlassian.net/l/cp/EtjB1ZLS document for restoring

   • Specify indices if partial restoration is required.

3. Monitor the restoration process:

   • Use GET _cat/recovery to track shard recovery progress.

4. Begin the re-indexing process in Elasticsearch.

Restoring Data During Different Scenarios

1. Restore Data from an Elasticsearch Snapshot

• Only the Elasticsearch data is restored, without impacting the droplets themselves.

• This approach is useful for recovering data from logical issues or accidental deletion.

2. Restore Droplet

• Restores the entire virtual machine, including Elasticsearch configuration, data, and other services on the droplet.

• Always restore Elasticsearch snapshots before starting the service on the restored droplet to avoid inconsistencies. But, if the cluster detects that the restored node’s data is outdated, it will re-synchronize the latest data from other nodes to maintain consistency.

• Suitable for recovering from full droplet corruption.

3. Create a New Droplet and Add it to the Existing Cluster

• A new node can be added by copying the docker-compose.yml file from an existing node and updating the .env file.

• The new node will sync data and settings automatically from the cluster.

• Useful for scaling the cluster or replacing a permanently failed droplet.

RT-5995

Content

General info

The Applications page in the CRM displays a list of applicants for various job vacancies. One of the key features of this page is the Source column, which provides information about where the applicant originated. Currently, the source data is gathered from cookies, but this implementation is prone to inaccuracies. The new requirement is to fetch the Source directly from the referral link, ensuring the correct attribution of the applicant’s origin. This enhancement improves data reliability and provides better insights for recruiters.

User story

As a CRM user,
I want to see the accurate source of an applicant in the Source column on the Applications page,
so that I can understand where the applicant originated and optimize recruitment strategies accordingly.

Visual design:

https://www.figma.com/design/I5CXH7H3ICD0vfA1kPbcVf/Rocken-Design?node-id=58547-180231&t=NxR24hy7v0v3zo13-4

List all Indexes in Elasticsearch:

curl -u "user:password" -X GET "localhost:9200/_cat/indices?v"

Important: Elasticsearch cannot restore an open index with the same name. You must delete or rename the existing index before restoring.

Option 1: Close the Existing Index

Close the index to allow restoration:

curl -u "user:password" -X POST "http://localhost:9200/products/_close"

Option 2: Delete the Existing Index

Delete the index to make way for the restoration:

curl -u "user:password" -X DELETE "http://localhost:9200/products"

List All Available Snapshots

Verify all snapshots stored in the repository (for quick checks and operational overviews):

curl -u "user:password" -X GET "http://localhost:9200/_cat/snapshots/test-elasticsearch-space?v&s=id&pretty"

To retrieve detailed information about a specific snapshot

curl -u "user:password" -X GET "http://localhost:9200/_snapshot/test-elasticsearch-space/snapshot_20241211144811?pretty"

Restore the Snapshot

Restore a specific index from the snapshot:

curl -u "user:password" -X POST "http://localhost:9200/_snapshot/test-elasticsearch-space/snapshot_20241211144811/_restore" -H 'Content-Type: application/json' -d'
{
  "indices": "products"
}
'

Restore with a New Name

Restore the snapshot to a new index name:

curl -u "user:password" -X POST "http://localhost:9200/_snapshot/test-elasticsearch-space/snapshot_20241211144811/_restore" -H 'Content-Type: application/json' -d'
{
  "indices": "products",
  "rename_pattern": "products",
  "rename_replacement": "restored_products"
}
'

Verify the Restored Index

Verify that the restored index is available:

curl -u "user:password" -X GET "http://localhost:9200/restored_products/_search?pretty"

Step 1: Get All Items in Elasticsearch

List all Indexes in Elasticsearch:

curl -u "user:password" -X GET "localhost:9200/_cat/indices?v"

List all items in the specific Index:

curl -u "user:password" -X GET "localhost:9200/products/_search?pretty"

Step 2: Create a Snapshot

Create a snapshot of your Elasticsearch data:

time curl -u "user:password" -X PUT "http://localhost:9200/_snapshot/test-elasticsearch-space/snapshot_$(date +%Y%m%d%H%M%S)?wait_for_completion=true&pretty" -H 'Content-Type: application/json' -d'
{
  "indices": "*",
  "ignore_unavailable": true,
  "include_global_state": true
}
'

where:

• http://127.0.0.1:9200: The Elasticsearch endpoint.

• /_snapshot/test-elasticsearch-space: Specifies the registered snapshot repository (test-elasticsearch-space).

• /snapshot_$(date +%Y%m%d%H%M%S): Dynamically generates a unique snapshot name using the current date and time in the format YYYYMMDDHHMMSS. This ensures that snapshots have unique names.

• ?wait_for_completion=true: Ensures the operation waits until the snapshot creation is complete before returning a response.

• &pretty: Formats the JSON response for better readability.

Verifying Snapshots

List All Available Snapshots

curl -u "user:password" -X GET "http://localhost:9200/_cat/snapshots/test-elasticsearch-space?v&s=id&pretty"

Verify all snapshots stored in the repository (for detailed inspection and debugging):

curl -u "user:password" -X GET "http://localhost:9200/_snapshot/test-elasticsearch-space/_all?pretty"

To retrieve detailed information about a specific snapshot:

curl -u "user:password" -X GET "http://localhost:9200/_snapshot/test-elasticsearch-space/snapshot_20241211144811?pretty"

Verifying Elasticsearch Data in DigitalOcean Spaces

1. From the left-hand menu, click Spaces Object Storage under the Manage section.

2. Select the Space where ElasticSearch snapshots are stored

3. In the Space, open the directory configured as the base_path in Elasticsearch

4. Inside the base_path directory, look for snapshot files. These may include:

• metadata-*.dat files: Metadata for the snapshots.

• snapshot-*.dat files: The actual snapshot data.

• index-* files: Index-specific metadata.

Deleting a Snapshot

curl -u "user:password" -X DELETE "http://localhost:9200/_snapshot/test-elasticsearch-space/snapshot_20241211144811?pretty"

Framework: Playwright

Language: TypeScript

IDE: VSCode

1. Create a GitLab account and add an SSH key:

1. Register on GitLab

2. Create an SSH Key on Your Machine.
   macOS:

   ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

   a. Replace "your_email@example.com" with your email address.

   2. Press Enter to save the key in the default location.

   3. Enter a passphrase if required (or press Enter to skip).

   4. Verify the key creation:

      ls ~/.ssh

   5. Add the SSH key to the agent:

      ssh-add ~/.ssh/id_rsa

   6. View the public key:

      cat ~/.ssh/id_rsa.pub

   7. Copy the entire output.

3. For Windows:
   a. Open Git Bash and run the commands as described for macOS.

2. Add the SSH Key to GitLab

1. Log in to your GitLab account.

2. Navigate to Preferences > SSH Keys.

3. Click the Add new key.

4. Paste your copied SSH public key and click Add key.

3. Install Node.js

1. Download and install Node.js from the official website.

2. Verify the installation by running the following commands:

   node -v
   npm -v

4. Download and set up VSCode:

1. Download VSCode from the official website.

2. Install VSCode on your machine.

3. Create a folder to store the project files.

4. Open VSCode and select the created folder.

5. Verify you’re in the correct folder by running:

   ls

6. Initialize a Node.js project in the terminal:

   npm init

5. Initialize Git and Clone a Project from GitLab

1. Initialize Git in the project folder:

   git init

2. Clone the project repository:

   git@gitlab.cheitgroup.com:talent/tests/end-2-end.git

3. Install all project dependencies:

   npm install

4. If Playwright is not installed, run:

   npm init playwright@latest

5. If errors occur, force installation with:

   npm init playwright@latest --force

6. Main Git commands:

7. Git flow:

         (master)
            │
       git status
            │
            ▼
git checkout -b feature-branch    // create & switch to feature-branch
            │
            │  (Make your changes)
            │  git add .
            │  git commit -m "BranchName Your Comment"
            │  git push -u origin feature-branch
            ▼
     git checkout master
            │
       git pull         // update local master from remote
            │
       git merge feature-branch // merge happens in GitLab (MR)
            │
       git pull         // fetch any remote changes post-merge
            │
            ▼
       (master updated)

Content

General info

The Head of Recruiting role in the CRM system is designed to streamline the management of recruiters and their activities. This role grants the Head of Recruiting access to all candidates and allows them to manage recruiters effectively. The Head of Recruiting can view and edit recruiter information, while other CRM user categories are excluded from their scope.

User story

As a Head of Recruiting,
I want to have access to all candidates and manage recruiters exclusively,
so that I can ensure efficient recruitment processes and oversee the recruiters’ activities.

Scheme:

Step 1: Add DigitalOcean Spaces Credentials to Elasticsearch Keystore

Login to each node in the Elasticsearch Cluster and add DigitalOcean Spaces access and secret keys to the Elasticsearch keystore:

docker exec -it elasticsearch-docker-node-1 /usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.access_key
docker exec -it elasticsearch-docker-node-1 /usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.secret_key

Step 2: Restart Elasticsearch

Restart Elasticsearch for the changes to take effect:

curl -u "user:password" -X POST "http://localhost:9200/_nodes/reload_secure_settings" -H 'Content-Type: application/json' -d'
{
   "secure_settings_password":""
}'

or

docker-compose restart

Step 3: Register the Snapshot Repository

Register the snapshot repository by sending a PUT request:

curl -u "user:password" -X PUT "http://localhost:9200/_snapshot/test-elasticsearch-space?pretty" -H 'Content-Type: application/json' -d'
{
    "type": "s3",
    "settings": {
        "bucket": "test-elasticsearch-space",
        "base_path": "elk-snapshots",
        "endpoint" : "fra1.digitaloceanspaces.com"
    }
}
'

where:

• bucket – name of the storage bucket in DigitalOcean Spaces where snapshots will be stored.

• base_path – the directory path within the bucket where snapshots will be stored.

• endpoint – the regional endpoint for DigitalOcean Space

Expected output:

{
  "acknowledged" : true
}

Step 4: Verify Snapshot Repository

Check if the repository is registered:

curl -u "user:password" -X GET "http://localhost:9200/_snapshot?pretty"

Delete the Snapshot Repository in Elasticsearch

• all snapshot repositories:

  curl -u "user:password" -X DELETE "http://localhost:9200/_snapshot/*"

• specific repo:

  curl -u "user:password" -X DELETE "http://localhost:9200/_snapshot/test-elasticsearch-space"

Troubleshooting

• verify access and secret key:

  docker exec -it elasticsearch-docker-node-1 /usr/share/elasticsearch/bin/elasticsearch-keystore show s3.client.default.access_key
  docker exec -it elasticsearch-docker-node-1 /usr/share/elasticsearch/bin/elasticsearch-keystore show s3.client.default.secret_key

• mount a DigitalOcean Space to a Droplet, in order to check permissions and availability:

  • Install s3fs-fuse:

    sudo apt update
    sudo apt install -y s3fs
    s3fs --version

  • Store credentials in a file:

    echo "ACCESS_KEY:SECRET_KEY" > ~/.passwd-s3fs
    chmod 600 ~/.passwd-s3fs

  • Create a Mount Point:

    sudo mkdir /mnt/do-space

  • Mount the Space

    s3fs test-elasticsearch-space /mnt/do-space -o url=https://fra1.digitaloceanspaces.com -o use_path_request_style -o allow_other

  • Verify the Mount

    df -h
    ls /mnt/do-space
    mount

  • command for debugging:

    s3fs test-elasticsearch-space /mnt/do-space -o url=https://fra1.digitaloceanspaces.com -o use_path_request_style -o allow_other -o dbglevel=info -f -o curldbg

Content

General info

The Personal Information Tab within the Interview section of the Rocken Talent (RT) platform provides recruiters with key personal details about the candidate. This section includes marital status, number of children, residency status, and nationality. This information helps recruiters understand the candidate’s personal background in the context of potential employment opportunities.

User story

As a recruiter or hiring manager,
I want to view the candidate’s personal information,
so that I can better understand their personal background and consider any relevant factors during the recruitment process.

Visual design:

https://www.figma.com/design/I5CXH7H3ICD0vfA1kPbcVf/Rocken-Design?node-id=47279-155823&t=RtaKHj5McNNv0wg6-4