Projects Incident Log

https://cheitgroup.atlassian.net/jira/software/c/projects/W4W/boards/120

Long page loading and problem with the server

21.06.2023

https://onelinemarketing.slack.com/archives/CQWNJ94M6/p1687346297968709
https://paste.pics/PSNWY

Alex Lozitskyi
Olga Gomon
Slava Moisieiev
Dima Sviridenko
Oleh Honcharuk
Ivan Atroshchenko

Boris Schaefer
Egzon Cekaj
Chiara Gragnano

Our steps:

• Review server settings and configurations.

• Check server logs for any anomalies or errors.

• Assess server performance and identify any potential issues.

• Examine the codebase for any errors or vulnerabilities.

• Collaborate with the team to brainstorm possible explanations and solutions.

Note: The same problem was a year ago due to the fact that they did not pay for hosting

1. add dmitriy.sviridenko@cheitgroup.com as admin to the account (not owner) he also will be aware.

2. pay for one year ahead 

3. put once a year 20.06 notification in calendar to check the server billing.

4. change owner to someone responsible for all servers/hosting/billings in company.

Casinor

NS name change was done 2 times and 301 redirects didn’t work for some time after launch that possibly lead to the rank drop.

1.09.2023

Hmm slightly confused here Kate as it seems there is non of the /go/urls pulled through into the new build?

Rankings and traffic is going down rapidly. Something is very wrong

Kate Hvozd

Alex Lozitskyi

Valeriy

Igor Korobeychenko (SEO)

Artem S (SEO)

Anna Nikitina (SEO) 

Dan, Mario

Our steps:

1. transferred 301 settings ASAP.

2. Full SEO audit with Google search console checkup.

3. Sending the client constant updates regarding our progress in searching the reasons.

4. Sending the client reports with analisys.

5. Checking the issue few days / weeks later

Note: similar rank drops were on the website a few times before the launch, so we are not sure that our mistakes realy influnced the ranking much.

1. After transfer check were the 301 settings tranferred

2. If the 301 were not tranferred – make the export import from stage carefully as the stage 301 settings will have stage domain “hardcoded”.

3. don’t change ns name more than 1 time during transfer.

4. If the website is SEO sensitive always provide SEO audit immediatelly after launch.

https://cheitgroup.atlassian.net/jira/software/c/projects/FGZ/boards/188

Stage indexing

18.09.2023

Egzon tagged Alex in slack

Anna Potiiuk
Alex Lozitskyi
Dima Nagaev
Oleh Honcharuk

Egzon Cekaj

• Figured out the reason why – when checking Page Speed the developer switched off htaccess and after forgot to switch on again.

• Checked all our stages to figure out if the same issue is reproducing.

Note: the same problem appeared on several staging site after checking

• Set the auto checking+closing each 15 min for all stages on our server

• If stage is on external server need to pay attention to this (PM+Developer)

Flytte

The emails are not sent

16.10.2023

We have an URGENT issue for flyttesiden and flytteskjema.

We have not received the emails with sign orders that should have been sent out 14th (Saturday)

Andrii Kupriianov, Mykola Pashchuck, Oleh Honcharuk

Mikael

Our steps:

• get access to the Google admin account (Workstation)

• setup new emails in workstation

• change SMTP settings

–

Chiidaspa

The main page lost the styles

31.10.2023

the Chiida Spa site is down: https://prnt.sc/LdPIYNS31oAF
Can you please check what the issue is and solve it?

https://cheitgroup.slack.com/archives/C018CAMQQ8G/p1698765620549709

Andrii Kupriianov
Valeriy Boyko

Danny Kurasinski

Our steps:

• Review cache settings

• Activate plugin Autoptimize

• Clear cache

–

Chiidaspa

Don’t work checkout on last step

02.11.2023

Booking Issue URGENT

Issue = https://prnt.sc/1-Njh6PANPT-
Link = https://chiidaspa.ch/termin-buchen/  

Andrii Kupriianov, Mykola Pashchuck

Danny Kurasinski

Our steps:

• understand how booking is formed (on js)

• disable js aggregation (because the order was collected on js)

• update the plugin and three lines to it

• write corrections in the template for the lines so that the form itself is not cached in any way

–

https://cheitgroup.atlassian.net/jira/software/c/projects/CC/boards/258

Broken styles on the new pages

14.03.2024

After upload of a new page to the live site, styles of that page got broken Screenshot

Daria Synenko Dmitriy Nagaev

Sam

Purge the server cache

After pushing each new page to live site, the styles on that page break, so after uploading to live, you need to clear the server cache on Cloudways.

https://cheitgroup.atlassian.net/jira/software/c/projects/FGZ/boards/188

WP-automatical update

10.04.2024

https://cheitgroup.slack.com/archives/C04G9107NER/p1712731382108639

https://prnt.sc/nJhONsmlPXak

Dmitiriy Nagaev
Pavel Trukhin

Luca Manz

1. Running a website test after the update

2. Turn off automatic updating

1. Raise a question about disabling automatic updates on Support projects.

2. Discuss this with the client on further projects at the beginning

Saclab

Incident #1 [Users can’t check out]

21.05.2024 1:49 pm

please check urgentlh it seems that users can’t check out

tested payment via Credit Card – failed. https://app.box.com/s/ko0a5y7s3jb2tk0tjvcigp77khuizazz
Affected methods reported by customers: CC, paypal, eventually Apple pay. Mollie Order ID note in failed order: https://app.box.com/s/3zgxnpefrkfdcuclmxrcyvpqlis8ex5o

Andrii Kupriianov Dima Savchenko (WP) Filipp Antonov

Thư Phan Alexander Böhringer

Deactivated and activated the Mollie plugin + updated the plugin.

Update the payment plugins as soon as a new version is released

Saclab

Incident #2 [Critical WP error on the front]

6.06.2024 3:52 pm

[Urgent] Critical WP error for client who wanted to schedule a pickup

SCLB-659

Andrii Kupriianov Dima Savchenko (WP)

Thư Phan Alexander Böhringer

update the cache

Purge the cache after update production

OneLine: Migrol

Incident #1 [Win email sent before winning]

30.05.2024 9:26 AM

(on live before launch)

“fyi: I don’t know if I should already start doing bugherd tasks, since you are also still testing, but I got the instant win email before I spun the wheel. The e-mail should be sent when I am on the instant prize page.”

Kate Hvozd val.b (Deactivated)

Artem Makarov

Pavlo Trukhin (Deactivated)

@Amire Berisha 

Rewrite the logic.

(PM captured this issue on the final tests, but developer said that it’s not easy to change the trigger, so we implemented some delay in email sending to cover this issue. However the client reported a bug anyway.)

If there are a few emails in the row – each should be tested separately to make sure it was trigerred by correct trigger (actition).

OneLine: Migrol

Incident #2 [some WP fields dissapered from theme settings because limits were reached]

31.05.2024 8:44AM

(on live before launch)

Kate Hvozd val.b (Deactivated)

Artem Makarov

Pavlo Trukhin (Deactivated)

@Amire Berisha 

Changed the limits in WP. (The issue apeared after we’ve implemented a few new fields).

Analyze the possible amount of fields and divide it accordinly to not exceed the limit or change the limit in advance.

OneLine: Migrol

Incident #3 [Links don’t work, the page reload on click – trigges a duplicate email sending]

3.06.2024 10:02 AM

(on live before launch)

•  IMPORTANT: ALL Links of the Prices don’t work. This Link here doesn’t work: https://prnt.sc/sU8p3Z5jI5JE It gets me over here

• Client called, they said they received 2 confirmation E-Mails when they won 5Rp, Mobility. Could you maybe check, I will as well

Kate Hvozd val.b (Deactivated)

Artem Makarov

Pavlo Trukhin (Deactivated)

@Amire Berisha 

Links fixed, so they stopped trigerring the page reload and duplicate email sending.

After each even small change the full functionality should be tested (even if it seems like the change is not connected to anything else. Including each email and their amaunt).

OneLine: Migrol

Incident #4 duplicated Reminder emails (and after fix appeared again)]

1.06 and 5.06

(on live before and after launch)

• 1.06 (before launch) – one more question: I did get the reminder Mail as well: https://prnt.sc/JJLDQGF5nhpcBut I did get like 3 of them, yours, mine and one with the name of the client. Do you know why?

• 5.06 (after launch) – Urgent Task again with the Reminder E-Mail. @Oliver Gehrig got like 12 Reminders: https://media.one-line.ch/QLqTXwA4

Kate Hvozd val.b (Deactivated)

Artem Makarov

Pavlo Trukhin (Deactivated)

@Amire Berisha 

• 1.06 (before launch) – changed the logic so users receive personalized emails and are o in the copy of each other

• 5.06 (after launch) – the fix made 1.06 was accidentally rewritten during some next transfer, so the fix was put back.

• Set up correct receivers for emails that are triggered by crons.

• Be careful with transfers to prevent the code rewriting

OneLine: Migrol

Incident #5 [Prize limits were reaching too fast as “-1” was on applied to more prizes then needed]

4.06.2024 4:37PM

(on live after launch)

 Car Wash for exmaple was won like more that 500 times today. In the backend I see this: https://prnt.sc/N5xSGxyHIAl_ and thats all of them.

Kate Hvozd val.b (Deactivated)

Artem Makarov

Pavlo Trukhin (Deactivated)

@Amire Berisha 

“-1” was applied to correct prize (not all the prizes form interests that were chosen)

Testing of the complicated functionality should include checkup of all the positive scenarios step by step. (To make sure that data saves correctely – accordingly to user actions)

OneLine: Migrol

Incident #6 [Sometimes some fields do not appear in the WP table (but saved in DB)

3.06 , 10.06.2024

(on live before and after launch)

3.06 – Question, all of the fields in the form are mandatory. But why was this data transferred from him to the backend? https://prnt.sc/ujBXgA1EzgVg 

10.06 –  I pulled out the lead list and in which ones the first or last name is missing.These are mandatory fields for us. Do you know why they are empty? https://prnt.sc/7haPj4nuYF7j

Kate Hvozd val.b (Deactivated)

Artem Makarov

Pavlo Trukhin (Deactivated)

@Amire Berisha 

• the first time it was reported there were a few such cases. We haven’t found the reason why it happened and were monitoring it.

• then we assumed that the code should be rewritten but it will be too dangerous, so we’ve just moved the data from DB to the table and DB.

After form testing – always make sure that all the fields data appeared in WP and emails.

Flytteskjema

Incident № 3. The emails are not sent

14.06.2024

We have an URGENT issue again

The sign orders (on email) stopped working once again….. We did not receive them for Flytteskjema

Andrii Kupriianov Dima Savchenko (WP)

Mikael Petrowski

Our steps:

• access to errors log on Flytteskjema server

• back to simple SMTP settings

The reason is the blocked email post@flytteskjema.no

need to investigate the reason of blocking and find the approach to avoid

One-Line: One-Line (support)

Broken animations around the website

12/07/2024 16:08 – 16:38

URGENT: on the whole website the animations are bugging and sometimes the user is not even able to scroll
Example: Screen Recording 2024-07-12 at 3.07.01 PM.mp4 Please fix asap, this can’t happen.

Anna Lukyan
Artem Gnibeda (Deactivated)

Illia Bilyk (Unlicensed)

Oliver Gehrig

Artem Gnibeda (Deactivated) reverted the changes implemented the day before (core dev and team lead were off that day)

1. Dev – test their changes, even though QA will test them later

2. QA – read comments more carefully

3. QA – even without additional comments, review the entire website after implementing those tasks where changes can affect more than just one page

4. PM – review the site after such changes as well

5. Dev – avoid manual deployments

Saclab

Incident #3 [Website is down for 2 hours]

20.08.2024 2:50 pm

could you check if the website is down? thanks

Andrii Kupriianov Dima Savchenko (WP) Roman Kliuiko

Thư Phan

Possible solutions:

1. Upgrade server plan, but let’s check several days the loading

2. Use security level in cloudflare during possible next downtime

Need more info, monitoring

Saclab

Incident #4 [Website has been down several times]

15.10.2024 11:49 am (Kyiv time)

the website was down again and some times was super slow for a brief moment. Could you please take a look and give me an update? thank you

Andrii Kupriianov Dima Savchenko (WP) Roman Kliuiko

Thư Phan

Solution: Upgraded server plan

Need more info, monitoring

Saclab

Incident #5 [Website has been down]

20.10.2024 3:38 pm (Kyiv time)

Website down

Dima Savchenko (WP)

Oliver

Solution: added new rule to the Confluence about user agent that attacked the website by link utm_source=google&utm_medium=sea&utm_campaign=fr_pmax_nbr_bags&gad_source=1&gclid=EAIaIQobChMI573P7rqciQMV9VcPAh3-ajmhEAQYByABEgJOZfD_BwE

Monitor new attacks and add new rules to the Confluence

Win4Win

Two step form is not working

27.11.2024
9:07 AM (Lisbon Time)

The form in french from tcs https://win4win.ch/fr/concours/concours-tcs/ is not working again, you can click on the first step button and nothing happend

alex.l@cheitgroup.com Anna Potiiuk Anna Aksonenko Pavlo Trukhin (Deactivated) Honcharuk Oleh

Chiara, Egzon, Labi

We are using here nonсes (for adding more security to cut off spam).
Before we excluded the nonces from caching, but they are still cached by the litespeed plugin, so the security check does not work.
Solution: The dev wrote all the nonces in the plugin in a more explicit way. According to the plugin documentation, it should work, but plugins sometimes just glitch, we cannot influence it on our part

Keep monitoring the form

Win4Win

Annoying recaptcha

27.11.2024
9:07 AM (Lisbon Time)

https://win4win.ch/fr/concours/concours-tcs/ – the rechaptcha is appearing, we whant to have this in the background

Anna Aksonenko

Chiara

Solution: reset recaptcha.
Here https://www.google.com/u/2/recaptcha/admin/site/713805202/settings
https://prnt.sc/mRUM1yVlFZ_s the Easiest for the users should be set

Specify with the client in advance before he creates the recaptcha, so he could set the Security level in settings.

Win4Win

Reminders sent a lot of times

26.11.2024
4:20 PM (Lisbon Time)

Some clients are complaining to receive the reminders 40 times. An other one said like 80 times. This one said even 150 times: https://media.one-line.ch/YPpmwivx

alex.l@cheitgroup.com Anna Aksonenko Honcharuk Oleh Pavlo Trukhin (Deactivated)

Chiara, Egzon, Labi

Solution:

• We have added a code that will prevent several crones from running in parallel, i.e. if one is already running, it will fully process the list, mark it processed, and only then the next crones will start and it will not get emails from the first crones again

• We also expanded the logs with a more accurate time mark and response status from Sendgrid. This is not directly related to resolving the problem, but it will provide more accurate information.

Test such user cases more, simulating a large number of simultaneous registrations.

OneLine: Play2win

Incident #1 [48h reminders stopped sending on competitions]

28.11.2024 3:35PM

(on live after launch)

client didn’t know about the issue (after win4win recent reminders bug we decided to fix Play2win bug quietly and track time to another task.)

Kate Hvozd

Anna Aksonenko (fixed)

Artur Bulan (main dev but was on the sick leave)

Artem Makarov

Pavlo Trukhin (Deactivated)

nobody

Request optimization: some checkups moved from code to DB request directly, so the data we receive became smaller and more actual.

Reason the issue appeared:

Request didn’t include strict timeframe (we were taking all old registrations created earlier than some date and later it lead to the accumulation) and in the code the registrations without partner ID were excluded (but they were not excluded from the request, so we received 1000+ rows, а потім у коді крутили їх виключаючи по одному).

If it’s a new code – the Code review was required and possibly there the logic problem could be captured.

Reminder log should be checked from time to time or additionally some monitoring with Slack notification applied. If the log was checked from time to time it should be possible to notice the gradual increase in reminders number.

(This issue couldn’t be captured by the one time testing)

Dan Aufhttps://cheitgroup.atlassian.net/jira/software/c/projects/HN/boards/124richtig: CasinoBonusDeals

Home page infinite redirect (other pages worked)

14.03.2025

Website is down

Roman Kliuiko

Honcharuk Oleh Eugene Rudyk

Dennis

Dan

Mario

https://jmp.sh/k8j6QCcT

(Custom SSL/TLS)
Changed the encryption mode that Cloudflare uses to connect to origin server.
From “Flexible” to “Full (Strict)”

task in Jira with full comment

It’s recommended to check the live site before making changes on the stage. If possible, review installed plugins that might affect the live site, even if they are only on the stage, as there was a suspicion this could have caused the issue.

https://cheitgroup.atlassian.net/jira/software/c/projects/HN/boards/124

All data from the files has disappeared on the country pages.

HN-371

25.09.2025

https://prnt.sc/za6IDY_rq9X4
Передавайте привіт розробнику. Без піздєца він не може. В сторінках країн зникли всі дані, які тягнулись з файлів.

Eugene Rudyk
Honcharuk Oleh

Client (Yehor Velychkovskyi)

Fixed the issue (was caused by the task that has beed implemented before –

HN-366

)

Що пропонуємо:

• Протягом наступних 2 місяців на всі завдання, де є функціональні зміни в коді (не стилістичні правки на кшталт розмірів шрифтів тощо), ми будемо залучати QA.

• QA перевірятиме не лише конкретну змінену таску, а й проходитиметься по основному функціоналу сайту.

• За цю додаткову роботу ми не будемо виставляти інвойс.

• Для якісного покриття нам буде потрібен від вас список ключових критично важливих сторінок/функціоналу, на які слід звертати особливу увагу під час перевірок.