Hello,
I will upgrade our Gerrit from 3.4.8 to 3.5.4. I have scheduled it for tomorrow November 17th at 9:00 UTC after the backport window.
The service will be unavailable for a few minutes while the upgrade occurs.
Release notes: https://www.gerritcodereview.com/3.5.html Our task: https://phabricator.wikimedia.org/T307334
I think the upgrade broke something.
* Gerrit loads very slow. Not on all actions, but on some of them. * Half of the patches appear empty, as if there are no changes. I suspect this is some timeout. The list of files never loads. * Whenever I try to add or edit a comment it fails with an error message "An error occurred. Error 500 (Server Error): Internal server error. Endpoint: /changes/*~*/revisions/*/drafts/*. Alert: Unable to save draft".
Better rollback?
Best Thiemo
Thiemo:
I wasn't involved in the upgrade, and I also saw a slowdown, but once it fully deauthenticated me and I logged back in, errors disappeared for me (although I haven't checked everything works yet). Could you try that if that helps?
On Thu, Nov 17, 2022 at 10:45 AM Thiemo Kreuz thiemo.kreuz@wikimedia.de wrote:
I think the upgrade broke something.
- Gerrit loads very slow. Not on all actions, but on some of them.
- Half of the patches appear empty, as if there are no changes. I
suspect this is some timeout. The list of files never loads.
- Whenever I try to add or edit a comment it fails with an error
message "An error occurred. Error 500 (Server Error): Internal server error. Endpoint: /changes/*~*/revisions/*/drafts/*. Alert: Unable to save draft".
Better rollback?
Best Thiemo _______________________________________________ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Hi,
Please stand by, Antoine is working on it.
Cheers,
On Thu, Nov 17, 2022 at 10:53 AM Jaime Crespo jcrespo@wikimedia.org wrote:
Thiemo:
I wasn't involved in the upgrade, and I also saw a slowdown, but once it fully deauthenticated me and I logged back in, errors disappeared for me (although I haven't checked everything works yet). Could you try that if that helps?
On Thu, Nov 17, 2022 at 10:45 AM Thiemo Kreuz thiemo.kreuz@wikimedia.de wrote:
I think the upgrade broke something.
- Gerrit loads very slow. Not on all actions, but on some of them.
- Half of the patches appear empty, as if there are no changes. I
suspect this is some timeout. The list of files never loads.
- Whenever I try to add or edit a comment it fails with an error
message "An error occurred. Error 500 (Server Error): Internal server error. Endpoint: /changes/*~*/revisions/*/drafts/*. Alert: Unable to save draft".
Better rollback?
Best Thiemo _______________________________________________ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
-- Jaime Crespo http://wikimedia.org _______________________________________________ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Le 17/11/2022 à 10:57, Clément Goubert a écrit :
Hi,
Please stand by, Antoine is working on it.
Hello,
It is taking way longer than expected. During the upgrade the root partition ended up filed which caused multiple issues such as caches not being writable anymore, indexing of changes to the Lucene secondary indices failing.
I have thus stopped Gerrit entirely.
I have cleared a lot of files to free up disk (old kernels, obsolete core dumps etc). I am currently running a full reindexing of all 847549 changes to ensure they are properly known and up-to-date with the version expected by Gerrit 3.5.
An alternatively would have been to rollback to Gerrit 3.4 which would have required a full reindexing to downgrade the schema. I have chose to pursue the upgrade.
I don't have a good estimate at how long it will take. As I write this it has done 740k changes out of 847k (87%).
Antoine Musso
wikitech-l@lists.wikimedia.org