Quantcast
Channel: Delphi – The Wiert Corner – irregular stream of stuff
Viewing all articles
Browse latest Browse all 1440

The Delphi documentation site docwiki.embarcadero.com has been down/up oscillating for 4 days is now down for almost a day.

$
0
0

The [Wayback/Archive] Embarcadero/IDERA Documentation Wiki has been mostly down since March 3rd, 2022 (not the main page, but almost all other pages are).

I modified [Wayback/Archive] Docwiki https – EmbarcaderoMonitoring to show the actual status of a deeper page as the (mostly static) top page is up, so monitoring that is useless as the deeper pages are down.

The deeper pages are dynamic and require a functioning MySQL database connection. That connection is mostly down (the error message is not clear, so this could be a network or a database server problem, or maybe even a loadbalancer gradually entering bit heaven).

Since it had been down for like 6 days in February*, I’d expect Idera to keep an eye on it and prepare for more downtime. Apparently that’s either not a 24×7 thing for them or  they missed the “pre” in preparation as it is dead-silent on .

It also runs on an unsupported version of Mediawiki 1.31** which by itself does not explain the outage, but does indicate that their idea if handling their internal lifetime management is different than what they advocate to clients in their software subscription model, see [Wayback/Archive] Delphi – Embarcadero store, [Wayback/Archive] Update Subscription – Embarcadero and [Wayback/Archive] Special Offers on RAD Studio, Delphi & C++Builder – Embarcadero:

Can I obtain updates and hotfixes without a subscription?

An active update subscription is the only way to obtain updates and hotfixes.

Back in the days Embarcadero were keen in advocating life cycle management. Maybe time to show they indeed still understand what that means.

*

The original 7 day outage from 20220217-20220224 was reported at [Wayback/Archive] docwiki.embarcadero.com is not working – General Help – Delphi-PRAXiS [en].

I amended some information at [Wayback/Archive] docwiki.embarcadero.com is not working – Page 2 – General Help – Delphi-PRAXiS [en] and redirected a [Wayback/Archive] UptimeRobot | Free Website Monitoring link at [Wayback/Archive] Docwiki https – EmbarcaderoMonitoring.

A few hours later it sort of looked to be operational again, but in the monitor actually showed it was having outages every few minutes for intervals of 5-45 minutes: [Wayback/Archive] Docwiki https – EmbarcaderoMonitoring.

**

The source of the docwiki home page at [Wayback] docwiki.embarcadero.com/ shows this:

   <meta name="generator" content="MediaWiki 1.31.1">

Mediawiki 1.31.1 is not even the most recent LTS version 1.31.16 (see [Wayback/Archive] Release notes/1.31 – MediaWiki), so here might even be CVE vulnerabilities.

When choosing for LTS, then they should go to the latest of 1.35 (at the time of writing 1.35.5), see [Wayback/Archive] Release notes/1.35 – MediaWiki.

Analysing the backtraces

First outage from [Wayback/Archive] docwiki.embarcadero.com is not working – General Help – Delphi-PRAXiS [en] and [Wayback/Archive] Access to the docwiki – ADUG Forums:

Sorry! This site is experiencing technical difficulties.

Try waiting a few minutes and reloading.

(Cannot access the database)

Backtrace:

#0 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1134): Wikimedia\Rdbms\Database->reportConnectionError('Connection refu...')
#1 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/loadbalancer/LoadBalancer.php(749): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#2 /var/www/html/shared/BaseWiki31/includes/GlobalFunctions.php(2801): Wikimedia\Rdbms\LoadBalancer->getConnection(0, Array, false)
#3 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LCStoreDB.php(45): wfGetDB(-1)
#4 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(412): LCStoreDB->get('en', 'deps')
#5 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(458): LocalisationCache->isExpired('en')
#6 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(334): LocalisationCache->initLanguage('en')
#7 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(371): LocalisationCache->loadItem('en', 'magicWords')
#8 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(292): LocalisationCache->loadSubitem('en', 'magicWords', 'tree')
#9 /var/www/html/shared/BaseWiki31/languages/Language.php(3177): LocalisationCache->getSubitem('en', 'magicWords', 'tree')
#10 /var/www/html/shared/BaseWiki31/includes/MagicWord.php(352): Language->getMagic(Object(MagicWord))
#11 /var/www/html/shared/BaseWiki31/includes/MagicWord.php(280): MagicWord->load('tree')
#12 /var/www/html/shared/BaseWiki31/includes/parser/Parser.php(4848): MagicWord::get('tree')
#13 /var/www/html/shared/BaseWiki31/extensions/TreeAndMenu/TreeAndMenu_body.php(24): Parser->setFunctionHook('tree', Array)
#14 /var/www/html/shared/BaseWiki31/includes/Setup.php(948): TreeAndMenu->setup()
#15 /var/www/html/shared/BaseWiki31/includes/WebStart.php(88): require_once('/var/www/html/s...')
#16 /var/www/html/shared/BaseWiki31/index.php(39): require('/var/www/html/s...')
#17 {main}

Second outage

[8c1240dd3d6ee2e766657d26] /RADStudio/Alexandria/en/Main_Page Wikimedia\Rdbms\DBQueryError from line 1457 of /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php: A connection error occured.
Query: SELECT lc_value FROM `rad_alexandria_en_l10n_cache` WHERE lc_lang = 'en' AND lc_key = 'preload' LIMIT 1
Function: LCStoreDB::get
Error: 2006 MySQL server has gone away (etnadocwikidb01)
Backtrace:
#0 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1427): Wikimedia\Rdbms\Database->makeQueryException(string, integer, string, string)
#1 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1200): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#2 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1653): Wikimedia\Rdbms\Database->query(string, string)
#3 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1479): Wikimedia\Rdbms\Database->select(string, string, array, string, array, array)
#4 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LCStoreDB.php(52): Wikimedia\Rdbms\Database->selectField(string, string, array, string)
#5 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(357): LCStoreDB->get(string, string)
#6 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(271): LocalisationCache->loadItem(string, string)
#7 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(471): LocalisationCache->getItem(string, string)
#8 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(334): LocalisationCache->initLanguage(string)
#9 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(371): LocalisationCache->loadItem(string, string)
#10 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(292): LocalisationCache->loadSubitem(string, string, string)
#11 /var/www/html/shared/BaseWiki31/languages/Language.php(3177): LocalisationCache->getSubitem(string, string, string)
#12 /var/www/html/shared/BaseWiki31/includes/MagicWord.php(352): Language->getMagic(MagicWord)
#13 /var/www/html/shared/BaseWiki31/includes/MagicWord.php(280): MagicWord->load(string)
#14 /var/www/html/shared/BaseWiki31/includes/parser/Parser.php(4848): MagicWord::get(string)
#15 /var/www/html/shared/BaseWiki31/extensions/TreeAndMenu/TreeAndMenu_body.php(24): Parser->setFunctionHook(string, array)
#16 /var/www/html/shared/BaseWiki31/includes/Setup.php(948): TreeAndMenu->setup()
#17 /var/www/html/shared/BaseWiki31/includes/WebStart.php(88): require_once(string)
#18 /var/www/html/shared/BaseWiki31/index.php(39): require(string)
#19 {main}

Failing method is called by the class starting at [Wayback/Archive] …/mediawiki/blob/1.31.1/includes/cache/localisation/LCStoreDB.php – Line 28.

I have bolded the line numbers of the failurs.

Line# Code
28 class LCStoreDB implements LCStore {
29
30         /** @var string */
31         private $currentLang;
32         /** @var bool */
33         private $writesDone = false;
34         /** @var IDatabase */
35         private $dbw;
36         /** @var array */
37         private $batch = [];
38         /** @var bool */
39         private $readOnly = false;
40
41         public function get( $code, $key ) {
42                 if ( $this->writesDone && $this->dbw ) {
43                         $db = $this->dbw; // see the changes in finishWrite()
44                 } else {
45                         $db = wfGetDB( DB_REPLICA );
46                 }
47
48         $value = $db->selectField(
49                'l10n_cache',
50                'lc_value',
51                [ 'lc_lang' => $code, 'lc_key' => $key ],
52                __METHOD__
53         );
54
55         return ( $value !== false ) ? unserialize( $db->decodeBlob( $value ) ) : null;
56 }

The first method gets the database object from the loadbalancer and is in [Wayback/Archive] mediawiki/GlobalFunctions.php at 1.31.1 · wikimedia/mediawiki:

Line# Code
2775 /**
2776  * Get a Database object.
2777  *
2778  * @param int $db Index of the connection to get. May be DB_MASTER for the
2779  * master (for write queries), DB_REPLICA for potentially lagged read
2780  * queries, or an integer >= 0 for a particular server.
2781  *
2782  * @param string|string[] $groups Query groups. An array of group names that this query
2783  * belongs to. May contain a single string if the query is only
2784  * in one group.
2785  *
2786  * @param string|bool $wiki The wiki ID, or false for the current wiki
2787  *
2788  * Note: multiple calls to wfGetDB(DB_REPLICA) during the course of one request
2789  * will always return the same object, unless the underlying connection or load
2790  * balancer is manually destroyed.
2791  *
2792  * Note 2: use $this->getDB() in maintenance scripts that may be invoked by
2793  * updater to ensure that a proper database is being updated.
2794  *
2795  * @todo Replace calls to wfGetDB with calls to LoadBalancer::getConnection()
2796  * on an injected instance of LoadBalancer.
2797  *
2798  * @return \Wikimedia\Rdbms\Database
2799  */
2100 function wfGetDB( $db, $groups = [], $wiki = false ) {
2101         return wfGetLB( $wiki )->getConnection( $db, $groups, $wiki );
2102 }
2103
2104 /**
2105  * Get a load balancer object.
2106  *
2107  * @deprecated since 1.27, use MediaWikiServices::getDBLoadBalancer()
2108  * or MediaWikiServices::getDBLoadBalancerFactory() instead.
2109  *
2110  * @param string|bool $wiki Wiki ID, or false for the current wiki
2111  * @return \Wikimedia\Rdbms\LoadBalancer
2112  */
2113 function wfGetLB( $wiki = false ) {
2114         if ( $wiki === false ) {
2115                 return MediaWikiServices::getInstance()->getDBLoadBalancer();
2116         } else {
2117                $factory = MediaWikiServices::getInstance()->getDBLoadBalancerFactory();
2118                return $factory->getMainLB( $wiki );
2119         }
2120 }

The selectField method is at [Wayback/Archive] mediawiki/Database.php at 1.31.1 · wikimedia/mediawiki – Line 1446.

Related class documentation (with diagrams how they related to the rest of the Mediawiki system) is at:

Based on these and some of the requests succeeding, there seemed to be three possibilities:

  1. Sometimes the load balancer cannot get a database connection at all
  2. Sometimes the load balancer gets a valid connection and that connection then fails returning a query
  3. Sometimes the load balancer gets a valid connection and that connection succeeds returning a query

But then [Wayback/Archive] Remy Lebeau (of Indy fame) kicked in with another backtrace at [Wayback/Archive] docwiki.embarcadero.com is not working – Page 3 – General Help – Delphi-PRAXiS [en]:

It is not just the load balancer that is having issues.  For example, trying to access several pages today, I’m running into a new kind of error:

Wikimedia\Rdbms\DBQueryError from line 1457 of /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading?
Query: SELECT lc_value FROM `lib_sydney_en_l10n_cache` WHERE lc_lang = 'en' AND lc_key = 'deps' LIMIT 1
Function: LCStoreDB::get
Error: 1146 Table 'wikidb.lib_sydney_en_l10n_cache' doesn't exist (10.50.1.120)
 
Backtrace:
#0 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1427): Wikimedia\Rdbms\Database->makeQueryException(string, integer, string, string)
#1 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1200): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#2 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1653): Wikimedia\Rdbms\Database->query(string, string)
#3 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1479): Wikimedia\Rdbms\Database->select(string, string, array, string, array, array)
#4 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LCStoreDB.php(52): Wikimedia\Rdbms\Database->selectField(string, string, array, string)
#5 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(412): LCStoreDB->get(string, string)
#6 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(458): LocalisationCache->isExpired(string)
#7 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(334): LocalisationCache->initLanguage(string)
#8 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(371): LocalisationCache->loadItem(string, string)
#9 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(292): LocalisationCache->loadSubitem(string, string, string)
#10 /var/www/html/shared/BaseWiki31/languages/Language.php(3177): LocalisationCache->getSubitem(string, string, string)
#11 /var/www/html/shared/BaseWiki31/includes/MagicWord.php(352): Language->getMagic(MagicWord)
#12 /var/www/html/shared/BaseWiki31/includes/MagicWord.php(280): MagicWord->load(string)
#13 /var/www/html/shared/BaseWiki31/includes/parser/Parser.php(4848): MagicWord::get(string)
#14 /var/www/html/shared/BaseWiki31/extensions/TreeAndMenu/TreeAndMenu_body.php(24): Parser->setFunctionHook(string, array)
#15 /var/www/html/shared/BaseWiki31/includes/Setup.php(948): TreeAndMenu->setup()
#16 /var/www/html/shared/BaseWiki31/includes/WebStart.php(88): require_once(string)
#17 /var/www/html/shared/BaseWiki31/index.php(39): require(string)
#18 {main}

I only realised after searching for the table name [Wayback/Archive] lib_sydney_en_l10n_cache – Google Search that it was odd.

I could reproduce it as well at [Archive] Internal error – RAD Studio: XE8 main page:

[53d58941e2d881306538a66d] /RADStudio/XE8/en/Main_Page Wikimedia\Rdbms\DBQueryError from line 1457 of /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading?
Query: SELECT lc_value FROM `rad_xe8_en_l10n_cache` WHERE lc_lang = 'en' AND lc_key = 'deps' LIMIT 1
Function: LCStoreDB::get
Error: 1146 Table 'wikidb.rad_xe8_en_l10n_cache' doesn't exist (10.50.1.120)
Backtrace:

#0 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1427): Wikimedia\Rdbms\Database->makeQueryException(string, integer, string, string)
#1 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1200): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#2 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1653): Wikimedia\Rdbms\Database->query(string, string)
#3 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1479): Wikimedia\Rdbms\Database->select(string, string, array, string, array, array)
#4 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LCStoreDB.php(52): Wikimedia\Rdbms\Database->selectField(string, string, array, string)
#5 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(412): LCStoreDB->get(string, string)
#6 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(458): LocalisationCache->isExpired(string)
#7 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(334): LocalisationCache->initLanguage(string)
#8 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(371): LocalisationCache->loadItem(string, string)
#9 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(292): LocalisationCache->loadSubitem(string, string, string)
#10 /var/www/html/shared/BaseWiki31/languages/Language.php(3177): LocalisationCache->getSubitem(string, string, string)
#11 /var/www/html/shared/BaseWiki31/includes/MagicWord.php(352): Language->getMagic(MagicWord)
#12 /var/www/html/shared/BaseWiki31/includes/MagicWord.php(280): MagicWord->load(string)
#13 /var/www/html/shared/BaseWiki31/includes/parser/Parser.php(4848): MagicWord::get(string)
#14 /var/www/html/shared/BaseWiki31/extensions/TreeAndMenu/TreeAndMenu_body.php(24): Parser->setFunctionHook(string, array)
#15 /var/www/html/shared/BaseWiki31/includes/Setup.php(948): TreeAndMenu->setup()
#16 /var/www/html/shared/BaseWiki31/includes/WebStart.php(88): require_once(string)
#17 /var/www/html/shared/BaseWiki31/index.php(39): require(string)
#18 {main}

and at [Archive] Internal error – RAD Studio Code Examples: Alexandria main page:

[e4184052a4f712d7d81e42b1] /CodeExamples/Alexandria/en/Main_Page Wikimedia\Rdbms\DBQueryError from line 1457 of /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading?
Query: SELECT lc_value FROM `cde_alexandria_en_l10n_cache` WHERE lc_lang = 'en' AND lc_key = 'deps' LIMIT 1
Function: LCStoreDB::get
Error: 1146 Table 'wikidb.cde_alexandria_en_l10n_cache' doesn't exist (10.50.1.120)
Backtrace:
#0 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1427): Wikimedia\Rdbms\Database->makeQueryException(string, integer, string, string)
#1 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1200): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#2 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1653): Wikimedia\Rdbms\Database->query(string, string)
#3 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1479): Wikimedia\Rdbms\Database->select(string, string, array, string, array, array)
#4 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LCStoreDB.php(52): Wikimedia\Rdbms\Database->selectField(string, string, array, string)
#5 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(412): LCStoreDB->get(string, string)
#6 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(458): LocalisationCache->isExpired(string)
#7 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(334): LocalisationCache->initLanguage(string)
#8 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(371): LocalisationCache->loadItem(string, string)
#9 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(292): LocalisationCache->loadSubitem(string, string, string)
#10 /var/www/html/shared/BaseWiki31/languages/Language.php(3177): LocalisationCache->getSubitem(string, string, string)
#11 /var/www/html/shared/BaseWiki31/includes/MagicWord.php(352): Language->getMagic(MagicWord)
#12 /var/www/html/shared/BaseWiki31/includes/MagicWord.php(280): MagicWord->load(string)
#13 /var/www/html/shared/BaseWiki31/includes/parser/Parser.php(4848): MagicWord::get(string)
#14 /var/www/html/shared/BaseWiki31/extensions/TreeAndMenu/TreeAndMenu_body.php(24): Parser->setFunctionHook(string, array)
#15 /var/www/html/shared/BaseWiki31/includes/Setup.php(948): TreeAndMenu->setup()
#16 /var/www/html/shared/BaseWiki31/includes/WebStart.php(88): require_once(string)
#17 /var/www/html/shared/BaseWiki31/index.php(39): require(string)
#18 {main}

Look at the table names:

  • lib_sydney_en_l10n_cache
  • rad_xe8_en_l10n_cache
  • cde_alexandria_en_l10n_cache

They have three different prefixes (lib, rad and cde) and three different product codes (sydney, xe8 and alexandria).

This looks like a a setup with each product version having at least three different Wikimedia databases each having an l10n_cache table. Searching for [Wayback/Archive] l10n_cache – Google Search resulted in [Wayback/Archive] Manual:l10n_cache table – MediaWiki (which describes the table for all ranges of Mediawiki versions) and a whole load of pages with various circumstances in which people bump into missing this table.

Then I looked into the current status monitor to see is anything substantial had changed since yesterday. And it did.

Current status

At [Wayback/ArchiveDocwiki https – EmbarcaderoMonitoring – docwiki https it looks someone started working on it almost 15 hours ago:

Response Time Last 2 days

 image.thumb.png.43a4fb1e533aac3e5f272b80bb3c96c2.png

5012.65ms / 10372.00ms / 214.00ms

Avg. response time / Max. response time / Min. response time

Recent events

Down for 14 h, 40 min

The reason is Internal Server Error.500
Details:The server encountered an unexpected condition that prevented it from fulfilling the request.
March 7, 2022, 17:57 GMT +00:00

Running again

March 7, 2022, 17:46 GMT +00:00

I really really hope they know what they are doing, as right now the databases don’t look well and things have not improved for more than 15 hours (I was interrupted while writing this reply).

Opinion

That their largest and most important site is still failing and there is no communication from Embarcadero on social media or 3rd party forums (they do not even have their own [Wayback/Archive] Embarcadero Discussion Forums and direct to 3rd parties at [Wayback/Archive] Community | Embarcadero RAD Studio, Delphi, & C++Builder Blogs) is inexcusable.

I searched social media. The only query returning anything useful was [Wayback/Archive] docwiki – Twitter Search / Twitter which only this Embarcadero response:

[Wayback/Archive] MarcoCantu on Twitter: “Yes, there is a technical issue with docwiki server our IT team is looking into” / Twitter (20220216)

Related

–jeroen

 


Viewing all articles
Browse latest Browse all 1440

Trending Articles