ci/nightly: stabilize UrlChecker

The `urlcheck` linter is intended to check for existence
of all URLs embedded in the sources. There are in fact two
different CI jobs which use the linter. `TestNightlyLint`
is executed by `Lint URLs (Bazel)` whereas
`URL Check (Bazel)` runs the linter directly (via main).
Both jobs are part of the same `Nightlies` flow. While
the latter checks _all_ URLs, the former checks only
URLs found directly in SQL `helpMessages`.

In the past year, exceedingly more websites are
now behind WAFs, many of which deny even a HEAD
request. The recent fix in [1] addressed the user-agent
to be compliant with Wikipedia's WAF, which restored
the `Lint URLs (Bazel)` CI job. Meanwhile, `URL Check (Bazel)`
has been (silently) failing for several months.

This PR fixes `URL Check (Bazel)` by extending the linter
to use the Github APIs for _all_ github URLs [2]. We also
added a simple cache of checked urls, stored in a gcs bucket.
The cache has a 7-day TTL and avoids thousands of redundant
URL requests. (Upon expiry, _all_ URLs are rechecked.)
A number of broken as well as paywalled URLs have been
updated with the corresponding archive.org URL.
A small number of paywalled URLs have been added to the
ignore list.

[1] #152764
[2] https://github.com/orgs/community/discussions/159123

Resolves: #146319
Epic: none
Release note: None

Author: srosenberg

build/teamcity/cockroach/nightlies/url_check.sh

@@ -14,5 +14,5 @@ source "$dir/teamcity-support.sh"  # For $root
 source "$dir/teamcity-bazel-support.sh"  # For run_bazel
 
 tc_start_block "Run url check"
-run_bazel build/teamcity/cockroach/nightlies/url_check_impl.sh
+BAZEL_SUPPORT_EXTRA_DOCKER_ARGS="-e GITHUB_API_TOKEN -e GOOGLE_EPHEMERAL_CREDENTIALS -e TC_BUILD_BRANCH" run_bazel build/teamcity/cockroach/nightlies/url_check_impl.sh
 tc_end_block "Run url check"

build/teamcity/cockroach/nightlies/url_check_impl.sh

@@ -8,4 +8,7 @@
 
 set -xeuo pipefail
 
+echo "$GOOGLE_EPHEMERAL_CREDENTIALS" > creds.json
+gcloud auth activate-service-account --key-file=creds.json
+
 bazel run //pkg/cmd/urlcheck --run_under="cd $PWD && "

docs/RFCS/20160331_index_hints.md

@@ -84,7 +84,7 @@ More syntax and detailed information [here][1].
 PG does not provide support for hints. Instead they provide various knobs for
 tuning the optimizer to do the right thing. Details [here][2].
 
-[2]: http://blog.2ndquadrant.com/hinting_at_postgresql/
+[2]: https://web.archive.org/web/20221224144457/http://blog.2ndquadrant.com/hinting_at_postgresql 
 
 ### Oracle
 

docs/RFCS/20160425_drain_modes.md

@@ -239,6 +239,6 @@ being if that seems opportune.
 * Change the lease transfer mechanism so a transferrer can transfer its
   timestamp cache's high water mark which would act as the low water mark of the
   recipient's timestamp cache. This is conditioned on not [inserting reads in
-  the command queue](https://forum.cockroachlabs.com/t/why-do-we-keep-read-commands-in-the-command-queue/360).
+  the command queue](https://web.archive.org/web/20210419190146/https://forum.cockroachlabs.com/t/why-do-we-keep-read-commands-in-the-command-queue/360).
 
 # Unresolved questions

docs/RFCS/20170117_enterprise.md

@@ -137,5 +137,5 @@ and/or in an email to the cluster administrator (address provided when they
 register for the license).
 
 [#14114]: https://github.com/cockroachdb/cockroach/pull/14114
-[business model]: https://www.cockroachlabs.com/blog/how-were-building-a-business-to-last/
+[business model]: https://web.archive.org/web/20240529205150/https://www.cockroachlabs.com/blog/how-were-building-a-business-to-last/
 [settings table]: https://github.com/cockroachdb/cockroach/pull/14230

docs/RFCS/20170517_algebraic_data_types.md

@@ -604,4 +604,4 @@ const (
 
   [\#16240]: https://github.com/cockroachdb/cockroach/pull/16240
   [separate RFC]: https://github.com/cockroachdb/cockroach/pull/10055
-  [Clang]: https://clang.llvm.org/docs/LibASTMatchers.html
+  [Clang]: https://web.archive.org/web/20250902012257/https://clang.llvm.org/docs/LibASTMatchers.html 

docs/RFCS/20170908_sql_optimizer_statistics.md

@@ -538,7 +538,7 @@ efficient than retrieving each histogram separately).
   data retrieved by the query could also be used to update the
   statistics, though this complicates normal query execution. See also
   [self-tuning
-  histograms](https://ashraf.aboulnaga.me/pubs/sigmod99sthist.pdf).
+  histograms](https://web.archive.org/web/20240413134934/https://ashraf.aboulnaga.me/pubs/sigmod99sthist.pdf).
 
 * If the stats haven't been updated in a long time and we happen to be
   doing a full table-range scan for some query, we might as well

docs/RFCS/20171214_computed_columns.md

@@ -168,4 +168,4 @@ contains the expression used to compute the column otherwise.
 # Unresolved Questions
 [Partitioning]: https://github.com/cockroachdb/cockroach/blob/aa61db043e9c54c0b83a405cd76ce0ec7cc6a35d/docs/RFCS/20170921_sql_partitioning.md
 [Query planning changes in the RFC]: https://github.com/cockroachdb/cockroach/blob/aa61db043e9c54c0b83a405cd76ce0ec7cc6a35d/docs/RFCS/20170921_sql_partitioning.md#query-planning-changes
-[MySQL's extension]: https://dev.mysql.com/doc/refman/5.7/en/columns-table.html
+[MySQL's extension]: https://dev.mysql.com/doc/refman/8.4/en/information-schema-columns-table.html 

docs/RFCS/20180501_change_data_capture.md

@@ -707,7 +707,7 @@ an alias if we build support for CockroachDB to CockroachDB replication.
 [kafka]: https://kafka.apache.org/intro
 [other cdc syntaxes]: #appendix-other-cdc-syntaxes
 [proposed and sent through raft]: https://github.com/cockroachdb/cockroach/blob/381e4dafa596c5f3621a48fcb5fce1f62b18c186/docs/RFCS/20160420_proposer_evaluated_kv.md
-[schema resolution]: http://avro.apache.org/docs/current/spec.html#Schema+Resolution
+[schema resolution]: https://avro.apache.org/docs/++version++/specification/#schema-resolution 
 [sql table sink]: #sql-table-sink
 [system jobs]: https://github.com/cockroachdb/cockroach/blob/381e4dafa596c5f3621a48fcb5fce1f62b18c186/docs/RFCS/20170215_system_jobs.md
 [transaction grouped changes]: #no-transaction-grouped-changes

docs/RFCS/20181204_copysets.md

@@ -375,7 +375,7 @@ function. This design in this RFC is something simple and the respective
 algorithms can be tweaked independently later.
 
 ### Chainsets
-[Chainsets](http://hackingdistributed.com/2014/02/14/chainsets/) is one way
+[Chainsets](https://web.archive.org/web/20240714222832/https://hackingdistributed.com/2014/02/14/chainsets/) is one way
 to make incremental changes to copysets, but again potentially at the cost
 of reduced locality diversity. The length of the chain used in chainsets
 could be considered equivalent to replication factor in cockroach.