- Analytics-announce - lists.wikimedia.org

Find websites
by thomasfamily8583＠gmail.com 19 May '24

19 May '24

Find websites to thomasfamily8583(a)gmail.com

1 0

Decommissioning of the older analytics clients (stats servers) scheduled for May 28th
by Ben Tullis 07 May '24

07 May '24

Hello, We need to plan to decommission several of the analytics clients <https://wikitech.wikimedia.org/wiki/Analytics/Systems/Clients>, also referred to as the stats servers, since they have reached their end of service date. The servers in question are: * stat1004 <https://wikitech.wikimedia.org/wiki/Stat1004> * stat1005 <https://wikitech.wikimedia.org/wiki/Stat1005> * stat1006 <https://wikitech.wikimedia.org/wiki/Stat1006> * stat1007 <https://wikitech.wikimedia.org/wiki/Stat1007> If you actively use these servers, please consider moving your work to alternative stat servers (namely, stat10[08-11]) as soon as reasonably possible. Similarly, should you have personal files in your home directory on any of these servers that you would like to retain, now would be a good time to consider moving them to a different server, or moving them to your HDFS home directory. There are some guides available on syncing files between stats servers <https://wikitech.wikimedia.org/wiki/Analytics/Systems/Clients#Rsync_between…> and also using the hdfs CLI <https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster#How_do_I...> to manage files, which may help you to clean up the necessary files. We would like to be able to decommission these servers *three weeks' from today*, which is on *Tuesday May 28th*. Please do feel free to get back to us if you feel that this timescale will not allow sufficient time for you to migrate your work to alternative servers, or if you have any other concerns about this plan. Kind regards, Ben -- *Ben Tullis*(he/him) Senior Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

Enabling canary events for all MediaWiki event streams
by Andrew Otto 11 Dec '23

11 Dec '23

tl;dr Ignore this email if you do not use MediaWiki event streams. On Monday December 11 2023, all MediaWiki related event streams will have artificial canary events <https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams#Canary_Even…> injected into them. If you use any of these streams, you should discard thesee canary events. Add code to your consumers that discards events where meta.domain == "canary". Canary Events At WMF, we use artificial 'canary' AKA 'heartbeat' events <https://wikitech.wikimedia.org/wiki/Event_Platform/Stream_Configuration#can…> to differentiate between a broken event stream and an empty event stream. Canary events should be produced at least once an hour. If there are no events in a stream for an hour, then something is likely broken with that stream. These artificial canary events can be identified by the fact that their meta.domain field is set to "canary". If you use any of the streams listed below, you will need to add code that discards any events where meta.domain == "canary". Back in 2020, we began producing canary events into all new streams, but we never got around to enabling these for streams that already existed. We needed to ensure that all consumers of these streams filtered out the canary events. We're just finally getting around to enabling canary events for all streams. We will enable canary event production <https://phabricator.wikimedia.org/T266798> for the following streams on Monday, December 11th, 2023: - mediawiki.recentchange - mediawiki.page-create - mediawiki.page-delete - mediawiki.page-links-change - mediawiki.page-move - mediawiki.page-properties-change - mediawiki.page-restrictions-change - mediawiki.page-suppress - mediawiki.page-undelete - mediawiki.revision-create - mediawiki.revision-visibility-change - mediawiki.user-blocks-change - mediawiki.centralnotice.campaign-change - mediawiki.centralnotice.campaign-create - mediawiki.centralnotice.campaign-delete If you consume any of these streams, either external to WMF networks using EventStreams, or internally using Kafka, please ensure that your consumer logic discards events where meta.domain == "canary" before this date. (Note that not all of these streams are exposed publicly at stream.wikimedia.org <https://stream.wikimedia.org/?doc#/streams>.) Thank you, -Andrew Otto & the WMF Data Engineering team <https://wikitech.wikimedia.org/wiki/Data_Engineering> References - T266798 - Enable canary events for all MediaWiki streams <https://phabricator.wikimedia.org/T266798> - T251609 - Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events <https://phabricator.wikimedia.org/T251609>

1 1

Download Chrome Browser
by Amir Hossain 24 Oct '23

24 Oct '23

You're one step closer to bringing the ease, control and security of Chrome Browser to your company. Simply open this email on a desktop computer and visit https://chromeenterprise.google/browser/download/ to get started. Need help with Chrome Browser? Chrome Help Center is just a click away. Visit https://support.google.com/chrome to find answers to your technical support related questions. http://ww12.cosmeticsjoint.com

1 2

Re: Your message to analytics-announce@lists.wikimedia.org awaits moderator approval
by Amir Hossain 23 Oct '23

23 Oct '23

On Mon, 23 Oct, 2023 at 9:14 AM < analytics-announce-bounces(a)lists.wikimedia.org> wrote: > Your mail to 'analytics-announce(a)lists.wikimedia.org' with the subject > > > > Is being held until the list moderator can review it for approval. > > The message is being held because: > > Message has more than 10 recipients > Message has no subject > > Either the message will get posted to the list, or you will receive > notification of the moderator's decision. >

1 0

Ordered
by Amir Hossain 16 Oct '23

16 Oct '23

https://lists.wikimedia.org/postorius/lists/

1 0

New private, granular pageview dataset
by Hal Triedman 21 Jun '23

21 Jun '23

Hello world! My name is Hal Triedman, and I’m a senior privacy engineer at WMF. I work to make data that WMF releases about reading, editing, and other on-wiki behavior safer, more granular, and more accessible to the world using differential privacy <https://en.wikipedia.org/wiki/Differential_privacy>. Today I’m reaching out to share that WMF has released almost 8 years (from 1 July 2015 to present) of privatized pageview data <https://diff.wikimedia.org/2023/06/21/new-dataset-uncovers-wikipedia-browsi…>, partitioned by country, project, and page. This data is significantly more granular than other datasets we release, and should help researchers to disambiguate both long- and short-term trends within languages on a country-by-country basis — several <https://phabricator.wikimedia.org/T207171> long-standing requests <https://phabricator.wikimedia.org/T267283> from Wikimedia communities. Due to various technical factors, there are three distinct datasets: - 1 July 2015 – 8 Feb 2017 <https://analytics.wikimedia.org/published/datasets/country_project_page_his…> / README <https://analytics.wikimedia.org/published/datasets/country_project_page_his…> (publishing threshold [1]: 3,500 pageviews) - 9 Feb 2017 – 5 Feb 2023 <https://analytics.wikimedia.org/published/datasets/country_project_page_his…> / README <https://analytics.wikimedia.org/published/datasets/country_project_page_his…> (publishing threshold: 450 pageviews) - 6 Feb 2023 – present <https://analytics.wikimedia.org/published/datasets/country_project_page/> / README <https://analytics.wikimedia.org/published/datasets/country_project_page/00_…> (publishing threshold: 90 pageviews) API access to this data should be coming in the next few months. In the interim, I’ve built an example python notebook <https://public-paws.wmcloud.org/67457802/private_pageview_data_access.ipynb> illustrating how one might access the data in its current csv format, as well as several different kinds of simple analyses that can be done with it. I also want to invite the analytics community to join me for a brief demo of this project at the July Research Showcase <https://www.mediawiki.org/wiki/Wikimedia_Research/Showcase>. In the meantime, please feel free to reach out with any questions on the project talk page <https://meta.wikimedia.org/wiki/Talk:Differential_privacy>. For more information about WMF’s work on differential privacy more generally, see the differential privacy homepage on meta <https://meta.wikimedia.org/wiki/Differential_privacy>. And in the future, look for more announcements of privatized datasets on editor behavior, on-wiki search, centralnotice impressions and clicks, and more. Best, Hal [1] “Publishing threshold” is the minimum value of a row in the dataset in order to be published.

1 0

Scheduled downtime for the analytics mariadb replicas: 2023/05/09 at 09:30 UTC
by Ben Tullis 05 May '23

05 May '23

Hello, We need to schedule a reboot of the servers that provide copies of the Mediawiki databases for analytics purposes. https://wikitech.wikimedia.org/wiki/Analytics/Systems/MariaDB These are the servers: dbstore1003,dbstore1005, and dbstore1007. I'm intending to carry out this work at 09:30 UTC next Tuesday the 9th of May. I will restart all three servers in succession, so I expect the maintenance to be complete within approximately 30 minutes. Please note that the Wiki Replica databases are not affected by this maintenance: https://wikitech.wikimedia.org/wiki/Wiki_Replicas Please do let me know if you have any queries or if this choice of maintenance window is likely to cause you any inconvenience. Kind regards, Ben -- *Ben Tullis*(he/him) Senior Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

Scheduled downtime for Hadoop services today (2023/04/18) from 13:30 UTC
by Ben Tullis 18 Apr '23

18 Apr '23

Hello, Apologies for the short notice. The SRE team will be carrying out an upgrade of the switches in eqiad row D later today (https://phabricator.wikimedia.org/T333377) at approximately 14:00 UTC. The network outage to this row resulting from this work is expected to be around 30 minutes, all being well. In support of this work, the Data Engineering team will be putting HDFS file system into safe mode at approximately 13:30 today, which means that write operations to the cluster will be refused. Jobs sent to the YARN cluster will also be refused from around the same time, so please try to plan any work that you may have for the cluster to avoid this maintenance window. Read-only access to Hive, Presto, Superset, Turnilo, should continue to function normally throughout the maintenance window. Finally, two of the stats servers (stat1005 and stat1006) will be unavailable, so please save any work that you may have on these servers before the loss of connectivity. Please do reach out via any of the normal channels (email: analytics(a)lists.wikimedia.org , IRC: #wikimedia-analytics , Slack #data-engineering ) if you have any queries or concerns. Kind regards, Ben -- *Ben Tullis*(he/him) Senior Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/>

1 1

Scheduled downtime for Hadoop and analytics services (2023/04/04) from 12:30 UTC
by Steve Munene 04 Apr '23

04 Apr '23

Hello, The SRE team will be carrying out an upgrade of the switches in eqiad row C: (https://phabricator.wikimedia.org/T331882) at 13:00 UTC. The network outage to this row resulting from this work is expected to be around 30 minutes, all being well. In support of this work, the Data Engineering team will be putting HDFS file system into safe mode at approximately 12:30 UTC today, which means that write operations to the cluster will be refused. Jobs sent to the YARN cluster will also be refused from around the same time, so please try to plan any work that you may have for the cluster to avoid this maintenance window. Some additional internal-facing services for analytics such as Hive, Superset, Presto, and the Druid-analytics cluster will also be largely unavailable for some periods while the switch upgrade takes place. The public-facing Analytics Query Service (AQS) will continue to function, albeit with a degraded response to some queries. However Wikistats ( stats.wikimedia.org) will be unavailable whilst the switch upgrade is in progress. Finally, two of the stats servers, stat1007 and stat1009, will be unavailable, so please save any work that you may have on these servers before the loss of connectivity. Please reach out via any of the normal channels (email: analytics(a)lists.wikimedia.org , IRC: #wikimedia-analytics , Slack #data-engineering ) if you have any queries or concerns. Kind regards, Steve Munene

1 1

2024

2023

2022

2021

2020

Analytics-announce