The German wikipedia was approached by freenet.de, who have been very kind hosting the German wikipedia CD image for us, about setting up a wikipedia.de mirror within their pages.
Nothing unusual there, but I had a thought: Wouldn't it be easier in the long run for us and other parties to * set up an additional MySQL replication slave at our site * probably without the passwords, or scramble them * have others (like freenet.de) replicate from that slave
All they'd have to do would be to set up a read-only MediaWiki, probably edit the skin to match their preferences, set the database to their replication server, and be done. They'd practically get a real-time mirror (maybe lacking a few minutes, but so what) without any maintenance.
Also, that would create an effective off-site backup of our db, very up-to-date, excluding the user passwords, which should be reconstructable from a "normal" dump for most users.
Alternatively, we could improve encryption of the passwords, and just have them replicate everything. Or, we just replicate the cur table (if that's possible) and write a "kiosk" mode into MediaWiki which ignores everything involving users, old versions, etc.
What do you think?
Magnus
On Fri, Oct 29, 2004 at 09:33:17AM +0200, Magnus Manske wrote:
Alternatively, we could improve encryption of the passwords, and just have them replicate everything. Or, we just replicate the cur table (if that's possible) and write a "kiosk" mode into MediaWiki which ignores everything involving users, old versions, etc.
I don't think that you are allowed to give the user-table away anyways. A kiosk would also make less traffic, and if you update only hourly you can skip many versions for often edited articles of fast deletitions.
Is the deletition log already a table?
ciao, tom
On Friday 29 October 2004 11:14, Thomas R. Koll wrote:
I don't think that you are allowed to give the user-table away anyways.
Of course you aren't.
On 10/29/2004 3:54 AM, NSK nsk2@wikinerds.org wrote:
On Friday 29 October 2004 11:14, Thomas R. Koll wrote:
I don't think that you are allowed to give the user-table away anyways.
Of course you aren't.
I don't think Magnus was referring to the mysql.user table.
Paul
Thomas R. Koll wrote:
I don't think that you are allowed to give the user-table away anyways. A kiosk would also make less traffic, and if you update only hourly you can skip many versions for often edited articles of fast deletitions.
Is the deletition log already a table?
Here's another thought (damn, more of these;-) : * have a PHP script in a cronjob (hourly, as you said), running on our to-be "outgoing replication master" that ** add/updates all changes to a cur table, since last "sync" ** removes all pages in the deletion log since last "sync", excluding those that have later dates in the add/update list above
So, not have a MySQL replication for out MySQL master, but rather out own software. That ensures * nothing we don't want can go through that door * it won't get out of sync even if the master is down for a while * probably requires less bandwidth/database stress than MySQL replication, since we would only sync the cur table * could serve as an update server for future CD/DVD-based wikipedia versions * could manage information about changed images (so these can be synced if needed) * could provide "reformatted" information like a list of authors for each article (more useful than a complete "old" table, and required by GFDL anyway)
Magnus
On Fri, Oct 29, 2004 at 12:37:05PM +0200, Magnus Manske wrote:
- it won't get out of sync even if the master is down for a while
and if the slave is down for a while it can d/l the archivied hourly or daily dumps.
- probably requires less bandwidth/database stress than MySQL
replication, since we would only sync the cur table
yes, files can be cached on the squids or redistributed.
- could provide "reformatted" information like a list of authors for
each article (more useful than a complete "old" table, and required by GFDL anyway)
For usage as a slave i wouldn't consider that as important, a link to wikipedia's history was and is enough in internet. But we could have a second output file which only carries the authors.
If we write our own software, we could use a smaller webserver for delivering the files (also for images), ain't there one in the kernel?
ciao, tom
Thomas R. Koll wrote: <snip>
Is the deletition log already a table?
In 1.4 they are (as well as protection, user rights ..).
Magnus Manske wrote:
- set up an additional MySQL replication slave at our site
- probably without the passwords, or scramble them
- have others (like freenet.de) replicate from that slave
I am in talks with a company (GuruNet) who would like to use our content via exactly this sort of system. They are willing to pay for this as a service, at a price which would reflect not just the cost to us, but also a generous goodwill contribution back to the community.
They actually called me yesterday just to let me know that they did a test with a remote slave to a mysql server (on their own network) and stress tested it. They said it worked just fine, which is an encouraging sign, although of course the devil is in the details.
--Jimbo
On Fri, Oct 29, 2004 at 08:42:08AM -0700, Jimmy (Jimbo) Wales wrote:
They actually called me yesterday just to let me know that they did a test with a remote slave to a mysql server (on their own network) and stress tested it. They said it worked just fine, which is an encouraging sign, although of course the devil is in the details.
Though I wouldn't prefer a mysql-only solution, it would increase security for our data/mysql-server if we maintain a single machine for only this purpose which would not include stuff like old and user tables. It ain't a good feeling for me to have outsiders have access to the same db where we store such sensible data.
ciao, tom
On 10/29/2004 12:23 PM, Thomas R. Koll tomk32@gmx.de wrote:
On Fri, Oct 29, 2004 at 08:42:08AM -0700, Jimmy (Jimbo) Wales wrote:
They actually called me yesterday just to let me know that they did a test with a remote slave to a mysql server (on their own network) and stress tested it. They said it worked just fine, which is an encouraging sign, although of course the devil is in the details.
Though I wouldn't prefer a mysql-only solution, it would increase security for our data/mysql-server if we maintain a single machine for only this purpose which would not include stuff like old and user tables.
This is a great suggestion. A simple way to achieve this would be to have a dedicated slave (S0) that (a) replicates from the master (M) and (b) acts as a master for the off-site slaves (S1, S2, S3...):
M -> S0 -> S1, S2, S3...
To limit which tables the off-site slaves can replicate, use replicate-do-table as necessary in S0's my.cnf:
replicate-do-table=wikipedia.cur
Without an intermediary slave, sensitive data could exist in the relay logs on the off-site slaves, even if the slaves were configured to ignore certain tables with replicate-ignore-table.
Paul
wikitech-l@lists.wikimedia.org