-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
A feature we may provide to users in the future is servers with fast local (not
NFS) storage for use by tools. This would be something like user-store, except
per-server. This storage would be redundant (RAID), but not backed up.
If anyone feels like they may make use of this, it would be useful to know
exactly how you plan to use it, e.g. how much storage you would need and for
how long, as well as the sort of tasks it would be used for.
- river.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (FreeBSD)
iEYEARECAAYFAkzkYpAACgkQIXd7fCuc5vJDygCghiTRX+/t62T4aAudXo0MoX1F
hCcAn22PuG2jWQCUnirte70sUwx2A/j4
=qPZo
-----END PGP SIGNATURE-----
Hi all, I'm a new and very ntimidited toolserver user :-)
Presently I'm doing some "Hello world" tests only, as I told in a previuos
message, but I'd like to run from toolserver my pywikipedia bot, Alebot,
that is a rather busy one (
http://stats.wikimedia.org/wikisource/EN/BotActivityMatrix.htm).<http://stats.wikimedia.org/wikisource/EN/BotActivityMatrix.htm>
Alebot reads at intervals itwikisource RecentChanges, selects new edits by
type, contributor and namespace, reads new or edited pages and "does
things". Such things often are nothing, often imply ad edit of one
wikisource page, sometimes are more complex, implying some more
readings/editing of some different, related pages (max 3-4
readings/editing, reading/updating of local pickle files).
My question is: how many pywikipedia readings/editing per hour are a "heavy"
toolserver job?
Alex
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
In the past, our policy on SGE (batch job scheduling) was that it should only
be used for jobs which run once then exit, and that it should not be used for
jobs which run continuously.
This policy has now changed, and SGE can be used for all jobs, including
long-running jobs. This effectively obsoletes Phoenix and similar tools, since
SGE with cronsub provides the same functionality.
Because SGE makes it easier for us to scale the Toolserver cluster, as well as
making your tools run more efficiently (and therefore quicker), we recommend
that all tools be converted to SGE where possible. The wiki page now has a
quick-start section explaining how to do this:
<https://wiki.toolserver.org/view/Batch_job_scheduling#Quick_start>
(While SGE has a lot of features and can be complicated, we've tried to make it
easy to use for the typical cases; if something is not clear, please let us
know.)
- river.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (FreeBSD)
iEYEARECAAYFAkziD9YACgkQIXd7fCuc5vLmewCeLTy/5zyomUajAttXfPoG7j+e
vToAnRD5yVeL5iccv/SfzIgGicHSoAS0
=fV93
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
The webservers (wolfsbane and ortelius) have been added as SGE submit hosts,
which means you can submit jobs from these servers (e.g. from CGI scripts).
However, jobs will not run on these servers.
- river.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (FreeBSD)
iEYEARECAAYFAkzgHFgACgkQIXd7fCuc5vJMsQCbBJfWa0Ii8akJowaN+/zHhdlH
y8wAoKhGCpaPfCiZIhzfA4QQkHpiw/zf
=2g0+
-----END PGP SIGNATURE-----
For info;
If you're finding the 'setlicense' script is ending unexpectedly, please be aware that it likes you to type 'yes' rather than just 'y' in response to questions.
Ho hum. Another half hour well spent ;)
> From: "Matthew P. Del Buono" <mpdelbuono(a)gmail.com>
>
> I'm in the conceptual phase of a new bot but it requires a bit of
> processing. Unfortunately the only way this processing can be done is by
> writing out two small files.
>
> However, the processing should be very fast, and after it's done the
> files can be trashed. For that reason, I don't think it's very logical
> to tax the hard drive with this processing.
Hi Matthew,
In my experience, the use of temporary files is not that bad (for
example, gcc uses multiple temporary files in the course of compiling
a .c file). I once did a test build of the Linux kernel on a RAM disk
instead of the real disk and the difference in build time was less
than 1%. The buffer cache is large enough that your program never
reads them from disk anyway. The only downside to using real files
instead of a RAM disk is that the system will eventually flush real
files to disk for durability - but if you delete them quickly enough,
this is unlikely to occur very often.
--
Derrick Coetzee
User:Dcoetzee
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hello all,
I searched for a bit but can't really find any advice on this problem.
I'm in the conceptual phase of a new bot but it requires a bit of
processing. Unfortunately the only way this processing can be done is by
writing out two small files.
However, the processing should be very fast, and after it's done the
files can be trashed. For that reason, I don't think it's very logical
to tax the hard drive with this processing.
Instead I was hoping to find a small RAM disk on toolserver that I could
use which would act like a file system without the hard drive
performance issues. Unfortunately, I didn't find one.
Is there (a) a RAM disk-like option on the toolserver, (b) a way to
request that one be made, or (c) a reason I shouldn't even bother
looking into this and just use real files instead?
Thanks for any input,
- -- Shirik
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJM38o/AAoJEN2UI7T9ssUI4S0IAIXAXme2GkNCTnU7Tf/Nj0n0
U+JXRMsn1z6ac+8ZfV2nMydN1AF+n2eKYQDkhMQ3/YNi1lQlZD+t80Fpb1uUrawU
qHPwu/2kDqhIMrdpVUk8qCJXtM9f8ykC4z9gP3xtxCLL+mwjKLw/gZ+uwpFGfr6a
V6OoCbj2gRlsaq3cRpp58hRb+BHEC1gXkmLkidw8xCPagy5z351CYSBwynm7LZ/6
ePsBU3s32t8QxhQ7Oop3wbixQzQgmZuIOyb4YiF1rnZR9zB/iJZ7kmInZGjBRH/G
pu27alXV05At00+pqCjEYjkxUu7+t8YuBH8lisATpWVmaPRu9MPYiBiPvzPAiew=
=wZPL
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
Since no major issues have been raised regarding the conversion of nightshade
to Solaris, this has been schedule for the maintenance window on January 3rd,
2011. For more information, see
<http://lists.wikimedia.org/pipermail/toolserver-announce/2010-September/000…>
During the maintenance window, we will back up crontabs to
$HOME/crontab.nightshade, and then reinstall the OS. This will take about 30
minutes. After the reinstall, the system will be identical to willow.
Users who currently use willow for cron will not have to make any changes.
Cronjobs using SGE will automatically be scheduled on nightshade as appropriate.
SGE jobs which force "-l arch=lx24-amd64" will fail to run, since there will be
no Linux hosts left. If you currently do this, you should remove the arch
limit.
Users who have crontabs on nightshade should review the old crontab file (which
will be saved as $HOME/crontab.nightshade), then run this command to install
it:
$ crontab $HOME/crontab.nightshade
Note there are some differences between Linux and Solaris cron; in particular,
"/" syntax is not supported. See <https://wiki.toolserver.org/view/Solaris#cron>
for details.
Any issues which prevent your tools running on Solaris must be fixed before the
conversion date. If you need system changes for this, file a request in JIRA
and include [solaris] in the summary. We will not delay the conversion unless
serious unanticipated issues are discovered.
- river.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (FreeBSD)
iEYEARECAAYFAkzgMHQACgkQIXd7fCuc5vJNUACdFP+v/BB8GJ1ZXOuJYGK6FVEf
0I4AoKDS1TdQ3zevlHenuWEg6opJ2wZA
=hzAC
-----END PGP SIGNATURE-----
Hi
I tried to provide a download link to a complete OSM Render Stack in a
Virtual Machine on the toolserver. The files are located at
/mnt/user-store and symlinked into my public_html.
This works for the small files (md5sums) but the 3.1G VMDK File is not
accessible: http://toolserver.org/~mazder/osm_vm/
Is it possible to serve such a large file via http?
Peter
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
Users were supposed to received an automated mail 28 days before account expiry
(which is currently 1 Dec for most users), but due to an issue with the program
which sends the mails, they were not sent. The problem has been fixed and
users will receive another mail 7 days before expiry, but in the mean time you
may wish to run 'acctrenew' on willow now to renew your account.
- river.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (FreeBSD)
iEYEARECAAYFAkzcNuAACgkQIXd7fCuc5vKdHwCeOtktiQ2Oao8Au4OvNeyifQLY
SgcAnjp7Ed+vKy3DzKmb2yfcQv81cJ2k
=QZ/G
-----END PGP SIGNATURE-----