Hello all,
as you may noticed nightshade is not re-setup yet and also yarrow is not back for use. That have a reason: We decided to get rid of Solaris on the userland- server again, because no real solaris-expert is left in the root-group; but there are 2 Debian-users in the group so we choose Debian as new operating system. The original plan was to use yarrow as a playground and when everything is fine there to switch willow. For some unknown reason, nightshade lost its solaris- installation shortly after or during the last colo-visit. So the plan was changed in the following manner: We use yarrow as playground, but try to speed up everything, and when we fine with it, we switch *nightshade* over. I hope to make all changes to puppet (our server-managment- system) during the next 7 days, but it may take few days longer. If you are a Debian-user too, you can help me a little bit: Insert needed packages at [1] and help other users to find packages for their needed liberies. Willow will keep Solaris for several more months (if nothing bad happens with it), so there will be no problems with non-running tools. At first everything will keep running on willow and when you are ready you can switch over your tools to Debian (if you use SGE, that will be very easy because only 1 flag has to set – more details later) – for most tools there should be no (big) problems.
Just as a information for you what is going on at the moment.
Sincerley, DaB.
[1] https://wiki.toolserver.org/view/User:Dab/Debian-Packages
Zawartość nagłówka ["Followup-To:" gmane.org.wikimedia.toolserver.]
DaB. WP@daniel.baur4.info wrote:
--===============3758331293830646509== Content-Type: multipart/signed; boundary="nextPart2441111.Px5et4TFKU"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit
--nextPart2441111.Px5et4TFKU Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable
Hello all,
as you may noticed nightshade is not re-setup yet and also yarrow is not ba= ck=20 for use. That have a reason: We decided to get rid of Solaris on the userla= nd- server again, because no real solaris-expert is left in the root-group; but= =20 there are 2 Debian-users in the group so we choose Debian as new operating= =20 system. The original plan was to use yarrow as a playground and when everything is = fine=20 there to switch willow. For some unknown reason, nightshade lost its solari= s- installation shortly after or during the last colo-visit.
DaB,
Please let me know if you need some help with Solaris.
//Saper
DaB,
On Sun, Mar 4, 2012 at 4:04 PM, Marcin Cieslak saper@saper.info wrote:
Please let me know if you need some help with Solaris.
Let me know if you need Solaris or other help.
Thanks, Gerald
"DaB." WP@daniel.baur4.info wrote:
[...] If you are a Debian-user too, you can help me a little bit: Insert needed packages at [1] and help other users to find packages for their needed liberies. [...]
Does this mean that all the accumulated JIRA requests for Perl modules & Co. will get scrapped? :-) I believe we had several issues already in the past when the installed soft- ware differed between the Solaris servers. Which brings me to: Does anyone know an established format a) in which pro- jects could write down their requirements and b) that covers both Debian and Solaris? So when admins need to (re-)in- stall a server, they wouldn't have to guess which packages are (still) required, but could just collect all $HOME/.requirements for active accounts and when one of these could not be satisfied, there would also be a person to contact before tools get broken.
Tim
On Sun, Mar 4, 2012 at 22:05, Tim Landscheidt tim@tim-landscheidt.de wrote:
Which brings me to: Does anyone know an established format a) in which pro- jects could write down their requirements and b) that covers both Debian and Solaris? So when admins need to (re-)in- stall a server, they wouldn't have to guess which packages are (still) required, but could just collect all $HOME/.requirements for active accounts and when one of these could not be satisfied, there would also be a person to contact before tools get broken.
I assume this is one of the reasons to use puppet?
Puppet manifests can have comments and the roots could establish a standardized way of writing (inline) why a package is needed. (e.g., a) assumed to be widely used like sed, grep or b) needed by the roots or a process that doesn't belong to a particular user or MMP or c) needed by users/MMP foo, baz, bar or some combination of those.)
Of course most people (whether roots or users or anyone else) won't do a very thorough job of enumerating dependencies when installing, updating, hacking software unless they first do an installation of that version on a brand new Debian install with a limited set of installed packages. i.e. most people won't notice that a package is needed or not already picked up some other way until they see something is broken because it's missing.
I'm not sure if I like ~/.requirements (and maybe it could be something like ~/.package-depends instead?) in place of puppet but at least it could be used as a failsafe if a root wanted to check after installing a new box or before removing a package.
This all got me thinking: can SGE be told that a job needs a certain package, choose a box that has it already installed, and keep relevant stats so that roots know if a package should be installed somewhere else as well?
-Jeremy
Jeremy Baron jeremy@tuxmachine.com wrote:
Which brings me to: Does anyone know an established format a) in which pro- jects could write down their requirements and b) that covers both Debian and Solaris? So when admins need to (re-)in- stall a server, they wouldn't have to guess which packages are (still) required, but could just collect all $HOME/.requirements for active accounts and when one of these could not be satisfied, there would also be a person to contact before tools get broken.
I assume this is one of the reasons to use puppet?
Puppet manifests can have comments and the roots could establish a standardized way of writing (inline) why a package is needed. (e.g., a) assumed to be widely used like sed, grep or b) needed by the roots or a process that doesn't belong to a particular user or MMP or c) needed by users/MMP foo, baz, bar or some combination of those.)
Of course most people (whether roots or users or anyone else) won't do a very thorough job of enumerating dependencies when installing, updating, hacking software unless they first do an installation of that version on a brand new Debian install with a limited set of installed packages. i.e. most people won't notice that a package is needed or not already picked up some other way until they see something is broken because it's missing.
I'm not sure if I like ~/.requirements (and maybe it could be something like ~/.package-depends instead?) in place of puppet but at least it could be used as a failsafe if a root wanted to check after installing a new box or before removing a package. [...]
I wouldn't want to put the burden on the roots because that would mean that they have to mangle all the small notes that project X needs package Y into the puppet manifest which in most cases - as you noted - will not have much effect. If instead it'd be up to the project's developers, the workload lies with who has a) the information needed and b) the moti- vation.
On second thought though, I think ~/.something is too hidden. We already have the nice [[Template:Tool]] on the Toolserver Wiki; it would be much better to store the depen- dencies as fields thus encouraging developers to update (or create) their pages there and give them more "publicity".
Tim
Hello, At Monday 05 March 2012 14:59:20 DaB. wrote:
Does this mean that all the accumulated JIRA requests for Perl modules & Co. will get scrapped? :-)
Let me descript the situation a bit: For solaris we do not use the solatris- package-manager to install packages, but compile and package the stuff on our own. For Debian we plan to use the package-manager of Debian as far as possible. In puppet (our server-managment-programm) I have now a HUGE list of liberies and software to install for the solaris-server – but the names of the packages does not match with the debian-package-names of course. I would need weeks to figure out what belongs to what – I do not have to time (and to admit: also not the motivation) to do this. So my plan is the following: I will install some basic stuff (like perl, python, gcc, libcurl – you know) and everything that people put in the list [1] – that's it. Then when I finish with the boxes, I will allow you all to login. You can test then your stuff and if do not work because something is missing, you can request to install that. There will be no hurry because willow (with a working solaris) is still there. Over the time, more and more tools will move to the Debian-boxes and more and more debian-packages will be installed (what makes the moving of other tools even easier). At the end, only the following stuff will remain at willow: *tools of people who are <s>too lazy</s>too busy to move their stuff *tools which can not run on Debian for some reasons *tools of users who left the TS
I believe we had several issues already in the past when the installed soft- ware differed between the Solaris servers. Which brings me to: Does anyone know an established format a) in which pro- jects could write down their requirements and b) that covers both Debian and Solaris? So when admins need to (re-)in- stall a server, they wouldn't have to guess which packages are (still) required, but could just collect all $HOME/.requirements for active accounts and when one of these could not be satisfied, there would also be a person to contact before tools get broken.
That is a nice plan, but it would not work, because most users are not capable to tell what liberies they need. And it is BTW not a problem to have a libery installed that is not used (because we have enough free space for that).
Sincerely, DaB.
[1] https://wiki.toolserver.org/view/User:Dab/Debian-Packages
On 05/03/12 15:20, DaB. wrote:
In puppet (our server-managment-programm) I have now a HUGE list of liberies and software to install for the solaris-server – but the names of the packages does not match with the debian-package-names of course. I would need weeks to figure out what belongs to what – I do not have to time (and to admit: also not the motivation) to do this.
Is the list available somewhere, to crowdsource the mapping? (the plan of starting from basic things looks good, too)
At the end, only the following stuff will remain at willow: *tools of people who are <s>too lazy</s>too busy to move their stuff *tools which can not run on Debian for some reasons *tools of users who left the TS
Remember that some TS-specific programs like setpass or acctrenew can only work in Solaris. They don't seem hard to rewrite, but it's something to keep into account.
That is a nice plan, but it would not work, because most users are not capable to tell what liberies they need. And it is BTW not a problem to have a libery installed that is not used (because we have enough free space for that).
Some automatic non-exhaustive dependency check would be easy. ldd on elf files, grep import on .py ones, etc.
Hello, At Tuesday 06 March 2012 16:42:01 DaB. wrote:
On 05/03/12 15:20, DaB. wrote:
In puppet (our server-managment-programm) I have now a HUGE list of liberies and software to install for the solaris-server – but the names of the packages does not match with the debian-package-names of course. I would need weeks to figure out what belongs to what – I do not have to time (and to admit: also not the motivation) to do this.
Is the list available somewhere, to crowdsource the mapping? (the plan of starting from basic things looks good, too)
at the moment the puppet-stuff is not public, because it contains some passwords, but I published the software-files at [1] (only the parts for solaris is interesting here). Please only put packages at [2] that you realy need or where you are realy sure that other people will need it.
At the end, only the following stuff will remain at willow: *tools of people who are <s>too lazy</s>too busy to move their stuff *tools which can not run on Debian for some reasons *tools of users who left the TS
Remember that some TS-specific programs like setpass or acctrenew can only work in Solaris. They don't seem hard to rewrite, but it's something to keep into account.
yes.
Sincerley, DaB.
[1] http://toolserver.org/~dab/puppet/ [2] https://wiki.toolserver.org/view/User:Dab/Debian-Packages
"DaB." WP@daniel.baur4.info wrote:
[...]
I believe we had several issues already in the past when the installed soft- ware differed between the Solaris servers. Which brings me to: Does anyone know an established format a) in which pro- jects could write down their requirements and b) that covers both Debian and Solaris? So when admins need to (re-)in- stall a server, they wouldn't have to guess which packages are (still) required, but could just collect all $HOME/.requirements for active accounts and when one of these could not be satisfied, there would also be a person to contact before tools get broken.
That is a nice plan, but it would not work, because most users are not capable to tell what liberies they need. And it is BTW not a problem to have a libery installed that is not used (because we have enough free space for that). [...]
It's not only a question of space, but also of availability. I don't know about Debian or Solaris, but on Fedora some packages are sent to a farm up north when they don't compile in the current release and noone volunteers to fix it. I wouldn't want the admins in this case to work on such prob- lems if there is no existing demand.
I'm more optimistic regarding the abilities of the tool- server users - if someone can write "import x" or "use y;" they should be able to copy and paste that to another file. And if they don't, that'd be a nice opportunity to ask for and share some knowledge.
Tim
On 05/03/12 23:36, Tim Landscheidt wrote:
It's not only a question of space, but also of availability. I don't know about Debian or Solaris, but on Fedora some packages are sent to a farm up north when they don't compile in the current release and noone volunteers to fix it. I wouldn't want the admins in this case to work on such prob- lems if there is no existing demand.
I'm more optimistic regarding the abilities of the tool- server users - if someone can write "import x" or "use y;" they should be able to copy and paste that to another file. And if they don't, that'd be a nice opportunity to ask for and share some knowledge.
Tim
That assumes they know what they are doing, instead of blindly copy&pasting recipes. I know that's not true for many toolserver users, where we have very valued developers. But I'm sure there is also a bunch of people which copied pywikipediabot following some instructions, and are only experts on its command line. Don't misinterpret me, I'm not meaning that they shouldn't have an account in the TS, or that they are second-class users. It's good to have them on board, each one has its own know-how, very much as not every php coder is a perl guru. But not everyone has the experience to identify the dependencies of what they're doing.
And then, even for those writing their own scripts, having to copy the requisites to a separate file and keeping them up-to-date is a burdensome task. I prefer the autodetection way as far as it's possible (which in most cases looks simple).
Platonides platonides@gmail.com wrote:
It's not only a question of space, but also of availability. I don't know about Debian or Solaris, but on Fedora some packages are sent to a farm up north when they don't compile in the current release and noone volunteers to fix it. I wouldn't want the admins in this case to work on such prob- lems if there is no existing demand.
I'm more optimistic regarding the abilities of the tool- server users - if someone can write "import x" or "use y;" they should be able to copy and paste that to another file. And if they don't, that'd be a nice opportunity to ask for and share some knowledge.
That assumes they know what they are doing, instead of blindly copy&pasting recipes. I know that's not true for many toolserver users, where we have very valued developers. But I'm sure there is also a bunch of people which copied pywikipediabot following some instructions, and are only experts on its command line. Don't misinterpret me, I'm not meaning that they shouldn't have an account in the TS, or that they are second-class users. It's good to have them on board, each one has its own know-how, very much as not every php coder is a perl guru. But not everyone has the experience to identify the dependencies of what they're doing.
Now you've scared me :-). In an ideal world I'd solve this by packaging pywikipediabot in the usual way so that the users would just have to declare their dependency on it, and someone experienced could make sure that only a) one b) cur- rent and c) secure snapshot is used.
And then, even for those writing their own scripts, having to copy the requisites to a separate file and keeping them up-to-date is a burdensome task. I prefer the autodetection way as far as it's possible (which in most cases looks simple).
As long as the user can specify additional dependencies, sure, why not.
Tim
toolserver-l@lists.wikimedia.org