For Java, we run an instance of Archiva: https://archiva.wikimedia.org/

It's not a perfect approach but I think we can and should move in that direction with all our other ecosystems (python, Javascript, PHP).  Our reduction in security-relevant surface area alone would be worth it.

On Fri, May 5, 2023 at 11:18 AM Kosta Harlan <kharlan@wikimedia.org> wrote:
Tangent: is it worthwhile to establish a consensus for best practices with package pinning and package management for Python projects in the Wikimedia ecosystem? When I last worked on a python project (https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing that we have so many different tools and approaches for doing these things, and seems like we'd benefit from having a standard, supported way. (Or maybe that already exists and I haven't found it?)

Kosta

On 5. May 2023, at 13:51, Slavina Stefanova <sstefanova@wikimedia.org> wrote:

Poetry is a modern lockfile-based packaging and dependency management tool worth looking into. It also supports exporting dependencies into a requirements.txt file, should you need that (nice if you want to containerize an app without bloating the image with Poetry, for instance). 

https://python-poetry.org/ 

--
Slavina Stefanova (she/her)
Software Engineer - Technical Engagement

Wikimedia Foundation


On Fri, May 5, 2023 at 1:38 PM Sebastian Berlin <sebastian.berlin@wikimedia.se> wrote:
A word of warning: using `pip freeze` to populate requirements.txt can result in a hard to read (very long) file and other issues: https://medium.com/@tomagee/pip-freeze-requirements-txt-considered-harmful-f0bce66cf895.

Sebastian Berlin
Utvecklare/Developer
Wikimedia Sverige (WMSE)

E-post/E-Mail: sebastian.berlin@wikimedia.se
Telefon/Phone: (+46) 0707 - 92 03 84


On Fri, 5 May 2023 at 13:17, Amir Sarabadani <ladsgroup@gmail.com> wrote:
You can also create an empty virtual env, install all requirements and then do
pip freeze > requirements.txt

That should take care of pinning

Am Fr., 5. Mai 2023 um 13:11 Uhr schrieb Lucas Werkmeister <lucas.werkmeister@wikimedia.de>:
For the general case of Python projects, I’d argue that a better solution is to adopt the lockfile pattern (package-lock.json, composer.lock, Cargo.lock, etc.) and pin all dependencies, and only update them when the new versions have been tested and are known to work. pip-tools can help with that, for example (requirements.in specifies “loose” dependencies; pip-compile creates a pinned requirements.txt; pip-sync installs it; pip-compile -U upgrades requirements.txt later; you check both requirements.in and requirements.txt into version control.) But I don’t know if that applies in your integration/config case.

Am Do., 4. Mai 2023 um 18:08 Uhr schrieb Antoine Musso <hashar@free.fr>:

Hello,

This is for python projects.

Today, May 4th, urllib3 has released a new major version 2.0.2 which breaks the extremely popular requests library.

The fix is to pin urllib3<2 to prevent the new major version from being installed (example).

https://phabricator.wikimedia.org/T335977

Upstream issue: https://github.com/psf/requests/issues/6432


Antoine "hashar" Musso
Wikimedia Release Engineering
_______________________________________________
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/


--
Lucas Werkmeister (he/er)
Software Engineer

Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Phone: +49 (0)30-577 11 62-0
https://wikimedia.de

Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision!
https://spenden.wikimedia.de

Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.
_______________________________________________
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/


--
Amir (he/him)

_______________________________________________
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
_______________________________________________
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
_______________________________________________
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

_______________________________________________
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/