Dear all,
Has anyone figured out how to use Toolforge Build Service for a tool (job + webservice) written in golang?
I'm trying to switch qrank.toolforge.org to Toolforge Build Service, but I'm struggling with it. To debug this, I wrote a minimal job that just prints a hello message to stdout: https://github.com/brawer/wikidata-qrank/blob/main/cmd/hello/main.go
Because Toolforge is emulating Heroku, it appears necessary to tell buildpack which binaries should get packaged into the container. The "// +heroku install" comment on line 14 of the following go.mod file seems to do the trick, together with listing the binaries again in a Heroku Procfile: https://github.com/brawer/wikidata-qrank/blob/main/go.mod#L14 https://github.com/brawer/wikidata-qrank/blob/main/Procfile
When running the following command on login.toolforge.org, the project seems to be built just fine: $ toolforge build start https://github.com/brawer/wikidata-qrank
Some interesting log message from build service: [step-build] 2024-04-30T18:13:31.170521374Z Building packages: [step-build] 2024-04-30T18:13:31.170585858Z - ./cmd/hello [step-build] 2024-04-30T18:13:31.170599159Z - ./cmd/qrank-builder [step-build] 2024-04-30T18:13:31.170608711Z - ./cmd/webserver [step-build] 2024-04-30T18:13:44.563334902Z [step-build] 2024-04-30T18:13:44.563404532Z [Setting launch table] [step-build] 2024-04-30T18:13:44.563417703Z Detected processes: [step-build] 2024-04-30T18:13:44.563427021Z - hello: hello [step-build] 2024-04-30T18:13:44.563437573Z - qrank-builder: qrank-builder [step-build] 2024-04-30T18:13:44.563446463Z - webserver: webserver [step-build] 2024-04-30T18:13:44.563488376Z - web: hello [step-build] 2024-04-30T18:13:44.577068596Z [step-build] 2024-04-30T18:13:44.577163104Z [Discovering process types] [step-build] 2024-04-30T18:13:44.579933448Z Procfile declares types -> web, qrank-builder, hello
The "web: hello" in the launch table looks very suspicious; it's not what my Procfile states (see link above). But at least the build seems to have been successful:
$ toolforge build show Build ID: qrank-buildpacks-pipelinerun-vhcrt Start Time: 2024-04-30T18:13:03Z End Time: 2024-04-30T18:14:00Z Status: ok Message: Tasks Completed: 1 (Failed: 0, Cancelled 0), Skipped: 0 Parameters: Source URL: https://github.com/brawer/wikidata-qrank Ref: N/A Envvars: N/A Destination Image: tools-harbor.wmcloud.org/tool-qrank/tool-qrank:latest
Now, I'd be happy to just run the "hello" job and see its output log. However, when I do this: $ toolforge jobs run --command hello --image tool-qrank/tool-qrank:latest --mount=all --filelog
The job runs for about five minutes (?!?!), and then: $ cat hello.out ERROR: failed to launch: bash exec: argument list too long
So clearly something is off... but what? How to run a golang tool on Toolforge when using Build Service?
Thanks for any help,
— Sascha
On Tue, Apr 30, 2024 at 12:26 PM Sascha Brawer via Cloud cloud@lists.wikimedia.org wrote:
Dear all,
Has anyone figured out how to use Toolforge Build Service for a tool (job + webservice) written in golang?
I got https://gitlab.wikimedia.org/toolforge-repos/bridgebot working last week, and in doing so I struggled with the same things you are reporting today.
I'm trying to switch qrank.toolforge.org to Toolforge Build Service, but I'm struggling with it. To debug this, I wrote a minimal job that just prints a hello message to stdout: https://github.com/brawer/wikidata-qrank/blob/main/cmd/hello/main.go
Because Toolforge is emulating Heroku, it appears necessary to tell buildpack which binaries should get packaged into the container. The "// +heroku install" comment on line 14 of the following go.mod file seems to do the trick, together with listing the binaries again in a Heroku Procfile: https://github.com/brawer/wikidata-qrank/blob/main/go.mod#L14 https://github.com/brawer/wikidata-qrank/blob/main/Procfile
When running the following command on login.toolforge.org, the project seems to be built just fine: $ toolforge build start https://github.com/brawer/wikidata-qrank
Some interesting log message from build service: [step-build] 2024-04-30T18:13:31.170521374Z Building packages: [step-build] 2024-04-30T18:13:31.170585858Z - ./cmd/hello [step-build] 2024-04-30T18:13:31.170599159Z - ./cmd/qrank-builder [step-build] 2024-04-30T18:13:31.170608711Z - ./cmd/webserver [step-build] 2024-04-30T18:13:44.563334902Z [step-build] 2024-04-30T18:13:44.563404532Z [Setting launch table] [step-build] 2024-04-30T18:13:44.563417703Z Detected processes: [step-build] 2024-04-30T18:13:44.563427021Z - hello: hello [step-build] 2024-04-30T18:13:44.563437573Z - qrank-builder: qrank-builder [step-build] 2024-04-30T18:13:44.563446463Z - webserver: webserver [step-build] 2024-04-30T18:13:44.563488376Z - web: hello [step-build] 2024-04-30T18:13:44.577068596Z [step-build] 2024-04-30T18:13:44.577163104Z [Discovering process types] [step-build] 2024-04-30T18:13:44.579933448Z Procfile declares types -> web, qrank-builder, hello
The "web: hello" in the launch table looks very suspicious; it's not what my Procfile states (see link above).
I noticed similar behavior when building my project. My sensemaking is that the golang buildpack is configured to always generate a "web" process because of Heroku's internal needs. In our stack Profile processing is done by a separate buildpack that the golang buildpack is not aware of.
But at least the build seems to have been successful:
$ toolforge build show Build ID: qrank-buildpacks-pipelinerun-vhcrt Start Time: 2024-04-30T18:13:03Z End Time: 2024-04-30T18:14:00Z Status: ok Message: Tasks Completed: 1 (Failed: 0, Cancelled 0), Skipped: 0 Parameters: Source URL: https://github.com/brawer/wikidata-qrank Ref: N/A Envvars: N/A Destination Image: tools-harbor.wmcloud.org/tool-qrank/tool-qrank:latest
Now, I'd be happy to just run the "hello" job and see its output log. However, when I do this: $ toolforge jobs run --command hello --image tool-qrank/tool-qrank:latest --mount=all --filelog
The job runs for about five minutes (?!?!), and then: $ cat hello.out ERROR: failed to launch: bash exec: argument list too long
So clearly something is off... but what? How to run a golang tool on Toolforge when using Build Service?
I ran into this behavior as well and reported it as https://phabricator.wikimedia.org/T363417. The workaround that I was able to find so I could move forward was to create a very slim shell script wrapper to launch my process from the Procfile: https://gitlab.wikimedia.org/toolforge-repos/bridgebot/-/commit/f05aa13d2309a55d5db286e7a38da45facb53b8a
In your case I think that you might be able to just put `/layers/heroku_go/go_target/bin/hello` as the command you want to run in either the Procfile or as the `--command` argument to `toolforge jobs`.
You can use `webservice --backend=kubernetes buildservice shell` to get an interactive shell inside of a container running the image you have built. This can be useful in debugging things. We have another task in the backlog about making this shell functionality a bit more discoverable now that we also have need for it with the jobs framework (https://phabricator.wikimedia.org/T311917).
I hope that all helps a bit. It is frustrating to be using new tools and run into such troubles. Thank you for taking the time to look for help on the mailing list though! Sharing our troubles and wins with each other helps build our community.
Bryan
Replying here just for documentation purposes (so anyone finding the thread knows it).
This was sorted out on toolforge (for the internal tricks used see https://phabricator.wikimedia.org/T363417), though it's an upstream issue so the "final" fix will come in some time.
On 04/30 13:52, Bryan Davis wrote:
On Tue, Apr 30, 2024 at 12:26 PM Sascha Brawer via Cloud cloud@lists.wikimedia.org wrote:
Dear all,
Has anyone figured out how to use Toolforge Build Service for a tool (job + webservice) written in golang?
I got https://gitlab.wikimedia.org/toolforge-repos/bridgebot working last week, and in doing so I struggled with the same things you are reporting today.
I'm trying to switch qrank.toolforge.org to Toolforge Build Service, but I'm struggling with it. To debug this, I wrote a minimal job that just prints a hello message to stdout: https://github.com/brawer/wikidata-qrank/blob/main/cmd/hello/main.go
Because Toolforge is emulating Heroku, it appears necessary to tell buildpack which binaries should get packaged into the container. The "// +heroku install" comment on line 14 of the following go.mod file seems to do the trick, together with listing the binaries again in a Heroku Procfile: https://github.com/brawer/wikidata-qrank/blob/main/go.mod#L14 https://github.com/brawer/wikidata-qrank/blob/main/Procfile
When running the following command on login.toolforge.org, the project seems to be built just fine: $ toolforge build start https://github.com/brawer/wikidata-qrank
Some interesting log message from build service: [step-build] 2024-04-30T18:13:31.170521374Z Building packages: [step-build] 2024-04-30T18:13:31.170585858Z - ./cmd/hello [step-build] 2024-04-30T18:13:31.170599159Z - ./cmd/qrank-builder [step-build] 2024-04-30T18:13:31.170608711Z - ./cmd/webserver [step-build] 2024-04-30T18:13:44.563334902Z [step-build] 2024-04-30T18:13:44.563404532Z [Setting launch table] [step-build] 2024-04-30T18:13:44.563417703Z Detected processes: [step-build] 2024-04-30T18:13:44.563427021Z - hello: hello [step-build] 2024-04-30T18:13:44.563437573Z - qrank-builder: qrank-builder [step-build] 2024-04-30T18:13:44.563446463Z - webserver: webserver [step-build] 2024-04-30T18:13:44.563488376Z - web: hello [step-build] 2024-04-30T18:13:44.577068596Z [step-build] 2024-04-30T18:13:44.577163104Z [Discovering process types] [step-build] 2024-04-30T18:13:44.579933448Z Procfile declares types -> web, qrank-builder, hello
The "web: hello" in the launch table looks very suspicious; it's not what my Procfile states (see link above).
I noticed similar behavior when building my project. My sensemaking is that the golang buildpack is configured to always generate a "web" process because of Heroku's internal needs. In our stack Profile processing is done by a separate buildpack that the golang buildpack is not aware of.
But at least the build seems to have been successful:
$ toolforge build show Build ID: qrank-buildpacks-pipelinerun-vhcrt Start Time: 2024-04-30T18:13:03Z End Time: 2024-04-30T18:14:00Z Status: ok Message: Tasks Completed: 1 (Failed: 0, Cancelled 0), Skipped: 0 Parameters: Source URL: https://github.com/brawer/wikidata-qrank Ref: N/A Envvars: N/A Destination Image: tools-harbor.wmcloud.org/tool-qrank/tool-qrank:latest
Now, I'd be happy to just run the "hello" job and see its output log. However, when I do this: $ toolforge jobs run --command hello --image tool-qrank/tool-qrank:latest --mount=all --filelog
The job runs for about five minutes (?!?!), and then: $ cat hello.out ERROR: failed to launch: bash exec: argument list too long
So clearly something is off... but what? How to run a golang tool on Toolforge when using Build Service?
I ran into this behavior as well and reported it as https://phabricator.wikimedia.org/T363417. The workaround that I was able to find so I could move forward was to create a very slim shell script wrapper to launch my process from the Procfile: https://gitlab.wikimedia.org/toolforge-repos/bridgebot/-/commit/f05aa13d2309a55d5db286e7a38da45facb53b8a
In your case I think that you might be able to just put `/layers/heroku_go/go_target/bin/hello` as the command you want to run in either the Procfile or as the `--command` argument to `toolforge jobs`.
You can use `webservice --backend=kubernetes buildservice shell` to get an interactive shell inside of a container running the image you have built. This can be useful in debugging things. We have another task in the backlog about making this shell functionality a bit more discoverable now that we also have need for it with the jobs framework (https://phabricator.wikimedia.org/T311917).
I hope that all helps a bit. It is frustrating to be using new tools and run into such troubles. Thank you for taking the time to look for help on the mailing list though! Sharing our troubles and wins with each other helps build our community.
Bryan
Bryan Davis Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808 _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/