i need some help re: a difference between commons beta cluster and production.
gwtoolset can download images from http://www.europeana1914-1918.eu without issue on beta, but not on production. the domain has been whitelisted on both servers in the wgCopyUploadsDomains array.
i have created a patch, https://gerrit.wikimedia.org/r/#/c/107038/, for the extension that will hopefully give us more insight on the issue, but in the mean time:
any ideas on how i can track down the issue? is there a log on production that might help?
thanks, dan
On Jan 13, 2014 9:41 AM, "dan entous" dan.entous.wikimedia@gmail.com wrote:
i need some help re: a difference between commons beta cluster and
production.
gwtoolset can download images from http://www.europeana1914-1918.eu without issue on beta, but not on production. the domain has been whitelisted on both servers in the wgCopyUploadsDomains array.
i have created a patch, https://gerrit.wikimedia.org/r/#/c/107038/, for the extension that will hopefully give us more insight on the issue, but in the mean time:
any ideas on how i can track down the issue? is there a log on production that might help?
thanks, dan
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Total guess, quite likely wrong: dont outbound http requests on cluster go through a proxy? Perhaps that proxy also has a whitelist.
-bawolff
Correct. All MW http requests are proxied in production. Ones headed externally are subject to a whitelist.
Not in front of my computer so I can't remember where...
-Chad On Jan 13, 2014 5:54 AM, "Brian Wolff" bawolff@gmail.com wrote:
On Jan 13, 2014 9:41 AM, "dan entous" dan.entous.wikimedia@gmail.com wrote:
i need some help re: a difference between commons beta cluster and
production.
gwtoolset can download images from http://www.europeana1914-1918.eu without issue on beta, but not on production. the domain has been whitelisted on both servers in the wgCopyUploadsDomains array.
i have created a patch, https://gerrit.wikimedia.org/r/#/c/107038/, for the extension that will hopefully give us more insight on the issue, but in the mean time:
any ideas on how i can track down the issue? is there a log on production that might help?
thanks, dan
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Total guess, quite likely wrong: dont outbound http requests on cluster go through a proxy? Perhaps that proxy also has a whitelist.
-bawolff _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Mon, Jan 13, 2014 at 7:54 AM, Chad innocentkiller@gmail.com wrote:
On Jan 13, 2014 5:54 AM, "Brian Wolff" bawolff@gmail.com wrote:
Total guess, quite likely wrong: dont outbound http requests on cluster go through a proxy? Perhaps that proxy also has a whitelist.
Correct. All MW http requests are proxied in production. Ones headed externally are subject to a whitelist.
Not in front of my computer so I can't remember where...
Looking through the operations/puppet git repository a bit leads me to believe that the url-downloader.wikimedia.org proxy is configured using the files/squid/copy-by-url-proxy.conf squid configuration file via the misc::url-downloader puppet class. I'm not seeing any acls in this configuration file that reference particular hosts (eg flickr.com & staticflickr.com).
Bryan
On Mon, Jan 13, 2014 at 10:03 AM, Bryan Davis bd808@wikimedia.org wrote:
On Mon, Jan 13, 2014 at 7:54 AM, Chad innocentkiller@gmail.com wrote:
On Jan 13, 2014 5:54 AM, "Brian Wolff" bawolff@gmail.com wrote:
Total guess, quite likely wrong: dont outbound http requests on cluster go through a proxy? Perhaps that proxy also has a whitelist.
Correct. All MW http requests are proxied in production. Ones headed externally are subject to a whitelist.
Not in front of my computer so I can't remember where...
Looking through the operations/puppet git repository a bit leads me to believe that the url-downloader.wikimedia.org proxy is configured using the files/squid/copy-by-url-proxy.conf squid configuration file via the misc::url-downloader puppet class. I'm not seeing any acls in this configuration file that reference particular hosts (eg flickr.com & staticflickr.com).
I'm fairly confident at this point that the problem is that Dan didn't know that direct use of MWHttpRequest in the WMF production cluster needs to explicitly set the 'proxy' option. There is a global that would do this automatically but apparently we don't set it in the production configuration.
Bryan
wikitech-l@lists.wikimedia.org