*** r-daneel has quit IRC | 00:00 | |
*** dingyichen has joined #openstack-infra | 00:01 | |
*** slaweq has joined #openstack-infra | 00:20 | |
*** slaweq has quit IRC | 00:25 | |
*** eernst has quit IRC | 00:27 | |
*** camunoz has quit IRC | 00:28 | |
mtreinish | I think paste.o.o is having issues, it's not responding for me | 00:31 |
---|---|---|
mtreinish | it looks like it's cpu is pegged: http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=145&rra_id=0&view_type=tree&graph_start=1524616238&graph_end=1524702638 | 00:31 |
*** hemna_ has quit IRC | 00:32 | |
*** dhill_ has quit IRC | 00:43 | |
*** namnh has joined #openstack-infra | 00:51 | |
*** dhill_ has joined #openstack-infra | 00:51 | |
*** yamamoto has quit IRC | 00:53 | |
*** yamamoto has joined #openstack-infra | 00:53 | |
*** shu-mutow-AWAY is now known as shu-mutow | 00:59 | |
*** agopi|zzz has joined #openstack-infra | 01:06 | |
*** kmalloc has quit IRC | 01:10 | |
*** kgiusti has left #openstack-infra | 01:21 | |
*** salv-orlando has joined #openstack-infra | 01:26 | |
*** iyamahat has quit IRC | 01:26 | |
*** salv-orl_ has quit IRC | 01:29 | |
*** yamahata has quit IRC | 01:32 | |
*** caphrim007_ has joined #openstack-infra | 01:33 | |
*** caphrim007 has quit IRC | 01:36 | |
*** yolanda__ has joined #openstack-infra | 01:40 | |
*** yolanda_ has quit IRC | 01:44 | |
*** rosmaita has quit IRC | 01:47 | |
*** rajinir has quit IRC | 01:48 | |
adriant | is paste.openstack.org down? | 01:55 |
adriant | I can't seem to access it | 01:55 |
adriant | I'm getting the spinning loading wheel of doom | 01:55 |
clarkb | adriant: mtreinish mentioned the cpu is pegged accorsing to cacti just before you in scrollback | 01:58 |
clarkb | I'm not in a position to debug it right now though | 01:58 |
adriant | oh, blast, I missed that, sorry clarkb | 01:59 |
*** ramishra has joined #openstack-infra | 02:05 | |
*** viks_ has joined #openstack-infra | 02:08 | |
*** rpioso|afk_ has joined #openstack-infra | 02:08 | |
*** gmann_ has joined #openstack-infra | 02:08 | |
*** jamespage_ has joined #openstack-infra | 02:08 | |
*** edwarnicke_ has joined #openstack-infra | 02:09 | |
*** beekneemech has joined #openstack-infra | 02:10 | |
*** zerick_ has joined #openstack-infra | 02:10 | |
*** ihar_ has joined #openstack-infra | 02:11 | |
*** dteselkin_ has joined #openstack-infra | 02:11 | |
*** Adri2000_ has joined #openstack-infra | 02:12 | |
*** rakhmerov__ has joined #openstack-infra | 02:12 | |
*** rakhmerov has quit IRC | 02:12 | |
*** edwarnicke has quit IRC | 02:12 | |
*** markmcclain has quit IRC | 02:12 | |
*** rpioso|afk has quit IRC | 02:12 | |
*** viks has quit IRC | 02:12 | |
*** jamespage has quit IRC | 02:12 | |
*** mordred has quit IRC | 02:12 | |
*** d0ugal has quit IRC | 02:12 | |
*** ihar has quit IRC | 02:12 | |
*** anteaya has quit IRC | 02:12 | |
*** gmann has quit IRC | 02:12 | |
*** Adri2000 has quit IRC | 02:12 | |
*** zerick has quit IRC | 02:12 | |
*** krenczewski has quit IRC | 02:12 | |
*** bnemec has quit IRC | 02:12 | |
*** oanson has quit IRC | 02:12 | |
*** dteselkin has quit IRC | 02:12 | |
*** ianw_pto has quit IRC | 02:12 | |
*** nhandler has quit IRC | 02:12 | |
*** rpioso|afk_ is now known as rpioso|afk | 02:12 | |
*** viks_ is now known as viks | 02:12 | |
*** krenczew1 has joined #openstack-infra | 02:12 | |
*** edwarnicke_ is now known as edwarnicke | 02:13 | |
*** jamespage_ is now known as jamespage | 02:13 | |
*** ianw has joined #openstack-infra | 02:13 | |
*** mordred has joined #openstack-infra | 02:13 | |
*** oanson has joined #openstack-infra | 02:13 | |
*** markmcclain has joined #openstack-infra | 02:13 | |
*** d0ugal has joined #openstack-infra | 02:13 | |
*** nhandler has joined #openstack-infra | 02:14 | |
*** bandini has quit IRC | 02:14 | |
*** cburgess has quit IRC | 02:15 | |
*** VW has joined #openstack-infra | 02:16 | |
*** dmueller[m] has quit IRC | 02:16 | |
*** kbaikov[m] has quit IRC | 02:16 | |
*** masayukig has quit IRC | 02:16 | |
*** warlord77[m] has quit IRC | 02:16 | |
*** aspiers[m] has quit IRC | 02:16 | |
*** ilpianista_ has quit IRC | 02:17 | |
*** bandini has joined #openstack-infra | 02:17 | |
*** cburgess has joined #openstack-infra | 02:18 | |
*** gongysh has joined #openstack-infra | 02:19 | |
*** edmondsw has joined #openstack-infra | 02:20 | |
*** xinliang3 has quit IRC | 02:22 | |
*** xinliang3 has joined #openstack-infra | 02:22 | |
*** hongbin has joined #openstack-infra | 02:23 | |
corvus | mtreinish, adriant, clarkb: i have no idea why it was stuck. i restarted it. | 02:43 |
corvus | [pid 4734] futex(0x2c04ad0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, ffffffffstrace: [ Process PID=4737 runs in x32 mode. ] | 02:43 |
corvus | that's all i got; and it's too late to gdb it. | 02:43 |
corvus | #status log restarted lodgeit on paste.o.o because it appeared hung | 02:44 |
openstackstatus | corvus: finished logging | 02:44 |
*** daidv has joined #openstack-infra | 02:46 | |
*** dhill_ has quit IRC | 02:46 | |
*** bobh has joined #openstack-infra | 02:47 | |
*** psachin has joined #openstack-infra | 02:48 | |
*** daidv has quit IRC | 02:49 | |
*** dhill_ has joined #openstack-infra | 02:54 | |
*** eernst has joined #openstack-infra | 03:01 | |
*** namnh_ has joined #openstack-infra | 03:06 | |
*** namnh has quit IRC | 03:08 | |
*** armaan has quit IRC | 03:16 | |
*** armaan has joined #openstack-infra | 03:17 | |
*** abelur_ has joined #openstack-infra | 03:19 | |
*** abelur_ has quit IRC | 03:19 | |
*** sree has joined #openstack-infra | 03:21 | |
*** andreas_s has joined #openstack-infra | 03:23 | |
*** sree has quit IRC | 03:26 | |
*** dpawlik has joined #openstack-infra | 03:27 | |
*** andreas_s has quit IRC | 03:28 | |
*** dpawlik has quit IRC | 03:32 | |
*** auristor has quit IRC | 03:32 | |
*** nicolasbock has quit IRC | 03:33 | |
*** rvba has joined #openstack-infra | 03:33 | |
*** rvba has quit IRC | 03:33 | |
*** rvba has joined #openstack-infra | 03:33 | |
*** dingyichen has quit IRC | 03:35 | |
*** dingyichen has joined #openstack-infra | 03:35 | |
*** HenryG has quit IRC | 03:37 | |
*** annp has joined #openstack-infra | 03:39 | |
*** auristor has joined #openstack-infra | 03:39 | |
*** lpetrut has joined #openstack-infra | 03:44 | |
*** HenryG has joined #openstack-infra | 03:45 | |
*** HenryG has quit IRC | 03:45 | |
*** HenryG has joined #openstack-infra | 03:46 | |
*** VW has quit IRC | 03:47 | |
*** VW has joined #openstack-infra | 03:48 | |
*** edmondsw has quit IRC | 03:50 | |
*** eernst has quit IRC | 03:52 | |
*** VW has quit IRC | 03:53 | |
*** janki has joined #openstack-infra | 03:55 | |
*** zhangfei has joined #openstack-infra | 03:55 | |
*** eernst has joined #openstack-infra | 03:55 | |
*** bobh has quit IRC | 04:00 | |
*** hongbin has quit IRC | 04:01 | |
*** dklyle has joined #openstack-infra | 04:02 | |
*** sree has joined #openstack-infra | 04:12 | |
*** lpetrut has quit IRC | 04:13 | |
*** sree_ has joined #openstack-infra | 04:14 | |
*** sree_ is now known as Guest35521 | 04:14 | |
*** niedbalski has quit IRC | 04:17 | |
*** niedbalski has joined #openstack-infra | 04:17 | |
*** sree has quit IRC | 04:17 | |
*** edmondsw has joined #openstack-infra | 04:18 | |
*** rakhmerov__ is now known as rakhmerov | 04:19 | |
*** edmondsw has quit IRC | 04:21 | |
*** edmondsw has joined #openstack-infra | 04:21 | |
*** edmondsw has quit IRC | 04:21 | |
*** edmondsw has joined #openstack-infra | 04:22 | |
*** slaweq has joined #openstack-infra | 04:22 | |
*** armaan has quit IRC | 04:22 | |
*** armaan has joined #openstack-infra | 04:23 | |
*** namnh has joined #openstack-infra | 04:23 | |
*** namnh_ has quit IRC | 04:23 | |
*** dklyle has quit IRC | 04:26 | |
*** edmondsw has quit IRC | 04:26 | |
*** slaweq has quit IRC | 04:26 | |
*** lpetrut has joined #openstack-infra | 04:27 | |
*** apetrich has quit IRC | 04:29 | |
*** apetrich has joined #openstack-infra | 04:30 | |
*** dklyle has joined #openstack-infra | 04:30 | |
*** dklyle has quit IRC | 04:35 | |
*** jchhatbar has joined #openstack-infra | 04:41 | |
*** janki has quit IRC | 04:44 | |
*** lpetrut has quit IRC | 04:46 | |
*** ccamacho has quit IRC | 04:48 | |
EmilienM | fungi, AJaeger : and the last patch: https://review.openstack.org/#/c/563200/ | 04:49 |
EmilienM | fungi, AJaeger : can someone from gerrit admin add me on https://review.openstack.org/#/admin/groups/1899,members please ? | 04:49 |
*** links has joined #openstack-infra | 04:53 | |
*** stakeda has joined #openstack-infra | 04:54 | |
*** eernst has quit IRC | 05:02 | |
*** yamahata has joined #openstack-infra | 05:02 | |
*** dhajare has joined #openstack-infra | 05:04 | |
*** lpetrut has joined #openstack-infra | 05:13 | |
*** e0ne has joined #openstack-infra | 05:16 | |
*** sree has joined #openstack-infra | 05:18 | |
*** Guest35521 has quit IRC | 05:19 | |
*** iranzo has joined #openstack-infra | 05:22 | |
*** lpetrut has quit IRC | 05:32 | |
openstackgerrit | Fatih Degirmenci proposed openstack-infra/zuul master: Fix zuul home directory in zuul from scratch document https://review.openstack.org/564386 | 05:37 |
openstackgerrit | Fatih Degirmenci proposed openstack-infra/zuul master: Remove terminal prompt from commands listed in the docs https://review.openstack.org/564387 | 05:41 |
*** quiquell|rover has joined #openstack-infra | 05:42 | |
*** e0ne has quit IRC | 05:43 | |
*** e0ne has joined #openstack-infra | 05:44 | |
*** yolanda__ is now known as yolanda | 05:45 | |
*** d0ugal has quit IRC | 05:46 | |
openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Add start and end timestamp to task result in zuul_json callback https://review.openstack.org/563888 | 05:52 |
*** shu-mutow is now known as shu-mutow-AWAY | 06:01 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/564389 | 06:03 |
AJaeger | EmilienM: I'm not an gerrit admin - better ask infra-root for help | 06:06 |
*** armaan has quit IRC | 06:12 | |
*** armaan has joined #openstack-infra | 06:13 | |
*** d0ugal has joined #openstack-infra | 06:13 | |
*** armaan has quit IRC | 06:13 | |
*** armaan has joined #openstack-infra | 06:13 | |
*** jtomasek has joined #openstack-infra | 06:18 | |
openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/564389 | 06:19 |
*** slaweq has joined #openstack-infra | 06:22 | |
EmilienM | AJaeger: ok thx anyway | 06:24 |
*** aeng has quit IRC | 06:26 | |
*** e0ne_ has joined #openstack-infra | 06:27 | |
openstackgerrit | Merged openstack/gertty master: Add rdoproject gerrit example https://review.openstack.org/428692 | 06:28 |
*** e0ne has quit IRC | 06:30 | |
*** hashar has joined #openstack-infra | 06:31 | |
frickler | EmilienM: done | 06:31 |
*** andreas_s has joined #openstack-infra | 06:35 | |
*** lpetrut has joined #openstack-infra | 06:38 | |
openstackgerrit | Merged openstack-infra/project-config master: Configure ansible-role-container-registry zuul jobs https://review.openstack.org/563200 | 06:39 |
frickler | infra-root: seeing proxy errors on http://paste.openstack.org , will take a close look | 06:46 |
*** sree_ has joined #openstack-infra | 06:48 | |
*** sree_ is now known as Guest50426 | 06:49 | |
csatari | Is this the right place to ask about the wiki? | 06:49 |
*** sree has quit IRC | 06:50 | |
*** dingyichen has quit IRC | 06:52 | |
*** dpawlik has joined #openstack-infra | 06:52 | |
*** amoralej|off is now known as amoralej | 06:55 | |
*** dpawlik has quit IRC | 06:56 | |
*** tesseract has joined #openstack-infra | 06:58 | |
*** dpawlik has joined #openstack-infra | 06:58 | |
*** tesseract has quit IRC | 07:00 | |
*** tesseract has joined #openstack-infra | 07:02 | |
*** ccamacho has joined #openstack-infra | 07:03 | |
*** udesale has joined #openstack-infra | 07:03 | |
*** gfidente has joined #openstack-infra | 07:04 | |
*** gfidente has joined #openstack-infra | 07:04 | |
*** jtomasek has quit IRC | 07:05 | |
*** quiquell|rover is now known as quique|rover|afk | 07:06 | |
*** shardy has joined #openstack-infra | 07:14 | |
*** yamamoto has quit IRC | 07:16 | |
*** yamamoto has joined #openstack-infra | 07:17 | |
frickler | infra-root: seems lodgeit needs to be started from its own directory, added that to the service definition and now runs fine. will do a proper patch later | 07:17 |
frickler | csatari: depends on the question, but give it a try I'd say | 07:18 |
*** andreas_s has quit IRC | 07:18 | |
csatari | I would like to display an image on this page: https://wiki.openstack.org/wiki/OpenStack_Edge_Discussions_Dublin_PTG | 07:19 |
*** andreas_s has joined #openstack-infra | 07:19 | |
csatari | But it looks like I do not have rights to upload a file and I could not figure out how to make it to render an image from an external source. | 07:20 |
csatari | frickler: This is it ;) | 07:20 |
*** tosky has joined #openstack-infra | 07:20 | |
*** ralonsoh has joined #openstack-infra | 07:20 | |
openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Add start and end timestamp to task and play result in zuul_json callback https://review.openstack.org/563888 | 07:21 |
*** andreas_s_ has joined #openstack-infra | 07:21 | |
*** andreas__ has joined #openstack-infra | 07:23 | |
frickler | csatari: ah, I think that functionality was disabled due to abuse by spammers. I think it may be possible to give proper credentials to your account. probably fungi can tell you more but you'd need to wait a couple of hours for that | 07:23 |
*** andreas_s has quit IRC | 07:23 | |
csatari | Okay, I'm not in a hurry. | 07:24 |
*** udesale has quit IRC | 07:24 | |
*** Guest50426 has quit IRC | 07:25 | |
*** sree has joined #openstack-infra | 07:25 | |
*** andreas__ has quit IRC | 07:26 | |
*** andreas_s_ has quit IRC | 07:26 | |
*** xarses has quit IRC | 07:28 | |
frickler | ttx looks to be a wiki admin, too. ^^ | 07:29 |
*** andreas_s has joined #openstack-infra | 07:30 | |
*** pcaruana has joined #openstack-infra | 07:31 | |
*** rcernin has quit IRC | 07:32 | |
*** kashyap has joined #openstack-infra | 07:34 | |
kashyap | Nice, 'devstack-platform-fedora-27', I don't think I've seen this job (or maybe I just missed to pay attention) | 07:35 |
*** jpena|off is now known as jpena | 07:35 | |
*** alexchadin has joined #openstack-infra | 07:36 | |
*** sree has quit IRC | 07:36 | |
*** andreas_s has quit IRC | 07:39 | |
*** andreas_s has joined #openstack-infra | 07:44 | |
ttx | csatari: hi! Let me see if I can help you | 07:47 |
ttx | csatari: if you have that image somwhere I can access it, I can try uploading it to the wiki | 07:48 |
ttx | then you can reference it in your page | 07:48 |
csatari | https://usercontent.irccloud-cdn.com/file/NIUo2xST/LocalAndRemoteSites.png | 07:49 |
csatari | Thanks ttx . | 07:49 |
csatari | I might need to update it in the future as this is made to trigger some discussions :) | 07:49 |
ttx | ok let's see if the wiki lets me upload things | 07:52 |
*** threestrands_ has quit IRC | 07:52 | |
ttx | OK you should be able to use [[File:LocalAndRemoteSites.png]] | 07:53 |
ttx | Let me see if I can bump your rights so that you can do that | 07:53 |
ttx | since it looks like a thing most people can do | 07:53 |
*** jpich has joined #openstack-infra | 07:54 | |
ttx | csatari: I made you a "verified user" which I think is what you missed for uploading by yourself | 07:55 |
ttx | You should be able to use https://wiki.openstack.org/wiki/Special:Upload now | 07:56 |
ttx | but I uploaded your file and you can include it by using [[File:LocalAndRemoteSites.png]] | 07:56 |
*** ssbarnea_ has joined #openstack-infra | 07:57 | |
*** andreas_s has quit IRC | 08:02 | |
csatari | Thanks ttx. | 08:06 |
*** ccamacho has quit IRC | 08:08 | |
*** andreas_s has joined #openstack-infra | 08:09 | |
*** ccamacho has joined #openstack-infra | 08:11 | |
*** yamamoto has quit IRC | 08:11 | |
*** Adri2000_ is now known as Adri2000 | 08:12 | |
*** Adri2000 has joined #openstack-infra | 08:12 | |
*** lucas-afk is now known as lucasagomes | 08:15 | |
*** sree has joined #openstack-infra | 08:15 | |
*** florianf has joined #openstack-infra | 08:15 | |
*** andreas_s has quit IRC | 08:18 | |
*** quique|rover|afk is now known as quiquell|rover | 08:19 | |
*** sshnaidm|off is now known as sshnaidm | 08:19 | |
*** owalsh is now known as owalsh_afk | 08:20 | |
openstackgerrit | Merged openstack-infra/irc-meetings master: neutron: remove jlibosva from chairing Neutron meetings https://review.openstack.org/564316 | 08:20 |
*** florianf has quit IRC | 08:22 | |
*** andreas_s has joined #openstack-infra | 08:23 | |
*** jaosorior has joined #openstack-infra | 08:24 | |
openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/diskimage-builder master: Move common ubuntu environment setting to ubuntu-common element https://review.openstack.org/563731 | 08:24 |
openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/diskimage-builder master: Remove redundant sources change/update https://review.openstack.org/563739 | 08:25 |
openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/diskimage-builder master: Add ubuntu-systemd-container operating-system element https://review.openstack.org/563748 | 08:25 |
*** yamamoto has joined #openstack-infra | 08:27 | |
*** dtruong_ has joined #openstack-infra | 08:27 | |
*** ramishra has quit IRC | 08:28 | |
*** dtruong has quit IRC | 08:28 | |
*** andreas_s has quit IRC | 08:29 | |
*** andreas_s has joined #openstack-infra | 08:29 | |
*** andreas_s has quit IRC | 08:30 | |
*** s-shiono has quit IRC | 08:32 | |
*** andreas_s_ has joined #openstack-infra | 08:33 | |
*** derekh has joined #openstack-infra | 08:34 | |
*** ramishra has joined #openstack-infra | 08:40 | |
*** rpittau has joined #openstack-infra | 08:42 | |
*** ramishra has quit IRC | 08:45 | |
*** masayukig has joined #openstack-infra | 08:46 | |
*** udesale has joined #openstack-infra | 08:46 | |
openstackgerrit | Fatih Degirmenci proposed openstack-infra/zuul master: Use code-block for admin documentation and remove terminal prompts https://review.openstack.org/564387 | 08:47 |
*** electrofelix has quit IRC | 08:47 | |
openstackgerrit | Fatih Degirmenci proposed openstack-infra/zuul master: Use code-block for admin documentation and remove terminal prompts https://review.openstack.org/564387 | 08:49 |
*** florianf has joined #openstack-infra | 08:50 | |
*** udesale has quit IRC | 08:51 | |
*** udesale has joined #openstack-infra | 08:55 | |
*** ramishra has joined #openstack-infra | 08:56 | |
*** shardy has quit IRC | 08:59 | |
*** shardy has joined #openstack-infra | 09:00 | |
*** sambetts|afk is now known as sambetts | 09:03 | |
*** panda|ruck|off is now known as panda|ruck | 09:10 | |
*** florianf has quit IRC | 09:12 | |
*** alexchadin has quit IRC | 09:12 | |
openstackgerrit | Ghanshyam Mann proposed openstack-dev/hacking master: Release notes for 1.1.0 https://review.openstack.org/561129 | 09:14 |
*** tushar has quit IRC | 09:15 | |
rnoriega | ianw, ping | 09:18 |
*** alexchadin has joined #openstack-infra | 09:18 | |
*** warlord77[m] has joined #openstack-infra | 09:23 | |
*** ilpianista_ has joined #openstack-infra | 09:23 | |
*** aspiers[m] has joined #openstack-infra | 09:23 | |
*** dmueller[m] has joined #openstack-infra | 09:23 | |
*** kbaikov[m] has joined #openstack-infra | 09:23 | |
*** zoli is now known as zoli|lunch | 09:32 | |
*** e0ne_ has quit IRC | 09:33 | |
*** electrofelix has joined #openstack-infra | 09:33 | |
*** derekh_ has joined #openstack-infra | 09:34 | |
*** jpich_ has joined #openstack-infra | 09:38 | |
*** masayukig has quit IRC | 09:38 | |
*** dtantsur|afk is now known as dtantsur | 09:38 | |
*** derekh has quit IRC | 09:39 | |
*** jpich has quit IRC | 09:39 | |
*** kbaikov[m] has quit IRC | 09:39 | |
*** warlord77[m] has quit IRC | 09:39 | |
*** dmueller[m] has quit IRC | 09:39 | |
*** ilpianista_ has quit IRC | 09:39 | |
*** pcichy has quit IRC | 09:40 | |
*** aspiers[m] has quit IRC | 09:40 | |
*** pcichy has joined #openstack-infra | 09:40 | |
*** mtreinish has quit IRC | 09:41 | |
*** jtomasek has joined #openstack-infra | 09:42 | |
*** dpawlik has quit IRC | 09:43 | |
*** ykarel has joined #openstack-infra | 09:45 | |
openstackgerrit | Fatih Degirmenci proposed openstack-infra/zuul master: Use code-block for admin documentation and terminal prompts https://review.openstack.org/564387 | 09:45 |
*** dpawlik has joined #openstack-infra | 09:46 | |
*** mtreinish has joined #openstack-infra | 09:47 | |
*** jaosorior has quit IRC | 09:49 | |
openstackgerrit | Fatih Degirmenci proposed openstack-infra/zuul master: Use code-block for admin documentation and add terminal prompts https://review.openstack.org/564387 | 09:49 |
*** udesale has quit IRC | 09:54 | |
*** dbecker has joined #openstack-infra | 09:54 | |
*** alexchadin has quit IRC | 09:59 | |
*** ramishra has quit IRC | 10:01 | |
*** gongysh has quit IRC | 10:01 | |
*** armaan has quit IRC | 10:03 | |
*** ramishra has joined #openstack-infra | 10:04 | |
*** alexchadin has joined #openstack-infra | 10:04 | |
*** ccamacho has quit IRC | 10:07 | |
*** ccamacho has joined #openstack-infra | 10:08 | |
*** ccamacho has joined #openstack-infra | 10:08 | |
*** alexchadin has quit IRC | 10:09 | |
*** alexchadin has joined #openstack-infra | 10:11 | |
*** vivsoni__ has quit IRC | 10:14 | |
*** namnh has quit IRC | 10:14 | |
*** masayukig has joined #openstack-infra | 10:16 | |
*** alexchadin has quit IRC | 10:16 | |
*** apetrich has quit IRC | 10:18 | |
*** alexchadin has joined #openstack-infra | 10:24 | |
*** alexchadin has quit IRC | 10:28 | |
*** alexchadin has joined #openstack-infra | 10:29 | |
*** anteaya has joined #openstack-infra | 10:32 | |
*** nicolasbock has joined #openstack-infra | 10:33 | |
*** alexchadin has quit IRC | 10:34 | |
openstackgerrit | Matthieu Huin proposed openstack-infra/os-loganalyze master: Support ANSI coloring in logs https://review.openstack.org/563947 | 10:36 |
*** boden has joined #openstack-infra | 10:37 | |
*** andreas_s_ has quit IRC | 10:37 | |
*** udesale has joined #openstack-infra | 10:37 | |
*** AJaeger has quit IRC | 10:40 | |
*** zoli|lunch is now known as zoli | 10:42 | |
*** udesale_ has joined #openstack-infra | 10:45 | |
*** dbecker_ has joined #openstack-infra | 10:46 | |
*** shardy_ has joined #openstack-infra | 10:46 | |
*** udesale has quit IRC | 10:47 | |
*** dbecker has quit IRC | 10:47 | |
*** shardy has quit IRC | 10:47 | |
*** yolanda_ has joined #openstack-infra | 10:53 | |
*** yolanda__ has joined #openstack-infra | 10:54 | |
*** ilpianista_ has joined #openstack-infra | 10:55 | |
*** warlord77[m] has joined #openstack-infra | 10:55 | |
*** aspiers[m] has joined #openstack-infra | 10:55 | |
*** kbaikov[m] has joined #openstack-infra | 10:55 | |
*** dmueller[m] has joined #openstack-infra | 10:55 | |
*** yamamoto has quit IRC | 10:55 | |
*** yolanda has quit IRC | 10:56 | |
*** yolanda_ has quit IRC | 10:58 | |
*** dpawlik has quit IRC | 10:59 | |
*** AJaeger has joined #openstack-infra | 10:59 | |
*** jpena is now known as jpena|lunch | 11:00 | |
*** threestrands_ has joined #openstack-infra | 11:01 | |
*** udesale_ has quit IRC | 11:01 | |
*** wolverineav has joined #openstack-infra | 11:04 | |
mrmartin | fungi: I patched the groups.o.o with this brand new SA: https://www.drupal.org/SA-CORE-2018-004 I'm travelling, and a real patch for the 7.59 upgrade will arrive soon. | 11:04 |
*** mriedem has joined #openstack-infra | 11:05 | |
*** alexchadin has joined #openstack-infra | 11:07 | |
*** e0ne has joined #openstack-infra | 11:11 | |
*** armaan has joined #openstack-infra | 11:12 | |
*** zhangfei has quit IRC | 11:13 | |
*** andreas_s has joined #openstack-infra | 11:17 | |
*** claudiub has joined #openstack-infra | 11:18 | |
*** udesale_ has joined #openstack-infra | 11:19 | |
*** pcichy has quit IRC | 11:24 | |
*** lucasagomes is now known as lucas-hungry | 11:24 | |
*** dpawlik has joined #openstack-infra | 11:26 | |
*** rh-jelabarre has joined #openstack-infra | 11:27 | |
*** e0ne has quit IRC | 11:35 | |
*** efried has joined #openstack-infra | 11:35 | |
*** rm_work has quit IRC | 11:39 | |
*** dtantsur has quit IRC | 11:39 | |
*** rm_work has joined #openstack-infra | 11:39 | |
*** dpawlik has quit IRC | 11:42 | |
*** dpawlik has joined #openstack-infra | 11:42 | |
*** dtantsur has joined #openstack-infra | 11:44 | |
*** dtantsur has quit IRC | 11:44 | |
*** dtantsur has joined #openstack-infra | 11:44 | |
*** dtantsur has quit IRC | 11:44 | |
*** armaan has quit IRC | 11:45 | |
openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/puppet-lodgeit master: Add WorkingDir to systemd service template https://review.openstack.org/564470 | 11:46 |
*** alex_xu has quit IRC | 11:49 | |
*** dtantsur has joined #openstack-infra | 11:49 | |
*** dtantsur has quit IRC | 11:49 | |
*** alex_xu has joined #openstack-infra | 11:49 | |
*** dtantsur has joined #openstack-infra | 11:49 | |
*** dtantsur has quit IRC | 11:49 | |
*** e0ne has joined #openstack-infra | 11:49 | |
*** yamamoto has joined #openstack-infra | 11:50 | |
*** yamamoto has quit IRC | 11:50 | |
*** alexchadin has quit IRC | 11:53 | |
*** alexchadin has joined #openstack-infra | 11:53 | |
*** ramishra has quit IRC | 11:54 | |
*** dtantsur has joined #openstack-infra | 11:54 | |
*** StevenK has quit IRC | 11:55 | |
*** ramishra has joined #openstack-infra | 11:55 | |
*** lpetrut_ has joined #openstack-infra | 11:56 | |
*** StevenK has joined #openstack-infra | 11:56 | |
*** edmondsw has joined #openstack-infra | 11:57 | |
*** edmondsw_ has joined #openstack-infra | 11:58 | |
*** eharney has joined #openstack-infra | 11:59 | |
*** lpetrut has quit IRC | 11:59 | |
*** edmondsw has quit IRC | 12:02 | |
*** Hal has joined #openstack-infra | 12:06 | |
*** Hal is now known as Guest44397 | 12:07 | |
*** edmondsw_ is now known as edmondsw | 12:08 | |
*** sshnaidm is now known as notsshnaidm | 12:10 | |
*** notsshnaidm is now known as nosshnaidm | 12:10 | |
*** nosshnaidm is now known as sshnaidm | 12:10 | |
*** niedbalski has quit IRC | 12:14 | |
*** niedbalski has joined #openstack-infra | 12:14 | |
*** tpsilva has joined #openstack-infra | 12:15 | |
*** apetrich has joined #openstack-infra | 12:16 | |
aspiers | question based on somewhat idle speculation: if we (OpenStack) "own" two Freenode channels via chanserv, is it possible to tell freenode to redirect people from one channel to the other when they /join? | 12:18 |
aspiers | I'm just thinking it might sense to redirect people from #openstack-ha to #openstack-self-healing | 12:18 |
aspiers | since I see HA as a subset of self-healing | 12:18 |
*** yamamoto has joined #openstack-infra | 12:18 | |
aspiers | I think it must be possible since IIRC there's a redirect from #suse to #opensuse, or maybe the other way around | 12:19 |
aspiers | ah, there's a +f mode | 12:20 |
cmurphy | aspiers: yep it's possible https://docs.openstack.org/infra/system-config/irc.html#renaming-an-irc-channel | 12:20 |
aspiers | cmurphy: thanks :) | 12:20 |
*** rosmaita has joined #openstack-infra | 12:20 | |
*** ralonsoh has quit IRC | 12:23 | |
*** jpena|lunch is now known as jpena | 12:24 | |
*** lucas-hungry is now known as lucasagomes | 12:30 | |
*** udesale_ has quit IRC | 12:32 | |
*** camunoz has joined #openstack-infra | 12:33 | |
*** dave-mccowan has joined #openstack-infra | 12:33 | |
*** trown|outtypewww is now known as trown | 12:34 | |
*** rlandy has joined #openstack-infra | 12:35 | |
*** ldnunes has joined #openstack-infra | 12:35 | |
*** e0ne has quit IRC | 12:37 | |
*** psachin has quit IRC | 12:38 | |
*** krenczew1 has quit IRC | 12:39 | |
*** dave-mcc_ has joined #openstack-infra | 12:40 | |
*** krenczew1 has joined #openstack-infra | 12:42 | |
smcginnis | I'm getting a proxy error from paste.o.o | 12:42 |
smcginnis | Reason: Error reading from remote server | 12:42 |
*** dave-mccowan has quit IRC | 12:43 | |
stephenfin | AJaeger, mordred: I assume I'm not allowed to post to zuul-announce. Could one of you send the email necessary to move this along? https://review.openstack.org/#/c/559348/ | 12:45 |
*** kgiusti has joined #openstack-infra | 12:49 | |
EmilienM | frickler: thanks | 12:49 |
*** quiquell|rover is now known as quique|rover|foo | 12:50 | |
*** jpich_ is now known as jpich | 12:52 | |
AJaeger | stephenfin: you're allowed to do so - corvus just needs to moderate it | 12:53 |
stephenfin | AJaeger: Oh, sweet. I'll do that now so | 12:53 |
AJaeger | stephenfin: great! | 12:53 |
*** quique|rover|foo is now known as quique|rover|lch | 12:55 | |
frickler | smcginnis: oh, did it break again? /me goes checking | 12:56 |
*** threestrands_ has quit IRC | 12:58 | |
*** dpawlik has quit IRC | 13:00 | |
openstackgerrit | Matthieu Huin proposed openstack-infra/os-loganalyze master: Support ANSI coloring in logs https://review.openstack.org/563947 | 13:00 |
*** dpawlik has joined #openstack-infra | 13:01 | |
frickler | smcginnis: restarted again, but doesn't look like that helps for long, someone will need to investigate further | 13:02 |
dhellmann | infra cores, could I get another review on https://review.openstack.org/#/c/562826/ please? | 13:03 |
frickler | dhellmann: done. also, the magic incantation is "infra-root". and I still wanted to make a patch to better document that. ;) | 13:05 |
*** eumel8 has joined #openstack-infra | 13:05 | |
*** dpawlik has quit IRC | 13:05 | |
dhellmann | frickler : oh, I didn't think it needed to be a root person to approve patches | 13:05 |
openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: ignore the series_status.yaml when processing release requests https://review.openstack.org/564342 | 13:06 |
*** zoli is now known as zoli|afk | 13:06 | |
frickler | dhellmann: ah, right, we do have "config-cores" for that | 13:06 |
dhellmann | I'll keep that one in mind, too | 13:06 |
frickler | though I think at this point in time AJaeger is the only person matching one and not the other ;) | 13:07 |
*** Goneri has joined #openstack-infra | 13:08 | |
*** VW has joined #openstack-infra | 13:09 | |
frickler | infra-root: restarting lodgeit seems to have helped only for a short time, if someone could do a more thorough investigation today, that would seem useful | 13:09 |
stephenfin | AJaeger, corvus: Sent that email off. Let me know if I wasn't clear | 13:09 |
*** jcoufal has joined #openstack-infra | 13:11 | |
openstackgerrit | Merged openstack-infra/project-config master: update the release tools to mark newton as closed https://review.openstack.org/562826 | 13:11 |
*** edmondsw has quit IRC | 13:12 | |
*** dpawlik has joined #openstack-infra | 13:12 | |
*** edmondsw has joined #openstack-infra | 13:12 | |
smcginnis | frickler: Thanks for the restart. Looks like it's already down again though. | 13:12 |
smcginnis | Not critical for me, but figured I should report it. | 13:12 |
smcginnis | frickler: Are we having known issues with that server? | 13:13 |
*** woid has joined #openstack-infra | 13:14 | |
*** edmondsw has quit IRC | 13:16 | |
woid | Hello everyone, I've got a question regarding zuul-jobs, specifically a configure-mirrors role. It does not support RHEL mirrors right now - is there a reason against doing that? | 13:17 |
frickler | smcginnis: yes, seems to have started tonight and I did another restart round earlier today | 13:17 |
smcginnis | frickler: OK, thanks. If I hear anyone else having issues I will let them know. | 13:18 |
frickler | smcginnis: cool, thanks and thanks for reporting this | 13:18 |
mordred | woid: there is no reason to not support one. we don't have it supported right now because infra just doesn't run any rhel mirrors (since we can't use rhel in the gate) | 13:18 |
mordred | woid: so I think if someone wanted to add support for rhel mirrors it would be a welcome addition | 13:19 |
woid | mordred: great, I guess I'll work on adding that and submit it for upstream then | 13:20 |
mordred | woid: \o/ | 13:20 |
openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove support for extensions as lists https://review.openstack.org/540890 | 13:20 |
woid | mordred: :) thanks | 13:21 |
openstackgerrit | Matthieu Huin proposed openstack-infra/os-loganalyze master: Support ANSI coloring in logs https://review.openstack.org/563947 | 13:21 |
frickler | infra-root: hmm, seem like it is always this query that is the first one thats failing after a restart "GET /json/?method=pastes.getRecent HTTP/1.1" | 13:25 |
*** pbourke has quit IRC | 13:25 | |
frickler | might overload the database with the number of pastes we have by now. maybe block that somehow? | 13:26 |
woid | mordred: btw. can you shed some light on why you can't use rhel for gate? | 13:27 |
*** pbourke has joined #openstack-infra | 13:27 | |
*** udesale_ has joined #openstack-infra | 13:27 | |
mordred | woid: rhel requires licenses, which is a bit problematic for our policy of not using any tools in the gate that a developer would not be able to use locally | 13:28 |
mordred | BUT | 13:28 |
frickler | infra-root: we also seem to have some spam content in there, not sure how to handle that | 13:28 |
*** sree_ has joined #openstack-infra | 13:28 | |
mordred | woid: we were willing to attempt to put that aside a few years ago - and Red Hat and Rackspace together could not figure out how to grant us licenses to have rhel test nodes | 13:29 |
*** sree_ is now known as Guest77133 | 13:29 | |
mordred | woid: the final story from Red Hat in the end was "just use centos, it's close enough" | 13:29 |
*** sree has quit IRC | 13:29 | |
*** quique|rover|lch is now known as quiquell|rover | 13:29 | |
woid | mordred: I see - developers can get a free RH development license now, though. But I don't think this makes the licensing in CI any easier... | 13:31 |
mordred | woid: yah. and honestly, in CI centos is going to do 95% of what you'd want | 13:32 |
*** alexchadin has quit IRC | 13:33 | |
pabelanger | I still don't know if the developer preview license could be used in a CI environment, I've asked a few times, got no response | 13:33 |
smcginnis | fungi or others: How does one go about running an experimental job using bionic? And would we want to do that for all "core" projects as a sanity test? | 13:35 |
*** jaosorior has joined #openstack-infra | 13:35 | |
*** dhill_ has quit IRC | 13:38 | |
pabelanger | smcginnis: yah, we can check to see if we rebuilt the official release today, then project should be able to start gating on them. I believe we have devstack jobs in place already | 13:39 |
*** alexchadin has joined #openstack-infra | 13:40 | |
*** lpetrut_ has quit IRC | 13:40 | |
smcginnis | pabelanger: Do you happen to know the job name? I'd like to add it to the cinder experimental set. | 13:41 |
*** Nil_ has quit IRC | 13:41 | |
pabelanger | smcginnis: I think frickler added it | 13:43 |
fungi | smcginnis: dependnig on what job it is you want to run on bionic, it may not be defined yet but you ought to be able to add a variant to the .zuul.yaml in teh cinder repo | 13:44 |
*** ediardo has quit IRC | 13:44 | |
smcginnis | Is it just a matter of having the line "nodeset: ubuntu-bionic" in the job definition? | 13:44 |
fungi | i think it's as simple as adding teh job to the experimental pipeline and then including a nodeset parameter to override the default nodeset to ubuntu-bionic | 13:45 |
fungi | yeah | 13:45 |
smcginnis | fungi, pabelanger: Cool, thanks for the help. I'll try it out. | 13:45 |
*** esberglu has joined #openstack-infra | 13:46 | |
*** Guest77133 has quit IRC | 13:46 | |
*** sree has joined #openstack-infra | 13:46 | |
*** dhill_ has joined #openstack-infra | 13:47 | |
fungi | frickler: in the past when this has happened it's been crawlers trying to index the server and/or abusive clients rapid-fire posting content every few seconds (or faster). i'll take a look at the apache logs and see if we need to block anyone | 13:50 |
fungi | the python process for lodgeit does seem to be consuming an entire cpu pegged to ~100% | 13:51 |
*** sree has quit IRC | 13:51 | |
*** zhipeng has joined #openstack-infra | 13:51 | |
fungi | i'm going to restart it and then tail the logs, since trying to analyze logs while it's not responsive may show pathological results | 13:53 |
*** e0ne has joined #openstack-infra | 13:55 | |
*** yamamoto has quit IRC | 13:56 | |
aspiers | where's the best place to ask for advice / mentoring from the TC? | 13:57 |
fungi | yeah, i have a feeling there's some sort of abuse going on because it nearly filled up the rootfs with logs before daily rotation kicked in | 13:57 |
aspiers | doh, stupid question | 13:57 |
* aspiers joins #openstack-tc | 13:57 | |
*** ihar_ is now known as ihar | 14:01 | |
*** pcichy has joined #openstack-infra | 14:01 | |
*** caphrim007_ has quit IRC | 14:03 | |
dmsimard | fungi: some log stats from paste.o.o: https://etherpad.openstack.org/p/oaCCfxYmVE | 14:03 |
pabelanger | frickler: dmellado: are we able to delete the held node in nodepool? | 14:03 |
*** caphrim007 has joined #openstack-infra | 14:04 | |
fungi | dmsimard: i have indeed seen more than a few requests come through looking for /json/?method=pastes.getRecent | 14:04 |
*** caphrim007 has quit IRC | 14:04 | |
fungi | not sure yet whether that's the culprit | 14:05 |
fungi | and yeah, that top offender address is allocated to "Hetzner Online GmbH" in germany | 14:06 |
*** e0ne has quit IRC | 14:06 | |
fungi | presumably a broadband provider? | 14:06 |
fungi | researching | 14:06 |
fungi | ahh, hosting company | 14:06 |
fungi | they seem to have rolled their own cloud service too | 14:11 |
fungi | openstackish but i don't think it's actually openstack | 14:11 |
*** e0ne has joined #openstack-infra | 14:11 | |
*** jaosorior has quit IRC | 14:12 | |
*** esarault has joined #openstack-infra | 14:12 | |
frickler | fungi: for the fill up of the rootfs, that has been mysql backups. I deleted 20-28+30 earlier today because / was at 92% | 14:13 |
*** jaosorior has joined #openstack-infra | 14:13 | |
AJaeger | fungi: Hetzner is indeed a larger hoster in GErmany. | 14:13 |
*** e0ne has quit IRC | 14:14 | |
frickler | pabelanger: deleted. that seems to have been a stray hold, but I didn't get to investigate | 14:14 |
*** zoli|afk is now known as zoli | 14:15 | |
*** hemna_ has joined #openstack-infra | 14:15 | |
*** zoli is now known as zoli|wfh | 14:15 | |
*** zoli|wfh is now known as zoli | 14:15 | |
frickler | fungi: we may need to reduce the number of backups we keep a bit in the long term | 14:15 |
frickler | fungi: or add a volume if we have quota for that | 14:16 |
fungi | yeah, we don't need a ton of database backup retention | 14:16 |
corvus | fungi: which logs were filling up the fs? | 14:16 |
frickler | corvus: mysql backups | 14:17 |
corvus | frickler: ah i see that now :) | 14:17 |
fungi | dmsimard: the second highest volume client ip address in your list is giving a wide variety of different browser id strings. reverse dns is "pastecrawler.prg.ff.avast.com." and it seems to be ignoring our http://paste.openstack.org/robots.txt | 14:19 |
*** edmondsw has joined #openstack-infra | 14:19 | |
fungi | its network is registered to "AVAST cloud" in prague too | 14:19 |
dmsimard | fungi: was sidetracked by a meeting -- do we know what "/json/?method=pastes.getRecent" is actually used for ? | 14:20 |
*** e0ne has joined #openstack-infra | 14:20 | |
corvus | fungi, frickler, dmsimard: what if we blocked all connections with iptables for a bit, so we can verify functionality without questions about what other users/bots might be doing at the same time? | 14:21 |
corvus | we could answer questions like: is it possible to add a single new paste at all? does that getRecent method return at all under no load? | 14:21 |
dmsimard | corvus: probably easier to shut down apache and expose the :5000 on 0.0.0.0 | 14:21 |
*** ssbarnea_ has quit IRC | 14:21 | |
corvus | dmsimard: ++ | 14:21 |
fungi | corvus: dmsimard: agreed, good next step | 14:21 |
dmsimard | ok, let me take a stab at that | 14:21 |
fungi | it does definitely seem to spike right back up to 100% cpu utilization after a service restart | 14:22 |
dmsimard | btw it seems like paste01 isn't running puppet for some reason | 14:22 |
*** rajinir has joined #openstack-infra | 14:22 | |
dmsimard | I noticed because the bash history isn't timestamps | 14:22 |
dmsimard | timestamped* | 14:22 |
* frickler goes to enjoy afternoon traffic, bbl | 14:22 | |
corvus | dmsimard: i thought i saw ansipuppet entries in the syslog last night | 14:22 |
fungi | dmsimard: i see timestamps when i do a history | 14:23 |
dmsimard | that's odd, I don't | 14:23 |
fungi | fungi@paste01:~$ history |head | 14:24 |
fungi | 1 2018-04-26T14:23:00+0000 uptime | 14:24 |
*** armaan has joined #openstack-infra | 14:24 | |
dmsimard | https://paste.fedoraproject.org/paste/nGIXBY5STrx2nj7t95VABg | 14:24 |
*** e0ne has quit IRC | 14:25 | |
dmsimard | my personal user has timestamps but not root | 14:25 |
dmsimard | heh | 14:25 |
fungi | ahh | 14:25 |
dmsimard | stopping paste and apache, I'll start paste in foreground on 0.0.0.0. | 14:25 |
dmsimard | I haven't checked yet if port 5000 is opened | 14:26 |
fungi | it's not | 14:26 |
dmsimard | I'll punch a hole momentarily to help us test | 14:26 |
corvus | fungi, dmsimard: maybe could just ssh port-forward | 14:26 |
fungi | iptables says we have ssh and http open | 14:26 |
fungi | yeah, port forwarding or local curl should be fine, no need for modifying the fw | 14:27 |
dmsimard | sure | 14:27 |
dmsimard | local curl works fine | 14:28 |
dmsimard | but "curl 127.0.0.1:5000/json/?method=pastes.getRecent" hangs immediately | 14:28 |
fungi | possible we have a problem with the trove instance for it | 14:28 |
dmsimard | one curl is enough to peg the cpu to 100% | 14:28 |
corvus | fungi, dmsimard: i manually queried and inserted a row in the db to check that; those operations seemed fine | 14:29 |
fungi | #status log ze09 was rebooted 2018-04-26 08:15z due to a provider incident (CSHD-1AG24176PQz) but is back in working order | 14:29 |
openstackstatus | fungi: finished logging | 14:29 |
*** dhajare has quit IRC | 14:29 | |
*** kzaitsev_pi has quit IRC | 14:30 | |
corvus | getRecent: http://git.openstack.org/cgit/openstack-infra/lodgeit/tree/lodgeit/lib/webapi.py#n111 | 14:31 |
dmsimard | needs a kill -9 to get rid of the process too eh | 14:31 |
*** kzaitsev_pi has joined #openstack-infra | 14:31 | |
pabelanger | fungi: the reboot of ze09, did we shut it down or was it rebooted automagically? I wonder if we need to check for stuck jobs in zuul, which can happen if an executor is not shutdown cleanly | 14:31 |
fungi | pabelanger: rebooted automatically (hung hypervisor host) for the second day in a row | 14:32 |
pabelanger | ack, will keep an eye | 14:32 |
*** dpawlik has quit IRC | 14:32 | |
*** hongbin_ has joined #openstack-infra | 14:32 | |
fungi | `curl 127.0.0.1:5000/show/719879/` returns fairly quickly for me | 14:33 |
dmsimard | I somehow got the following warning in foreground | 14:33 |
dmsimard | /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/default_comparator.py:161: SAWarning: The IN-predicate on "pastes.parent_id" was invoked with an empty sequence. This results in a contradiction, which nonetheless can be expensive to evaluate. Consider alternative strategies for improved performance. 'strategies for improved performance.' % expr) | 14:33 |
fungi | looks like it's just sqla being nice and warning us we have an inefficient query | 14:34 |
dmsimard | yeah | 14:34 |
dmsimard | figure I'd mention it | 14:34 |
fungi | i have a suspicion that this avast.com paste crawler which is ignoring our robots.txt and repeatedly polling the very expensive pastes.getRecent method may be the cause | 14:35 |
corvus | if we don't git getRecent, does everything else work? | 14:35 |
corvus | s/git/hit/ | 14:35 |
fungi | yeah, show/719879 worked fine for me | 14:35 |
dmsimard | yeah the home page works quickly and so does a show on a regular paste | 14:36 |
fungi | though i'll dig through the apache logs and see if the hits to getRecent started around the same time as our cpu pegging | 14:36 |
*** ssbarnea_ has joined #openstack-infra | 14:36 | |
*** beekneemech is now known as bnemec | 14:37 | |
*** sree has joined #openstack-infra | 14:37 | |
dmsimard | I'm wondering if there's just one of the pastes which is corrupted for some reason (some fancy emoji?) and since it's being pulled up by the getRecent amount it's freezing it | 14:38 |
dmsimard | I'm browsing through the recent pastes | 14:38 |
corvus | fungi, dmsimard: i'd like to try something which may hold a db lock and affect other testing... can you let me know when it's okay to do so? | 14:38 |
fungi | the getRecent queries go back two weeks at a steady pace, so probably not the trigger | 14:38 |
fungi | corvus: go for it | 14:39 |
fungi | or at least combined with something more recent to start triggering this (some strange paste added around 23:35z yesterday might explain it) | 14:40 |
dmsimard | corvus: I've done the query I needed, go ahead | 14:40 |
fungi | since that's when the cpu first jumped up to 100% and never came back | 14:41 |
*** eernst has joined #openstack-infra | 14:41 | |
corvus | fungi, dmsimard: pastes have timestamps in the db if you want to look | 14:41 |
corvus | (my test isn't working yet, so i'm not mucking anything up at the moment, need a bit more infrastructure) | 14:42 |
*** sree has quit IRC | 14:42 | |
*** pcaruana has quit IRC | 14:42 | |
dmsimard | corvus: yeah, see the end of https://etherpad.openstack.org/p/oaCCfxYmVE -- there's a paste that stands out, it somehow has an old date but it pulls up fine when doing a curl on it | 14:42 |
dmsimard | the dates are also out of order despite being ordered by paste_id.. that's odd | 14:43 |
openstackgerrit | Merged openstack-infra/nodepool master: Fix awkward merging of code https://review.openstack.org/564045 | 14:43 |
corvus | dmsimard: one of those is probably me, i copied a timestamp. but only exactly one. | 14:43 |
corvus | the content would be 'testing' or something | 14:43 |
corvus | dmsimard: it's probably the highest paste #, meaning no new pastes since this started. | 14:44 |
dmsimard | oh | 14:44 |
dmsimard | ok that makes sense if it was created manually, sorry :P | 14:44 |
fungi | 719959 is from right around when the graph went bonkers though | 14:44 |
dmellado | pabelanger, sure, thanks! | 14:45 |
*** alexchadin has quit IRC | 14:45 | |
*** r-daneel has joined #openstack-infra | 14:45 | |
*** Nil_ has joined #openstack-infra | 14:46 | |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul-website master: WIP Use gatsby for static site generation https://review.openstack.org/564524 | 14:46 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul-website master: Migrate the content https://review.openstack.org/564525 | 14:46 |
corvus | fungi, dmsimard: i'm going to try to get an interactive python repl with lodgeit, so we can try some of the underlying queries | 14:46 |
dmsimard | corvus: want me to kill the foreground lodgeit ? | 14:46 |
fungi | corvus: sounds great | 14:46 |
corvus | dmsimard: nah, should be fine to continue poking. i need a few minutes to set up db connections | 14:46 |
dmsimard | corvus: if you do this in a screen session I'd like to watch :D | 14:47 |
*** quiquell|rover is now known as quiquell|off | 14:47 | |
openstackgerrit | Merged openstack-infra/nodepool master: Fix race in test_hold_expiration_no_default https://review.openstack.org/564314 | 14:47 |
corvus | dmsimard: okay, i'll finish prepping, then restart in screen when i have it figgered | 14:47 |
*** dklyle has joined #openstack-infra | 14:47 | |
*** apetrich has quit IRC | 14:49 | |
fungi | i wonder if show/719959 would also hang and spike cpu utilization | 14:49 |
fungi | since that was the last paste to be successfully added and occurred right during the cacti sample which took off | 14:50 |
fungi | and was a good 2.5 hours after the next most recent paste posted | 14:50 |
dmsimard | fungi: it seems like I've hanged it with exactly that paste actually | 14:50 |
*** udesale_ has quit IRC | 14:50 | |
dmsimard | unless corvus has locked things up | 14:51 |
corvus | nope | 14:51 |
dmsimard | ok, restarted the process and I'll try again. | 14:52 |
fungi | i'm betting something about 719959 is triggering whatever this behavior is, and any hits to getRecent are going to do the same while that's one of the most recent pastes | 14:52 |
fungi | and since there's a robot out there hitting getRecent every few minutes, explains why utilization pretty much immediately goes back to 100% | 14:52 |
corvus | yeah. i'm looking at it in the db. maybe syntax highlighting. | 14:52 |
dmsimard | fungi: yup, exactly the direction I was looking | 14:52 |
dmsimard | I added a thing on the etherpad, we can trigger the lock by hitting it | 14:53 |
dmsimard | locked again on that paste. | 14:53 |
*** edmondsw has quit IRC | 14:54 | |
corvus | i'm going to abandon my db/repl line of inquiry | 14:54 |
*** edmondsw has joined #openstack-infra | 14:54 | |
corvus | dmsimard: i'd like to modify that paste slightly and have you try again, ready for me to do that? | 14:55 |
*** jaosorior has quit IRC | 14:55 | |
dmsimard | corvus: I added a select on that paste_id in the pad | 14:56 |
corvus | dmsimard: i changed it from java to text; can you try loading it again? | 14:57 |
dmsimard | corvus: sure, let me try | 14:57 |
*** yamamoto has joined #openstack-infra | 14:57 | |
dmsimard | corvus: worked | 14:57 |
dmsimard | corvus: so something wrong with java highlighting ? | 14:57 |
corvus | seems that way | 14:57 |
*** armaan has quit IRC | 14:58 | |
*** felipemonteiro_ has joined #openstack-infra | 14:58 | |
dmsimard | I never realized lodgeit was something out of openstack-infra, is it really ? No commits since 2015 | 14:58 |
corvus | it's a fork of an abandoned project | 14:58 |
dmsimard | I know "if it's not broken don't fix it" | 14:58 |
dmsimard | ah. | 14:59 |
fungi | we're the last (as far as we know) remaining "maintained" fork | 14:59 |
fungi | so sort of de facto upstream | 14:59 |
*** felipemonteiro__ has joined #openstack-infra | 14:59 | |
*** eumel8 has quit IRC | 14:59 | |
dmsimard | corvus, fungi: now that this is resolved for the time being, I'll bring the service back up -- it should be fairly easy to reproduce and troubleshoot elsewhere | 15:00 |
corvus | dmsimard: wfm | 15:00 |
fungi | sounds good. thanks! | 15:00 |
*** r-daneel has quit IRC | 15:01 | |
*** sree has joined #openstack-infra | 15:01 | |
dmsimard | infra-root: paste.o.o is back up. Is it status worthy ? #status notice We've successfully troubleshooted the issue that prevented paste.openstack.org from loading and it's now back online, thank you for your patience. | 15:02 |
*** yamamoto has quit IRC | 15:03 | |
*** dbecker_ has quit IRC | 15:03 | |
fungi | sure | 15:03 |
*** rpioso|afk is now known as rpioso | 15:03 | |
*** felipemonteiro_ has quit IRC | 15:03 | |
*** niska has quit IRC | 15:03 | |
*** jistr|mtgs is now known as jistr | 15:04 | |
dmsimard | #status notice We've successfully troubleshooted the issue that prevented paste.openstack.org from loading and it's now back online, thank you for your patience. | 15:04 |
openstackstatus | dmsimard: sending notice | 15:04 |
*** rosmaita has quit IRC | 15:04 | |
*** sree has quit IRC | 15:04 | |
*** dpawlik has joined #openstack-infra | 15:04 | |
*** sree has joined #openstack-infra | 15:05 | |
clarkb | morning | 15:05 |
dmsimard | If we ever want to change our mind about lodgeit, paste.fedoraproject.org uses https://github.com/LINKIWI/modern-paste | 15:05 |
-openstackstatus- NOTICE: We've successfully troubleshooted the issue that prevented paste.openstack.org from loading and it's now back online, thank you for your patience. | 15:05 | |
*** links has quit IRC | 15:06 | |
corvus | dmsimard, fungi: i've reprod the problem with just pygments | 15:06 |
dmsimard | latest version of pygments ? | 15:07 |
corvus | dmsimard: version installed on paste | 15:07 |
dmsimard | I guess we're not pinning it | 15:07 |
corvus | i put code in the etherpad | 15:07 |
openstackstatus | dmsimard: finished sending notice | 15:07 |
*** zhipeng has quit IRC | 15:07 | |
corvus | dmsimard: we're using distro package: 2.1+dfsg-1 | 15:08 |
fungi | corvus: oh, wow, yep | 15:09 |
corvus | i'll try in a venv with latest | 15:09 |
dmsimard | Pygments==2.1 -- latest would be 2.2.0, there's a few fixes between 2.1 and 2.2.0 but I don't know if there's anything in there to help us http://pygments.org/docs/changelog/ | 15:09 |
dmsimard | corvus: good idea | 15:09 |
*** dpawlik has quit IRC | 15:09 | |
corvus | latest works | 15:09 |
dmsimard | "Added “Rainbow Dash” style. (PR#623)" well now I want the latest release of pygments. | 15:10 |
corvus | bionic has 2.2.0 | 15:11 |
*** rkukura has joined #openstack-infra | 15:11 | |
corvus | so we could play the odds until we upgrade. if this is a once-in-6-years event, hopefully we'll be upgraded by then :) | 15:12 |
dmsimard | corvus: 2.2.0 works, 2.1.3 doesn't. So there's something in 2.2.0 specifically for that. | 15:12 |
dmsimard | 2.2.0 is from January 2017 too, huh. | 15:12 |
mrhillsman | forumtopics.openstack.org unreacheable | 15:13 |
mrhillsman | connection refused | 15:14 |
*** bobh has joined #openstack-infra | 15:14 | |
corvus | i don't see forumtopics.openstack.org in our system manifest | 15:14 |
mrhillsman | not sure if we shut it down as it was time to or actual error/issue | 15:14 |
clarkb | mrhillsman: that is a mostly not managed by infra service, ttx is who you want to talk to | 15:14 |
clarkb | iirc we booted the instance in our tenant and handed it over to ttx | 15:14 |
mrhillsman | thx | 15:14 |
corvus | clarkb: :( | 15:14 |
clarkb | corvus: its the old old summit planning server iirc | 15:14 |
corvus | it sounds like the new summit planning server | 15:15 |
mordred | the summit planning server is dead, long live the summit planning server | 15:15 |
mrhillsman | lol | 15:15 |
corvus | clarkb, ttx: we have a world class system that lets anyone in the community help sysadmin all of our services. it'd be nice to use it. | 15:16 |
mrhillsman | people were still submitting even though deadline passed, thx | 15:16 |
mrhillsman | ttx disabled | 15:16 |
*** niska has joined #openstack-infra | 15:17 | |
corvus | clarkb, ttx: what's the plan there? | 15:21 |
corvus | clarkb, ttx: can we move it into puppet? | 15:21 |
clarkb | corvus: I want to say fungi was involved /me double checks on where it is hosted | 15:21 |
clarkb | I think the original plan was no more summit means we don't need it anymore | 15:22 |
clarkb | then forum things revived it? | 15:22 |
fungi | clarkb: corvus: ttx: i thought the last incarnation of forumtopics.o.o was hosted from the foundation's rax tenant, not ours | 15:22 |
clarkb | fungi: ah that may have been what happened, was old summit host was removed from our tenant then went to foundations as forum | 15:23 |
fungi | that is to say, bit infra's | 15:23 |
fungi | s/bit/not/ | 15:23 |
corvus | that's still :( | 15:23 |
fungi | agreed | 15:23 |
corvus | this should be something that's managed by the community and doesn't rely on just one person | 15:23 |
fungi | it's constantly lived as an experimental service, i think | 15:23 |
clarkb | and ya not seeing any hosts matching forum or summit in our main tenant in dfw | 15:23 |
*** rosmaita has joined #openstack-infra | 15:26 | |
clarkb | fungi: I'm shutting down our root screen on puppetmaster now (or at least leaving it if you want to and haven't already) | 15:27 |
clarkb | er no I will properly close the processes | 15:27 |
clarkb | infra-root https://review.openstack.org/#/c/564338/ is an update to how we run kick.sh in resposne to a semi painful experience yesterday | 15:28 |
clarkb | I would appreciate reviews to make sure I am not missing something | 15:28 |
fungi | clarkb: thanks | 15:29 |
ttx | Its disablement was actually a feature | 15:31 |
*** sree has quit IRC | 15:33 | |
corvus | ttx: yeah, mrhillsman let us know. but if you're going to keep running this service for future forums, can we run it as an infra service? we can still disable it at the appropriate times :) | 15:33 |
*** sree has joined #openstack-infra | 15:33 | |
*** esarault_ has joined #openstack-infra | 15:34 | |
ttx | corvus: yeah, so.. we stopped using it, then people said we could still hack it to do the proposal part, hence its survival up to today... I think we should just kill it and use some online form | 15:35 |
corvus | ttx: even better :) | 15:35 |
ttx | since people did not use the commenting function that much | 15:35 |
mrhillsman | would be good to have a submission closed message hehe | 15:35 |
ttx | I mean, if we keep it, it needs to be fixed, puppetized, etc high cost for very low value | 15:35 |
dmsimard | I've removed paste01 from the emergency file | 15:36 |
*** rosmaita has quit IRC | 15:36 | |
ttx | people still using it was just the path of least resistance. If we say " we can continue using it but we can't continue to run it so hackishly" then we'll just switch to something else | 15:36 |
ttx | It used to run the whole thing (submission, review, scheduling) and now it's just used for submission | 15:37 |
*** esarault has quit IRC | 15:37 | |
ttx | The code is full of kill switches to disable 99% of its features | 15:38 |
*** sree has quit IRC | 15:38 | |
ttx | makes me sad | 15:38 |
ttx | So in summary, not worth the time to properly run it since we only use 1% of its feature set those days | 15:39 |
corvus | ttx: ethercalc supports forms: https://ethercalc.openstack.org/wl98zy6oh8kj/app | 15:39 |
*** yamahata has quit IRC | 15:39 | |
*** gyee has joined #openstack-infra | 15:40 | |
ttx | corvus: yeah, that was my plan, but I delegated the work and then people revived forumtopics for one more round | 15:40 |
corvus | apparently you can make something happen after the send using a "panel" formula and a "copyvalue" formula. i'll leave that for later. it's enough to know you can do that for now. :) | 15:41 |
*** ykarel is now known as ykarel|away | 15:43 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-ethercalc master: Fix defined type namespacing https://review.openstack.org/564535 | 15:45 |
*** AJaeger has quit IRC | 15:46 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-drupal master: Fix defined type namespacing https://review.openstack.org/564537 | 15:47 |
*** ramishra has quit IRC | 15:49 | |
clarkb | Shrews: I'm eating breakfast but nodepool restarts were mentioned. Is that something you want to help do soonish? | 15:49 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-gerrit master: [DNM] Test puppet-4 puppet-apply jobs https://review.openstack.org/557515 | 15:50 |
*** ykarel|away has quit IRC | 15:50 | |
*** esarault_ has quit IRC | 15:50 | |
clarkb | Shrews: I figured we'd start with the builders as they are the easy one. If we restart the launchers that doesn't invalidate any in use nodes that zuul has locks on right? | 15:50 |
*** dhill_ has quit IRC | 15:53 | |
*** iranzo has quit IRC | 15:53 | |
*** jamesmcarthur has joined #openstack-infra | 15:53 | |
openstackgerrit | Eric Brown proposed openstack-infra/project-config master: Remove the project bandit from openstack https://review.openstack.org/564539 | 15:54 |
Shrews | clarkb: currently out for an appointment, but should be back in an hour or so. Feel free to go ahead if you like though. | 15:57 |
*** dpawlik has joined #openstack-infra | 15:57 | |
Shrews | clarkb: everything should be handled by zuul and nodepool gracefully | 15:58 |
*** hamzy has quit IRC | 15:59 | |
*** caphrim007 has joined #openstack-infra | 15:59 | |
*** yamamoto has joined #openstack-infra | 15:59 | |
clarkb | Shrews: cool thanks for confirming. Probably do rolling restarts then across launchers once the builders are done | 16:00 |
*** dhill_ has joined #openstack-infra | 16:01 | |
*** andreas_s has quit IRC | 16:01 | |
*** dpawlik has quit IRC | 16:01 | |
*** caphrim007 has quit IRC | 16:02 | |
*** caphrim007 has joined #openstack-infra | 16:02 | |
*** caphrim007 has quit IRC | 16:02 | |
*** caphrim007 has joined #openstack-infra | 16:02 | |
*** caphrim007 has quit IRC | 16:02 | |
*** d0ugal has quit IRC | 16:03 | |
*** caphrim007 has joined #openstack-infra | 16:04 | |
*** jamesmcarthur has quit IRC | 16:04 | |
*** d0ugal has joined #openstack-infra | 16:04 | |
*** andreas_s_ has joined #openstack-infra | 16:04 | |
*** yamamoto has quit IRC | 16:04 | |
*** andreas_s_ has quit IRC | 16:08 | |
*** jamesmcarthur has joined #openstack-infra | 16:09 | |
ttx | corvus: one issue with EtherCalc data forms last time I considered them is that anyone can modify everything, so it's not a lot better than an Etherpad | 16:10 |
corvus | ttx: yeah, more like etherpad with prompts | 16:10 |
*** agopi|zzz has quit IRC | 16:11 | |
*** owalsh_afk has quit IRC | 16:11 | |
ttx | so we kept using forumtopics since it's still incrementally better | 16:12 |
*** owalsh has joined #openstack-infra | 16:12 | |
*** jtomasek has quit IRC | 16:14 | |
*** jchhatbar has quit IRC | 16:14 | |
*** rosmaita has joined #openstack-infra | 16:16 | |
fungi | teh alternative of using a survey platform is also problematic, i suppose, in that it doesn't provide a means for revising or subsequent commenting on proposed sessions either | 16:18 |
*** yamahata has joined #openstack-infra | 16:18 | |
clarkb | I hate using gerrit as a hammer to every nail but doesn't gerrit kinda solve this problem | 16:19 |
anteaya | fungi I think there may be a setting in limesurvey that allows for editing of ones response | 16:19 |
clarkb | you propose content that can be updated but has strong historical tracking of edits | 16:19 |
clarkb | then you approve all the changes that are "accepted" | 16:19 |
anteaya | fungi however the editing may only be allowable prior to submission | 16:19 |
clarkb | but maybe gerrit is too much work for some of the intended user audience | 16:20 |
fungi | anteaya: cool, so partly viable, if we handle the subsequent publication and follow-up responses in other ways | 16:20 |
*** gfidente has quit IRC | 16:20 | |
fungi | also having private submission followed by public reporting at the end loses the opportunity to see if someone else has already proposed a similar/identical session before you duplicate it | 16:21 |
*** rosmaita has quit IRC | 16:21 | |
anteaya | fungi I'm looking for confirmation of my assumptions now | 16:21 |
*** owalsh has quit IRC | 16:24 | |
openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: update the announce-release job to look in releases repo for tools https://review.openstack.org/564329 | 16:26 |
clarkb | dhellmann: smcginnis ^ fyi | 16:26 |
*** owalsh has joined #openstack-infra | 16:26 | |
anteaya | have to go make a phone call, will continue to search upon my return | 16:27 |
smcginnis | clarkb: Thank you | 16:30 |
*** edmondsw has quit IRC | 16:31 | |
*** pblaho has quit IRC | 16:33 | |
*** agopi|zzz has joined #openstack-infra | 16:34 | |
mugsie | hey - is there any chance someone could add me to https://review.openstack.org/#/admin/groups/1874,members ? | 16:37 |
*** lucasagomes is now known as lucas-afk | 16:38 | |
mugsie | (I created the group, and forgot to get myself added) | 16:38 |
clarkb | mugsie: looking | 16:40 |
mugsie | clarkb: thanks | 16:40 |
*** jpich has quit IRC | 16:41 | |
corvus | clarkb: git is the 'stupid content tracker'. it's not just for source code :) | 16:42 |
clarkb | mugsie: done | 16:42 |
mugsie | clarkb: ty | 16:42 |
*** felipemonteiro__ has quit IRC | 16:45 | |
*** felipemonteiro has joined #openstack-infra | 16:45 | |
*** jpena is now known as jpena|off | 16:45 | |
*** panda|ruck is now known as panda|ruck|off | 16:49 | |
openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools master: finish moving announce.sh to releases repo by deleting it https://review.openstack.org/564559 | 16:50 |
*** germs has joined #openstack-infra | 16:50 | |
*** germs has quit IRC | 16:50 | |
*** germs has joined #openstack-infra | 16:50 | |
openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools master: finish moving announce.sh to releases repo by deleting it https://review.openstack.org/564559 | 16:50 |
clarkb | I am restarting the nodepool builder on nb01 now then will do 02 | 16:51 |
clarkb | corvus: Shrews: e9b82226a5641042e1aad1329efa6e3b376e7f3a is the sha1 on the builders | 16:52 |
openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Switch to stestr https://review.openstack.org/536862 | 16:52 |
clarkb | now I'm going to figure out what our least active nl0x host is and restart it first and make sure it can do the nodepooling before doing the other 3 | 16:54 |
*** ediardo has joined #openstack-infra | 16:55 | |
*** agopi|zzz has quit IRC | 16:55 | |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Cache configuration objects in addition to YAML dicts https://review.openstack.org/564061 | 16:55 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Change TestMaxTimeout to not run ansible https://review.openstack.org/564562 | 16:56 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Store source context on all config objects https://review.openstack.org/564563 | 16:56 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Stop deep-copying job variables https://review.openstack.org/564564 | 16:56 |
clarkb | nl02 is our lucky winner, restarting it now | 16:56 |
*** lbragstad[m] has joined #openstack-infra | 16:57 | |
clarkb | limestone which is nl02 cloud just marked a ndoe ready | 16:59 |
clarkb | I haven't noticed any tracebacks in the log tail either | 16:59 |
clarkb | I'll give it a few more minutes then restart the other three | 17:00 |
openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Switch openstack-tox-py36 to ubuntu-bionic https://review.openstack.org/564565 | 17:01 |
clarkb | same sha1 on the launchers too | 17:01 |
*** yamamoto has joined #openstack-infra | 17:01 | |
pabelanger | clarkb: fungi: dhellmann: Updates openstack-tox-py36 to ubuntu-bionic^, we can then start to add that into projects for non-voting | 17:02 |
*** derekh_ has quit IRC | 17:03 | |
*** yamamoto has quit IRC | 17:06 | |
*** lbragstad has quit IRC | 17:07 | |
*** baoli has joined #openstack-infra | 17:08 | |
dhellmann | pabelanger : nice. did you see smcginnis' question on the ML about using py3 instead of py36? http://lists.openstack.org/pipermail/openstack-dev/2018-April/129888.html | 17:08 |
pabelanger | dhellmann: I did, I can reply with thoughts | 17:10 |
clarkb | #status log nb0[12], nl0[1-4] restarted nodepool services to pick up recent chagnes to nodepool. All running e9b82226a5641042e1aad1329efa6e3b376e7f3a of nodepool now. | 17:10 |
openstackstatus | clarkb: finished logging | 17:10 |
clarkb | Shrews: corvus ^ I guess watch it and release on monday? | 17:10 |
*** agopi|zzz has joined #openstack-infra | 17:10 | |
*** dtantsur is now known as dtantsur|afk | 17:10 | |
clarkb | dhellmann: pabelanger I too was going to reply | 17:10 |
clarkb | py3 isn't a thing in tox like py3x is aiui | 17:10 |
clarkb | we can make our own py3 but its less toxic/tox like? | 17:11 |
dhellmann | I thought smcginnis said it worked for him? | 17:11 |
dhellmann | maybe I misunderstood what parts worked | 17:11 |
* clarkb double checks but is fairly certain he tested this at one point and it wasn't special like the built in py envs | 17:11 | |
dhellmann | but yeah, if it's not automatic with tox we shouldn't do it | 17:11 |
*** dklyle has quit IRC | 17:12 | |
*** hamzy has joined #openstack-infra | 17:12 | |
*** germs has quit IRC | 17:13 | |
*** trown is now known as trown|lunch | 17:13 | |
*** germs has joined #openstack-infra | 17:13 | |
*** germs has quit IRC | 17:13 | |
*** germs has joined #openstack-infra | 17:13 | |
*** ssbarnea_ has quit IRC | 17:14 | |
*** ansiwen[q] has joined #openstack-infra | 17:15 | |
clarkb | I am/was wrong. Looks like tox added support for py3 in 2016 | 17:15 |
clarkb | (py2 is also valid) | 17:16 |
dhellmann | that makes it interesting | 17:16 |
dhellmann | so then the trade off is between using explicit versions so we can decide when to update to a new version and using the implied version so we don't have to take the explicit step but possibly getting different things on different platforms | 17:17 |
*** baoli has quit IRC | 17:17 | |
pabelanger | I've replied to ML | 17:18 |
corvus | clarkb, Shrews: sounds like a plan | 17:18 |
clarkb | dhellmann: yup | 17:18 |
clarkb | dhellmann: I think that is less of an issue with python3 after 3.5 (at least compared to 3.6 and 3.7) | 17:19 |
fungi | confirmed, one of my personal projects which is python 3.x only uses basepython = python3 in its tox.ini to much contentment | 17:19 |
dhellmann | clarkb : I think we've had one or two issues, but in general I agree | 17:19 |
dhellmann | fungi : oh, yeah, that's definitely possible. the question is whether we can use an env named "py3" | 17:19 |
fungi | oh | 17:19 |
clarkb | dhellmann: so we might generally say python3.5 is expected to wrok and explicitly test on that version too | 17:20 |
fungi | right i hadn't tested that but glad to know it works | 17:20 |
clarkb | dhellmann: then py3 rolling forward? I don't know what the lifespan of python3.5 is supposed to be but it ended up on a lot of distros aiui | 17:20 |
clarkb | a244dec0cc4ba4e85e42893e6c3ce1a0e3c1290f is the commit that added support for py3 and py2 to tox fwiw | 17:20 |
fungi | my test environments have python minor versions 3.3 through 3.7 and i test against all of them so no "default" | 17:20 |
dhellmann | the uncertainty of what is going to actually be tested makes me a little uncomfortable | 17:20 |
clarkb | it was easy with python2 when we really only had two options to worry about | 17:21 |
dhellmann | fungi : I suspect you would get whatever version you end up with when you run "python3" on the command line outside of tox or a virtualenv | 17:21 |
fungi | dhellmann: right, which i do control (via a symlink in ~/bin) | 17:21 |
*** AJaeger has joined #openstack-infra | 17:21 | |
clarkb | you need a ~tox===2.5.0 to support py2/py3 which isn't that new | 17:21 |
clarkb | from a tooling perspective I think this is fine then | 17:22 |
dhellmann | yeah, I think we *could* do it, I'm still not sure we *should* | 17:22 |
clarkb | yup definitely a is this good for our testing needs question on top of can the tools support it | 17:22 |
dhellmann | right | 17:23 |
dhellmann | let's see where that ML thread takes us, then? | 17:23 |
clarkb | sure | 17:23 |
fungi | yeah, i can see positives and negatives for both options | 17:23 |
dhellmann | maybe we want to use a job template that is just python3 but have more explicit job names used in that template | 17:24 |
* mordred has started using pyenv for python versions and has given up on his distro-provided python | 17:24 | |
dhellmann | that way we can change the template easily as we upgrade or add multiple versions | 17:24 |
*** germs has quit IRC | 17:24 | |
fungi | mordred: me too re distro python, though i just build from source | 17:24 |
mordred | yah. I just let pyenv build from source for me | 17:25 |
*** germs has joined #openstack-infra | 17:25 | |
*** germs has quit IRC | 17:25 | |
*** germs has joined #openstack-infra | 17:25 | |
fungi | it's only ~4 commands, so hasn't been that taxing when i want to update | 17:25 |
AJaeger | corvus, if we retire a project, should we remove .zuul.yaml from all stable branches as well? I think we should to not be stuck with any dependencies... | 17:25 |
fungi | most of the hassle is keeping track of what build optimization flags were supported for which 3.x series | 17:26 |
*** ykarel|away has joined #openstack-infra | 17:26 | |
*** germs has quit IRC | 17:26 | |
*** germs has joined #openstack-infra | 17:26 | |
*** germs has quit IRC | 17:26 | |
*** germs has joined #openstack-infra | 17:26 | |
clarkb | looks like python3.6 will be supported for 24 months after 3.7 is released and 3.7 hasn't released yet | 17:26 |
fungi | AJaeger: we could, but i think once the repo gets removed from the projects list zuul will ignore those anyway? | 17:26 |
* mordred can't wait until we're on python 3.6 | 17:27 | |
clarkb | I can't find a similar document for 3.5 | 17:27 |
fungi | clarkb: yeah, i've been building 3.7 from release candidates | 17:27 |
fungi | and alpha/beta tags before that | 17:27 |
corvus | AJaeger: we can drop it from the zuul tenant config and then we won't care | 17:27 |
clarkb | but I think 3.5 is a reasonable python3 line in sand for now since the changes to 3.6 and 3.7 are relatively minor aiui | 17:27 |
clarkb | and you get the full power of asyncio | 17:27 |
AJaeger | corvus: ah, indeed - then we're fine. | 17:28 |
fungi | er, i guess v3.7.0b3 is current, so no rc yet (dunno why i thought there was) | 17:28 |
clarkb | ya py37 seems like small backward compatible improvements unless you really hate utf8 | 17:30 |
*** shardy_ has quit IRC | 17:31 | |
mordred | clarkb: 3.6 has format strings. I am bursting at the seems to be able to switch to using them | 17:32 |
mordred | also - according to a python core dev I was talking to at pyconit - the format strings are also more efficient - as are dicts in py3.6 | 17:32 |
mordred | but yes, 3.5is a fine choice for today | 17:32 |
clarkb | mordred: the major dict efficiency work is in 3.5 too aiui its why we have had so many bugs with yaml and dicts in zuul | 17:33 |
clarkb | turns out ansible is a good test case for that particular problem area | 17:33 |
*** ykarel|away has quit IRC | 17:33 | |
mordred | clarkb: the 3.6 dicts are apparently better with memory - and also are ordered by default now | 17:33 |
*** SumitNaiksatam has joined #openstack-infra | 17:33 | |
fungi | nowai! | 17:34 |
*** eernst has quit IRC | 17:34 | |
mordred | according to Christian Heimes | 17:34 |
clarkb | I'm guessing the order is hash specific | 17:34 |
mordred | no - insert order ordered | 17:34 |
fungi | so with different 3.6 patch versions on random distros, my dict ordering will still be consistent? | 17:34 |
mordred | like the ordereddict impl | 17:34 |
mordred | yup | 17:34 |
clarkb | mordred: wait isn't that a security issue and why python3 went random by default? | 17:35 |
* fungi swoons | 17:35 | |
*** e0ne has joined #openstack-infra | 17:35 | |
openstackgerrit | Eric Brown proposed openstack-infra/project-config master: Remove the project bandit from openstack https://review.openstack.org/564539 | 17:35 |
mordred | clarkb: I do not know | 17:35 |
mordred | fungi: right? | 17:35 |
*** eernst has joined #openstack-infra | 17:35 | |
clarkb | oh I bet its still hash specific as far as table collisions go | 17:35 |
clarkb | since the actual issue was less order and being able to predict hash collisions iirc | 17:35 |
fungi | clarkb: right | 17:36 |
mordred | ah - yah | 17:36 |
clarkb | (you just happened to get deterministic order out of predictable hashing) | 17:36 |
corvus | (as ordereddict shows, you can maintain a key order independent of the underlying hash) | 17:36 |
fungi | cve-2012-1150 for reference | 17:36 |
mordred | ++ | 17:36 |
corvus | though -- generally i would think that involves extra memory (an extra pointer) | 17:37 |
corvus | so now i'm curious about "use less memory plus be ordered" :) | 17:37 |
fungi | https://bugs.python.org/issue13703 | 17:37 |
clarkb | TIL @ is an operator as of 3.5 | 17:38 |
clarkb | (matrix multiplication) | 17:38 |
fungi | ooh! | 17:38 |
* fungi wants to math geek on that but has other things to work on already | 17:38 | |
mordred | >>> a=1 | 17:40 |
mordred | >>> f'this is {a}' | 17:40 |
mordred | that's my favorite 3.6 thing | 17:40 |
openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add openstack-python36-jobs / nonvoting templates https://review.openstack.org/564569 | 17:41 |
mordred | no more 'this is {a}'.format(a=a) ... AND it's apparently also more efficient | 17:41 |
fungi | danger will robinson | 17:41 |
fungi | at least with foo.format(a=a) you're being explicit about which variables are allowed in your format string | 17:42 |
*** apetrich has joined #openstack-infra | 17:42 | |
*** esberglu has quit IRC | 17:42 | |
fungi | if you have sensitive values in context and operate on user-supplied format strings (bad idea, yes) then things could go bad really quickly for you | 17:42 |
mordred | fungi: the f'' prefix is required for the magic | 17:42 |
mordred | so you can't have user-supplied format strings and use f to my konwledge | 17:42 |
openstackgerrit | Merged openstack-infra/zuul-jobs master: Add installation/deprecation documentation https://review.openstack.org/560463 | 17:43 |
*** zoli is now known as zoli|gone | 17:43 | |
fungi | well, seems like you could recast a string as a format string, but i'd have to test that | 17:43 |
*** zoli|gone is now known as zoli | 17:43 | |
fungi | anyway, yes, terrible idea to allow that | 17:43 |
fungi | ahh, f'blah' is interpreted at the time of assignment | 17:44 |
fungi | this does make it a little easier to be safe about | 17:45 |
fungi | so if you a=1;foo=f'{a}';a=2;print(foo) you get '1' | 17:45 |
*** germs has quit IRC | 17:46 | |
*** amoralej is now known as amoralej|off | 17:46 | |
*** germs has joined #openstack-infra | 17:47 | |
*** germs has quit IRC | 17:47 | |
*** germs has joined #openstack-infra | 17:47 | |
fungi | so f'blah' is not a different type than 'blah' like b'blah' is (or u'blah' used to be) | 17:47 |
fungi | f is more like an operator | 17:47 |
mordred | yes! | 17:47 |
fungi | but what gets stored in memory is just a normal str type | 17:48 |
fungi | okay, i can get behind this | 17:48 |
mordred | https://www.python.org/dev/peps/pep-0498/#examples-from-python-s-source-code <-- has some fun examples for comparisons | 17:48 |
fungi | inflexible as it makes it harder to reuse predefined format strings with different inputs i suppose | 17:49 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Allow extra time for some ansible tests https://review.openstack.org/564572 | 17:49 |
mordred | fungi: yay! I was going to be sad if we were going to be cagematching over use of the feature :) | 17:49 |
fungi | ready... fight! | 17:50 |
* corvus puts away popcorn | 17:50 | |
*** felipemonteiro has quit IRC | 17:51 | |
*** felipemonteiro has joined #openstack-infra | 17:51 | |
*** rpittau_ has joined #openstack-infra | 17:57 | |
fungi | i need to step out for a bit, but should be back in less than an hour | 17:57 |
*** dpawlik has joined #openstack-infra | 17:58 | |
*** e0ne has quit IRC | 18:00 | |
mordred | corvus: can I have some of the popcorn - that sounds tasty | 18:02 |
*** yamamoto has joined #openstack-infra | 18:02 | |
*** jamesmcarthur has quit IRC | 18:02 | |
*** dpawlik has quit IRC | 18:03 | |
*** yamamoto has quit IRC | 18:07 | |
*** eernst has quit IRC | 18:08 | |
* Shrews hands mordred a box of rhinoceros flavored popcorn | 18:08 | |
mordred | mmm. rhinocerosy | 18:10 |
mordred | I can't even say that word in my head | 18:10 |
andreykurilin | hi folks! do we have any plan(dates) about using poly-gerrit? | 18:13 |
*** trown|lunch is now known as trown | 18:16 | |
clarkb | andreykurilin: it requires us to upgrade to 2.14 or 2.15 so when that happens we'll get poly-gerrit | 18:17 |
*** felipemonteiro_ has joined #openstack-infra | 18:17 | |
clarkb | andreykurilin: right now step 0 in that process is upgrading the base os for gerrit on may 2nd (you may have seen emails to the dev list about that) which gives us new enough java (8) to run 2.14/2.15 | 18:17 |
clarkb | andreykurilin: then we have to sort out how long of an upgrade process it will be to convert our database to nodedb and do the install and take the downtime | 18:18 |
andreykurilin | clarkb: thanks for explanation | 18:18 |
clarkb | andreykurilin: its also possible we can incrementally convert the notedb before upgrading, then take a shorter downtime to do the final sync. But this is completely untested | 18:19 |
*** felipemonteiro has quit IRC | 18:21 | |
*** lbragstad has joined #openstack-infra | 18:22 | |
*** dklyle has joined #openstack-infra | 18:22 | |
*** e0ne has joined #openstack-infra | 18:22 | |
*** sree has joined #openstack-infra | 18:23 | |
*** tesseract has quit IRC | 18:24 | |
smcginnis | clarkb, dhellmann: Yep, just doing tox -e py3 worked for me. | 18:25 |
smcginnis | We may want tooling and things that aren't that critical as to specific minor version to just define "basepython = python3" to minimize updates over time. If we still want to be explicit with ut and functional test runs. | 18:28 |
*** sree has quit IRC | 18:28 | |
pabelanger | clarkb: fungi: could I get a +3 on ianw changes for gerrit: https://review.openstack.org/#/q/status:open+project:openstack-infra/gerrit+branch:openstack/2.13 includes some changes for bindep and removal of install-distro-packages.sh | 18:30 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-drupal master: Fix defined type namespacing https://review.openstack.org/564537 | 18:30 |
*** lpetrut has joined #openstack-infra | 18:32 | |
*** esberglu has joined #openstack-infra | 18:32 | |
clarkb | pabelanger: ya I'll take a look after lunch | 18:34 |
*** SumitNaiksatam has left #openstack-infra | 18:34 | |
*** ssbarnea_ has joined #openstack-infra | 18:35 | |
pabelanger | thanks | 18:35 |
*** ssbarnea_ has quit IRC | 18:36 | |
*** ssbarnea_ has joined #openstack-infra | 18:37 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-zuul master: Fix operatingsystem major release fact name https://review.openstack.org/564577 | 18:40 |
openstackgerrit | Goutham Pacha Ravi proposed openstack-infra/project-config master: Gerritbot: Add stable branch notifications to manila https://review.openstack.org/564578 | 18:41 |
*** harlowja has joined #openstack-infra | 18:46 | |
*** eernst has joined #openstack-infra | 18:48 | |
*** sree has joined #openstack-infra | 18:50 | |
*** rkukura has quit IRC | 18:51 | |
*** xinliang has quit IRC | 18:51 | |
*** sree has quit IRC | 18:55 | |
*** rpittau_ has quit IRC | 18:57 | |
*** mriedem has quit IRC | 18:58 | |
anteaya | sorry for the delay in reporting back, what I thought was the ability to edit responses appears to not be correct | 19:01 |
anteaya | if the survey admin toggles the 'update response' function, a given token can be used more than once | 19:02 |
anteaya | however when reusing a token for submitted responses, the first set of responses are not visible to the token holder, just the questions | 19:02 |
anteaya | then any submitted responses have their own id and are saved as an additional set of responses associated with that token | 19:03 |
anteaya | limesurvey's definition of 'update' and mine seem to be a bit at odds | 19:04 |
*** yamamoto has joined #openstack-infra | 19:04 | |
*** mriedem has joined #openstack-infra | 19:06 | |
*** xinliang has joined #openstack-infra | 19:07 | |
*** xinliang has quit IRC | 19:07 | |
*** xinliang has joined #openstack-infra | 19:07 | |
*** yamamoto has quit IRC | 19:09 | |
*** sree has joined #openstack-infra | 19:10 | |
fungi | ahh, so the user can retake the survey and get their original responses replaced by whatever the enter the next time | 19:10 |
fungi | er, they enter | 19:11 |
fungi | but not exactly go back and edit their old responses | 19:11 |
*** e0ne has quit IRC | 19:14 | |
*** agopi|zzz has quit IRC | 19:14 | |
*** imacdonn has quit IRC | 19:14 | |
anteaya | and not replaced, saved as well | 19:14 |
*** imacdonn has joined #openstack-infra | 19:14 | |
fungi | oh, interesting | 19:14 |
fungi | yeah, rereading what you said | 19:15 |
fungi | "additional set of responses associated with that token" | 19:15 |
anteaya | and yes, so far I don't see the functionality of being able to see your old answers to say for instance add punctuation | 19:15 |
anteaya | yes, additional set | 19:15 |
*** yamahata has quit IRC | 19:15 | |
anteaya | now the interesting thing is the admin can edit the responses | 19:17 |
anteaya | which to me feels odd but the functionality is there | 19:17 |
*** armaan has joined #openstack-infra | 19:19 | |
anteaya | the admin can edit in place | 19:19 |
*** sree has quit IRC | 19:20 | |
*** e0ne has joined #openstack-infra | 19:21 | |
*** e0ne has quit IRC | 19:21 | |
*** armaan has quit IRC | 19:22 | |
*** ediardo has quit IRC | 19:23 | |
*** eharney has quit IRC | 19:24 | |
*** sree has joined #openstack-infra | 19:24 | |
fungi | does this look to anyone else like we're missing a task to put these files on the tarballs site? http://logs.openstack.org/3b/3b1f9de37fb54d692706fda36fc3580f9489f4a0/post/legacy-laravel-openstackid-release-branch/3994868/job-output.txt.gz#_2018-04-25_19_10_51_894399 | 19:32 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-openstackci master: Fix scope of vhost template variable https://review.openstack.org/564600 | 19:36 |
pabelanger | fungi: yah, it isn't using the publish-openstack-artifacts job | 19:36 |
fungi | role? | 19:38 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-storyboard master: Use versioncmp function https://review.openstack.org/564601 | 19:38 |
pabelanger | fungi: no, it will need to parent to it because it will need to use the secret for tarballs.o.o SSH key | 19:38 |
fungi | ahh... | 19:38 |
pabelanger | my guess, is we never updated it to the new process | 19:39 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-refstack master: Use versioncmp function https://review.openstack.org/564602 | 19:40 |
fungi | well, this was an imported v2 job which has been broken ever since, so trying to work out the missing bits and make sure something is working before we embark on trying to completely rewrite it | 19:40 |
*** sree has quit IRC | 19:40 | |
fungi | especially since i'm not too strong on php distribution building frameworks | 19:40 |
*** VW has quit IRC | 19:42 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Use instance variables in puppet template https://review.openstack.org/564603 | 19:43 |
pabelanger | I think we could write a publish-openstack-php-content job in project-config, that would parent to publish-openstack-artifact here | 19:43 |
*** jamesmcarthur has joined #openstack-infra | 19:44 | |
fungi | pabelanger: so here https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-jobs.yaml#n670 i can just change the parent from legacy-base to legacy-publish-openstack-artifacts for now, i guess? | 19:44 |
pabelanger | looking | 19:44 |
pabelanger | fungi: ah, yes. I think that is right. | 19:45 |
fungi | i'll give it a shot | 19:45 |
pabelanger | I didn't realize we had legacy-publish-openstack-artifacts | 19:45 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-subunit2sql master: Use instance variables in puppet template https://review.openstack.org/564606 | 19:45 |
*** felipemonteiro__ has joined #openstack-infra | 19:46 | |
*** felipemonteiro_ has quit IRC | 19:46 | |
*** rosmaita has joined #openstack-infra | 19:47 | |
openstackgerrit | Jeremy Stanley proposed openstack-infra/openstack-zuul-jobs master: More fix laravel branch and master publish jobs https://review.openstack.org/564608 | 19:48 |
*** dhill_ has quit IRC | 19:48 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-kibana master: Use instance variables in puppet template https://review.openstack.org/564609 | 19:48 |
fungi | pabelanger: ^ like that | 19:48 |
fungi | (564608) | 19:48 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-etherpad_lite master: Fix defined type namespacing https://review.openstack.org/564610 | 19:50 |
pabelanger | looking | 19:51 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-gerrit master: [DNM] Test puppet-4 puppet-apply jobs https://review.openstack.org/557515 | 19:53 |
pabelanger | fungi: I think i see an issue, one moment | 19:53 |
fungi | k | 19:54 |
fungi | this is why we have reviews! ;) | 19:54 |
pabelanger | fungi: okay, I believe you will also have to update http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/laravel-openstackid-release-master/post.yaml#n30 and http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/laravel-openstackid-release-branch/post.yaml#n30 to use the new folder structure of 'artifacts' like we did in: | 19:58 |
pabelanger | http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/post-sahara-extra-artifacts/post.yaml | 19:58 |
pabelanger | otherwise, we'll sync back the data from the node to the executor to the wrong folder | 19:58 |
pabelanger | and the upload to tarballs.o.o will fail: http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/publish-artifacts-to-fileserver/tasks/main.yaml | 19:58 |
fungi | oh, good catch | 19:59 |
*** dpawlik has joined #openstack-infra | 19:59 | |
*** dhill_ has joined #openstack-infra | 20:00 | |
slaweq | hi, people with +2 power in project-config repo :) Please check small patch https://review.openstack.org/#/c/561930/ if You will have a minute | 20:02 |
slaweq | thx in advance | 20:02 |
*** lpetrut has quit IRC | 20:02 | |
*** dpawlik has quit IRC | 20:03 | |
*** yamamoto has joined #openstack-infra | 20:05 | |
openstackgerrit | Jeremy Stanley proposed openstack-infra/openstack-zuul-jobs master: More fix laravel branch and master publish jobs https://review.openstack.org/564608 | 20:06 |
fungi | pabelanger: okay, so like that ^ | 20:06 |
*** sree has joined #openstack-infra | 20:10 | |
*** salv-orl_ has joined #openstack-infra | 20:11 | |
*** salv-orlando has quit IRC | 20:11 | |
*** yamamoto has quit IRC | 20:11 | |
fungi | ianw: heads up, in case you missed it, pabelanger had a comment along with his +2 on https://review.openstack.org/555602 with a suggestion for some followup cleanup | 20:13 |
*** sree has quit IRC | 20:14 | |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Cache configuration objects in addition to YAML dicts https://review.openstack.org/564061 | 20:15 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Stop deep-copying job variables https://review.openstack.org/564564 | 20:15 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Allow extra time for some ansible tests https://review.openstack.org/564572 | 20:15 |
pabelanger | fungi: yah, that looks right | 20:15 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-refstack master: Use versioncmp function https://review.openstack.org/564602 | 20:16 |
*** bobh has quit IRC | 20:25 | |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Cache configuration objects in addition to YAML dicts https://review.openstack.org/564061 | 20:33 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Stop deep-copying job variables https://review.openstack.org/564564 | 20:33 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Allow extra time for some ansible tests https://review.openstack.org/564572 | 20:33 |
*** kgiusti has left #openstack-infra | 20:37 | |
*** agopi|zzz has joined #openstack-infra | 20:37 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-zuul master: Install libre2-dev package https://review.openstack.org/564621 | 20:38 |
*** eernst has quit IRC | 20:38 | |
*** yolanda__ has quit IRC | 20:38 | |
*** bobh has joined #openstack-infra | 20:39 | |
*** sshnaidm is now known as sshnaidm|off | 20:39 | |
openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: import the git-nit repository https://review.openstack.org/564622 | 20:40 |
*** yolanda__ has joined #openstack-infra | 20:43 | |
*** eharney has joined #openstack-infra | 20:43 | |
*** hamzy has quit IRC | 20:45 | |
corvus | dhellmann: there's a two-step process for adding things now; the zuul.d/projects.yaml part has to happen in a second change: https://docs.openstack.org/infra/manual/creators.html#adding-the-project-to-the-ci-system | 20:46 |
dhellmann | ok, I was just re-reading that | 20:46 |
corvus | dhellmann: the *good news* is that it's no longer a 3-step process :) | 20:46 |
dhellmann | we used to have great big notes "stop here and wait" when we needed to do that; is this change new | 20:47 |
dhellmann | well, heh, ok | 20:47 |
dhellmann | so I do the gerrit stuff in 1 patch and the zuul stuff in another? | 20:47 |
fungi | the two changes can be (and are preferred to be) pushed together | 20:47 |
fungi | it's just that the second one will fail tests until the first merges | 20:47 |
dhellmann | yeah, I may not have read closely enough to realize that I needed 2 patches | 20:48 |
corvus | dhellmann: gerrit+zuul/main.yaml in one. zuul.d/projects.yaml in the second. | 20:48 |
dhellmann | I mean, clearly I didn't | 20:48 |
fungi | but they can be pushed at the same time and approved in very rapid succession | 20:48 |
dhellmann | corvus : ack, thanks | 20:48 |
corvus | yeah, it's still a lot faster than before | 20:48 |
corvus | the second patch will continue to fail with the same error, but shortly after the first one lands, it can be rechecked and it'll turn green. | 20:48 |
openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: import the git-nit repository https://review.openstack.org/564622 | 20:48 |
openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add test jobs for openstack/git-nit repository https://review.openstack.org/564625 | 20:48 |
corvus | that looks good | 20:49 |
fungi | clarkb: cmurphy: so... launch-node.py --environment fails to find private hiera keys. this is not (as far as i know) a regression in the most recent refactor as i ran into and worked around this previously, but as a heads up it does seem to be broken (still) | 20:49 |
fungi | maybe i can work out why from the output | 20:50 |
*** iranzo has joined #openstack-infra | 20:50 | |
dhellmann | corvus : thanks | 20:50 |
*** eernst has joined #openstack-infra | 20:51 | |
dhellmann | ah, I do see the "submitting infra change for review" step in there now | 20:52 |
corvus | clarkb: cmurphy's change made me realize we may not be running the version of zuul we expected | 20:53 |
corvus | i think we may be failing installs right now because of that | 20:53 |
fungi | clarkb: cmurphy: it created /etc/puppet/hieradata/fungi/common.yaml on the launched instance, but did not put anything in .../group/ or .../fqdn/ | 20:53 |
*** trown is now known as trown|outtypewww | 20:56 | |
*** sree has joined #openstack-infra | 20:58 | |
mgagne | pabelanger: working on inap-mtl01 to fixed instances stuck in deleting state | 20:59 |
clarkb | corvus: oh | 20:59 |
clarkb | corvus: zuul web should report it too right? | 20:59 |
corvus | clarkb: it'll tell us, but you have to count commits; we should see if we can get the git sha in there | 20:59 |
*** pcichy has quit IRC | 20:59 | |
fungi | in theory, just hooking into the version variables pbr provides should do that for you? | 21:03 |
*** sree has quit IRC | 21:03 | |
fungi | pbr.version.VersionInfo('zuul') | 21:03 |
fungi | i think? | 21:03 |
clarkb | corvus: thinkign about timeline we restarted zuul before we did the puppet changes | 21:04 |
clarkb | so I think nodepool would be the one affected by those changes | 21:04 |
clarkb | at least relatively to the most recent zuul restart | 21:04 |
corvus | self.zuul_version = zuul_version.version_info.release_string() | 21:05 |
clarkb | corvus: pbr freeze seems to think we've installed 3.0.2 | 21:05 |
clarkb | zuul==3.0.2.dev37 # git sha d4e2d97 | 21:05 |
clarkb | on ze01 | 21:05 |
corvus | clarkb: that's pre 3.0.2 | 21:05 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-kibana master: Use instance variables in puppet template https://review.openstack.org/564609 | 21:05 |
clarkb | oh | 21:05 |
*** witek has quit IRC | 21:05 | |
* clarkb compares to git log | 21:06 | |
corvus | clarkb: the change that broke zuul installation is when we added the re2 dependency | 21:06 |
clarkb | ah | 21:06 |
corvus | the release_string ^ thing is what we do now | 21:06 |
*** yamamoto has joined #openstack-infra | 21:07 | |
corvus | and that gets us '3.0.2.dev37' but not the sha | 21:08 |
*** witek has joined #openstack-infra | 21:08 | |
*** dklyle has quit IRC | 21:08 | |
*** david-lyle has joined #openstack-infra | 21:08 | |
corvus | mordred: do you know off the top of your head how we can ask pbr for either "3.0.2" if it's exactly a release, but something that includes the git sha if it isn't? | 21:09 |
corvus | it doesn't have to be one function call, we can munge two things together here | 21:10 |
fungi | worst case, might be able to check whether the release string includes any characters in [a-z] and if so, append the abbreviated commit ref | 21:11 |
fungi | though i suppose we really only care if it includes the 'dev' substring in this case | 21:11 |
fungi | as any other letters would denote prerelease tags | 21:12 |
corvus | yeah, but i don't know how to get the sha | 21:12 |
clarkb | pbr freeze does it, we should start there. Looking | 21:13 |
corvus | zuul.version.version_info.semantic_version().version_tuple() | 21:13 |
corvus | (3, 0, 2, 'dev', 21) | 21:13 |
corvus | that's helpful | 21:13 |
*** yamamoto has quit IRC | 21:13 | |
corvus | (the last # there is the dev count; my checkout currently reports "dev22") | 21:13 |
clarkb | pkg_resources.get_distribution(package_name).get_metadata('pbr.json')) has a git_version attribute | 21:14 |
corvus | pkg_resources.get_distribution('zuul').get_metadata('pbr.json') | 21:14 |
corvus | '{"git_version": "796329d", "is_release": false}' | 21:14 |
clarkb | pbr.json isn't guaranteed to be there if pbr is older (at this point should be very uncommon) | 21:15 |
*** ldnunes has quit IRC | 21:15 | |
clarkb | so we could probably do a if its there use it else print NA | 21:15 |
fungi | maybe that's a better solution anyway as it stops treating pbr as a runtime dependency? | 21:15 |
*** Goneri has quit IRC | 21:16 | |
mordred | corvus: I'm not sure I understand the questions? | 21:16 |
mordred | corvus: ah - I've read more scrollback | 21:17 |
fungi | mordred: wanting a way in zuul-dashboard to display the git sha of the running zuul if it's not a tagged release (e.g., continuous deployment) | 21:17 |
corvus | i think i'm about ready to write a patch, why don't i do that real quick | 21:17 |
mordred | ++ | 21:17 |
fungi | though... hooking into pkg_resources like that is going to report the version details of the currently installed zuul, which may not be the currently running zuul, right? i suppose if we query that at start and keep it stashed somewhere in memory it gets around that issue. just don't call that method after starting | 21:18 |
clarkb | fungi: oh right good point | 21:19 |
clarkb | fungi: ya I think that is what we should do, load the info on process start, its still a minor race there but liveable | 21:19 |
openstackgerrit | Adam Harwell proposed openstack-infra/devstack-gate master: Align timing guidance to reality https://review.openstack.org/430393 | 21:20 |
clarkb | double checking nodepool now and it appears to be fine | 21:24 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Report git sha in status page version https://review.openstack.org/564628 | 21:24 |
clarkb | corvus: as for fixing our installations any work done on that yet? | 21:24 |
clarkb | fungi: for debugging that it would probably help to hold the node and see what the hiera setup looks like on the remote | 21:25 |
corvus | clarkb, fungi, mordred: ^ how's that look? i'm happy to change the remaining pbr thing to pkg_resources too if that's better, but could be a followup. | 21:25 |
corvus | clarkb: cmurphy fixes it here: https://review.openstack.org/564621 | 21:25 |
clarkb | cool will review both changes | 21:25 |
cmurphy | the beaker tests caught it | 21:26 |
fungi | corvus: yeah, i suppose Scheduler.init() only gets called at daemon start, so the race i was concerned about isn't an issue | 21:26 |
corvus | fungi: yep. and beyond that, even the first import should fix those things. | 21:27 |
clarkb | cmurphy: corvus I left a -1 though beaker tests catching it makes me second guess my -1 | 21:27 |
clarkb | I guess wedon't have a +1 from zuul yet so maybe I'm right still :) | 21:27 |
cmurphy | oh woops | 21:28 |
corvus | clarkb: oh yeah i think you may be right | 21:28 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-zuul master: Install libre2-dev package https://review.openstack.org/564621 | 21:29 |
fungi | corvus: mordred: i left a comment on the version change... i notice just playing around that pkg_resources will raise IOError if pbr.json is not available. is it possible this might happen in some (say linux distro oriented) packaging circumstances we need to care about? | 21:31 |
fungi | or should we just expect that a pbr.json will always be available to properly-packaged zuul deployments? | 21:32 |
clarkb | corvus: fungi and I left similar comments. I think my suggested solution will address it | 21:33 |
clarkb | fungi: no I don't think we can rely on it the more I think about it. Particularly if packagers aren't including pbr.json | 21:33 |
clarkb | fungi: because maybe they patched pbr out like some packagers do :( | 21:33 |
fungi | clarkb: wellm if they completely patched pbr out then the existing import pbr.version would explode anyway | 21:35 |
clarkb | oh good point | 21:35 |
fungi | i was thinking more about a future state where we replace the pbr runtime dependency with all pkg_resources | 21:35 |
clarkb | similar issue in that case | 21:36 |
fungi | but i suppose a try/except can be added at that point | 21:36 |
fungi | and we can just consider that a pbr runtime dep implies a pbr.json present in the package metadata | 21:36 |
fungi | for now | 21:36 |
* clarkb reads some setuptools source | 21:37 | |
corvus | yeah, let's invert the logic there so we check "is False" so we know that we got pbr.json. and then add a try/except in version.py. | 21:38 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Report git sha in status page version https://review.openstack.org/564628 | 21:39 |
corvus | clarkb, fungi, mordred: ^ i think that should take care of it? | 21:39 |
*** dhill_ has quit IRC | 21:39 | |
clarkb | corvus: the default value for is_release is None but then you check is False. I think you want if not is_release instead | 21:40 |
clarkb | otherwise yup lgtm | 21:40 |
corvus | clarkb: the "is False" means "we do have pbr.json and it's not a release" | 21:41 |
clarkb | oh right | 21:41 |
clarkb | because we inverted | 21:41 |
corvus | right | 21:41 |
clarkb | shipit | 21:41 |
*** hashar has quit IRC | 21:44 | |
rm_work | hey so, I want to go from "CR link" to "cherry-pick the latest patchset" in code | 21:44 |
rm_work | is there a straightforward way to calculate or otherwise ask gerrit how to get the latest PS link? | 21:45 |
fungi | rm_work: you have to query it to figure out the patchset sequence number | 21:45 |
rm_work | so if I have like, https://review.openstack.org/#/c/558962/ | 21:45 |
rm_work | I can browse to it, and go to the Download -> cherrypick thing | 21:45 |
fungi | there is no special "most recent patchset" ref | 21:45 |
rm_work | so is there an API? | 21:46 |
rm_work | I want to do it in a script | 21:46 |
*** jamesmcarthur has quit IRC | 21:46 | |
rm_work | have a list of CR links, and have my script do cherry-picks for them | 21:46 |
fungi | rm_work: there is a rest api, though `git review -x 558962` does what you're talking about too | 21:46 |
rm_work | yeah, avoiding git-review because i may not have that | 21:46 |
rm_work | so gerrit has a rest-api i can query for that info, k | 21:47 |
* rm_work looks for docs | 21:47 | |
* rm_work found them | 21:47 | |
fungi | https://review.openstack.org/Documentation/rest-api.html | 21:47 |
*** dhill_ has joined #openstack-infra | 21:47 | |
fungi | yeah, documentation->rest api at the top of the webui | 21:47 |
fungi | will get you to the version relevant to the deployed system | 21:48 |
*** boden has quit IRC | 21:49 | |
*** agopi|zzz has quit IRC | 21:52 | |
rm_work | hmmm most of these use change-id | 21:52 |
rm_work | i was hoping to get there from the CR# the way git-review does | 21:52 |
rm_work | maybe i can pick apart git-review | 21:52 |
rm_work | ahh i guess the git-review command doesn't rely on all of the normal git-review setup, so maybe it's ok for me | 21:58 |
rm_work | thanks for the tip! | 21:58 |
dhellmann | corvus : I don't understand what "not normalized" means for my change to import git-nit. | 21:59 |
*** slaweq has quit IRC | 22:00 | |
dhellmann | is it complaining about a missing newline at the end of the file? | 22:00 |
*** dpawlik has joined #openstack-infra | 22:00 | |
dhellmann | let's see if that's it | 22:01 |
openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: import the git-nit repository https://review.openstack.org/564622 | 22:01 |
openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add test jobs for openstack/git-nit repository https://review.openstack.org/564625 | 22:01 |
*** dave-mcc_ has quit IRC | 22:02 | |
*** VW has joined #openstack-infra | 22:04 | |
corvus | dhellmann: yeah, i think that's what that one is | 22:05 |
pabelanger | clarkb: did we ever land the change to collect more debug info when we hit v4 v6 route failure to git.o.o? I am hitting it pretty often on fedora-27 | 22:05 |
*** dpawlik has quit IRC | 22:05 | |
pabelanger | I suspect we don't have something configured properly | 22:05 |
clarkb | pabelanger: no, because corvus want it to be tested which required changes to stuff | 22:05 |
clarkb | pabelanger: I think at this point it is tested if we want to merge it before the changes to update testing | 22:05 |
pabelanger | k | 22:09 |
*** VW has quit IRC | 22:09 | |
*** yamamoto has joined #openstack-infra | 22:09 | |
*** ssbarnea_ has quit IRC | 22:11 | |
*** yamamoto has quit IRC | 22:14 | |
*** rlandy is now known as rlandy|afk | 22:19 | |
*** rkukura has joined #openstack-infra | 22:20 | |
*** rcernin has joined #openstack-infra | 22:23 | |
*** tpsilva has quit IRC | 22:26 | |
*** felipemonteiro__ has quit IRC | 22:28 | |
*** agopi|zzz has joined #openstack-infra | 22:31 | |
*** rkukura has quit IRC | 22:31 | |
*** rkukura has joined #openstack-infra | 22:32 | |
notmyname | lifeless: any suggestions for how to run bindep on alpine linux? it doesn't have `lsb_release` available | 22:33 |
*** david-lyle is now known as dklyle | 22:33 | |
timburke | notmyname: something something /etc/os-release ? | 22:36 |
clarkb | ya we probably want to check if /etc/os-release exists and if so use that else try lsb_release | 22:37 |
notmyname | timburke: I think "something something" may be spelled "patch/rewrite bindep". but *surely* someone's already got this working | 22:37 |
notmyname | yeah, alpine has /etc/os-release | 22:37 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix race in test_bubblewrap_leak https://review.openstack.org/564640 | 22:38 |
clarkb | ID and VERSION_ID appear to be the two fields that bindep would be most interested in | 22:39 |
clarkb | and VERSION_CODENAME? | 22:39 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Remove source_context argument to Pipeline https://review.openstack.org/564642 | 22:42 |
*** VW has joined #openstack-infra | 22:42 | |
*** wolverineav has quit IRC | 22:42 | |
*** wolverineav has joined #openstack-infra | 22:43 | |
corvus | http://git.openstack.org/cgit/openstack-infra/bindep/tree/bindep/depends.py#n278 | 22:43 |
*** agopi|zzz has quit IRC | 22:44 | |
*** rpioso is now known as rpioso|afk | 22:44 | |
corvus | you know, someone should standardize how to to describe and report this sort of information. | 22:44 |
*** salv-orl_ has quit IRC | 22:45 | |
clarkb | on ubuntu lsb_release -cirs and os-release agree on everything but character case | 22:45 |
*** salv-orlando has joined #openstack-infra | 22:45 | |
clarkb | on opensuse tumbleweed they don't agree on id | 22:45 |
clarkb | we'll have to account for that but should be doable | 22:46 |
*** VW has quit IRC | 22:46 | |
clarkb | centos doesn't agree on version id | 22:47 |
clarkb | 7.4.1708 via lsb vs 7 via lsb_release | 22:47 |
*** hongbin_ has quit IRC | 22:47 | |
*** wolverineav has quit IRC | 22:47 | |
clarkb | er os-release is just 7 | 22:48 |
pabelanger | for glean, we used the distro library for python (vendored actually): https://review.openstack.org/545314/ could do the same thing in bindep | 22:48 |
clarkb | looks like it does what I suggested, checks os-release first if that isn't available uses lsb_release | 22:50 |
clarkb | so ya maybe just use that to avoid reimplementing that | 22:50 |
*** salv-orlando has quit IRC | 22:50 | |
clarkb | (it also checks uname for bsd support) | 22:50 |
*** esberglu has quit IRC | 22:54 | |
openstackgerrit | Merged openstack-infra/puppet-zuul master: Install libre2-dev package https://review.openstack.org/564621 | 22:57 |
fungi | given bindep already has a dependency (parsley) adding one more probably won't hurt so long as it's pure pythn | 23:00 |
*** iranzo has quit IRC | 23:01 | |
fungi | we could also look at dropping bindep's runtime pbr dep | 23:01 |
openstackgerrit | Merged openstack-infra/system-config master: Remove debian-jessie from reprepro https://review.openstack.org/564263 | 23:01 |
fungi | not that it has anything to do with this situation, just noticing we list it in requirements.txt and really don't need it for anything important | 23:01 |
*** VW has joined #openstack-infra | 23:06 | |
*** bobh has quit IRC | 23:09 | |
*** camunoz has quit IRC | 23:10 | |
*** VW has quit IRC | 23:10 | |
*** yamamoto has joined #openstack-infra | 23:11 | |
*** camunoz has joined #openstack-infra | 23:13 | |
*** annp has quit IRC | 23:13 | |
*** annp has joined #openstack-infra | 23:14 | |
*** dpawlik has joined #openstack-infra | 23:14 | |
*** wolverineav has joined #openstack-infra | 23:15 | |
*** yamamoto has quit IRC | 23:16 | |
*** jmccrory has quit IRC | 23:17 | |
*** jmccrory has joined #openstack-infra | 23:18 | |
*** yamahata has joined #openstack-infra | 23:19 | |
*** dpawlik has quit IRC | 23:19 | |
*** wolverineav has quit IRC | 23:20 | |
*** wolverineav has joined #openstack-infra | 23:25 | |
*** camunoz has quit IRC | 23:27 | |
*** dingyichen has joined #openstack-infra | 23:30 | |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Simplify UnparsedConfig.copy https://review.openstack.org/564647 | 23:31 |
*** sree has joined #openstack-infra | 23:36 | |
*** claudiub has quit IRC | 23:39 | |
*** ihar has quit IRC | 23:40 | |
*** sree has quit IRC | 23:40 | |
*** wolverineav has quit IRC | 23:41 | |
*** salv-orlando has joined #openstack-infra | 23:46 | |
*** rlandy|afk is now known as rlandy | 23:48 | |
*** caphrim007_ has joined #openstack-infra | 23:48 | |
*** caphrim007 has quit IRC | 23:49 | |
*** tosky has quit IRC | 23:49 | |
*** caphrim007_ has quit IRC | 23:51 | |
*** eernst has quit IRC | 23:51 | |
*** salv-orlando has quit IRC | 23:51 | |
openstackgerrit | Anita Kuno proposed openstack-infra/system-config master: This patch impliments the survey spec: I3c389596373b94459a32a4e540d514a2941acbb1 https://review.openstack.org/557979 | 23:51 |
*** xarses has joined #openstack-infra | 23:53 | |
openstackgerrit | Clark Boylan proposed openstack-infra/bindep master: Use the distro package to determine distro https://review.openstack.org/564650 | 23:53 |
clarkb | pabelanger: fungi notmyname ^ that won't pass tests as is because the entire test framework needs to be updated | 23:54 |
openstackgerrit | Anita Kuno proposed openstack-infra/system-config master: This patch impliments the survey spec: I3c389596373b94459a32a4e540d514a2941acbb1 https://review.openstack.org/557979 | 23:54 |
clarkb | but it works for me on my local opensuse tumbleweed machine | 23:54 |
clarkb | if someone wants to work on porting over some of the test machinery from distro itself so that we can override the os-release info that would be excellent | 23:54 |
*** gyee has quit IRC | 23:54 | |
*** jamesmcarthur has joined #openstack-infra | 23:56 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!