| *** yamamoto has quit IRC | 00:01 | |
| *** EricGonczer_ has joined #openstack-infra | 00:03 | |
| *** yamamoto has joined #openstack-infra | 00:05 | |
| *** dingyichen has joined #openstack-infra | 00:08 | |
| *** xyang1 has quit IRC | 00:11 | |
| *** ramishra has joined #openstack-infra | 00:14 | |
| *** gmann has quit IRC | 00:16 | |
| *** gmann has joined #openstack-infra | 00:16 | |
| *** gordc has joined #openstack-infra | 00:24 | |
| *** EricGonczer_ has quit IRC | 00:27 | |
| *** EricGonczer_ has joined #openstack-infra | 00:28 | |
| *** ramishra has quit IRC | 00:28 | |
| *** gordc has quit IRC | 00:29 | |
| *** aeng has quit IRC | 00:29 | |
| *** mriedem has quit IRC | 00:30 | |
| *** EricGonczer_ has quit IRC | 00:33 | |
| *** armax has quit IRC | 00:34 | |
| *** slaweq has joined #openstack-infra | 00:36 | |
| *** mriedem has joined #openstack-infra | 00:38 | |
| *** harlowja has quit IRC | 00:38 | |
| *** gordc has joined #openstack-infra | 00:41 | |
| *** slaweq has quit IRC | 00:42 | |
| *** zhurong has joined #openstack-infra | 00:43 | |
| *** thorst_afk has joined #openstack-infra | 00:46 | |
| *** thorst_afk has quit IRC | 00:54 | |
| *** https_GK1wmSU has joined #openstack-infra | 00:55 | |
| *** https_GK1wmSU has quit IRC | 00:56 | |
| *** https_GK1wmSU has joined #openstack-infra | 01:00 | |
| *** https___GK1wmSU has joined #openstack-infra | 01:02 | |
| *** https__GK1wmSU has joined #openstack-infra | 01:02 | |
| *** https___GK1wmSU has quit IRC | 01:03 | |
| *** https__GK1wmSU has quit IRC | 01:03 | |
| *** https_GK1wmSU has quit IRC | 01:03 | |
| *** dougwig has quit IRC | 01:05 | |
| *** rkukura has quit IRC | 01:06 | |
| *** s-shiono has joined #openstack-infra | 01:07 | |
| *** liujiong has joined #openstack-infra | 01:08 | |
| *** sdague has quit IRC | 01:08 | |
| *** EricGonczer_ has joined #openstack-infra | 01:09 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move selinux packages to redhat-common https://review.openstack.org/489434 | 01:10 |
|---|---|---|
| *** pahuang has quit IRC | 01:18 | |
| *** ramishra has joined #openstack-infra | 01:20 | |
| *** yamamoto has quit IRC | 01:25 | |
| *** thorst_afk has joined #openstack-infra | 01:29 | |
| *** rkukura has joined #openstack-infra | 01:29 | |
| *** yamamoto has joined #openstack-infra | 01:29 | |
| *** rkukura has quit IRC | 01:30 | |
| *** gcb has joined #openstack-infra | 01:30 | |
| *** EricGonczer_ has quit IRC | 01:31 | |
| *** stakeda has joined #openstack-infra | 01:33 | |
| *** rkukura has joined #openstack-infra | 01:34 | |
| *** pahuang has joined #openstack-infra | 01:35 | |
| *** gcb has quit IRC | 01:37 | |
| *** thorst_afk has quit IRC | 01:38 | |
| *** thorst_afk has joined #openstack-infra | 01:38 | |
| *** gcb has joined #openstack-infra | 01:39 | |
| *** xinliang has joined #openstack-infra | 01:40 | |
| *** xinliang has quit IRC | 01:40 | |
| *** xinliang has joined #openstack-infra | 01:40 | |
| *** https_GK1wmSU has joined #openstack-infra | 01:40 | |
| *** https_GK1wmSU has left #openstack-infra | 01:41 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 01:42 | |
| *** thorst_afk has quit IRC | 01:43 | |
| *** EricGonczer_ has joined #openstack-infra | 01:47 | |
| *** Apoorva has quit IRC | 01:48 | |
| *** gordc has quit IRC | 01:49 | |
| *** pahuang has quit IRC | 01:51 | |
| *** yamamoto has quit IRC | 01:51 | |
| *** tuanluong has joined #openstack-infra | 01:51 | |
| *** camunoz has quit IRC | 01:52 | |
| *** armax has joined #openstack-infra | 01:52 | |
| *** pahuang has joined #openstack-infra | 01:53 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: External Calendar Sync https://review.openstack.org/487683 | 01:54 |
| *** thorst_afk has joined #openstack-infra | 01:55 | |
| *** thorst_afk has quit IRC | 01:55 | |
| *** yamamoto has joined #openstack-infra | 01:56 | |
| *** rkukura has quit IRC | 01:59 | |
| *** EricGonczer_ has quit IRC | 01:59 | |
| *** slaweq has joined #openstack-infra | 01:59 | |
| *** slaweq has quit IRC | 02:04 | |
| *** yamamoto has quit IRC | 02:07 | |
| *** yamahata has quit IRC | 02:08 | |
| openstackgerrit | Kien Nguyen proposed openstack-infra/project-config master: Add topology flag to Heat job template https://review.openstack.org/482048 | 02:08 |
| *** EricGonczer_ has joined #openstack-infra | 02:09 | |
| *** slaweq has joined #openstack-infra | 02:09 | |
| *** mriedem has quit IRC | 02:10 | |
| *** thorst_afk has joined #openstack-infra | 02:11 | |
| *** slaweq has quit IRC | 02:15 | |
| *** rossella_s has quit IRC | 02:19 | |
| *** yamamoto has joined #openstack-infra | 02:19 | |
| *** rossella_s has joined #openstack-infra | 02:21 | |
| *** hongbin has joined #openstack-infra | 02:21 | |
| *** harlowja has joined #openstack-infra | 02:22 | |
| *** thorst_afk has quit IRC | 02:25 | |
| *** bobh has joined #openstack-infra | 02:26 | |
| *** EricGonczer_ has quit IRC | 02:33 | |
| *** esberglu has joined #openstack-infra | 02:39 | |
| *** panda has quit IRC | 02:44 | |
| *** harlowja has quit IRC | 02:44 | |
| *** esberglu has quit IRC | 02:44 | |
| *** panda has joined #openstack-infra | 02:47 | |
| *** esberglu has joined #openstack-infra | 02:49 | |
| *** liujiong has quit IRC | 02:50 | |
| *** baoli_ has quit IRC | 02:50 | |
| *** liujiong has joined #openstack-infra | 02:51 | |
| *** esikachev has joined #openstack-infra | 02:54 | |
| *** raddaoui has quit IRC | 02:54 | |
| *** hongbin_ has joined #openstack-infra | 02:55 | |
| *** hongbin has quit IRC | 02:57 | |
| *** esikachev has quit IRC | 02:58 | |
| *** hongbin has joined #openstack-infra | 02:59 | |
| *** aeng has joined #openstack-infra | 03:00 | |
| *** hongbin_ has quit IRC | 03:00 | |
| *** yamamoto has quit IRC | 03:01 | |
| *** yamamoto has joined #openstack-infra | 03:06 | |
| *** lbragstad_ has joined #openstack-infra | 03:14 | |
| *** lbragstad has quit IRC | 03:14 | |
| *** yamamoto has quit IRC | 03:21 | |
| *** bobh has quit IRC | 03:23 | |
| *** sree_ has joined #openstack-infra | 03:23 | |
| *** sree_ is now known as Guest35159 | 03:23 | |
| *** yamamoto has joined #openstack-infra | 03:24 | |
| *** gouthamr has quit IRC | 03:28 | |
| *** yamamoto has quit IRC | 03:36 | |
| *** marst has joined #openstack-infra | 03:36 | |
| *** slaweq has joined #openstack-infra | 03:37 | |
| *** harlowja has joined #openstack-infra | 03:38 | |
| *** slaweq has quit IRC | 03:42 | |
| *** yamamoto has joined #openstack-infra | 03:44 | |
| *** nicolasbock has joined #openstack-infra | 03:45 | |
| *** dave-mcc_ has quit IRC | 03:48 | |
| *** ykarel has joined #openstack-infra | 03:52 | |
| *** ijw has quit IRC | 03:57 | |
| EmilienM | fungi: hey, I'm wondering if https://review.openstack.org/#/c/487187/ has been applied on logservers | 03:58 |
| EmilienM | I can't recall the url to find out puppet dashboar | 03:59 |
| clarkb | EmilienM: it should be applied since it is merged | 04:02 |
| clarkb | all puppet updates should be applied within an hour of merging | 04:03 |
| EmilienM | clarkb: ok, I'm wondering why we still don't have index page | 04:03 |
| *** mtanino has quit IRC | 04:04 | |
| *** yamamoto has quit IRC | 04:04 | |
| *** hongbin has quit IRC | 04:05 | |
| EmilienM | clarkb: not sure how to debug it, tbh. If you have an idea | 04:06 |
| *** links has joined #openstack-infra | 04:07 | |
| *** yamamoto has joined #openstack-infra | 04:09 | |
| clarkb | EmilienM: do you have an example job log it would apply to? | 04:10 |
| *** psachin has joined #openstack-infra | 04:10 | |
| EmilienM | clarkb: http://logs.openstack.org/42/487742/1/gate/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-puppet/ed7e7cd/ | 04:10 |
| clarkb | EmilienM: http://logs.openstack.org/28/489428/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/79a12aa/logs/ the tempest one is no longer applied either | 04:14 |
| clarkb | EmilienM: Jul 31 19:56:27 static puppet-user[8814]: (/Stage[main]/Httpd/Service[httpd]) ReadmeName takes one argument, a filename <- is from syslog | 04:15 |
| clarkb | but I don't see ReadmeName anywhere in /etc/apache2/sites-enabled | 04:16 |
| EmilienM | oops | 04:17 |
| clarkb | I think the changes to iterate over the readme vars must have a bug | 04:18 |
| EmilienM | yeah i'm digging the infra puppet apache | 04:18 |
| EmilienM | clarkb: you're using https://github.com/openstack-infra/puppet-httpd right? | 04:18 |
| clarkb | yes I think so | 04:18 |
| clarkb | you may have to do a global lookup of readmes for that to work | 04:19 |
| clarkb | because the erb is being evaluated from puppet-httpd not openstackci | 04:19 |
| clarkb | ::openstackci::readmes | 04:19 |
| EmilienM | right | 04:20 |
| EmilienM | clarkb: neither in /etc/apache2/sites-available? | 04:22 |
| clarkb | EmilienM: no, we don't put the file in sites available (since its controlled by puppet we just add and remove to sites-enabled directly) | 04:23 |
| EmilienM | ah I think I found | 04:23 |
| EmilienM | https://github.com/openstack-infra/puppet-openstackci/blob/fdc4e9cc045a79d8542023a08527c4ea2d97eac7/templates/logs.vhost.erb#L71 | 04:23 |
| EmilienM | <Directory <%= @docroot %><%= pattern %>> | 04:23 |
| EmilienM | I think it should be @pattern | 04:23 |
| clarkb | I don't think so because in that context you are using the |pattern| iteration variable | 04:24 |
| clarkb | right? I thought ruby didn't require you to prefix local vars | 04:24 |
| EmilienM | ah right it's a local var | 04:25 |
| EmilienM | clarkb: can you pastbine the full catalog log please? | 04:25 |
| EmilienM | clarkb: and maybe some apache logs, I want to debug why puppet doesn't apply it | 04:25 |
| EmilienM | clarkb: also, did Puppet failed to apply the catalog or passed but the resource didnd't work? | 04:26 |
| *** thorst_afk has joined #openstack-infra | 04:26 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add buildlogs.centos.org to reverse proxy cache https://review.openstack.org/489462 | 04:27 |
| clarkb | EmilienM: http://paste.openstack.org/show/617083/ is the last puppet apply run. I think puppet succeeded it just couldn't properly restart apache | 04:27 |
| EmilienM | clarkb: when you restart apache, what does it do? | 04:28 |
| EmilienM | wait | 04:29 |
| EmilienM | clarkb: can you paste /etc/apache2/sites-enabled/50-logs.openstack.org.conf please? | 04:29 |
| clarkb | EmilienM: ya sites-enabled doesn't have that not sure if puppet-httpd is doing its own check before applying or something | 04:29 |
| clarkb | so I don't know what undid it | 04:30 |
| *** thorst_afk has quit IRC | 04:31 | |
| EmilienM | clarkb: ^ can you pastbin plz? | 04:31 |
| *** adisky__ has joined #openstack-infra | 04:33 | |
| clarkb | EmilienM: http://paste.openstack.org/show/617084/ is the relevant bits | 04:33 |
| clarkb | EmilienM: 13f3e35d98ee04b21173c138b9db3c54e5276f03 is the version of puppet-openstackci installed on the puppetmaster which is copied to the individual node before running puppet apply | 04:36 |
| clarkb | 2017-07-31 20:13:15,311 p=14431 u=root | static.openstack.org : ok=7 changed=2 unreachable=0 failed=1 is the last time ansible logs running puppet on that server | 04:37 |
| clarkb | but ansible has run more recently than that | 04:37 |
| clarkb | looks like someone disabled puppet on static.o.o | 04:38 |
| clarkb | and must've manually cleaned up that file | 04:38 |
| *** treaki__ has quit IRC | 04:38 | |
| clarkb | EmilienM: 20:10:06 fungi | EmilienM: looks like we have a bug... the erb change in https://review.openstack.org/487120 resulted in readmefile entries like this: http://paste.openstack.org/show/617059 | 04:38 |
| *** treaki_ has joined #openstack-infra | 04:38 | |
| EmilienM | indeed | 04:39 |
| clarkb | .each seems to give you the key value pairs | 04:40 |
| clarkb | do you need |pattern, helpfile| instead? also you might want to update your default value for readmes to {} instead of [] to better reflect the type | 04:40 |
| *** jhesketh__ has joined #openstack-infra | 04:44 | |
| *** jhesketh__ has quit IRC | 04:44 | |
| EmilienM | clarkb: yes it sounds like, I'm looking into it | 04:44 |
| *** jhesketh_ has joined #openstack-infra | 04:44 | |
| *** jhesketh_ has quit IRC | 04:45 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/puppet-openstackci master: Iterate readmes as an hash https://review.openstack.org/489466 | 04:53 |
| EmilienM | clarkb: ^ can you run a puppet apply with ^ ? please | 04:54 |
| clarkb | EmilienM: I'm not going to run tonight, its bed time :) | 04:54 |
| clarkb | I'll give it a quick review though | 04:54 |
| EmilienM | clarkb: ok thx | 04:54 |
| EmilienM | good night! & thx for your help | 04:55 |
| clarkb | EmilienM: and done | 04:56 |
| *** slaweq has joined #openstack-infra | 04:59 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/puppet-openstackci master: Iterate readmes as an hash https://review.openstack.org/489466 | 04:59 |
| EmilienM | clarkb: done, adding the @ you missed in comment | 04:59 |
| *** rkukura has joined #openstack-infra | 05:02 | |
| *** slaweq has quit IRC | 05:04 | |
| *** prateek has joined #openstack-infra | 05:06 | |
| *** harlowja has quit IRC | 05:10 | |
| *** https__GK1wmSU has joined #openstack-infra | 05:10 | |
| *** https_GK1wmSU has joined #openstack-infra | 05:11 | |
| *** https_GK1wmSU has quit IRC | 05:12 | |
| *** https__GK1wmSU has quit IRC | 05:12 | |
| *** https___GK1wmSU has joined #openstack-infra | 05:15 | |
| *** https__GK1wmSU has joined #openstack-infra | 05:15 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Fix test_cache_hard_links when on tmpfs https://review.openstack.org/489377 | 05:16 |
| *** https___GK1wmSU has left #openstack-infra | 05:16 | |
| *** https__GK1wmSU has left #openstack-infra | 05:16 | |
| *** jtomasek has joined #openstack-infra | 05:20 | |
| *** esikachev has joined #openstack-infra | 05:21 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] larger LVM testing https://review.openstack.org/486864 | 05:25 |
| *** gcb has quit IRC | 05:29 | |
| *** gcb_ has joined #openstack-infra | 05:29 | |
| *** https_GK1wmSU has joined #openstack-infra | 05:41 | |
| *** https___GK1wmSU has joined #openstack-infra | 05:43 | |
| *** https___GK1wmSU has quit IRC | 05:44 | |
| *** https_GK1wmSU has quit IRC | 05:45 | |
| *** bhavik1 has joined #openstack-infra | 05:48 | |
| *** rlandy has joined #openstack-infra | 05:49 | |
| *** rlandy is now known as rlandy|ci_mtgs | 05:50 | |
| *** aviau has quit IRC | 05:50 | |
| *** aviau has joined #openstack-infra | 05:51 | |
| *** harlowja has joined #openstack-infra | 05:51 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Improve cleanup of test disk accountant https://review.openstack.org/489477 | 05:52 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Add max-nodes-per-job tenant setting https://review.openstack.org/489481 | 05:56 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Add max-nodes-per-job tenant setting https://review.openstack.org/489481 | 05:58 |
| *** yamahata has joined #openstack-infra | 06:01 | |
| *** cshastri has joined #openstack-infra | 06:04 | |
| *** jaosorior has quit IRC | 06:05 | |
| *** esberglu has quit IRC | 06:06 | |
| *** esberglu has joined #openstack-infra | 06:06 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Don't ignore inexistent jobs in config https://review.openstack.org/488758 | 06:11 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Don't ignore inexistent jobs in config https://review.openstack.org/488758 | 06:12 |
| *** slaweq has joined #openstack-infra | 06:13 | |
| *** esberglu has quit IRC | 06:15 | |
| *** wolverineav has joined #openstack-infra | 06:18 | |
| *** wolverineav has quit IRC | 06:22 | |
| *** jaosorior has joined #openstack-infra | 06:24 | |
| *** https_GK1wmSU has joined #openstack-infra | 06:27 | |
| *** bhavik1 has quit IRC | 06:27 | |
| *** thorst_afk has joined #openstack-infra | 06:28 | |
| *** https_GK1wmSU has quit IRC | 06:30 | |
| *** kjackal_ has quit IRC | 06:30 | |
| *** thorst_afk has quit IRC | 06:32 | |
| *** https_GK1wmSU has joined #openstack-infra | 06:34 | |
| AJaeger | ianw: you have a few older open changes in project-config - will you continue on these and push them forward? https://review.openstack.org/#/q/owner:iwienand%2540redhat.com+status:open+project:openstack-infra/project-config | 06:35 |
| *** https_GK1wmSU has quit IRC | 06:36 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Use zuul attributes in semaphore documentation https://review.openstack.org/489499 | 06:40 |
| *** panda has quit IRC | 06:43 | |
| *** panda has joined #openstack-infra | 06:47 | |
| *** harlowja has quit IRC | 06:51 | |
| *** ralonsoh has joined #openstack-infra | 06:52 | |
| *** rcernin has joined #openstack-infra | 06:55 | |
| *** alexchadin has joined #openstack-infra | 06:56 | |
| *** kjackal_ has joined #openstack-infra | 06:58 | |
| *** dtantsur|afk is now known as dtantsur | 07:01 | |
| *** pcaruana has joined #openstack-infra | 07:06 | |
| *** markus_z has joined #openstack-infra | 07:10 | |
| *** gibi_ is now known as gibi | 07:11 | |
| openstackgerrit | Alexander Chadin proposed openstack-infra/project-config master: Add gate-watcherclient-dsvm-functional-ubuntu-xenial to check and gate pipelines https://review.openstack.org/489504 | 07:12 |
| *** yolanda__ has joined #openstack-infra | 07:14 | |
| *** ccamacho has quit IRC | 07:14 | |
| *** ccamacho has joined #openstack-infra | 07:14 | |
| *** yolanda__ is now known as yolanda | 07:15 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Use zuul attributes in nodeset section https://review.openstack.org/489506 | 07:17 |
| *** vsaienk0 has joined #openstack-infra | 07:20 | |
| *** vsaienk0 has left #openstack-infra | 07:20 | |
| *** aarefiev_afk is now known as aarefiev | 07:28 | |
| *** dizquierdo has joined #openstack-infra | 07:28 | |
| *** dizquierdo has quit IRC | 07:28 | |
| *** dizquierdo has joined #openstack-infra | 07:29 | |
| ianw | AJaeger: the coverage ones i'd still like to have implemented, but it's stuck on a spec that hasn't got much discussion | 07:34 |
| yuval | AJaeger: Hey, remember the issue with the queued jobs from yesterday? I think it happens now as well. Looking at the test node graph at http://status.openstack.org/zuul/ , it seems like there is a pattern. Maybe some periodic jobs occuring every day? | 07:40 |
| *** slaweq_ has joined #openstack-infra | 07:41 | |
| *** rcarrillocruz has quit IRC | 07:45 | |
| *** rcarrillocruz has joined #openstack-infra | 07:45 | |
| *** slaweq_ has quit IRC | 07:47 | |
| *** dtantsur is now known as dtantsur|bbl | 07:47 | |
| *** rcernin has quit IRC | 07:48 | |
| AJaeger | ianw: so bring the spec up for the next team meeting (today?) ? | 07:48 |
| AJaeger | ianw: ^ | 07:48 |
| *** LindaWang has joined #openstack-infra | 07:48 | |
| AJaeger | ianw: can you check nodepool status, please? | 07:49 |
| *** rcernin has joined #openstack-infra | 07:50 | |
| AJaeger | infra-root, looks like we have *again* a stuck nodepool - see http://grafana.openstack.org/dashboard/db/nodepool | 07:50 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Add max-nodes-per-job tenant setting https://review.openstack.org/489481 | 07:54 |
| *** egonzalez has joined #openstack-infra | 07:56 | |
| AJaeger | ianw, jhesketh, yolanda : Anybody of you around to help, please? ^ | 07:59 |
| yolanda | hi AJaeger , same issue again? | 07:59 |
| yolanda | that's so weird, again stuck from 1:30 min | 08:00 |
| *** esikachev has quit IRC | 08:00 | |
| yolanda | latest activity in the launcher is from 1:30 min ago | 08:00 |
| yolanda | this time the process was not even running, i started it | 08:02 |
| *** LindaWang has quit IRC | 08:02 | |
| *** clarkb has quit IRC | 08:04 | |
| *** hashar has joined #openstack-infra | 08:06 | |
| *** ryotagami has quit IRC | 08:07 | |
| yolanda | ok, back to life | 08:08 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Refactor provider config to driver module https://review.openstack.org/488384 | 08:08 |
| yolanda | #status info Yolanda has started nodepool-launcher process because it was stopped for more than one hour | 08:08 |
| openstackstatus | yolanda: unknown command | 08:08 |
| yolanda | #status log Yolanda has started nodepool-launcher process because it was stopped for more than one hour | 08:09 |
| openstackstatus | yolanda: finished logging | 08:09 |
| *** LindaWang has joined #openstack-infra | 08:09 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Update feature branch creation instructions https://review.openstack.org/489356 | 08:10 |
| AJaeger | thanks, yolanda | 08:11 |
| yolanda | i don't know what's causing it, nothing strange in the logs, just lack of activity | 08:11 |
| openstackgerrit | Igor Litmanovich proposed openstack-infra/git-review master: Fix type error applying len() on int (change number) https://review.openstack.org/489271 | 08:12 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix test_cache_hard_links when on tmpfs https://review.openstack.org/489377 | 08:16 |
| yuval | AJaeger: just wondering, maybe there should be some monitoring on these kind of issues? Health check or something? | 08:16 |
| *** yolanda has quit IRC | 08:17 | |
| *** clarkb has joined #openstack-infra | 08:17 | |
| *** efoley has joined #openstack-infra | 08:17 | |
| *** efoley has quit IRC | 08:18 | |
| *** yolanda has joined #openstack-infra | 08:18 | |
| *** efoley has joined #openstack-infra | 08:18 | |
| openstackgerrit | Béla Vancsics proposed openstack-infra/shade master: Use more specific asserts in tests https://review.openstack.org/489524 | 08:18 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Add max-nodes-per-job tenant setting https://review.openstack.org/489481 | 08:19 |
| *** s-shiono has quit IRC | 08:19 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add ENABLED_SERVICES to Mogan job template https://review.openstack.org/489178 | 08:22 |
| *** jascott1 has quit IRC | 08:22 | |
| *** jascott1 has joined #openstack-infra | 08:23 | |
| openstackgerrit | Nisha Agarwal proposed openstack-infra/project-config master: Moves the ironic python2 experimental job as non-voting job https://review.openstack.org/489278 | 08:25 |
| *** lucas-afk is now known as lucasagomes | 08:26 | |
| *** gmann_ has joined #openstack-infra | 08:26 | |
| *** thorst_afk has joined #openstack-infra | 08:27 | |
| *** jascott1 has quit IRC | 08:27 | |
| *** markvoelker has quit IRC | 08:30 | |
| *** markvoelker has joined #openstack-infra | 08:31 | |
| AJaeger | yuval: Yes, best talk to the rest of the team - US morning time. | 08:31 |
| *** thorst_afk has quit IRC | 08:32 | |
| *** markvoelker has quit IRC | 08:35 | |
| *** pgadiya has joined #openstack-infra | 08:35 | |
| *** e0ne has joined #openstack-infra | 08:43 | |
| *** esikachev has joined #openstack-infra | 08:43 | |
| *** electrofelix has joined #openstack-infra | 08:44 | |
| *** priteau has joined #openstack-infra | 08:48 | |
| *** Guest35159 has quit IRC | 08:50 | |
| *** sree_ has joined #openstack-infra | 08:51 | |
| *** sree_ is now known as Guest35970 | 08:51 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/489533 | 08:51 |
| *** dingyichen has quit IRC | 08:53 | |
| *** zeih has joined #openstack-infra | 08:53 | |
| *** alexchadin has quit IRC | 08:53 | |
| ianw | yolanda saves the day again! thanks | 08:55 |
| *** Guest35970 has quit IRC | 08:55 | |
| *** alexchadin has joined #openstack-infra | 08:56 | |
| *** martinkopec has joined #openstack-infra | 09:02 | |
| *** kjackal_ has quit IRC | 09:03 | |
| esikachev | hey guys. do you have an unit-file for zuul? (ubuntu 16.04) | 09:03 |
| AJaeger | esikachev: clarkb answered this last night - did you see it in scrollback? You can read in eavesdrop.openstack.org as well. | 09:06 |
| esikachev | sorry, I was disconnected this time | 09:06 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a static driver for Nodepool https://review.openstack.org/468624 | 09:07 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement an OpenContainer driver https://review.openstack.org/468753 | 09:07 |
| *** ralonsoh has quit IRC | 09:10 | |
| *** ralonsoh has joined #openstack-infra | 09:11 | |
| *** lrossetti has quit IRC | 09:11 | |
| openstackgerrit | Merged openstack-infra/project-config master: Increase timeout of mixmatch functional job https://review.openstack.org/489369 | 09:16 |
| openstackgerrit | Merged openstack-infra/project-config master: Promote patrole py35 job to voting https://review.openstack.org/489394 | 09:16 |
| *** yamahata has quit IRC | 09:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add missing release notes jobs to sushy https://review.openstack.org/489165 | 09:17 |
| openstackgerrit | Merged openstack-infra/project-config master: Add readthedocs hook to publish networking-cisco docs https://review.openstack.org/489247 | 09:17 |
| *** slaweq_ has joined #openstack-infra | 09:20 | |
| *** sdague has joined #openstack-infra | 09:20 | |
| *** tosky has joined #openstack-infra | 09:24 | |
| *** slaweq_ has quit IRC | 09:24 | |
| *** sambetts|afk is now known as sambetts | 09:27 | |
| *** lrossetti has joined #openstack-infra | 09:31 | |
| *** coolsvap has joined #openstack-infra | 09:32 | |
| *** prateek has quit IRC | 09:33 | |
| *** yamamoto has quit IRC | 09:37 | |
| *** marst_ has joined #openstack-infra | 09:38 | |
| *** marst has quit IRC | 09:38 | |
| *** ramishra has quit IRC | 09:40 | |
| *** ramishra has joined #openstack-infra | 09:41 | |
| *** kjackal_ has joined #openstack-infra | 09:43 | |
| openstackgerrit | Nisha Agarwal proposed openstack-infra/project-config master: Moves the ironic python3 experimental job as non-voting job https://review.openstack.org/489278 | 09:45 |
| *** pgadiya has quit IRC | 09:46 | |
| *** kjackal_ has quit IRC | 09:46 | |
| *** yushb has joined #openstack-infra | 09:47 | |
| *** esberglu has joined #openstack-infra | 09:48 | |
| *** pgadiya has joined #openstack-infra | 09:49 | |
| *** esberglu has quit IRC | 09:53 | |
| *** esikachev has quit IRC | 09:56 | |
| openstackgerrit | Sean Handley proposed openstack-infra/project-config master: Add Public Cloud WG project. https://review.openstack.org/489548 | 09:57 |
| *** esikachev has joined #openstack-infra | 09:57 | |
| *** yamamoto has joined #openstack-infra | 09:57 | |
| *** wolverineav has joined #openstack-infra | 09:59 | |
| *** esikachev has quit IRC | 10:02 | |
| *** esikachev has joined #openstack-infra | 10:03 | |
| openstackgerrit | Sean Handley proposed openstack-infra/project-config master: Add Public Cloud WG project. https://review.openstack.org/489548 | 10:03 |
| *** seanhandley has joined #openstack-infra | 10:04 | |
| *** gmann_ has quit IRC | 10:06 | |
| *** esikachev has quit IRC | 10:08 | |
| *** yamamoto has quit IRC | 10:08 | |
| seanhandley | I'm setting up a new repo so the public cloud working group can create specs: https://review.openstack.org/#/c/489548/2 | 10:09 |
| seanhandley | Eyeballs appreciated <3 | 10:09 |
| *** treaki_ has quit IRC | 10:10 | |
| *** dtantsur|bbl is now known as dtantsur | 10:11 | |
| *** liujiong has quit IRC | 10:12 | |
| *** lewo has quit IRC | 10:15 | |
| *** lewo has joined #openstack-infra | 10:15 | |
| *** yushb has quit IRC | 10:17 | |
| *** prateek has joined #openstack-infra | 10:19 | |
| *** prateek has quit IRC | 10:20 | |
| *** prateek has joined #openstack-infra | 10:21 | |
| *** mugsie has quit IRC | 10:22 | |
| *** mugsie has joined #openstack-infra | 10:25 | |
| *** mugsie has quit IRC | 10:25 | |
| *** mugsie has joined #openstack-infra | 10:25 | |
| *** yamamoto has joined #openstack-infra | 10:25 | |
| openstackgerrit | Sean Handley proposed openstack-infra/project-config master: Add Public Cloud WG project. https://review.openstack.org/489548 | 10:26 |
| *** slaweq_ has joined #openstack-infra | 10:27 | |
| *** thorst_afk has joined #openstack-infra | 10:28 | |
| *** mugsie has quit IRC | 10:31 | |
| *** markvoelker has joined #openstack-infra | 10:31 | |
| *** thorst_afk has quit IRC | 10:32 | |
| *** mugsie has joined #openstack-infra | 10:33 | |
| *** mugsie has quit IRC | 10:33 | |
| *** mugsie has joined #openstack-infra | 10:33 | |
| *** slaweq_ has quit IRC | 10:34 | |
| *** alexchadin has quit IRC | 10:35 | |
| *** sree has joined #openstack-infra | 10:35 | |
| *** alexchadin has joined #openstack-infra | 10:36 | |
| *** katkapilatova has joined #openstack-infra | 10:36 | |
| AJaeger | seanhandley: you need to link the two changes - via needed-by/depends-on, see https://docs.openstack.org/infra/manual/creators.html#add-new-repository-to-the-governance-repository | 10:44 |
| *** panda has quit IRC | 10:44 | |
| *** jkilpatr has quit IRC | 10:44 | |
| *** panda has joined #openstack-infra | 10:44 | |
| seanhandley | AJaeger: Out of interest, have you ever seen anyone follow that document correctly the first time? | 10:45 |
| seanhandley | also, thanks for the help :) | 10:45 |
| seanhandley | I'm basing my work here on the API working group | 10:46 |
| seanhandley | and they don't have a project in the governance repo | 10:46 |
| seanhandley | but they *do* have a repo | 10:46 |
| AJaeger | seanhandley, if you have ideas on how to improve the document, tell me... | 10:46 |
| AJaeger | seanhandley: they have - see reference/technical-committee-repos.yaml in governance repo | 10:47 |
| seanhandley | oic | 10:48 |
| openstackgerrit | Sean Handley proposed openstack-infra/project-config master: Add Public Cloud WG project. https://review.openstack.org/489548 | 10:49 |
| seanhandley | AJaeger: Ok the changes are linked by Needed-By and Depends-On | 10:49 |
| seanhandley | (TIL they exist) | 10:49 |
| *** esikachev has joined #openstack-infra | 10:50 | |
| openstackgerrit | Sean Handley proposed openstack-infra/project-config master: Add Public Cloud WG project. https://review.openstack.org/489548 | 10:53 |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/489533 | 10:57 |
| *** stakeda has quit IRC | 11:01 | |
| *** prateek_ has joined #openstack-infra | 11:03 | |
| *** markvoelker has quit IRC | 11:05 | |
| *** prateek has quit IRC | 11:05 | |
| *** alexchadin has quit IRC | 11:08 | |
| *** EricGonczer_ has joined #openstack-infra | 11:08 | |
| *** slagle has quit IRC | 11:08 | |
| *** alexchadin has joined #openstack-infra | 11:12 | |
| *** jkilpatr has joined #openstack-infra | 11:15 | |
| *** EricGonczer_ has quit IRC | 11:15 | |
| *** yamamoto has quit IRC | 11:17 | |
| *** alexchadin has quit IRC | 11:22 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/groups master: Add absolute urls to embedded map https://review.openstack.org/489568 | 11:23 |
| *** sree has quit IRC | 11:28 | |
| *** sree has joined #openstack-infra | 11:30 | |
| *** markvoelker has joined #openstack-infra | 11:30 | |
| *** yamamoto has joined #openstack-infra | 11:32 | |
| *** pgadiya has quit IRC | 11:35 | |
| sambetts | AJaeger: I'm seeing really weird failures on a patch https://review.openstack.org/#/c/486934/ the test env doesn't seem to get set up correctly | 11:35 |
| *** pgadiya has joined #openstack-infra | 11:36 | |
| sambetts | AJaeger: and the logs where the error should be are non-existent | 11:36 |
| openstackgerrit | Sean Handley proposed openstack-infra/project-config master: Add Public Cloud WG project. https://review.openstack.org/489548 | 11:37 |
| *** esberglu has joined #openstack-infra | 11:37 | |
| *** thorst_afk has joined #openstack-infra | 11:40 | |
| *** prateek_ has quit IRC | 11:41 | |
| *** esberglu has quit IRC | 11:42 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Make github ssl verification configurable https://review.openstack.org/489573 | 11:46 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Make github ssl verification configurable https://review.openstack.org/489573 | 11:48 |
| openstackgerrit | Merged openstack-infra/groups master: Add absolute urls to embedded map https://review.openstack.org/489568 | 11:49 |
| *** yamamoto has quit IRC | 11:49 | |
| *** lucasagomes is now known as lucas-hungry | 11:52 | |
| *** slagle has joined #openstack-infra | 11:53 | |
| *** ldnunes has joined #openstack-infra | 11:58 | |
| *** ccamacho has quit IRC | 12:01 | |
| *** ccamacho has joined #openstack-infra | 12:01 | |
| *** claudiub has joined #openstack-infra | 12:06 | |
| *** tuanluong has quit IRC | 12:08 | |
| *** atarakt has joined #openstack-infra | 12:17 | |
| *** slagle has quit IRC | 12:17 | |
| *** thorst_afk has quit IRC | 12:22 | |
| *** esberglu has joined #openstack-infra | 12:24 | |
| *** dpaterson has quit IRC | 12:25 | |
| *** slagle has joined #openstack-infra | 12:25 | |
| *** mriedem has joined #openstack-infra | 12:27 | |
| *** mhickey has joined #openstack-infra | 12:29 | |
| *** slaweq_ has joined #openstack-infra | 12:31 | |
| *** esikachev has quit IRC | 12:36 | |
| *** slaweq_ has quit IRC | 12:36 | |
| *** dprince has joined #openstack-infra | 12:38 | |
| *** baoli has joined #openstack-infra | 12:39 | |
| *** esberglu has quit IRC | 12:39 | |
| *** links has quit IRC | 12:40 | |
| *** slagle has quit IRC | 12:47 | |
| *** rhallisey has joined #openstack-infra | 12:48 | |
| *** yamamoto has joined #openstack-infra | 12:49 | |
| *** kgiusti has joined #openstack-infra | 12:50 | |
| *** lucas-hungry is now known as lucasagomes | 12:50 | |
| *** gordc has joined #openstack-infra | 12:50 | |
| *** dangers_away has quit IRC | 12:52 | |
| *** electrofelix has quit IRC | 12:52 | |
| *** dangers_away has joined #openstack-infra | 12:54 | |
| *** dave-mccowan has joined #openstack-infra | 12:54 | |
| *** esberglu has joined #openstack-infra | 12:55 | |
| *** mriedem has quit IRC | 12:57 | |
| *** thorst_afk has joined #openstack-infra | 12:57 | |
| *** jcoufal has joined #openstack-infra | 12:58 | |
| *** electrofelix has joined #openstack-infra | 12:58 | |
| *** yamamoto has quit IRC | 12:59 | |
| mordred | AJaeger: are we having any super weird problems with git cloning? | 13:00 |
| mordred | sambetts: ah - that seems to be the same thing I'm seeing | 13:01 |
| sambetts | \o/ I'm not the only one | 13:01 |
| *** ilpianista_ has quit IRC | 13:02 | |
| *** masayukig[m] has quit IRC | 13:02 | |
| mordred | sambetts: I had *some* logs show up for mine (took a little while): http://logs.openstack.org/24/489524/1/check/gate-dsvm-nodepool-src/7cfd364/logs/devstack-gate-setup-workspace-new.txt | 13:02 |
| mordred | if you skip to 2017-08-01 08:35:47.083 you can see it takes 4 minutes to do git branch -a | grep remotes/origin/master | 13:03 |
| *** gouthamr has joined #openstack-infra | 13:03 | |
| *** ykarel has quit IRC | 13:04 | |
| mordred | and at 2017-08-01 09:23:26.028 the same thing takes takes 26 minutes for neutron | 13:04 |
| sambetts | oh wow... :/ | 13:04 |
| mordred | I don't see anything that's changed in devstack-gate or zuul-cloner ... | 13:05 |
| sambetts | it might be the zuul git server itself | 13:06 |
| *** mriedem has joined #openstack-infra | 13:06 | |
| mordred | well.... | 13:07 |
| mordred | sambetts: any idea when you started seeing these failures? | 13:07 |
| mordred | sambetts: (it's extra weird because git fetch doesn't seem to take extra time - just git branch -a) | 13:07 |
| *** eharney has joined #openstack-infra | 13:08 | |
| sambetts | this morning was the first time, patch set kicked off the job at 8:03AM GMT | 13:09 |
| *** LindaWang has quit IRC | 13:09 | |
| sambetts | results came in at 11:13 GMT | 13:09 |
| mordred | sambetts: yah. I have a successful set of jobs from 6 hours ago and a failed set from 3 hours ago | 13:10 |
| mordred | we had new images go live roughly 6 hours ago | 13:11 |
| *** ihrachys has joined #openstack-infra | 13:12 | |
| *** yushb has joined #openstack-infra | 13:14 | |
| *** dizquierdo is now known as dizquierdo_afk | 13:15 | |
| *** rloo has joined #openstack-infra | 13:17 | |
| *** yushb has quit IRC | 13:18 | |
| *** esikachev has joined #openstack-infra | 13:20 | |
| electrofelix | Anyone to review https://review.openstack.org/479192? Ensures git-review works with Gerrit 2.14 | 13:22 |
| mordred | electrofelix: lgtm | 13:23 |
| electrofelix | thanks, just spotted another patch submitted for the same thing so figured it must be impacting | 13:23 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Move the fakeprovider module to the fake driver https://review.openstack.org/488383 | 13:24 |
| *** esikachev has quit IRC | 13:25 | |
| *** claudiub has quit IRC | 13:25 | |
| *** tosky has quit IRC | 13:31 | |
| *** tosky has joined #openstack-infra | 13:31 | |
| *** esikachev has joined #openstack-infra | 13:32 | |
| *** slaweq_ has joined #openstack-infra | 13:32 | |
| AJaeger | mordred: no idea, didn't check further | 13:33 |
| EmilienM | fungi: good morning - fyi https://review.openstack.org/#/c/489466/ | 13:36 |
| *** LindaWang has joined #openstack-infra | 13:36 | |
| fungi | EmilienM: i saw your discussion with clarkb last night. looks like you missed my discussion with cmurphy but presumably you came to the same conclusion on syntax | 13:37 |
| *** slaweq_ has quit IRC | 13:37 | |
| fungi | i was planning to write the fix this morning, ran out of steam last night | 13:37 |
| EmilienM | fungi: ah indeed I missed it | 13:37 |
| *** hongbin has joined #openstack-infra | 13:38 | |
| fungi | just about reached the necessary caffeine levels to review that now that i know what it should have looked like | 13:38 |
| *** masayukig[m] has joined #openstack-infra | 13:40 | |
| *** Goneri has joined #openstack-infra | 13:41 | |
| cmurphy | fungi: EmilienM I have more faith that it's the right conclusion since we came to it independently :) | 13:43 |
| fungi | cmurphy: indeed--thanks! | 13:43 |
| EmilienM | cmurphy, fungi: cool, let's see how it works now :) | 13:44 |
| fungi | once it's merged and i see our ansible run has completed a cycle i'll un-emergency the server and watch the logs again | 13:47 |
| fungi | mordred: need help looking into git server performance? | 13:47 |
| mordred | fungi: I'm fairly stumped at the moment | 13:48 |
| *** liujiong has joined #openstack-infra | 13:48 | |
| fungi | i don't suspect changes in dib itself, its git log has been quiet for days | 13:49 |
| mordred | fungi: my next plan was to put in an autohold and retrigger a shade job - or to try rolling back to the previous image | 13:49 |
| fungi | from the symptoms it sounded like we had maybe regressed pre-caching branches in the local copies on images | 13:49 |
| openstackgerrit | Merged openstack-infra/puppet-openstackci master: Iterate readmes as an hash https://review.openstack.org/489466 | 13:50 |
| fungi | or maybe this is a short-term symptom due to creation of lots of stable/pike branches? not sure exactly how though | 13:50 |
| fungi | load averages are pretty high on our per-provider mirrors | 13:52 |
| *** mat128 has joined #openstack-infra | 13:53 | |
| fungi | huh, we still don't have the citycloud mirrors in cacti either | 13:54 |
| *** ijw has joined #openstack-infra | 13:54 | |
| mordred | fungi: yah - the weird thing is that it doesn't seem to be slow during fetch | 13:56 |
| openstackgerrit | Merged openstack-infra/puppet-translation_checksite master: Update Gemfile https://review.openstack.org/477153 | 13:56 |
| *** ilpianista_ has joined #openstack-infra | 13:57 | |
| fungi | granted there was a bit of a backlog on jobs this morning too... second day in a row yolanda ended up restarting a hung nodepool-launcher daemon | 13:57 |
| fungi | i haven't looked at the logs for it yet, but wonder if some recent change is causing it to have issues during log rotation or something | 13:58 |
| *** mat128 has quit IRC | 13:58 | |
| *** lrossetti has quit IRC | 13:58 | |
| *** camunoz has joined #openstack-infra | 13:59 | |
| *** lrossetti has joined #openstack-infra | 13:59 | |
| *** yushb has joined #openstack-infra | 13:59 | |
| openstackgerrit | Merged openstack-infra/puppet-translation_checksite master: Fix beaker on xenial https://review.openstack.org/477144 | 14:00 |
| *** zeih has quit IRC | 14:02 | |
| *** zeih has joined #openstack-infra | 14:02 | |
| yolanda | hi fungi , i found nodepool-launcher just stopped this morning | 14:03 |
| *** pgadiya has quit IRC | 14:03 | |
| yolanda | more or less it was at the same hour | 14:03 |
| yolanda | as yesterday | 14:03 |
| fungi | yolanda: did it coincide with anything in particular, like log rotation signal handling? | 14:03 |
| yolanda | logs stopped at 6:10am | 14:03 |
| yolanda | is that a familiar time? | 14:03 |
| *** EricGonczer_ has joined #openstack-infra | 14:03 | |
| fungi | 06:10 utc? | 14:04 |
| *** lbragstad_ is now known as lbragstad | 14:04 | |
| *** slaweq has quit IRC | 14:05 | |
| mordred | fungi: fwiw, I had a job complete successfully at 8:21 UTC - but then one fail at 11:05 UTC | 14:06 |
| mordred | fungi: (or report failure at 11:05 UTC) | 14:06 |
| fungi | due to job timeout? | 14:07 |
| mordred | fungi: sambetts reports one reporting failure at 11:13 UTC | 14:07 |
| mordred | fungi: yes | 14:07 |
| mordred | http://logs.openstack.org/24/489524/1/check/gate-shade-functional/e90bb3c/ is an example of a failing job | 14:07 |
| mordred | oh - actually - that one has zero-size workspace setup log ... | 14:07 |
| mordred | http://logs.openstack.org/24/489524/1/check/gate-dsvm-nodepool-src/7cfd364/ is the one where you can see the workspace setup log | 14:08 |
| mordred | http://logs.openstack.org/24/489524/1/check/gate-dsvm-nodepool-src/7cfd364/logs/devstack-gate-setup-workspace-new.txt | 14:08 |
| *** mat128 has joined #openstack-infra | 14:08 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add citycloud mirrors to cacti https://review.openstack.org/489614 | 14:09 |
| fungi | ^ before i forget yet again that it hasn't been done | 14:09 |
| mordred | http://logs.openstack.org/htmlify/24/489524/1/check/gate-dsvm-nodepool-src/7cfd364/logs/devstack-gate-setup-workspace-new.txt#_2017-08-01_09_23_26_028 | 14:10 |
| *** rosmaita has joined #openstack-infra | 14:10 | |
| *** lrossetti has quit IRC | 14:10 | |
| *** mat128 has quit IRC | 14:10 | |
| mordred | is a link to one of the times right before it skips (I added the htmlify to the beginning of the url so we'd get the links) | 14:10 |
| *** mat128 has joined #openstack-infra | 14:11 | |
| mordred | fungi: I'm rechecking that shade patch to see if it was just a temporal abnormality | 14:11 |
| fungi | mordred: both examples ran in infracloud | 14:11 |
| mordred | oh - well that's good to note | 14:12 |
| *** EricGonc_ has joined #openstack-infra | 14:13 | |
| *** markus_z has quit IRC | 14:13 | |
| *** EricGonczer_ has quit IRC | 14:14 | |
| *** Julien-zte has joined #openstack-infra | 14:15 | |
| *** liujiong has quit IRC | 14:15 | |
| *** pgadiya has joined #openstack-infra | 14:15 | |
| *** jamesden_ has joined #openstack-infra | 14:16 | |
| *** jamesdenton has quit IRC | 14:18 | |
| fungi | yeah, wondering if we're still overrunning our real capacity there | 14:19 |
| fungi | or if something has happened to networking for it | 14:20 |
| *** rbrndt has joined #openstack-infra | 14:20 | |
| fungi | or maybe some misbehaving compute nodes | 14:20 |
| *** martinkopec has quit IRC | 14:20 | |
| fungi | but a sample size of 2 is not terribly representative so hard to draw good conclusions just from that | 14:20 |
| yolanda | fungi hep, 6:10 utc | 14:22 |
| dmsimard | I think I've broken python on my local machine after installing gertty ? Does this ring a bell to anyone: http://paste.openstack.org/raw/617126/ | 14:22 |
| *** vponomaryov has left #openstack-infra | 14:22 | |
| *** LindaWang has quit IRC | 14:22 | |
| *** LindaWang has joined #openstack-infra | 14:23 | |
| fungi | dmsimard: looks like something left a dangling setuptools entrypoint for oslo_serialization behind? | 14:24 |
| mordred | fungi: well - ALL of the non-unittest jobs for https://review.openstack.org/#/c/489524/ failed | 14:24 |
| mordred | fungi: http://logs.openstack.org/24/489524/1/check/gate-shade-functional-legacy/de81810/console.html is from osic | 14:24 |
| mordred | although it has a zero-length devstack-gate-setup-workspace-new.txt | 14:25 |
| fungi | mordred: cool, so probably a good counterexample | 14:25 |
| *** jascott1 has joined #openstack-infra | 14:26 | |
| dmsimard | fungi: yeah.. tried reinstalling pip, setuptools and virtualenv, I'll dig some more | 14:27 |
| dmsimard | the fact that I started noticing it after installing gertty might be a coincidence | 14:27 |
| *** marst_ has quit IRC | 14:28 | |
| fungi | i don't think gertty uses anything from oslo besides maybe pbr | 14:29 |
| *** vponomaryov has joined #openstack-infra | 14:30 | |
| mordred | I'm certain it does not | 14:30 |
| *** jascott1 has quit IRC | 14:30 | |
| mordred | dmsimard: looking at your paste | 14:31 |
| mordred | dmsimard: is virtualenv installed on your main machine via pip? | 14:31 |
| mordred | dmsimard: oh - wait ... | 14:32 |
| mordred | dmsimard: in your current directory - is there a file called json.py ? | 14:32 |
| *** eharney has quit IRC | 14:33 | |
| *** slaweq has joined #openstack-infra | 14:33 | |
| mordred | dmsimard: that looks lke potentially pip/_vendor/requests/compat.py is doing an 'import json' which is finding a json.py in your current dir instead of the json module in the stdlib | 14:33 |
| *** slaweq has quit IRC | 14:33 | |
| dmsimard | mordred: omg that's it :/ | 14:34 |
| fungi | if it helps, i don't ever pip install anything into the system python env. i use my distro's virtualenv package to bootstrap a virtualenv that i install latest virtualenv into, and then link it in ~/bin and use that to crease virtualenv sof rmy other python-based tools which i also symlink similarly. reduces questions over what i might have pip-installed to break my environment | 14:34 |
| dmsimard | mordred: you're my hero, I had a junk json.py script I hacked together to test something in /tmp :/ | 14:34 |
| mordred | dmsimard: :) | 14:34 |
| dmsimard | which happened to use oslo_serialization | 14:34 |
| fungi | fun! | 14:35 |
| *** marst has joined #openstack-infra | 14:35 | |
| mordred | it's happened to me before | 14:35 |
| mordred | fungi: amusingly enough - I go the exact opposite route and just make sure I never install any python modules at the system level | 14:35 |
| mordred | from distro packages | 14:35 |
| mordred | that is | 14:35 |
| *** slaweq has joined #openstack-infra | 14:35 | |
| *** pgadiya has quit IRC | 14:35 | |
| fungi | there's a great thread going on the distutils-sig ml right now covering, among other things, whether eth cwd should be in the import path when building wheels (and whether it should be at the begining or end of the path) | 14:35 |
| *** slaweq_ has joined #openstack-infra | 14:36 | |
| fungi | because of people reporting bugs against pip/setuptools/wheel for problems just like that one | 14:36 |
| fungi | mordred: for me, it's mainly because i'd rather not trust arbitrary python packages' setup.py with sudo access | 14:37 |
| fungi | then again, i've also taken to compiling my own builds of latest python 2.7, 3.3, 3.4, 3.5 and 3.6 point releases in my homedir and linking those in ~/bin as well | 14:38 |
| fungi | since testing system python packages is not quite actually testing python | 14:38 |
| mordred | fungi: indeed | 14:38 |
| *** dizquierdo_afk is now known as dizquierdo | 14:39 | |
| clarkb | its not easy for me to build python on suse for that reason :( | 14:39 |
| *** slagle has joined #openstack-infra | 14:39 | |
| clarkb | so many distro specififc hacks | 14:39 |
| mordred | fungi: that makes me want us to make an additional base image that has upstream python installed | 14:39 |
| fungi | clarkb: you can't easily compile python from source on opensuse? | 14:39 |
| mordred | that we could run jobs on as a verification that we're not accidentally depending on non-upstream vendor hacks | 14:40 |
| *** slaweq_ has quit IRC | 14:40 | |
| mordred | maybe after v3 goes live I'll waste a weekend poking at that idea | 14:40 |
| clarkb | fungi: nope because things like curses is all derpy. I can eiyher vendor underpy libs in and ppint python at them of use a patch stack like 40 deep to get it all working | 14:41 |
| fungi | yeah, for some of my personal projects where i want to commit to supporting >=3.3, it nice to be able to test against latest of all the (currently 4) minor versions | 14:41 |
| clarkb | there are also hacks for finding default python path | 14:41 |
| mordred | oh - interesting ... | 14:42 |
| *** psachin has quit IRC | 14:42 | |
| mordred | so - fwiw, the docker "python" image is, in fact, compling python from source not installing it from distro package | 14:42 |
| fungi | clarkb: on debian/unstable at least, i just make sure i have all the dev packages for relevant libs installed and then it's as simple as ./compile --prefix=~/whatever;make;make altinstall | 14:42 |
| fungi | er, ./configure not ./compile | 14:43 |
| *** cshastri has quit IRC | 14:43 | |
| *** panda has quit IRC | 14:43 | |
| mordred | https://github.com/docker-library/python/blob/master/3.5/jessie/slim/Dockerfile fwiw | 14:43 |
| *** panda has joined #openstack-infra | 14:43 | |
| clarkb | ya I found python build support for random unixes and things in tree but $linuxdistro just patches locally | 14:44 |
| mordred | so they grab build-depends, build and install python from upstream source with no patches, then install pip from get-pip.py | 14:45 |
| *** esikachev has quit IRC | 14:45 | |
| mordred | it would be kinda nice to have a similar image in dib - it's a neat concept | 14:46 |
| fungi | shouldn't need a separate get-pip.py since the built python 3 ought to include ensurepip? | 14:46 |
| fungi | latest 2.7 versions ought to as well if memory serves | 14:46 |
| fungi | https://docs.python.org/3/library/ensurepip.html | 14:47 |
| mordred | fungi: they actually disable ensurepip - I think that's becaue they want pip to exist on the system when you create a container from the image - and ensurepip is the tool that's there to allow someone to install pip if they want, right? | 14:47 |
| mordred | hrm - they could really just use python -m ensurepip instead of disabling and then curling | 14:48 |
| fungi | that was my point, yes | 14:48 |
| mordred | I'm mildly curious why they do one over the other... | 14:48 |
| *** eharney has joined #openstack-infra | 14:48 | |
| fungi | cargo cult? | 14:48 |
| fungi | no need to separately retrieve get-pip.py if you build your own python from source. just as a workaround to distro-built pythons which disable it so you'll use separately-packaged pip | 14:49 |
| mordred | fungi: ah ... no - I think I get it | 14:50 |
| fungi | granted if you want a newer version of pip than the bundled one from when you compiled python, you need an extra step to pip install --upgrade pip setuptools wheel (or whatever) | 14:50 |
| mordred | fungi: ensurepip ensures you have pip - and ensurepip --upgrade ensures you have at least as new a pip as bundled with your python | 14:50 |
| mordred | yah- you'd need to do "ensurepip --upgrade && pip install -U pip" | 14:50 |
| mordred | which isn't a super win from a build-time perspective as just grabbing get-pip and running it once | 14:51 |
| fungi | which is still at least a little less cringeworthy that curl|sudo bash | 14:51 |
| fungi | er, than | 14:51 |
| *** xarses has joined #openstack-infra | 14:51 | |
| *** LindaWang has quit IRC | 14:52 | |
| *** zeih has quit IRC | 14:52 | |
| *** annegentle has joined #openstack-infra | 14:53 | |
| *** EricGonc_ has quit IRC | 14:53 | |
| *** ramishra has quit IRC | 14:54 | |
| fungi | one thing i haven't checked yet is whether you can influence the version of pip you bundle in your python build from source, in which case you can sidestep all that | 14:54 |
| *** annegentle has quit IRC | 14:54 | |
| *** annegentle has joined #openstack-infra | 14:55 | |
| EmilienM | fungi, clarkb, cmurphy: thanks for your help; index works now: http://logs.openstack.org/50/487450/8/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/f93b0bd/logs/ | 14:55 |
| *** ramishra has joined #openstack-infra | 14:56 | |
| fungi | EmilienM: yep, i was watching the puppet logs for the server and seems it applied successfully this time | 14:56 |
| jeblair | fungi, mordred: wow, i just read a screenfull about python deps and then it suddenly occurred -- wait? what ever happened with the slow git repo branch stuff? | 14:56 |
| EmilienM | fungi: w00t | 14:56 |
| clarkb | EmilienM: next is http://logs.openstack.org/50/487450/8/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/f93b0bd/logs/undercloud/etc/ cleanup? (note there are many other /etc log locations that need the same treatment) | 14:56 |
| fungi | jeblair: i'm still digging around in cacti graphs | 14:56 |
| jeblair | fungi: i have +3d cacti add | 14:57 |
| fungi | not sure if anyone else has hit on any new evidence yet | 14:57 |
| fungi | thanks jeblair! | 14:57 |
| fungi | jeblair: other thing that still needs a deep-dive is figuring out why nodepool-launcher has hung/died around 06:10 utc two days in a row | 14:58 |
| mordred | jeblair: I also rechecked a patch that had failed earlier to see if it was moment-in-time | 14:58 |
| fungi | mordred: yeah, report on 489524 from 10:05 failed most jobs, then at 14:54 succeeded | 14:59 |
| jeblair | tagged zuul-sphinx 0.1.2 | 14:59 |
| jeblair | (just wanted to get the machinery moving on that for later) | 14:59 |
| EmilienM | clarkb: yes there are patches in progress but people are on PTO this week | 14:59 |
| * fungi needs to jump into a meeting now | 14:59 | |
| fungi | but will try to continue digging in parallel | 14:59 |
| *** mhickey has quit IRC | 15:00 | |
| *** jamesden_ has quit IRC | 15:02 | |
| mordred | fungi: ah - so it did | 15:02 |
| mordred | fungi: so EXCELLENT - it was a moment-in-time problem | 15:02 |
| *** EricGonczer_ has joined #openstack-infra | 15:02 | |
| *** felipemonteiro has joined #openstack-infra | 15:02 | |
| mordred | jeblair: any thoughts on why git fetch would be quick but git branch -a would take 20 minutes? | 15:02 |
| mordred | jeblair: I'm not sure there is anything more we can push on here - it's a super weird and non-reproducible currently :( | 15:03 |
| jeblair | mordred: not one clue | 15:03 |
| jeblair | mordred: i mean, not anything logical. only things like network/disk/etc problems/contention come to mind. | 15:04 |
| openstackgerrit | Merged openstack-infra/system-config master: Add citycloud mirrors to cacti https://review.openstack.org/489614 | 15:04 |
| *** jamesdenton has joined #openstack-infra | 15:05 | |
| clarkb | http://paste.openstack.org/show/617134/ appears to be the launcher logs from when things happened :/ | 15:05 |
| *** LindaWang has joined #openstack-infra | 15:06 | |
| jeblair | clarkb: yeah, i'm not sure if there's much more we can do without a stack trace | 15:06 |
| *** bobh has joined #openstack-infra | 15:06 | |
| clarkb | would be good to hit it with sigusr2 next time and see if it will dump threads | 15:07 |
| jeblair | clarkb: iirc, i think this happened once recently and sigusr2 didn't work? | 15:08 |
| clarkb | jeblair: I know it happened yesterday and today (UTC time) but I don't know if sigusr2 was attempted in those cases | 15:08 |
| jeblair | 2017-07-19 23:07:08 UTC restarted nodepool-launcher which had frozen (did not respond to SIGUSR2) | 15:08 |
| jeblair | clarkb: that's ^ the time i was thinking of | 15:08 |
| *** felipemonteiro has quit IRC | 15:08 | |
| clarkb | may need strace then | 15:09 |
| jeblair | so it's happened 3 times recently, that's the first | 15:09 |
| fungi | according to yolanda, today the daemon had completely died, no process running for it, so nothing to handle any signal there | 15:10 |
| *** LindaWang has quit IRC | 15:10 | |
| jeblair | ah, that's different | 15:10 |
| *** mhickey has joined #openstack-infra | 15:11 | |
| yolanda | yep, logs stopped, i did a ps and was no process. So i just started it | 15:12 |
| jeblair | nothing in syslog around 6:30 | 15:12 |
| yolanda | yesterday it was different, process was still active, but looked stalled | 15:12 |
| yolanda | i had to kill it and start it again, it was not even stopping | 15:12 |
| *** rcernin has quit IRC | 15:14 | |
| yolanda | and both days, i found the situation at the same time, i cannot tell you exact time of the first day, but just when i started working, and the process was inactive for 1:30-2hours | 15:14 |
| clarkb | weird | 15:17 |
| yolanda | maybe just coincidence, if it continues to happen, then it's going to be a pattern | 15:19 |
| *** makowals has quit IRC | 15:19 | |
| fungi | i love tautologies | 15:21 |
| *** pcaruana has quit IRC | 15:22 | |
| *** makowals has joined #openstack-infra | 15:23 | |
| fungi | interesting that we seem to have apparmor running on nodepool.o.o and denying lots of mysql operations | 15:27 |
| fungi | at least according to dmesg | 15:28 |
| fungi | is apparmor intentional there? | 15:28 |
| *** claudiub has joined #openstack-infra | 15:29 | |
| fungi | goes back nearly a month, filling up the ring buffer | 15:30 |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: Update URIs linked in TripleO-CI documentation https://review.openstack.org/489636 | 15:33 |
| *** zz_dimtruck has quit IRC | 15:33 | |
| *** sree has quit IRC | 15:35 | |
| *** armax has quit IRC | 15:36 | |
| *** sree has joined #openstack-infra | 15:36 | |
| *** lrossetti has joined #openstack-infra | 15:36 | |
| *** ykarel has joined #openstack-infra | 15:38 | |
| *** katkapilatova has left #openstack-infra | 15:39 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement an OpenContainer driver https://review.openstack.org/468753 | 15:39 |
| *** pcaruana has joined #openstack-infra | 15:40 | |
| *** armax has joined #openstack-infra | 15:40 | |
| *** sree has quit IRC | 15:40 | |
| fungi | jeblair: clarkb: mordred: on the trail of strange/sluggish git behavior, i think i found a breadcrumb: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=135 | 15:41 |
| *** andreas_s has quit IRC | 15:41 | |
| clarkb | fungi: I think apparmor is default in ubuntu it just isn't appleid to much so typically can be ignored | 15:41 |
| fungi | something seems to have had zm04 so slammed that it didn't respond to snmp queries for over an hour | 15:42 |
| fungi | network traffic seems to have spiked and topped out at over 100mbps | 15:42 |
| fungi | cpu was all the way up to 100% roughly half of which seems to have been system | 15:43 |
| *** eharney_ has joined #openstack-infra | 15:43 | |
| clarkb | might want to check the merger log from that time frame it is possible that git dos'd itself | 15:43 |
| jeblair | fungi: aha | 15:45 |
| *** annegentle has quit IRC | 15:46 | |
| fungi | the mass of failed jobs reporting at 10:05 on https://review.openstack.org/489524 had their zuul refs assigned from zm04 so this looks like more than coincidence | 15:46 |
| *** eharney has quit IRC | 15:46 | |
| fungi | traffic spike was outbound, so maybe we have a repo with a really, really huge ref? | 15:47 |
| fungi | or could be a really terribly-designed job | 15:47 |
| *** eharney_ is now known as eharney | 15:48 | |
| *** annegentle has joined #openstack-infra | 15:48 | |
| *** rossella_s has quit IRC | 15:48 | |
| jeblair | fungi: 8:30 is when that period starts and it's also shortly after nodepool was restarted | 15:49 |
| jeblair | paste.o.o seems unhappy | 15:49 |
| *** rcernin has joined #openstack-infra | 15:49 | |
| fungi | looking at the graph, the traffic spike seems to be starting at 08:05-08:10 utc | 15:49 |
| clarkb | 8:01 is when nodepool launcher started | 15:49 |
| *** hamzy has quit IRC | 15:49 | |
| jeblair | fungi: well that corresponds even better with nodepool onlining | 15:49 |
| fungi | agreed | 15:50 |
| *** e0ne has quit IRC | 15:50 | |
| *** xarses has quit IRC | 15:50 | |
| fungi | wonder why it slammed zm04 and not any of the other 7? | 15:50 |
| *** rossella_s has joined #openstack-infra | 15:50 | |
| fungi | jeblair: paste seems snappy for me. maybe you caught a db timeout/reconnect | 15:51 |
| fungi | as for the timing coinciding between the nodepool-launcher restart and the spike on zm04, easily explained if whatever job(s) triggered this behavior were sitting queued | 15:52 |
| jeblair | fungi: better now | 15:52 |
| dhellmann | hey, folks, it looks like we ended up with a race between 2 publishing jobs for docs.o.o out of openstack-manuals and the result is not quite reflecting what's in master in that repo. Is there a way for us to serialize those jobs so the site is updated in the order we land patches? | 15:53 |
| fungi | and once nodes got assigned, they started slamming zm04 | 15:53 |
| jeblair | fungi, clarkb: a hacky histogram: http://paste.openstack.org/show/617141/ | 15:53 |
| *** yushb has quit IRC | 15:53 | |
| jeblair | dhellmann: i have a good plan for that immediately post-v3 (a new pipeline manager which only enqueues one change past the one currently running, so if 5 changes land, we run the first and the last of those 5).... but in the interim.... | 15:54 |
| fungi | dhellmann: we talked about a new queue type for that in zuul v3, but for now it's "eventually consistent" | 15:54 |
| jeblair | dhellmann: maybe we could stick a mutex on it? | 15:55 |
| sdague | so... we've got the redirects working for nova docs thanks to dhellmann | 15:55 |
| sdague | is there a way we could get http logs of 404s with nova in their url so we can see what's getting lost and build the rest of the redirects? | 15:56 |
| jeblair | dhellmann: the mutex would slow it down, because it would mean strict serialization of all those jobs | 15:56 |
| fungi | a workaround in zuul v2 is to assign those jobs a custom node label corresponding to a persistent node and that way only one instance of the job can run at a time (but it's one more server to manage and if there are a lot of those jobs they may create a bit of a backlog) | 15:56 |
| fungi | sdague: if someone gets time, though obviously lower priority than the several broken systems we're currently attempting to troubleshoot | 15:57 |
| mordred | infra-root: https://etherpad.openstack.org/p/ImHsFudauN draft email about PTG rollout to openstack-dev - that look good to folks? | 15:58 |
| *** ykarel has quit IRC | 15:59 | |
| jeblair | mordred: wow we have the same color | 16:00 |
| *** Apoorva has joined #openstack-infra | 16:00 | |
| fungi | jeblair: comparing your hacky histogram to zm01, looks like zm04 got several times as many hits | 16:01 |
| fungi | so, again, could have been a particular build doing an unnatural amount of git access to its assigned merger | 16:01 |
| fungi | also the number of hits on zm04 likely trails off because after performance topped out jobs stopped getting quick (or any) responses from it | 16:03 |
| fungi | otherwise would have continued to rise to something like an order of magnitude more than the other mergers were seeing | 16:04 |
| *** egonzalez has quit IRC | 16:04 | |
| pabelanger | mordred: ++ | 16:04 |
| fungi | zm02 seems consistent with 01 | 16:05 |
| fungi | hits to zm03 seem way lower than 01 or 02 for the same timeframe though | 16:06 |
| clarkb | sdague: fungi I've got some command history to generate general list of 404s will be easy to generate list for nova | 16:06 |
| fungi | zm05 looks similar to 03 | 16:07 |
| *** makowals has quit IRC | 16:07 | |
| fungi | zm06 is in the same range as 01 and 02 | 16:08 |
| clarkb | sdague: http://paste.openstack.org/show/617142/ | 16:08 |
| *** claudiub has quit IRC | 16:08 | |
| fungi | as is 07 | 16:09 |
| *** makowals has joined #openstack-infra | 16:09 | |
| fungi | though 08 is a bit more than what 01/02/06/07 were seeing | 16:10 |
| fungi | so another theory is we ended up with a really uneven distribution of refs to mergers somehow | 16:10 |
| clarkb | on the frontend of the gearman worker queue we pull jobs off as we are able to process them one by one which does basic load balancing. But on the backend of jobs fetching refs we may not be quite so balanced as a single merger must host all refs for a single job right? | 16:11 |
| clarkb | (if I've interpreted that correctly I think it lends weight to the idea that distribution of refs was uneven) | 16:11 |
| sdague | clarkb: thanks | 16:12 |
| *** links has joined #openstack-infra | 16:12 | |
| fungi | clarkb: sure, and also we don't have any way of knowing the number od hits the merger is going to get from a given job | 16:12 |
| fungi | nor the implied load of any particular request it will service | 16:13 |
| jeblair | clarkb, fungi: yeah, skimming the access log, i'm seeing a bunch of related requests from the same hosts | 16:13 |
| *** rlandy|ci_mtgs has quit IRC | 16:14 | |
| fungi | i tend to suspect the misbehaving job (or jobs, but for the same ref) theory over the uneven distribution in the scheduler theory | 16:15 |
| fungi | occam's razor | 16:15 |
| jeblair | though... comparing hits/ip from 08:20 to 15:20, there are actually fewer earlier than later | 16:15 |
| fungi | the performance impact to the merger could probably explain that | 16:16 |
| fungi | particularly if these were something like a poorly-written job git cloning nova from the merger, or handling a merger on some repo full of huge binary blobs | 16:18 |
| fungi | i can certainly imagine a bespoke job blindly assuming that $ZUUL_URL is a suitable place to clone all its repos in full | 16:19 |
| fungi | rather than relying on devstack-gate or zuul-cloner to do repo setup | 16:19 |
| jeblair | fungi, clarkb: sorting by the number of hits to urls, the 08:20 block had 206 devstack-gate fetches, the most of any url, accounding for 4.3% of all hits. the 15:20 block had 20 devstack-gate fetches, was in the middle of the pack as far as frequency (requirements had 40 fetches) and accounted for 1.4% of hits. | 16:20 |
| jeblair | fungi, clarkb: that makes me think that it ended up with an unusually high number of devstack-gate jobs | 16:21 |
| fungi | http://codesearch.openstack.org/?q=ZUUL_URL is mildly disturbing | 16:21 |
| jeblair | fungi: someone has rendered jjb xml checked into their repo? | 16:22 |
| mordred | yah. that's ... what the heck is that? | 16:22 |
| fungi | though skimming, the majority of $ZUUL_URL uses seem to be accompanied by fetching $ZUUL_REF so most of these are likely okay | 16:22 |
| *** lucasagomes is now known as lucas-afk | 16:23 | |
| pabelanger | clarkb: I'd like to add buildlogs.centos.org to our reverse proxy to help mitigate job timeouts for tripleo / puppet: https://review.openstack.org/#/c/489462/ | 16:23 |
| mordred | have I mentioned how nice zuul pushing git repos to nodes is for things lke this? | 16:23 |
| *** aarefiev is now known as aarefiev_afk | 16:23 | |
| pabelanger | clarkb: I'll propose a change to configure_mirror.sh too | 16:23 |
| *** pblaho has quit IRC | 16:24 | |
| clarkb | pabelanger: is the build logs content reasonably cacheable? | 16:24 |
| clarkb | pabelanger: there was some content somewhere (rdo maybe?) that was per commit so not really cacheable in a reasonable manner | 16:25 |
| pabelanger | clarkb: I believe it changes about the same rate as RDO does today | 16:25 |
| *** jascott1 has joined #openstack-infra | 16:25 | |
| *** jascott1 has quit IRC | 16:25 | |
| pabelanger | clarkb: ya, RDO we couldn | 16:25 |
| clarkb | also do we know what we are downloading from there? | 16:25 |
| pabelanger | 't RSYNC fast enough | 16:25 |
| pabelanger | clarkb: https://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-pike/ is the URL that is being accessed the moest | 16:26 |
| pabelanger | most* | 16:26 |
| *** EricGonczer_ has quit IRC | 16:26 | |
| pabelanger | which is a little different then data in RDO | 16:26 |
| pabelanger | I think this is further down their release pipeline, since it is getting built on centos hardware | 16:27 |
| clarkb | looks like that data is fairly stable, it isn't updating everyday | 16:27 |
| *** rosmaita has left #openstack-infra | 16:28 | |
| pabelanger | clarkb: Oh, now I understand. It is where they push there -rc RPM packages before releasing them | 16:28 |
| pabelanger | they test against that in the gate first, to help catch things | 16:28 |
| fungi | jeblair: mordred: hah, so this seems to be where compass-core grew ci system configuration files: https://review.openstack.org/208083 (looks like it's the configuration for their third-party ci system maybe?) | 16:29 |
| clarkb | as long as the data is fairly stable and not updating everyday (like some other repos) that should be fine | 16:29 |
| jeblair | fungi: that is a fascinating use of a system which was explicitly designed to eliminate the need for xml files in git repos. | 16:30 |
| fungi | indeed | 16:31 |
| fungi | "you're doing it wrong" | 16:32 |
| *** yamahata has joined #openstack-infra | 16:32 | |
| clarkb | pabelanger: left one comment which is mostly a nit today in hopes of avoiding headache in the future | 16:32 |
| dhellmann | jeblair : a mutex might be ok. We could also take care to not approve large batches of patches. Let me discuss it with the docs team | 16:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add buildlogs.centos.org to reverse proxy cache https://review.openstack.org/489462 | 16:34 |
| pabelanger | clarkb: updated, we do the same for registry-1.docker.io below | 16:34 |
| fungi | jeblair: actually, seems they're not using jjb and just happened to have used a similar directory name for their jenkins configs... i don't see the tell-tale jjb comment block in their xml files | 16:34 |
| jeblair | fungi: pretty sure they used it at least once. :) | 16:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-zuul master: Add /keys to apache vhost https://review.openstack.org/489658 | 16:36 |
| pabelanger | jeblair: mordred: ^ pass thru keys for zuulv3.o.o secrets | 16:36 |
| openstackgerrit | Alex Schultz proposed openstack-infra/yaml2ical master: Add Adhoc frequency https://review.openstack.org/489659 | 16:37 |
| fungi | jeblair: yeah, i do see some clues to suggest that, like their use of gerrit-git-prep.sh and run-tox.sh, and similar log publisher filepaths... odds are they started with our project-config anyway | 16:37 |
| fungi | but instead of keeping the yaml, they seem to have kept the xml instead | 16:38 |
| fungi | and removed the managed-by-jjb comment block | 16:38 |
| AJaeger | mordred, jeblair, pabelanger, is the Zuul v3 naming proposal good to merge now? https://review.openstack.org/487848 might be something mordred wants to mention in his email... | 16:41 |
| openstackgerrit | Alex Schultz proposed openstack-infra/irc-meetings master: Update Puppet OpenStack meeting information https://review.openstack.org/489662 | 16:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Switch to using zuul-sphinx https://review.openstack.org/489409 | 16:44 |
| *** ijw has quit IRC | 16:44 | |
| *** vhosakot has joined #openstack-infra | 16:44 | |
| *** hashar has quit IRC | 16:44 | |
| jeblair | AJaeger: lgtm. and yeah, i was thinking that we may want to start building out some more 'zuulv3 for openstack devs' docs in the manual, including that section. | 16:47 |
| *** bobh has quit IRC | 16:47 | |
| AJaeger | jeblair: yeah, some more docs would be great... | 16:48 |
| pabelanger | jeblair: mordred: clarkb: fungi: I'd like to add openstack-dev/sandbox to zuulv3.o.o for testing our release pipeline. This would mean creating new tags and uploading to testpypi.python.org, any objections or suggestions for different project? | 16:48 |
| AJaeger | jeblair: but they can be out-of-tree first - e.g. in presentation form | 16:48 |
| fungi | pabelanger: no objection on my end | 16:49 |
| fungi | seems a pragmatic choice | 16:49 |
| *** ramishra has quit IRC | 16:49 | |
| mordred | pabelanger: wfm | 16:50 |
| *** zeih has joined #openstack-infra | 16:53 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add openstack-dev/sandbox to zuulv3.o.o https://review.openstack.org/489667 | 16:54 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Add /keys to apache vhost https://review.openstack.org/489658 | 16:55 |
| *** slaweq has quit IRC | 16:56 | |
| openstackgerrit | Nisha Agarwal proposed openstack-infra/project-config master: Moves the ironic python3 experimental job as non-voting job https://review.openstack.org/489278 | 16:57 |
| *** zeih has quit IRC | 16:58 | |
| *** bobh has joined #openstack-infra | 16:58 | |
| *** pcaruana has quit IRC | 16:58 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Push translated pages to docs-draft https://review.openstack.org/489669 | 16:58 |
| AJaeger | dhellmann: ^ | 16:59 |
| jeblair | AJaeger, mordred: i'm going to try to write a skeleton zuulv3 for devs doc for the infra manual real quick | 16:59 |
| *** jistr is now known as jistr|afk | 17:00 | |
| AJaeger | jeblair: great! | 17:00 |
| *** baoli has quit IRC | 17:00 | |
| *** ralonsoh has quit IRC | 17:02 | |
| *** hamzy has joined #openstack-infra | 17:03 | |
| *** dizquierdo has quit IRC | 17:03 | |
| *** baoli has joined #openstack-infra | 17:05 | |
| *** slaweq has joined #openstack-infra | 17:07 | |
| *** efoley has quit IRC | 17:07 | |
| *** rkukura has quit IRC | 17:08 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add buildlogs.centos to nodepool variables https://review.openstack.org/489673 | 17:09 |
| *** mhickey has quit IRC | 17:11 | |
| *** baoli has quit IRC | 17:11 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Push translated pages to docs-draft https://review.openstack.org/489669 | 17:12 |
| *** dtantsur is now known as dtantsur|afk | 17:13 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: Enable Django 1.10 test for Sahara UI only from Pike+ https://review.openstack.org/489674 | 17:14 |
| *** jamesmcarthur has joined #openstack-infra | 17:14 | |
| *** baoli has joined #openstack-infra | 17:15 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add buildlogs.centos.org to reverse proxy cache https://review.openstack.org/489462 | 17:18 |
| petevg | Hello, everyone. I have some tests that trigger in the gate which download various packages, and I'd like to cache some of those downloads to save bandwidth and address some reliability issues. Are there docs that y'all can point me at outlining best practices for that sort of thing? | 17:18 |
| petevg | (beisner_ thought that there might be.) | 17:18 |
| *** rbrndt has quit IRC | 17:19 | |
| clarkb | petevg: we already mirror all of pypi and then ubuntu and centos (and I think fedora) where do these packages come from? | 17:19 |
| petevg | clarkb: is the mirror something that I'm going to end up using automatically, or do I need to point my tests at the mirror? | 17:20 |
| AJaeger | petevg: it all depends on what you're doing - please tell us more first | 17:21 |
| clarkb | petevg: for those mirrors it should all be used automatically as long as you aren't using overlay fileystems in something like a container that would overwrite the config we write down in the base os | 17:21 |
| fungi | (or using configuration management within the job which is trying to directly manage those system configs) | 17:21 |
| petevg | clarkb, fungi: got it. I don't think that I'm doing anything to bypass the mirror. I did see a handful of tests fail last week due to slow and failed downloads. Is the mirror going to get upset with me if I run 3 or 4 tests in parallel that all want to do something fairly heavy, like fetch an image from the Ubuntu archive? | 17:23 |
| *** sambetts is now known as sambetts|afk | 17:24 | |
| fungi | petevg: i don't think we mirror any ubuntu image sites | 17:24 |
| clarkb | fungi: correct | 17:24 |
| fungi | what images are these? | 17:24 |
| *** rloo is now known as rloo|afk | 17:24 | |
| fungi | or from where are you trying to obtain them? | 17:25 |
| clarkb | (it would probably help to have information on what you are trying to do and how you are currently doing it) | 17:25 |
| fungi | we mirror ubuntu _packages_ | 17:25 |
| petevg | fungi: xenial images, I think from the Ubuntu cloud archive. The tests install a basic Openstack with snaps, and verify that it can create a VM. | 17:25 |
| fungi | and the package repositories configured in the sources.list refer to our closest package mirrors | 17:25 |
| clarkb | petevg: none of that should require an ubuntu image right? you just need the snaps (which I'm not sure we mirror because they aren't debs and we use reprepro) and then you can use the cirros image we bake into the test nodes right? | 17:27 |
| petevg | I guess that returns me to my original questions. Are there best practices for caching arbitrary stuff in the gate? Or am I free to improvise? :-) | 17:27 |
| clarkb | petevg: there is but we'd like to make sure we aren't already solving the problem for you before we go add caches for $thing | 17:27 |
| petevg | clarkb: same here :-) | 17:28 |
| petevg | The tests need network for two things: | 17:28 |
| petevg | 1) After openstack is setup, they go and fetch images from the cloud archive, and mount them. | 17:28 |
| petevg | 2) Prior to that, the snaps all build themselves, which involves pull source code packages from anywhere that the author specified in the snapcraft.yaml. | 17:29 |
| petevg | *pulling | 17:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-manual master: WIP: add zuul v3 migration doc https://review.openstack.org/489677 | 17:29 |
| jeblair | AJaeger, fungi, mordred: ^ what do you think of the approach I outlined there? | 17:29 |
| clarkb | petevg: for 1) could you just boot the cirros image we bake onto the test nodes? | 17:29 |
| clarkb | petevg: you shouldn't need an ubuntu image to confirm that openstack functions correct? | 17:30 |
| petevg | clarkb: That is a possibility. The tests are supposed to work in people's dev environments as well as in the gate, but I do already have some code that does special things when in the gate. | 17:30 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-zuul master: Use zuul_status_url for /keys endpoint https://review.openstack.org/489679 | 17:30 |
| clarkb | petevg: you may want to consider cirros for local dev anyways because it is much smaller | 17:30 |
| petevg | clarkb: I will read up on it. Thank you. | 17:31 |
| pabelanger | jeblair: mordred: ^ fixes a typeo with /keys endpoint. We should be using zuul_status_url, not zuul_web_url | 17:31 |
| fungi | and so won't beat up their local internet connection or take up as much room on disk | 17:31 |
| fungi | or cause their testing to take longer while waiting for larger images to download | 17:31 |
| clarkb | for 2) http://mirror.dfw.rax.openstack.org/ubuntu/ and http://mirror.dfw.rax.openstack.org/ubuntu-cloud-archive/ are what we have currently mirrored for ubuntu packages | 17:31 |
| petevg | Yep. The tests are basically a first pass at things so, I'm all ears when it comes to stuff that will make them more efficient :-) | 17:31 |
| pabelanger | clarkb: fungi: jeblair: mordred: also, do you mind a +3 on https://review.openstack.org/489667 adding openstack-dev/sandbox to zuulv3.o.o | 17:31 |
| clarkb | petevg: can you check if that includes what you need? | 17:32 |
| jeblair | pabelanger: done and done | 17:32 |
| pabelanger | ty | 17:32 |
| petevg | clarkb: that also looks like it contains what I need. Between that and cirros, I think that we've addressed issue number 1. | 17:33 |
| clarkb | petevg: and 2? | 17:34 |
| petevg | Issue #2, "snapcraft.yaml can get source from anywhere", is a bit trickier. | 17:34 |
| clarkb | cirros for 1 and package mirrosr for 2 unless I've misunderstood | 17:34 |
| *** rkukura has joined #openstack-infra | 17:34 | |
| fungi | apparently they don't use ubuntu packages exclusively in snaps? | 17:35 |
| petevg | clarkb, fungi: correct. Snaps are a package format. They build from source, not debs. | 17:35 |
| petevg | ... and that source can live anywhere. | 17:35 |
| clarkb | petevg: where do sources currently live? | 17:35 |
| fungi | but could be retrieving stuff from random github repos and compiling, or from sdists on pypi, or js tarballs from npm, or... | 17:35 |
| fungi | ? | 17:36 |
| *** jascott1 has joined #openstack-infra | 17:36 | |
| fungi | petevg: have a link to a good example snapcraft.yaml that gets sources from multiple places? | 17:36 |
| fungi | sounds more like freebsd ports | 17:37 |
| petevg | fungi, clarkb: pulling up some examples right now ... | 17:37 |
| fungi | where you just keep the build recipes, but the build system snags source code from various upstreams | 17:37 |
| petevg | fungi, clarkb: snap-nova is a good example. If you grep through https://github.com/openstack/snap-nova/blob/master/snapcraft.yaml for the word "source", you'll see it a healthy variety :-) | 17:39 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Switch to using zuul-sphinx https://review.openstack.org/489409 | 17:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Add openstack-dev/sandbox to zuulv3.o.o https://review.openstack.org/489667 | 17:39 |
| petevg | fungi: you are correct. Lots of upstream sources. | 17:39 |
| *** tosky has quit IRC | 17:39 | |
| openstackgerrit | Merged openstack-infra/shade master: Use more specific asserts in tests https://review.openstack.org/489524 | 17:39 |
| fungi | yeah, caching this is fairly intractable for our current ci system design | 17:40 |
| mordred | it is | 17:40 |
| petevg | fungi: no worries. I mainly just wanted to know if the a system was in place. And it sounds like there is one for half of my problem. | 17:40 |
| petevg | I can work on addressing the other half. | 17:41 |
| fungi | we'd need a full proxy solution for the test nodes, and since we can't predict the addresses/networks they'll come from and don't want to run an open proxy for obvious security reasons... | 17:41 |
| mordred | also - I would HIGHLY recommend changing lines like https://github.com/openstack/snap-nova/blob/master/snapcraft.yaml#L66 to point at git.openstack.org/openstack instead of github | 17:41 |
| clarkb | fungi: right thats the biggest issue | 17:41 |
| openstackgerrit | Merged openstack-infra/shade master: router: Ignore L3 HA ports when listing interfaces https://review.openstack.org/482153 | 17:41 |
| clarkb | I don't want to be an open proxy on the internet | 17:41 |
| openstackgerrit | Merged openstack-infra/shade master: Add Neutron QoS minimum bandwidth rule commands https://review.openstack.org/489390 | 17:41 |
| *** gyee has joined #openstack-infra | 17:41 | |
| petevg | mordred: noted. I'll tag that as a TODO. | 17:41 |
| mordred | petevg: we'll also want to sync up to figure out a way to tell snappy that we've already cloned something for it | 17:42 |
| petevg | mordred: +1 | 17:42 |
| mordred | petevg: because in v3 you can tell a job that it requires other repos to run -so zuul will happily put git.openstack.org/openstack/snap.openstack into th esource dir for you | 17:42 |
| petevg | Nice. | 17:43 |
| fungi | we _do_ at least have a targeted proxy for tarballs.openstack.org available, so you could in theory try to find a way to leverage that in your jobs for items you're grabbing from there (e.g. those nova tarballs) | 17:43 |
| mordred | yah - I think it has the potential to work really nicely for files like that | 17:43 |
| clarkb | and today we've got /opt/git/openstack/nova too... | 17:43 |
| mordred | yup | 17:43 |
| *** yamahata has quit IRC | 17:43 | |
| *** electrofelix has quit IRC | 17:43 | |
| mordred | but it should be extra easy to find git+https://git.openstack.org/openstack/snap.openstack and replace with git+file:///home/zuul/src/git.openstack.org/openstack/snap.openstack | 17:44 |
| *** Goneri has quit IRC | 17:44 | |
| mordred | and could potentially be pretty neat if it can be sorted out generally | 17:45 |
| fungi | similarly stream edit the tarballs.openstack.org urls with equivalent proxy urls from the shell snipped we provide on the nodes | 17:45 |
| AJaeger | jeblair: one nit commented, otherwise, I like structure, let's merge and iterate... | 17:45 |
| fungi | s/snipped/snippet/ | 17:45 |
| jeblair | AJaeger: cool, i started on the second section locally | 17:45 |
| jeblair | AJaeger: heh, yep. | 17:45 |
| petevg | mordred, fungi: that makes a lot of sense. Thank you. | 17:46 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-manual master: WIP: add zuul v3 migration doc https://review.openstack.org/489677 | 17:46 |
| mordred | petevg: sure thing - after the PTG I'll have more bandwidth to help think about ways to make that part nicer/more seamless | 17:47 |
| jeblair | AJaeger: ^ updated | 17:47 |
| *** esikachev has joined #openstack-infra | 17:47 | |
| fungi | so while we don't have a general solution for improving access to random stuff from the internet, we do cache some things many jobs are likely to want nearby (on the local filesystem and/or available from caches located in the same clouds) which could help some if you take advantage of them | 17:47 |
| mordred | jeblair: lgtm - WIP still in commit title | 17:47 |
| AJaeger | jeblair: LGTM - and I agree with mordred | 17:47 |
| AJaeger | mordred, pabelanger, is the Zuul v3 naming proposal good to merge now? https://review.openstack.org/487848 | 17:48 |
| mordred | yah - +3 | 17:49 |
| petevg | fungi: ack. I've saved off notes here. Is there a doc that I can read when/if I want to know what else is currently proxied? | 17:49 |
| AJaeger | thanks, mordred | 17:50 |
| *** Julien-zte has quit IRC | 17:50 | |
| petevg | mordred: I'll be at the PTG, and would be happy to buy you a beverage and chat about stuff, if you have the time. | 17:50 |
| clarkb | petevg: http://mirror.dfw.rax.openstack.org will give you everything we mirror | 17:51 |
| *** Julien-zte has joined #openstack-infra | 17:51 | |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-manual master: Add zuul v3 migration doc https://review.openstack.org/489677 | 17:51 |
| petevg | clarkb: thx! | 17:51 |
| jeblair | mordred, AJaeger: ^ :) | 17:52 |
| clarkb | petevg: for reverse proxy caches we don't have a good way of advertising those yet I don't think (though there is a bash script that can be sourced to load up the info in jobs, https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/configure_mirror.sh#n105) | 17:52 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Describe Zuul v3 job naming https://review.openstack.org/487848 | 17:52 |
| clarkb | petevg: if you scroll up from that link you'll see what is currently in that file | 17:52 |
| petevg | Cool. Thank you again. | 17:52 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Use zuul_status_url for /keys endpoint https://review.openstack.org/489679 | 17:53 |
| *** dmsimard is now known as dms | 17:54 | |
| *** dms is now known as dmsimard | 17:55 | |
| *** slaweq has quit IRC | 17:56 | |
| AJaeger | jeblair: you have my +2 - mordred, do yo uwant to +2A 489677 ? | 17:56 |
| *** slaweq has joined #openstack-infra | 17:57 | |
| *** links has quit IRC | 17:57 | |
| fungi | petevg: come to the infra room at the ptg monday/tuesday and we're happy to help work through some of this stuff there too if you get stuck | 17:58 |
| fungi | we have a "helproom" the first two days intended for stuff like this | 17:58 |
| petevg | fungi: I will do so. Thank you! | 17:58 |
| fungi | or, you know, beer works too ;) | 17:58 |
| AJaeger | project-config cores, could you review a small update for translated manuals, please? https://review.openstack.org/489669 | 17:58 |
| petevg | Beer is always good. :-) | 17:59 |
| *** baoli has quit IRC | 17:59 | |
| *** Apoorva_ has joined #openstack-infra | 17:59 | |
| *** dmsimard is now known as dms | 17:59 | |
| *** dms is now known as dmsimard | 17:59 | |
| *** baoli has joined #openstack-infra | 18:00 | |
| *** baoli has quit IRC | 18:00 | |
| *** Apoorva_ has quit IRC | 18:00 | |
| *** baoli has joined #openstack-infra | 18:01 | |
| *** Apoorva has quit IRC | 18:01 | |
| *** Apoorva_ has joined #openstack-infra | 18:01 | |
| *** slaweq has quit IRC | 18:01 | |
| *** slaweq has joined #openstack-infra | 18:01 | |
| *** dmsimard is now known as dms | 18:02 | |
| *** markvoelker has quit IRC | 18:02 | |
| *** markvoelker has joined #openstack-infra | 18:02 | |
| *** dms is now known as dmsimard | 18:03 | |
| *** tosky has joined #openstack-infra | 18:03 | |
| *** esikachev has quit IRC | 18:04 | |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-manual master: Add more content to zuul v3 migration https://review.openstack.org/489691 | 18:04 |
| *** kjackal_ has joined #openstack-infra | 18:04 | |
| *** apetrich has quit IRC | 18:04 | |
| jeblair | AJaeger, mordred: ^ another chunk. i have to afk for a bit now. | 18:05 |
| *** apetrich has joined #openstack-infra | 18:05 | |
| *** jamesmcarthur has quit IRC | 18:05 | |
| *** pvaneck has joined #openstack-infra | 18:05 | |
| *** dmsimard is now known as dmoreaus | 18:07 | |
| *** dmoreaus is now known as dmsimard | 18:07 | |
| *** ijw has joined #openstack-infra | 18:07 | |
| *** ijw has joined #openstack-infra | 18:07 | |
| *** lrossetti has quit IRC | 18:08 | |
| *** lrossetti has joined #openstack-infra | 18:08 | |
| *** slagle has quit IRC | 18:09 | |
| *** slagle has joined #openstack-infra | 18:11 | |
| *** lrossetti_ has joined #openstack-infra | 18:13 | |
| *** lrossetti has quit IRC | 18:13 | |
| *** esikachev has joined #openstack-infra | 18:14 | |
| *** EmilienM is now known as emacchi | 18:14 | |
| *** emacchi is now known as EmilienM | 18:14 | |
| AJaeger | jeblair: reads nice, commented | 18:18 |
| *** atarakt has quit IRC | 18:18 | |
| *** atarakt has joined #openstack-infra | 18:18 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Disable inactive infra-root accounts https://review.openstack.org/489698 | 18:18 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: tripleo: move container upgrade jobs to check https://review.openstack.org/489702 | 18:19 |
| clarkb | fungi: did you see my comments on https://review.openstack.org/#/c/483974/ ? was that script already used to generate voter rolls? wondering how much updating you want to do to it on this first pass | 18:19 |
| fungi | clarkb: saw, thanks! haven't had time to address yet but it's on my agenda for later today | 18:20 |
| *** yamahata has joined #openstack-infra | 18:20 | |
| fungi | i did attempt to make the alteration as unobtrusive to the current logic in the script as possible to avoid any concerns over it modifying the result in unintended ways | 18:21 |
| *** rbrndt has joined #openstack-infra | 18:21 | |
| fungi | the existing script (prior to the proposed changes) has been used for electoral roll generation for a few cycles | 18:21 |
| fungi | with minor updates to deal with changes in schema for the governance projects list | 18:22 |
| clarkb | ya mostly just thinking we should record when something breaks | 18:22 |
| clarkb | so that we can go back easily and say "oh ya you didn't have a matching email addr" | 18:22 |
| fungi | well, an early iteration did output warnings for the ones it found, but as some ~12% of the contributors don't have corresponding foundation memberships that got really noisy. the information is already easy to obtain from the _all_owners.yaml file produced by the script | 18:24 |
| fungi | in fact, the voter rolls can be completely recreated based on the yaml output, they're just built by the script as an added convenience | 18:24 |
| clarkb | oh I didn't realize it produced a full yaml output at the end I guess taht would include a lack of membership id | 18:24 |
| clarkb | nice | 18:24 |
| fungi | so _all_owners.yaml has everyone who contributed to those repos along with an optional field that includes their member id if it was able to find one | 18:25 |
| *** slaweq has quit IRC | 18:25 | |
| clarkb | great then much less concerend about handling errors early | 18:26 |
| *** slaweq has joined #openstack-infra | 18:26 | |
| fungi | yeah, treating them as warnings was just overwhelming from a volume perspective | 18:26 |
| fungi | here's what the yaml blob for me looks like from a recent run i did to get a list of people for discount invites who contributed during the current cycle: http://paste.openstack.org/show/617158/ | 18:27 |
| clarkb | and we just don't print member: XYZ at all if it is missing ya? | 18:28 |
| fungi | right, there is simply no member key in the entry if it couldn't find a member | 18:28 |
| clarkb | perfect | 18:28 |
| clarkb | this way we know the user was checked and failed to find a member rather than just $bug | 18:28 |
| *** Goneri has joined #openstack-infra | 18:29 | |
| fungi | preferred contains the gerrit preferred e-mail address, extra contains any additional e-mail addresses that account has in gerrit, and the script iterates through all of them asking the foundation member directory "do you have a member profile for this e-mail address?" until it gets a hit | 18:29 |
| *** slaweq has quit IRC | 18:29 | |
| *** slaweq has joined #openstack-infra | 18:30 | |
| fungi | also all the necessary info is there if someone needs to double-check an entry, including repeating the foundation member directory api query if there's suspicion the membership does actually exist for some reason | 18:30 |
| *** slaweq has quit IRC | 18:31 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config master: Notify changes in OSC feature/osc4 branch to IRC https://review.openstack.org/489703 | 18:31 |
| fungi | one other thing i was planning to do in the next few days is do a prelim run for the full tc roll but with the ptl elections timeframe and put together a list of the names and corresponding member id numbers, then a second list of the names which had no member ids found, and follow up to the ml post about the election changes with links to both preliminary lists | 18:32 |
| fungi | that way anyone who wants to know if they're going to be eligible to vote or need to go update their member profiles before the election can do that | 18:32 |
| fungi | (this was suggested by dims as an extra precaution) | 18:32 |
| *** annegentle has quit IRC | 18:33 | |
| fungi | i did perform a bunch of spot checks of the output too, just to make sure i didn't find any obvious inaccuracies, but it's a very large data set after all | 18:34 |
| *** jistr|afk is now known as jistr | 18:34 | |
| clarkb | that sounds like a good idea (though may need to make it less spammer friendly?) | 18:35 |
| *** slaweq has joined #openstack-infra | 18:35 | |
| fungi | i won't be including e-mail addresses, so hopefully it wouldn't be? | 18:35 |
| fungi | just names and member id numbers | 18:35 |
| clarkb | oh I see you'd just pount out the ya that | 18:35 |
| clarkb | ++ | 18:35 |
| fungi | and also invite anyone who thinks there's a discrepancy in who should be on which lists to reach out to me so i can spot the bug if there is one | 18:36 |
| clarkb | that sounds like a great idea | 18:37 |
| *** ijw has quit IRC | 18:39 | |
| *** slaweq has quit IRC | 18:40 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Update tox-tarball to use tox role https://review.openstack.org/489705 | 18:43 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Use py35 tox envlist for documentation https://review.openstack.org/489707 | 18:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Use py35 tox envlist for documentation https://review.openstack.org/489707 | 18:45 |
| *** ijw has joined #openstack-infra | 18:46 | |
| *** jkilpatr has quit IRC | 18:47 | |
| *** yushb has joined #openstack-infra | 18:50 | |
| *** jamesmcarthur has joined #openstack-infra | 18:51 | |
| *** zeih has joined #openstack-infra | 18:54 | |
| *** yushb has quit IRC | 18:54 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Update tox-tarball to use tox role https://review.openstack.org/489705 | 18:56 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Clean up accounts for former root sysadmins https://review.openstack.org/489711 | 18:57 |
| *** adisky__ has quit IRC | 18:58 | |
| *** bobh has quit IRC | 18:58 | |
| *** jkilpatr has joined #openstack-infra | 18:58 | |
| fungi | it's that (weekly infra team meeting) time again! find us in #openstack-meeting for the next hour | 19:00 |
| *** zeih has quit IRC | 19:00 | |
| * AJaeger walks over | 19:00 | |
| pabelanger | fungi: clarkb: +2 on 489711 will comment | 19:02 |
| clarkb | pabelanger: hrm fun | 19:03 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Pass config from Stream to FailEvent https://review.openstack.org/397973 | 19:03 |
| pabelanger | clarkb: ya, I've been meaning to update the ansible task for ssh keys to support edit, where it will delete and create if data is different | 19:06 |
| *** gordc has quit IRC | 19:06 | |
| *** EricGonczer_ has joined #openstack-infra | 19:06 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Update tox-tarball to use tox role https://review.openstack.org/489705 | 19:06 |
| clarkb | pabelanger: thinking about it more we can probably just switch it out quickly and be fine. Only nodes booted between deleting and creating will fail and since its nodepool thats fine | 19:07 |
| pabelanger | clarkb: exactly | 19:07 |
| pabelanger | we'll have to delete key first, then re-run cloud-launcher | 19:08 |
| jeblair | fungi: if time permits, can we sneak in a zuulv3 meeting topic? | 19:10 |
| fungi | jeblair: i'll do that next! | 19:10 |
| *** ijw has quit IRC | 19:12 | |
| *** beisner_ is now known as beisner | 19:12 | |
| *** zeus has quit IRC | 19:13 | |
| *** zeus has joined #openstack-infra | 19:15 | |
| *** zeus is now known as Guest47987 | 19:15 | |
| *** jamesmcarthur has quit IRC | 19:17 | |
| *** annegentle has joined #openstack-infra | 19:17 | |
| *** jamesmcarthur has joined #openstack-infra | 19:19 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Rename Node.hold_reason to 'comment' https://review.openstack.org/489717 | 19:20 |
| *** hashar has joined #openstack-infra | 19:21 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Update tox-tarball to use tox role https://review.openstack.org/489705 | 19:26 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Simplify run tox task https://review.openstack.org/487551 | 19:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 19:30 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add --detail option to nodepool list command https://review.openstack.org/489720 | 19:30 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add hold job to nodepool list output https://review.openstack.org/489721 | 19:30 |
| *** baoli has quit IRC | 19:31 | |
| *** rcernin has quit IRC | 19:36 | |
| *** slaweq has joined #openstack-infra | 19:36 | |
| *** jamesmcarthur has quit IRC | 19:37 | |
| *** baoli has joined #openstack-infra | 19:38 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Update tox-tarball to use tox role https://review.openstack.org/489705 | 19:39 |
| *** slaweq has quit IRC | 19:41 | |
| *** jkilpatr_ has joined #openstack-infra | 19:44 | |
| *** nicolasbock has quit IRC | 19:45 | |
| *** jkilpatr has quit IRC | 19:46 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: DNM - testing https://review.openstack.org/489724 | 19:48 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: DNM - testing https://review.openstack.org/489724 | 19:53 |
| *** wolverineav has quit IRC | 19:53 | |
| *** wolverineav has joined #openstack-infra | 19:54 | |
| *** rloo|afk is now known as rloo | 19:54 | |
| *** eharney has quit IRC | 19:56 | |
| EmilienM | clarkb: just a heads-up on tripleo/etc topic - we have some patches in progress but folks who work on it are on PTO and it's a low-activity week for us | 19:58 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: DNM - testing https://review.openstack.org/489724 | 19:58 |
| EmilienM | just to let you know, it's something we have in the backlog and we're working on it | 19:58 |
| clarkb | EmilienM: ok thanks for the heads up | 19:58 |
| *** bobh has joined #openstack-infra | 20:00 | |
| cmurphy | I'm looking for reviews to get the puppet-4 ball rolling https://review.openstack.org/#/c/481955/ | 20:00 |
| cmurphy | from there we can start getting the puppet-apply jobs to test puppet 4, totally independent of any beaker work | 20:01 |
| *** priteau has quit IRC | 20:01 | |
| *** dprince has quit IRC | 20:04 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: DNM - testing https://review.openstack.org/489724 | 20:06 |
| *** e0ne has joined #openstack-infra | 20:09 | |
| *** eharney has joined #openstack-infra | 20:11 | |
| cmurphy | thanks jeblair | 20:14 |
| *** wolverineav has quit IRC | 20:17 | |
| *** wolverineav has joined #openstack-infra | 20:17 | |
| *** wolverineav has quit IRC | 20:17 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: DNM - testing https://review.openstack.org/489724 | 20:18 |
| *** jtomasek has quit IRC | 20:20 | |
| ianw | greghaynes / anyone : we had a fire-drill once over /etc/machine-id in images. i'm trying to remember the exact details, if it rings any bells | 20:22 |
| greghaynes | ianw: I want to say it was ironic related | 20:24 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Update tox-tarball to use tox role https://review.openstack.org/489705 | 20:24 |
| greghaynes | ianw: the issue was we were having a static machine id set iirc | 20:25 |
| *** e0ne has quit IRC | 20:25 | |
| *** esikachev has quit IRC | 20:25 | |
| greghaynes | There may be a lp issue? | 20:25 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Update tox-tarball to use tox role https://review.openstack.org/489705 | 20:26 |
| ianw | greghaynes: yeah ... which seems similar to https://review.openstack.org/#/c/489013/ (Clear /etc/machine-id to avoid duplicate machine-ids) | 20:26 |
| *** kgiusti has left #openstack-infra | 20:26 | |
| *** slaweq has joined #openstack-infra | 20:27 | |
| greghaynes | hrm I thought I made a patch which did this at some point | 20:28 |
| *** baoli has quit IRC | 20:29 | |
| *** e0ne has joined #openstack-infra | 20:30 | |
| greghaynes | ianw: yep that looks like the kind of issue I remember | 20:32 |
| jeblair | clarkb, infra-root: it looks like bandersnatch is having problems. i'm looking into it. | 20:33 |
| *** sree_ has joined #openstack-infra | 20:37 | |
| *** sree_ is now known as Guest16991 | 20:37 | |
| *** baoli has joined #openstack-infra | 20:38 | |
| jeblair | http://paste.openstack.org/show/617170/ | 20:38 |
| ianw | greghaynes: not totally convinced ... i'm wondering what's actually making that. dropped a comment too | 20:39 |
| *** Guest16991 has quit IRC | 20:41 | |
| *** dizquierdo has joined #openstack-infra | 20:41 | |
| jeblair | infra-root: anyone understand that error? | 20:42 |
| *** gordc has joined #openstack-infra | 20:43 | |
| *** ijw has joined #openstack-infra | 20:43 | |
| fungi | jeblair: how long has it been doing that? | 20:44 |
| fungi | i can't look just now, about to sit down to eat | 20:45 |
| jeblair | fungi: last success 2017-08-01 02:16:38,045 | 20:45 |
| ianw | jeblair: any upgrades run? old virtualenv new system type thing? | 20:45 |
| fungi | wondering if they upgraded the https cert today to one requests doesn't recognize | 20:46 |
| jeblair | ianw: i don't see anything relevant in syslog at that time | 20:47 |
| fungi | hrm, though if it really is a 403 we're getting past the https negotiation anyway | 20:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 20:49 |
| fungi | last bandersnatch release was a few months ago, so wouldn't be that | 20:49 |
| ianw | it works "by hand" -> http://paste.openstack.org/show/617174/ | 20:50 |
| ianw | <Response [200]> | 20:50 |
| jeblair | ianw: what if you set user agent to 'bandersnatch/1.11 (CPython 2.7.6-final0, Linux 3.13.0-116-generic x86_64)' | 20:51 |
| fungi | wondering if fastly has changed something on the cdn that breaks bandersnatch, yeah | 20:51 |
| * fungi disappears to eat | 20:51 | |
| ianw | jeblair: <Response [403]> | 20:52 |
| jeblair | neat | 20:52 |
| ianw | well there you go | 20:52 |
| ianw | interesting, it is very specific | 20:54 |
| ianw | change anything and it seems to work ... it's not just matching on "bandersnatch" | 20:55 |
| *** zeih has joined #openstack-infra | 20:55 | |
| jeblair | ianw: if i change the version to 1.10 it fails, but if i change to to 2.0 it succeeds | 20:55 |
| ianw | yep, seems it's "bandersnatch/1.?" that fails | 20:56 |
| *** EricGonczer_ has quit IRC | 20:57 | |
| ianw | not sure how to tell the difference between "we have a legitimate problem with this version" and "oops, we typed in the wrong regex" | 20:57 |
| *** EricGonczer_ has joined #openstack-infra | 20:57 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add --detail option to nodepool list command https://review.openstack.org/489720 | 20:58 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add hold job to nodepool list output https://review.openstack.org/489721 | 20:58 |
| *** jcoufal has quit IRC | 20:59 | |
| *** camunoz has quit IRC | 20:59 | |
| *** zeih has quit IRC | 21:00 | |
| *** rhallisey has quit IRC | 21:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 21:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add README for tox role https://review.openstack.org/489750 | 21:01 |
| openstackgerrit | Merged openstack/diskimage-builder master: Move selinux packages to redhat-common https://review.openstack.org/489434 | 21:02 |
| ianw | jeblair: so all the hash-index changes made it in to 2.0.0 IIRC (https://bitbucket.org/pypa/bandersnatch/commits/ef50efc1d2a80715b499725331c2dccd45153151) | 21:04 |
| ianw | maybe we just update? | 21:04 |
| jeblair | ianw: <sigh> i guess? | 21:05 |
| jeblair | not how i wanted to spend my day | 21:05 |
| ianw | jeblair: i can take a look if you like, i already spent some time on that merging those changes | 21:06 |
| ianw | i also asked in #pypa, although i doubt anyone there knows | 21:07 |
| *** Guest47987 is now known as zeus` | 21:07 | |
| jeblair | ianw: heh, i also asked #python-dev. your idea was probably better. :) | 21:07 |
| *** zeus` is now known as zeus | 21:07 | |
| *** zeus has quit IRC | 21:07 | |
| *** zeus has joined #openstack-infra | 21:07 | |
| ianw | jeblair: iirc, the holdup was just confirming that 2.0 wouldn't completely destroy everything but doing a test clone | 21:08 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add --detail option to nodepool list command https://review.openstack.org/489720 | 21:08 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add hold job to nodepool list output https://review.openstack.org/489721 | 21:08 |
| *** marst has quit IRC | 21:09 | |
| jeblair | ianw: okay cool. tbh, we're probably in a good position here -- we can avoid releasing the volume until we're sure 2.0 works. | 21:09 |
| ianw | yeah, that's what i'm thinking, although if it is screwed, i guess it means a resync | 21:10 |
| mordred | golly here's hoping that bandersnatch v2 doesn't just completely rewrite all the files on disk | 21:11 |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/shade master: Add support for get details of available QoS rule type https://review.openstack.org/489751 | 21:11 |
| *** camunoz has joined #openstack-infra | 21:11 | |
| jeblair | ianw: i'm holding a flock on mirror-update, so current bandersnatch shouldn't run. but hang on before you do anything to the volume. there's activity in #python-dev | 21:12 |
| *** baoli has quit IRC | 21:13 | |
| *** baoli has joined #openstack-infra | 21:13 | |
| jeblair | ianw: they would like us to try again | 21:15 |
| ianw | yep, returns 200 for me now | 21:15 |
| *** ihrachys has quit IRC | 21:15 | |
| jeblair | ianw: cool, am i clear to do a manual bandersnatch run? (i'm assuming you haven't installed v2 globally yet, right?) | 21:16 |
| ianw | jeblair: i have not installed or modified anything | 21:16 |
| jeblair | okay, it's off and running | 21:16 |
| *** yamamoto has joined #openstack-infra | 21:17 | |
| fungi | okay, i'm back | 21:17 |
| ianw | fungi: it's ok, jeblair fixed it :) | 21:17 |
| fungi | i want to say my fix for the exit code did _not_ make it into the last release | 21:18 |
| fungi | so we were running the previous release with that hand-applied | 21:18 |
| jeblair | fungi: oh i saw something in 2.0 about exit codes but not sure if it's that one | 21:19 |
| fungi | or something like that | 21:19 |
| *** baoli has quit IRC | 21:19 | |
| fungi | oh, yep, that's mentioned in the release notes | 21:19 |
| fungi | maybe we didn't notice they released a 2.0 | 21:20 |
| fungi | i expect we were still on 1.11 with my exit code fix applied | 21:20 |
| jeblair | yeah, we're running 1.11 | 21:20 |
| jeblair | so sounds like multiple reasons to upgrade to 2.0. at our leisure. :) | 21:20 |
| *** yamamoto has quit IRC | 21:21 | |
| fungi | ahh, need to reinstall it for python 3 though | 21:21 |
| jeblair | is it 3.5+ only? | 21:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 21:21 |
| fungi | preferably minimum of python 3.5 so it seems, though might work with 3.3 and 3.4 | 21:21 |
| *** yamamoto has joined #openstack-infra | 21:21 | |
| fungi | official support starts with 3.5 per release notes | 21:21 |
| jeblair | neat | 21:21 |
| fungi | and plans to start relying on >=3.5 features in future releases | 21:22 |
| mordred | at least it won't be our first production python3 service :) | 21:22 |
| *** ldnunes has quit IRC | 21:23 | |
| fungi | well, mirror-update.o.o is still on trusty and so python 3.4.3 | 21:23 |
| fungi | so we may need to consider replacing it with a xenial server | 21:23 |
| ianw | hang on, just let me get the yak, it needs a shave :) | 21:25 |
| *** yamamoto has quit IRC | 21:26 | |
| *** eharney has quit IRC | 21:27 | |
| *** dave-mccowan has quit IRC | 21:27 | |
| jeblair | i let #openstack-release know about the mirror issue | 21:28 |
| fungi | thanks jeblair! | 21:28 |
| jeblair | #status log pypi mirrors have not updated since 2:15 UTC due to issue with pypi.python.org. reported issue, since corrected. mirror updates now in progress. | 21:28 |
| openstackstatus | jeblair: finished logging | 21:28 |
| *** e0ne has quit IRC | 21:28 | |
| fungi | doesn't appear that #python-dev discussions are logged anywhere i can find | 21:32 |
| ianw | jeblair / fungi: should i perhaps attach a large volume, clone pypi to it and run a manual test with 2.0 on trusty? | 21:32 |
| ianw | then, assuming that all works, we could just bring up a xenial host for long-term saftey after that | 21:33 |
| fungi | ianw: sounds like a great idea | 21:33 |
| clarkb | so was the problem upstream stopped letting bandersnatch 1.x function? | 21:33 |
| fungi | that's what i'm wondering, which is why i went hunting for channel logs | 21:33 |
| ianw | clarkb: <EWDurbin> Apologies!! My regex was bad and I should fee bad | 21:34 |
| ianw | so it seem to be totally unintentional | 21:34 |
| clarkb | ah cool | 21:34 |
| fungi | good | 21:34 |
| fungi | thanks ianw | 21:34 |
| clarkb | do we need to be thinking about upgrading real soon now? | 21:34 |
| fungi | seems like we should upgrade soon-ish since we're running on a fork which should in theory no longer be required | 21:35 |
| ianw | fungi: ok, i'll take it as an #action item to show that 2.0 won't destroy things terribly over the next little while, and we can take it from there | 21:37 |
| fungi | ianw: note that copying the current data set means pulling hundreds of gigabytes from afs so could be on the slowish side | 21:37 |
| fungi | unless you were thinking some afs volume cloning operation | 21:37 |
| fungi | but "attach" makes me think you're talking about cinder | 21:38 |
| ianw | yeah, but the alternative seems to be to destroy it an have to re-sync it ... so i think slow and steady wins? | 21:38 |
| fungi | totally, not objecting, just reminding ;) | 21:38 |
| ianw | i was thinking a large cinder volume, copy it, run 2.0 against it, compare and contrast, release volume | 21:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 21:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Rename tox_command_line in docs to tox_extra_args https://review.openstack.org/489758 | 21:41 |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/shade master: Use valid_kwargs decorator in QoS related functions https://review.openstack.org/489759 | 21:43 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: DNM: Override tox requirments with zuul git repos https://review.openstack.org/489761 | 21:45 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Docs: add a glossary https://review.openstack.org/489270 | 21:46 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Docs: add a :default: argument to zuul:attr https://review.openstack.org/489303 | 21:48 |
| cmurphy | ianw: any reason not to +A this one? https://review.openstack.org/#/c/477228/ it's totally separate from whatever we decide re: install_puppet.sh | 21:48 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use zuul attributes in semaphore documentation https://review.openstack.org/489499 | 21:48 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove zuul_url from merger config https://review.openstack.org/489378 | 21:48 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Improve cleanup of test disk accountant https://review.openstack.org/489477 | 21:48 |
| *** jkilpatr_ has quit IRC | 21:49 | |
| *** jamesmcarthur has joined #openstack-infra | 21:51 | |
| *** ijw has quit IRC | 21:52 | |
| *** dizquierdo has quit IRC | 21:52 | |
| *** EricGonczer_ has quit IRC | 21:52 | |
| *** camunoz has quit IRC | 21:55 | |
| *** jamesmcarthur has quit IRC | 21:56 | |
| *** esberglu has quit IRC | 21:58 | |
| *** slaweq has quit IRC | 22:03 | |
| *** slaweq has joined #openstack-infra | 22:03 | |
| *** mikal has quit IRC | 22:04 | |
| ianw | cmurphy: i guess not :) | 22:04 |
| *** thorst_afk has quit IRC | 22:05 | |
| *** mikal has joined #openstack-infra | 22:06 | |
| *** annegentle has quit IRC | 22:06 | |
| cmurphy | ianw: cool, thanks :D | 22:07 |
| *** slaweq has quit IRC | 22:08 | |
| *** jkilpatr has joined #openstack-infra | 22:08 | |
| openstackgerrit | Merged openstack-infra/puppet-openstack_infra_spec_helper master: Move spec_helper_acceptance into helper gem https://review.openstack.org/477228 | 22:08 |
| *** esberglu has joined #openstack-infra | 22:12 | |
| jeblair | releasing the pypi mirror | 22:14 |
| *** gordc has quit IRC | 22:15 | |
| jeblair | done | 22:15 |
| jeblair | and i've released the lock | 22:15 |
| *** bobh has quit IRC | 22:16 | |
| jeblair | #status log pypi mirror manually updated and released | 22:16 |
| openstackstatus | jeblair: finished logging | 22:16 |
| *** esberglu has quit IRC | 22:17 | |
| *** thorst_afk has joined #openstack-infra | 22:19 | |
| *** yamamoto has joined #openstack-infra | 22:23 | |
| *** esberglu has joined #openstack-infra | 22:23 | |
| *** ijw has joined #openstack-infra | 22:23 | |
| *** thorst_afk has quit IRC | 22:24 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Simplify run tox task https://review.openstack.org/487551 | 22:25 |
| *** yamamoto has quit IRC | 22:25 | |
| *** yamamoto has joined #openstack-infra | 22:25 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Simplify github status descriptions https://review.openstack.org/489767 | 22:26 |
| *** hashar has quit IRC | 22:28 | |
| *** Goneri has quit IRC | 22:32 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for missing host mapping bug 1708039 https://review.openstack.org/489771 | 22:33 |
| openstack | bug 1708039 in grenade "gate-grenade-dsvm-neutron-multinode-ubuntu-xenial fails with "No host-to-cell mapping found for selected host"" [Undecided,New] https://launchpad.net/bugs/1708039 | 22:33 |
| *** Goneri has joined #openstack-infra | 22:34 | |
| *** sdague has quit IRC | 22:35 | |
| mnaser | hey folks | 22:40 |
| mnaser | http://zuul.openstack.org/status.json is not loading | 22:40 |
| mnaser | okay, it is loading, very very slowly | 22:40 |
| mnaser | ok, chrome takes a while to load it, but curl is okay. probably client side | 22:41 |
| fungi | mnaser: odds are it's huge nested data and chrome is attempting some pretty rendering | 22:42 |
| mnaser | fungi: ah i see | 22:42 |
| mnaser | maybe i have some json prettify extension | 22:43 |
| fungi | i like to pipe it through https://pypi.org/project/flamel/ | 22:43 |
| *** panda has quit IRC | 22:43 | |
| *** dave-mccowan has joined #openstack-infra | 22:43 | |
| clarkb | mnaser: I think firefox and chrome both have built in prettify for json | 22:43 |
| clarkb | mnaser: not sure if you can turn it off | 22:44 |
| mnaser | oh wow pypi is pretty now | 22:44 |
| fungi | mnaser: oh, that's new pypi. use pypi.python.org if you like the old interface | 22:44 |
| fungi | pypi.org is "warehouse" | 22:45 |
| mnaser | i kinda like this anyways :-P | 22:45 |
| *** annegentle has joined #openstack-infra | 22:45 | |
| fungi | good, since it will eventually replace the old cheeseshop pypi implementation | 22:45 |
| fungi | better than hating it and being forlorn that the version you like is obsolete and getting decommissioned eventually | 22:46 |
| mnaser | typical release cycle, everyone complains then accepts it :p | 22:46 |
| *** panda has joined #openstack-infra | 22:48 | |
| fungi | i think the gate job running for 489677,3 is hung somehow. last entry in the console log: | 22:49 |
| fungi | 2017-08-01 22:20:54.125265 | venv create: /home/jenkins/workspace/gate-infra-manual-docs-ubuntu-xenial/.tox/venv | 22:49 |
| fungi | nearly 30 minutes since it said that, and still nothing | 22:50 |
| *** rbrndt has quit IRC | 22:50 | |
| *** rbrndt has joined #openstack-infra | 22:50 | |
| *** rbrndt has quit IRC | 22:50 | |
| fungi | think i'll hold and jump on that node to see what's happening | 22:50 |
| *** https_GK1wmSU has joined #openstack-infra | 22:52 | |
| *** pbourke has quit IRC | 22:53 | |
| *** https_GK1wmSU has left #openstack-infra | 22:53 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Docs: use zuul:attr in project definition https://review.openstack.org/489774 | 22:55 |
| fungi | i'm at a loss... http://paste.openstack.org/show/617190/ | 22:55 |
| *** pbourke has joined #openstack-infra | 22:55 | |
| fungi | hung doing things with our pypi mirror i guess? | 22:56 |
| *** zeih has joined #openstack-infra | 22:56 | |
| clarkb | what does python2 - setuptools pip wheel do? or is that some mangled get-pip.py process? | 22:56 |
| *** ijw has quit IRC | 22:58 | |
| fungi | my guess is it's something like: tox invokes virtualenv, virtualenv invokes python2 venv, python2 forks and makes up that process name | 22:58 |
| fungi | but yes something to do with bootstrapping through the low-level python venv module | 22:59 |
| fungi | if i run `virtualenv foo` on that same system from the shell, it stops after "Installing setuptools, pip, wheel..." and just sits there | 23:00 |
| *** zeih has quit IRC | 23:01 | |
| clarkb | fungi: try python3 instead? | 23:01 |
| fungi | `virtualenv -p python3 foo` does exactly the same | 23:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add README for tox role https://review.openstack.org/489750 | 23:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 23:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Rename tox_command_line in docs to tox_extra_args https://review.openstack.org/489758 | 23:02 |
| fungi | aha! if i attempt to ssh into mirror.regionone.osic-cloud1.openstack.org it hangs | 23:02 |
| clarkb | I'm assuming that means "it has begun" | 23:03 |
| clarkb | cloudnull: ^ | 23:03 |
| fungi | quite possible, might be we should turn down osic in our nodepool but i'm poking at the nova api now | 23:03 |
| *** annegentle has quit IRC | 23:04 | |
| *** calebb has quit IRC | 23:05 | |
| fungi | aha, it's reachable again now | 23:05 |
| fungi | load average had spiked up to at least 40 | 23:06 |
| *** esberglu has quit IRC | 23:06 | |
| *** pvaneck has quit IRC | 23:06 | |
| *** calebb has joined #openstack-infra | 23:06 | |
| *** slaweq has joined #openstack-infra | 23:07 | |
| fungi | disk anyone? http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=4935&rra_id=all | 23:07 |
| fungi | [Tue Aug 1 23:03:59 2017] kernel BUG at /var/lib/dkms/openafs/1.6.7/build/src/libafs/MODLOAD-3.13.0-96-generic-SP/osi_file.c:53! | 23:08 |
| fungi | looks like starting around 22:23:52 per dmesg, we started getting "ext4_end_bio:317: I/O error -5 writing to inode" for device dm-0 | 23:09 |
| fungi | and then loads of "Buffer I/O error on device dm-0" | 23:09 |
| clarkb | did we run out of inodes or disk? | 23:10 |
| fungi | and eventually lost page writes, unable to read itable blocks... | 23:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Send the root logger to debug and normal logs https://review.openstack.org/489775 | 23:10 |
| fungi | i think the cinder volume went to lunch for a good long while, and afs was keeping its cache there | 23:11 |
| *** slaweq has quit IRC | 23:11 | |
| fungi | hence the eventual openafs driver backtraces | 23:11 |
| *** esberglu has joined #openstack-infra | 23:12 | |
| fungi | i'm going to try to give it a reboot, but it's possible you're right and whatever was providing cinder volumes has been yanked | 23:13 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Stop using the OSIC environment https://review.openstack.org/489776 | 23:16 |
| fungi | if it doesn't come back up in a few minutes or comes up with the cinder volume unusable, i'll approve that ^ | 23:16 |
| fungi | though it seems to have not obeyed my reboot request from the command-line so may need a nova reboot issued | 23:17 |
| smcginnis | fungi: What percent of the pool is 768 servers? | 23:18 |
| fungi | smcginnis: right now, around 42% | 23:18 |
| smcginnis | fungi: Dang | 23:18 |
| smcginnis | Good thing we're past pike-3 I guess. | 23:18 |
| fungi | that was very recently increased from 512 a couple weeks ago and we also have 50 temporarily offline in citynet while they work through a network issue there, so by a couple weeks ago numbers osic was more like 27% of our total | 23:19 |
| clarkb | fungi: I left a comment but it should be fine to merge as is as well | 23:19 |
| fungi | smcginnis: also we have other providers interested in donating, so given we're past feature freeze as you noted i think we'll be fine for a little while anyway | 23:20 |
| smcginnis | fungi: Cool, doesn't sound like we'll be in too bad of shape then. | 23:20 |
| *** nunchuck has quit IRC | 23:21 | |
| fungi | nova reboot --hard has that instance in an error state now too | 23:22 |
| clarkb | fungi: you could try booting a new one | 23:22 |
| clarkb | since its a stateless server | 23:22 |
| clarkb | but that would require dns records to timeout and all that | 23:22 |
| fungi | and reattaching the cinder volume | 23:23 |
| fungi | assuming i can even get it to detach from the error state instance | 23:23 |
| clarkb | you could use a new volume too | 23:23 |
| clarkb | since its purely cache | 23:24 |
| clarkb | but chances are if its dead then none of that will work either | 23:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Wrap handle_keys with debug statement https://review.openstack.org/489777 | 23:24 |
| fungi | for some strange reason gertty stuck your comment for that highlighted block of text in an unrelated location in the diff | 23:24 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Stop using the OSIC environment https://review.openstack.org/489776 | 23:25 |
| fungi | clarkb: ^ | 23:25 |
| fungi | addressed your comment | 23:25 |
| openstackgerrit | Mike Perez proposed openstack-infra/statusbot master: Implement #thanks command in statusbot https://review.openstack.org/473534 | 23:26 |
| clarkb | fungi: +2 | 23:27 |
| fungi | i'm counting the odds i'll need to manually submit that patch to merge fairly high | 23:27 |
| clarkb | :/ | 23:27 |
| fungi | i'm going to go ahead and just manually apply it on nodepool.o.o and put the server in the emergency disable list | 23:30 |
| *** hongbin has quit IRC | 23:30 | |
| *** jaypipes has quit IRC | 23:31 | |
| *** jascott1 has quit IRC | 23:32 | |
| fungi | manually deleting the remaining ready nodes in osic now | 23:33 |
| fungi | at least in this state we won't have negative impact on new jobs while fiddling with getting the mirror back online | 23:34 |
| clarkb | sounds good | 23:34 |
| pabelanger | fungi: pause: true in images will stop uploads | 23:35 |
| pabelanger | over deleting all images in nodepool.yaml | 23:35 |
| *** jamesden_ has joined #openstack-infra | 23:35 | |
| *** jamesdenton has quit IRC | 23:35 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix webapp path parsing https://review.openstack.org/489779 | 23:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Try handling keys before removing tenant name https://review.openstack.org/489780 | 23:38 |
| fungi | does max-servers: -1 no longer have any meaning separate from 0? | 23:38 |
| *** jamesden_ has quit IRC | 23:39 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix webapp path parsing https://review.openstack.org/489779 | 23:40 |
| clarkb | fungi: correct | 23:41 |
| *** jamesdenton has joined #openstack-infra | 23:41 | |
| fungi | pabelanger: is pause: true preferable over emptying the images list then? | 23:41 |
| clarkb | pabelanger: the problem with not deleting them now is that if the cloud is gone then we'll have to delete them all by hand I think or at least I had to hand edit zk records last time | 23:41 |
| clarkb | fungi: I think pause:true is good if you expect the cloud to come back | 23:41 |
| jlvillal | Is the gate broken? And if so, any chance of an announcement about it? Or has it been fixed already? | 23:41 |
| * jlvillal is noticing a lot of failures | 23:42 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix webapp path parsing https://review.openstack.org/489779 | 23:42 |
| fungi | jlvillal: we just took osic out of service. what symptoms are you seeing? so far i had one obviously impacted job but not sure what the expected fallout was | 23:42 |
| jlvillal | fungi: Jobs that failed on OSIC | 23:42 |
| jlvillal | fungi: Couldn't install pypi packages | 23:43 |
| fungi | that sounds right. i've just deleted the remaining nodes in a used state in osic so that zuul should hopefully requeue them instead of incurring more failures | 23:43 |
| jlvillal | fungi: Maybe apt-get packages, not pypi | 23:43 |
| jlvillal | fungi: Okay thanks. | 23:43 |
| *** gouthamr has quit IRC | 23:43 | |
| jeblair | status notice osic nodes have been removed from nodepool due to a problem with the mirror host there. please recheck any jobs with failures installing packages. | 23:44 |
| jeblair | how's that look? | 23:44 |
| fungi | jeblair: that's close enough to what i'd just about finished writing | 23:45 |
| fungi | could note the start time | 23:45 |
| fungi | ~22:20 utc | 23:45 |
| pabelanger | clarkb: fungi: ah, we don't expect OSIC to come back, ya removal is better then | 23:45 |
| jeblair | #status notice osic nodes have been removed from nodepool due to a problem with the mirror host beginning around 22:20 UTC. please recheck any jobs with failures installing packages. | 23:45 |
| openstackstatus | jeblair: sending notice | 23:45 |
| fungi | thanks jeblair! | 23:46 |
| jeblair | np | 23:46 |
| fungi | only one node left in osic now, and it's the held one i was using as a test client | 23:47 |
| -openstackstatus- NOTICE: osic nodes have been removed from nodepool due to a problem with the mirror host beginning around 22:20 UTC. please recheck any jobs with failures installing packages. | 23:47 | |
| * jlvillal grumbles that someone added a >65 character subject check to Gerrit | 23:47 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix webapp path parsing https://review.openstack.org/489779 | 23:48 |
| jlvillal | Since it means his cherry-pick of an old commit won't be accepted | 23:48 |
| fungi | jlvillal: what repo? | 23:48 |
| jlvillal | fungi: openstack/ironic-tempest-plugin | 23:48 |
| jeblair | yeah, pretty sure we settled that one 6 years ago | 23:48 |
| jlvillal | fungi: I'm trying to port all of our ironic_tempest_plugin/ directory code to that repo. | 23:48 |
| openstackstatus | jeblair: finished sending notice | 23:49 |
| jlvillal | remote: (W) 3cb8459: commit subject >65 characters; use shorter first paragraph is error I got | 23:49 |
| cloudnull | rutro! | 23:49 |
| fungi | cloudnull: i'm still troubleshooting, but what are the odds that whatever was serving cinder volumes just went offline around 1.5 hours ago? | 23:50 |
| cloudnull | IDk . let me go find out :) | 23:51 |
| jlvillal | fungi: Is the 65 character limit something that can be removed from: https://review.openstack.org/#/admin/projects/openstack/ironic-tempest-plugin ? | 23:51 |
| cloudnull | I know the demise is soon, but i didn | 23:51 |
| cloudnull | 't think it was today | 23:51 |
| * cloudnull goes to see if someone did something bad | 23:51 | |
| jeblair | jlvillal, fungi: apparently we have the commit-message-length-validator gerrit plugin installed, and it defaults to 65 | 23:52 |
| fungi | jlvillal: i thought it was configurable per project in gerrit and off by default, but need to look deeper | 23:52 |
| jeblair | jlvillal, fungi: and apparently it's system-wide config | 23:52 |
| fungi | huh, i didn't even realize that was there | 23:52 |
| jeblair | did it sneak in when we switched to plugins? | 23:52 |
| fungi | i wonder if it's in the default set | 23:52 |
| jlvillal | jeblair: fungi Thanks. I guess it wasn't there on December 9th, 2016. | 23:52 |
| cloudnull | looks like the instance "mirror.regionone.osic-cloud1.openstack.org" is in error | 23:52 |
| *** ijw has joined #openstack-infra | 23:52 | |
| jlvillal | When the patch I am cherry-picking was first posted. | 23:53 |
| fungi | i don't have a personal grudge against long git subjects, i just set my commit message editor to warn me when i get over 50 characters there | 23:53 |
| * jlvillal does the same | 23:53 | |
| fungi | cloudnull: yeah, i tried to reboot it after i couldn't get the cinder volume attached to it back online | 23:53 |
| clarkb | the way we install plugins is to grab all the defaults then add in melody and whatever hte others are | 23:53 |
| clarkb | pretty sure we don't explicitly install it so must come from defaults | 23:53 |
| cloudnull | im jumping into the env now | 23:53 |
| *** markvoelker has quit IRC | 23:53 | |
| fungi | cloudnull: and then the instance went into error state on a nova reboot --hard (after ignoring attempts to reboot from the command line and attempts to nova reboot without --hard) | 23:54 |
| fungi | cloudnull: so wondering if whatever happened with the volume resulted in the instance going into error when it couldn't reconnect the volume at boot | 23:55 |
| fungi | worst case i was going to boot a replacement instance and create a replacement volume... there's nothing especially stateful there it's just cache space | 23:55 |
| *** jascott1 has joined #openstack-infra | 23:56 | |
| jeblair | jlvillal, fungi, clarkb: https://review.openstack.org/316197 is relevant | 23:56 |
| fungi | interesting that it's still complaining at 65 | 23:57 |
| jlvillal | jeblair: Maybe need to add for maxSubjectLength https://review.openstack.org/plugins/commit-message-length-validator/Documentation/config.html | 23:57 |
| cloudnull | fungi: quite possibily | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!