| *** bobh has quit IRC | 00:04 | |
| *** bobh has joined #openstack-infra | 00:06 | |
| *** bobh has quit IRC | 00:11 | |
| *** bobh has joined #openstack-infra | 00:22 | |
| *** longkb has joined #openstack-infra | 00:23 | |
| *** jamesmcarthur has quit IRC | 00:24 | |
| *** bobh has quit IRC | 00:26 | |
| *** bobh_ has joined #openstack-infra | 00:26 | |
| *** bobh_ has quit IRC | 00:31 | |
| *** bobh has joined #openstack-infra | 00:36 | |
| *** rkukura has quit IRC | 00:43 | |
| *** fried_rolls1 has joined #openstack-infra | 00:48 | |
| *** bobh has quit IRC | 00:49 | |
| *** fried_rolls has quit IRC | 00:50 | |
| *** fried_rolls1 is now known as fried_rolls | 00:50 | |
| pabelanger | dhellmann: looks like testr is missing: http://logs.openstack.org/53/604653/4/check/tox-cover/fb44cc8/job-output.txt.gz#_2018-09-24_00_01_51_323616 | 00:55 |
|---|---|---|
| pabelanger | looks like runtests command wasn't updated properly | 00:56 |
| *** fried_rolls is now known as efried | 01:07 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/storyboard master: Update coverage environment https://review.openstack.org/604667 | 01:09 |
| AJaeger | dhellmann: let's try this ^ | 01:09 |
| *** imacdonn has quit IRC | 01:17 | |
| *** imacdonn has joined #openstack-infra | 01:17 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: lay down gentoo use file for openvswitch https://review.openstack.org/604655 | 01:18 |
| AJaeger | config-core, karbor and cinder are ready with python3-first - please review https://review.openstack.org/#/c/594834/ and https://review.openstack.org/#/c/601127 | 01:20 |
| *** mrsoul has joined #openstack-infra | 01:22 | |
| *** rcernin has quit IRC | 01:40 | |
| *** rcernin has joined #openstack-infra | 01:40 | |
| prometheanfire | AJaeger: we need a new gentoo image first don't we? | 01:47 |
| prometheanfire | maybe not, since it's ansible | 01:47 |
| *** weshay is now known as weshay_afk | 01:49 | |
| *** hongbin has joined #openstack-infra | 01:49 | |
| *** Bhujay has joined #openstack-infra | 02:04 | |
| *** Bhujay has quit IRC | 02:12 | |
| prometheanfire | AJaeger: I'm guessing lineinfile doesn't create the file if it doesn't already exist? | 02:18 |
| prometheanfire | lesigh | 02:18 |
| prometheanfire | Destination /etc/portage/package.use/ovs does not exist ! | 02:18 |
| openstackgerrit | Matthew Thode proposed openstack-infra/zuul-jobs master: create the pacage.use file... https://review.openstack.org/604671 | 02:20 |
| prometheanfire | yet another round... | 02:20 |
| *** jamesmcarthur has joined #openstack-infra | 02:24 | |
| *** pcrews has joined #openstack-infra | 02:26 | |
| *** jamesmcarthur has quit IRC | 02:28 | |
| *** imacdonn has quit IRC | 02:35 | |
| *** psachin has joined #openstack-infra | 02:40 | |
| *** yamamoto has quit IRC | 02:42 | |
| *** jamesmcarthur has joined #openstack-infra | 02:43 | |
| *** bobh has joined #openstack-infra | 02:43 | |
| *** imacdonn has joined #openstack-infra | 02:50 | |
| *** jamesmcarthur has quit IRC | 02:54 | |
| *** annp has joined #openstack-infra | 03:07 | |
| *** bobh has quit IRC | 03:15 | |
| *** vivsoni has joined #openstack-infra | 03:25 | |
| *** ijw has joined #openstack-infra | 03:28 | |
| *** hongbin has quit IRC | 03:39 | |
| *** psachin has quit IRC | 03:48 | |
| *** udesale has joined #openstack-infra | 03:50 | |
| *** icey has quit IRC | 03:56 | |
| *** icey has joined #openstack-infra | 03:57 | |
| *** psachin has joined #openstack-infra | 04:00 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/storyboard master: Update coverage environment https://review.openstack.org/604667 | 04:05 |
| AJaeger | prometheanfire: you can test with depends-on... | 04:06 |
| *** yamamoto has joined #openstack-infra | 04:08 | |
| *** vivsoni has quit IRC | 04:10 | |
| *** _ari_ has quit IRC | 04:11 | |
| prometheanfire | ok | 04:16 |
| openstackgerrit | Matthew Thode proposed openstack-infra/openstack-zuul-jobs master: add Gentoo jobs and fix install test https://review.openstack.org/602439 | 04:18 |
| *** ykarel has joined #openstack-infra | 04:22 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: add support for generic build resource https://review.openstack.org/570668 | 04:29 |
| *** vivsoni has joined #openstack-infra | 04:30 | |
| *** vivsoni has quit IRC | 04:37 | |
| *** vivsoni has joined #openstack-infra | 04:37 | |
| *** vivsoni has quit IRC | 04:43 | |
| *** vivsoni has joined #openstack-infra | 04:46 | |
| *** jamesmcarthur has joined #openstack-infra | 04:50 | |
| *** jamesmcarthur has quit IRC | 04:54 | |
| *** Bhujay has joined #openstack-infra | 04:58 | |
| *** ramishra has joined #openstack-infra | 04:59 | |
| *** rkukura has joined #openstack-infra | 05:00 | |
| *** spsurya has joined #openstack-infra | 05:02 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Kubernetes driver https://review.openstack.org/535557 | 05:09 |
| prometheanfire | lol, neutron type errror | 05:14 |
| openstackgerrit | Matthew Thode proposed openstack-infra/zuul-jobs master: use find instead of ls to list interfaces https://review.openstack.org/604677 | 05:22 |
| openstackgerrit | Matthew Thode proposed openstack-infra/openstack-zuul-jobs master: add Gentoo jobs and fix install test https://review.openstack.org/602439 | 05:23 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Support node caching in the nodeIterator https://review.openstack.org/604648 | 05:25 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Use node cache in quota calculations https://review.openstack.org/604649 | 05:25 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Cache iterations over ready nodes https://review.openstack.org/604650 | 05:25 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Use cache when counting poolnodes https://review.openstack.org/604651 | 05:25 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Use cache when deleting oldest unused nodes https://review.openstack.org/604652 | 05:25 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix race in test_launchNode_delete_error https://review.openstack.org/604678 | 05:25 |
| *** yamamoto has quit IRC | 05:30 | |
| *** dave-mccowan has joined #openstack-infra | 05:36 | |
| *** jtomasek has joined #openstack-infra | 05:37 | |
| *** yamamoto has joined #openstack-infra | 05:39 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Support node caching in the nodeIterator https://review.openstack.org/604648 | 05:40 |
| *** dayou has quit IRC | 05:43 | |
| *** dayou has joined #openstack-infra | 05:44 | |
| *** quique|rover|off is now known as quiquell|rover | 05:45 | |
| *** jiapei has joined #openstack-infra | 05:46 | |
| *** Bhujay has quit IRC | 05:53 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move storyboard cover job in-repo https://review.openstack.org/604679 | 05:57 |
| *** Bhujay has joined #openstack-infra | 05:59 | |
| *** dpawlik has joined #openstack-infra | 06:03 | |
| *** ajo has joined #openstack-infra | 06:03 | |
| *** pcaruana has joined #openstack-infra | 06:05 | |
| *** apetrich has joined #openstack-infra | 06:05 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add labels page https://review.openstack.org/604682 | 06:13 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add nodes page https://review.openstack.org/604683 | 06:13 |
| AJaeger | frickler, ianw, karbor and cinder are ready with python3-first - could either of you review https://review.openstack.org/#/c/594834/ and https://review.openstack.org/#/c/601127 , please? | 06:18 |
| AJaeger | and also https://review.openstack.org/#/c/604529/ and https://review.openstack.org/604608 and https://review.openstack.org/601946, please | 06:19 |
| prometheanfire | AJaeger: looks like 'Validate ipv4 private firewall configuration' is failing, have any insight? http://logs.openstack.org/39/602439/7/check/openstack-infra-multinode-integration-gentoo-17-0-systemd/eeaf2c8/ara-report/reports/3eb0c0c2-a3ed-4787-8fd4-de5d388fe78f.html | 06:21 |
| * prometheanfire is running a recheck on it just in case | 06:22 | |
| AJaeger | prometheanfire: sorry, no ;( | 06:23 |
| *** e0ne has joined #openstack-infra | 06:23 | |
| prometheanfire | hopefully recheck fixes it, otherwise I'll have to do debug statements I think | 06:23 |
| prometheanfire | this looks wrong though http://logs.openstack.org/39/602439/7/check/openstack-infra-multinode-integration-gentoo-17-0-systemd/eeaf2c8/job-output.txt.gz#_2018-09-24_05_59_48_770056 | 06:24 |
| *** longkb has quit IRC | 06:25 | |
| frickler | prometheanfire: seems the firewall rules generation is still skipped completely http://logs.openstack.org/39/602439/7/check/openstack-infra-multinode-integration-gentoo-17-0-systemd/eeaf2c8/job-output.txt.gz#_2018-09-24_05_59_46_181335 | 06:25 |
| *** longkb has joined #openstack-infra | 06:25 | |
| prometheanfire | frickler: I think that's a limitation of ansible itself... | 06:28 |
| frickler | prometheanfire: not sure what you mean by that. to me it looks like the rules aren't getting installed which then causes the checks whether they have been installed to fail | 06:30 |
| prometheanfire | 2018-09-24 05:59:46.707012 | WARNING: Gentoo is not supported by this role yet. The execution of the job will continue without persisting iptables rules. | 06:30 |
| prometheanfire | where does that role exist? | 06:30 |
| frickler | let me check | 06:31 |
| prometheanfire | oh.. found it lol | 06:31 |
| prometheanfire | was looking in files, not for files | 06:31 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Support node caching in the nodeIterator https://review.openstack.org/604648 | 06:31 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Use node cache in quota calculations https://review.openstack.org/604649 | 06:31 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Cache iterations over ready nodes https://review.openstack.org/604650 | 06:31 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Use cache when counting poolnodes https://review.openstack.org/604651 | 06:31 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Use cache when deleting oldest unused nodes https://review.openstack.org/604652 | 06:31 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Use zuul-sphinx for configuration layout https://review.openstack.org/604274 | 06:33 |
| *** chandankumar has joined #openstack-infra | 06:33 | |
| *** chandankumar is now known as chkumar|ruck | 06:34 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: log-inventory: remove inventory_file usage https://review.openstack.org/578238 | 06:34 |
| prometheanfire | frickler: should I be saving rules before shutdown or letting ansible handle all rule generation? | 06:36 |
| *** ijw has quit IRC | 06:36 | |
| frickler | prometheanfire: iiuc what the test does is flush all the iptables rules and then check that they get created again after the iptables service is restarted | 06:38 |
| frickler | see http://logs.openstack.org/39/602439/7/check/openstack-infra-multinode-integration-gentoo-17-0-systemd/eeaf2c8/job-output.txt.gz#_2018-09-24_05_59_47_159963 ff. | 06:38 |
| openstackgerrit | Matthew Thode proposed openstack-infra/zuul-jobs master: Add Gentoo iptables handling https://review.openstack.org/604688 | 06:38 |
| frickler | not sure how to best achieve that in gentoo, but I think it needs to happen on the os side and not in ansible | 06:38 |
| prometheanfire | frickler: yep, I'm not saving it on shutdown ^ | 06:38 |
| AJaeger | prometheanfire: add a depends-on again to test it, please | 06:39 |
| prometheanfire | frickler: as long as nothing overwrites the file it'll restore to it | 06:39 |
| prometheanfire | yep | 06:39 |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for cinder repositories https://review.openstack.org/601127 | 06:39 |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for karbor repositories https://review.openstack.org/594834 | 06:39 |
| frickler | prometheanfire: o.k., that looks reasonable to me, let's see how the test goes | 06:39 |
| prometheanfire | this'll be my last one before slee | 06:39 |
| prometheanfire | this'll be my last one before sleep | 06:39 |
| openstackgerrit | Matthew Thode proposed openstack-infra/openstack-zuul-jobs master: add Gentoo jobs and fix install test https://review.openstack.org/602439 | 06:40 |
| *** e0ne_ has joined #openstack-infra | 06:41 | |
| prometheanfire | ok, nn | 06:42 |
| AJaeger | prometheanfire: good night! | 06:43 |
| *** e0ne has quit IRC | 06:44 | |
| openstackgerrit | Merged openstack-infra/project-config master: Avoid double root-marker https://review.openstack.org/601946 | 06:54 |
| *** adrianreza has quit IRC | 06:56 | |
| *** quiquell|rover is now known as quique|rover|brb | 06:58 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/zuul-jobs master: use find instead of ls to list interfaces https://review.openstack.org/604677 | 06:58 |
| prometheanfire | ok, now I sleep | 07:00 |
| *** ginopc has joined #openstack-infra | 07:03 | |
| *** ijw has joined #openstack-infra | 07:06 | |
| *** rcernin has quit IRC | 07:06 | |
| *** ijw has quit IRC | 07:10 | |
| *** gfidente has joined #openstack-infra | 07:13 | |
| *** aojea has joined #openstack-infra | 07:13 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: create the pacage.use file... https://review.openstack.org/604671 | 07:17 |
| *** holser_ has joined #openstack-infra | 07:23 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Cache node iterations of DeletedNodeWorker https://review.openstack.org/604691 | 07:23 |
| *** shardy has joined #openstack-infra | 07:27 | |
| *** ykarel is now known as ykarel|lunch | 07:27 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Cache node iterations in cleanup workers https://review.openstack.org/604691 | 07:29 |
| *** quique|rover|brb is now known as quiquell|rover | 07:30 | |
| *** gfidente has quit IRC | 07:31 | |
| *** gfidente has joined #openstack-infra | 07:36 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Allow debootstrap to clenaup without a kernel https://review.openstack.org/604692 | 07:37 |
| *** roman_g has joined #openstack-infra | 07:43 | |
| *** dtantsur|afk is now known as dtantsur | 07:46 | |
| AJaeger | config-core, https://review.openstack.org/604526 is ready (retire fuxi) and https://review.openstack.org/604529 (retire charm-ceph) needs second +2 - please review | 07:47 |
| *** jpena|off is now known as jpena | 07:49 | |
| *** alexchadin has joined #openstack-infra | 07:51 | |
| *** tosky has joined #openstack-infra | 07:51 | |
| ianw | AJaeger: first one is WIP but i assume it's ready as all deps are | 07:52 |
| *** jpich has joined #openstack-infra | 07:56 | |
| openstackgerrit | Merged openstack-infra/project-config master: Retire charm-ceph (step 4) https://review.openstack.org/604529 | 07:57 |
| *** longkb has quit IRC | 08:03 | |
| *** hashar has joined #openstack-infra | 08:06 | |
| *** jcoufal has joined #openstack-infra | 08:07 | |
| *** ginopc has quit IRC | 08:09 | |
| *** alexchadin has quit IRC | 08:14 | |
| AJaeger | ianw: Oh, yes - will remove now. Thanks | 08:17 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Doc: executor operations - explain jobs will be restarted at restart https://review.openstack.org/603136 | 08:18 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Doc: executor operations - explain jobs will be restarted at restart https://review.openstack.org/603136 | 08:21 |
| AJaeger | thanks frickler and ianw for reviewing! | 08:22 |
| *** openstackgerrit has quit IRC | 08:22 | |
| *** openstackgerrit has joined #openstack-infra | 08:30 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove migrated keystone swift jobs https://review.openstack.org/603199 | 08:30 |
| openstackgerrit | Merged openstack-infra/project-config master: Retire fuxi projects (step 4) https://review.openstack.org/604526 | 08:30 |
| AJaeger | dmellado: ^ | 08:31 |
| dmellado | AJaeger: awesome, I was just checking that on the gerrit as well ;) | 08:31 |
| AJaeger | dmellado: https://review.openstack.org/#/c/604527/ (the governance change) will be merged in a week if there are no objections. So, our work should be done here ;) | 08:32 |
| dmellado | \o/ | 08:33 |
| *** janki has joined #openstack-infra | 08:33 | |
| *** janki has quit IRC | 08:34 | |
| *** janki has joined #openstack-infra | 08:34 | |
| *** holser_ has quit IRC | 08:35 | |
| *** holser_ has joined #openstack-infra | 08:36 | |
| *** holser_ has left #openstack-infra | 08:36 | |
| openstackgerrit | Merged openstack-infra/python-storyboardclient master: Update the URL in HACKING.rst https://review.openstack.org/604639 | 08:37 |
| *** ykarel|lunch is now known as ykarel | 08:38 | |
| *** Bhujay has quit IRC | 08:38 | |
| *** finucannot is now known as stephenfin | 08:38 | |
| *** rossella_s has quit IRC | 08:41 | |
| *** alexchadin has joined #openstack-infra | 08:42 | |
| *** derekh has joined #openstack-infra | 08:44 | |
| *** yamamoto has quit IRC | 08:45 | |
| *** jamesmcarthur has joined #openstack-infra | 08:50 | |
| *** jamesmcarthur has quit IRC | 08:55 | |
| openstackgerrit | Antonio Ojea proposed openstack-infra/openstack-zuul-jobs master: Add openSUSE Leap 15.0 job for platform testing https://review.openstack.org/604563 | 09:00 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Allow debootstrap to clenaup without a kernel https://review.openstack.org/604692 | 09:03 |
| *** openstackgerrit has quit IRC | 09:05 | |
| *** gfidente is now known as gfidente|afk | 09:05 | |
| *** jcoufal has quit IRC | 09:07 | |
| *** kukacz has quit IRC | 09:10 | |
| *** jaosorior has quit IRC | 09:11 | |
| *** kukacz has joined #openstack-infra | 09:12 | |
| *** openstackgerrit has joined #openstack-infra | 09:13 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove most job entries for microversion-parse https://review.openstack.org/599453 | 09:13 |
| *** rpittau has joined #openstack-infra | 09:20 | |
| *** pbourke has joined #openstack-infra | 09:21 | |
| *** ianychoi_ has joined #openstack-infra | 09:30 | |
| *** Bhujay has joined #openstack-infra | 09:32 | |
| *** ianychoi has quit IRC | 09:34 | |
| *** adrianreza has joined #openstack-infra | 09:36 | |
| *** Bhujay has quit IRC | 09:37 | |
| *** e0ne_ has quit IRC | 09:38 | |
| *** e0ne has joined #openstack-infra | 09:38 | |
| *** yamamoto has joined #openstack-infra | 09:40 | |
| *** hrw has quit IRC | 09:44 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Update coverage environment https://review.openstack.org/604667 | 09:56 |
| dtantsur | hi all! does anyone know why https://docs.openstack.org/metalsmith/ does not redirect to https://docs.openstack.org/metalsmith/latest/ ? | 09:57 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/storyboard master: skip some alembic migrations for sqlite https://review.openstack.org/604653 | 09:57 |
| AJaeger | dtantsur: is metalsmith listed in openstack-manuals? | 10:00 |
| dtantsur | AJaeger: unlikely, but nor are many other small projects, I suspect | 10:01 |
| AJaeger | dtantsur: best to follow https://docs.openstack.org/doc-contrib-guide/doc-index.html | 10:01 |
| *** Emine has joined #openstack-infra | 10:01 | |
| dtantsur | AJaeger: oh, are all projects added there manually? | 10:02 |
| AJaeger | dtantsur: the latest is generated automatically from governance repo, and metalsmith was only recently added to governance. Might be we did not publish from openstack-manuals since then... | 10:03 |
| AJaeger | dtantsur: so, two things: To add to an index page, send change for openstack-manuals | 10:03 |
| AJaeger | dtantsur: and to create latest, wait until next merge of teh repo, we go over governance list of projects automatically | 10:04 |
| dtantsur | ah, okay. I think the latter is enough | 10:04 |
| AJaeger | dtantsur: you can add metalsmith to the "Bare Metal subprojects" list on https://docs.openstack.org/rocky/projects.html as well | 10:05 |
| AJaeger | (for master -> stein release) | 10:05 |
| AJaeger | bbl | 10:06 |
| dtantsur | stein is not a thing yet | 10:06 |
| AJaeger | dtantsur: /latest will be stein | 10:07 |
| dtantsur | AJaeger: yeah, but you said it's auto-generated, no? | 10:07 |
| *** e0ne has quit IRC | 10:08 | |
| *** janki is now known as janki|lunch | 10:14 | |
| *** jaosorior has joined #openstack-infra | 10:22 | |
| *** ssbarnea|bkp has joined #openstack-infra | 10:23 | |
| *** jiapei has quit IRC | 10:23 | |
| *** e0ne has joined #openstack-infra | 10:27 | |
| *** e0ne has quit IRC | 10:40 | |
| *** Bhujay has joined #openstack-infra | 10:40 | |
| AJaeger | dtantsur: I mean https://docs.openstack.org/latest/projects.html -> in prepartion for stein | 10:46 |
| *** e0ne has joined #openstack-infra | 10:49 | |
| slaweq | AJaeger: hi | 10:49 |
| AJaeger | hi slaweq... | 10:49 |
| slaweq | AJaeger: can You maybe put some job for autohold for me? | 10:49 |
| slaweq | AJaeger: I need to debug once again what I still miss in job neutron-tempest-plugin-dvr-multinode-scenario-zuulv3 that it is still not working properly | 10:50 |
| slaweq | it's in neutron-tempest-plugin repo | 10:50 |
| AJaeger | slaweq: no, cannot - you need an adin for that. Please ask "infra-root" - and feel free to leave a message here in general | 10:50 |
| slaweq | patch https://review.openstack.org/#/c/578796/ | 10:50 |
| AJaeger | infra-root, can you help slaweq, please? ^ | 10:50 |
| slaweq | ok, sorry AJaeger, I though that maybe You can also do that :) | 10:51 |
| AJaeger | slaweq: no worries | 10:52 |
| *** jcoufal has joined #openstack-infra | 10:56 | |
| *** e0ne has quit IRC | 10:57 | |
| frickler | slaweq: autohold set up, which ssh key should I use once the job finishes? | 11:00 |
| *** Bhujay has quit IRC | 11:02 | |
| *** Bhujay has joined #openstack-infra | 11:05 | |
| *** Bhujay has quit IRC | 11:06 | |
| *** Bhujay has joined #openstack-infra | 11:07 | |
| *** Bhujay has quit IRC | 11:08 | |
| *** Bhujay has joined #openstack-infra | 11:08 | |
| *** Bhujay has quit IRC | 11:09 | |
| *** janki|lunch is now known as janki | 11:10 | |
| *** Bhujay has joined #openstack-infra | 11:10 | |
| *** jpena is now known as jpena|lunch | 11:10 | |
| *** Bhujay has quit IRC | 11:11 | |
| *** Bhujay has joined #openstack-infra | 11:11 | |
| openstackgerrit | Merged openstack-dev/pbr master: Fix typo in contribution instructions. https://review.openstack.org/601281 | 11:14 |
| slaweq | frickler: thx, I just sent You my pub key in priv | 11:18 |
| *** alexchadin has quit IRC | 11:20 | |
| *** udesale has quit IRC | 11:25 | |
| *** aidin has joined #openstack-infra | 11:36 | |
| *** quiquell|rover is now known as quique|rover|brb | 11:40 | |
| *** ansmith has quit IRC | 11:43 | |
| *** quique|rover|brb is now known as quiquell|rover | 11:45 | |
| *** rh-jelabarre has joined #openstack-infra | 11:46 | |
| *** _ari_ has joined #openstack-infra | 11:56 | |
| *** alexchadin has joined #openstack-infra | 11:59 | |
| *** alexchadin has quit IRC | 12:05 | |
| *** gfidente|afk is now known as gfidente | 12:06 | |
| *** alexchadin has joined #openstack-infra | 12:06 | |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Add an option to skip generating RELEASENOTES.rst https://review.openstack.org/587204 | 12:06 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: docs: Add docs for reno integration https://review.openstack.org/604754 | 12:06 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Remove my_ip from generated wsgi script https://review.openstack.org/601325 | 12:07 |
| *** e0ne has joined #openstack-infra | 12:10 | |
| *** boden has joined #openstack-infra | 12:11 | |
| *** trown|outtypewww is now known as trown | 12:11 | |
| *** jamesmcarthur has joined #openstack-infra | 12:11 | |
| *** psachin has quit IRC | 12:13 | |
| *** jpena|lunch is now known as jpena | 12:14 | |
| *** graphene has joined #openstack-infra | 12:15 | |
| *** Bhujay has quit IRC | 12:15 | |
| *** tpsilva has joined #openstack-infra | 12:16 | |
| *** Bhujay has joined #openstack-infra | 12:16 | |
| *** rfolco has joined #openstack-infra | 12:17 | |
| *** aidin has quit IRC | 12:18 | |
| mordred | AJaeger: if you have a second, I got a failure on an sdk patch that's similar to ones I've been getting a bunch recently: https://review.openstack.org/#/c/604628 | 12:21 |
| mordred | AJaeger: specifically you can see openstack.tests.functional.network.v2.test_trunk.TestTrunk.test_subports failed with "StringException" - which usually means it was timed out | 12:22 |
| AJaeger | ;( | 12:22 |
| mordred | I have timeout set to 9000 http://git.openstack.org/cgit/openstack/openstacksdk/tree/.zuul.yaml#n59 | 12:23 |
| mordred | and the indiviaul test timeout set to 90: http://git.openstack.org/cgit/openstack/openstacksdk/tree/tox.ini#n34 | 12:24 |
| AJaeger | looking at http://logs.openstack.org/28/604628/2/check/openstacksdk-functional-devstack-tips/bfabb8e/job-output.txt.gz#_2018-09-23_18_24_01_666698 | 12:24 |
| AJaeger | and the killed 2 lines below, I see 109 seconds | 12:25 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: rewrite interface in react https://review.openstack.org/591604 | 12:25 |
| AJaeger | so, that looks like your individual test timeout of 90 - doesn't it? | 12:25 |
| mordred | AJaeger: oh! so it does | 12:25 |
| AJaeger | mordred: sorry, no expert in those... | 12:25 |
| AJaeger | mordred: please double check! | 12:25 |
| mordred | yeah - I think you're right ... thanks, that's a good place to look | 12:26 |
| mordred | also - I have a lot of those "resetting dropped connection" - I might also need to look in to what I need to do to not have keepalives going | 12:26 |
| AJaeger | config-core, could you review https://review.openstack.org/604679 and https://review.openstack.org/604608, please? | 12:26 |
| AJaeger | mordred: yes, worth doing | 12:27 |
| *** jamesmcarthur has quit IRC | 12:27 | |
| *** jamesmcarthur has joined #openstack-infra | 12:28 | |
| *** aidin has joined #openstack-infra | 12:30 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: rewrite interface in react https://review.openstack.org/591604 | 12:33 |
| *** jamesmcarthur has quit IRC | 12:33 | |
| dhellmann | AJaeger , pabelanger : thanks! | 12:33 |
| *** sambetts has joined #openstack-infra | 12:36 | |
| *** kgiusti has joined #openstack-infra | 12:38 | |
| *** Bhujay has quit IRC | 12:39 | |
| *** Bhujay has joined #openstack-infra | 12:40 | |
| *** yumiriam has joined #openstack-infra | 12:40 | |
| *** agopi has quit IRC | 12:41 | |
| *** Bhujay has quit IRC | 12:41 | |
| *** Bhujay has joined #openstack-infra | 12:41 | |
| *** jaosorior has quit IRC | 12:42 | |
| *** jcoufal has quit IRC | 12:43 | |
| *** alexchadin has quit IRC | 12:43 | |
| *** jaosorior has joined #openstack-infra | 12:44 | |
| *** udesale has joined #openstack-infra | 12:46 | |
| *** jiapei has joined #openstack-infra | 12:47 | |
| *** jamesmcarthur has joined #openstack-infra | 12:51 | |
| *** yumiriam has quit IRC | 12:52 | |
| *** alexchadin has joined #openstack-infra | 12:53 | |
| *** icey3 has joined #openstack-infra | 12:58 | |
| *** jrist has joined #openstack-infra | 12:59 | |
| *** icey3 has quit IRC | 13:01 | |
| *** ansmith has joined #openstack-infra | 13:02 | |
| *** bobh has joined #openstack-infra | 13:02 | |
| *** agopi has joined #openstack-infra | 13:04 | |
| *** yamamoto has quit IRC | 13:07 | |
| *** yamamoto has joined #openstack-infra | 13:08 | |
| *** mriedem has joined #openstack-infra | 13:09 | |
| mriedem | does putting a project in required-projects in a job definition make zuul-cloner clone that repo before devstack/grenade ever run? | 13:13 |
| mriedem | like http://logs.openstack.org/54/604454/4/check/neutron-grenade/9883195/job-output.txt.gz#_2018-09-22_19_12_52_391704 | 13:13 |
| *** smarcet has joined #openstack-infra | 13:13 | |
| AJaeger | mriedem: zuul-cloner is Zuul v2 and needs a list of repos to clone. It can only clone repos in required-projects. | 13:14 |
| mriedem | b/c i'm getting this when trying to git clone openstack/placement in a grenade run http://logs.openstack.org/54/604454/4/check/neutron-grenade/9883195/logs/grenade.sh.txt.gz#_2018-09-22_20_08_04_799 | 13:14 |
| mriedem | AJaeger: yeah if it clones the repo that's fine, | 13:15 |
| mriedem | but i'm not sure where it is cloning it to | 13:15 |
| mriedem | b/c for grenade we need it to be cloned to /opt/stack/new | 13:15 |
| AJaeger | with zuul legacy jobs ,you need to add to required-projects *and* update the zuul-cloner invocation. | 13:15 |
| mriedem | i've got the former, | 13:17 |
| *** agopi is now known as agopi|ansible|tr | 13:17 | |
| mriedem | not sure what the latter is, | 13:17 |
| mriedem | is it this? http://git.openstack.org/cgit/openstack/neutron/tree/playbooks/legacy/neutron-grenade/run.yaml#n10 | 13:17 |
| *** lbragstad has joined #openstack-infra | 13:18 | |
| *** aidin has quit IRC | 13:19 | |
| *** sthussey has joined #openstack-infra | 13:21 | |
| mriedem | i think i see what was missing, | 13:29 |
| mriedem | still need to list openstack/placement in the d-g PROJECTS variable to get it cloned into /opt/stack/new/ | 13:30 |
| AJaeger | mriedem: I think so - but those legacy jobs are old, I would rather touch the Zuul v3 ones. Unfortunately grenade is not ready yet. | 13:33 |
| fungi | infra-root: we may want to keep an eye on these and prod for a sru importing the fixes from debian if they drag on: https://people.canonical.com/~ubuntu-security/cve/2018/CVE-2018-16947.html https://people.canonical.com/~ubuntu-security/cve/2018/CVE-2018-16948.html https://people.canonical.com/~ubuntu-security/cve/2018/CVE-2018-16949.html | 13:33 |
| fungi | and if they don't patch the versions in trusty, we might want to consider accelerating the replacement of those servers with xenial sooner than some others | 13:34 |
| *** icey has quit IRC | 13:34 | |
| *** SteelyDan is now known as dansmith | 13:34 | |
| *** e0ne has quit IRC | 13:38 | |
| *** jamesmcarthur has quit IRC | 13:40 | |
| fungi | mmm, another wave of wikispam today. just mass-deleted several hundred pages created by half a dozen new accounts | 13:45 |
| slaweq | frickler: this job which You hold for me failed already: http://logs.openstack.org/96/578796/23/check/neutron-tempest-plugin-dvr-multinode-scenario-zuulv3/f51be2a/ | 13:46 |
| slaweq | frickler: can You add there my ssh key? | 13:46 |
| fungi | the usual scam phone numbers in search engine results tactic, this time advertising prescription drugs | 13:46 |
| slaweq | mordred: hi, or maybe You can add it if frickler is not around? ^^ :) | 13:46 |
| fungi | oh, also a smattering of help desk scam phone numbers | 13:46 |
| *** e0ne has joined #openstack-infra | 13:47 | |
| *** joabdearaujo has quit IRC | 13:48 | |
| *** edmondsw has joined #openstack-infra | 13:49 | |
| mordred | slaweq: ips are 38.108.68.22 and 38.108.68.41 | 13:52 |
| mordred | adding keys | 13:52 |
| slaweq | mordred: thx a lot :) | 13:52 |
| mordred | slaweq: k - you should be good on both | 13:53 |
| mordred | and you're welcome! | 13:53 |
| slaweq | mordred: thx | 13:53 |
| openstackgerrit | Matthew Thode proposed openstack-infra/openstack-zuul-jobs master: add Gentoo jobs and vars and also fix install test https://review.openstack.org/602439 | 13:54 |
| *** yamamoto has quit IRC | 13:54 | |
| *** yamamoto has joined #openstack-infra | 13:55 | |
| *** yamamoto has quit IRC | 13:55 | |
| *** yamamoto has joined #openstack-infra | 13:55 | |
| *** jaosorior has quit IRC | 13:55 | |
| *** yamamoto has quit IRC | 13:55 | |
| *** bdodd has joined #openstack-infra | 13:55 | |
| *** beekneemech is now known as bnemec | 13:57 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/openstack-zuul-jobs master: add Gentoo jobs and vars and also fix install test https://review.openstack.org/602439 | 13:58 |
| *** agopi|ansible|tr is now known as agopi|ansible | 13:58 | |
| openstackgerrit | Merged openstack-dev/cookiecutter master: Update the outdated URL in README.rst https://review.openstack.org/604598 | 13:59 |
| *** ykarel is now known as ykarel|away | 14:03 | |
| *** dhill_ has quit IRC | 14:06 | |
| *** dhill_ has joined #openstack-infra | 14:06 | |
| *** chkumar|ruck has quit IRC | 14:08 | |
| *** icey has joined #openstack-infra | 14:09 | |
| *** jamesmcarthur has joined #openstack-infra | 14:12 | |
| *** chandankumar has joined #openstack-infra | 14:14 | |
| *** ramishra has quit IRC | 14:14 | |
| *** chandankumar is now known as chkumar|ruck | 14:15 | |
| *** quiquell|rover is now known as quique|rover|off | 14:15 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: added public endpoint to get summit by id https://review.openstack.org/604801 | 14:30 |
| *** roman_g has quit IRC | 14:32 | |
| *** roman_g has joined #openstack-infra | 14:32 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: added public endpoint to get summit by id https://review.openstack.org/604801 | 14:33 |
| corvus | fungi: happily we don't run the butc openafs process | 14:36 |
| fungi | awesome, that was the one i was most worried about out of the three, but wasn't 100% sure how to confirm | 14:36 |
| *** dpawlik has quit IRC | 14:37 | |
| fungi | the second one is mildly worrying if it can leak process memory which contains some credential leading to compromise of the service, but reading through the implication is that's nontrivial to exploit usefully | 14:37 |
| *** dulek has quit IRC | 14:38 | |
| fungi | the third one is really just a risk that some bored script kiddie decides it would be fun to knock the service offline | 14:38 |
| *** dulek has joined #openstack-infra | 14:39 | |
| *** dpawlik has joined #openstack-infra | 14:40 | |
| *** dpawlik has quit IRC | 14:40 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for ironic repositories https://review.openstack.org/592472 | 14:43 |
| *** armax has joined #openstack-infra | 14:45 | |
| *** eernst has joined #openstack-infra | 14:46 | |
| *** dave-mccowan has quit IRC | 14:50 | |
| *** aojea has quit IRC | 14:50 | |
| ssbarnea | did anyone over here ever created openstack.atlassian.net instance? I wanted to register it for some testing but it seems to be already taken. | 14:58 |
| *** chkumar|ruck is now known as chkumar|off | 15:02 | |
| fungi | not to my knowledge | 15:03 |
| fungi | also, heading out to run some lunch errands, but should be back in not too long | 15:03 |
| *** Bhujay has quit IRC | 15:07 | |
| dhellmann | fungi : it looks like our pypi upload job is reporting failures even when it can upload the wheel - http://logs.openstack.org/6e/6e3778dea94f5b3b215706042c32c2125d0a2ce9/release/release-openstack-python/69d77c6/ara-report/result/68f058f3-f727-40f9-bf7e-d3ba16ddbc1f/ | 15:08 |
| dhellmann | I wonder if the legacy API is reporting errors to try to move folks to a newer one? | 15:11 |
| dhellmann | I have at least 3 examples of that failure and it looks like several more | 15:12 |
| mordred | ssbarnea: I certainly have not. I feel like back in the day there was a foundation-sponsored metrics project (bitergia) that for reasons I don't understand used confluence for something | 15:12 |
| *** dave-mccowan has joined #openstack-infra | 15:12 | |
| mordred | ssbarnea: so it's concievable that someone at the foundation registered that | 15:12 |
| mordred | jbryce: ^^ maybe you know? | 15:12 |
| *** yamamoto has joined #openstack-infra | 15:13 | |
| ssbarnea | fungi: yep, I had two remove the old legacy pypi upload address to make it upload packages again (no openstack infra). If I remember well solution was to remove the URL completly to make it work, from the config. | 15:14 |
| *** gsanso_ has joined #openstack-infra | 15:16 | |
| *** dave-mccowan has quit IRC | 15:17 | |
| mordred | so maybe we should prioritize getting to the new api? | 15:17 |
| dhellmann | does twine not use that automatically? | 15:19 |
| dhellmann | ssbarnea : when you say "remove the old legacy pypi upload address" where did you remove it from? | 15:20 |
| dhellmann | and what value did you replace it with? | 15:21 |
| ssbarnea|bkp | dhellmann: nothing, twine default behavior is to use correct url, but you need to avoid creating a config file. put credentials in env vars and it will work. i may be able to find the bug that caused this. | 15:22 |
| dhellmann | ssbarnea : that would be great, thanks | 15:22 |
| mordred | dhellmann: we're setting | 15:22 |
| mordred | it | 15:22 |
| mordred | repository_url: https://test.pypi.org/legacy/ | 15:22 |
| *** weshay_afk is now known as weshay | 15:23 | |
| dhellmann | why are we using the test server? | 15:23 |
| mordred | oh - wait - nevermind - that's in our test pypi secret | 15:23 |
| mordred | we do not set that in production | 15:23 |
| *** dave-mccowan has joined #openstack-infra | 15:23 | |
| ssbarnea|bkp | sounds very similar to my issues, had the same both production and testing, i had to drop testing and started to publish only on production, both releases and prereleases. | 15:23 |
| mordred | ok, I'm satisfied that we should not be writing a repository url into our pypirc files | 15:24 |
| mordred | we are writing index-servers = pypi and then [pypi]\nusername:{{ username }}\npassword:{{ password }} | 15:25 |
| mordred | oh - wait | 15:25 |
| mordred | ssbarnea|bkp: you're saying that we can't put credentials into file and have this work? | 15:26 |
| *** jesusaur has quit IRC | 15:26 | |
| mordred | that's ... exciting | 15:26 |
| ssbarnea|bkp | mordred not sure about that, you need to test. removing url from config may be enough. | 15:26 |
| *** smarcet has quit IRC | 15:28 | |
| *** tbachman has joined #openstack-infra | 15:29 | |
| *** gagehugo has quit IRC | 15:29 | |
| tbachman | question for process/procedure to the channel — if I need to delete a tag (carelessly applied against the wrong branch) in an upstream repo, is there a process for doing this? | 15:30 |
| *** jesusaur has joined #openstack-infra | 15:31 | |
| mordred | ssbarnea|bkp: ok. we shouldn't be setting it - but obviously something is unhappy somewhere :) | 15:32 |
| mordred | ssbarnea|bkp, dhellmann: we could also just put https://upload.pypi.org/ in as our repository_url | 15:32 |
| ssbarnea | mordred: i found https://github.com/pypa/twine/issues/289 but not much else. I didn't had a rc file on CI (travis) but had one locally. Just remove "repository=" and it will start working. | 15:32 |
| *** gagehugo has joined #openstack-infra | 15:33 | |
| ssbarnea | i checked my local file and is commented too | 15:33 |
| mordred | ssbarnea: ok - so like this? http://paste.openstack.org/show/730640/ (also, is username: really a thing?) | 15:34 |
| dhellmann | mordred : this feels like something we should not set, if the PyPA folks are updating twine in advance of breaking changes to the API | 15:34 |
| *** gsanso_ has quit IRC | 15:34 | |
| mordred | dhellmann: yes. I agree- I would definitely like to not set it | 15:35 |
| ssbarnea|bkp | mordred: yep your file looks correct, almost identical with my local copy. | 15:36 |
| mordred | cool. this: https://github.com/pypa/twine/blob/cdd5b2d602d8966fac5ecc127e26c5095360b001/tests/test_utils.py#L28-L47 seems to be their test that such a setup should result in utils.DEFAULT_REPOSITORY in repository | 15:36 |
| ssbarnea|bkp | twine issues are related to pypi CDN migration that happened few months bach, not sure if is their fault. | 15:37 |
| *** efried has quit IRC | 15:37 | |
| *** dpawlik has joined #openstack-infra | 15:38 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Use = not : for ini settings in pypirc https://review.openstack.org/604820 | 15:39 |
| *** efried has joined #openstack-infra | 15:39 | |
| mordred | no clue if that has any impact at all | 15:39 |
| *** dpawlik has quit IRC | 15:42 | |
| dhellmann | I have a long list of the releases that seem to have failed | 15:44 |
| dhellmann | I'll put those into an etherpad so we can redo them | 15:45 |
| dhellmann | mordred : did you have a patch to not set the URL? or do we not need that? | 15:45 |
| mordred | dhellmann: we shouldn't need it - we shoudln't be setting the url currently | 15:45 |
| dhellmann | oh, so then we don't know why the job is failing? | 15:46 |
| mordred | nope - nor why it's trying legacy.pypi.org | 15:46 |
| mordred | http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/secrets.yaml#n60 is the secret definition | 15:46 |
| dhellmann | hmm | 15:46 |
| mordred | compared to http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/secrets.yaml#n42 which is the test_pypi_secret which shows setting a url | 15:47 |
| mordred | and then the defaults for the role: http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/upload-pypi/defaults/main.yaml | 15:47 |
| *** alexchadin has quit IRC | 15:47 | |
| mordred | and the pypirc jinja template: http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/upload-pypi/templates/.pypirc.j2 | 15:47 |
| openstackgerrit | Clark Boylan proposed openstack-infra/glean master: Add option to ignore config drive interfaces info https://review.openstack.org/604193 | 15:53 |
| clarkb | prometheanfire: mordred ianw ^ now with testing | 15:53 |
| clarkb | I'm going to see if I can get the integration tests to do a one off test of that for us too | 15:53 |
| prometheanfire | cool | 15:54 |
| mordred | ++ | 15:54 |
| mordred | clarkb: the patch seems sensible | 15:54 |
| dhellmann | there's an unfortunately long list of releases to re-enqueue https://etherpad.openstack.org/p/DCFL3JyTQH | 15:55 |
| prometheanfire | ya, oslo.messaging-8.1.1 wasn't in pypi | 15:55 |
| dhellmann | oh, so some of them aren't being uploaded? | 15:56 |
| dhellmann | some were | 15:56 |
| dhellmann | at least I thought so | 15:56 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool master: Do Not Merge Testing glean + nodepool integration https://review.openstack.org/604826 | 15:56 |
| clarkb | mordred: ^ I think that will test the nodepool piece with glean that ianw has been working on (we have to be able to set the configuration properties on the nodepool side) | 15:57 |
| clarkb | mordred: fwiw I don't know if the instance metadata keys go in the meta dict or if they are top level | 15:57 |
| clarkb | I guess I should sort that out next | 15:57 |
| clarkb | tbachman: not sure if you got an answer yet, but there isn't really a great answer for that as tags don't update on pull | 15:58 |
| clarkb | tbachman: so you can't expect the tag to actually move for all of your downstream consumers | 15:58 |
| tbachman | clarkb: thx for the response! I acknowledge this is a mess for downstream consumers. | 16:00 |
| dhellmann | tbachman : we usually just tag again in this case (using a new version number) | 16:01 |
| tbachman | I’m left with the worse of two evils: keep the tag, which is incorrectly applied against a branch, or delete it and deal with anything downstream | 16:01 |
| *** e0ne has quit IRC | 16:01 | |
| tbachman | dhellmann: thx for the reply! Let me see if there’s a way I can make creating a new tag work. The problem is that the tag has a higher semantic version than what’s usually used for this branch. That said, let me try some things to see if I can make it work. | 16:02 |
| AJaeger | config-core, could you review https://review.openstack.org/604679 and https://review.openstack.org/604608, please? | 16:03 |
| clarkb | AJaeger: looking | 16:03 |
| mordred | clarkb: I think they go into the meta dict | 16:04 |
| AJaeger | thanks, clarkb | 16:04 |
| mordred | oh - wait - of what? | 16:05 |
| AJaeger | clarkb: could you also review corvus's change to add cherrypy to 3rdparty-ci, please? https://review.openstack.org/#/c/600846/ | 16:06 |
| dhellmann | mordred : this looks suspicious, is this the template? http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/pypirc.erb#n6 | 16:06 |
| AJaeger | bbl | 16:07 |
| clarkb | AJaeger: done. That should be safe because it isn't in any pipelines yet so we won't spam them | 16:07 |
| clarkb | (also I think they may need to set up the application on the github side to get the real spam going | 16:07 |
| mordred | dhellmann: hrm. it shouldn't be - that should only have impact if puppet ran somewhere, and we don't run puppet on our nodepool nodes anymore | 16:07 |
| mordred | we should go ahead and remove that line though | 16:07 |
| clarkb | mordred: ya I'm booting a new test instance in ovh to double check | 16:07 |
| *** agopi|ansible is now known as agopi|training | 16:08 | |
| mordred | clarkb: sorry - if we were talking about the server dict from openstacksdk, the metadata should be found in the metadata key | 16:09 |
| clarkb | mordred: this is the meta_data.json file in the config drive | 16:09 |
| mordred | yeah - there it looks like it's in the meta key | 16:10 |
| *** lbragstad has quit IRC | 16:10 | |
| mordred | looking at a node nodepool booted | 16:10 |
| clarkb | and I forgot to boot it with a config drive | 16:10 |
| mordred | \o/ | 16:12 |
| openstackgerrit | Merged openstack-infra/project-config master: Move storyboard cover job in-repo https://review.openstack.org/604679 | 16:12 |
| clarkb | mordred: yup it goes in the 'meta' dict, that means my change to glean should now have valid testing | 16:14 |
| slaweq | mordred: You can destroy "my" nodes now | 16:15 |
| slaweq | mordred: thx a lot once again | 16:15 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove extra branches attributes https://review.openstack.org/604608 | 16:15 |
| openstackgerrit | Merged openstack-infra/project-config master: Add cherrypy to third-party ci https://review.openstack.org/600846 | 16:15 |
| *** lbragstad has joined #openstack-infra | 16:18 | |
| dhellmann | mordred : so, how do i debug this? it looks like we're using a role defined way down in zuul-jobs, so any debugging would have to land and then I would have to run the job, right? | 16:19 |
| *** janki has quit IRC | 16:20 | |
| dhellmann | https://github.com/pypa/twine/issues/403 makes me think it's a twine bug? | 16:21 |
| dhellmann | ah, and twine 1.12.0 was release today | 16:23 |
| dhellmann | yay | 16:23 |
| clarkb | dhellmann: coffee time? | 16:25 |
| clarkb | we can probably pin twine if we have to | 16:25 |
| *** spotz is now known as spotz_ | 16:25 | |
| clarkb | or just wait for a fix and release | 16:25 |
| *** spotz_ is now known as spotz | 16:25 | |
| dhellmann | it appears that the pypi uploads actually worked, but our tarball uploads failed | 16:25 |
| dhellmann | I don't remember why those are happening after the pypi uploads | 16:26 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for cinder test_retype_multiattach bug 1794143 https://review.openstack.org/604836 | 16:26 |
| openstack | bug 1794143 in Cinder "cinder.tests.unit.volume.test_volume_retype.VolumeRetypeTestCase.test_retype_multiattach intermittently fails" [Medium,Confirmed] https://launchpad.net/bugs/1794143 | 16:26 |
| dhellmann | clarkb : so if there's some way to run the tarball uploads from your side, I think I can do the announce and constraint updates | 16:27 |
| dhellmann | and then we can just not release anything else today and see if twine is fixed | 16:27 |
| clarkb | we probably download the tarball from pypi, check the hash then put that on tarballs.o.o? | 16:27 |
| dhellmann | I don't know how that step works | 16:27 |
| dhellmann | I would expect us to sign and upload the thing we just uploaded to pypi but maybe you're right | 16:28 |
| clarkb | ya I mean for after the fact fixing now | 16:28 |
| dhellmann | oh, yeah | 16:28 |
| clarkb | I want to say those artifacts are ephemeral until we upload them to pypi and tarballs so if we want a tarball that matches the one on pypi we need to publish the one that is there | 16:28 |
| *** icey has quit IRC | 16:29 | |
| dhellmann | yes, you're right about that | 16:29 |
| dhellmann | sorry, I thought you were asking about the job | 16:29 |
| dhellmann | near the top of https://etherpad.openstack.org/p/DCFL3JyTQH there are some lists of the things that exist on pypi but not tarballs.o.o | 16:29 |
| bnemec | Hey, can someone add me to https://review.openstack.org/#/admin/groups/1946,members ? | 16:30 |
| clarkb | bnemec: done | 16:30 |
| bnemec | clarkb: Thanks! | 16:30 |
| *** aidin has joined #openstack-infra | 16:30 | |
| clarkb | hrm we don't upload the signatures to pypi do we. Preserving those may make this more complicated | 16:32 |
| *** icey has joined #openstack-infra | 16:33 | |
| pabelanger | no, we opted not to becase they don't use them any more | 16:33 |
| *** aidin has quit IRC | 16:35 | |
| clarkb | fungi: ^ are we going to have to download from pypi, sign with release key, then push to tarballs in this case? | 16:37 |
| *** dave-mccowan has quit IRC | 16:38 | |
| clarkb | dhellmann: in the list of missing files karbor is only missing a signature file and not the tarball? | 16:43 |
| clarkb | thinking about this more I wonder if we want to sign these files with the CI system if we are pulling from pypi. Harder to verify the CI system actually generated them | 16:43 |
| *** jiapei has quit IRC | 16:43 | |
| fungi | clarkb: dhellmann: in the past we've just lived with some of the release artifacts lacking signatures. this was intentionally designed around previously when we used multiple jobs since we could have a dependent job retrieve the artifacts from tarballs.o.o, check their signatures, then upload the retrieved artifacts to pypi. with the push to combine all this into fewer jobs we lost those safety | 16:44 |
| fungi | checks | 16:44 |
| fungi | clarkb: dhellmann: also looks like they started work on releasing twine 1.12.1 around 16:30z | 16:48 |
| fungi | https://github.com/pypa/twine/pull/406 | 16:49 |
| *** jpich has quit IRC | 16:50 | |
| *** trown is now known as trown|lunch | 16:50 | |
| *** diablo_rojo has joined #openstack-infra | 16:51 | |
| clarkb | fungi: in that case just upload the tarballs on pypi to tarballs.o.o without signatures? | 16:54 |
| clarkb | I also wonder if we should flip the order of operations here so that things we control happen before things we don't control | 16:54 |
| *** hashar is now known as hasharDinner | 16:55 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver https://review.openstack.org/604404 | 16:55 |
| *** rkukura has quit IRC | 16:56 | |
| fungi | yes, i think we should | 16:57 |
| fungi | (on both) | 16:57 |
| *** hamzy_ has joined #openstack-infra | 16:58 | |
| fungi | using the old order it was simpler since we could take the artifacts we already published to tarballs.o.o and manually upload them to pypi if necessary | 16:58 |
| *** derekh has quit IRC | 16:59 | |
| *** hamzy has quit IRC | 16:59 | |
| *** dpawlik has joined #openstack-infra | 17:00 | |
| clarkb | ya and we could rerun the job if not already on pypi | 17:01 |
| clarkb | but pypi will error if there is already a package there | 17:01 |
| fungi | rerunning individual jobs is also nontrivial | 17:02 |
| fungi | we can reenqueue tag refs and rerun _all_ the jobs for them | 17:03 |
| clarkb | reading the job defs we use a list of post run playbooks the first of which is the tarballs upload playbook then the pypi upload playbook. I thought that we ran those in order | 17:04 |
| clarkb | now I'm extra confused over why we don't have things on tarballs.o.o | 17:05 |
| clarkb | oh wait we don't actually upload to tarballs in this jobs? | 17:07 |
| fungi | are we still talking about http://logs.openstack.org/6e/6e3778dea94f5b3b215706042c32c2125d0a2ce9/release/release-openstack-python/69d77c6/ or some other build? | 17:09 |
| clarkb | fungi: that is what I am looking at | 17:09 |
| clarkb | basically where do we attempt to publish to tarballs.o.o? that can't be part of the twine playbook can it? (maybe it is | 17:09 |
| *** udesale has quit IRC | 17:10 | |
| fungi | looks like it would have happened in openstack-artifacts.yaml but was scheduled after pypi.yaml and so its tasks were skipped when the predecessor failed | 17:10 |
| *** sshnaidm has joined #openstack-infra | 17:11 | |
| fungi | the publish-artifacts-to-fileserver role | 17:12 |
| clarkb | ah it is the behavior of post run ordering of parents | 17:12 |
| clarkb | the job order is like a matryoshka doll | 17:13 |
| clarkb | s/job/play/ | 17:13 |
| *** sshnaidm is now known as sshnaidm|pto | 17:13 | |
| *** bobh has quit IRC | 17:14 | |
| clarkb | now to understand why we skip the subsequent playbook there instead of running it | 17:14 |
| clarkb | (Ithought post always ran?) | 17:14 |
| clarkb | when: zuul_success | bool | 17:15 |
| dhellmann | those steps check if the job is failing | 17:15 |
| *** bobh has joined #openstack-infra | 17:15 | |
| dhellmann | yeah | 17:15 |
| clarkb | the mismatch here then is we don't want to publish tarballs if pypi fails to publish | 17:16 |
| clarkb | but twine is confusing things | 17:16 |
| dhellmann | when I'm done with lunch I will work out how to run just the announce and constraint jobs for those releases | 17:16 |
| dhellmann | yeah | 17:16 |
| dhellmann | is there any way to change the order of those steps without rebuilding the whole job? | 17:16 |
| clarkb | dhellmann: no I think zuul hardcodes that matryoshka behavior of the inherited pre and post plays | 17:18 |
| clarkb | maybe this is a reason to go back to the upload to tarballs job then download and push to pypi job split out that we had before? | 17:18 |
| *** jamesmcarthur has quit IRC | 17:19 | |
| *** sshnaidm|pto has quit IRC | 17:20 | |
| *** rkukura has joined #openstack-infra | 17:20 | |
| *** bobh has quit IRC | 17:20 | |
| *** jpena is now known as jpena|off | 17:21 | |
| dhellmann | hmm | 17:21 |
| openstackgerrit | Merged openstack-infra/nodepool master: Add instance boot properties https://review.openstack.org/604248 | 17:23 |
| *** anteaya has joined #openstack-infra | 17:25 | |
| *** smarcet has joined #openstack-infra | 17:25 | |
| *** tosky has quit IRC | 17:30 | |
| *** sshnaidm|pto has joined #openstack-infra | 17:33 | |
| *** dave-mccowan has joined #openstack-infra | 17:35 | |
| *** jamesmcarthur has joined #openstack-infra | 17:35 | |
| *** harlowja has joined #openstack-infra | 17:36 | |
| *** sambetts is now known as sambetts|afk | 17:39 | |
| *** jamesmcarthur has quit IRC | 17:40 | |
| clarkb | dhellmann: smcginnis: should I go ahead and start copying tarballs from pypi to tarballs and not bother with the signature files? | 17:45 |
| dhellmann | clarkb : yes, please | 17:45 |
| dhellmann | I'm working on the script to run the announce and constraint updates | 17:45 |
| dhellmann | but I have a meeting starting in a bit so it may take me a while | 17:46 |
| clarkb | dhellmann: do we also copy the wheels to tarballs? | 17:46 |
| clarkb | for example os-net-config only has a .whl file | 17:47 |
| clarkb | we must upload .whl before the .tar.gz? | 17:47 |
| clarkb | now the question becomes do we want to generate sdists by hand where they are missing? | 17:48 |
| dhellmann | oh, I didn't check that | 17:48 |
| dhellmann | ugh | 17:48 |
| * dhellmann makes a note to always run a test release before doing any real releases | 17:49 | |
| dhellmann | I wonder if it would be easier to just re-tag all of these things | 17:50 |
| dhellmann | is there some way to pin the version of twine we use? | 17:50 |
| mordred | dhellmann: yah - we install it in the ensure-twine role | 17:50 |
| *** jamesmcarthur has joined #openstack-infra | 17:51 | |
| clarkb | that is in zuul-jobs | 17:51 |
| mordred | dhellmann: it might, in fact, be easier to just do a patch-level version bump - but I don't know if that's the rightchoice | 17:51 |
| *** dpawlik has quit IRC | 17:51 | |
| *** trown|lunch is now known as trown | 17:52 | |
| fungi | the fix was merged to twine a couple hours ago. i was hoping they'd have merged the 1.12.1 release pr shortly thereafter | 17:52 |
| *** jamesmcarthur has quit IRC | 17:53 | |
| *** bobh has joined #openstack-infra | 17:53 | |
| dhellmann | mordred : I think it probably will be. I'll start working on that patch now and then we can figure out how to do the right thing with twine | 17:54 |
| mordred | fungi: so we want either 1.11 or 1.12.1 but not 1.12.0 right? | 17:55 |
| clarkb | mordred: yes | 17:55 |
| fungi | correct | 17:55 |
| mordred | fungi: we could just directly put "twine!=1.12.0" in the role | 17:55 |
| fungi | yes | 17:55 |
| mordred | and that should pin us to 1.11 until 1.12.1 is out | 17:55 |
| fungi | precisely | 17:55 |
| *** dtantsur is now known as dtantsur|afk | 17:56 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Block twine 1.12.0 when we install it https://review.openstack.org/604862 | 17:57 |
| *** harlowja has quit IRC | 17:58 | |
| fungi | i don't suppose you need to quote the parameter when passing it in ansible like that? | 17:58 |
| fungi | the ! isn't going to get interpreted by the shell it's passed off to, or it will get quoted by ansible itself? | 17:58 |
| *** akhila has joined #openstack-infra | 17:58 | |
| clarkb | fungi: I typed != into bash and and it say != command not found | 17:59 |
| clarkb | so I think that is ok | 17:59 |
| fungi | cool | 17:59 |
| clarkb | and yaml should be fine with the ! as long as it isn't the first character in the string? | 17:59 |
| fungi | i know > and < in pip install versions tend to make it ornery | 17:59 |
| mordred | yah- I tried the pip install command locally in the shell and it worked | 17:59 |
| fungi | awesome | 18:00 |
| clarkb | you need a space after ! for it to be parsed as negation by bash looks like | 18:00 |
| clarkb | !false doesn't work but ! false does | 18:00 |
| openstack | clarkb: Error: "false" is not a valid command. | 18:00 |
| fungi | meetbot agrees | 18:02 |
| AJaeger | I have a docs update for bindep - anybody wants to review it, please? https://review.openstack.org/599755 | 18:04 |
| *** e0ne has joined #openstack-infra | 18:06 | |
| *** ediardo has joined #openstack-infra | 18:07 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool master: Add release note for instance-properties config https://review.openstack.org/604863 | 18:10 |
| *** coreycb has joined #openstack-infra | 18:14 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/bindep master: Update docs building https://review.openstack.org/599755 | 18:16 |
| AJaeger | fungi: addressed your comment ^ | 18:16 |
| *** dpawlik has joined #openstack-infra | 18:17 | |
| *** dpawlik_ has joined #openstack-infra | 18:17 | |
| prometheanfire | AJaeger: I think the gentoo stuff 'works' now, just waiting on multinode fedora | 18:18 |
| fungi | thanks AJaeger! | 18:18 |
| *** dpawlik has quit IRC | 18:18 | |
| clarkb | corvus: when you have a second we should do a quick chat about the state of zuul triggering ansible on bridge.o.o | 18:19 |
| *** bobh has quit IRC | 18:20 | |
| *** bobh has joined #openstack-infra | 18:20 | |
| prometheanfire | one thing that may want to be looked at is adding a binhost/mirror for binkpkgs to save on gate time | 18:20 |
| clarkb | prometheanfire: that is precompiled packages for gentoo? | 18:21 |
| prometheanfire | ya | 18:21 |
| prometheanfire | I do that for my home infra | 18:21 |
| prometheanfire | and https://review.openstack.org/604268 does it for DIB | 18:21 |
| prometheanfire | would just have to have a job that compiled some of the bigger programs and stored it in a mirror location | 18:22 |
| AJaeger | fungi, thanks! | 18:22 |
| prometheanfire | https://wiki.gentoo.org/wiki/Binary_package_guide | 18:22 |
| fungi | AJaeger: thanks for fixing it up! | 18:22 |
| corvus | clarkb: how about after lunch? | 18:25 |
| clarkb | corvus: sure | 18:26 |
| prometheanfire | clarkb: in order to do it though we'd need a list of packages and a periodic job type setup | 18:26 |
| clarkb | prometheanfire: left some notes, not sure I understand enough of gentoo packaging to actually know why you've made those choices | 18:26 |
| *** jamesmcarthur has joined #openstack-infra | 18:32 | |
| prometheanfire | ianw: clarkb: would you rather me clean it up now or later? | 18:32 |
| *** sshnaidm|pto has quit IRC | 18:36 | |
| *** ykarel|away has quit IRC | 18:39 | |
| prometheanfire | too late | 18:39 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: clean up gentoo overlay provisioning https://review.openstack.org/604873 | 18:39 |
| dhellmann | mordred, fungi, clarkb : it looks like twine 1.12.1 is out now. I have a patch up to run a release test. Should I wait for any other changes to land before approving that? | 18:48 |
| mordred | dhellmann: I don't think you should need to - each job installs twine from scratch | 18:49 |
| dhellmann | mordred : ok, thanks | 18:50 |
| *** ianychoi_ is now known as ianychoi | 18:54 | |
| *** panda has quit IRC | 18:56 | |
| *** yumiriam has joined #openstack-infra | 18:58 | |
| *** jmorgan1 has quit IRC | 19:06 | |
| mwhahaha | clarkb: did we ever get caching stats from the mirrors recorded somewhere? we're trying to track down why container fetching sometimes takes longer than other runs | 19:06 |
| clarkb | mwhahaha: no, I pushed my scrupt up for basic numbers but ut isnt publishing anywhere | 19:08 |
| *** anteaya has quit IRC | 19:08 | |
| fungi | i guess i or someone can run that over current logs and put up a paste of the results? | 19:10 |
| *** jmorgan1 has joined #openstack-infra | 19:11 | |
| *** tosky has joined #openstack-infra | 19:11 | |
| *** shardy has quit IRC | 19:12 | |
| *** pcaruana has quit IRC | 19:17 | |
| *** jamesmcarthur has quit IRC | 19:27 | |
| slaweq | clarkb: hi, can You help me maybe with network config for dvr multinode job? | 19:30 |
| slaweq | clarkb: I was trying to do what we discussed few days ago | 19:30 |
| clarkb | slaweq: sure I'm eating lunch right now thoufh | 19:30 |
| slaweq | clarkb: ok, please ping me when You will have some time later :) | 19:30 |
| slaweq | clarkb: bon appetit :) | 19:31 |
| *** jamesmcarthur has joined #openstack-infra | 19:32 | |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Adding openstack/octavia-lib project https://review.openstack.org/604889 | 19:33 |
| *** jamesmcarthur_ has joined #openstack-infra | 19:33 | |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Adding openstack/octavia-lib project https://review.openstack.org/604889 | 19:35 |
| *** jamesmcarthur has quit IRC | 19:37 | |
| *** panda has joined #openstack-infra | 19:37 | |
| corvus | clarkb: o/ | 19:37 |
| clarkb | corvus: Im finishing uo lunch about 10-15 minutes away | 19:38 |
| corvus | k | 19:38 |
| *** graphene has quit IRC | 19:49 | |
| *** graphene has joined #openstack-infra | 19:50 | |
| *** harlowja has joined #openstack-infra | 19:50 | |
| *** dpawlik_ has quit IRC | 19:51 | |
| clarkb | corvus: ok the first thing is that nil nodeset jobs are weird. We don't capture the inventory (useful for the zuul vars and related data) and we don't remove the master key from the ssh agent because the ssh setup role runs against all. | 19:51 |
| clarkb | corvus: the other thing is that bridge.o.o doesn't allow root login over ssh generally. You have to ssh from one of the two bastions for that to work | 19:52 |
| clarkb | for the first thing we should probably sort out how we want to deal with nil nodeset jobs (maybe a new base job?) for the bridge ssh problem maybe we should add a zuul user to bridge.o.o? | 19:52 |
| clarkb | fungi: if you get a chance to run that it would probably be helpful | 19:53 |
| fungi | clarkb: where was the script? is there a review containing it or something? | 19:54 |
| *** dpawlik has joined #openstack-infra | 19:55 | |
| *** brwyatt_ has joined #openstack-infra | 19:56 | |
| *** dpawlik has quit IRC | 19:56 | |
| clarkb | fungi: system-config/modules/openstack_project/files/mirror/cache-stats.sh | 19:57 |
| clarkb | it merged looks like | 19:57 |
| fungi | oh, excellent. thanks! | 19:58 |
| *** njohnston has joined #openstack-infra | 19:59 | |
| *** brwyatt_ is now known as brwyatt | 19:59 | |
| clarkb | mordred: what is the current state of group membership configuration in ansible? I need to rebase https://review.openstack.org/#/c/591017/1 to modify groups for etherpad servers and don't want to end up conflicting again | 19:59 |
| fungi | i have noted the upgraded configdrive/metadata problem with recent servers i've booted in my personal ovh account too. cloud-init similarly acts on them, but if i go in and replace the cloud-init recorded static addressing and routing with auto dhcp then everything just works | 20:00 |
| fungi | i'll probably add some ovh-specific cloud-init overrides in userdata to work around it at first boot | 20:00 |
| njohnston | Hi! Quick question - I am working on setting up a job that will run some neutron tests in zuul using a Fedora 28 image, and it looks like something (probably devstack-gate) assumed that `yum` will be assumed on RPM-based systems. But the dnf-yum package is not installed so it looks like there is no yum executable. | 20:04 |
| njohnston | Is this something where it would be better to add to the image, or add something to install yum in devstack-gate or whatever is the right place in the toolchain? | 20:05 |
| njohnston | error in question: http://logs.openstack.org/30/600530/8/check/neutron-tempest-iptables_hybrid-fedora/f21b11a/job-output.txt.gz#_2018-09-24_17_27_49_170132 | 20:06 |
| clarkb | slaweq: I'm around the computer now to talk networking things | 20:06 |
| slaweq | clarkb: thx | 20:07 |
| clarkb | njohnston: new jobs shouldn't use devstack-gate, it is basically on life support to support the existing legacy jobs that we had pre zuulv3 | 20:07 |
| clarkb | njohnston: devstack itself should use dnf iirc | 20:07 |
| slaweq | clarkb: basically I did something related to what we discussed few days ago in this dvr multinode job | 20:07 |
| slaweq | clarkb: I had to propose additional patch https://review.openstack.org/#/c/604285/ to create patch ports between br-ex and br-infra | 20:08 |
| slaweq | clarkb: I also changed devstack config that it will configure public_gateway_ip on br-infra bridge instead of br-ex directly, it's in https://review.openstack.org/#/c/578796/ | 20:09 |
| slaweq | clarkb: but even with that this job failed again | 20:09 |
| slaweq | clarkb: but now I just found that maybe BUILD_TIMEOUT was too short there so I'm trying again | 20:09 |
| slaweq | clarkb: so for now only thing I would like from You is to take a look at https://review.openstack.org/#/c/604285/ and tell me if such change makes any sense for You :) | 20:10 |
| clarkb | slaweq: ok, I'm not sure why the public_gateway_ip would need to be on br-infra instread of br-ex if the two are bridged together but otherwise that seems right | 20:10 |
| clarkb | slaweq: I think you should do that on the devstack side of things and not in the generic bridge role. The zuul-jobs roles are itnended to be generally useful and not encode openstack specific things | 20:10 |
| slaweq | clarkb: ok, so are You saying that it should be done directly in my job's definition somehow or in some devstack role (where I can find it?) | 20:12 |
| clarkb | slaweq: yes, because people using the zuul-jobs roles shouldn't be faced with neutron options. All of the devstack setup should be in devstack itself | 20:13 |
| pabelanger | clarkb: ianw: so, if we +3 https://review.openstack.org/#/c/585574/ for glean, do you think we can cut a release today? I am at a loss why nodepool booting fedora fails on glean, but manually it works | 20:13 |
| clarkb | slaweq: you probably want to add a play or role to devstack to configure the additional networking if dvr is enabled | 20:13 |
| pabelanger | unless, I am some how using different shade flags manually, then nodepool is | 20:13 |
| *** dpawlik has joined #openstack-infra | 20:13 | |
| clarkb | pabelanger: I need more context, but I would be very surprized if that fixed anything for you | 20:14 |
| slaweq | clarkb: ok, I will take a look where to add it then :) | 20:14 |
| slaweq | thx for explanation | 20:14 |
| fungi | clarkb: mwhahaha: mnaser: looks like for mirror.ca-ymq-1.vexxhost (the one in question in the slow job linked from #tripleo earlier) this is the cache breakdown over the past few days: http://paste.openstack.org/show/730669 | 20:14 |
| pabelanger | clarkb: yah, I don't think it is the fix either. But happy to explain more. | 20:15 |
| fungi | fairly high cache miss percentage on the port 8080 proxy | 20:15 |
| *** agopi|training has quit IRC | 20:15 | |
| clarkb | fungi: we should update the script to look at port 80 now since pypi is a proxy there too | 20:15 |
| pabelanger | clarkb: basically, trying to get fedora-minimal DIB to boot in vexxhost from nodepool, using glean, but for some reason doesn't properly setup the network. The bad part, if I manually boot a VM using shade, I don't have a failure with glean. | 20:16 |
| fungi | i'll try that | 20:16 |
| mwhahaha | which port is for which service? | 20:16 |
| mnaser | fungi: clarkb mwhahaha could we just be unlucky and not have a lot of hits lately..? | 20:16 |
| clarkb | mnaser: it could be due to new data (so not cached), expired data needing updates, data too big to be cached (I think our limit is like 5gb though which is huge) | 20:17 |
| mnaser | clarkb: it *seems* like this issue seems to be limited to our mirror server though | 20:17 |
| mnaser | but i'm assuming things, mwhahaha can comment | 20:17 |
| *** dpawlik has quit IRC | 20:18 | |
| fungi | looks like the script will need a bit of refactoring to accommodate port 80 since it's not going to appear in the log filenames | 20:18 |
| clarkb | fungi: it shouldn't | 20:20 |
| pabelanger | clarkb: also, if I could figure out how to redirect glean.sh to console log for openstack, that should explain what is failing | 20:20 |
| fungi | mwhahaha: the configuration is here: https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/mirror.vhost.erb | 20:20 |
| clarkb | fungi: ya just need to make that loop a function and pass in the file path as an arg probably | 20:21 |
| mwhahaha | so 8181 is docker bits | 20:21 |
| clarkb | mwhahaha: it depends on which version of docker, because well docker | 20:21 |
| *** harlowja has quit IRC | 20:22 | |
| mwhahaha | well from those stats, clearly we're not using 8282 :D | 20:22 |
| pabelanger | there is also the issue that docker invalids the cache every 24 hours, for some reason | 20:22 |
| clarkb | ya I think that was set up for loci as they are bleeding edge docker and they don't run many jobs | 20:22 |
| clarkb | pabelanger: we do that in apache | 20:22 |
| pabelanger | clarkb: right, but if we refersh, it gets reset doesn't it? | 20:22 |
| clarkb | pabelanger: it shouldn't be an issue apache will check if the file has changed since it cached it and reset the counter | 20:22 |
| clarkb | pabelanger: what are the chacnes you forgot to turn on config drive with your fedora problem? | 20:23 |
| clarkb | since that is what breaks glean for me 99% of the time | 20:23 |
| mwhahaha | the docker stuff seems to be 88% hits which is not too bad | 20:23 |
| pabelanger | clarkb: will confirm again, that nodepool.yaml had it enabled | 20:23 |
| mwhahaha | so that would seem to point to transit to the mirror from the vms | 20:24 |
| clarkb | pabelanger: if you openstack server show the instance it will tell you if config drive is enabled or not, probably good idea to double check if it is actually set on that side | 20:24 |
| pabelanger | clarkb: docker expires, this is outside of our apache refresh however. They do something to their headers, regardless of blob age, force expire after 24 hours | 20:24 |
| *** harlowja has joined #openstack-infra | 20:25 | |
| clarkb | pabelanger: yes thats fine, all that means is apache has to check the validity of the data on the next request, it only redownloads it if the content has changed | 20:25 |
| clarkb | it uses an HTTP HEAD for that iirc | 20:25 |
| pabelanger | clarkb: okay, I'd have to look again, but think it did results of us missing a cache and redownloading | 20:26 |
| pabelanger | but been a few months since I last looked | 20:26 |
| clarkb | pabelanger: in fungi's paste, http://paste.openstack.org/show/730669/, the cache refresh is this situation | 20:26 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Update cache-stats.sh to include the port 80 proxy https://review.openstack.org/604897 | 20:27 |
| clarkb | if the remote is down or if the data did update then we have to redownload. But since docker images are sha256sum identified we should only ever reconfirm the data is still valid and use what is cached | 20:28 |
| pabelanger | clarkb: yah, I guess for 8081, we'd need to check the last time tripleo create a new image, because if they are rotating them a lot, then I'd expect misses to be high, if they don't rebuild often, misses should be lower | 20:28 |
| fungi | http://paste.openstack.org/show/730670 to include the port 80 proxy (per patch in 604897) | 20:28 |
| clarkb | pabelanger: yup | 20:28 |
| pabelanger | http://paste.openstack.org/show/725348/ | 20:29 |
| fungi | oh, i have an expansion error in there. fixing | 20:29 |
| pabelanger | that was from my last round of testing | 20:29 |
| pabelanger | sha256 blobs expire every 4 hours, sorry | 20:29 |
| *** kgiusti has left #openstack-infra | 20:29 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Update cache-stats.sh to include the port 80 proxy https://review.openstack.org/604897 | 20:31 |
| fungi | http://paste.openstack.org/show/730671 is the output from the latest patchset | 20:31 |
| pabelanger | clarkb: config_drive: True | 20:32 |
| *** akhila has quit IRC | 20:32 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Add digit support to etherpad servers https://review.openstack.org/591017 | 20:33 |
| *** sshnaidm|pto has joined #openstack-infra | 20:34 | |
| clarkb | pabelanger: ok so not config drive, is this fedora 28 like what we are booting for openstack? | 20:34 |
| pabelanger | clarkb: I am doing boot-from-volume, so I guess I could try to snapshot or copy volume and look at logs | 20:34 |
| pabelanger | clarkb: no, not the same. I am using some of the same elements, -minimal, growroot, etc, but different | 20:35 |
| pabelanger | clarkb: this does work with centos-7 as the same elements as fedora-28 | 20:36 |
| pabelanger | so, something on fedora side is different | 20:36 |
| clarkb | pabelanger: and how are you observing the failure? is it that nodepool can't ssh in? any idea if you can ping the instances? | 20:36 |
| clarkb | I wonder if it is a glean issue vs a services on the host issue | 20:37 |
| *** e0ne has quit IRC | 20:37 | |
| pabelanger | clarkb: yah, nodepool return NODE_FAILURE, I manually grab the console log via openstack client and see: [[0;1;31mFAILED[0m] Failed to start Glean for interface ens3. | 20:38 |
| pabelanger | but glean doesn't output debug to console | 20:39 |
| clarkb | pabelanger: it does if you pass --debug I think | 20:39 |
| clarkb | you could modify your simple-init init scripts to add that flag | 20:39 |
| pabelanger | yah, did, nothing sadly | 20:39 |
| pabelanger | I think systemd blocks it | 20:40 |
| pabelanger | let me confirm | 20:40 |
| *** dpawlik has joined #openstack-infra | 20:40 | |
| pabelanger | yah, working VM no info in console, but is in journald | 20:40 |
| pabelanger | StandardOutput=journal+console | 20:41 |
| pabelanger | should fix that however | 20:41 |
| *** jamesmcarthur_ has quit IRC | 20:41 | |
| clarkb | systemd.journald.forward_to_console will set it globally too I think | 20:42 |
| clarkb | probably too noisy to do that | 20:42 |
| *** jamesmcarthur has joined #openstack-infra | 20:42 | |
| *** jamesmcarthur_ has joined #openstack-infra | 20:44 | |
| *** jamesmcarthur has quit IRC | 20:44 | |
| pabelanger | yah, pushed up patch, rebuilding image | 20:44 |
| *** jamesmcarthur has joined #openstack-infra | 20:44 | |
| clarkb | pabelanger: also that comes from log.debug which may go to stderror | 20:45 |
| pabelanger | clarkb: ++, did both to be safe | 20:45 |
| *** jamesmcarthur_ has quit IRC | 20:48 | |
| clarkb | mwhahaha: fungi mnaser the high cache miss rate on port 80 appears to be due to pypi setting a short ttl (uncacheable basically) on pypi indexes | 20:49 |
| clarkb | I think the packages themselves are being cached | 20:49 |
| *** jackatrack_ has joined #openstack-infra | 20:51 | |
| clarkb | mwhahaha: fungi mnaser on the docker side of things grepping for images that were missed shows they were only ever used once | 20:51 |
| clarkb | (at least a small handful of semi randomly chosen examples) | 20:51 |
| mwhahaha | Likely less frequent jobs on older branches | 20:52 |
| *** ansmith has quit IRC | 20:53 | |
| clarkb | mwhahaha: tripleorocky/centos-binary-rsyslog-base seems to be one of the examples of missed and never used again | 20:54 |
| mwhahaha | That's odd | 20:54 |
| *** slaweq has quit IRC | 21:01 | |
| jackatrack_ | Hopefully I'm in the right place! I wanted to bring up a possible bug that is noticeable while looking at the “Nodepool Images” panels on http://grafana.openstack.org/d/rZtIH5Imz/nodepool?orgId=1 , all of the graphs appear to report a minimum constant of one (1) build in progress. I would expect the minimum to be zero. thanks | 21:03 |
| *** trown is now known as trown|outtypewww | 21:05 | |
| clarkb | jackatrack_: I think we don't send any data to statsd when we are not doing work | 21:05 |
| *** evrardjp has joined #openstack-infra | 21:06 | |
| clarkb | jackatrack_: we send data when doing work so it gets set to 1 then never back to 0 | 21:06 |
| jackatrack_ | I see, that makes sense. Any thoughts as to how to fix it? even if it's a work around/ | 21:06 |
| clarkb | we could probably send an event when we complete a task to decrement the counter? corvus may have ideas for that | 21:07 |
| *** bobh has quit IRC | 21:08 | |
| jackatrack_ | out of curiosity, can you point me to the code that is responsible for updating statsd ? | 21:08 |
| *** sshnaidm|pto has quit IRC | 21:10 | |
| *** diablo_rojo_phon has quit IRC | 21:10 | |
| clarkb | jackatrack_: https://git.zuul-ci.org/cgit/nodepool/tree/nodepool/stats.py I think that is the library that nodepool calls into to do those updates | 21:10 |
| jackatrack_ | that looks like it | 21:12 |
| *** dpawlik has quit IRC | 21:12 | |
| *** rh-jelabarre has quit IRC | 21:13 | |
| pabelanger | clarkb: bingo: http://paste.openstack.org/show/730674/ | 21:13 |
| pabelanger | hostnname too long | 21:14 |
| pabelanger | :) | 21:14 |
| fungi | dhellmann: if you need, we can bump your test release change into the gate so release requests don't get held up any longer than necessary | 21:14 |
| clarkb | pabelanger: the template for the names is configurable | 21:14 |
| clarkb | pabelanger: but ya thats a limit defined by dns | 21:14 |
| pabelanger | clarkb: ++ | 21:14 |
| pabelanger | yah, ever hit that before | 21:14 |
| pabelanger | ansible-fedora-28-vexxhost-ansible-networking-ca-ymq-1-0000188812 | 21:15 |
| pabelanger | is what it is trying to set | 21:15 |
| clarkb | 64 per dot and 256 total? | 21:15 |
| clarkb | something like that | 21:15 |
| corvus | clarkb: ah, i see you're suggesting a new base job for zero-node jobs which runs add-build-sshkey on localhost? | 21:15 |
| pabelanger | clarkb: yah, need to read up and confirm | 21:16 |
| clarkb | corvus: ya, though maybe we need a new role entirely? I pushed up a thing to make the existing job work but I'm not sure I care for it. Let me see if I can find the links | 21:16 |
| pabelanger | my hostname is 65 | 21:16 |
| clarkb | corvus: https://review.openstack.org/#/c/603240/ and https://review.openstack.org/#/c/603241/ | 21:16 |
| fungi | ietf rfc 1035: "labels 63 octets or less, names 255 octets or less" | 21:17 |
| clarkb | I was off by one! | 21:18 |
| fungi | you were basically right | 21:18 |
| clarkb | pabelanger: maybe we should update the simple init systemd config to write to console then if someone adds debug it just works for them | 21:20 |
| clarkb | (worth noting the file close fix would not change this behavior for you) | 21:20 |
| pabelanger | clarkb: think that is a good idea, will propose a patch | 21:21 |
| openstackgerrit | Merged openstack-infra/nodepool master: Add release note for instance-properties config https://review.openstack.org/604863 | 21:23 |
| *** Tim_ok has joined #openstack-infra | 21:23 | |
| openstackgerrit | Merged openstack-infra/bindep master: Update docs building https://review.openstack.org/599755 | 21:24 |
| corvus | clarkb: yeah, that's a puzzle. lemme cipher on that. the second thing -- a zuul user on bridge vs logging in as root -- that seems okay to me, and it can 'become' root to run ansible-playbook on bridge. let's run that by mordred if he hasn't seen that yet. | 21:27 |
| clarkb | corvus: I mentioned it to him I think he was ok with that plan but lets double check | 21:27 |
| mordred | yah - I think it's a good plan | 21:28 |
| dhellmann | fungi : that may be a good idea. I'm going offline for a bit now, but I can come back and check on things later this evening | 21:31 |
| pabelanger | clarkb: and glean fixed. Thanks for being sounding board :) | 21:32 |
| pabelanger | clarkb: maybe we should / could update nodepool to warn about hostname > 64 too | 21:32 |
| *** jtomasek has quit IRC | 21:37 | |
| *** sshnaidm|pto has joined #openstack-infra | 21:40 | |
| *** agopi|training has joined #openstack-infra | 21:43 | |
| *** bobh has joined #openstack-infra | 21:45 | |
| *** gfidente has quit IRC | 21:47 | |
| *** ansmith has joined #openstack-infra | 21:49 | |
| clarkb | I'm going to install https://review.openstack.org/#/c/604193/ on a test node in bhs1, remove the existing eth0 config and reboot and see if glean sets things up with dhcp as expected | 21:51 |
| clarkb | woot it works | 21:55 |
| *** jamesmcarthur has quit IRC | 21:56 | |
| *** bobh has quit IRC | 21:56 | |
| *** bobh has joined #openstack-infra | 21:57 | |
| clarkb | infra-root I think https://review.openstack.org/#/c/604193/4 is ready to go as a workaround for new bhs1 behavior. Reviews would be much apprecated as the loss of capacity is part of the reason for slow test processing last week and this week | 21:57 |
| *** mriedem is now known as mriedem_away | 21:57 | |
| fungi | i wonder if ovh is going to find a more general solution to that, because the problem affects cloud-init in exactly the same way | 22:01 |
| * fungi has instances in ovh to prove it | 22:02 | |
| *** Emine has quit IRC | 22:03 | |
| *** emerson has joined #openstack-infra | 22:04 | |
| *** elod has quit IRC | 22:04 | |
| *** graphene has quit IRC | 22:05 | |
| clarkb | fungi: amorin mentioned on the change that they would like to find a way to fix it | 22:05 |
| *** bobh has quit IRC | 22:06 | |
| *** boden has quit IRC | 22:06 | |
| *** graphene has joined #openstack-infra | 22:06 | |
| *** anteaya has joined #openstack-infra | 22:07 | |
| *** evrardjp has quit IRC | 22:07 | |
| ianw | i guess nodepool will need a restart along with adding the flag to project-config | 22:09 |
| clarkb | ianw: yup | 22:10 |
| *** AJaeger_ has joined #openstack-infra | 22:10 | |
| *** elod has joined #openstack-infra | 22:11 | |
| *** AJaeger has quit IRC | 22:11 | |
| clarkb | mordred: corvus for the zuul user being added to bridge.o.o do we want ot treat that like a normal system user via the extra_users mechanism? we can set the public key that way too | 22:14 |
| clarkb | or would you rather see it set up as a system user? | 22:14 |
| clarkb | I'm sort of waffling on this one | 22:14 |
| *** emerson has quit IRC | 22:15 | |
| corvus | extra_users sounds fine to me | 22:15 |
| *** evrardjp has joined #openstack-infra | 22:15 | |
| *** dpawlik has joined #openstack-infra | 22:16 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-sphinx master: Add example and type options to attributes https://review.openstack.org/604267 | 22:17 |
| mordred | clarkb: +2 on the glean - left it un-+Ad in case you want to manage that | 22:17 |
| mordred | clarkb: and yeah - I think extra_users sounds good | 22:17 |
| *** dpawlik has quit IRC | 22:20 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Add zuul user to bridge.openstack.org https://review.openstack.org/604925 | 22:22 |
| clarkb | mordred: corvus ^ I don't actually know if it will handle the mutliple keys there properly | 22:22 |
| corvus | clarkb: not using the url makes me very sad | 22:23 |
| clarkb | we might be able to have a jinja template that pulls the multiple keys down from zuul somehow? | 22:24 |
| clarkb | I'm not really sure how that would work with the current user management | 22:24 |
| corvus | or adjust the role to handle key being a list of urls? | 22:24 |
| corvus | (or, well, list of keys for that matter, it doesn't have to care it's a url) | 22:24 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Add zuul user to bridge.openstack.org https://review.openstack.org/604925 | 22:25 |
| clarkb | updated the job to use that new user | 22:25 |
| *** tbachman has left #openstack-infra | 22:26 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-sphinx master: Add example and type options to attributes https://review.openstack.org/604267 | 22:26 |
| corvus | though, if we did have the users role fetch the key(s), we could keep using the exclusive option which would be nice | 22:26 |
| clarkb | looks like it takes a url | 22:26 |
| clarkb | so ya maybe the role just needs to be updated to handle that as a list | 22:26 |
| corvus | authorized_key takes a url, but it doesn't take a list of urls | 22:27 |
| clarkb | right we'd have to do the iteration a layer above (and then lose the exclusive flag on those) | 22:27 |
| corvus | but if we did the fetch ourselves, we could combine them and keep using exclusive | 22:30 |
| corvus | best of both worlds, but more work | 22:30 |
| prometheanfire | probably the most reviewed glean change | 22:30 |
| corvus | "make cloud work" changes tend to get attention | 22:31 |
| clarkb | I've just melted my brain trying to understand nested loops where inner loop refers to outer loop in ansible | 22:31 |
| prometheanfire | yep | 22:32 |
| *** rkukura has quit IRC | 22:32 | |
| clarkb | is include_tasks really the easiest way to describe this? | 22:33 |
| clarkb | thats kind of amazing | 22:33 |
| clarkb | does ansible preserve python pass by reference behavior with dicts? eg if I have a piece of ansible update part of the host_vars will that be exposed to subsequent tasks? | 22:37 |
| *** panda is now known as panda|off | 22:38 | |
| prometheanfire | ya, ansible changed inlcude to include_tasks/roles | 22:44 |
| *** rcernin has joined #openstack-infra | 22:45 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for cinder test_retype_multiattach bug 1794143 https://review.openstack.org/604836 | 22:45 |
| openstack | bug 1794143 in Cinder "cinder.tests.unit.volume.test_volume_retype.VolumeRetypeTestCase.test_retype_multiattach intermittently fails" [Medium,Confirmed] https://launchpad.net/bugs/1794143 | 22:45 |
| clarkb | prometheanfire: mostly I'm amazed that this is the way to do a nested loop more so than that change | 22:45 |
| clarkb | I've decided that its better to write pythonfor this as a result | 22:46 |
| prometheanfire | :D | 22:47 |
| *** andreww has joined #openstack-infra | 22:51 | |
| *** emerson has joined #openstack-infra | 22:51 | |
| *** lathiat_ has joined #openstack-infra | 22:53 | |
| *** owalsh_ has joined #openstack-infra | 22:53 | |
| *** kukacz_ has joined #openstack-infra | 22:55 | |
| *** emerson has quit IRC | 22:57 | |
| *** dims_ has joined #openstack-infra | 22:59 | |
| *** tomaw- has joined #openstack-infra | 22:59 | |
| *** _d34dh0r53_ has joined #openstack-infra | 23:00 | |
| *** stephenfin_ has joined #openstack-infra | 23:00 | |
| *** kukacz has quit IRC | 23:00 | |
| *** _ari_ has quit IRC | 23:00 | |
| *** owalsh has quit IRC | 23:00 | |
| *** tdasilva has quit IRC | 23:00 | |
| *** dims has quit IRC | 23:00 | |
| *** d34dh0r53 has quit IRC | 23:00 | |
| *** Diabelko has quit IRC | 23:00 | |
| *** jlvillal has quit IRC | 23:00 | |
| *** lathiat has quit IRC | 23:00 | |
| *** xarses_ has quit IRC | 23:00 | |
| *** cburgess has quit IRC | 23:00 | |
| *** strigazi has quit IRC | 23:00 | |
| *** xinliang has quit IRC | 23:00 | |
| *** weshay has quit IRC | 23:00 | |
| *** bandini has quit IRC | 23:00 | |
| *** wznoinsk has quit IRC | 23:00 | |
| *** dabukalam has quit IRC | 23:00 | |
| *** stephenfin has quit IRC | 23:00 | |
| *** tomaw has quit IRC | 23:00 | |
| *** eglute has quit IRC | 23:00 | |
| *** kambiz has quit IRC | 23:00 | |
| *** emerson has joined #openstack-infra | 23:02 | |
| *** andreykurilin has quit IRC | 23:03 | |
| *** ruhe has quit IRC | 23:03 | |
| *** tosky has quit IRC | 23:03 | |
| *** jamesmcarthur has joined #openstack-infra | 23:04 | |
| *** sshnaidm|pto has quit IRC | 23:04 | |
| *** jamesmcarthur has quit IRC | 23:05 | |
| *** jamesmcarthur has joined #openstack-infra | 23:05 | |
| *** hasharDinner has quit IRC | 23:05 | |
| *** andreykurilin has joined #openstack-infra | 23:05 | |
| *** ruhe has joined #openstack-infra | 23:06 | |
| *** evrardjp has quit IRC | 23:07 | |
| *** Diabelko has joined #openstack-infra | 23:08 | |
| *** evrardjp has joined #openstack-infra | 23:08 | |
| *** rkukura has joined #openstack-infra | 23:08 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Manage user ssh keys from urls https://review.openstack.org/604932 | 23:10 |
| clarkb | mordred: corvus ^ I really have no idea if I am doing that right and considering how important it is that we be able to ssh into the bastion... | 23:10 |
| clarkb | I'll WIP it for now but I think that is roughly what corvus would like | 23:11 |
| clarkb | I've approved the glean change. We won't actually use it until we make a glean release and rebuild images | 23:14 |
| clarkb | I likely won't get around to that today (and considering we have to restart nodepool and update nodepool config anyway that is probably fine) | 23:14 |
| clarkb | ianw: ^ unsure if you are interested in doing follow the sun on that though | 23:14 |
| ianw | i'll take a look | 23:16 |
| ianw | is there anything else in glean that might cause bigger issues? | 23:17 |
| clarkb | ianw: this would be the only change on top of the last release I think | 23:17 |
| clarkb | also we should decide if we want to update nodepool side first or glean and images first | 23:18 |
| ianw | glean and images i think | 23:19 |
| ianw | did we want to add that close() thing too? | 23:19 |
| clarkb | ianw: I'm happy to add the close thing too fi we can get another infra root to review it | 23:20 |
| clarkb | infra-root https://review.openstack.org/#/c/585574/ is the glean close() thing ianw is talking about | 23:20 |
| *** anteaya has quit IRC | 23:21 | |
| pabelanger | +2 since I was looking at it today | 23:22 |
| pabelanger | I won't get my | 23:23 |
| pabelanger | err | 23:23 |
| pabelanger | I won't get my stderr redirect to console patch for today, so will be the next glean tag | 23:23 |
| ianw | there's https://review.openstack.org/#/c/493443/ which has been there for a year | 23:23 |
| *** aidin has joined #openstack-infra | 23:25 | |
| clarkb | ianw: prometheanfire ^ comment on | 23:26 |
| clarkb | I think the 0 should be dropped | 23:26 |
| *** jamesmcarthur has quit IRC | 23:28 | |
| clarkb | and now for something completely different | 23:31 |
| clarkb | the ovh gra1 mirror seems to be down | 23:31 |
| clarkb | it pings but ssh isn't responding | 23:32 |
| clarkb | also e-r indicates that the pypi proxy doesn't work | 23:32 |
| clarkb | [17984452.588512] blk_update_request: I/O error, dev sda, sector 92290184 says the console log | 23:33 |
| clarkb | before that [740409.091725] INFO: task apache2:8599 blocked for more than 120 seconds. | 23:33 |
| clarkb | I'm going to try a reboot | 23:33 |
| ianw | ++ | 23:34 |
| clarkb | seems like it is up now | 23:36 |
| fungi | live migration gone wrong during upgrade maintenance maybe? | 23:37 |
| clarkb | this is gra1 where we aren't supposed to have done upgrades yet | 23:38 |
| clarkb | but could be upgrade prep | 23:38 |
| *** tpsilva has quit IRC | 23:43 | |
| clarkb | http://status.openstack.org/elastic-recheck/gate.html#1449136 should hopefully fall back to zero now | 23:43 |
| *** smarcet has quit IRC | 23:47 | |
| *** sthussey has quit IRC | 23:51 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool master: Do Not Merge Testing glean + nodepool integration https://review.openstack.org/604826 | 23:54 |
| clarkb | silly yaml types | 23:54 |
| clarkb | ianw: ^ thats sort of a hgih level sanity check | 23:54 |
| clarkb | it failed because Boolean types are not string in python | 23:54 |
| *** eernst has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!