| corvus | all the session files are identical | 00:00 |
|---|---|---|
| corvus | i wonder if the internal api calls create sessions | 00:00 |
| *** jamesmcarthur has joined #openstack-infra | 00:00 | |
| mordred | I betcha they do | 00:03 |
| corvus | yeah, looks that way | 00:03 |
| corvus | i deleted them all, then observed that there were 2 during a lull in replication; then a bunch of replication events and now we're back up to 1500 | 00:04 |
| mordred | that may warrant a bug | 00:04 |
| corvus | it accounts for about 5GiB of our raw used space :) | 00:05 |
| mordred | wow :) | 00:05 |
| corvus | the internal sessions have uid=0; there is no uid=0 in the db, so i assume that means "the internal user" | 00:05 |
| corvus | i found my web session, i'm uid=2 (as expected according to the db) | 00:05 |
| corvus | i'm going to eod now; by tomorrow we should have gotten through the replication backlog and completed the backfill, so we can poke at things with a bit more clarity | 00:07 |
| *** Jeffrey4l has joined #openstack-infra | 00:08 | |
| mordred | corvus: cool. talk to you then! | 00:08 |
| *** jamesmcarthur has quit IRC | 00:09 | |
| weshay|ruck | if anyone could put https://review.openstack.org/#/c/628047/ at the top of the tripleo gate queue, we'd be much obliged or something like that :) | 00:21 |
| *** wolverineav has quit IRC | 00:33 | |
| clarkb | weshay|ruck: are the test failures for that change a concern? | 00:35 |
| *** bobh has joined #openstack-infra | 00:37 | |
| *** rascasoft has joined #openstack-infra | 00:39 | |
| mwhahaha | clarkb: no those are upgrade (non-voting) jobs which are unrelated to the change. the standalone jobs passed which proves the change to the tempest config is fine | 00:39 |
| mwhahaha | specifically the change ends up http://logs.openstack.org/47/628047/1/check/tripleo-ci-centos-7-standalone/0430d8b/logs/undercloud/home/zuul/skip_file.gz which is used by the tempest run | 00:39 |
| mwhahaha | and the Attach interfasces tests aren't listed in http://logs.openstack.org/47/628047/1/check/tripleo-ci-centos-7-standalone/0430d8b/logs/undercloud/home/zuul/tempest/tempest.html.gz | 00:40 |
| clarkb | ok its enqueued to the gate now. promoiting it will restart the gate queue | 00:42 |
| clarkb | I guess thats fine | 00:42 |
| mwhahaha | better than 50 resets overnight :D | 00:42 |
| clarkb | running now | 00:42 |
| mwhahaha | thanks | 00:42 |
| *** rascasoft has quit IRC | 00:46 | |
| *** rh-jelabarre has quit IRC | 00:47 | |
| *** wolverineav has joined #openstack-infra | 00:53 | |
| *** wolverineav has quit IRC | 00:57 | |
| *** wolverineav has joined #openstack-infra | 00:57 | |
| *** sthussey has quit IRC | 01:05 | |
| *** tosky has quit IRC | 01:06 | |
| *** rascasoft has joined #openstack-infra | 01:06 | |
| *** rascasoft has quit IRC | 01:14 | |
| *** smarcet has joined #openstack-infra | 01:21 | |
| *** dave-mccowan has quit IRC | 01:23 | |
| *** mriedem has quit IRC | 01:25 | |
| *** markvoelker has quit IRC | 01:26 | |
| *** markvoelker has joined #openstack-infra | 01:26 | |
| *** markvoelker has quit IRC | 01:31 | |
| *** wolverineav has quit IRC | 01:33 | |
| *** wolverineav has joined #openstack-infra | 01:35 | |
| *** efried has joined #openstack-infra | 01:47 | |
| *** efried has quit IRC | 01:50 | |
| *** wolverineav has quit IRC | 01:55 | |
| *** diablo_rojo has quit IRC | 01:56 | |
| *** smarcet has quit IRC | 02:00 | |
| *** rascasoft has joined #openstack-infra | 02:03 | |
| *** jamesmcarthur has joined #openstack-infra | 02:05 | |
| *** bobh_ has joined #openstack-infra | 02:05 | |
| *** wolverineav has joined #openstack-infra | 02:06 | |
| *** bobh has quit IRC | 02:09 | |
| *** jamesmcarthur has quit IRC | 02:09 | |
| *** rascasoft has quit IRC | 02:10 | |
| *** wolverineav has quit IRC | 02:18 | |
| *** jamesmcarthur has joined #openstack-infra | 02:23 | |
| *** markvoelker has joined #openstack-infra | 02:27 | |
| *** hongbin has joined #openstack-infra | 02:29 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:35 | |
| *** rlandy|rover|bbl is now known as rlandy|rover | 02:46 | |
| *** dave-mccowan has joined #openstack-infra | 02:50 | |
| *** markvoelker has quit IRC | 02:50 | |
| *** markvoelker has joined #openstack-infra | 02:50 | |
| *** jrist has quit IRC | 03:07 | |
| *** jamesmcarthur has quit IRC | 03:20 | |
| *** rlandy|rover has quit IRC | 03:20 | |
| *** owalsh has quit IRC | 03:25 | |
| *** psachin has joined #openstack-infra | 03:36 | |
| *** bobh has joined #openstack-infra | 03:39 | |
| *** owalsh has joined #openstack-infra | 03:39 | |
| *** bobh_ has quit IRC | 03:42 | |
| *** bhavikdbavishi has quit IRC | 03:46 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:46 | |
| *** bhavikdbavishi has quit IRC | 03:48 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:48 | |
| *** roman_g has quit IRC | 03:53 | |
| *** bobh has quit IRC | 03:54 | |
| *** armax has quit IRC | 03:59 | |
| *** ykarel|away has joined #openstack-infra | 04:00 | |
| *** wolverineav has joined #openstack-infra | 04:11 | |
| *** ramishra has joined #openstack-infra | 04:17 | |
| *** ykarel|away is now known as ykarel | 04:17 | |
| *** anteaya has quit IRC | 04:17 | |
| *** jamesmcarthur has joined #openstack-infra | 04:35 | |
| *** udesale has joined #openstack-infra | 04:36 | |
| *** bobh has joined #openstack-infra | 04:46 | |
| *** eernst_ has quit IRC | 04:52 | |
| *** dave-mccowan has quit IRC | 04:56 | |
| *** psachin has quit IRC | 05:00 | |
| *** hongbin has quit IRC | 05:05 | |
| *** whoami-rajat has joined #openstack-infra | 05:05 | |
| *** markvoelker has quit IRC | 05:17 | |
| *** bobh has quit IRC | 05:17 | |
| *** wolverineav has quit IRC | 05:26 | |
| *** psachin has joined #openstack-infra | 05:33 | |
| *** markvoelker has joined #openstack-infra | 05:49 | |
| *** jamesmcarthur has quit IRC | 05:54 | |
| *** wolverineav has joined #openstack-infra | 06:07 | |
| *** markvoelker has quit IRC | 06:22 | |
| *** markvoelker has joined #openstack-infra | 06:23 | |
| *** markvoelker has quit IRC | 06:27 | |
| *** udesale has quit IRC | 06:33 | |
| *** udesale has joined #openstack-infra | 06:34 | |
| *** ccamacho has quit IRC | 06:41 | |
| *** jtomasek has joined #openstack-infra | 06:49 | |
| *** jtomasek has quit IRC | 06:49 | |
| *** jtomasek has joined #openstack-infra | 06:49 | |
| *** wolverineav has quit IRC | 06:53 | |
| *** wolverineav has joined #openstack-infra | 06:54 | |
| *** wolverineav has quit IRC | 06:58 | |
| *** rossella_s has joined #openstack-infra | 07:00 | |
| *** yboaron has joined #openstack-infra | 07:04 | |
| *** rossella_s has quit IRC | 07:05 | |
| *** rcernin has quit IRC | 07:09 | |
| *** quiquell|off is now known as quiquell | 07:18 | |
| *** pgaxatte has joined #openstack-infra | 07:22 | |
| *** ccamacho has joined #openstack-infra | 07:24 | |
| *** ccamacho has quit IRC | 07:24 | |
| *** ccamacho has joined #openstack-infra | 07:25 | |
| *** hwoarang has quit IRC | 07:47 | |
| *** apetrich has joined #openstack-infra | 07:49 | |
| *** hwoarang has joined #openstack-infra | 07:50 | |
| *** yboaron has quit IRC | 07:54 | |
| *** jamesmcarthur has joined #openstack-infra | 07:55 | |
| *** rascasoft has joined #openstack-infra | 07:55 | |
| *** jamesmcarthur has quit IRC | 07:59 | |
| *** dpawlik has joined #openstack-infra | 08:01 | |
| *** aojea has joined #openstack-infra | 08:02 | |
| *** ginopc has joined #openstack-infra | 08:03 | |
| *** lpetrut has joined #openstack-infra | 08:03 | |
| *** ramishra has quit IRC | 08:05 | |
| *** slaweq has joined #openstack-infra | 08:06 | |
| *** ccamacho has quit IRC | 08:09 | |
| *** ramishra has joined #openstack-infra | 08:09 | |
| *** markvoelker has joined #openstack-infra | 08:23 | |
| *** tosky has joined #openstack-infra | 08:27 | |
| *** xek has joined #openstack-infra | 08:51 | |
| *** ccamacho has joined #openstack-infra | 08:54 | |
| *** dtantsur|afk is now known as dtantsur | 08:59 | |
| *** jpich has joined #openstack-infra | 09:00 | |
| *** rossella_s has joined #openstack-infra | 09:05 | |
| *** shardy has joined #openstack-infra | 09:06 | |
| *** shardy has quit IRC | 09:06 | |
| *** shardy has joined #openstack-infra | 09:07 | |
| *** whoami-rajat has quit IRC | 09:13 | |
| *** derekh has joined #openstack-infra | 09:18 | |
| *** whoami-rajat has joined #openstack-infra | 09:25 | |
| *** apetrich has quit IRC | 09:29 | |
| *** amorin has quit IRC | 09:30 | |
| *** lpetrut has quit IRC | 09:32 | |
| *** jpich has quit IRC | 09:37 | |
| *** apetrich has joined #openstack-infra | 09:37 | |
| *** gfidente has joined #openstack-infra | 09:41 | |
| *** e0ne has joined #openstack-infra | 09:42 | |
| *** jpich has joined #openstack-infra | 09:51 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:52 | |
| *** jamesmcarthur has joined #openstack-infra | 09:56 | |
| *** yamamoto has joined #openstack-infra | 09:58 | |
| *** jamesmcarthur has quit IRC | 10:00 | |
| *** shardy has quit IRC | 10:10 | |
| *** shardy has joined #openstack-infra | 10:10 | |
| *** bhavikdbavishi has quit IRC | 10:13 | |
| *** ykarel is now known as ykarel|lunch | 10:16 | |
| *** yamamoto has quit IRC | 10:17 | |
| *** yamamoto has joined #openstack-infra | 10:28 | |
| *** yamamoto has quit IRC | 10:30 | |
| *** smcginnis_away is now known as smcginnis | 10:34 | |
| *** yamamoto has joined #openstack-infra | 10:36 | |
| *** yamamoto has quit IRC | 10:38 | |
| *** yamamoto has joined #openstack-infra | 10:40 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on future parser for one git worker https://review.openstack.org/615966 | 10:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on future parser for all git workers https://review.openstack.org/615967 | 10:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on future parser for git load balancer https://review.openstack.org/615968 | 10:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on future parser for mirror-update.o.o https://review.openstack.org/615991 | 10:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for rax mirror https://review.openstack.org/615992 | 10:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for all mirrors https://review.openstack.org/615993 | 10:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for files.openstack.org https://review.openstack.org/615994 | 10:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on future parser for one zookeeper instance https://review.openstack.org/615995 | 10:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on future parser for all zookeeper instances https://review.openstack.org/615996 | 10:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for one nameserver https://review.openstack.org/615998 | 10:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for all nameservers https://review.openstack.org/615999 | 10:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for master nameserver https://review.openstack.org/616000 | 10:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for openstackid-dev https://review.openstack.org/616001 | 10:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for openstackid https://review.openstack.org/616002 | 10:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on future parser for one nodepool launcher https://review.openstack.org/616288 | 10:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on future parser for all nodepool launchers https://review.openstack.org/616289 | 10:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on future parser for one nodepool builder https://review.openstack.org/616290 | 10:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on future parser for all nodepool builders https://review.openstack.org/616291 | 10:48 |
| * cmurphy climbs out of rebase hell | 10:49 | |
| *** ykarel|lunch is now known as ykarel | 10:52 | |
| *** amorin has joined #openstack-infra | 10:59 | |
| *** pbourke has quit IRC | 11:01 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for refstack https://review.openstack.org/628153 | 11:01 |
| *** pbourke has joined #openstack-infra | 11:02 | |
| *** dkehn has quit IRC | 11:04 | |
| *** smarcet has joined #openstack-infra | 11:21 | |
| *** janki has joined #openstack-infra | 11:21 | |
| *** janki has quit IRC | 11:22 | |
| *** janki has joined #openstack-infra | 11:23 | |
| *** smarcet has quit IRC | 11:23 | |
| *** udesale has quit IRC | 11:26 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Fix puppet beaker jobs for bundler 2.0 https://review.openstack.org/628159 | 11:26 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for refstack https://review.openstack.org/628153 | 11:27 |
| *** smarcet has joined #openstack-infra | 11:28 | |
| *** smarcet has quit IRC | 11:33 | |
| *** janki has quit IRC | 11:41 | |
| *** whoami-rajat has quit IRC | 11:43 | |
| *** janki has joined #openstack-infra | 11:44 | |
| *** yamamoto has quit IRC | 11:44 | |
| *** agopi has joined #openstack-infra | 11:48 | |
| *** gfidente has quit IRC | 11:50 | |
| *** Vadmacs has joined #openstack-infra | 11:54 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Pin bundler for puppet rspec jobs https://review.openstack.org/628159 | 12:01 |
| *** dosaboy has quit IRC | 12:02 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Pin bundler for puppet rspec jobs https://review.openstack.org/628159 | 12:03 |
| *** dosaboy has joined #openstack-infra | 12:07 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:29 | |
| *** yamamoto has joined #openstack-infra | 12:30 | |
| *** gfidente has joined #openstack-infra | 12:36 | |
| *** yamamoto has quit IRC | 12:36 | |
| *** smarcet has joined #openstack-infra | 12:37 | |
| *** smarcet has quit IRC | 12:41 | |
| *** rlandy has joined #openstack-infra | 12:42 | |
| *** rlandy is now known as rlandy|rover | 12:43 | |
| *** rh-jelabarre has joined #openstack-infra | 12:49 | |
| *** gfidente has quit IRC | 12:53 | |
| *** bhavikdbavishi has quit IRC | 12:54 | |
| ssbarnea | i seen workflowed change from yesterday which in not in any zuul queue: https://review.openstack.org/#/c/627178/ | 12:57 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings master: Consolidate openstack-meeting channels https://review.openstack.org/626182 | 12:59 |
| *** agopi_ has joined #openstack-infra | 12:59 | |
| frickler | ssbarnea: the dependency isn't approved yet | 13:00 |
| *** bhavikdbavishi has joined #openstack-infra | 13:01 | |
| *** agopi has quit IRC | 13:02 | |
| *** agopi_ is now known as agopi | 13:02 | |
| ssbarnea | frickler: not sure about what dependency are you talking. | 13:02 |
| ssbarnea | frickler: i just did a recheck and now i see gerrit running the check gates again, and I suppose it will run the gate after, as it was already workflowed. | 13:03 |
| fungi | ssbarnea: Depends-On: https://review.openstack.org/#/c/627186/ | 13:03 |
| ssbarnea | ouch, i missed to see that. | 13:03 |
| fungi | and 627186 hasn't been approved yet | 13:03 |
| *** dpawlik has quit IRC | 13:03 | |
| AJaeger | ssbarnea: and it will not - since the depends-on is on requrirements and tripleo-upgrade and requirements don't share the same queue | 13:03 |
| AJaeger | ssbarnea: so, will need toggle of +Workflow once 627186 is merged | 13:04 |
| fungi | cmurphy: sorry about the rebase hell, that was likely my fault updating all the inventory group globs | 13:04 |
| cmurphy | fungi: no worries | 13:04 |
| cmurphy | fungi: you can make it up to me https://review.openstack.org/628159 | 13:04 |
| fungi | cmurphy: you bet! | 13:04 |
| *** agopi has quit IRC | 13:06 | |
| fungi | thanks for fixing that | 13:06 |
| cmurphy | no problem | 13:07 |
| *** agopi has joined #openstack-infra | 13:08 | |
| *** yamamoto has joined #openstack-infra | 13:10 | |
| *** trown|outtypewww is now known as trown | 13:11 | |
| *** jcoufal has joined #openstack-infra | 13:11 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Pin bundler for puppet rspec jobs https://review.openstack.org/628159 | 13:13 |
| *** quiquell is now known as quiquell|brb | 13:15 | |
| *** ianychoi has joined #openstack-infra | 13:17 | |
| *** markvoelker has quit IRC | 13:18 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/elastic-recheck master: add query for intermittent container failure https://review.openstack.org/626962 | 13:18 |
| *** dave-mccowan has joined #openstack-infra | 13:18 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/elastic-recheck master: add query for intermittent container failure https://review.openstack.org/626962 | 13:19 |
| *** markvoelker has joined #openstack-infra | 13:20 | |
| *** yamamoto has quit IRC | 13:21 | |
| *** derekh has quit IRC | 13:24 | |
| *** whoami-rajat has joined #openstack-infra | 13:30 | |
| *** gfidente has joined #openstack-infra | 13:32 | |
| *** gfidente is now known as gfidente|afk | 13:33 | |
| *** quiquell|brb is now known as quiquell | 13:35 | |
| *** bhavikdbavishi has quit IRC | 13:44 | |
| sshnaidm | clarkb, fungi, pabelanger do you know if we have a feature in CI to release some external resources in aborted job before node is completely destroyed? | 13:54 |
| sshnaidm | not sure feature of what it should be - zuul or nodepool | 13:55 |
| *** lujinluo has joined #openstack-infra | 13:55 | |
| fungi | sshnaidm: there's been a thread on the zuul-discuss ml about adding something like that, but it's not something which has been implemented yet | 13:55 |
| sshnaidm | fungi, ok, thanks, is it planned at all? Or better to create a story in storyboard? | 13:56 |
| fungi | sshnaidm: the discussion thread begins at http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-December/000653.html | 13:56 |
| fungi | sshnaidm: those are questions for #zuul (remember zuul and nodepool are now independent from the openstack infra team, and openstack as a whole even) | 13:57 |
| *** kgiusti has joined #openstack-infra | 13:57 | |
| *** lbragstad has joined #openstack-infra | 13:57 | |
| sshnaidm | fungi, cool, will ask there, thanks | 13:57 |
| *** jistr is now known as jistr|mtg | 14:04 | |
| *** derekh has joined #openstack-infra | 14:04 | |
| *** smarcet has joined #openstack-infra | 14:04 | |
| *** smarcet has quit IRC | 14:09 | |
| *** sthussey has joined #openstack-infra | 14:09 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/ansible-role-puppet master: Revert "Don't log puppot module sync" https://review.openstack.org/628189 | 14:09 |
| *** boden has joined #openstack-infra | 14:10 | |
| mwhahaha | so question, looks like 627522,2 is stuck in queued for like 7 hours and is currently at the top of the gate. any idea why it's stuck? | 14:14 |
| *** mriedem has joined #openstack-infra | 14:14 | |
| pabelanger | clarkb: yay hostids | 14:15 |
| fungi | mwhahaha: i can look after the tc meeting concludes, but hopefully some other infra-root can dig into scheduler debug logs sooner | 14:15 |
| mwhahaha | thanks | 14:16 |
| fungi | i'm worried it could be a new regression brought in with last night's restart, but really have no evidence to back that up | 14:16 |
| *** dkehn has joined #openstack-infra | 14:16 | |
| Shrews | mwhahaha: fungi: i'll see if i can dig something up in the logs | 14:17 |
| fungi | thanks Shrews!!! | 14:18 |
| *** mriedem has quit IRC | 14:19 | |
| *** dpawlik has joined #openstack-infra | 14:20 | |
| *** mriedem has joined #openstack-infra | 14:21 | |
| *** udesale has joined #openstack-infra | 14:22 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/project-config master: Add openstack-virtual-baremetal to openstack https://review.openstack.org/620613 | 14:24 |
| *** dpawlik has quit IRC | 14:24 | |
| *** dpawlik has joined #openstack-infra | 14:24 | |
| *** bobh has joined #openstack-infra | 14:26 | |
| Shrews | hrm, some odd warnings and exceptions around that change that are somewhat cryptic | 14:26 |
| *** dpawlik has quit IRC | 14:28 | |
| *** dpawlik has joined #openstack-infra | 14:28 | |
| *** lujinluo has quit IRC | 14:29 | |
| *** bobh has quit IRC | 14:29 | |
| *** smarcet has joined #openstack-infra | 14:34 | |
| openstackgerrit | kevin raymond proposed openstack-infra/git-review master: Warn which patchset are going to be obsoleted. https://review.openstack.org/456937 | 14:37 |
| Shrews | mwhahaha: fungi: all of the queued jobs for that change seem to have these messages: http://paste.openstack.org/show/739869/ The exception being the openstack-tox-pep8 job | 14:38 |
| fungi | Shrews: could a change have merged immediately ahead of it and triggered a reconfiguration which somehow went wrong, leaving that change in limbo? grasping at straws | 14:42 |
| mwhahaha | the change itself is a zuul change | 14:43 |
| * mwhahaha also has no idea | 14:43 | |
| fungi | yeah, those job names are definitely displayed in the status data, so curious why the scheduler thinks the queueitem doesn't contain them | 14:43 |
| Shrews | fungi: not sure how we could get to that state :/ | 14:44 |
| Shrews | the change itself looks safe enough | 14:44 |
| mwhahaha | shall we just kick that change out of the gate or does anyone want to look into it further | 14:45 |
| Shrews | might be worth having corvus take a look when he's around | 14:46 |
| *** gfidente|afk is now known as gfidente | 14:46 | |
| mwhahaha | k | 14:46 |
| fungi | he tends to show up not too much later than nowish | 14:47 |
| fungi | so i agree, it could be worth waiting a bit. if we _do_ decide to dequeue it then let's at least take a thread dump | 14:47 |
| *** lbragstad has quit IRC | 14:50 | |
| *** smarcet has quit IRC | 14:50 | |
| *** lbragstad has joined #openstack-infra | 14:53 | |
| *** jistr|mtg is now known as jistr | 14:55 | |
| ssbarnea | clarkb: mriedem: can you please help with this e-r query https://review.openstack.org/#/c/626962/ ? | 14:57 |
| ssbarnea | is quite an important source of problems | 14:57 |
| *** Goneri has joined #openstack-infra | 14:57 | |
| slaweq | hi infra team | 14:59 |
| mriedem | looking | 14:59 |
| slaweq | I'm checking why from time to time we have failed db migration tests in neutron, like e.g. http://logs.openstack.org/30/626830/9/check/neutron-functional/e66470c/logs/dsvm-functional-logs/neutron.tests.functional.db.test_migrations.TestModelsMigrationsMysql.test_check_mysql_engine.txt.gz | 14:59 |
| slaweq | it's always MySQL tests which are failing | 15:00 |
| slaweq | what I found is that during e.g. this failed test there was less than 100 IOPS for most of the time in dstat | 15:00 |
| slaweq | do You know about any performance issues which may leads to such issues? | 15:00 |
| fungi | slaweq: if those ran in limestone-regionone, logan- found yesterday that cpu passthrough wasn't enabled on some hosts there and corrected it | 15:04 |
| slaweq | fungi: yes, those which I found in last few days were in this region | 15:05 |
| fungi | slaweq: has it continued to happen today? | 15:06 |
| slaweq | fungi: last time I found it was today morning | 15:07 |
| slaweq | fungi: see http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22bug%201687027%5C%22 | 15:07 |
| *** udesale has quit IRC | 15:08 | |
| *** bhavikdbavishi has joined #openstack-infra | 15:10 | |
| fungi | thanks, that's good feedback. could indicate that there's something else happening in limestone-regionone independent of the cpu setting | 15:10 |
| *** bobh has joined #openstack-infra | 15:14 | |
| *** wolverineav has joined #openstack-infra | 15:15 | |
| *** sshnaidm has quit IRC | 15:15 | |
| *** Bhujay has joined #openstack-infra | 15:15 | |
| *** bhavikdbavishi has quit IRC | 15:16 | |
| *** Bhujay has quit IRC | 15:16 | |
| *** ykarel is now known as ykarel|away | 15:17 | |
| *** wolverineav has quit IRC | 15:20 | |
| *** armstrong has joined #openstack-infra | 15:20 | |
| *** efried has joined #openstack-infra | 15:25 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: add query for intermittent container failure https://review.openstack.org/626962 | 15:26 |
| *** sshnaidm has joined #openstack-infra | 15:26 | |
| *** anteaya has joined #openstack-infra | 15:28 | |
| *** e0ne has quit IRC | 15:28 | |
| *** quiquell is now known as quiquell|off | 15:28 | |
| *** whoami-rajat has quit IRC | 15:39 | |
| *** bhavikdbavishi has joined #openstack-infra | 15:41 | |
| *** xek_ has joined #openstack-infra | 15:46 | |
| *** xek has quit IRC | 15:48 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: Add new Sahara repositories for split plugins https://review.openstack.org/628209 | 15:55 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: Add new Sahara repositories for split plugins https://review.openstack.org/628209 | 15:56 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: Add new Sahara repositories for split plugins https://review.openstack.org/628209 | 16:00 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Extract out common config parsing for ConfigPool https://review.openstack.org/621642 | 16:05 |
| ssbarnea | mriedem: thanks for the e-r help! | 16:09 |
| *** sshnaidm has quit IRC | 16:15 | |
| corvus | Shrews, fungi: i'm digging through logs; i think it would be okay to dequeue that change now if you want. no stack dump necessary. | 16:19 |
| fungi | corvus: thanks. mwhahaha i'll dequeue it now | 16:21 |
| *** armax has joined #openstack-infra | 16:21 | |
| fungi | er, actually it doesn't seem to be queued any longer | 16:22 |
| fungi | ahh, weshay|ruck abandoned and restored it 12 minutes ago | 16:23 |
| clarkb | fungi: cmurphy I plan to start approving future parser changes again today now that Im mostly caught up post vacation | 16:23 |
| weshay|ruck | ya.. it seemed stuck | 16:23 |
| clarkb | the next set should be straightofrward | 16:24 |
| cmurphy | clarkb: awesome thanks | 16:24 |
| fungi | weshay|ruck: it was indeed stuck. we'd been discussing it in here with mwhahaha starting a couple hours aho | 16:24 |
| fungi | er, ago | 16:24 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: Add new Sahara repositories for split plugins https://review.openstack.org/628209 | 16:24 |
| weshay|ruck | sorry if I messed you guys up | 16:25 |
| cmurphy | clarkb: i think i have reviews up for all nodes except for the afs and kdc servers and the backup server | 16:25 |
| clarkb | cmurphy: we also need to fix the globs for list servers | 16:27 |
| openstackgerrit | Amy Marrich (spotz) proposed openstack-infra/irc-meetings master: Updated meeting time for D&I WG https://review.openstack.org/628214 | 16:27 |
| clarkb | current entries dont match hosts so noop | 16:27 |
| cmurphy | clarkb: lists[0-9]*.katacontainers.io and lists[0-9]*.open*.org is wrong? | 16:28 |
| *** sshnaidm has joined #openstack-infra | 16:28 | |
| clarkb | cmurphy: ya the current hosts don't have digits and since its a glob not a regex that means we must have at least one digit in the hostname suffix | 16:30 |
| cmurphy | oh oops | 16:31 |
| clarkb | if it was a regex it would be fine | 16:31 |
| cmurphy | yeah it's confusing that it has \d* in site.pp | 16:31 |
| fungi | right, in globspeak [0-9]* means a digit followed by anything, not zero or more digits | 16:31 |
| cmurphy | i assumed there were actually digits | 16:31 |
| *** sshnaidm_ has joined #openstack-infra | 16:32 | |
| clarkb | fungi: care to quickly review https://review.openstack.org/#/c/615966/4 and a handful of its children so I can get those through today? | 16:32 |
| fungi | \d* in the regex is future-proofing for zero or more digits which we can change to \d+ once there actually are | 16:32 |
| fungi | clarkb: sure! | 16:32 |
| *** sshnaidm has quit IRC | 16:35 | |
| *** sshnaidm__ has joined #openstack-infra | 16:35 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Fix glob for lists.katacontainers.io https://review.openstack.org/628216 | 16:36 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Fix glob for lists.o.o https://review.openstack.org/628217 | 16:36 |
| weshay|ruck | dmsimard, limestone http://logs.openstack.org/70/628070/1/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/01e7110/job-output.txt.gz#_2019-01-03_16_20_58_123909 | 16:36 |
| *** sshnaidm has joined #openstack-infra | 16:37 | |
| *** sshnaidm_ has quit IRC | 16:38 | |
| *** sshnaidm__ has quit IRC | 16:40 | |
| *** sshnaidm_ has joined #openstack-infra | 16:40 | |
| *** sshnaidm has quit IRC | 16:43 | |
| clarkb | fungi: slaweq fwiw the IO struggles haven't been specific to a single cloud. I do wonder if 1) we are our own noisy neighbor in many of these cases and 2) if kernel updates (or similar) have regressed performance somewhere between disks filesystems kvm qemu and our workloads | 16:45 |
| clarkb | it will be interesting to see if behavior like that changes with the bionic switch (either better or worse which may imply there is a kernel component) | 16:45 |
| fungi | agreed | 16:46 |
| *** auristor has quit IRC | 16:53 | |
| fungi | cmurphy: clarkb: i stopped short of the openstackid server changes since those are in flux | 16:54 |
| *** ginopc has quit IRC | 16:54 | |
| *** ykarel|away has quit IRC | 16:54 | |
| dmsimard | weshay|ruck: that occurred on a proper e5-2650 cpu so it wasn't it | 16:55 |
| cmurphy | fungi: you mean openstackid will be renamed? | 16:55 |
| *** ccamacho has quit IRC | 16:55 | |
| clarkb | fungi: ok. I'll be happy to get through the git server set today. But since I Got an early start may go further | 16:55 |
| fungi | cmurphy: no, but it's got puppeting disabled on the dev server while updates to the php framework are underway | 16:55 |
| cmurphy | fungi: ah okay | 16:56 |
| fungi | still, that's the next dozen-ish changes +2'd | 16:56 |
| clarkb | then tomorrow is likely going to be a paperwork administrivia start of the year day for me | 16:56 |
| *** auristor has joined #openstack-infra | 16:56 | |
| clarkb | fungi: ya should be plenty for today | 16:57 |
| *** dpawlik has quit IRC | 17:00 | |
| *** dpawlik has joined #openstack-infra | 17:00 | |
| *** auristor has quit IRC | 17:01 | |
| openstackgerrit | Merged openstack-infra/zuul master: Switch back to three columns for mid sized screens https://review.openstack.org/627997 | 17:02 |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on future parser for one git worker https://review.openstack.org/615966 | 17:04 |
| *** dpawlik has quit IRC | 17:05 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Fix glob for lists.katacontainers.io https://review.openstack.org/628216 | 17:07 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Fix glob for lists.o.o https://review.openstack.org/628217 | 17:07 |
| *** auristor has joined #openstack-infra | 17:08 | |
| *** pgaxatte has quit IRC | 17:12 | |
| *** janki has quit IRC | 17:13 | |
| *** larainema has quit IRC | 17:14 | |
| fungi | looks like we've got ~950 gerrit push tasks waiting to git@38.108.68.66 (new gitea server?) since 22:17:40 yesterday | 17:16 |
| clarkb | mriedem: I was thinking about sending an email re job and gate stability that basically says our data set is small now due to holidays, so please remember to check back in in a week or two. | 17:17 |
| clarkb | mriedem: anything else you know of we should call out? | 17:17 |
| mriedem | clarkb: no not really, haven't been watching it the past 2 weeks | 17:19 |
| mriedem | http://status.openstack.org/elastic-recheck/data/integrated_gate.html looks f'ed though | 17:19 |
| mriedem | Overall Categorization Rate: 0.0% | 17:19 |
| clarkb | ya but only 4 failures when I last looked | 17:19 |
| mriedem | yeah Total: 4 - Found: 0 = Unclassified: 4 | 17:20 |
| mriedem | i've seen more failures than that though, but maybe in non-integrated gate jobs | 17:20 |
| *** bobh has quit IRC | 17:21 | |
| *** lujinluo has joined #openstack-infra | 17:23 | |
| *** ykarel|away has joined #openstack-infra | 17:24 | |
| *** gyee has joined #openstack-infra | 17:25 | |
| slaweq | clarkb: ok, we have marked those tests as "unstable" for now so they are skipped instead of failing, but I will remove those markers and lets see how often it will happen. Also we will then see if that will happen more often when we move this job to bionic | 17:26 |
| slaweq | clarkb: fungi I will get back to You if it will happen more often again :) | 17:26 |
| clarkb | slaweq: thank you. | 17:26 |
| *** panda is now known as panda|off | 17:27 | |
| *** lujinluo has quit IRC | 17:28 | |
| fungi | who here groks gerrit's submodule handling? i think the upgrade to gerrit 2.13 has broken updating of openstack/openstack due to https://www.gerritcodereview.com/2.13.html#Access-control-for-git-submodule-subscriptions | 17:30 |
| fungi | i have a feeling we need to start manually setting all the submodules in https://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/acls/openstack/openstack.config | 17:31 |
| *** sshnaidm__ has joined #openstack-infra | 17:31 | |
| clarkb | I want to say mordred? | 17:31 |
| fungi | apparently the linux foundation tracks the activity in that repo and brought it to our attention | 17:32 |
| fungi | er, mentioned it to hogepodge who mentioned it to me and clarkb anyway | 17:32 |
| * hogepodge waves to everyone | 17:33 | |
| hogepodge | This was the link LF pointed me to. https://landscape.cncf.io/grouping=no&license=open-source&selected=open-stack&sort=latest-commi | 17:34 |
| *** sshnaidm_ has quit IRC | 17:34 | |
| hogepodge | They have an interactive landscape which tracks a lot of projects that have integrations with CNCF projects. | 17:34 |
| mordred | clarkb, fungi we also need to figure out updating the submodules in there | 17:36 |
| mordred | I pinged the release team about it a few months ago - but I don't think anyone has had a particularly great inspiration as to what the right way to do that is | 17:36 |
| clarkb | mordred: you mean updating the commits for each submodule or updating the list of submodules | 17:37 |
| mordred | list of submodules | 17:37 |
| fungi | the release team has a lot of metadata. they probably have some idea of what projects "participate" in the release and we could generate the list from that | 17:37 |
| clarkb | seems like for the list of submodules that could be part of the TC adds project to openstack step N is add it to list of submodules | 17:37 |
| mordred | yah - that's what I was thinking - because it shoudl be possible to have things properly included per-release | 17:37 |
| *** ykarel|away has quit IRC | 17:37 | |
| fungi | sure, putting all official openstack deliverables in there could be an alternative approach | 17:38 |
| mordred | clarkb: yeah - and removes it similarly ... and that somewhere in the release process we update the submodule subscriptions in a stable branch of openstack/openstack to point to the stable branches ... or something | 17:38 |
| mordred | fungi: I think we could do either - anything that participates, or anythiung that is official | 17:38 |
| mordred | either would be a form of correct | 17:39 |
| clarkb | considering the git hosting plans of the future I think we should focus on anything that is official for this | 17:39 |
| clarkb | and move away from being a catch all | 17:39 |
| mordred | ++ | 17:40 |
| *** jpich has quit IRC | 17:40 | |
| mordred | I believe that repo was originally "everything official" - it's just bitrotted | 17:40 |
| fungi | and now ceased updating entirely even thanks to non-backward-compatible changes in gerrit 2.13 | 17:41 |
| *** trown is now known as trown|lunch | 17:41 | |
| openstackgerrit | Chris Dent proposed openstack-infra/project-config master: Add publish-to-pypi template to placement https://review.openstack.org/628240 | 17:41 |
| mordred | fungi: yeah. although it seems like we can make one change to All-Projects and be back in business | 17:42 |
| fungi | is that where it's set? | 17:42 |
| fungi | not in the project acl? | 17:42 |
| clarkb | ya maybe we should get the current list working again then sort out the submodule list separately | 17:42 |
| mordred | we don't have All-Projects in our gerrit/acls dir yet do we? | 17:43 |
| clarkb | mordred: no | 17:43 |
| clarkb | because you need all-projects in place in order to manage it with jeepyb (which uses gerrit/acls) | 17:43 |
| clarkb | so its chicken and eggy and ya | 17:43 |
| mordred | nod. | 17:43 |
| clarkb | also we may not want all git repo submodules to auto update? | 17:43 |
| clarkb | so probably should go in openstack/openstack's config? | 17:43 |
| fungi | according to https://groups.google.com/forum/#!msg/repo-discuss/KzLJiNqu2AM/1ppoBsYaCgAJ wmf needed to add both allow and subscribe entries for every submodule | 17:44 |
| mordred | no - it's an all-projects setting we need to do - or we need to do a setting on each submodule repo | 17:44 |
| mordred | if we put: | 17:44 |
| *** panda|off has quit IRC | 17:44 | |
| mordred | [allowSuperproject "my-only-superproject"] | 17:44 |
| mordred | matching = refs/heads/*:refs/heads/* | 17:44 |
| mordred | into All Projects - (setting "my-only-superproject" to openstack/openstack) ... then we should be good to go to have openstack/openstack subscribe to whichever projects it wants to | 17:45 |
| fungi | but we still need to put the subscribe entries in the openstack/openstack acl? | 17:45 |
| clarkb | mordred: ah then other repos would manage submodules normally | 17:45 |
| fungi | (and is that in addition to the submodule entries in its .gitmodules file?) | 17:45 |
| mordred | yah | 17:46 |
| clarkb | fungi: reading the docs I don't think so | 17:46 |
| clarkb | "Gerrit will only automatically update superprojects where the submodules are hosted on the same Gerrit instance as the superproject. Gerrit determines this by checking that the URL of the submodule specified in the .gitmodules file starts with gerrit.canonicalWebUrl. The protocol part is ignored in this check." | 17:46 |
| mordred | yah- only the all projects for acls - but we need to put the submodule entries in | 17:46 |
| clarkb | I think ^ means that if we set up the superproject then any submodule in the repo itself will just work with that rule | 17:46 |
| *** panda has joined #openstack-infra | 17:48 | |
| clarkb | I'm +2 on adding that to all projects which gets things working for existing submodules | 17:48 |
| *** ykarel|away has joined #openstack-infra | 17:48 | |
| clarkb | then we can followup with TC/releaseteam/whoever on how they want to manage the list of submodules | 17:48 |
| mordred | fungi, clarkb: http://paste.openstack.org/show/739880/ | 17:48 |
| mordred | I cloned All-Projects, checked out the meta config ref, and made that chnage to project.config file | 17:49 |
| mordred | does that look correct to you? | 17:49 |
| clarkb | mordred: refs/meta/config doesn't look right as the refs | 17:49 |
| clarkb | refs/heads/*:refs/heads/* is what it should be right? | 17:50 |
| mordred | clarkb: thanks - total bad copy-pasta | 17:50 |
| mordred | http://paste.openstack.org/show/739881/ | 17:51 |
| fungi | seems reasonable. so what needs to happen with the [subscribe] sections mentioned in that wmf bug/discussion? | 17:51 |
| clarkb | fungi: aiui nothing | 17:51 |
| clarkb | fungi: as long as our submodules match the rules in the docs quote above | 17:51 |
| mordred | yah | 17:52 |
| fungi | oh, maybe they have submodules outside their gerrit? | 17:52 |
| clarkb | ya could be | 17:52 |
| clarkb | mordred: newer diff lgtm, also gerrit will reject invalid syntax so pretty safe to push I think | 17:53 |
| mordred | ok. cool. | 17:53 |
| fungi | so we should still be able to manage submodules for openstack/openstack by pushing updates to its .gitmodules file? | 17:53 |
| *** ykarel|away has quit IRC | 17:54 | |
| clarkb | http://git01.openstack.org:8080/cgit/openstack/nova is still working after the future parser applied to that host | 17:54 |
| clarkb | I'm going to approve the chagne to update all our git backends to use future parser now | 17:54 |
| clarkb | fungi: yes as long as we host teh submodules on the gerrit too | 17:54 |
| *** aojea has quit IRC | 17:54 | |
| fungi | and as an aside, what's the process for updating .gitmodules? do we just push directly to gerrit bypassing change review? or push a review and then manually submit it to merge? | 17:54 |
| *** lewo has quit IRC | 17:55 | |
| clarkb | I've always done it as a change (usually updating the gerrit repos submodules) | 17:55 |
| mordred | https://review.openstack.org/#/c/628242/ | 17:55 |
| fungi | oh, i tried doing that (a few years ago i think?) and someone told me that wasn't how it worked | 17:55 |
| clarkb | mordred: lgtm but I don't have voting rights so someone will have to project creator that in or push it directly | 17:56 |
| mordred | following the instructions got me to creating that - although I think just pushing it directly might make more sense, since we don't have review acls | 17:56 |
| mordred | yeah | 17:56 |
| mordred | how about I just push directly | 17:56 |
| mordred | hah | 17:57 |
| fungi | https://review.openstack.org/318661 | 17:57 |
| clarkb | fungi: https://review.openstack.org/#/c/480644/ is an example submodule update | 17:57 |
| mordred | remote: You are not allowed to perform this operation. | 17:57 |
| fungi | was my abandoned attempt | 17:57 |
| mordred | remote: Configuration changes can only be pushed by project owners | 17:57 |
| mordred | remote: who also have 'Push' rights on refs/meta/config | 17:57 |
| mordred | remote: User: mordred | 17:57 |
| mordred | remote: Please read the documentation and contact an administrator | 17:57 |
| mordred | remote: if you feel the configuration is incorrect | 17:57 |
| mordred | remote: Processing changes: refs: 1, done | 17:57 |
| mordred | I guess I need to put myself in bootstrappers first | 17:57 |
| clarkb | mordred: ya | 17:58 |
| clarkb | and if you don't --force it should be safe | 17:58 |
| mordred | k. done | 17:58 |
| *** gfidente has quit IRC | 17:58 | |
| fungi | clarkb: any idea what about 318661 led me to be told it was not the correct way? | 17:58 |
| clarkb | fungi: no, maybe we had a different set of rules / expectations for the super project? | 17:59 |
| fungi | i'll see if i can find it in my irc log archive | 17:59 |
| mordred | also - we should update the urls in the file to be relative paths | 17:59 |
| clarkb | mordred: note that is what we undo with the gerrit submodules because we don't fork the submodules, only gerrit tiself | 18:00 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: Add new Sahara repositories for split plugins https://review.openstack.org/628209 | 18:02 |
| mordred | yeah. but this is only an informational repo, so that shouldn't be an issue for folks - but it should hopefully cause mirrors like github or gitea or whatnot to be more able to produce working hyperlinks? | 18:02 |
| clarkb | oh ya | 18:03 |
| mordred | clarkb, fungi: oh! that generate gitmodules script could probably just be updated to read projects.yaml from governance | 18:03 |
| mordred | http://git.openstack.org/cgit/openstack/openstack/tree/generate-gitmodules | 18:03 |
| clarkb | ++ | 18:03 |
| *** derekh has quit IRC | 18:03 | |
| * mordred makes patch | 18:05 | |
| fungi | also, it looks like the reason i abandoned 318661 was that we set the acl to only allow the release managers to push http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-07-07.log.html#t2016-07-07T14:57:17 | 18:07 |
| hogepodge | Thank you everybody! | 18:08 |
| clarkb | mordred: we should also double check that the superproject magic works on relative paths. The docs say it checks if the url starts with the canonical web url for gerrit which may break with relative paths | 18:09 |
| clarkb | (this is somethign we can test) | 18:09 |
| fungi | hogepodge: thanks for bringing to our attention that we neglected to update our configuration for a gerrit upgrade we performed 8 months ago ;) | 18:09 |
| *** jamesmcarthur has joined #openstack-infra | 18:10 | |
| *** diablo_rojo has joined #openstack-infra | 18:11 | |
| fungi | hrm, though actually the gerrit upgrade to 2.13 wasn't at the beginning of may, the gerrit server upgrade from trusty to xenial was | 18:12 |
| fungi | i wonder why that seems to have been what precipitated the cessation of updates for the superrepo | 18:12 |
| clarkb | new java version behavior possibly? | 18:12 |
| *** bnemec has quit IRC | 18:13 | |
| fungi | not sure... seems like based on what we've found configuration-wise the 2.13 update should have stopped it... | 18:14 |
| mordred | remote: https://review.openstack.org/628246 Update generate-gitmodules script for official deliverables | 18:14 |
| mordred | remote: https://review.openstack.org/628247 Update git submodules based on new script | 18:14 |
| fungi | well, at any rate, if this gets it flowing again i don't care ;) | 18:14 |
| mordred | clarkb: the docs go on to say that you can use relative paths | 18:14 |
| mordred | https://gerrit-documentation.storage.googleapis.com/Documentation/2.13/user-submodules.html#_relative_submodules | 18:14 |
| *** diablo_rojo has quit IRC | 18:15 | |
| clarkb | mordred: cool | 18:15 |
| clarkb | fungi: ya we should know soon enough if this is the fix | 18:15 |
| *** diablo_rojo has joined #openstack-infra | 18:15 | |
| mordred | we could probably make a zuul job to trigger on patches landed to projects.yaml that does a proposal bot submission | 18:15 |
| mordred | clarkb: looks like we had it set up to do relative paths already in the gitmodules file - just not in the script | 18:17 |
| clarkb | ah | 18:18 |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on future parser for all git workers https://review.openstack.org/615967 | 18:23 |
| mordred | clarkb: hrm. I donm't think that's gonna fully do it - I think we need to do a submodule add first - updates coming | 18:23 |
| *** jamesmcarthur has quit IRC | 18:25 | |
| *** whoami-rajat has joined #openstack-infra | 18:25 | |
| *** bnemec has joined #openstack-infra | 18:26 | |
| *** jamesmcarthur has joined #openstack-infra | 18:28 | |
| *** jamesmcarthur has quit IRC | 18:28 | |
| *** jamesmcarthur has joined #openstack-infra | 18:29 | |
| *** armstrong has quit IRC | 18:30 | |
| *** smarcet has joined #openstack-infra | 18:30 | |
| smarcet | fungi: afternoon and happy new year ! could u review this one https://review.openstack.org/#/c/625691/ ? thx ! | 18:30 |
| *** bnemec has quit IRC | 18:30 | |
| fungi | smarcet: happy new year to you as well! we should probably move those laravel jobs into the openstackid repo at some point and get them out of the openstack-zuul-jobs repo, but the nodeset change there is fairly unobtrusive for now | 18:33 |
| smarcet | fungi: yes i know, its on my list, just wanna to speed up the migration to php 7.x and then will tacke that :) | 18:34 |
| *** bnemec has joined #openstack-infra | 18:34 | |
| fungi | sounds great, thanks! | 18:37 |
| clarkb | next run will apply 615967, about ~40 minutes away? | 18:38 |
| clarkb | (I'm watching those changes as I approve them) | 18:38 |
| *** bnemec has quit IRC | 18:39 | |
| *** jamesmcarthur has quit IRC | 18:40 | |
| *** jamesmcarthur has joined #openstack-infra | 18:40 | |
| *** ramishra has quit IRC | 18:43 | |
| *** dtantsur is now known as dtantsur|afk | 18:43 | |
| *** wolverineav has joined #openstack-infra | 18:52 | |
| *** bnemec has joined #openstack-infra | 18:52 | |
| *** bnemec has quit IRC | 18:58 | |
| *** wolverineav has quit IRC | 18:59 | |
| *** trown|lunch is now known as trown | 19:00 | |
| mordred | clarkb, fungi: remote: https://review.openstack.org/628246 Update generate-gitmodules script for official deliverables | 19:01 |
| mordred | remote: https://review.openstack.org/628247 Update git submodules based on new script | 19:01 |
| mordred | updated - this time using submodule add properly | 19:01 |
| mordred | fungi: ++ to your suggestion about reworking to use openstack_governance/projects.py ... | 19:02 |
| clarkb | mordred: I only have +/-1 on that | 19:02 |
| clarkb | do we need to have release team review? | 19:02 |
| mordred | yeah. I think so | 19:02 |
| mordred | I kinda think we should actuallky move the script to governance repo - and set up a proposal job there in post ... but this should get us somewhere for today | 19:03 |
| clarkb | fungi: mordred http://git.openstack.org/cgit/openstack/openstack/log/ fyi it seems to be doing things again now | 19:07 |
| clarkb | hogepodge: ^ you too I guess | 19:07 |
| mordred | woot | 19:07 |
| clarkb | (we aren't going to replay the lost 8 months of history though | 19:07 |
| mordred | clarkb, hogepodge: I just pinged folks in #openstack-release | 19:07 |
| clarkb | I suppose someone could create that but it would liekly be more effort than its worth, but if you do it we can push a merge commit to the repo | 19:07 |
| fungi | clarkb: deleting that repo should re-replicate it all, right? | 19:07 |
| mordred | we should maybe stick an apache license file in that repo too | 19:08 |
| clarkb | fungi: I cloned from gerrit directly and the interim content doesn't seem to exist | 19:08 |
| fungi | huh, interesting | 19:08 |
| mordred | no - gerrit adds commits for each submodule commit | 19:08 |
| fungi | ohhh | 19:08 |
| mordred | we'd have to script *thousands* of commits | 19:08 |
| mordred | I donm't think it's worth it tbh | 19:08 |
| fungi | agreed | 19:08 |
| clarkb | mordred: someone would have to then push a merge commit. I don't think we need to :) | 19:08 |
| clarkb | and ya not worth it | 19:09 |
| mordred | clarkb, fungi: https://review.openstack.org/628269 | 19:10 |
| mordred | as well | 19:10 |
| *** bnemec has joined #openstack-infra | 19:14 | |
| clarkb | mordred: left a bunch of comments on https://review.openstack.org/#/c/628246/2 I don't think individualy any one demands a -1, but together may consider reworking it? | 19:18 |
| clarkb | I +0 | 19:18 |
| clarkb | s | 19:18 |
| clarkb | er | 19:18 |
| clarkb | 0 | 19:18 |
| clarkb | bah | 19:18 |
| *** wolverineav has joined #openstack-infra | 19:20 | |
| *** wolverineav has quit IRC | 19:21 | |
| *** wolverineav has joined #openstack-infra | 19:21 | |
| *** wolverineav has quit IRC | 19:21 | |
| *** wolverineav has joined #openstack-infra | 19:21 | |
| mordred | clarkb: I agree - updated based on your feedback | 19:21 |
| *** mriedem has quit IRC | 19:22 | |
| *** mriedem has joined #openstack-infra | 19:24 | |
| clarkb | +1 now | 19:24 |
| *** irclogbot_1 has quit IRC | 19:24 | |
| *** bhavikdbavishi has quit IRC | 19:25 | |
| clarkb | git backends look happy | 19:26 |
| *** irclogbot_1 has joined #openstack-infra | 19:26 | |
| clarkb | going to approve change for the load balancer now | 19:26 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Update openstackid jobs https://review.openstack.org/625691 | 19:30 |
| *** bobh has joined #openstack-infra | 19:37 | |
| *** bobh has quit IRC | 19:41 | |
| clarkb | cmurphy: fungi: uhm puppet on git.openstack.org is version 4 | 19:50 |
| clarkb | which means ansible role puppet is failing to puppet that node beacuse /etc/puppetlabs/* doesn't exist | 19:50 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Pin bundler in puppet-lint and -syntax jobs https://review.openstack.org/628283 | 19:51 |
| clarkb | wow on git backends we install puppet from puppetlabs | 19:51 |
| clarkb | (good) | 19:51 |
| clarkb | on git.o.o we install it from the rdo ocata repo for some reason | 19:52 |
| cmurphy | clarkb: oh that is surprising | 19:52 |
| clarkb | so I think the fix here is remove the ocata repo, reinstall puppet, then puppet will be 3 as epxected | 19:52 |
| cmurphy | yes | 19:52 |
| clarkb | ok I'm not going to surgery that before lunch | 19:52 |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on future parser for git load balancer https://review.openstack.org/615968 | 19:53 |
| clarkb | maybe someone else a bit more familiar with centos wants to double check ^ for me | 19:53 |
| clarkb | I'm going to afk for a bit for lunch and some quick errands | 19:53 |
| clarkb | maybe there is a way to audit other packages that come from that repo? | 19:54 |
| clarkb | so that we have an idea of what will be changing? I don't know how to do that without digging into docs | 19:54 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-pip master: Fix openstack_pip provider for pip 18 https://review.openstack.org/606021 | 19:54 |
| fungi | i wonder if we added it to get newer haproxy or something? | 19:56 |
| cmurphy | might be easier to just switch directly to puppet-4 from the puppetlabs repo | 19:58 |
| clarkb | fungi: I doubt it. I think we added that repo for our test images when they puppeted and this leaked across | 19:58 |
| *** smarcet has quit IRC | 19:58 | |
| clarkb | the when rdo updated puppet versions we got it? | 19:58 |
| *** smarcet has joined #openstack-infra | 19:59 | |
| fungi | ahh | 20:00 |
| fungi | yeah, i'd buy that 'splanation | 20:00 |
| clarkb | we should also double check all git backends. I've only looked at 01 and 02 so far | 20:03 |
| *** smarcet has quit IRC | 20:07 | |
| *** smarcet has joined #openstack-infra | 20:11 | |
| *** bobh has joined #openstack-infra | 20:12 | |
| fungi | a quick ssh loop says puppet-3.8.7-1.el7.noarch is installed on all 8 git servers | 20:12 |
| fungi | no variations | 20:13 |
| *** smarcet has quit IRC | 20:15 | |
| *** ccamacho has joined #openstack-infra | 20:20 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: Add new Sahara repositories for split plugins https://review.openstack.org/628209 | 20:20 |
| *** e0ne has joined #openstack-infra | 20:21 | |
| *** ccamacho has quit IRC | 20:22 | |
| clarkb | `yum list installed` shows you packages and where they came from. Its not just puppet in the list from rdo | 20:30 |
| clarkb | I think we disable CentOS-OpenStack-ocata.repo CentOS-Ceph-Jewel.repo CentOS-QEMU-EV.repo and CentOS-Storage-common.repo repos, then do a yum update to realign everything? | 20:30 |
| *** smarcet has joined #openstack-infra | 20:30 | |
| clarkb | dmsimard: ^ would you expect that to work? | 20:30 |
| clarkb | fungi: ^ I'm not quite back yet. Just poking at this quickly between other tasks, but I think that is what I'll go with | 20:32 |
| fungi | yeah, your guess is as good as mine there | 20:32 |
| fungi | most of my red hat distro knowledge hails from the time when you still directly invoked the rpm command-line for all your package management needs | 20:33 |
| fungi | and indeed, i used `rpm -q puppet` to find out what version was installed on the git servers :0 | 20:33 |
| *** apetrich has quit IRC | 20:33 | |
| *** whoami-rajat has quit IRC | 20:35 | |
| *** smarcet has quit IRC | 20:36 | |
| *** Vadmacs has quit IRC | 20:36 | |
| *** e0ne has quit IRC | 20:37 | |
| clarkb | heh if you disable the repo you can't reinstall a package for it to find the package in another repo? | 20:39 |
| clarkb | (I'm almost inclined to rebuild this host instead of figuring out yum) | 20:39 |
| clarkb | infra-root http://paste.openstack.org/show/739885/ seems to be the thing we want | 20:46 |
| clarkb | note this will only change puppet and not the other packages installed by the rdo pacakge repo(s) | 20:46 |
| clarkb | should I go ahead and say yes to the prompt? | 20:46 |
| *** apetrich has joined #openstack-infra | 20:46 | |
| mordred | clarkb: yeah. I think that looks good | 20:46 |
| clarkb | we can pick off the other packages as we like or potentially just reinstall the host later? | 20:47 |
| clarkb | I'll hit y at the prompt now | 20:47 |
| fungi | lgtm, thanks! | 20:47 |
| fungi | i think this is fine to get past the current situation at least | 20:48 |
| *** psachin has quit IRC | 20:48 | |
| clarkb | puppet --version reports the correct version now and the install seemed to run without unexpected behavior | 20:48 |
| fungi | package downgrades are always scary because it's hard to build in the ability to undo actions which arbitrary future versions might take | 20:49 |
| clarkb | ya though looking at the list I think its likely relatively safe to do so for the packages that leaked across | 20:50 |
| clarkb | some mariadb client and lib stuff, some python2 libs, and some ceph client/lib stuff | 20:51 |
| clarkb | we don't consume any of that stuff on this host so should be able to reconcile it pretty easily or just ignore them because we don't use it and a rebuild or gitea will clean it up | 20:51 |
| *** jamesmcarthur has quit IRC | 20:53 | |
| *** e0ne has joined #openstack-infra | 20:54 | |
| *** bobh has quit IRC | 20:55 | |
| fungi | seems reasonable, agreed | 20:57 |
| clarkb | skimming through some rpm -q --whatrequires foo output I'm somewhat inclined to just leave things as they are | 20:57 |
| fungi | i think ignoring is fine until we discover an issue which demands we fix another one of those packages | 20:58 |
| fungi | yep | 20:58 |
| clarkb | I don't think the overlap with what the server actually does is really there | 20:58 |
| clarkb | cmurphy: http://paste.openstack.org/show/739887/ that looks ok for today | 20:59 |
| clarkb | but definitely some cleanup if this ever goes to puppet 4 | 21:00 |
| clarkb | ok going to go back to lunch/errands after being sufficiently distracted by this :) | 21:02 |
| clarkb | I've also approved the mirror update change since thats a low impact change if it goes wrong | 21:02 |
| *** jamesmcarthur has joined #openstack-infra | 21:03 | |
| cmurphy | clarkb: it should be fine going to puppet 4, those warnings are for pre-puppet-5 and the haproxy module | 21:03 |
| clarkb | cmurphy: ah so future parser saying next major release means 5 not 4? | 21:04 |
| clarkb | I guess that makes sense as futureparser ~= 4 | 21:04 |
| cmurphy | yeah it's treating it as if it was puppet 4 | 21:04 |
| *** xek_ has quit IRC | 21:07 | |
| *** jamesmcarthur has quit IRC | 21:08 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Remove unecessary finally clauses https://review.openstack.org/628299 | 21:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix items stuck in queue pending node requests https://review.openstack.org/628300 | 21:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Be more aggressive in canceling node requests https://review.openstack.org/628301 | 21:08 |
| clarkb | cmurphy: https://review.openstack.org/#/c/615998/6 comment on that one as I'm going through some of the list past the git servers | 21:09 |
| clarkb | cmurphy: basically the dns servers for opendev.org are ansible only without puppet. So we should be explicit there I think | 21:09 |
| cmurphy | clarkb: okay i agree with that | 21:10 |
| cmurphy | clarkb: is it just the dns servers or are there others that are different between opendev and openstack? | 21:11 |
| clarkb | cmurphy: just the dns servers so far. | 21:11 |
| cmurphy | ok | 21:12 |
| *** Qiming has quit IRC | 21:20 | |
| *** Qiming has joined #openstack-infra | 21:22 | |
| *** jamesmcarthur has joined #openstack-infra | 21:28 | |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on future parser for mirror-update.o.o https://review.openstack.org/615991 | 21:28 |
| mordred | infra-root: I'm going to be offline for the next little bit - the magical human has arrived to bring me more network packets | 21:29 |
| mordred | in theory, I should wind up having 1G of them after the magical human is done | 21:33 |
| *** kgiusti has left #openstack-infra | 21:33 | |
| mordred | in practice, of course, who the heck know | 21:34 |
| fungi | moar packets! | 21:35 |
| *** bobh has joined #openstack-infra | 21:38 | |
| mordred | I just hope I have enough closet space to store them all in | 21:39 |
| corvus | that's the problem, you have to convince them to take the packets away too. they often aren't as excited about that. | 21:39 |
| mordred | I can never seem to get rid of all of my excess packets | 21:39 |
| corvus | you should definitely have that looked at | 21:39 |
| mordred | although I hear I might get me some of the ipv6 too - so maybe that'll help | 21:40 |
| clarkb | from frontier? | 21:42 |
| clarkb | what magic words do I use for this? | 21:42 |
| mordred | clarkb: no - from spectrum | 21:42 |
| *** jamesmcarthur has quit IRC | 21:43 | |
| *** jamesmcarthur has joined #openstack-infra | 21:48 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Cache empty branch config to prevent spurious reconfig https://review.openstack.org/628307 | 21:48 |
| *** e0ne has quit IRC | 21:50 | |
| smcginnis | fungi: Thanks for the Review-Priority sanity check. | 21:51 |
| fungi | np | 21:52 |
| *** e0ne has joined #openstack-infra | 21:52 | |
| *** jamesmcarthur has quit IRC | 21:53 | |
| *** e0ne has quit IRC | 21:57 | |
| corvus | clarkb: https://review.openstack.org/628299 failed due to mysql being slow, on limestone. | 22:05 |
| clarkb | corvus: I think fungi and slaweq were looking at that this morning? | 22:06 |
| clarkb | I'm not sure if logan- has seen though | 22:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Be more aggressive in canceling node requests https://review.openstack.org/628301 | 22:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Cache empty branch config to prevent spurious reconfig https://review.openstack.org/628307 | 22:08 |
| clarkb | it is somewhat curious to me that multiple clouds start exhibiting that behavior together, but the ovh side didn't seem to think it was openstack related | 22:08 |
| clarkb | logan-: in the ovh case it was hypervisors swapping (we think) anything obivous like that on your end? | 22:09 |
| *** slaweq has quit IRC | 22:11 | |
| *** trown is now known as trown|outtypewww | 22:13 | |
| *** bobh has quit IRC | 22:30 | |
| *** efried has quit IRC | 22:35 | |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on the future parser for rax mirror https://review.openstack.org/615992 | 22:35 |
| *** efried has joined #openstack-infra | 22:35 | |
| clarkb | The call for PTG participation in denver is going out now. I'll add it to the meeting agenda, but wanted to let people know to think about that | 22:38 |
| *** efried has quit IRC | 22:39 | |
| timburke | fyi, i'm reviving https://review.openstack.org/#/c/212824 -- it looks like it caused some trouble in the gate in the past? but the traceback AJaeger linked to (http://paste.openstack.org/show/488308/) seemed to be bubbling up out of *jenkins*; i'm not sure how relevant that is anymore... | 22:41 |
| *** rcernin has joined #openstack-infra | 22:42 | |
| timburke | i'll try to keep an eye on the in-progress jobs and push up another patchset with some early returns in the problematic tests if it seems to be spinning, but i wanted to give you guys a heads up, in case it starts eating nodes again and i miss it | 22:43 |
| clarkb | timburke: thanks, our jobs should have a 3 hour timeout max though it is lower for the python unittest jobs | 22:43 |
| clarkb | timburke: so you may see them timeout more aggressively now | 22:43 |
| clarkb | other than that should be fine | 22:43 |
| timburke | if it takes 3 hrs, my patch has other problems ;-) | 22:45 |
| *** jcoufal has quit IRC | 22:49 | |
| *** boden has quit IRC | 22:53 | |
| *** apetrich has quit IRC | 22:56 | |
| openstackgerrit | Merged openstack-infra/zuul master: Remove unecessary finally clauses https://review.openstack.org/628299 | 23:00 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix items stuck in queue pending node requests https://review.openstack.org/628300 | 23:00 |
| openstackgerrit | Merged openstack-infra/zuul master: Be more aggressive in canceling node requests https://review.openstack.org/628301 | 23:03 |
| openstackgerrit | Merged openstack-infra/zuul master: Cache empty branch config to prevent spurious reconfig https://review.openstack.org/628307 | 23:03 |
| corvus | it would not be a bad idea for us to restart with that -- zuul is performing a lot of extra reconfigs | 23:04 |
| *** dustinc has joined #openstack-infra | 23:04 | |
| clarkb | ++ I'm around though trying to keep track of puppet things | 23:05 |
| corvus | i can do it in a bit when i finish up something | 23:05 |
| * mordred has internets! | 23:09 | |
| anteaya | welcome back | 23:10 |
| anteaya | do the extra packets fit? | 23:10 |
| mordred | also - props to the installer dude, I pointed at my router and said "I want the internet to come in to that" and he totally understood | 23:10 |
| mordred | which shouldn't be, you know, outstanding - but it still is :) | 23:10 |
| *** rascasoft has quit IRC | 23:11 | |
| mordred | anteaya: I think so? we'll have to see if they accumulate under the sofa or anything | 23:11 |
| clarkb | mordred: I got cat5 installed without even asking | 23:11 |
| corvus | clarkb: cat5e i hope? | 23:11 |
| clarkb | corvus: its definitely not 6 and its only 100/100 so not sure actually (I've not bothered to check and can run my own cable if I need to | 23:12 |
| clarkb | corvus: but its not docsis or moca over a coax cable | 23:12 |
| corvus | oh yeah i guess if you only have a decigigabit it's not important | 23:12 |
| corvus | mordred has *10* decigigabits | 23:13 |
| *** dustinc has quit IRC | 23:13 | |
| clarkb | ya matters more for mordred | 23:13 |
| mordred | yes! we actually speed tested 934 | 23:13 |
| *** apetrich has joined #openstack-infra | 23:13 | |
| mordred | so - yay for uplink speed - and yay for router routing it | 23:13 |
| *** dustinc_ has joined #openstack-infra | 23:13 | |
| mordred | I've also got the nice new shiny cat6 lines run from the main switch to 2 of the 3 leaf switches too (the 3rd involves attic crawling, so is currently being defered) | 23:14 |
| clarkb | I also said things like "spoof my mac address" when he was concerned that testing the connection on my laptop would later cause problems when I hooked up my router | 23:14 |
| clarkb | mordred: next is 10Gbe for local devices? :P | 23:15 |
| corvus | i ran 5e instead of 6, because it involved so many tight places, and the spool was $10 cheaper. | 23:15 |
| *** dustinc_ is now known as dustinc | 23:15 | |
| mordred | corvus: yeah - I went ahead and got the 6 spool because bigger numbers - it's almost certainly a waste of the extra $10 :) | 23:17 |
| mordred | clarkb: if only I had something that could produce 10Gbe of traffic for a local device to consume :) | 23:17 |
| *** jamesmcarthur has joined #openstack-infra | 23:20 | |
| corvus | pabelanger: there's a constant stream of wrong password errors on the pbx; is there a way to disable those from being printed to the console? | 23:21 |
| *** yamamoto has joined #openstack-infra | 23:29 | |
| *** hamerins has joined #openstack-infra | 23:33 | |
| *** owalsh_ has joined #openstack-infra | 23:38 | |
| *** hamerins has quit IRC | 23:40 | |
| *** owalsh has quit IRC | 23:40 | |
| *** owalsh_ has quit IRC | 23:42 | |
| *** owalsh has joined #openstack-infra | 23:43 | |
| *** jcoufal has joined #openstack-infra | 23:47 | |
| *** jcoufal has quit IRC | 23:47 | |
| clarkb | I've approved the change to future parser files*.o.o. I've reviewed the manifests to double check it looks safe and ya its a vhost that appears to properly look up vars applied several times with different values | 23:48 |
| clarkb | (mostly a heads up) | 23:48 |
| *** rascasoft has joined #openstack-infra | 23:49 | |
| fungi | i picked up shielded cat5e with plenum jacketing for here, wasn't too expensive by the 1kft spool, though i don't recall exactly what i paid | 23:51 |
| fungi | shielded because i don't trust what the conduits are running near, and plenum jacket because, well it's going inside walls/ceilings and so there's fire code to consider | 23:52 |
| corvus | i'll restart the scheduler now | 23:54 |
| *** bobh has joined #openstack-infra | 23:54 | |
| *** rascasoft has quit IRC | 23:55 | |
| *** dustinc has quit IRC | 23:57 | |
| corvus | #status log restarted zuul scheduler at 2fd688352f5e220fda0dfc72b164144910670d95 | 23:57 |
| openstackstatus | corvus: finished logging | 23:57 |
| *** mriedem has quit IRC | 23:58 | |
| clarkb | ui shows that its up again | 23:59 |
| clarkb | is that quicker than last time? | 23:59 |
| corvus | a little yeah; probably had the good luck to run the nova cat jobs on a smaller repo | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!