| *** eernst_ has quit IRC | 00:03 | |
| *** mattw4 has quit IRC | 00:07 | |
| *** HenryG has joined #openstack-infra | 00:07 | |
| *** jamesmcarthur has quit IRC | 00:08 | |
| *** sthussey has quit IRC | 00:09 | |
| *** eernst has joined #openstack-infra | 00:09 | |
| *** eernst has quit IRC | 00:11 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** eernst has joined #openstack-infra | 00:11 | |
| *** yamamoto has joined #openstack-infra | 00:12 | |
| *** eernst_ has joined #openstack-infra | 00:13 | |
| *** eernst has quit IRC | 00:13 | |
| fungi | hrm, all the executors *except* ze10 have git sha a13d922 installed now | 00:13 |
|---|---|---|
| fungi | ah, no, it does too but also seems to have pbr installed under python2 for some strange reason so it's the only one where `pbr freeze` doesn't list zuul | 00:14 |
| *** eernst_ has quit IRC | 00:17 | |
| *** markvoelker has quit IRC | 00:19 | |
| fungi | anyone object to me restarting executors? | 00:19 |
| fungi | looks like they're all updated with the regression revert now | 00:20 |
| *** yamamoto has quit IRC | 00:20 | |
| *** jamesmcarthur has joined #openstack-infra | 00:20 | |
| clarkb | no objection from me | 00:20 |
| *** slaweq has quit IRC | 00:24 | |
| fungi | looks like in the past i've done this: | 00:25 |
| fungi | sudo ansible ze*.openstack.org -m shell -a 'systemctl stop zuul-executor' | 00:25 |
| fungi | and then similar for start after confirming they all died | 00:25 |
| fungi | is there a preferred method lately? | 00:26 |
| clarkb | I think that is it if not using the full restart playbook | 00:26 |
| clarkb | and we dont want a full restart as that will undo the reverts for the memory leak | 00:26 |
| fungi | yep | 00:27 |
| fungi | okay, will get it underway in a moment | 00:27 |
| *** ijw has quit IRC | 00:29 | |
| *** bobh has joined #openstack-infra | 00:36 | |
| *** jamesmcarthur has quit IRC | 00:36 | |
| fungi | all have stopped except ze01 and ze12 now | 00:37 |
| *** bobh has quit IRC | 00:37 | |
| *** hwoarang has quit IRC | 00:37 | |
| fungi | now only ze01 remains | 00:41 |
| *** hwoarang has joined #openstack-infra | 00:41 | |
| fungi | and all clear. starting them all now | 00:42 |
| fungi | looks like they've all stabilized now | 00:53 |
| fungi | oddly, our mergers graph suggests only the built-ins from our executors are registered | 00:53 |
| fungi | dropped from 12 to 0 when i stopped the executors, and then went back to 12 when i started them again | 00:54 |
| fungi | is that expected? | 00:54 |
| corvus | did i forget to restart them...? | 00:54 |
| corvus | yeah, i think that's what happened, sorry | 00:54 |
| corvus | fungi: do you still have a bridge shell? you want to restart them? or should i? | 00:54 |
| fungi | yeah, graph says it dropped from 20 to 12 late on the 23rd | 00:55 |
| corvus | and systemd doesn't report any activity after that | 00:55 |
| fungi | i can start them, no problem | 00:56 |
| corvus | fungi: thx | 00:56 |
| fungi | did: sudo ansible zm*.openstack.org -m shell -a 'systemctl start zuul-merger' | 00:56 |
| fungi | still not running though. looking closer at one | 00:58 |
| fungi | Active: active (exited) since Tue 2019-04-23 19:15:28 UTC; 2 days ago | 00:58 |
| fungi | hrm... | 00:58 |
| fungi | doing a stop on them all and then will start again | 00:59 |
| fungi | okay, that seems to have done the trick | 00:59 |
| corvus | i'm sure that makes sense to a systemd expert. | 01:00 |
| fungi | yeah, this is probably an improvement over system v, and some day someone will explain to me why | 01:01 |
| corvus | the graph looks better now, and zm01 seems to be running | 01:01 |
| pabelanger | speaking of mergers, I was thinking we might want to rotate each one out and remove old repos on local file system from opendev migration (or delete all) so reclaim disk space | 01:01 |
| pabelanger | to* | 01:01 |
| fungi | i've re-enqueued the 1.4.2 tag for openstack/release-test | 01:05 |
| *** slaweq has joined #openstack-infra | 01:11 | |
| fungi | success! | 01:13 |
| pabelanger | yay | 01:14 |
| *** slaweq has quit IRC | 01:24 | |
| *** jamesmcarthur has joined #openstack-infra | 01:27 | |
| *** dave-mccowan has quit IRC | 01:28 | |
| *** whoami-rajat has joined #openstack-infra | 01:31 | |
| *** michael-beaver has quit IRC | 01:34 | |
| *** jamesmcarthur has quit IRC | 01:34 | |
| *** jamesmcarthur has joined #openstack-infra | 01:35 | |
| *** ykarel|away has joined #openstack-infra | 01:36 | |
| *** rosmaita has quit IRC | 01:41 | |
| *** gmann_afk is now known as gmann | 01:42 | |
| *** hongbin has joined #openstack-infra | 01:44 | |
| *** rosmaita has joined #openstack-infra | 01:46 | |
| *** aaronsheffield has quit IRC | 01:47 | |
| *** ykarel|away is now known as ykarel | 01:49 | |
| *** apetrich has quit IRC | 01:57 | |
| *** hwoarang has quit IRC | 01:59 | |
| *** hwoarang has joined #openstack-infra | 02:00 | |
| openstackgerrit | Merged zuul/zuul master: web: add triggers information to pipeline list https://review.opendev.org/637670 | 02:09 |
| *** jcoufal has joined #openstack-infra | 02:12 | |
| *** slaweq has joined #openstack-infra | 02:15 | |
| *** jamesmcarthur has quit IRC | 02:18 | |
| *** slaweq has quit IRC | 02:24 | |
| *** jamesmcarthur has joined #openstack-infra | 02:26 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:28 | |
| *** mriedem has quit IRC | 02:32 | |
| *** jamesmcarthur has quit IRC | 02:43 | |
| *** jamesmcarthur has joined #openstack-infra | 02:45 | |
| *** bhavikdbavishi has quit IRC | 02:59 | |
| *** samueldmq has quit IRC | 03:04 | |
| *** jamesmcarthur has quit IRC | 03:15 | |
| *** slaweq has joined #openstack-infra | 03:16 | |
| *** nicolasbock has quit IRC | 03:21 | |
| *** slaweq has quit IRC | 03:24 | |
| *** psachin has joined #openstack-infra | 03:32 | |
| *** psachin has quit IRC | 03:33 | |
| *** psachin has joined #openstack-infra | 03:35 | |
| *** hongbin has quit IRC | 03:39 | |
| *** ykarel has quit IRC | 03:43 | |
| *** jamesmcarthur has joined #openstack-infra | 03:45 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:50 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: trigger: add job filter event https://review.opendev.org/639905 | 03:51 |
| *** jamesmcarthur has quit IRC | 03:51 | |
| *** ykarel has joined #openstack-infra | 03:59 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: webtrigger: add initial driver and event https://review.opendev.org/555153 | 03:59 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: webtrigger: add web route and rpclistener https://review.opendev.org/554839 | 03:59 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: add build button to trigger job https://review.opendev.org/635716 | 03:59 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: webtrigger: add support for parameterized trigger https://review.opendev.org/644484 | 03:59 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: add build modal with a parameter form https://review.opendev.org/644485 | 03:59 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: add support for checkbox and list parameters https://review.opendev.org/648661 | 03:59 |
| *** udesale has joined #openstack-infra | 04:00 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 04:01 | |
| *** bhavikdbavishi has quit IRC | 04:03 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 04:03 | |
| *** ykarel_ has joined #openstack-infra | 04:05 | |
| *** ykarel has quit IRC | 04:07 | |
| *** rh-jelabarre has quit IRC | 04:09 | |
| *** slaweq has joined #openstack-infra | 04:15 | |
| *** jamesmcarthur has joined #openstack-infra | 04:20 | |
| *** pcaruana has joined #openstack-infra | 04:23 | |
| *** slaweq has quit IRC | 04:24 | |
| *** jamesmcarthur has quit IRC | 04:27 | |
| *** bhavikdbavishi has quit IRC | 04:52 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:53 | |
| *** jamesmcarthur has joined #openstack-infra | 05:00 | |
| *** jamesmcarthur has quit IRC | 05:05 | |
| *** jcoufal has quit IRC | 05:11 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** slaweq has quit IRC | 05:22 | |
| *** jamesmcarthur has joined #openstack-infra | 05:34 | |
| *** jamesmcarthur has quit IRC | 05:39 | |
| *** igordc has joined #openstack-infra | 05:49 | |
| *** iurygregory has joined #openstack-infra | 05:53 | |
| *** igordc has quit IRC | 05:54 | |
| *** igordc has joined #openstack-infra | 05:54 | |
| *** ccamacho has quit IRC | 05:57 | |
| *** igordc has quit IRC | 06:02 | |
| *** quiquell has joined #openstack-infra | 06:06 | |
| *** rtjure has joined #openstack-infra | 06:08 | |
| *** dpawlik has joined #openstack-infra | 06:10 | |
| *** kopecmartin|off is now known as kopecmartin | 06:14 | |
| *** slaweq has joined #openstack-infra | 06:16 | |
| *** electrofelix has joined #openstack-infra | 06:16 | |
| *** yamamoto has joined #openstack-infra | 06:18 | |
| *** markvoelker has joined #openstack-infra | 06:26 | |
| *** jamesmcarthur has joined #openstack-infra | 06:35 | |
| *** yamamoto has quit IRC | 06:37 | |
| *** jamesmcarthur has quit IRC | 06:40 | |
| *** yamamoto has joined #openstack-infra | 06:43 | |
| *** yamamoto has quit IRC | 06:44 | |
| *** apetrich has joined #openstack-infra | 06:47 | |
| *** pcaruana has quit IRC | 06:48 | |
| *** e0ne has joined #openstack-infra | 06:49 | |
| *** kjackal has joined #openstack-infra | 06:50 | |
| *** yamamoto has joined #openstack-infra | 06:55 | |
| *** pcaruana has joined #openstack-infra | 06:55 | |
| openstackgerrit | Jason Lee proposed opendev/storyboard master: RC2: Blueprint Import works, formatting fixes, pep8 cleanup https://review.opendev.org/654812 | 07:01 |
| *** rpittau|afk is now known as rpittau | 07:02 | |
| *** ramishra has joined #openstack-infra | 07:02 | |
| *** rcernin has quit IRC | 07:05 | |
| *** iurygregory is now known as iurygregory_wfh | 07:10 | |
| *** kjackal has quit IRC | 07:12 | |
| *** jamesmcarthur has joined #openstack-infra | 07:13 | |
| *** happyhemant has joined #openstack-infra | 07:14 | |
| *** slaweq has quit IRC | 07:15 | |
| *** slaweq has joined #openstack-infra | 07:17 | |
| *** ykarel__ has joined #openstack-infra | 07:18 | |
| *** ykarel__ is now known as ykarel | 07:18 | |
| *** lpetrut has joined #openstack-infra | 07:19 | |
| *** rtjure has quit IRC | 07:19 | |
| *** ykarel_ has quit IRC | 07:20 | |
| *** udesale has quit IRC | 07:20 | |
| *** tosky has joined #openstack-infra | 07:20 | |
| *** flaper87 has quit IRC | 07:21 | |
| *** udesale has joined #openstack-infra | 07:23 | |
| *** jamesmcarthur has quit IRC | 07:25 | |
| *** flaper87 has joined #openstack-infra | 07:27 | |
| *** rtjure has joined #openstack-infra | 07:28 | |
| slaweq | frickler: hello | 07:37 |
| slaweq | frickler: may ask You about favor (again)? | 07:37 |
| slaweq | frickler: can You maybe set for me job neutron-tempest-plugin-dvr-multinode-scenario to "on-hold" for one run? I will do recheck on some DNM patch for that | 07:38 |
| slaweq | frickler: because I would like to try to debug why some random tests are failing mostly on this job with SSH problems | 07:38 |
| *** ccamacho has joined #openstack-infra | 07:42 | |
| *** jpena|off is now known as jpena | 07:43 | |
| *** ykarel is now known as ykarel|lunch | 07:45 | |
| *** yamamoto has quit IRC | 07:49 | |
| *** yamamoto has joined #openstack-infra | 07:51 | |
| *** lpetrut has quit IRC | 08:02 | |
| *** jpich has joined #openstack-infra | 08:07 | |
| *** lucasagomes has joined #openstack-infra | 08:21 | |
| *** tesseract has joined #openstack-infra | 08:22 | |
| *** tkajinam has quit IRC | 08:25 | |
| AJaeger | infra-root, looking at zuul.openstack.org/status I see some changes waiting for a long time for nodes to start testing, this looks strange. Digging into http://grafana.openstack.org/d/rZtIH5Imz/nodepool?orgId=1&fullscreen&panelId=17&from=now-30d&to=now I see a higher "Error Node launch Attempts" rate than normal. | 08:25 |
| *** dtantsur|afk is now known as dtantsur | 08:26 | |
| *** gfidente has joined #openstack-infra | 08:26 | |
| *** rcernin has joined #openstack-infra | 08:32 | |
| *** ykarel|lunch is now known as ykarel | 08:32 | |
| *** derekh has joined #openstack-infra | 08:32 | |
| *** Lucas_Gray has joined #openstack-infra | 08:34 | |
| *** ralonsoh has joined #openstack-infra | 08:35 | |
| *** tesseract has quit IRC | 08:37 | |
| *** tesseract has joined #openstack-infra | 08:37 | |
| *** iurygregory_wfh has quit IRC | 08:50 | |
| *** yamamoto has quit IRC | 08:54 | |
| *** yamamoto has joined #openstack-infra | 08:55 | |
| *** yamamoto has quit IRC | 08:55 | |
| *** yamamoto has joined #openstack-infra | 08:56 | |
| *** yamamoto has quit IRC | 09:00 | |
| openstackgerrit | Merged zuul/zuul master: Fix race in test_job_pause_pre_skipped_child https://review.opendev.org/655808 | 09:03 |
| *** shardy has joined #openstack-infra | 09:07 | |
| *** Lucas_Gray has quit IRC | 09:15 | |
| *** panda|off is now known as panda | 09:34 | |
| *** yamamoto has joined #openstack-infra | 09:52 | |
| openstackgerrit | caoyuan proposed openstack/os-testr master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/655062 | 10:11 |
| *** yamamoto has quit IRC | 10:19 | |
| *** nicolasbock has joined #openstack-infra | 10:24 | |
| *** bhavikdbavishi has quit IRC | 10:36 | |
| *** yamamoto has joined #openstack-infra | 10:57 | |
| *** yamamoto has quit IRC | 11:05 | |
| *** panda is now known as panda|lunch | 11:10 | |
| *** ykarel is now known as ykarel|afk | 11:12 | |
| *** yamamoto has joined #openstack-infra | 11:14 | |
| *** derekh has quit IRC | 11:15 | |
| *** derekh has joined #openstack-infra | 11:15 | |
| *** rcernin has quit IRC | 11:16 | |
| *** yamamoto has quit IRC | 11:19 | |
| *** yamamoto has joined #openstack-infra | 11:20 | |
| *** tesseract has quit IRC | 11:21 | |
| *** tesseract has joined #openstack-infra | 11:21 | |
| *** yamamoto has quit IRC | 11:22 | |
| *** udesale has quit IRC | 11:22 | |
| *** yamamoto has joined #openstack-infra | 11:23 | |
| *** jpena is now known as jpena|lunch | 11:33 | |
| *** yamamoto has quit IRC | 11:40 | |
| *** jamesmcarthur has joined #openstack-infra | 11:45 | |
| *** yamamoto has joined #openstack-infra | 11:48 | |
| *** rh-jelabarre has joined #openstack-infra | 11:49 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:49 | |
| *** jamesmcarthur has quit IRC | 11:50 | |
| *** quiquell is now known as quiquell|lunch | 11:51 | |
| *** panda|lunch is now known as panda | 11:57 | |
| *** kjackal has joined #openstack-infra | 11:58 | |
| *** tesseract has quit IRC | 11:59 | |
| *** tesseract has joined #openstack-infra | 12:00 | |
| *** jamespage has quit IRC | 12:01 | |
| *** MarkMaglana has quit IRC | 12:01 | |
| *** sparkycollier has quit IRC | 12:01 | |
| *** evgenyl has quit IRC | 12:01 | |
| *** yamamoto has quit IRC | 12:02 | |
| *** JpMaxMan has quit IRC | 12:02 | |
| *** rajinir has quit IRC | 12:02 | |
| *** sparkycollier has joined #openstack-infra | 12:02 | |
| *** evgenyl has joined #openstack-infra | 12:03 | |
| *** jamespage has joined #openstack-infra | 12:03 | |
| *** rajinir has joined #openstack-infra | 12:03 | |
| *** MarkMaglana has joined #openstack-infra | 12:03 | |
| *** JpMaxMan has joined #openstack-infra | 12:03 | |
| *** yamamoto has joined #openstack-infra | 12:06 | |
| *** pkopec has joined #openstack-infra | 12:08 | |
| *** tesseract has quit IRC | 12:09 | |
| *** tesseract has joined #openstack-infra | 12:10 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Recover cached repos from corrupt object files https://review.opendev.org/655890 | 12:11 |
| *** tesseract has quit IRC | 12:12 | |
| *** tesseract has joined #openstack-infra | 12:13 | |
| *** ykarel|afk is now known as ykarel | 12:13 | |
| *** jamesmcarthur has joined #openstack-infra | 12:17 | |
| *** kgiusti has joined #openstack-infra | 12:18 | |
| *** kjackal has quit IRC | 12:24 | |
| *** jamesmcarthur has quit IRC | 12:30 | |
| *** EmilienM is now known as EvilienM | 12:31 | |
| *** jpena|lunch is now known as jpena | 12:32 | |
| *** mriedem has joined #openstack-infra | 12:33 | |
| *** markvoelker has quit IRC | 12:40 | |
| *** quiquell|lunch is now known as quiquell | 12:40 | |
| openstackgerrit | Dirk Mueller proposed opendev/system-config master: Properly mirror the lxc container images as well https://review.opendev.org/655895 | 12:43 |
| openstackgerrit | Merged zuul/zuul-jobs master: Don't repeat the etc/alias setup for buildset registry pushes https://review.opendev.org/655802 | 12:47 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Finish removing git.openstack.org references https://review.opendev.org/654040 | 12:47 |
| *** yamamoto has quit IRC | 12:47 | |
| *** yamamoto has joined #openstack-infra | 12:49 | |
| *** jamesmcarthur has joined #openstack-infra | 12:50 | |
| AJaeger | config-core, the usual small housekeeping for review, please: https://review.opendev.org/#/c/655234/ https://review.opendev.org/655809 - plus a review to reduce some IRC announcements on our channels: https://review.opendev.org/655761 | 12:50 |
| *** kjackal has joined #openstack-infra | 12:50 | |
| *** aaronsheffield has joined #openstack-infra | 13:00 | |
| *** eharney has quit IRC | 13:00 | |
| *** dpawlik has quit IRC | 13:07 | |
| *** yamamoto has quit IRC | 13:09 | |
| *** jcoufal has joined #openstack-infra | 13:11 | |
| *** MarkMaglana has quit IRC | 13:14 | |
| *** jamespage has quit IRC | 13:14 | |
| *** jamespage has joined #openstack-infra | 13:14 | |
| *** jamesmcarthur has quit IRC | 13:16 | |
| *** lseki has joined #openstack-infra | 13:17 | |
| openstackgerrit | Merged openstack/project-config master: Don't announce osf/ projects in openstack-infra/opendev https://review.opendev.org/655761 | 13:17 |
| openstackgerrit | Merged openstack/project-config master: Add glance, tempest, and nova stable* to ceph grafana https://review.opendev.org/655809 | 13:17 |
| openstackgerrit | Merged openstack/project-config master: Track neutron uwsgi jobs move to check queue https://review.opendev.org/655234 | 13:17 |
| *** kranthikirang has joined #openstack-infra | 13:20 | |
| *** kranthikirang1 has joined #openstack-infra | 13:23 | |
| *** kranthikirang has quit IRC | 13:26 | |
| *** openstackgerrit has quit IRC | 13:27 | |
| *** kjackal has quit IRC | 13:28 | |
| *** MarkMaglana has joined #openstack-infra | 13:29 | |
| pabelanger | AJaeger: vexxhost doesn't look happy: http://grafana.openstack.org/dashboard/db/nodepool-vexxhost wonder if we leaked a volume again | 13:39 |
| pabelanger | AJaeger: and looks like an outage in inap? http://grafana.openstack.org/dashboard/db/nodepool-inap | 13:39 |
| mnaser | pabelanger: I think I have seen some over quota stuff | 13:40 |
| mnaser | so I think it is leaked volumes or quota going out of sync again :\ | 13:41 |
| mnaser | hopefully not the latter | 13:41 |
| *** sthussey has joined #openstack-infra | 13:41 | |
| pabelanger | mnaser: thanks, I'll login shortly and look | 13:42 |
| pabelanger | mnaser: also, is other region down for a reason? Or just no capacity? | 13:42 |
| pabelanger | canont remember why disabled | 13:42 |
| *** yamamoto has joined #openstack-infra | 13:42 | |
| *** raissa has joined #openstack-infra | 13:43 | |
| mnaser | pabelanger: just disabled for capacity reasons at the time | 13:44 |
| mnaser | we can look at enabling it (but not now, because ptg) | 13:44 |
| *** yamamoto has quit IRC | 13:44 | |
| pabelanger | kk | 13:44 |
| clarkb | if our images have rebuilt with 1.0.0.1 dns and base job updated for that we can try turning ovh bhs1 back on | 13:48 |
| pabelanger | | 0003587456 | vexxhost-sjc1 | fedora-29-vexxhost | 62e01c16-019d-46b8-b92b-b3e409161e97 | 38.108.68.35 | | ready | 49:00:39:29 | unlocked | | 13:48 |
| pabelanger | maybe we should set fedora-29-vexxhost nodes to min-ready 0 | 13:49 |
| pabelanger | doesn't look like they are used very often | 13:49 |
| *** yamamoto has joined #openstack-infra | 13:49 | |
| *** openstackgerrit has joined #openstack-infra | 13:52 | |
| openstackgerrit | Paul Belanger proposed openstack/project-config master: Set min-ready 0 on vexxhost specific labels https://review.opendev.org/655907 | 13:52 |
| pabelanger | clarkb: mnaser: ^should help with some capacity | 13:52 |
| pabelanger | given have nodepool works now, I'd also be okay setting all other labels to min-ready 0 too | 13:53 |
| *** eharney has joined #openstack-infra | 13:54 | |
| pabelanger | since node-requests are randomized across clouds | 13:54 |
| pabelanger | mnaser: clarkb: mordred: okay, 3 leaked volumes in vexxhost, I've deleted them. I think our next steps here, is to maybe add quota support into nodepool for volumes, and try to delete leaked ones like we do with FIPs. IIRC, we first need to add support into openstacksdk for that? | 13:55 |
| pabelanger | fungi: logan-: there is also a 12 day old auto-hold, is it safe to delete that? | 13:56 |
| Shrews | pabelanger: the logic for deciding if a resource is leaked is within nodepool itself. openstacksdk only needs to support volume delete (which it should already) | 13:59 |
| fungi | pabelanger: i'm guessing we can delete it but should confirm with logan- | 13:59 |
| *** kjackal has joined #openstack-infra | 14:00 | |
| mgagne | pabelanger: we see exceeded quota errors on our side for inap-mtl01. Most usages are in "reversed" state (we do not run placement) so I'm not sure how nodepool deals with it. | 14:00 |
| pabelanger | fungi: thanks | 14:00 |
| pabelanger | mnaser: is it possible to see why 8619a73b-f08f-4c3e-9f20-ec62b69eb2a0 failed to launch, we only get http://paste.openstack.org/show/749814/ from nodepool currently | 14:02 |
| openstackgerrit | Sean McGinnis proposed openstack/reviewstats master: Switch to stestr https://review.opendev.org/655506 | 14:02 |
| openstackgerrit | Sean McGinnis proposed openstack/reviewstats master: Raise hacking version and fix pep8 errors https://review.opendev.org/655911 | 14:02 |
| openstackgerrit | Sean McGinnis proposed openstack/reviewstats master: Drop pypy default tox env https://review.opendev.org/655912 | 14:02 |
| mnaser | pabelanger: quota exceeded errors that im seeing | 14:03 |
| pabelanger | mnaser: that would be on volumes? | 14:04 |
| mnaser | yes, volume quota exceeded | 14:04 |
| pabelanger | hmm | 14:04 |
| mgagne | pabelanger mnaser : has anything changed in quota handling in zuul/nodepool ? | 14:04 |
| mordred | pabelanger, Shrews: I don't think we should have nodepool delete volumes it didn't make - that's really dangerous | 14:04 |
| pabelanger | mgagne: I don't think so, but maybe mordred / Shrews know more | 14:04 |
| *** bnemec is now known as beekneemech | 14:04 | |
| Shrews | nodepool hasn't changed anything wrt volumes | 14:05 |
| mgagne | what about servers? | 14:05 |
| mordred | (volumes can actually contain precious data, and if someone has an unattached volume in their project, I don't think we can know for sure "this is a volume that was created via nova for boot-from-volume that nova didn't delete properly" vs "this is a volume that's not attached to anything" | 14:06 |
| Shrews | mgagne: not that i recall. the openstack driver has been pretty stable | 14:06 |
| mnaser | mordred: I guess the only way to go about this is to stop letting nova create volumes for bfv and instead handle creating volumes by node pool directly | 14:06 |
| mordred | maybe I'm wrong about that - and maybe there is a way to detect that - or maybe we could have nodepool set a piece of metadata on volumes associated with servers it booted using boot-from-volume or something | 14:06 |
| mordred | but I'd want to be VERY careful | 14:07 |
| mordred | mnaser: god I would find that so annoying ... but you might be right | 14:07 |
| pabelanger | mnaser: I see 42 volumes online, can you remind we what the quota is? | 14:07 |
| mordred | mnaser: we can set arbitrary metadata on volumes, right? | 14:07 |
| pabelanger | mordred: I think we do already today | 14:07 |
| mnaser | so instead of: nova boot-me-a-server-and-create-a-volume, you would: cinder create-me-a-volume-with-some-nodepooly-metadata; nova boot-me-a-server-with-that-volume; | 14:07 |
| mnaser | pabelanger: I can double heck | 14:08 |
| mnaser | check | 14:08 |
| mordred | yeah. then "nova delete the server and then cinder delete that volume" | 14:08 |
| mordred | yeah. if we do that, then I think it can be safe | 14:08 |
| mnaser | and you can filter metadata | 14:08 |
| mnaser | if node pool forgot to delete something | 14:08 |
| mordred | I just **REALLY** don't want to be responsible for deleting someone's actual data :) | 14:08 |
| mnaser | setup some really cryptic metadata | 14:08 |
| mordred | mnaser: we already set metadata on servers to indicate they belong to a given nodepool | 14:09 |
| mnaser | OR something like __nodepool_instance_id metadata and then check if that instance still exists | 14:09 |
| fungi | that's basically how we handle servers today | 14:09 |
| mordred | IIRC | 14:09 |
| fungi | yeah | 14:09 |
| Shrews | mordred: yeah, me too. /me recalls deleting all of tobiash's ports | 14:09 |
| mordred | Shrews: ++ | 14:09 |
| mordred | because we've had people running two nodepools in the same openstack project before | 14:09 |
| mnaser | ahhh okay the instance id won't be a clean solution then | 14:09 |
| pabelanger | mnaser: so, it looks like in nodepool we say we can boot 60 servers, 50 pool + 10 pool, however I can only see we are making out at about 42 volumes | 14:09 |
| mordred | yeah - but we can re-use what we're doing for servers almost certainly | 14:09 |
| mnaser | mordred: I guess the only difference is servers is a tracked resources in node pool but volumes are not | 14:10 |
| pabelanger | so, is it possible our nodepool config is too high for volume quota? | 14:10 |
| mnaser | pabelanger: I am pretty sure the quota is probably for 50.. but the 10 pool is probably the kata one right? | 14:10 |
| mnaser | we should one day rename the project name for the node pool account | 14:10 |
| mnaser | 'openstackjenkins' cmon | 14:10 |
| pabelanger | mnaser: yah | 14:10 |
| mordred | mnaser: well, we could just use the server tracking info for the volume | 14:11 |
| mordred | mnaser: opendevzuul sounds more like it :) | 14:11 |
| mnaser | mordred: jokes on ya, can't delete a volume if it's still attached, can't detach a root volume, so you have to delete the server first | 14:11 |
| mnaser | and if for some reason it doesn't get deleted, it'll be orphan again | 14:11 |
| pabelanger | mnaser: but because the same provider, they will try to launch 60 servers combined | 14:11 |
| mnaser | is the kata pool even being used sad | 14:11 |
| mnaser | :( | 14:12 |
| pabelanger | mnaser: I'm starting to wonder if maybe quota is out of sync some how in the provider? | 14:12 |
| pabelanger | mnaser: that that I seen for fedora-28 above | 14:12 |
| pabelanger | 47+ days of ready | 14:12 |
| mnaser | dont think thats the most popular flavor | 14:12 |
| mnaser | unless the fedora-28 was in the kata pool! | 14:12 |
| pabelanger | mnaser: https://review.opendev.org/655907/ will help limited that pool | 14:13 |
| pabelanger | and only boot those images when requested | 14:13 |
| *** ccamacho has quit IRC | 14:13 | |
| mnaser | looks like there is no limits on instance # or ram | 14:14 |
| mnaser | the only quota is 6144 gb of volume storage | 14:14 |
| pabelanger | but think we need to drop https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl03.openstack.org.yaml#L233 to 40, to deal with 10 from kata pool. Or bump volume quota on cloud side by 10 | 14:14 |
| mnaser | im not sure how we calculated that 6144... | 14:14 |
| mnaser | because you boot 80gb volumes | 14:14 |
| mnaser | so thats 76.8 instances anyways | 14:14 |
| pabelanger | yah | 14:14 |
| pabelanger | something out of sync? | 14:15 |
| mnaser | this isn't the first time the cinder quota has gone out of sync. | 14:15 |
| mnaser | gr | 14:15 |
| *** kjackal has quit IRC | 14:16 | |
| mnaser | yep | 14:16 |
| mnaser | they went out of sync again | 14:16 |
| mnaser | https://www.irccloud.com/pastebin/T6pMzItA/ | 14:17 |
| pabelanger | Yay, we figured it out... boo sync issue | 14:17 |
| mnaser | pabelanger: resync'd with my 'old school' db update way | 14:18 |
| *** dpawlik has joined #openstack-infra | 14:19 | |
| pabelanger | won't ask that that is :) | 14:19 |
| pabelanger | okay, looking into inap now | 14:20 |
| mgagne | pabelanger: I think there is something going on, I found a lot of zombies instances (again). I'm getting quota errors, I'm getting scheduling errors because there are more cores used on the compute than expected so even if you have quota, it can't find a host. | 14:21 |
| pabelanger | mgagne: yup, I just seen: https://pastebin.com/raw/XvQWXQXk | 14:23 |
| *** dpawlik has quit IRC | 14:23 | |
| *** raissa has quit IRC | 14:24 | |
| pabelanger | I can only see 114 instances in nodepool | 14:24 |
| *** raissa has joined #openstack-infra | 14:25 | |
| *** gfidente has quit IRC | 14:25 | |
| pabelanger | mnaser: thanks, things look to be better now: http://grafana.openstack.org/dashboard/db/nodepool-vexxhost cc AJaeger | 14:25 |
| pabelanger | I still think we should do https://review.opendev.org/655907/ | 14:26 |
| mgagne | pabelanger: I took a look at a zombie instance and tried to trace its lifecycle. I see the delete request coming in nova-api but can't find anything in other nova services. | 14:26 |
| *** iurygregory has joined #openstack-infra | 14:27 | |
| pabelanger | mgagne: yah, I think we might have a few stuck instances in delete state | 14:27 |
| pabelanger | mgagne: IIRC, don't we usually toggle the state on those type of instances, so we can issue delete again, and they get cleaned up? | 14:28 |
| Shrews | pabelanger: nodepool will repeatedly try to delete the instances, but i've seen cases where the cloud just doesn't let it for reasons | 14:29 |
| *** raissa has quit IRC | 14:29 | |
| mgagne | pabelanger: I think it depends on how it got stuck in deleting state. One "bad" reason is that nova-compute has a delete task stuck on compute and you need to restart the service. Otherwise you can't restart a new delete request. | 14:29 |
| pabelanger | Shrews: yah, in the past, sometimes if you toggle state of the VM (needs admin cloud side) and you delete again, openstack deletes it, but I cannot remember under what conditions that works | 14:30 |
| pabelanger | mgagne: ack | 14:30 |
| mgagne | pabelanger: it works if nova-compute isn't stuck which might be caused by (I think) listening/publishing on a queue without producers/consumers. Restarting the service usually fixes it. | 14:31 |
| pabelanger | ok | 14:32 |
| mgagne | I'll see what I can do, I might restart everything and delete zombie instances and see how it goes. | 14:32 |
| *** Lucas_Gray has joined #openstack-infra | 14:33 | |
| pabelanger | mnaser: only think I am seeing is http://paste.openstack.org/show/749819/ now | 14:34 |
| pabelanger | possible there is a bad port some place? | 14:34 |
| pabelanger | let me get UUID | 14:34 |
| pabelanger | 2019-04-26 14:28:16,535 DEBUG nodepool.NodeLauncher-0005800188: Node 0005800188 is running [region: sjc1, az: nova, ip: 2604:e100:3:0:f816:3eff:feae:5d33 ipv4: 38.108.68.198, ipv6: 2604:e100:3:0:f816:3eff:feae:5d33, hostid: 60404687b8e801db0abdd0a229e1c2af5d6574294b1ee44a80f0cc6e] | 14:34 |
| *** gfidente has joined #openstack-infra | 14:36 | |
| pabelanger | 2019-04-26 14:27:52,096 DEBUG nodepool.NodeLauncher-0005800170: Node 0005800170 is running [region: sjc1, az: nova, ip: 2604:e100:3:0:f816:3eff:fe4a:3459 ipv4: 38.108.68.179, ipv6: 2604:e100:3:0:f816:3eff:fe4a:3459, hostid: f95beda436c786f440a7a0140042bec3b4b7b815654fdd766d0d2781] | 14:36 |
| pabelanger | is another | 14:37 |
| pabelanger | mgagne: ack, thanks. Let me know if you need anything on our side | 14:37 |
| *** kranthikirang1 has quit IRC | 14:38 | |
| *** yamamoto has quit IRC | 14:40 | |
| pabelanger | logan-: it seems a9686573-82ef-4d4d-a8b5-550723d3b631 | ubuntu-xenial-limestone-regionone-0004606772 | BUILD might be a zombie too | 14:41 |
| pabelanger | openstack server show cannot find it, but it shows up in openstack server list | 14:41 |
| pabelanger | AJaeger: I have to stop for now, but at least identified some of the issues | 14:42 |
| openstackgerrit | Sean McGinnis proposed zuul/zuul-jobs master: Revert "Add environment debugging to ensure-twine role" https://review.opendev.org/655916 | 14:46 |
| fungi | this morning i'm mostly focused on getting a couple dozen previously-failed release jobs rerun now that we got the executors restarted | 14:46 |
| fungi | though i'll probably take the afternoon off for travel prep | 14:46 |
| clarkb | fungi: any idea if the 1.0.0.1 changes have made it in? | 14:50 |
| clarkb | if so we can likely also revert the revert of the revert to enable that | 14:50 |
| fungi | lookin' | 14:50 |
| *** kjackal has joined #openstack-infra | 14:51 | |
| fungi | 655687 merged 13:47z yesterday, so it's been in for 24 hours. i've un-wipped 655689 | 14:52 |
| *** smarcet has joined #openstack-infra | 14:52 | |
| *** Lucas_Gray has quit IRC | 14:53 | |
| clarkb | fwiw I think those didn't need to be a strict dependency | 14:53 |
| *** kopecmartin is now known as kopecmartin|off | 14:53 | |
| clarkb | one handles the pre job stuff workign and theo ther makes the jobs even more reliable | 14:53 |
| fungi | ahh, in that case i'll approve it now | 14:53 |
| clarkb | ya the base image change is so that things mostly work on boot. Then on job start we refine it further to avoid going through NAT if on ipv6 cloud | 14:54 |
| openstackgerrit | Merged opendev/base-jobs master: Switch from 1.1.1.1 to 1.0.0.1 https://review.opendev.org/655689 | 14:59 |
| *** ianychoi_ has joined #openstack-infra | 15:00 | |
| *** gfidente has quit IRC | 15:01 | |
| *** gfidente has joined #openstack-infra | 15:01 | |
| mgagne | pabelanger: restarted everything and deleted zombies. | 15:02 |
| *** ianychoi has quit IRC | 15:03 | |
| *** Emine has joined #openstack-infra | 15:05 | |
| *** tesseract has quit IRC | 15:06 | |
| *** iurygregory has quit IRC | 15:08 | |
| mgagne | pabelanger: I think I found something more serious. There is a discrepancy between Nova API cell database and Nova Compute cell database. Will have to look into it. | 15:09 |
| *** eernst has joined #openstack-infra | 15:11 | |
| *** Goneri has joined #openstack-infra | 15:12 | |
| corvus | infra-root: which of us used the openstackadmin account to "thumbsup" my pr at https://github.com/go-gitea/gitea/pull/6758 ? | 15:13 |
| mgagne | pabelanger: I suggest we disable inap-mtl01. I have no idea yet how I'm going to fix that one. | 15:13 |
| mordred | corvus: oh crap - I didn't realize I was still logged in as openstackadmin | 15:15 |
| mordred | corvus: I am now nolonger logged in to that account | 15:16 |
| *** zbr|rover is now known as zbr|over | 15:16 | |
| AJaeger | thanks, pabelanger ! | 15:16 |
| *** ykarel is now known as ykarel|afk | 15:21 | |
| openstackgerrit | Mathieu Gagné proposed openstack/project-config master: Disable inap-mtl01 https://review.opendev.org/655924 | 15:21 |
| *** rpittau is now known as rpittau|afk | 15:27 | |
| mordred | smcginnis: why revert the debug message fix? | 15:30 |
| smcginnis | mordred: I wasn't sure it would be necessary anymore. I suppose it could help if there are ever any other issues. What do you think? | 15:30 |
| *** Goneri has quit IRC | 15:30 | |
| mordred | smcginnis: oh - wait - I understand what's going on there ... I have a different feedback | 15:31 |
| smcginnis | If we want to keep it, it could probably be improved a bit. | 15:32 |
| smcginnis | But I think as far as ensure-twine goes, we're good now that we're not picking up ansibles venv python. | 15:32 |
| mordred | smcginnis: nah - the thing that caught my eye turns out that I thnik we're not reverting enough | 15:32 |
| mordred | smcginnis: left an inline | 15:32 |
| smcginnis | Ummm.... rebase error? Yeah, that's definitely not right and not what I had intended to do. | 15:33 |
| smcginnis | I'll fix that up. | 15:33 |
| *** smarcet has quit IRC | 15:35 | |
| openstackgerrit | Sean McGinnis proposed zuul/zuul-jobs master: Revert "Add environment debugging to ensure-twine role" https://review.opendev.org/655916 | 15:35 |
| *** gyee has joined #openstack-infra | 15:36 | |
| mordred | smcginnis: looks great, thanks! | 15:36 |
| smcginnis | mordred: Sorry about that, I usually pay more attention when needing to do things like squashing commits. | 15:37 |
| smcginnis | I blame waking up too early and not enough coffee. | 15:37 |
| mordred | smcginnis: those are both the worst things | 15:37 |
| pabelanger | corvus: not I, I've not logged in using that account is a while | 15:39 |
| corvus | pabelanger: we got to the bottom of it, thx | 15:40 |
| pabelanger | ack | 15:40 |
| corvus | infra-root: ^ question re openstackadmin account has been answered | 15:40 |
| pabelanger | mgagne: okay, if you want to propose a change, I'll +2 | 15:40 |
| mgagne | pabelanger: https://review.opendev.org/655924 | 15:40 |
| pabelanger | +2 | 15:41 |
| openstackgerrit | Merged zuul/zuul master: Recover cached repos from corrupt object files https://review.opendev.org/655890 | 15:43 |
| corvus | that's a good sign ^ means the registry jobs still work after the recent change | 15:44 |
| *** Goneri has joined #openstack-infra | 15:44 | |
| corvus | should we restart the scheduler with only the cancel-job patch reverted? | 15:44 |
| pabelanger | corvus: is that related to the memory leak for zuul? | 15:47 |
| corvus | ya | 15:47 |
| fungi | i can take a break from reenqueuing tags for a moment if we want a scheduler restart | 15:47 |
| pabelanger | wfm | 15:47 |
| corvus | we narrowed it down to 2 patches, some of us think that one is the most likely of the two | 15:48 |
| corvus | clarkb, mordred, tobiash: ^ | 15:48 |
| fungi | but will want the ones already enqueued to finish their jobs firs | 15:48 |
| fungi | t | 15:48 |
| corvus | that seems entirely reasonable :) | 15:48 |
| corvus | fungi: so yeah, let's go ahead and start quiescing | 15:48 |
| *** ykarel|afk has quit IRC | 15:49 | |
| fungi | cool. i have ~9 more i'll enqueue once the scheduler is back up | 15:49 |
| corvus | smcginnis: ^ fyi | 15:49 |
| fungi | been taking them in small batches since we had a couple dozen | 15:49 |
| smcginnis | Got it, thanks! | 15:49 |
| smcginnis | Shouldn't be much longer on the current release jobs completing. | 15:50 |
| corvus | after those, we wait for the follow-on tag jobs, right? | 15:51 |
| corvus | oh those are the tag jobs | 15:52 |
| corvus | nm | 15:52 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: WIP: Run dstat and generate graphs in unit tests https://review.opendev.org/610100 | 15:52 |
| smcginnis | corvus: No, these are the final ones that normally are triggered after the tagging. | 15:52 |
| smcginnis | Two left. | 15:52 |
| smcginnis | And then there were none. | 15:52 |
| corvus | i'll maybe wait another 5m for the current post pipeline to clear out? | 15:53 |
| *** psachin has quit IRC | 15:54 | |
| *** smarcet has joined #openstack-infra | 15:54 | |
| corvus | meanwhile, i've staged in ~corvus/zuul a tree with the opendev migration patch followed by a revert of 9f7c642ae1dc5ac7de1cb0ff5c7e32d6426bd1b3 -- "Tolerate missing project" | 15:54 |
| corvus | er | 15:55 |
| corvus | that's the opposite of what i said i'd do | 15:55 |
| corvus | glad i checked | 15:55 |
| corvus | meanwhile, i've staged in ~corvus/zuul a tree with the opendev migration patch followed by a revert of 3704095c7927568a1f32317337c3646a9d15769e -- "Centralize job canceling" | 15:55 |
| corvus | that should be the same as clarkb's tree from 2 days ago but with a single revert instead of 2 | 15:56 |
| corvus | so we won't have any of the other changes recently merged (including the looping warnings) | 15:56 |
| corvus | i'll restart now | 15:57 |
| *** e0ne has quit IRC | 15:58 | |
| *** smarcet has quit IRC | 16:00 | |
| *** psachin has joined #openstack-infra | 16:00 | |
| *** electrofelix has quit IRC | 16:00 | |
| *** ekultails has joined #openstack-infra | 16:04 | |
| corvus | #status log restarted zuul scheduler at 6afa22c9949bbe769de8e54fd27bc0aad14298bc with a local revert of 3704095c7927568a1f32317337c3646a9d15769e to confirm it is cause of memory leak | 16:04 |
| openstackstatus | corvus: finished logging | 16:04 |
| corvus | fungi: you should be gtg | 16:04 |
| fungi | thanks! | 16:06 |
| *** shardy has quit IRC | 16:07 | |
| *** shardy has joined #openstack-infra | 16:09 | |
| *** lucasagomes has quit IRC | 16:09 | |
| corvus | re-enqueue is finished | 16:10 |
| *** smarcet has joined #openstack-infra | 16:12 | |
| *** chandankumar is now known as raukadah | 16:12 | |
| *** ykarel has joined #openstack-infra | 16:15 | |
| *** ramishra has quit IRC | 16:16 | |
| *** igordc has joined #openstack-infra | 16:18 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Revert "Add environment debugging to ensure-twine role" https://review.opendev.org/655916 | 16:20 |
| *** mattw4 has joined #openstack-infra | 16:20 | |
| *** jpich has quit IRC | 16:21 | |
| *** smarcet has quit IRC | 16:23 | |
| *** smarcet has joined #openstack-infra | 16:25 | |
| AJaeger | infra-root, do we want to try enabling ovh again? Or is it bad timing? | 16:26 |
| *** xarses_ has joined #openstack-infra | 16:27 | |
| clarkb | I think we can I justwont be able to watch it much | 16:27 |
| clarkb | so defer to those more here | 16:27 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Revert "Revert "Revert "nodepool: pause ovh""" https://review.opendev.org/655945 | 16:28 |
| AJaeger | here's the proposal for the revert ^ | 16:28 |
| AJaeger | but let's those infra-root that are here and have time vote and decide ^ | 16:29 |
| *** andreww has quit IRC | 16:30 | |
| *** smarcet has quit IRC | 16:32 | |
| *** ekultails has quit IRC | 16:33 | |
| *** roman_g has quit IRC | 16:34 | |
| openstackgerrit | Merged openstack/project-config master: Disable inap-mtl01 https://review.opendev.org/655924 | 16:36 |
| *** jpena is now known as jpena|off | 16:42 | |
| openstackgerrit | Jason Lee proposed opendev/storyboard master: RC3: Add feature to migrate Launchpad blueprints to Storyboard https://review.opendev.org/654812 | 16:45 |
| *** ccamacho has joined #openstack-infra | 16:45 | |
| *** ccamacho has quit IRC | 16:46 | |
| *** ekultails has joined #openstack-infra | 16:51 | |
| corvus | i'm +2 on that if someone else is :) | 16:55 |
| *** panda is now known as panda|off | 16:56 | |
| *** zxiiro has quit IRC | 17:03 | |
| *** zxiiro has joined #openstack-infra | 17:04 | |
| *** derekh has quit IRC | 17:05 | |
| *** smarcet has joined #openstack-infra | 17:08 | |
| *** eharney has quit IRC | 17:09 | |
| *** Emine has quit IRC | 17:09 | |
| openstackgerrit | Elod Illes proposed openstack/openstack-zuul-jobs master: Use tox-docs job in periodic for Stein https://review.opendev.org/655972 | 17:10 |
| openstackgerrit | Merged openstack/project-config master: Revert "Revert "Revert "nodepool: pause ovh""" https://review.opendev.org/655945 | 17:12 |
| *** tjgresha has left #openstack-infra | 17:14 | |
| *** tjgresha has joined #openstack-infra | 17:15 | |
| *** Weifan has joined #openstack-infra | 17:16 | |
| *** tjgresha has quit IRC | 17:19 | |
| *** tjgresha has joined #openstack-infra | 17:19 | |
| *** tjgresha has quit IRC | 17:19 | |
| *** tjgresha has joined #openstack-infra | 17:20 | |
| *** guimaluf has joined #openstack-infra | 17:22 | |
| *** ralonsoh has quit IRC | 17:22 | |
| openstackgerrit | Adam Spiers proposed openstack/ptgbot master: Switch example IRC port to 6697 https://review.opendev.org/655976 | 17:27 |
| openstackgerrit | Adam Spiers proposed openstack/ptgbot master: Change OpenStack to Open Infrastructure in README.rst title https://review.opendev.org/655977 | 17:27 |
| *** jamesmcarthur has joined #openstack-infra | 17:30 | |
| *** tjgresha has quit IRC | 17:31 | |
| *** tjgresha has joined #openstack-infra | 17:31 | |
| *** stewie925 has joined #openstack-infra | 17:31 | |
| stewie925 | hi good morning OPENDEV team | 17:31 |
| stewie925 | our team were trying to do a git merge but getting "you are not allowed to upload merges: | 17:33 |
| stewie925 | https://opendev.org/x/ranger/ | 17:33 |
| *** Weifan has quit IRC | 17:33 | |
| stewie925 | how do we get the upload merges permissions? | 17:34 |
| clarkb | stewie925: you'll need to update the project acls. Let me find an examae | 17:36 |
| *** tjgresha has quit IRC | 17:36 | |
| *** ccamacho has joined #openstack-infra | 17:38 | |
| clarkb | stewie925: https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls/openstack/swift.config#L1-2 is how swiftdoes it | 17:38 |
| clarkb | you'll want asimilar block in ranger's acl file in that repo I linked above (you can propose that change and we have to review and merge it) | 17:38 |
| stewie925 | hi thank you let me take a look | 17:40 |
| *** kjackal has quit IRC | 17:46 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix memory leak in job cancelation https://review.opendev.org/655982 | 17:49 |
| *** jamespage has quit IRC | 17:50 | |
| *** jamespage has joined #openstack-infra | 17:50 | |
| stewie925 | clarkb: thanks again, we'll send a commit to request that (patterned after swift.config) | 17:51 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix memory leak in job cancelation https://review.opendev.org/655982 | 17:52 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix memory leak in job cancellation https://review.opendev.org/655982 | 17:53 |
| *** Goneri has quit IRC | 17:54 | |
| *** michael-beaver has joined #openstack-infra | 17:54 | |
| *** MarkMaglana has quit IRC | 17:56 | |
| *** MarkMaglana has joined #openstack-infra | 17:56 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Update test coverage for openSUSE-minial to 15.0 https://review.opendev.org/610436 | 17:57 |
| *** mattw4 has quit IRC | 17:58 | |
| *** ccamacho has quit IRC | 17:58 | |
| *** shardy has quit IRC | 17:59 | |
| *** shardy has joined #openstack-infra | 18:01 | |
| *** mattw4 has joined #openstack-infra | 18:02 | |
| *** Goneri has joined #openstack-infra | 18:05 | |
| openstackgerrit | Adam Spiers proposed openstack/ptgbot master: Add presence tracking commands https://review.opendev.org/655986 | 18:08 |
| *** jbadiapa has quit IRC | 18:10 | |
| *** jaosorior has quit IRC | 18:11 | |
| *** cmurphy is now known as cmurpheus | 18:17 | |
| *** happyhemant has quit IRC | 18:21 | |
| openstackgerrit | Ian Pittwood proposed openstack/project-config master: Add spyglass-plugin-xls project https://review.opendev.org/655991 | 18:21 |
| *** eharney has joined #openstack-infra | 18:29 | |
| *** jamesmcarthur has quit IRC | 18:34 | |
| *** jamesmcarthur has joined #openstack-infra | 18:34 | |
| *** jamesmcarthur has quit IRC | 18:36 | |
| *** mattw4 has quit IRC | 18:45 | |
| *** mattw4 has joined #openstack-infra | 18:48 | |
| *** Weifan has joined #openstack-infra | 18:49 | |
| *** jamesmcarthur has joined #openstack-infra | 18:49 | |
| *** gfidente is now known as gfidente|afk | 18:54 | |
| *** Weifan has quit IRC | 18:55 | |
| *** jamesmcarthur has quit IRC | 18:56 | |
| *** Weifan has joined #openstack-infra | 18:57 | |
| *** dtantsur is now known as dtantsur|afk | 18:57 | |
| *** jamesmcarthur has joined #openstack-infra | 18:59 | |
| *** pkopec has quit IRC | 19:02 | |
| *** mattw4 has quit IRC | 19:03 | |
| openstackgerrit | Adam Spiers proposed openstack/project-config master: Notify #openstack-ptg IRC channel on ptgbot dev activity https://review.opendev.org/655999 | 19:09 |
| *** Weifan has quit IRC | 19:12 | |
| fungi | stewie925: be forewarned that more often than not, that rejection when pushing merge commits acts as a safeguard against someone with a dirty pull into a branch accidentally pushing loads of irrelevant commits, so we don't recommend turning it on except for contributors who are extra careful inspecting their branches prior to pushing | 19:14 |
| openstackgerrit | STEW TY proposed openstack/project-config master: Add Project Configs for ranger and ranger-agent https://review.opendev.org/656000 | 19:15 |
| *** ykarel is now known as ykarel|away | 19:21 | |
| *** stewie925 has quit IRC | 19:24 | |
| *** dims has quit IRC | 19:25 | |
| *** psachin has quit IRC | 19:25 | |
| *** dims has joined #openstack-infra | 19:30 | |
| *** dims has quit IRC | 19:31 | |
| *** dims has joined #openstack-infra | 19:31 | |
| *** stewie925 has joined #openstack-infra | 19:32 | |
| *** zul has quit IRC | 19:34 | |
| openstackgerrit | STEW TY proposed openstack/project-config master: Add Project Configs for ranger and ranger-agent https://review.opendev.org/656000 | 19:37 |
| *** dims has quit IRC | 19:38 | |
| stewie925 | Hello, need help clarifying the comments made in https://review.opendev.org/656000 - appreciate it. | 19:41 |
| stewie925 | I have updated the commit message | 19:41 |
| fungi | stewie925: be forewarned that more often than not, that rejection when pushing merge commits acts as a safeguard against someone with a dirty pull into a branch accidentally pushing loads of irrelevant commits, so we don't recommend turning it on except for contributors who are extra careful inspecting their branches prior to pushing. about the only time you want to push a merge commit is if you're | 19:44 |
| fungi | merging between two branches which are already in gerrit | 19:44 |
| stewie925 | fungi: hi | 19:44 |
| *** jbadiapa has joined #openstack-infra | 19:45 | |
| *** Emine has joined #openstack-infra | 19:46 | |
| AJaeger | stewie925: your repo is called x/ranger, not openstack/ranger, you touch the wrong file. And don't copy the full content, just the two lines tht fungi mentioned earlier | 19:46 |
| fungi | er, well, that was clarkb not me | 19:47 |
| AJaeger | sorry... | 19:47 |
| clarkb | swift limits who can do it which was my example | 19:47 |
| clarkb | did I miss something else? | 19:48 |
| *** dims has joined #openstack-infra | 19:48 | |
| stewie925 | AJaeger: whats the diff between x/ranger instead of openstack/ranger | 19:51 |
| fungi | there is no openstack/ranger, that's just a legacy redirect to x/ranger | 19:51 |
| fungi | and there's probably also a stale defunct openstack/ranger on github | 19:52 |
| *** dims has quit IRC | 19:52 | |
| stewie925 | I will abandon https://review.opendev.org/656000 for now. | 19:53 |
| fungi | did you determine you don't need the ability to push merge commits to x/ranger? | 19:54 |
| stewie925 | ok, let me confer with my team before I do anything with the commit - thank you | 19:54 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Fix stale job dir deletion on startup https://review.opendev.org/656003 | 19:55 |
| *** jcoufal has quit IRC | 19:56 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix race in test_job_pause_retry https://review.opendev.org/656004 | 19:56 |
| *** dims has joined #openstack-infra | 19:59 | |
| *** ykarel|away has quit IRC | 19:59 | |
| *** kgiusti has left #openstack-infra | 20:00 | |
| *** dims has quit IRC | 20:04 | |
| *** dims has joined #openstack-infra | 20:05 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Fix deletion of stale build dirs on startup https://review.opendev.org/620697 | 20:08 |
| stewie925 | AJaeger: I see what you mean about touching the "wrong file" | 20:09 |
| stewie925 | since I put then under openstack/config - and there is no x/config | 20:09 |
| stewie925 | correction - i meant gerri/acls/openstack not openstack/config | 20:11 |
| stewie925 | and there is no gerrit/acls/x | 20:11 |
| *** Weifan has joined #openstack-infra | 20:11 | |
| fungi | there's not? https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls/x | 20:12 |
| AJaeger | stewie925: better update the project-config repo ^ | 20:12 |
| AJaeger | https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls/x/ranger.config is the file you need to update | 20:13 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add more debugging to tests https://review.opendev.org/656006 | 20:13 |
| stewie925 | oh thanks! | 20:14 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Rename request: airship-in-a-bottle https://review.opendev.org/656008 | 20:22 |
| *** stewie925 has quit IRC | 20:37 | |
| openstackgerrit | Merged zuul/zuul master: Fix memory leak in job cancellation https://review.opendev.org/655982 | 20:42 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Rename request: airship-in-a-bottle https://review.opendev.org/656008 | 20:46 |
| *** Emine has quit IRC | 20:48 | |
| *** mriedem has quit IRC | 20:52 | |
| *** tjgresha has joined #openstack-infra | 20:53 | |
| *** smarcet has quit IRC | 20:57 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Fix deletion of stale build dirs on startup https://review.opendev.org/620697 | 20:58 |
| *** smarcet has joined #openstack-infra | 21:00 | |
| *** beekneemech has quit IRC | 21:06 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Move legacy-periodic-package-stackviz-element to stackviz repo https://review.opendev.org/641923 | 21:09 |
| *** rosmaita has quit IRC | 21:10 | |
| *** whoami-rajat has quit IRC | 21:10 | |
| *** pcaruana has quit IRC | 21:16 | |
| openstackgerrit | Ian Pittwood proposed openstack/project-config master: Add spyglass-plugin-xls project https://review.opendev.org/655991 | 21:22 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Fix test race in test_job_pause_post_fail https://review.opendev.org/656019 | 21:25 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Increase zookeeper priority during tests https://review.opendev.org/656021 | 21:32 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Fix test race in test_job_pause_post_fail https://review.opendev.org/656019 | 21:36 |
| *** mattw4 has joined #openstack-infra | 21:39 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Fix test race in test_job_pausee https://review.opendev.org/656024 | 21:40 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Fix test race in test_job_pause https://review.opendev.org/656024 | 21:41 |
| *** eharney has quit IRC | 21:42 | |
| *** igordc has quit IRC | 21:55 | |
| *** imacdonn has quit IRC | 22:02 | |
| *** imacdonn has joined #openstack-infra | 22:03 | |
| *** Lucas_Gray has joined #openstack-infra | 22:13 | |
| clarkb | not sure if everyone has seen but monday night to tuesday it will apparently snow in the denver area. So lowry looking less good monday night and instead the barcade option looking much better | 22:13 |
| clarkb | also pack a jacket and shoes :P | 22:13 |
| *** aaronsheffield has quit IRC | 22:17 | |
| fungi | yeah, unfortunate. my carry-on is going to be pushed to its limits | 22:17 |
| *** ekultails has quit IRC | 22:20 | |
| corvus | inconceivable! | 22:24 |
| *** slaweq has quit IRC | 22:24 | |
| paladox | it snows in may? :P | 22:32 |
| *** Weifan has quit IRC | 22:34 | |
| *** sshnaidm has joined #openstack-infra | 22:37 | |
| *** smarcet has quit IRC | 22:38 | |
| fungi | the elevation there is fairly high | 22:40 |
| *** sshnaidm is now known as sshnaidm|off | 22:43 | |
| *** sshnaidm|off has quit IRC | 22:46 | |
| *** Weifan has joined #openstack-infra | 22:50 | |
| *** EvilienM is now known as EmilienM | 22:51 | |
| *** sshnaidm has joined #openstack-infra | 22:53 | |
| *** sshnaidm is now known as sshnaidm|off | 22:54 | |
| *** eernst has quit IRC | 22:56 | |
| mordred | fungi: I think something something "a mile" something | 23:04 |
| *** jamesmcarthur has quit IRC | 23:05 | |
| *** tosky has quit IRC | 23:08 | |
| *** Weifan has quit IRC | 23:11 | |
| *** sshnaidm|off has quit IRC | 23:24 | |
| *** Weifan has joined #openstack-infra | 23:27 | |
| *** Weifan has quit IRC | 23:32 | |
| *** Weifan has joined #openstack-infra | 23:34 | |
| *** yamamoto has joined #openstack-infra | 23:35 | |
| *** yamamoto has quit IRC | 23:37 | |
| *** Weifan has quit IRC | 23:39 | |
| *** Weifan has joined #openstack-infra | 23:40 | |
| *** nicolasbock has quit IRC | 23:40 | |
| *** slaweq has joined #openstack-infra | 23:42 | |
| *** bdodd has quit IRC | 23:45 | |
| *** slaweq has quit IRC | 23:46 | |
| *** yamamoto has joined #openstack-infra | 23:48 | |
| *** gyee has quit IRC | 23:49 | |
| *** yamamoto has quit IRC | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!