| *** cdearborn has quit IRC | 00:01 | |
| *** tetsuro has joined #openstack-infra | 00:03 | |
| *** seco has quit IRC | 00:16 | |
| *** hongbin has quit IRC | 00:22 | |
| *** jamesmcarthur has joined #openstack-infra | 00:27 | |
| *** hongbin has joined #openstack-infra | 00:29 | |
| *** jamesmcarthur has quit IRC | 00:36 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: nodepool: use job inheritance https://review.opendev.org/713158 | 00:46 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add ubuntu-bionic-plain to more regions https://review.opendev.org/720316 | 00:48 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add ubuntu-bionic-plain to all regions https://review.opendev.org/720316 | 00:52 |
| * jlvillal wonders who is in charge of updating the links for this project https://pypi.org/project/bash8/ | 00:54 | |
| *** jamesmcarthur has joined #openstack-infra | 00:58 | |
| *** prometheanfire has quit IRC | 01:00 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: nodepool: Add more plain images https://review.opendev.org/720318 | 01:07 |
| *** prometheanfire has joined #openstack-infra | 01:07 | |
| fungi | jlvillal: i suspect we could just delete that, as it got renamed to https://pypi.org/project/bashate/ immediately after its initial release | 01:19 |
| ianw | i agree, i can remove that | 01:22 |
| fungi | jlvillal: https://governance.openstack.org/tc/reference/projects/quality-assurance.html#bashate looks like it's an official deliverable of the openstack qa team, so if they give the go ahead i'd be happy to do whatever they want with it | 01:22 |
| fungi | the renaming, if memory serves, occurred after some distros who were interested in packaging it raised the concern that it might look like we were claiming it was some new bash version 8 | 01:23 |
| fungi | (which would also be quite the version skip for bash, but point taken) | 01:24 |
| *** jamesmcarthur has quit IRC | 01:26 | |
| *** jamesmcarthur has joined #openstack-infra | 01:26 | |
| fungi | now it just looks like we hate some dude named bas, but that's okay, he's probably a jerk anyway | 01:26 |
| *** ricolin has joined #openstack-infra | 01:32 | |
| *** jamesmcarthur has quit IRC | 01:32 | |
| ianw | fungi: i don't see that it being there is anything but confusing, i think we should just delete it? | 01:33 |
| *** tetsuro has quit IRC | 01:47 | |
| fungi | that's fine, i just don't want to make that call for some other team without at least giving them a heads up, since it's irreversible | 01:48 |
| fungi | and there technically is a release uploaded there, and we often complain about how people break things by deleting releases from pypi | 01:48 |
| *** tetsuro has joined #openstack-infra | 01:54 | |
| *** jamesmcarthur has joined #openstack-infra | 02:04 | |
| *** jamesmcarthur_ has joined #openstack-infra | 02:07 | |
| *** apetrich has quit IRC | 02:08 | |
| *** __ministry has joined #openstack-infra | 02:09 | |
| *** jamesmcarthur has quit IRC | 02:11 | |
| *** jamesmcarthur_ has quit IRC | 02:13 | |
| ianw | as a core of bashate, i feel confident enough to make the call that we should remove this | 02:18 |
| *** artom has quit IRC | 02:30 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add ubuntu-bionic-plain to all regions https://review.opendev.org/720316 | 02:30 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: nodepool: Add more plain images https://review.opendev.org/720318 | 02:31 |
| fungi | ianw: that works for me then | 02:39 |
| *** lifeless_ is now known as liffeless | 02:40 | |
| *** liffeless is now known as lifeless | 02:40 | |
| *** jamesmcarthur has joined #openstack-infra | 02:41 | |
| *** jamesmcarthur has quit IRC | 02:43 | |
| *** jamesmcarthur has joined #openstack-infra | 02:44 | |
| *** hongbin has quit IRC | 02:50 | |
| *** xinranwang has joined #openstack-infra | 03:01 | |
| *** jamesmcarthur_ has joined #openstack-infra | 03:02 | |
| *** jamesmcarthur has quit IRC | 03:05 | |
| *** ociuhandu has joined #openstack-infra | 03:14 | |
| *** ociuhandu has quit IRC | 03:19 | |
| *** psachin has joined #openstack-infra | 03:26 | |
| *** threestrands has joined #openstack-infra | 03:29 | |
| *** tetsuro has quit IRC | 03:30 | |
| *** jep28 has joined #openstack-infra | 03:49 | |
| *** zxiiro has joined #openstack-infra | 03:56 | |
| *** gyee has quit IRC | 04:02 | |
| *** jamesmcarthur has joined #openstack-infra | 04:06 | |
| *** jamesmcarthur_ has quit IRC | 04:09 | |
| *** osmanlicilegi has joined #openstack-infra | 04:14 | |
| *** lmiccini has joined #openstack-infra | 04:16 | |
| *** jamesmcarthur has quit IRC | 04:22 | |
| *** jamesmcarthur has joined #openstack-infra | 04:23 | |
| *** ykarel|away is now known as ykarel | 04:28 | |
| *** jamesmcarthur has quit IRC | 04:28 | |
| *** jamesmcarthur has joined #openstack-infra | 04:30 | |
| *** jamesmcarthur has quit IRC | 04:30 | |
| *** jamesmcarthur has joined #openstack-infra | 04:36 | |
| *** evrardjp has quit IRC | 04:37 | |
| *** evrardjp has joined #openstack-infra | 04:37 | |
| *** jamesmcarthur has quit IRC | 04:38 | |
| *** jamesmcarthur has joined #openstack-infra | 04:39 | |
| *** udesale has joined #openstack-infra | 04:43 | |
| *** jamesmcarthur has quit IRC | 04:47 | |
| *** tetsuro has joined #openstack-infra | 04:48 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: nb03: use linaro-us mirror https://review.opendev.org/720342 | 04:51 |
| *** jep28 has quit IRC | 05:08 | |
| *** jamesmcarthur has joined #openstack-infra | 05:14 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Use TOX_CONSTRAINTS_FILE in release script https://review.opendev.org/720265 | 05:36 |
| *** ysandeep is now known as ysandeep|brb | 05:38 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Use TOX_CONSTRAINTS_FILE in release script https://review.opendev.org/720265 | 05:41 |
| *** jtomasek has joined #openstack-infra | 05:45 | |
| openstackgerrit | Merged openstack/project-config master: nb03: use linaro-us mirror https://review.opendev.org/720342 | 05:49 |
| *** ralonsoh has joined #openstack-infra | 05:56 | |
| openstackgerrit | Merged openstack/project-config master: Use TOX_CONSTRAINTS_FILE in release script https://review.opendev.org/720265 | 05:58 |
| *** ociuhandu has joined #openstack-infra | 06:11 | |
| *** jamesmcarthur_ has joined #openstack-infra | 06:16 | |
| *** xinranwang has quit IRC | 06:19 | |
| *** jamesmcarthur has quit IRC | 06:20 | |
| *** jamesmcarthur_ has quit IRC | 06:21 | |
| *** ysandeep|brb is now known as ysandeep | 06:34 | |
| *** seco has joined #openstack-infra | 06:34 | |
| *** dklyle has quit IRC | 06:36 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: nodepool: use job inheritance https://review.opendev.org/713158 | 06:37 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add ubuntu-bionic-plain to all regions https://review.opendev.org/720316 | 06:37 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: nodepool: Add more plain images https://review.opendev.org/720318 | 06:37 |
| *** drifterza has joined #openstack-infra | 06:38 | |
| *** ociuhandu has quit IRC | 06:38 | |
| *** ociuhandu has joined #openstack-infra | 06:39 | |
| *** ociuhandu has quit IRC | 06:44 | |
| *** nightmare_unreal has joined #openstack-infra | 06:47 | |
| *** dpawlik has joined #openstack-infra | 06:49 | |
| *** apetrich has joined #openstack-infra | 06:51 | |
| *** jamesmcarthur has joined #openstack-infra | 06:54 | |
| *** ccamacho has joined #openstack-infra | 07:01 | |
| *** yolanda has quit IRC | 07:08 | |
| *** apetrich has quit IRC | 07:08 | |
| *** yolanda has joined #openstack-infra | 07:09 | |
| *** slaweq has joined #openstack-infra | 07:12 | |
| *** jep28 has joined #openstack-infra | 07:13 | |
| jep28 | hi anyone can help me on the glance location-add command, it gives me an HTTP400 invalid location error when adding vsphere template | 07:13 |
| *** jcapitao has joined #openstack-infra | 07:14 | |
| *** amoralej|off is now known as amoralej | 07:24 | |
| *** apetrich has joined #openstack-infra | 07:26 | |
| *** ociuhandu has joined #openstack-infra | 07:30 | |
| *** rpittau|afk is now known as rpittau | 07:34 | |
| *** apetrich has quit IRC | 07:38 | |
| *** tosky has joined #openstack-infra | 07:38 | |
| *** apetrich has joined #openstack-infra | 07:42 | |
| *** ykarel is now known as ykarel|lunch | 07:43 | |
| *** witek has joined #openstack-infra | 07:44 | |
| *** jpena|off is now known as jpena | 07:53 | |
| *** zxiiro has quit IRC | 07:55 | |
| openstackgerrit | Roman Gorshunov proposed openstack/project-config master: Retire airship-in-a-bottle https://review.opendev.org/720160 | 08:03 |
| *** happyhemant has joined #openstack-infra | 08:14 | |
| *** pkopec has joined #openstack-infra | 08:15 | |
| *** ociuhandu has quit IRC | 08:25 | |
| *** dtantsur|afk is now known as dtantsur | 08:28 | |
| *** ociuhandu has joined #openstack-infra | 08:36 | |
| *** threestrands has quit IRC | 08:45 | |
| *** tkajinam has quit IRC | 08:50 | |
| *** ysandeep is now known as ysandeep|lunch | 08:53 | |
| *** jamesmcarthur has quit IRC | 08:55 | |
| *** noama has joined #openstack-infra | 08:56 | |
| *** gfidente has joined #openstack-infra | 08:59 | |
| *** hrw has left #openstack-infra | 09:05 | |
| *** ociuhandu has quit IRC | 09:05 | |
| *** jep28 has quit IRC | 09:08 | |
| *** ociuhandu has joined #openstack-infra | 09:08 | |
| *** ykarel|lunch is now known as ykarel | 09:21 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/project-config master: Add CentOS 8 AArch64 nodes https://review.opendev.org/720167 | 09:22 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/project-config master: Add CentOS 8 AArch64 nodes https://review.opendev.org/720167 | 09:30 |
| *** smcginnis has quit IRC | 09:52 | |
| *** tetsuro has quit IRC | 09:52 | |
| *** tetsuro has joined #openstack-infra | 09:54 | |
| *** lpetrut has joined #openstack-infra | 09:56 | |
| *** ociuhandu has quit IRC | 10:02 | |
| *** ociuhandu has joined #openstack-infra | 10:03 | |
| brtknr | hi there | 10:05 |
| brtknr | I am trying to figure out how to reenable functional tests for our CI since we had to previously disable it due to some weird errors related to kernel incompatibily | 10:06 |
| brtknr | looks like in the mean time, Zuul has moved on and we are seeing errors related to Tempest not being found | 10:06 |
| brtknr | https://zuul.opendev.org/t/openstack/build/5661943cb2d24c11934f6f44e115e764/console | 10:08 |
| brtknr | grep for ModuleNotFoundError: No module named 'tempest' | 10:08 |
| brtknr | 2 | 10:08 |
| *** elod_ has joined #openstack-infra | 10:09 | |
| *** elod_ has quit IRC | 10:09 | |
| AJaeger | brtknr: better ask on #openstack-qa. Looking at your job, I do not see where your require tempest, best to double check that. | 10:10 |
| brtknr | ok thanks | 10:11 |
| *** os-devan777 has joined #openstack-infra | 10:13 | |
| *** rpittau is now known as rpittau|bbl | 10:19 | |
| *** ysandeep|lunch is now known as ysandeep | 10:22 | |
| *** yolanda has quit IRC | 10:26 | |
| *** tetsuro has quit IRC | 10:28 | |
| openstackgerrit | Merged openstack/project-config master: Add CentOS 8 AArch64 nodes https://review.opendev.org/720167 | 10:37 |
| *** yolanda has joined #openstack-infra | 10:38 | |
| *** rfolco has joined #openstack-infra | 10:50 | |
| *** jamesmcarthur has joined #openstack-infra | 10:53 | |
| *** jcapitao is now known as jcapitao_lunch | 10:53 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: nodepool: use job inheritance https://review.opendev.org/713158 | 10:55 |
| *** derekh has joined #openstack-infra | 10:58 | |
| *** jtomasek has quit IRC | 11:13 | |
| *** jtomasek has joined #openstack-infra | 11:13 | |
| *** ysandeep is now known as ysandeep|afk | 11:15 | |
| *** drifterza has quit IRC | 11:22 | |
| *** artom has joined #openstack-infra | 11:29 | |
| *** jpena is now known as jpena|lunch | 11:30 | |
| *** cdearborn has joined #openstack-infra | 11:37 | |
| *** witek has quit IRC | 11:40 | |
| *** ysandeep|afk is now known as ysandeep | 11:47 | |
| *** dpawlik has quit IRC | 11:50 | |
| *** dpawlik has joined #openstack-infra | 11:50 | |
| *** rh-jelabarre has joined #openstack-infra | 11:56 | |
| *** Goneri has joined #openstack-infra | 11:59 | |
| *** smcginnis has joined #openstack-infra | 12:01 | |
| *** ociuhandu has quit IRC | 12:04 | |
| *** ociuhandu has joined #openstack-infra | 12:05 | |
| *** jcapitao_lunch is now known as jcapitao | 12:07 | |
| *** rpittau|bbl is now known as rpittau | 12:07 | |
| nightmare_unreal | has anyone tried to make a second devstack as compute node for first devstack here ? | 12:08 |
| nightmare_unreal | I am having issues in doing so | 12:08 |
| *** ociuhandu has quit IRC | 12:09 | |
| *** hashar has joined #openstack-infra | 12:09 | |
| *** dpawlik has quit IRC | 12:09 | |
| *** dpawlik has joined #openstack-infra | 12:10 | |
| *** amoralej is now known as amoralej|lunch | 12:13 | |
| os-devan777 | nightmare_unreal : I dont think its a good idea to add whole devstack units as compute nodes | 12:15 |
| nightmare_unreal | it's just for test environment | 12:15 |
| nightmare_unreal | if there is much simpler way ? | 12:15 |
| *** psachin has quit IRC | 12:16 | |
| nightmare_unreal | like 1 devstack and just 1 additional server for compute node | 12:16 |
| os-devan777 | I think easier way would be to configure the controller to accommodate another compute node | 12:16 |
| nightmare_unreal | like this ? I was looking at this : https://docs.openstack.org/nova/queens/install/compute-install-ubuntu.html | 12:17 |
| nightmare_unreal | os-devan777: if there is another link/doc that can help me please share :) Thanks | 12:18 |
| os-devan777 | one thing you can do is maybe setup another node for nova and configure it to use the controller endpoint of your existing devstack unit | 12:19 |
| os-devan777 | I havent tested if it works that way, but better idea maybe to have separate compute and controller unit and configure them independently and scale them as per your test requirements as per docs you are reffering? | 12:22 |
| nightmare_unreal | well I need devstack too for dev work. | 12:25 |
| nightmare_unreal | since I want to work on live migration related tasks I need 1 more compute node | 12:25 |
| nightmare_unreal | so that's the idea behind my requirement | 12:25 |
| *** zxiiro has joined #openstack-infra | 12:27 | |
| *** ociuhandu has joined #openstack-infra | 12:28 | |
| *** jpena|lunch is now known as jpena | 12:31 | |
| os-devan777 | nightmare_unreal : I think slight modifications to this setup will suit your test requirements https://docs.openstack.org/devstack/latest/guides/multinode-lab.html | 12:31 |
| *** udesale_ has joined #openstack-infra | 12:31 | |
| nightmare_unreal | I will take a look os-devan777 thanks :) | 12:32 |
| *** udesale has quit IRC | 12:34 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: dhall-diff: add new job https://review.opendev.org/718694 | 12:44 |
| *** ociuhandu has quit IRC | 12:46 | |
| *** ociuhandu has joined #openstack-infra | 12:46 | |
| *** ociuhandu has quit IRC | 12:51 | |
| *** yolanda has quit IRC | 12:55 | |
| *** yolanda has joined #openstack-infra | 12:55 | |
| *** amoralej|lunch is now known as amoralej | 13:10 | |
| *** ysandeep is now known as ysandeep|mtg | 13:14 | |
| *** ociuhandu has joined #openstack-infra | 13:16 | |
| *** eharney has joined #openstack-infra | 13:26 | |
| *** ykarel is now known as ykarel|afk | 13:32 | |
| *** os-devan777 has quit IRC | 13:36 | |
| *** kevinz has quit IRC | 13:51 | |
| *** roman_g has quit IRC | 14:02 | |
| *** ociuhandu has quit IRC | 14:17 | |
| *** dklyle has joined #openstack-infra | 14:24 | |
| *** lpetrut has quit IRC | 14:34 | |
| *** os-devan777 has joined #openstack-infra | 14:42 | |
| *** gyee has joined #openstack-infra | 14:42 | |
| *** ykarel|afk is now known as ykarel | 14:45 | |
| *** __ministry1 has joined #openstack-infra | 14:49 | |
| *** panda is now known as panda|ruck | 14:49 | |
| *** __ministry has quit IRC | 14:50 | |
| *** __ministry1 is now known as __ministry | 14:50 | |
| *** ysandeep|mtg is now known as ysandeep | 14:59 | |
| *** roman_g has joined #openstack-infra | 15:05 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Add openstack-tox-functional-py38 job https://review.opendev.org/720257 | 15:08 |
| *** ysandeep is now known as ysandeep|away | 15:11 | |
| fungi | nightmare_unreal: also, there are probably more devstack maintainers in the #openstack-qa channel since it's their project | 15:12 |
| *** happyhemant has quit IRC | 15:14 | |
| *** os-devan777 has quit IRC | 15:14 | |
| clarkb | nightmare_unreal: you should look at devstack multinode testing in zuul. The way it works is you run devstack on each node separately with configs for the node type that tells it about the cluster | 15:21 |
| clarkb | the controller is built first then compute nodes added to it | 15:21 |
| *** dpawlik has quit IRC | 15:26 | |
| *** dpawlik has joined #openstack-infra | 15:26 | |
| *** tbarron_ has joined #openstack-infra | 15:34 | |
| *** ykarel is now known as ykarel|away | 15:35 | |
| *** ttx has quit IRC | 15:38 | |
| *** ttx has joined #openstack-infra | 15:38 | |
| melwitt | it seems like there might be something wrong with the gate, the change at the front has been "queued" for neutron-tempest-dvr job for > 6 hours | 15:40 |
| clarkb | melwitt: I think that implies it is still waiting for nodepool to return a test node for it. I wonder why that got lost | 15:44 |
| melwitt | yeah, that's also my understanding, waiting for a node | 15:45 |
| clarkb | oh it just siwtched to "waiting" | 15:45 |
| clarkb | which is the state when a node is ready but it needs to find an executor | 15:45 |
| *** ociuhandu has joined #openstack-infra | 15:45 | |
| clarkb | and back to queued now interesting | 15:46 |
| clarkb | corvus: ^ fyi this is an odd zuul behavior. I've started trying to grep logs to see what is going on | 15:46 |
| corvus | clarkb: ack; let me know if you want me to take a look | 15:47 |
| clarkb | 2020-04-16 07:18:49,467 DEBUG zuul.nodepool: [e: 34044aa217a94a1592b19dca9b2237b8] Unable to revise locked node request <NodeRequest 200-0008451921 <NodeSet openstack-single-node-bionic [<Node None ('controller',):ubuntu-bionic>]>> | 15:49 |
| corvus | harmless | 15:49 |
| corvus | (that's the relative priority thing, where it changes the priority of a node request based on current conditions; it's expected to fail like that some times) | 15:49 |
| corvus | http://zuul.openstack.org/stream/2cbdd037fb95486584f05c2aed752c46?logfile=console.log is running | 15:50 |
| *** ociuhandu has quit IRC | 15:50 | |
| clarkb | corvus: ya the logs show ^ starting was just going to start putting a paste together | 15:50 |
| clarkb | I guess now I work backward in the logs and see why it wasn't starting before | 15:51 |
| *** lmiccini has quit IRC | 15:53 | |
| clarkb | if I'm grepping properly that is the first tiem the job was started | 15:54 |
| clarkb | (which I guess is good because that means there were no untracked retries) | 15:54 |
| clarkb | 34044aa217a94a1592b19dca9b2237b8 is the triggering event I think | 15:56 |
| clarkb | 200-0008451921 is the node request I think it may have actually taken nodepool that long to fulfill it for some reason | 15:58 |
| clarkb | so I'm switching to nodepool logs now | 15:58 |
| openstackgerrit | James Page proposed openstack/project-config master: Add TrilioVault charms https://review.opendev.org/720534 | 16:01 |
| melwitt | wow. yeah, I wasn't sure. I've been watching it since last night and when I woke up this morning, it was in the exact same state was before I went to sleep. so I got concerned | 16:01 |
| melwitt | *as | 16:01 |
| *** ociuhandu has joined #openstack-infra | 16:02 | |
| clarkb | I think I'm seeing the inap provider in nodepool having it as an active request from 2020-04-16 05:50:12,714 to 2020-04-16 15:48:16,937 | 16:03 |
| clarkb | it looks like there were moer than one requests in that state but the list has since gone down to 1 | 16:03 |
| clarkb | I see a lot of 2020-04-16 08:30:42,815 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node for inap nodes | 16:05 |
| clarkb | not sure if that is related yet | 16:05 |
| clarkb | that warning comes from quota calculations which will cause us to undercount quota so yes I think it likely was related | 16:06 |
| clarkb | we'd accept more noderequests than we can handle but then due to being over quota it will be a long slog until we can actually fulfill the request (as observed with this job) | 16:07 |
| clarkb | now why did that happen. infra-root were there nodepool provider or clouds.yaml changes in the last ~12 hours? | 16:07 |
| clarkb | https://opendev.org/zuul/nodepool/src/branch/master/nodepool/driver/openstack/provider.py#L159-L166 is the code we are looking at | 16:08 |
| corvus | clarkb: we changed the files on the builders yesterday, but i don't think anything significant should have been seen on the launchers | 16:09 |
| *** jcapitao has quit IRC | 16:10 | |
| clarkb | node.pool is None there | 16:11 |
| clarkb | (this was logged thankfully) | 16:11 |
| clarkb | a bug in writes to zk maybe? | 16:11 |
| mordred | clouds.yaml is still valid on nl01 - as is nodepool.yaml - and neither has changed recently | 16:12 |
| *** __ministry has quit IRC | 16:12 | |
| clarkb | ok I think these unknown servers were maybe half deleted | 16:13 |
| mordred | same on nl03 | 16:14 |
| mordred | (which is where inap is defined) | 16:14 |
| clarkb | http://paste.openstack.org/show/792241/ nodes that have no pool seem to show up that way | 16:14 |
| clarkb | I'm guessing we maybe update pool to None too early? | 16:15 |
| clarkb | I'm now looking through the delete path | 16:15 |
| clarkb | (but my hunch is we tried to delete those servers and they failed to delete, this caused them to still count against quota but from our side we weren't able to add them to our quota usage) | 16:15 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove pypy job from bindep https://review.opendev.org/720543 | 16:17 |
| *** sshnaidm has joined #openstack-infra | 16:17 | |
| *** rpittau is now known as rpittau|afk | 16:18 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove pypy templates https://review.opendev.org/720544 | 16:19 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove pypy template https://review.opendev.org/720544 | 16:22 |
| melwitt | clarkb: thanks for taking a look, apologies if there was nothing wrong | 16:30 |
| fungi | so it did eventually get a node, looks like? | 16:32 |
| clarkb | melwitt: well I think I figured it out maybe? And it might even be a nova bug :) | 16:32 |
| fungi | score! | 16:32 |
| clarkb | give me a minute to put a paste together with some links to code and stuff | 16:32 |
| fungi | they need more of those | 16:32 |
| melwitt | lol ... yeah right! | 16:32 |
| *** udesale_ has quit IRC | 16:35 | |
| *** dtantsur is now known as dtantsur|afk | 16:36 | |
| frickler | clarkb: corvus: I've seen a backlog on the check queue of 6-8h for the last couple of days, seemed to me to be the past-easter pre-release rush | 16:37 |
| frickler | having 14h on integrated gate is new, though | 16:37 |
| *** evrardjp has quit IRC | 16:37 | |
| clarkb | ya I think I have it figured out just trying to type it up | 16:37 |
| *** evrardjp has joined #openstack-infra | 16:37 | |
| clarkb | http://paste.openstack.org/show/792243/ | 16:42 |
| clarkb | oh wait I've misread the meta stuff thats our own data colelction | 16:43 |
| clarkb | so this is actually even simpler which is nice | 16:43 |
| clarkb | basically nodepool asked the cloud to delete the instances. It said it did, but it didn't. This broke our ability to properly account for the quota associated with those instances which abcked things up | 16:43 |
| melwitt | ok, delete is async, so 202 or 204 means request was successfully initiated, but could potentially fail if something goes wrong on the compute node in the middle of a delete | 16:48 |
| *** ricolin has quit IRC | 16:48 | |
| melwitt | ok, so metadata was deleted but the server itself was not. would it be possible for us to look at the nova logs to see what happened? | 16:51 |
| fungi | that would likely need mgagne next time he's around | 16:53 |
| melwitt | oh, right, because it's in the node provider | 16:54 |
| *** TomStappaerts has joined #openstack-infra | 16:59 | |
| TomStappaerts | Hi guys, Anyone know when we can expect Centos8 support for Ussuri? We seem to be falling in a gap now where we cannot test it on centos7 because support is dropped and we cannot test it on centos8 because it is not yet supported. | 17:00 |
| *** gfidente is now known as gfidente|afk | 17:04 | |
| AJaeger | TomStappaerts: are you an operator to deploy OpenStack or are you looking at developing an OpenStack project? | 17:07 |
| frickler | TomStappaerts: I'm assuming you are talking about devstack support, to discuss that, the -qa channel would be better. the short answer is: when someone provided the necessary patches | 17:07 |
| frickler | the opendev infrastructure in generel supports centos8 just fine | 17:08 |
| TomStappaerts | Yes, thx. This is indeed for devstack support | 17:08 |
| TomStappaerts | Should have made that clear, I will ask my question in the qa channel :) | 17:09 |
| clarkb | melwitt: I think I decided I was wrong about the metadat not deleting (sorry noted that in irc after making the paste) | 17:09 |
| clarkb | melwitt: I think it was just that the server stuck around and enough things ended up in that state that our quota ccounting got all out of sync and that is what we use to determine if we can handle more requests | 17:10 |
| melwitt | clarkb: thanks, sorry I missed the meaning of that. so it is a problem of nova somehow failing to finish deleting instances. dang, wish I could look at the logs | 17:11 |
| clarkb | melwitt: ya and I think we expect that hence the handler code in nodepool for it. My hunch is the thing that made this extra painful is we had a larger normal of leaked instances than normal | 17:13 |
| clarkb | so we weren't floating around that point and were well over it | 17:13 |
| melwitt | I see | 17:13 |
| *** ociuhandu has quit IRC | 17:13 | |
| melwitt | whoa, a ton of stuff just merged. sweet | 17:15 |
| clarkb | yay zuul is working even when clouds are silly :) | 17:16 |
| *** jpena is now known as jpena|off | 17:18 | |
| melwitt | :) | 17:21 |
| *** seco has quit IRC | 17:22 | |
| clarkb | fwiw we probably don't need to spend too much time on this unless it becomes persistent | 17:24 |
| melwitt | yeah, understood | 17:26 |
| fungi | though we might want to give mgagne a heads up if the deletion problems there persist | 17:27 |
| melwitt | yeah. we'll definitely need to look at compute logs to investigate if this is a recurring issue | 17:29 |
| *** sshnaidm has quit IRC | 17:37 | |
| *** rcernin has quit IRC | 17:37 | |
| *** TomStappaerts has left #openstack-infra | 17:39 | |
| *** diablo_rojo has joined #openstack-infra | 17:51 | |
| *** amoralej is now known as amoralej|off | 17:59 | |
| *** hashar is now known as hasharAway | 18:03 | |
| *** ociuhandu has joined #openstack-infra | 18:09 | |
| *** yolanda has quit IRC | 18:14 | |
| *** ralonsoh has quit IRC | 18:16 | |
| *** diablo_rojo has quit IRC | 18:18 | |
| *** nightmare_unreal has quit IRC | 18:23 | |
| *** icarusfactor has joined #openstack-infra | 18:24 | |
| *** factor has quit IRC | 18:27 | |
| *** yolanda has joined #openstack-infra | 18:28 | |
| *** ociuhandu has quit IRC | 18:31 | |
| *** Lucas_Gray has joined #openstack-infra | 18:34 | |
| *** jamesmcarthur has quit IRC | 18:44 | |
| *** jamesmcarthur has joined #openstack-infra | 18:44 | |
| *** rfolco is now known as rfolco|bbl | 19:01 | |
| *** hasharAway is now known as hashar | 19:05 | |
| *** ociuhandu has joined #openstack-infra | 19:16 | |
| *** ociuhandu has quit IRC | 19:22 | |
| *** jamesmcarthur has quit IRC | 19:29 | |
| *** jamesmcarthur has joined #openstack-infra | 19:33 | |
| *** jamesmcarthur has quit IRC | 19:37 | |
| *** jamesmcarthur has joined #openstack-infra | 19:38 | |
| *** rfolco|bbl is now known as rfolco | 19:46 | |
| *** jamesmcarthur has quit IRC | 20:14 | |
| *** jamesmcarthur has joined #openstack-infra | 20:20 | |
| *** jamesmcarthur has quit IRC | 20:20 | |
| *** derekh has quit IRC | 20:26 | |
| *** derekh has joined #openstack-infra | 20:35 | |
| *** rfolco has quit IRC | 20:36 | |
| *** ccamacho has quit IRC | 20:38 | |
| *** sreejithp has joined #openstack-infra | 20:39 | |
| *** eharney has quit IRC | 20:40 | |
| sreejithp | qq what will be the procedure to delete old email from review.opendev.org | 20:43 |
| *** ociuhandu has joined #openstack-infra | 20:45 | |
| *** xek has quit IRC | 20:47 | |
| *** ociuhandu has quit IRC | 20:49 | |
| fungi | sreejithp: go to https://review.opendev.org/#/settings/contact and it should let you delete any non-preferred address you've registered to your account | 20:54 |
| fungi | if the one you want to delete is marked as your preferred address, switch a different one to be preferred first | 20:55 |
| *** noama has quit IRC | 20:58 | |
| *** armax has quit IRC | 21:04 | |
| *** armax has joined #openstack-infra | 21:05 | |
| *** rfolco has joined #openstack-infra | 21:10 | |
| *** slaweq has quit IRC | 21:14 | |
| sreejithp | fungi, i did change that.. i dont have the old account credentials to delete them | 21:16 |
| sreejithp | will it be possible to delete without credentials | 21:17 |
| *** hashar has quit IRC | 21:17 | |
| clarkb | sreejithp: are you wanting to recover the account or disable it entirely? | 21:24 |
| clarkb | (I think the process might be slightly different depending on what the goal is) | 21:24 |
| sreejithp | clarkb: disable it entirely | 21:24 |
| clarkb | sreejithp: are you able to send email from one of the associated email addresses (this can serve as confirmation that you own the account) if so I think you can email myself or fungi and we can disable the account for you | 21:26 |
| sreejithp | sure | 21:27 |
| clarkb | fungi: ^ fyi (and yes I recognize email si weak verification but unsure we can do much better, also easy enough to reenable an account in the futurei f that is necessary) | 21:32 |
| fungi | yes, i would just set-account --inactive it anyway | 21:34 |
| fungi | so we can always set-account --active again later if requested | 21:34 |
| *** ramishra has quit IRC | 21:37 | |
| *** ociuhandu has joined #openstack-infra | 21:38 | |
| clarkb | sreejithp: fungi account 9079 has been marked inactive | 21:42 |
| *** ociuhandu has quit IRC | 21:44 | |
| sreejithp | thanks | 21:44 |
| openstackgerrit | Luigi Toscano proposed openstack/devstack-gate master: Set up stable/ussuri https://review.opendev.org/715272 | 21:47 |
| fungi | #status log marked gerrit account 9079 inactive at request of sreejithp | 21:53 |
| openstackstatus | fungi: finished logging | 21:53 |
| *** vkmc has joined #openstack-infra | 21:53 | |
| *** pkopec has quit IRC | 21:53 | |
| clarkb | I guess there is a second account looking for that one now | 21:53 |
| clarkb | sreejithp: fungi looking at the db I think this account captures both email addrs | 21:55 |
| clarkb | ya I'm not finding a second account so I think we are done | 21:55 |
| sreejithp | thanks clark, fungi | 21:56 |
| vkmc | hey all... quick question | 21:58 |
| vkmc | I want to add a redirect from https://docs.openstack.org/api-ref/shared-file-systems | 21:58 |
| vkmc | to the actual api-ref that lives in https://docs.openstack.org/api-ref/shared-file-system/ | 21:58 |
| vkmc | I saw this htaccess here https://github.com/openstack/api-site/blob/master/www/.htaccess#L5 | 21:59 |
| vkmc | but seems something is not working? | 21:59 |
| vkmc | maybe I'm looking at the wrong place? | 21:59 |
| *** mihalis68_ has quit IRC | 22:01 | |
| vkmc | clarkb, ^ | 22:01 |
| clarkb | vkmc: is that not a redirect to itself? but I think AJaeger and fungi were debugging issues with fully rooted redirects? | 22:01 |
| vkmc | no... the final s makes all the difference | 22:02 |
| clarkb | I wasn't able to follow that conversation (it had to do with glance docs) but they may have learned somethign that is useful in this context | 22:02 |
| vkmc | I can ask them tomorrow :) | 22:02 |
| fungi | i'm digging into it | 22:04 |
| fungi | i think there's confusion on this one as to which site you're looking at | 22:04 |
| fungi | the .htaccess there seems to be for redirecting from the old api site like http://api.openstack.org/api-ref/shared-file-systems/ | 22:04 |
| fungi | i think you want the docs.openstack.org .htaccess file instead | 22:04 |
| fungi | vkmc: i think https://opendev.org/openstack/openstack-manuals/src/branch/master/www/.htaccess is probably what you're looking for now that api-site content is served from docs.o.o instead of api.o.o | 22:06 |
| clarkb | as a sanity check the vhost on that site allows htaccess redirects and redirectmatches | 22:06 |
| fungi | the glance/cinder specs thing i suspect is an absolute vs relative redirect match problem but i haven't had time to test that theory yet | 22:07 |
| fungi | api.openstack.org is apparently also developer.openstack.org | 22:08 |
| fungi | yep, the .htaccess file for api-site is actually published into afs here: /afs/openstack.org/developer-docs/.htaccess | 22:09 |
| fungi | so it's used for developer.openstack.org urls, not docs.openstack.org urls | 22:09 |
| fungi | the one in openstack-manuals is for docs.openstack.org | 22:09 |
| clarkb | vkmc: ^ I think that means you have to update openstack-manuals .htaccess file | 22:11 |
| fungi | yes, that's what i was trying to say | 22:11 |
| fungi | though folks in the technical writing sig probably know more than we do about it | 22:12 |
| fungi | #openstack-doc channel i think? | 22:12 |
| *** Lucas_Gray has quit IRC | 22:21 | |
| vkmc | clarkb, fungi, oh I see | 22:28 |
| vkmc | sorry, stepped away for a moment | 22:28 |
| vkmc | ok so I can check that repo, maybe ask doc folks | 22:29 |
| fungi | cool, and yeah no need to apologize, i did too ;) | 22:30 |
| vkmc | thanks for the help clarkb and fungi! | 22:34 |
| openstackgerrit | Merged openstack/project-config master: nodepool: use job inheritance https://review.opendev.org/713158 | 22:34 |
| fungi | vkmc: any time! ;) | 22:35 |
| *** rcernin has joined #openstack-infra | 22:39 | |
| *** tkajinam has joined #openstack-infra | 22:39 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add ubuntu-bionic-plain to all regions https://review.opendev.org/720316 | 22:42 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: nodepool: Add more plain images https://review.opendev.org/720318 | 22:42 |
| *** bdodd has quit IRC | 22:46 | |
| *** jamesmcarthur has joined #openstack-infra | 22:48 | |
| *** bdodd has joined #openstack-infra | 22:50 | |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-pip: export ensure_pip_virtualenv_command https://review.opendev.org/718224 | 23:01 |
| *** tosky has quit IRC | 23:09 | |
| *** threestrands has joined #openstack-infra | 23:14 | |
| *** threestrands has quit IRC | 23:15 | |
| *** threestrands has joined #openstack-infra | 23:15 | |
| *** jamesmcarthur has quit IRC | 23:19 | |
| *** rfolco has quit IRC | 23:21 | |
| openstackgerrit | Merged zuul/zuul-jobs master: fetch-zuul-cloner: use ensure-pip https://review.opendev.org/717882 | 23:25 |
| *** dchen has quit IRC | 23:35 | |
| *** dchen has joined #openstack-infra | 23:36 | |
| *** sreejithp has quit IRC | 23:45 | |
| *** sreejithp has joined #openstack-infra | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!