| *** heyongli has quit IRC | 00:00 | |
| *** heyongli has joined #openstack-infra | 00:00 | |
| *** heyongli has quit IRC | 00:10 | |
| *** heyongli has joined #openstack-infra | 00:11 | |
| *** heyongli has quit IRC | 00:20 | |
| *** heyongli has joined #openstack-infra | 00:21 | |
| *** heyongli has quit IRC | 00:31 | |
| *** heyongli has joined #openstack-infra | 00:31 | |
| *** heyongli has quit IRC | 00:41 | |
| *** heyongli has joined #openstack-infra | 00:41 | |
| *** kzaitsev_pi has quit IRC | 00:43 | |
| *** kzaitsev_pi has joined #openstack-infra | 00:51 | |
| *** heyongli has quit IRC | 00:51 | |
| *** heyongli has joined #openstack-infra | 00:52 | |
| *** heyongli has quit IRC | 01:01 | |
| *** heyongli has joined #openstack-infra | 01:02 | |
| *** heyongli has quit IRC | 01:12 | |
| *** heyongli has joined #openstack-infra | 01:12 | |
| *** sthussey has joined #openstack-infra | 01:15 | |
| *** heyongli has quit IRC | 01:22 | |
| *** heyongli has joined #openstack-infra | 01:22 | |
| *** felipemonteiro has quit IRC | 01:22 | |
| *** felipemonteiro has joined #openstack-infra | 01:28 | |
| *** heyongli has quit IRC | 01:32 | |
| *** heyongli has joined #openstack-infra | 01:33 | |
| portdirect | hey - I think since the change to ansible 2.5 we are seeing some odd behavior on our gate roles that I cannot reproduce locally | 01:35 |
|---|---|---|
| clarkb | portdirect: have an example? | 01:36 |
| portdirect | https://review.openstack.org/#/c/574055/3/roles/deploy-kubeadm-aio-node/tasks/main.yaml | 01:36 |
| clarkb | (I'm not sure I'll be able to debug it now but it will help to have specific examples and logs) | 01:36 |
| portdirect | l41 | 01:36 |
| portdirect | for some reason this (and the following similar command) | 01:37 |
| portdirect | now always fail on the 1st node | 01:37 |
| portdirect | http://logs.openstack.org/55/574055/3/check/openstack-helm-infra-ubuntu/9e9ac15/ara-report/result/54b51530-2857-4003-83e2-53fe446808b0/ | 01:37 |
| portdirect | but run fine on others: http://logs.openstack.org/55/574055/3/check/openstack-helm-infra-ubuntu/9e9ac15/ara-report/result/b173a14c-c8a3-407d-9e7b-c35b44fdee5c/ | 01:37 |
| portdirect | I suspect i can just get round this by running as root, but it seems a bit dirty | 01:38 |
| clarkb | those tracebacks are related to our callback module writing to the log file I think (this is what gives us the console log) | 01:39 |
| clarkb | but rather than console-None it should be console-builduuid iirc | 01:39 |
| portdirect | any idea how i could hack round this in the short-term? | 01:40 |
| *** felipemonteiro has quit IRC | 01:40 | |
| clarkb | I think we need to figure out why the value there is None instead of the uuid | 01:41 |
| clarkb | at least to understand what is going on | 01:41 |
| clarkb | I wonder if that pause is breaking it | 01:42 |
| clarkb | maybe instead of a pause do a command sleep 60? | 01:42 |
| *** felipemonteiro has joined #openstack-infra | 01:42 | |
| *** heyongli has quit IRC | 01:42 | |
| *** heyongli has joined #openstack-infra | 01:43 | |
| portdirect | i put the pause in to try and see what was going on ;) | 01:44 |
| portdirect | we hit this in master atm - so since thurs all our gates started to fail :( | 01:45 |
| portdirect | this is how its manifesting in master atm: http://logs.openstack.org/07/573007/4/check/openstack-helm-infra-ubuntu/4c07def/ara-report/result/d87ec2dc-4047-4ff9-8f97-4f5bc3c74b49/ | 01:46 |
| portdirect | https://github.com/openstack/openstack-helm-infra/blame/master/roles/deploy-kubeadm-aio-node/tasks/main.yaml#L38-L44 | 01:47 |
| portdirect | which was running ok for the last 8 months | 01:47 |
| clarkb | that doesn't appear to have crashing console logs | 01:47 |
| clarkb | (makes me wonder if the pause is actually at least partially at fault for those) | 01:47 |
| clarkb | portdirect: the error there is stdout isn't valid on that dict | 01:51 |
| clarkb | the internet has examples using stdout_lines. I'm trying to find more definitive info though | 01:51 |
| portdirect | i was assuming that that was as it couldnt open somthing to dump stdout to? | 01:51 |
| clarkb | or it is evaluating the until prior to the command running the first time? looks like stdout and stdout_lines should both be valid | 01:53 |
| *** heyongli has quit IRC | 01:53 | |
| portdirect | as it allways seems to effect only the 1st node that trys to run a delegated task that uses stdout | 01:53 |
| portdirect | eg: http://logs.openstack.org/07/573007/4/check/openstack-helm-infra-ubuntu/4c07def/ara-report/result/436bb033-12c5-420f-a76b-0365f6171229/ | 01:53 |
| clarkb | 'Error from server (NotFound): nodes "ubuntu-xenial-rax-dfw-0004350526" not found' is suspicuous though | 01:53 |
| *** heyongli has joined #openstack-infra | 01:53 | |
| clarkb | http://logs.openstack.org/07/573007/4/check/openstack-helm-infra-ubuntu/4c07def/zuul-info/inventory.yaml ya that node isn't in the inventory | 01:53 |
| portdirect | this is normal for the 1st 15 seconds or so | 01:54 |
| clarkb | I think the delegate_to may be at fault if it is delegating to ubuntu-xenial-rax-dfw-0004350526 | 01:54 |
| clarkb | that is my best guess at this point. Maybe try hardcoding it to node-1 or node-2 or primary depending on which is the kube master to check? | 01:56 |
| portdirect | are you getting that from the command thats being run? | 01:57 |
| portdirect | (the host?) | 01:57 |
| portdirect | as that should be reffering to the hostname of the node, not the ansible node-name | 01:57 |
| clarkb | I'm getting that from your ara link above http://logs.openstack.org/07/573007/4/check/openstack-helm-infra-ubuntu/4c07def/ara-report/result/d87ec2dc-4047-4ff9-8f97-4f5bc3c74b49/ | 01:57 |
| clarkb | says "Error from server (NotFound): nodes "ubuntu-xenial-rax-dfw-0004350526" not found | 01:58 |
| clarkb | and you have a delegate to in that task | 01:58 |
| *** claudiub has quit IRC | 01:58 | |
| portdirect | that command looks right | 01:58 |
| portdirect | its the hostname of node-1: http://logs.openstack.org/07/573007/4/check/openstack-helm-infra-ubuntu/4c07def/zuul-info/host-info.node-1.yaml | 01:58 |
| clarkb | I'm reading that to mean the delegate to is an invalid (not in inventory) host | 01:58 |
| portdirect | its being delegated to this node: https://github.com/openstack/openstack-helm-infra/blame/master/roles/deploy-kubeadm-aio-node/tasks/main.yaml#L20 | 01:58 |
| clarkb | sure but its not what is in the invetory? http://logs.openstack.org/07/573007/4/check/openstack-helm-infra-ubuntu/4c07def/zuul-info/inventory.yaml | 01:59 |
| clarkb | portdirect: where do you include these tasks? is it being run against all nodes? | 02:02 |
| clarkb | (other idea is maybe the k8s cluster task is killing networking? | 02:02 |
| portdirect | I think we may be slightly talking accross each other | 02:03 |
| *** heyongli has quit IRC | 02:03 | |
| portdirect | the node names being refered to in the output - refer to the fdqn's of the nodes, not their names in the inventory | 02:03 |
| *** heyongli has joined #openstack-infra | 02:03 | |
| clarkb | portdirect: right but it is saying the node is not found implying the value set in the delegate to is not correct | 02:04 |
| clarkb | (at least that is how I am reading it) | 02:04 |
| portdirect | its been deligated fine (why the command is running) | 02:04 |
| *** calebb has quit IRC | 02:04 | |
| portdirect | but the node is noth found - as k8s has not yet started up | 02:04 |
| clarkb | but it isn't running? | 02:04 |
| portdirect | this is the wait for k8s to be up on that node | 02:05 |
| clarkb | ah ok I was misreading the output as from asnible | 02:05 |
| portdirect | :) | 02:05 |
| clarkb | got it | 02:05 |
| portdirect | so yeah - since thurday is seems that this fails for the 1st node to get to the point of delegating this task to the node used to bootstrap the cluster | 02:06 |
| clarkb | "The conditional check 'task_result.stdout == 'Ready'' failed. The error was: error while evaluating conditional (task_result.stdout == 'Ready'): 'dict object' has no attribute 'stdout' | 02:06 |
| clarkb | is the error then | 02:06 |
| portdirect | yup | 02:07 |
| portdirect | even if the command fails, we should have a stdout attribute in the dict should we not? | 02:07 |
| portdirect | even if its just "" | 02:08 |
| *** calebb has joined #openstack-infra | 02:09 | |
| *** psachin` has joined #openstack-infra | 02:09 | |
| clarkb | ya reading the command module it seems to default to b'' if nothing else | 02:10 |
| clarkb | (for stdout) | 02:10 |
| clarkb | also ara shows stdout there | 02:10 |
| clarkb | the task itself is set to retry 120 times but ara is recording it as the 121st occurence | 02:11 |
| *** slaweq has joined #openstack-infra | 02:11 | |
| portdirect | clarkb: this seems to be working: https://review.openstack.org/#/c/574055/4/roles/deploy-kubeadm-aio-node/tasks/main.yaml | 02:11 |
| portdirect | (the 1st new task is irrelevant) | 02:12 |
| clarkb | current hunch is that stdout isn't there because it isn't actually running the command on the 121st time though | 02:12 |
| clarkb | and its just failing 120 times | 02:12 |
| *** jesslampe has joined #openstack-infra | 02:12 | |
| *** heyongli has quit IRC | 02:13 | |
| *** heyongli has joined #openstack-infra | 02:14 | |
| portdirect | it looks to be failing on the 1st ? http://logs.openstack.org/07/573007/4/check/openstack-helm-infra-ubuntu/4c07def/job-output.txt.gz#_2018-06-07_20_21_25_776164 | 02:15 |
| portdirect | just from the timestamps | 02:15 |
| *** slaweq has quit IRC | 02:16 | |
| clarkb | right I think the host not found error is happening 120 times | 02:16 |
| clarkb | then on the 121st there is no stdout because the command doesn't actually run | 02:16 |
| portdirect | it cant be running it 121 limes in a second? | 02:17 |
| portdirect | lol - times sry | 02:17 |
| clarkb | it says retries 121 | 02:17 |
| clarkb | but ya delay is 5 seconds so I dunno | 02:18 |
| portdirect | anyway - moving to run as root seems to fix it | 02:18 |
| portdirect | I'll try and get a zuul up locally (as im ironically working on a chart for it atm :) ) | 02:18 |
| clarkb | to test this I would remove the until and retries | 02:19 |
| portdirect | but i suspect theres an issue with the console streamer with deligation | 02:19 |
| portdirect | roger | 02:19 |
| portdirect | I'll get another ps up | 02:19 |
| clarkb | run it once, wait some long period of time, try it again and check for stdout being happier and if the result changes on the command itself | 02:19 |
| clarkb | mostly I suspect some weird behavior in new ansible around until and retries | 02:19 |
| portdirect | Im running with root here and all seems well: http://zuul.openstack.org/stream.html?uuid=cf56688233c94493a79115e884725803&logfile=console.log | 02:19 |
| clarkb | is it possible that kubectl needs root on that node for some reason? | 02:20 |
| clarkb | (needing to write to a file owned by root or read a config file owned by root, etc) | 02:20 |
| portdirect | shouldnt | 02:22 |
| portdirect | and we run these playbooks internally without any issue | 02:22 |
| clarkb | what version of ansible ? | 02:22 |
| portdirect | 2.5.4 | 02:22 |
| clarkb | (we went from 2.3 to 2.5) | 02:22 |
| portdirect | we launch about 20-50 clusters a day using this in our internal ci | 02:23 |
| clarkb | in that case ya I think the thing to check is if not using until and retries a better or different error message | 02:23 |
| *** heyongli has quit IRC | 02:23 | |
| clarkb | then work backward from there | 02:24 |
| portdirect | and from looking at our gates - it was the cut from 2.3.3 to 2.5 that caused breakage in -infra | 02:24 |
| *** heyongli has joined #openstack-infra | 02:24 | |
| clarkb | /etc/kubernetes/admin.conf is readable by the zuul user? | 02:24 |
| portdirect | no | 02:24 |
| clarkb | (just noticing that in the task itself) | 02:24 |
| clarkb | wouldn't that require a become then? | 02:25 |
| portdirect | but theres a copy of that at ~/.kube/config that is | 02:25 |
| portdirect | ans used by kubectl by default | 02:25 |
| portdirect | so we only need to manually spec it when using root | 02:25 |
| clarkb | maybe double check that file exists and is readable by the zuul user too? | 02:26 |
| clarkb | (just looking at the delta between what works and doesn't | 02:26 |
| clarkb | and that is something in that delta) | 02:27 |
| portdirect | http://logs.openstack.org/16/572116/1/check/openstack-helm-infra-ubuntu/1f16ccb/job-output.txt.gz#_2018-06-04_15_03_09_145858 | 02:27 |
| portdirect | ^^ last gate that ran on 2.3.3 | 02:27 |
| portdirect | nothing has changed/merged since. | 02:27 |
| portdirect | checks that were passing, now fail | 02:28 |
| clarkb | ya could be a difference in default perms for file resources in new ansible for example | 02:31 |
| clarkb | I'm just trying to rule out things that could cause kubectl to fail and things that could cause ansible to fail as a result | 02:31 |
| clarkb | yes the ansible update may be at fault, but unless we understand the root cause we aren't going to get anywhere fixing it | 02:32 |
| portdirect | clarkb: I'll try just running an `ls` command - i suspect this will have the same effect | 02:32 |
| clarkb | possibly old ansible assigned ownership and perms on files based on teh current user and new one assumes root if not set and if you run your playbooks as root locally then they work but zuul uses the zuul user (lots of variables at play and I'm just trying to make sure we understand what broke) | 02:33 |
| *** heyongli has quit IRC | 02:34 | |
| clarkb | (I don't think that is likely, btu I do think it is possible) | 02:34 |
| *** heyongli has joined #openstack-infra | 02:34 | |
| portdirect | i dont run as root locally :( | 02:36 |
| portdirect | thanks for your help dude - I'll ping you when i get results from this: https://review.openstack.org/#/c/574065/1/roles/deploy-kubeadm-aio-node/tasks/main.yaml | 02:36 |
| *** felipemonteiro has quit IRC | 02:37 | |
| clarkb | cool, its dinner time so I may not catch up until tomorrow morning, but I think these couple debugging tasks will likely help. Worst case we learn nothing and its deep dive into zuul (maybe holding a node for you) | 02:37 |
| portdirect | gotta eat :D cheers clarkb | 02:38 |
| *** heyongli has quit IRC | 02:44 | |
| *** heyongli has joined #openstack-infra | 02:44 | |
| *** dbecker has quit IRC | 02:47 | |
| *** felipemonteiro has joined #openstack-infra | 02:50 | |
| *** heyongli has quit IRC | 02:54 | |
| *** heyongli has joined #openstack-infra | 02:55 | |
| *** dbecker has joined #openstack-infra | 03:02 | |
| *** heyongli has quit IRC | 03:04 | |
| *** heyongli has joined #openstack-infra | 03:05 | |
| *** heyongli has quit IRC | 03:15 | |
| *** heyongli has joined #openstack-infra | 03:15 | |
| *** slaweq has joined #openstack-infra | 03:22 | |
| *** sthussey has quit IRC | 03:25 | |
| *** heyongli has quit IRC | 03:25 | |
| *** heyongli has joined #openstack-infra | 03:25 | |
| *** slaweq has quit IRC | 03:27 | |
| *** heyongli has quit IRC | 03:35 | |
| *** heyongli has joined #openstack-infra | 03:36 | |
| *** jesslampe has quit IRC | 03:37 | |
| *** heyongli has quit IRC | 03:45 | |
| *** heyongli has joined #openstack-infra | 03:46 | |
| *** heyongli has quit IRC | 03:56 | |
| *** heyongli has joined #openstack-infra | 03:56 | |
| portdirect | clarkb: seeing the same thing with just an echo | 04:04 |
| portdirect | http://logs.openstack.org/65/574065/3/check/openstack-helm-infra-centos/07e424c/ara-report/result/81d61ab5-3771-43ea-9dba-a6ddd210f60d/ | 04:04 |
| portdirect | https://review.openstack.org/#/c/574065/3/roles/deploy-kubeadm-aio-node/tasks/main.yaml | 04:04 |
| *** heyongli has quit IRC | 04:06 | |
| *** heyongli has joined #openstack-infra | 04:06 | |
| *** felipemonteiro has quit IRC | 04:11 | |
| *** jesslampe has joined #openstack-infra | 04:15 | |
| *** heyongli has quit IRC | 04:16 | |
| *** heyongli has joined #openstack-infra | 04:17 | |
| *** heyongli has quit IRC | 04:26 | |
| *** heyongli has joined #openstack-infra | 04:27 | |
| *** heyongli has quit IRC | 04:37 | |
| *** heyongli has joined #openstack-infra | 04:37 | |
| *** heyongli has quit IRC | 04:47 | |
| *** heyongli has joined #openstack-infra | 04:47 | |
| *** hongbin has joined #openstack-infra | 04:50 | |
| *** hongbin has quit IRC | 04:52 | |
| *** heyongli has quit IRC | 04:57 | |
| *** heyongli has joined #openstack-infra | 04:58 | |
| *** heyongli has quit IRC | 05:07 | |
| *** heyongli has joined #openstack-infra | 05:08 | |
| *** heyongli has quit IRC | 05:18 | |
| *** heyongli has joined #openstack-infra | 05:18 | |
| *** yamamoto has joined #openstack-infra | 05:26 | |
| *** heyongli has quit IRC | 05:28 | |
| *** heyongli has joined #openstack-infra | 05:28 | |
| *** yamamoto has quit IRC | 05:31 | |
| *** heyongli has quit IRC | 05:38 | |
| *** heyongli has joined #openstack-infra | 05:39 | |
| *** xinliang3 has quit IRC | 05:48 | |
| *** heyongli has quit IRC | 05:48 | |
| *** xinliang3 has joined #openstack-infra | 05:49 | |
| *** heyongli has joined #openstack-infra | 05:49 | |
| *** heyongli has quit IRC | 05:59 | |
| *** heyongli has joined #openstack-infra | 05:59 | |
| *** slaweq has joined #openstack-infra | 06:05 | |
| *** heyongli has quit IRC | 06:09 | |
| *** heyongli has joined #openstack-infra | 06:09 | |
| *** slaweq has quit IRC | 06:10 | |
| *** lifeless has joined #openstack-infra | 06:12 | |
| *** heyongli has quit IRC | 06:19 | |
| *** heyongli has joined #openstack-infra | 06:20 | |
| *** heyongli has quit IRC | 06:29 | |
| *** heyongli has joined #openstack-infra | 06:30 | |
| *** iranzo has joined #openstack-infra | 06:33 | |
| *** heyongli has quit IRC | 06:40 | |
| *** heyongli has joined #openstack-infra | 06:40 | |
| *** heyongli has quit IRC | 06:50 | |
| *** heyongli has joined #openstack-infra | 06:50 | |
| *** heyongli has quit IRC | 07:00 | |
| *** heyongli has joined #openstack-infra | 07:01 | |
| *** lifeless_ has joined #openstack-infra | 07:10 | |
| *** lifeless has quit IRC | 07:10 | |
| *** heyongli has quit IRC | 07:10 | |
| *** heyongli has joined #openstack-infra | 07:11 | |
| *** jbadiapa_ has quit IRC | 07:13 | |
| *** pcichy has quit IRC | 07:17 | |
| *** pcichy has joined #openstack-infra | 07:17 | |
| *** jbadiapa_ has joined #openstack-infra | 07:18 | |
| *** lifeless_ is now known as lifeless | 07:18 | |
| *** Bhujay has joined #openstack-infra | 07:19 | |
| *** heyongli has quit IRC | 07:21 | |
| *** heyongli has joined #openstack-infra | 07:21 | |
| *** heyongli has quit IRC | 07:31 | |
| *** heyongli has joined #openstack-infra | 07:31 | |
| *** andreww has joined #openstack-infra | 07:38 | |
| *** jesslampe has quit IRC | 07:39 | |
| *** jesslampe has joined #openstack-infra | 07:39 | |
| *** Bhujay has quit IRC | 07:39 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Add release note about upgrading for multi-label https://review.openstack.org/573827 | 07:40 |
| *** xarses_ has quit IRC | 07:40 | |
| *** Bhujay has joined #openstack-infra | 07:41 | |
| frickler | infra-root: I'm seeing this console-None issue also for devstack-multinode, which hasn't been modified lately I think: http://logs.openstack.org/20/564220/1/check/devstack-multinode/f1fe43d/ara-report/result/70872a34-300e-4da0-b94f-7f58daf30517/ | 07:41 |
| *** heyongli has quit IRC | 07:41 | |
| frickler | so guessing some behaviour change in zuul could be triggering this | 07:41 |
| *** heyongli has joined #openstack-infra | 07:42 | |
| frickler | ah, or ansible change. seems the common factor might indeed be a "become: yes" together with a non-root "become_user" | 07:44 |
| *** heyongli has quit IRC | 07:51 | |
| *** heyongli has joined #openstack-infra | 07:52 | |
| *** heyongli has quit IRC | 08:02 | |
| *** heyongli has joined #openstack-infra | 08:02 | |
| frickler | possibly unrelated, I'm still seeing POST_FAILURE on osc-functional-devstack after the original tox failure got fixed http://logs.openstack.org/02/573202/2/check/osc-functional-devstack/fc3d20a/ | 08:03 |
| frickler | I can see no error at all there in the logs, so probably failing very late. guess I'll hold another node to debug that | 08:04 |
| *** dhajare has joined #openstack-infra | 08:09 | |
| *** heyongli has quit IRC | 08:12 | |
| *** heyongli has joined #openstack-infra | 08:12 | |
| *** dklyle has quit IRC | 08:14 | |
| *** heyongli has quit IRC | 08:22 | |
| *** jesslampe has quit IRC | 08:23 | |
| *** heyongli has joined #openstack-infra | 08:23 | |
| AJaeger | portdirect, clarkb: seeing your discussion, are you sure it's ansible 2.5? https://review.openstack.org/#/c/572313/ changes hostname - that merged also last week. | 08:31 |
| *** heyongli has quit IRC | 08:32 | |
| *** heyongli has joined #openstack-infra | 08:33 | |
| *** heyongli has quit IRC | 08:43 | |
| *** heyongli has joined #openstack-infra | 08:43 | |
| *** armaan has joined #openstack-infra | 08:51 | |
| *** heyongli has quit IRC | 08:53 | |
| *** heyongli has joined #openstack-infra | 08:53 | |
| *** benj_ has quit IRC | 09:02 | |
| *** heyongli has quit IRC | 09:03 | |
| *** heyongli has joined #openstack-infra | 09:04 | |
| *** claudiub has joined #openstack-infra | 09:11 | |
| *** heyongli has quit IRC | 09:13 | |
| *** heyongli has joined #openstack-infra | 09:14 | |
| *** psachin` has quit IRC | 09:22 | |
| *** heyongli has quit IRC | 09:24 | |
| *** heyongli has joined #openstack-infra | 09:24 | |
| *** heyongli has quit IRC | 09:34 | |
| *** heyongli has joined #openstack-infra | 09:34 | |
| *** yamamoto has joined #openstack-infra | 09:34 | |
| *** pbourke has quit IRC | 09:42 | |
| *** pbourke has joined #openstack-infra | 09:43 | |
| *** heyongli has quit IRC | 09:44 | |
| *** heyongli has joined #openstack-infra | 09:45 | |
| *** heyongli has quit IRC | 09:54 | |
| *** heyongli has joined #openstack-infra | 09:55 | |
| *** yamamoto has quit IRC | 10:03 | |
| *** heyongli has quit IRC | 10:05 | |
| *** heyongli has joined #openstack-infra | 10:05 | |
| *** sshnaidm has joined #openstack-infra | 10:15 | |
| *** heyongli has quit IRC | 10:15 | |
| *** heyongli has joined #openstack-infra | 10:15 | |
| *** heyongli has quit IRC | 10:25 | |
| *** heyongli has joined #openstack-infra | 10:26 | |
| *** sshnaidm has quit IRC | 10:28 | |
| *** lifeless_ has joined #openstack-infra | 10:29 | |
| *** lifeless has quit IRC | 10:30 | |
| *** heyongli has quit IRC | 10:35 | |
| *** heyongli has joined #openstack-infra | 10:36 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Improve logging around tenant reconfiguration https://review.openstack.org/574077 | 10:39 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: tox-cover https://review.openstack.org/574078 | 10:40 |
| *** yamamoto has joined #openstack-infra | 10:42 | |
| *** heyongli has quit IRC | 10:46 | |
| *** heyongli has joined #openstack-infra | 10:46 | |
| *** yamamoto has quit IRC | 10:48 | |
| *** jtomasek has joined #openstack-infra | 10:50 | |
| *** heyongli has quit IRC | 10:56 | |
| *** heyongli has joined #openstack-infra | 10:56 | |
| *** heyongli has quit IRC | 11:06 | |
| *** heyongli has joined #openstack-infra | 11:07 | |
| *** zaro__ has quit IRC | 11:09 | |
| *** heyongli has quit IRC | 11:16 | |
| *** heyongli has joined #openstack-infra | 11:17 | |
| *** yamamoto has joined #openstack-infra | 11:22 | |
| *** heyongli has quit IRC | 11:27 | |
| *** heyongli has joined #openstack-infra | 11:27 | |
| *** yamamoto has quit IRC | 11:32 | |
| *** yamamoto has joined #openstack-infra | 11:33 | |
| *** heyongli has quit IRC | 11:37 | |
| *** heyongli has joined #openstack-infra | 11:37 | |
| *** yamamoto has quit IRC | 11:38 | |
| *** armaan has quit IRC | 11:40 | |
| *** e0ne has joined #openstack-infra | 11:40 | |
| *** armaan has joined #openstack-infra | 11:41 | |
| *** armaan has quit IRC | 11:45 | |
| *** heyongli has quit IRC | 11:47 | |
| *** heyongli has joined #openstack-infra | 11:48 | |
| *** yamamoto has joined #openstack-infra | 11:48 | |
| *** yamamoto has quit IRC | 11:53 | |
| *** heyongli has quit IRC | 11:57 | |
| *** heyongli has joined #openstack-infra | 11:58 | |
| *** tosky has joined #openstack-infra | 12:00 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix tox-cover https://review.openstack.org/574080 | 12:01 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Improve logging around tenant reconfiguration https://review.openstack.org/574077 | 12:03 |
| *** heyongli has quit IRC | 12:08 | |
| *** heyongli has joined #openstack-infra | 12:08 | |
| *** armaan has joined #openstack-infra | 12:08 | |
| *** heyongli has quit IRC | 12:18 | |
| *** heyongli has joined #openstack-infra | 12:18 | |
| *** sshnaidm has joined #openstack-infra | 12:19 | |
| *** heyongli has quit IRC | 12:28 | |
| *** heyongli has joined #openstack-infra | 12:29 | |
| *** yamamoto has joined #openstack-infra | 12:33 | |
| *** e0ne has quit IRC | 12:34 | |
| *** yamamoto has quit IRC | 12:37 | |
| *** heyongli has quit IRC | 12:38 | |
| *** heyongli has joined #openstack-infra | 12:39 | |
| fungi | mtreinish: just a heads up, https://review.openstack.org/538574 seems to break openid logins (and presumably other actions but i wasn't able to log in to test further). logs indicate "Lock wait timeout exceeded; try restarting transaction" when attempting "UPDATE users SET updated_at=%(updated_at)s, last_login=%(last_login)s WHERE users.id = %(users_id)s" | 12:40 |
| *** armaan has quit IRC | 12:41 | |
| *** armaan has joined #openstack-infra | 12:41 | |
| fungi | i've rolled back the installation on sb.o.o as of yesterday morning to the previous commit while we work out whether we need to revert or whether there's a quick fix (but rereading that change, it's not obvious to me how it causes that) | 12:42 |
| fungi | i did at least roll forward/back a few times bisecting, before confirming it's that specific commit triggering the issue | 12:43 |
| *** armaan has quit IRC | 12:46 | |
| *** yamamoto has joined #openstack-infra | 12:48 | |
| *** heyongli has quit IRC | 12:49 | |
| *** heyongli has joined #openstack-infra | 12:49 | |
| *** yamamoto has quit IRC | 12:53 | |
| *** heyongli has quit IRC | 12:59 | |
| *** heyongli has joined #openstack-infra | 12:59 | |
| *** heyongli has quit IRC | 13:09 | |
| *** heyongli has joined #openstack-infra | 13:10 | |
| *** jtomasek has quit IRC | 13:14 | |
| *** trown|outtypewww has quit IRC | 13:18 | |
| *** heyongli has quit IRC | 13:19 | |
| *** heyongli has joined #openstack-infra | 13:20 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Also cache empty unparsed config https://review.openstack.org/574086 | 13:20 |
| *** e0ne has joined #openstack-infra | 13:20 | |
| *** e0ne has quit IRC | 13:22 | |
| *** dave-mcc_ has joined #openstack-infra | 13:23 | |
| *** heyongli has quit IRC | 13:30 | |
| *** heyongli has joined #openstack-infra | 13:30 | |
| *** e0ne has joined #openstack-infra | 13:32 | |
| *** yamamoto has joined #openstack-infra | 13:33 | |
| *** e0ne has quit IRC | 13:37 | |
| *** Bhujay has quit IRC | 13:37 | |
| *** yamamoto has quit IRC | 13:38 | |
| *** Bhujay has joined #openstack-infra | 13:38 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Also cache empty unparsed config https://review.openstack.org/574086 | 13:39 |
| *** heyongli has quit IRC | 13:40 | |
| *** heyongli has joined #openstack-infra | 13:40 | |
| *** yamamoto has joined #openstack-infra | 13:48 | |
| *** heyongli has quit IRC | 13:50 | |
| *** heyongli has joined #openstack-infra | 13:51 | |
| *** yamamoto has quit IRC | 13:53 | |
| *** e0ne has joined #openstack-infra | 13:55 | |
| *** heyongli has quit IRC | 14:00 | |
| *** heyongli has joined #openstack-infra | 14:01 | |
| *** tosky has quit IRC | 14:02 | |
| frickler | the POST_FAILURE seems to stem from a missing role, not sure why there isn't anything in the log, this is from the executor log http://paste.openstack.org/show/723114/ | 14:03 |
| *** dhill_ has joined #openstack-infra | 14:03 | |
| *** e0ne has quit IRC | 14:07 | |
| frickler | also the devstack-multinode jobs passed on a recheck. seems the console-None issue might randomly hit tasks and fails if it hits twice with conflicting users/permissions | 14:08 |
| frickler | this is what I see on the held osc node http://paste.openstack.org/show/723113/ | 14:08 |
| *** heyongli has quit IRC | 14:11 | |
| *** heyongli has joined #openstack-infra | 14:11 | |
| *** e0ne has joined #openstack-infra | 14:15 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Change test prints to log.info https://review.openstack.org/554058 | 14:18 |
| *** yamamoto has joined #openstack-infra | 14:18 | |
| *** sshnaidm has quit IRC | 14:18 | |
| *** yamamoto has quit IRC | 14:18 | |
| *** heyongli has quit IRC | 14:21 | |
| *** heyongli has joined #openstack-infra | 14:21 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix logging in tests to be quiet when expected https://review.openstack.org/554054 | 14:25 |
| frickler | oh, seems AJaeger made a patch that needs to be cherry-picked to stable branches, https://review.openstack.org/574089 | 14:25 |
| *** armaan has joined #openstack-infra | 14:26 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix logging in tests to be quiet when expected https://review.openstack.org/554054 | 14:27 |
| openstackgerrit | Merged openstack-infra/zuul master: Cleanup log streaming error handling https://review.openstack.org/572513 | 14:27 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix remote stream log test https://review.openstack.org/572574 | 14:28 |
| *** e0ne has quit IRC | 14:29 | |
| *** heyongli has quit IRC | 14:31 | |
| *** heyongli has joined #openstack-infra | 14:32 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix logging in tests to be quiet when expected https://review.openstack.org/554054 | 14:33 |
| *** e0ne has joined #openstack-infra | 14:39 | |
| *** heyongli has quit IRC | 14:41 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix logging in tests to be quiet when expected https://review.openstack.org/554054 | 14:42 |
| *** heyongli has joined #openstack-infra | 14:42 | |
| *** e0ne has quit IRC | 14:51 | |
| *** heyongli has quit IRC | 14:52 | |
| *** heyongli has joined #openstack-infra | 14:52 | |
| *** slaweq has joined #openstack-infra | 14:52 | |
| *** dhajare has quit IRC | 14:54 | |
| *** slaweq has quit IRC | 14:58 | |
| *** e0ne has joined #openstack-infra | 14:59 | |
| *** armaan has quit IRC | 15:01 | |
| *** armaan has joined #openstack-infra | 15:02 | |
| *** heyongli has quit IRC | 15:02 | |
| *** heyongli has joined #openstack-infra | 15:02 | |
| *** e0ne has quit IRC | 15:03 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/gerritlib master: Reuse same paramiko connection recreating on errors https://review.openstack.org/412535 | 15:07 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/gerritlib master: Support querying for single group info https://review.openstack.org/412536 | 15:10 |
| *** heyongli has quit IRC | 15:12 | |
| *** heyongli has joined #openstack-infra | 15:13 | |
| *** e0ne has joined #openstack-infra | 15:16 | |
| *** slaweq has joined #openstack-infra | 15:17 | |
| *** slaweq has quit IRC | 15:18 | |
| *** slaweq has joined #openstack-infra | 15:18 | |
| *** yamamoto has joined #openstack-infra | 15:19 | |
| *** e0ne has quit IRC | 15:21 | |
| *** heyongli has quit IRC | 15:22 | |
| *** heyongli has joined #openstack-infra | 15:23 | |
| *** slaweq has quit IRC | 15:23 | |
| *** yamamoto has quit IRC | 15:23 | |
| *** e0ne has joined #openstack-infra | 15:29 | |
| *** felipemonteiro has joined #openstack-infra | 15:31 | |
| *** heyongli has quit IRC | 15:33 | |
| *** heyongli has joined #openstack-infra | 15:33 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/git-review master: Limit reviews by branch when specified https://review.openstack.org/312056 | 15:38 |
| *** armaan has quit IRC | 15:43 | |
| *** heyongli has quit IRC | 15:43 | |
| *** armaan has joined #openstack-infra | 15:43 | |
| *** heyongli has joined #openstack-infra | 15:43 | |
| *** armaan has quit IRC | 15:48 | |
| *** heyongli has quit IRC | 15:53 | |
| *** heyongli has joined #openstack-infra | 15:54 | |
| *** apetrich has quit IRC | 16:00 | |
| *** apetrich has joined #openstack-infra | 16:00 | |
| *** heyongli has quit IRC | 16:03 | |
| *** heyongli has joined #openstack-infra | 16:04 | |
| *** r-daneel has joined #openstack-infra | 16:12 | |
| *** iranzo has quit IRC | 16:12 | |
| *** heyongli has quit IRC | 16:14 | |
| *** heyongli has joined #openstack-infra | 16:14 | |
| *** felipemonteiro has quit IRC | 16:23 | |
| *** heyongli has quit IRC | 16:24 | |
| *** heyongli has joined #openstack-infra | 16:24 | |
| *** heyongli has quit IRC | 16:34 | |
| *** heyongli has joined #openstack-infra | 16:35 | |
| *** r-daneel has quit IRC | 16:35 | |
| *** rkukura has quit IRC | 16:41 | |
| *** heyongli has quit IRC | 16:44 | |
| *** heyongli has joined #openstack-infra | 16:45 | |
| *** Bhujay has quit IRC | 16:47 | |
| *** germs has joined #openstack-infra | 16:48 | |
| *** apetrich has quit IRC | 16:50 | |
| *** apetrich has joined #openstack-infra | 16:50 | |
| *** e0ne has quit IRC | 16:54 | |
| *** heyongli has quit IRC | 16:55 | |
| *** auristor has quit IRC | 16:55 | |
| *** heyongli has joined #openstack-infra | 16:55 | |
| *** auristor has joined #openstack-infra | 16:59 | |
| *** sshnaidm has joined #openstack-infra | 17:00 | |
| *** e0ne has joined #openstack-infra | 17:04 | |
| *** heyongli has quit IRC | 17:05 | |
| *** heyongli has joined #openstack-infra | 17:05 | |
| *** heyongli has quit IRC | 17:15 | |
| *** heyongli has joined #openstack-infra | 17:16 | |
| *** lifeless_ has quit IRC | 17:16 | |
| *** heyongli has quit IRC | 17:25 | |
| *** slaweq has joined #openstack-infra | 17:26 | |
| *** heyongli has joined #openstack-infra | 17:26 | |
| *** felipemonteiro has joined #openstack-infra | 17:27 | |
| *** slaweq_ has joined #openstack-infra | 17:33 | |
| *** heyongli has quit IRC | 17:36 | |
| *** slaweq has quit IRC | 17:36 | |
| *** heyongli has joined #openstack-infra | 17:36 | |
| *** hongbin has joined #openstack-infra | 17:40 | |
| *** heyongli has quit IRC | 17:46 | |
| *** heyongli has joined #openstack-infra | 17:46 | |
| *** jbadiapa_ has quit IRC | 17:56 | |
| *** heyongli has quit IRC | 17:56 | |
| *** heyongli has joined #openstack-infra | 17:57 | |
| *** heyongli has quit IRC | 18:07 | |
| *** heyongli has joined #openstack-infra | 18:07 | |
| *** heyongli has quit IRC | 18:17 | |
| *** heyongli has joined #openstack-infra | 18:17 | |
| *** slaweq_ has quit IRC | 18:24 | |
| *** heyongli has quit IRC | 18:27 | |
| *** heyongli has joined #openstack-infra | 18:27 | |
| *** felipemonteiro has quit IRC | 18:33 | |
| *** heyongli has quit IRC | 18:37 | |
| *** heyongli has joined #openstack-infra | 18:38 | |
| *** agopi has joined #openstack-infra | 18:41 | |
| *** heyongli has quit IRC | 18:48 | |
| *** heyongli has joined #openstack-infra | 18:48 | |
| *** heyongli has quit IRC | 18:58 | |
| *** heyongli has joined #openstack-infra | 18:58 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-cgit master: Update Gemfile for Zuulv3 https://review.openstack.org/571987 | 19:04 |
| *** jamesmcarthur has joined #openstack-infra | 19:04 | |
| *** heyongli has quit IRC | 19:08 | |
| *** heyongli has joined #openstack-infra | 19:08 | |
| *** agopi has quit IRC | 19:13 | |
| *** heyongli has quit IRC | 19:18 | |
| *** heyongli has joined #openstack-infra | 19:19 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/storyboard master: use required enums to validate 'type' args https://review.openstack.org/545170 | 19:19 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/storyboard master: mark worklist filter_criteria as a required field https://review.openstack.org/545405 | 19:19 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/storyboard master: mark FilterCriterion title as a mandatory field https://review.openstack.org/545406 | 19:19 |
| *** felipemonteiro has joined #openstack-infra | 19:24 | |
| *** heyongli has quit IRC | 19:29 | |
| *** heyongli has joined #openstack-infra | 19:29 | |
| *** bobh has joined #openstack-infra | 19:34 | |
| *** heyongli has quit IRC | 19:39 | |
| *** heyongli has joined #openstack-infra | 19:39 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-cgit master: Update Gemfile for Zuulv3 https://review.openstack.org/571987 | 19:42 |
| *** bobh has quit IRC | 19:45 | |
| *** lifeless has joined #openstack-infra | 19:45 | |
| *** heyongli has quit IRC | 19:49 | |
| *** tosky has joined #openstack-infra | 19:49 | |
| *** heyongli has joined #openstack-infra | 19:49 | |
| *** heyongli has quit IRC | 19:59 | |
| *** heyongli has joined #openstack-infra | 20:00 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-cgit master: Update Gemfile for Zuulv3 https://review.openstack.org/571987 | 20:03 |
| *** jamesmcarthur has quit IRC | 20:09 | |
| *** heyongli has quit IRC | 20:10 | |
| *** heyongli has joined #openstack-infra | 20:10 | |
| *** jamesmcarthur has joined #openstack-infra | 20:13 | |
| *** jamesmcarthur has quit IRC | 20:19 | |
| *** heyongli has quit IRC | 20:20 | |
| *** heyongli has joined #openstack-infra | 20:20 | |
| *** jamesmcarthur has joined #openstack-infra | 20:24 | |
| *** jamesmcarthur has quit IRC | 20:29 | |
| *** heyongli has quit IRC | 20:30 | |
| *** heyongli has joined #openstack-infra | 20:30 | |
| *** logan- has quit IRC | 20:31 | |
| *** logan- has joined #openstack-infra | 20:32 | |
| *** felipemonteiro has quit IRC | 20:35 | |
| *** jamesmcarthur has joined #openstack-infra | 20:35 | |
| *** jamesmcarthur has quit IRC | 20:39 | |
| *** heyongli has quit IRC | 20:40 | |
| *** heyongli has joined #openstack-infra | 20:41 | |
| *** heyongli has quit IRC | 20:51 | |
| *** heyongli has joined #openstack-infra | 20:51 | |
| *** slaweq has joined #openstack-infra | 20:52 | |
| *** heyongli has quit IRC | 21:01 | |
| *** heyongli has joined #openstack-infra | 21:01 | |
| *** agopi has joined #openstack-infra | 21:02 | |
| *** jamesmcarthur has joined #openstack-infra | 21:10 | |
| *** heyongli has quit IRC | 21:11 | |
| *** heyongli has joined #openstack-infra | 21:11 | |
| *** jamesmcarthur has quit IRC | 21:14 | |
| *** heyongli has quit IRC | 21:21 | |
| *** heyongli has joined #openstack-infra | 21:22 | |
| *** jamesmcarthur has joined #openstack-infra | 21:27 | |
| *** hongbin has quit IRC | 21:28 | |
| *** jamesmcarthur has quit IRC | 21:31 | |
| *** heyongli has quit IRC | 21:32 | |
| *** heyongli has joined #openstack-infra | 21:32 | |
| *** jamesmcarthur has joined #openstack-infra | 21:36 | |
| *** xarses_ has joined #openstack-infra | 21:39 | |
| *** r-daneel has joined #openstack-infra | 21:40 | |
| *** heyongli has quit IRC | 21:42 | |
| *** heyongli has joined #openstack-infra | 21:42 | |
| *** andreww has quit IRC | 21:42 | |
| *** slaweq has quit IRC | 21:48 | |
| *** r-daneel has quit IRC | 21:48 | |
| *** jamesmcarthur has quit IRC | 21:49 | |
| *** heyongli has quit IRC | 21:52 | |
| *** heyongli has joined #openstack-infra | 21:52 | |
| *** jamesmcarthur has joined #openstack-infra | 21:53 | |
| *** nhicher has joined #openstack-infra | 21:55 | |
| *** hongbin has joined #openstack-infra | 21:57 | |
| *** jamesmcarthur has quit IRC | 21:58 | |
| *** jamesmcarthur has joined #openstack-infra | 21:59 | |
| *** heyongli has quit IRC | 22:02 | |
| *** heyongli has joined #openstack-infra | 22:03 | |
| *** r-daneel has joined #openstack-infra | 22:03 | |
| *** jamesmcarthur has quit IRC | 22:07 | |
| *** jamesmcarthur has joined #openstack-infra | 22:08 | |
| *** heyongli has quit IRC | 22:13 | |
| *** heyongli has joined #openstack-infra | 22:13 | |
| *** kien-ha has joined #openstack-infra | 22:19 | |
| *** e0ne has quit IRC | 22:20 | |
| *** jamesmcarthur has quit IRC | 22:20 | |
| *** jamesmcarthur has joined #openstack-infra | 22:21 | |
| *** heyongli has quit IRC | 22:23 | |
| *** heyongli has joined #openstack-infra | 22:23 | |
| *** bobh has joined #openstack-infra | 22:25 | |
| *** bobh has quit IRC | 22:28 | |
| *** rcernin has joined #openstack-infra | 22:32 | |
| *** lifeless has quit IRC | 22:33 | |
| *** heyongli has quit IRC | 22:33 | |
| *** heyongli has joined #openstack-infra | 22:33 | |
| *** lifeless has joined #openstack-infra | 22:39 | |
| *** krtaylor has quit IRC | 22:40 | |
| *** heyongli has quit IRC | 22:43 | |
| *** heyongli has joined #openstack-infra | 22:44 | |
| *** heyongli has quit IRC | 22:54 | |
| *** heyongli has joined #openstack-infra | 22:54 | |
| *** krtaylor has joined #openstack-infra | 22:55 | |
| *** hongbin has quit IRC | 22:56 | |
| *** dbecker has quit IRC | 22:58 | |
| *** jamesmcarthur has quit IRC | 23:03 | |
| *** heyongli has quit IRC | 23:04 | |
| *** heyongli has joined #openstack-infra | 23:04 | |
| *** aeng has joined #openstack-infra | 23:11 | |
| *** heyongli has quit IRC | 23:14 | |
| *** heyongli has joined #openstack-infra | 23:14 | |
| *** bobh has joined #openstack-infra | 23:15 | |
| *** dingyichen has joined #openstack-infra | 23:16 | |
| *** agopi has quit IRC | 23:16 | |
| *** felipemonteiro has joined #openstack-infra | 23:18 | |
| *** heyongli has quit IRC | 23:24 | |
| *** heyongli has joined #openstack-infra | 23:25 | |
| *** heyongli has quit IRC | 23:35 | |
| *** heyongli has joined #openstack-infra | 23:35 | |
| *** heyongli has quit IRC | 23:45 | |
| *** heyongli has joined #openstack-infra | 23:45 | |
| openstackgerrit | Tin Lam proposed openstack-infra/project-config master: Add py35 and readthedocs jobs https://review.openstack.org/574102 | 23:47 |
| *** bobh has quit IRC | 23:48 | |
| *** claudiub has quit IRC | 23:50 | |
| *** tosky has quit IRC | 23:52 | |
| *** bobh has joined #openstack-infra | 23:52 | |
| *** heyongli has quit IRC | 23:55 | |
| *** heyongli has joined #openstack-infra | 23:55 | |
| *** agopi has joined #openstack-infra | 23:56 | |
| *** slaweq has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!