| *** UtahDave has joined #openstack-infra | 00:00 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Added documentation for REST API layer. https://review.openstack.org/69212 | 00:02 |
|---|---|---|
| *** UtahDave has quit IRC | 00:06 | |
| *** UtahDave has joined #openstack-infra | 00:07 | |
| *** matsuhashi has joined #openstack-infra | 00:09 | |
| *** dcramer_ has joined #openstack-infra | 00:16 | |
| openstackgerrit | A change was merged to openstack/requirements: Add python-openstackclient https://review.openstack.org/64562 | 00:17 |
| *** Guest57266 is now known as med_ | 00:26 | |
| *** UtahDave has quit IRC | 00:27 | |
| *** med_ is now known as medberry | 00:27 | |
| *** medberry is now known as med_ | 00:27 | |
| *** matsuhashi has quit IRC | 00:33 | |
| lifeless | fungi: hey | 00:34 |
| lifeless | gate-tripleo-deploy: queued (non-voting) | 00:34 |
| lifeless | fungi: for all our jobs - how can I debug whatever is wrong ? | 00:34 |
| lifeless | (or clarkb or mordred ^ ) | 00:35 |
| * lifeless doesn't want to disturb the other one who AIUI is still unwell/jetlagged :P) | 00:35 | |
| *** matsuhashi has joined #openstack-infra | 00:48 | |
| lyxus | I am trying to setup my ci infa (http://ci.openstack.org/running-your-own.html) It does say to follow http://ci.openstack.org/puppet.html#id2 but use my own repo. In the puppet slave it says to use the ci-puppetmaster and review. In this case, shouldn't I use my own ? | 00:49 |
| lifeless | yes, you should | 00:52 |
| lyxus | It's my first intro with puppet, then i am not sure what is wrong "server=openstack-pmaster.mydomain.com certname=openstack-pmaster.mydomain.com" | 00:55 |
| lyxus | I got on the slave "err: Could not request certificate: The certificate retrieved from the master does not match the agent's private key." | 00:56 |
| lyxus | I did not add anything to the ssl_client_header | 00:56 |
| clarkb | lifeless: if the jobs queue like that it means there are no available slaves to run the job. You can check jenkins0X for available slaves and or look at the tenant in the tripleo cloud | 00:58 |
| *** senk has joined #openstack-infra | 00:58 | |
| clarkb | lyxus: puppet uses client certs to identify puppet agents to the puppet master | 00:58 |
| clarkb | lyxus: I believe you want certname on the agent to reflect the agents name and not the masters name | 00:58 |
| *** nosnos has joined #openstack-infra | 00:58 | |
| *** senk has quit IRC | 01:00 | |
| *** senk has joined #openstack-infra | 01:00 | |
| *** ianw has quit IRC | 01:00 | |
| lyxus | clarkb, are the agent name are ci-puppetmaster, review ? how can i identify their name ? | 01:01 |
| clarkb | lyxus: it is arbitrary | 01:02 |
| clarkb | lyxus: for most of our hosts the certname is the same as the hostname | 01:02 |
| clarkb | and is typically what people do | 01:02 |
| *** hashar has left #openstack-infra | 01:03 | |
| lyxus | clarkb, this is what i tried to do here. | 01:04 |
| *** lyxus has left #openstack-infra | 01:04 | |
| *** lyxus has joined #openstack-infra | 01:04 | |
| bknudson | where does the keystone-coverage job output wind up? | 01:04 |
| clarkb | bknudson: http://logs.openstack.org/first_two_chars_of_sha1/sha1/ | 01:04 |
| bknudson | e.g. https://jenkins05.openstack.org/job/keystone-coverage/5/ | 01:04 |
| *** UtahDave has joined #openstack-infra | 01:05 | |
| *** nati_ueno has joined #openstack-infra | 01:06 | |
| lyxus | clarkb, do you need I need 3 server (1 agent for the review, 1 for the cert, 1 for the master) | 01:09 |
| lifeless | clarkb: there is a running slave in th ci-cloud | 01:09 |
| lifeless | clarkb: will any of the jenkins answer ? | 01:09 |
| lifeless | clarkb: or does it have to be a specific one ? | 01:09 |
| clarkb | lifeless: you want to check if the slave is attached to any of them | 01:09 |
| clarkb | lyxus: cert and master can be separate but we run them together | 01:10 |
| lyxus | clarkb, Can i run all of them together ? | 01:10 |
| bknudson | clarkb: that must be the sha1 of the merge commit and not the original commit. | 01:11 |
| clarkb | lyxus: sure, but the agent does the work so you need to run it wherever you want puppet to mange things | 01:11 |
| clarkb | it can manage the master and you can run all of them together | 01:11 |
| lifeless | clarkb: neutron-dhcp-agent had lost the plot again | 01:11 |
| clarkb | but you also want to run agents on individual hosts | 01:11 |
| lifeless | clarkb: there's a stuck fedora template, could you delete that so nodepool gets back on track ? | 01:12 |
| *** jcooley_ has joined #openstack-infra | 01:12 | |
| clarkb | lifeless: not right now, I am not on a machine with keys | 01:12 |
| lifeless | ack | 01:12 |
| clarkb | but a stuck fedora node shouldn't affect the other node types | 01:12 |
| lifeless | I know, but it offends me :P | 01:13 |
| clarkb | also if you nuke it on your end I would expect nodepool and jenkins to catch on | 01:13 |
| lifeless | clarkb: no, nodepool record will be BUILDING still I am fairly sure | 01:13 |
| lifeless | anyhow | 01:13 |
| lifeless | as you say | 01:13 |
| lifeless | it can wait | 01:13 |
| lyxus | clarkb, do you know where I can find the doc to install the cert ? I used this (http://ci.openstack.org/puppet.html#id2) | 01:16 |
| clarkb | lyxus: http://docs.puppetlabs.com/references/3.4.0/man/cert.html | 01:16 |
| lyxus | clarkb, thanks | 01:17 |
| *** gokrokve has joined #openstack-infra | 01:19 | |
| lifeless | right, its building now | 01:22 |
| lifeless | erm no | 01:22 |
| lifeless | my mistake | 01:22 |
| *** nati_ueno has quit IRC | 01:23 | |
| *** gokrokve has quit IRC | 01:23 | |
| lifeless | clarkb: where in the jenkins ui do you look for ttached slaved ? | 01:24 |
| *** markwash has joined #openstack-infra | 01:24 | |
| clarkb | lifeless: main page along the left side lists all of the attached slaves | 01:24 |
| lifeless | found it - /computer | 01:25 |
| lifeless | and one finally attached, yay. | 01:27 |
| lifeless | jenkins02 seems a little overloaded... | 01:29 |
| *** masayukig has joined #openstack-infra | 01:29 | |
| *** markwash has quit IRC | 01:31 | |
| *** markwash has joined #openstack-infra | 01:34 | |
| *** praneshp has quit IRC | 01:35 | |
| *** ok_delta has quit IRC | 01:36 | |
| clarkb | lifeless: it and 01 both have static slaves attached to them | 01:37 |
| clarkb | the remaining masters do not, so there is a bit of an imbalance | 01:37 |
| *** fallenpegasus has joined #openstack-infra | 01:39 | |
| *** ok_delta has joined #openstack-infra | 01:40 | |
| lifeless | \o/ we have progress | 01:41 |
| lifeless | clarkb: https://jenkins02.openstack.org/job/gate-tripleo-deploy/15/console | 01:41 |
| lifeless | clarkb: is there a jenkins job timeout on these things? | 01:41 |
| lifeless | clarkb: we might need to make it larger if so | 01:42 |
| clarkb | lifeless: the timeout is 2 hours | 01:43 |
| clarkb | http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/tripleo.yaml#n16 | 01:44 |
| lifeless | cool | 01:44 |
| clarkb | lifeless: do you expect 2 hours to be plenty? | 01:49 |
| lifeless | clarkb: depends; we don't have a warmed up squid cache or pip cache on the nodes | 01:50 |
| lifeless | so I expect building to be slow | 01:50 |
| lifeless | we'll see about adding an ubuntu mirror and pip mirror into the deployed test infrastructure soon | 01:50 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/devstack-gate: Enable pidstat by default https://review.openstack.org/69253 | 01:53 |
| *** nosnos_ has joined #openstack-infra | 01:53 | |
| *** vladan has quit IRC | 01:53 | |
| *** nosnos has quit IRC | 01:54 | |
| *** vladan has joined #openstack-infra | 01:57 | |
| *** gokrokve has joined #openstack-infra | 01:59 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/devstack-gate: Set concurrency to num CPUS minus 2 https://review.openstack.org/69256 | 02:00 |
| *** hashar has joined #openstack-infra | 02:00 | |
| *** fallenpegasus has quit IRC | 02:01 | |
| *** fallenpegasus has joined #openstack-infra | 02:03 | |
| *** gokrokve has quit IRC | 02:04 | |
| *** senk has quit IRC | 02:04 | |
| *** gokrokve has joined #openstack-infra | 02:06 | |
| *** gokrokve_ has joined #openstack-infra | 02:08 | |
| *** gokrokve has quit IRC | 02:11 | |
| *** gokrokv__ has joined #openstack-infra | 02:11 | |
| *** gokrokve_ has quit IRC | 02:13 | |
| *** starmer has joined #openstack-infra | 02:13 | |
| stevebaker | hai! | 02:16 |
| *** gokrokv__ has quit IRC | 02:16 | |
| stevebaker | I'm getting closer to the reason for heat-slow tests failing in gate | 02:17 |
| stevebaker | In devstack-gate localrc, SERVICE_HOST is set to 127.0.0.1 | 02:17 |
| stevebaker | and heat uses SERVICE_HOST in heat.conf heat_metadata_server_url, which doesn't work obviously | 02:19 |
| clarkb | why wouldn't that work? everything is on localhost right? | 02:24 |
| lifeless | clarkb: VM's talking to heat. ... ? | 02:24 |
| lifeless | clarkb: the VMs will be trying to talk to themselves | 02:24 |
| clarkb | the VMs talk to heat? | 02:25 |
| * clarkb imagined it would bte the other way around, but I suppose that is easier on firewalls | 02:25 | |
| *** matsuhashi has quit IRC | 02:26 | |
| lifeless | clarkb: its a hard requirement | 02:26 |
| *** AlexF_ has joined #openstack-infra | 02:26 | |
| stevebaker | clarkb: yes, for waitconditions the VMs signal heat that they are "done", and get data out for other orchestration tasks | 02:26 |
| lifeless | clarkb: unless you make every neutron network have a heat IP on it | 02:26 |
| openstackgerrit | Masayuki Igawa proposed a change to openstack-infra/elastic-recheck: Add one more fingerprint for bug 1097592 https://review.openstack.org/69259 | 02:26 |
| stevebaker | I'm going to try HOST_IP | 02:26 |
| clarkb | you will probably run into iptables problems if talking to the external address | 02:27 |
| clarkb | may need to relax the rules to allow local traffic to the external address | 02:27 |
| portante | sdague, clarkb, fungi, others: nice job on a virtually empty gate | 02:28 |
| lifeless | stevebaker: if only heat used the nova metadata API | 02:28 |
| stevebaker | lets suck it and see | 02:28 |
| stevebaker | lifeless: yeah, but it would be nice if the requests proxied all the way to heat | 02:30 |
| lifeless | stevebaker: a small matter of code :P | 02:30 |
| stevebaker | lifeless: would you see nova-compute or the neutron-metadata-proxy doing that? | 02:31 |
| lifeless | compute | 02:31 |
| lifeless | not everyone uses the metadata proxy | 02:31 |
| lifeless | e.g. baremetal | 02:31 |
| stevebaker | right | 02:31 |
| *** markmcclain has joined #openstack-infra | 02:31 | |
| lifeless | arguably it should be a separate process that asks nova-api/neutron-api/keystone-api etc as appropriate | 02:32 |
| *** fallenpegasus has quit IRC | 02:32 | |
| *** matsuhashi has joined #openstack-infra | 02:32 | |
| stevebaker | surely if nova proxies to heat, then everyone will want nova to proxy to them too. So we'll need a general mechanism | 02:32 |
| stevebaker | https://review.openstack.org/#/c/69261/ | 02:33 |
| *** AlexF_ has quit IRC | 02:35 | |
| *** praneshp has joined #openstack-infra | 02:42 | |
| *** dims has quit IRC | 02:50 | |
| *** dkranz has quit IRC | 02:51 | |
| *** sdake has joined #openstack-infra | 02:53 | |
| *** sdake has quit IRC | 02:53 | |
| *** sdake has joined #openstack-infra | 02:53 | |
| openstackgerrit | lifeless proposed a change to openstack/requirements: Mirror gear - it's needed by tripleo-ci. https://review.openstack.org/69264 | 02:53 |
| lifeless | clarkb: / mordred: any chance of an APRV on that ^ | 02:54 |
| *** emagana has joined #openstack-infra | 02:59 | |
| *** thuc has joined #openstack-infra | 02:59 | |
| *** thuc_ has joined #openstack-infra | 03:11 | |
| *** thuc has quit IRC | 03:14 | |
| *** mriedem has quit IRC | 03:15 | |
| *** matsuhashi has quit IRC | 03:18 | |
| clarkb | lifeless: it has my +2. btw what is that secure sms app you and jeblair use? | 03:25 |
| lifeless | textsecure | 03:26 |
| lifeless | same folk that make redphone | 03:26 |
| clarkb | thanks | 03:26 |
| openstackgerrit | Masayuki Igawa proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1101147 https://review.openstack.org/69268 | 03:26 |
| *** AlexF_ has joined #openstack-infra | 03:30 | |
| *** AlexF_ has quit IRC | 03:31 | |
| stevebaker | clarkb: this seems to be stuck https://jenkins07.openstack.org/job/gate-tempest-dsvm-neutron-heat-slow/1/console | 03:38 |
| *** hashar has quit IRC | 03:38 | |
| stevebaker | ...aaaand its unstuck | 03:38 |
| lifeless | clarkb: any guesses here - | 03:38 |
| lifeless | http://logs.openstack.org/67/69267/2/check/gate-tripleo-deploy/9664531/console.html ? | 03:38 |
| lifeless | 2014-01-27 03:34:45.897 | 2014-01-27 03:34:40,828 - testenv-client - INFO - Running command "./toci_devtest.sh" | 03:39 |
| lifeless | 2014-01-27 03:34:45.899 | You are not currently on a branch, so I cannot use any | 03:39 |
| *** gokrokve has joined #openstack-infra | 03:39 | |
| clarkb | lifeless: is there any text after 'any'? not sure what any it can't use | 03:39 |
| lifeless | clarkb: yes,didn't want to spam the channel | 03:43 |
| lifeless | clarkb: click on th url | 03:43 |
| *** gokrokve has quit IRC | 03:44 | |
| lifeless | clarkb: oh, interesting - it's reusing the node | 03:45 |
| lifeless | clarkb: which we can work with, I just didn't expect that | 03:45 |
| clarkb | lifeless: so, it looks liek the repo ends up on a detached head which is normally fine, but toci is trying to merge another branch in which git can't do | 03:45 |
| lifeless | AFAICT - https://jenkins02.openstack.org/computer/tripleo-precise-tripleo-test-cloud-1212713/builds | 03:45 |
| *** praneshp has quit IRC | 03:49 | |
| *** hub_cap has quit IRC | 03:51 | |
| *** david-lyle has joined #openstack-infra | 03:51 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/devstack-gate: Pretty up the post-run df call https://review.openstack.org/69157 | 03:51 |
| openstackgerrit | lifeless proposed a change to openstack-infra/devstack-gate: Capture disk space available at the start of a run https://review.openstack.org/69156 | 03:51 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Run gate-triple-deploy on tripleo-ci changes. https://review.openstack.org/69272 | 03:58 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Run gate-tripleo-deploy on devstack-gate changes. https://review.openstack.org/69273 | 03:58 |
| *** ok_delta has quit IRC | 03:59 | |
| *** UtahDave has quit IRC | 03:59 | |
| *** senk has joined #openstack-infra | 04:03 | |
| *** markmcclain has quit IRC | 04:05 | |
| stevebaker | clarkb: switching to HOST_IP didn't help. Is there somewhere I can poke at iptables rules? | 04:07 |
| *** chandankumar_ has joined #openstack-infra | 04:10 | |
| clarkb | stevebaker: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/template.pp#n17 is setting them up in the image | 04:10 |
| openstackgerrit | Masayuki Igawa proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1263417 https://review.openstack.org/69275 | 04:11 |
| clarkb | stevebaker: via http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/slave_template.pp | 04:11 |
| clarkb | you can probably edit them there or have d-g or devstack do it | 04:11 |
| stevebaker | ok | 04:14 |
| lifeless | clarkb: ok so how do we start archiving the seed.qcow2 when a run is successful ? | 04:17 |
| lifeless | clarkb: such that we can use it in the next run ? | 04:17 |
| lifeless | clarkb: or do we build it into the image cache and accept 24 hour latency on updates? | 04:17 |
| lifeless | righto, thats much more happily running | 04:18 |
| lifeless | it's odd that we had any success at all | 04:19 |
| lifeless | since that directory should have existed the whole time | 04:19 |
| clarkb | we need to copy it off somewhere | 04:19 |
| lifeless | hmm, I should EOD for a while at least | 04:20 |
| lifeless | so I don't become a dull boy | 04:20 |
| clarkb | do you want to update the image only when tests are successful and code merges? | 04:20 |
| clarkb | if they don't need to be published publicly (at least not right away), we could write it into a dir via AFS | 04:21 |
| clarkb | mordred: jeblair ^ was that the appropriate answer? I think I am learning | 04:21 |
| lifeless | clarkb: so the idea is that rather than running three or four tests A B C D | 04:23 |
| lifeless | we run A | 04:23 |
| lifeless | and we run B with the prior output of A | 04:23 |
| lifeless | and we run C with the prior output of B | 04:23 |
| lifeless | and so on | 04:23 |
| lifeless | the interfaces between these things are stable | 04:23 |
| lifeless | since A is a cloud | 04:23 |
| lifeless | and B is a cloud | 04:23 |
| lifeless | I'm not sure how best to do this | 04:23 |
| lifeless | one way is to skip most but not all of A setup - the image build | 04:23 |
| lifeless | another is to have dedicated workers for A/B/C/D etc and then we could just build new versions on the TE hosts in cron | 04:24 |
| *** amotoki has quit IRC | 04:24 | |
| lifeless | off of trunk | 04:24 |
| clarkb | I think baking it into the daily cached image is probably simplest, similar to how we bake in cirros, but cirros isn't expected to do much | 04:24 |
| *** chandankumar_ has quit IRC | 04:24 | |
| *** amotoki has joined #openstack-infra | 04:25 | |
| *** chandankumar_ has joined #openstack-infra | 04:29 | |
| openstackgerrit | Steve Baker proposed a change to openstack-infra/config: Open port 8000 for heat-api-cfn https://review.openstack.org/69276 | 04:30 |
| lifeless | hmm | 04:30 |
| lifeless | 9 207M 9 19.1M 0 0 4069k 0 0:00:52 0:00:04 0:00:48 4302kFATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel | 04:30 |
| lifeless | 2014-01-27 04:24:02.939 | hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel | 04:31 |
| lifeless | 2014-01-27 04:24:02.939 | at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41) | 04:31 |
| lifeless | 2014-01-27 04:24:02.939 | at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34) | 04:31 |
| lifeless | 2014-01-27 04:24:02.940 | at hudson.remoting.Request.call(Request.java:174) | 04:31 |
| lifeless | 2014-01-27 04:24:02.940 | at hudson.remoting.Channel.call(Channel.java:722) | 04:31 |
| *** fallenpegasus has joined #openstack-infra | 04:31 | |
| lifeless | cloud endpoint offline, debugging | 04:31 |
| *** fallenpegasus has quit IRC | 04:34 | |
| lifeless | needing to ip link set eth2 down; ip link set eth2 up and it was happy | 04:35 |
| lifeless | f* mellanox | 04:36 |
| *** coolsvap has joined #openstack-infra | 04:38 | |
| *** gokrokve has joined #openstack-infra | 04:38 | |
| *** matsuhashi has joined #openstack-infra | 04:40 | |
| *** gokrokve has quit IRC | 04:43 | |
| *** senk has quit IRC | 04:45 | |
| *** thedodd has joined #openstack-infra | 04:49 | |
| *** thuc has joined #openstack-infra | 04:49 | |
| *** jcooley_ has quit IRC | 04:50 | |
| lifeless | clarkb: fungi: nodepool seems to have given up on ci-overcloud, but AFAICT it's up | 04:51 |
| lifeless | is anyone around that can give it a good thump? | 04:51 |
| *** thuc_ has quit IRC | 04:53 | |
| *** chandankumar_ has quit IRC | 04:53 | |
| *** thuc has quit IRC | 04:53 | |
| *** thedodd has quit IRC | 04:54 | |
| *** starmer has quit IRC | 04:59 | |
| *** jcooley_ has joined #openstack-infra | 05:11 | |
| *** jcooley_ has quit IRC | 05:19 | |
| *** jcooley_ has joined #openstack-infra | 05:20 | |
| *** chandankumar_ has joined #openstack-infra | 05:23 | |
| *** jcooley_ has quit IRC | 05:24 | |
| *** markwash has quit IRC | 05:26 | |
| *** markwash has joined #openstack-infra | 05:30 | |
| *** nicedice_ has quit IRC | 05:35 | |
| mordred | clarkb: yes. the appropriate answer is always AFS. well done | 05:49 |
| mordred | clarkb, lifeless: we don't have a great systemic answer to "pass this artifact produced by job a to exist as an input for job b" if the job a artifact is not intended for public publication | 05:51 |
| *** chandankumar_ has quit IRC | 05:52 | |
| *** talluri has joined #openstack-infra | 05:59 | |
| *** jcooley_ has joined #openstack-infra | 06:01 | |
| *** DinaBelova_ is now known as DinaBelova | 06:01 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 06:04 | |
| *** gokrokve has joined #openstack-infra | 06:08 | |
| *** gokrokve has quit IRC | 06:13 | |
| *** jcooley_ has quit IRC | 06:14 | |
| *** markwash has quit IRC | 06:23 | |
| lifeless | mordred: it being public would be fine | 06:24 |
| openstackgerrit | Kei YAMAZAKI proposed a change to openstack-infra/jenkins-job-builder: Added support for Builds chain fingerprinter https://review.openstack.org/69284 | 06:25 |
| *** AaronGr is now known as AaronGr_Zzz | 06:29 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 06:31 | |
| *** DinaBelova is now known as DinaBelova_ | 06:32 | |
| *** miqui has quit IRC | 06:33 | |
| *** VijayT has joined #openstack-infra | 06:33 | |
| *** miqui has joined #openstack-infra | 06:33 | |
| *** miqui has quit IRC | 06:33 | |
| *** boris-42 has quit IRC | 06:34 | |
| *** chandankumar_ has joined #openstack-infra | 06:38 | |
| *** chandankumar_ has quit IRC | 06:46 | |
| openstackgerrit | Guido Günther proposed a change to openstack-infra/jenkins-job-builder: Support site monitor publisher https://review.openstack.org/69290 | 06:54 |
| openstackgerrit | Kei YAMAZAKI proposed a change to openstack-infra/jenkins-job-builder: Added support for Builds chain fingerprinter https://review.openstack.org/69284 | 06:56 |
| *** DinaBelova_ is now known as DinaBelova | 06:57 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 06:57 | |
| openstackgerrit | Kei YAMAZAKI proposed a change to openstack-infra/jenkins-job-builder: Added support for Builds chain fingerprinter https://review.openstack.org/69284 | 07:00 |
| *** nati_ueno has joined #openstack-infra | 07:06 | |
| *** gokrokve has joined #openstack-infra | 07:08 | |
| *** yolanda_ has joined #openstack-infra | 07:10 | |
| *** gokrokve has quit IRC | 07:13 | |
| *** nati_ueno has quit IRC | 07:13 | |
| *** yolanda_ has quit IRC | 07:15 | |
| *** starmer has joined #openstack-infra | 07:17 | |
| *** jcoufal has joined #openstack-infra | 07:18 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 07:20 | |
| openstackgerrit | Guido Günther proposed a change to openstack-infra/jenkins-job-builder: Support site monitor publisher https://review.openstack.org/69290 | 07:21 |
| *** yamahata has joined #openstack-infra | 07:24 | |
| *** yolanda_ has joined #openstack-infra | 07:30 | |
| *** DinaBelova is now known as DinaBelova_ | 07:30 | |
| *** afazekas has joined #openstack-infra | 07:32 | |
| *** AlexF_ has joined #openstack-infra | 07:36 | |
| *** flaper87|afk is now known as flaper87 | 07:37 | |
| *** lttrl has joined #openstack-infra | 07:38 | |
| *** coolsvap has quit IRC | 07:39 | |
| *** ociuhandu has joined #openstack-infra | 07:44 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 07:48 | |
| *** DinaBelova_ is now known as DinaBelova | 07:49 | |
| *** VijayT has quit IRC | 07:55 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 08:03 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 08:03 | |
| *** pblaho has joined #openstack-infra | 08:04 | |
| *** gokrokve has joined #openstack-infra | 08:08 | |
| *** gokrokve has quit IRC | 08:12 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 08:17 | |
| *** luqas has joined #openstack-infra | 08:17 | |
| *** markmc has joined #openstack-infra | 08:18 | |
| *** coolsvap has joined #openstack-infra | 08:21 | |
| *** ociuhandu has quit IRC | 08:25 | |
| *** AlexF_ has quit IRC | 08:30 | |
| *** DinaBelova is now known as DinaBelova_ | 08:34 | |
| *** boris-42 has joined #openstack-infra | 08:36 | |
| *** matsuhashi has quit IRC | 08:36 | |
| *** matsuhashi has joined #openstack-infra | 08:37 | |
| *** che-arne has quit IRC | 08:40 | |
| *** roeyc has joined #openstack-infra | 08:47 | |
| *** ociuhandu has joined #openstack-infra | 08:52 | |
| *** jpich has joined #openstack-infra | 08:59 | |
| *** bookwar has joined #openstack-infra | 09:01 | |
| *** lttrl has quit IRC | 09:05 | |
| *** dizquierdo has joined #openstack-infra | 09:05 | |
| *** yassine has joined #openstack-infra | 09:06 | |
| *** gokrokve has joined #openstack-infra | 09:08 | |
| *** jamielennox|away has quit IRC | 09:09 | |
| *** dizquierdo has quit IRC | 09:09 | |
| *** jamielennox|away has joined #openstack-infra | 09:12 | |
| *** gokrokve has quit IRC | 09:13 | |
| *** matrohon has joined #openstack-infra | 09:15 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Load projects from yaml file https://review.openstack.org/66280 | 09:18 |
| *** derekh has joined #openstack-infra | 09:18 | |
| *** rossella_s has joined #openstack-infra | 09:19 | |
| *** dizquierdo has joined #openstack-infra | 09:22 | |
| *** fbo_away is now known as fbo | 09:25 | |
| *** _ruhe is now known as ruhe | 09:35 | |
| *** dizquierdo has quit IRC | 09:36 | |
| *** starmer has quit IRC | 09:38 | |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/nodepool: Support nodes with launch condition https://review.openstack.org/65261 | 09:43 |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/nodepool: Support install phase with nodepool https://review.openstack.org/61463 | 09:43 |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/nodepool: Support nodes with launch condition https://review.openstack.org/65261 | 09:46 |
| *** afazekas has quit IRC | 09:52 | |
| *** luqas has quit IRC | 09:52 | |
| *** flaper87 is now known as flaper87|afk | 09:55 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Use the openstack pypi mirror for tripleo images. https://review.openstack.org/69309 | 09:56 |
| *** jp_at_hp has joined #openstack-infra | 09:56 | |
| *** gokrokve has joined #openstack-infra | 10:08 | |
| *** gokrokve has quit IRC | 10:13 | |
| *** chandankumar_ has joined #openstack-infra | 10:13 | |
| *** che-arne has joined #openstack-infra | 10:14 | |
| *** afazekas has joined #openstack-infra | 10:15 | |
| *** max_lobur_afk is now known as max_lobur | 10:18 | |
| *** wenlock has quit IRC | 10:21 | |
| openstackgerrit | Peter Liljenberg proposed a change to openstack-infra/jenkins-job-builder: Added support for Jenkins Campfire plugin https://review.openstack.org/69315 | 10:30 |
| *** luqas has joined #openstack-infra | 10:34 | |
| *** chandankumar_ has quit IRC | 10:37 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 10:47 | |
| *** ArxCruz has joined #openstack-infra | 10:48 | |
| *** alexpilotti has joined #openstack-infra | 10:58 | |
| *** chandankumar_ has joined #openstack-infra | 11:02 | |
| *** alexpilotti has quit IRC | 11:03 | |
| *** coolsvap_away has joined #openstack-infra | 11:05 | |
| *** coolsvap has quit IRC | 11:06 | |
| *** gokrokve has joined #openstack-infra | 11:08 | |
| *** chandankumar_ has quit IRC | 11:08 | |
| *** roeyc has quit IRC | 11:11 | |
| *** ruhe is now known as _ruhe | 11:12 | |
| *** gokrokve has quit IRC | 11:13 | |
| *** chandankumar_ has joined #openstack-infra | 11:15 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 11:17 | |
| *** flaper87|afk is now known as flaper87 | 11:18 | |
| *** alexpilotti has joined #openstack-infra | 11:25 | |
| *** rfolco has joined #openstack-infra | 11:27 | |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/nodepool: Support nodes with launch condition https://review.openstack.org/65261 | 11:29 |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/nodepool: Support install phase with nodepool https://review.openstack.org/61463 | 11:29 |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 11:33 | |
| *** coolsvap_away is now known as coolsvap | 11:36 | |
| *** DinaBelova_ is now known as DinaBelova | 11:39 | |
| *** chandankumar_ has quit IRC | 11:39 | |
| *** _ruhe is now known as ruhe | 11:39 | |
| *** matsuhashi has quit IRC | 11:40 | |
| *** emagana has quit IRC | 11:41 | |
| *** matsuhashi has joined #openstack-infra | 11:45 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/nodepool: Switch node id and ip in debug output https://review.openstack.org/69334 | 11:47 |
| *** malini_afk is now known as malini | 11:47 | |
| *** gokrokve has joined #openstack-infra | 11:59 | |
| *** Ryan_Lane has quit IRC | 12:01 | |
| *** malini has left #openstack-infra | 12:01 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Load projects from yaml file https://review.openstack.org/66280 | 12:01 |
| *** Ryan_Lane has joined #openstack-infra | 12:03 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: remove graphite graphs from these pages https://review.openstack.org/69337 | 12:03 |
| *** chandankumar_ has joined #openstack-infra | 12:05 | |
| *** chandankumar_ has quit IRC | 12:08 | |
| *** kashyap has joined #openstack-infra | 12:11 | |
| *** matsuhashi has quit IRC | 12:12 | |
| *** amotoki has quit IRC | 12:12 | |
| *** matsuhashi has joined #openstack-infra | 12:13 | |
| *** DinaBelova is now known as DinaBelova_ | 12:21 | |
| *** markwash has joined #openstack-infra | 12:21 | |
| *** dims has joined #openstack-infra | 12:23 | |
| openstackgerrit | Flavio Percoco proposed a change to openstack-infra/devstack-gate: Archive config files along with logs https://review.openstack.org/69344 | 12:25 |
| *** jasondotstar has quit IRC | 12:26 | |
| *** mkerrin has joined #openstack-infra | 12:33 | |
| *** smarcet has joined #openstack-infra | 12:35 | |
| *** coolsvap has quit IRC | 12:36 | |
| *** gsamfira has joined #openstack-infra | 12:36 | |
| *** masayukig has quit IRC | 12:37 | |
| *** luqas has quit IRC | 12:40 | |
| *** emagana has joined #openstack-infra | 12:41 | |
| *** CaptTofu has joined #openstack-infra | 12:43 | |
| *** ewindisch has quit IRC | 12:46 | |
| *** salv-orlando has joined #openstack-infra | 12:48 | |
| *** talluri has quit IRC | 12:49 | |
| *** emagana has quit IRC | 12:50 | |
| *** ruhe is now known as _ruhe | 12:52 | |
| *** heyongli has joined #openstack-infra | 12:52 | |
| *** chandankumar_ has joined #openstack-infra | 12:54 | |
| *** _ruhe is now known as ruhe | 12:54 | |
| ttx | SergeyLukjanov_: any idea what the errors at https://review.openstack.org/#/c/68471/ actually mean ? | 12:56 |
| ttx | fungi, sdague: the gate queue looks a bit frozen | 12:57 |
| sdague | yep, something's not right with the way nodes are allocating | 12:57 |
| sdague | but someone with actual access to look into it needs to be one | 12:58 |
| sdague | on | 12:58 |
| ttx | relatively recent | 12:58 |
| * ttx runs an errand | 12:58 | |
| salv-orlando | good morning/afternoon/evening. How's gate bug squashing going so far? | 12:59 |
| salv-orlando | I see just 21 people in #openstack-gate so I guess it's just starting? | 12:59 |
| *** Steap has quit IRC | 12:59 | |
| sdague | salv-orlando: honestly, I'm still waking up | 13:00 |
| *** Steap has joined #openstack-infra | 13:00 | |
| salv-orlando | sdague: makes sense. Usually at 8AM I'm not even at the keyboard. | 13:01 |
| sdague | heh | 13:01 |
| *** david-lyle has quit IRC | 13:05 | |
| *** max_lobur is now known as max_lobur_afk | 13:07 | |
| sdague | fungi: as soon as you are at your keyboard, something is seriously not right with nodepool/zuul | 13:10 |
| *** CaptTofu has quit IRC | 13:11 | |
| openstackgerrit | Julien Vey proposed a change to openstack/requirements: Add decorator to the global requirements https://review.openstack.org/69364 | 13:11 |
| *** CaptTofu has joined #openstack-infra | 13:11 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: remove graphite graphs from these pages https://review.openstack.org/69337 | 13:13 |
| *** ewindisch has joined #openstack-infra | 13:14 | |
| *** CaptTofu has quit IRC | 13:16 | |
| *** rossella_s has quit IRC | 13:16 | |
| *** vkozhukalov has joined #openstack-infra | 13:20 | |
| *** chandankumar_ has quit IRC | 13:22 | |
| *** chandankumar_ has joined #openstack-infra | 13:23 | |
| *** markwash has quit IRC | 13:24 | |
| *** DinaBelova_ is now known as DinaBelova | 13:25 | |
| *** sandywalsh has joined #openstack-infra | 13:27 | |
| *** weshay has joined #openstack-infra | 13:29 | |
| *** rossella_s has joined #openstack-infra | 13:29 | |
| *** DinaBelova is now known as DinaBelova_ | 13:31 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 13:31 | |
| *** Ajaeger has joined #openstack-infra | 13:34 | |
| *** jasondotstar has joined #openstack-infra | 13:34 | |
| *** oubiwann_ has joined #openstack-infra | 13:35 | |
| *** johnthetubaguy has joined #openstack-infra | 13:36 | |
| *** DinaBelova_ is now known as DinaBelova | 13:37 | |
| *** talluri has joined #openstack-infra | 13:37 | |
| *** luqas has joined #openstack-infra | 13:37 | |
| *** b3nt_pin is now known as beagles | 13:40 | |
| *** hashar has joined #openstack-infra | 13:41 | |
| *** dizquierdo has joined #openstack-infra | 13:43 | |
| *** chandankumar_ has quit IRC | 13:44 | |
| *** dcramer_ has quit IRC | 13:47 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: fix classification rate https://review.openstack.org/69367 | 13:47 |
| *** markwash has joined #openstack-infra | 13:48 | |
| ttx | gate looks unstuck but the graph really looks funny | 13:50 |
| sdague | it's not unstuck | 13:50 |
| sdague | it's just going to keep cycling like this, there is something fundamentally wrong | 13:50 |
| *** gokrokve has quit IRC | 13:50 | |
| ttx | sdague: at least the top of the pipe is running tests, so it's not completely stuck. | 13:51 |
| sdague | sure, but yuo go 3 deep and they still don't have d-g nodes | 13:51 |
| openstackgerrit | Julien Danjou proposed a change to openstack/requirements: Add support for Python 3 requirements https://review.openstack.org/58770 | 13:52 |
| openstackgerrit | Guido Günther proposed a change to openstack-infra/jenkins-job-builder: Support site monitor publisher https://review.openstack.org/69290 | 13:52 |
| SergeyLukjanov | ttx, the first idea was that pypi.o.o is now used for building release-tools, I'll digg into it soon | 13:53 |
| sdague | it's days like today I really wish there was someone in .eu that had access to these boxes. | 13:53 |
| SergeyLukjanov | sdague, heh, it looks like there were already a lot of such days | 13:54 |
| sdague | heh | 13:54 |
| anteaya | sdague: we are working on bringing mattoliverau up to speed | 13:54 |
| anteaya | he is in oz | 13:54 |
| anteaya | I stole him from mikal | 13:55 |
| anteaya | well he is in mikal's team | 13:55 |
| sdague | so given that everything was working prior to the nodepool changes to support tripleo, I'm going to point at that as the most probably cause | 13:55 |
| *** gokrokve has joined #openstack-infra | 13:55 | |
| ttx | not sure that would have helped in this case. Most of OZ is sleeping right now. | 13:55 |
| anteaya | but I absconded him for -infra | 13:55 |
| anteaya | doesn't help us today though | 13:55 |
| anteaya | ttx: true, but if it started several hours ago he might have caught it | 13:56 |
| anteaya | I'm open to absconding someone from eu as well | 13:56 |
| sdague | ttx: yeh, this was busted all yesterday as well | 13:56 |
| *** freyes has joined #openstack-infra | 13:56 | |
| sdague | there just wasn't any load on the system | 13:56 |
| sdague | so someone in .au would have helped | 13:56 |
| ttx | hah | 13:57 |
| *** thuc has joined #openstack-infra | 13:57 | |
| freyes | hi all | 13:58 |
| *** rcleere has quit IRC | 13:59 | |
| *** skraynev has quit IRC | 14:00 | |
| ProfFalken | if I wanted to setup a "test rig" to understand Zuul better, where would I start? Is there documentation on requirements and deploy proceedures? | 14:01 |
| ProfFalken | I want to understand it better, and in my experience deploying things myself is the best way to learn! | 14:01 |
| *** skraynev has joined #openstack-infra | 14:01 | |
| *** dkranz has joined #openstack-infra | 14:01 | |
| *** heyongli has quit IRC | 14:01 | |
| *** mrmartin has joined #openstack-infra | 14:02 | |
| BobBall | Is there a nice diagram somewhere which shows how the -infra services (I'm mostly interested in nodepool) interact with each other? | 14:03 |
| anteaya | ProfFalken: this might be a good place to begin: http://ci.openstack.org/running-your-own.html | 14:03 |
| BobBall | heh | 14:04 |
| anteaya | BobBall: nothing that i am aware of the includes nodepool | 14:04 |
| BobBall | I should have read that | 14:04 |
| anteaya | you still can | 14:04 |
| BobBall | that's probably enough for now | 14:04 |
| anteaya | great, have fun | 14:04 |
| *** dprince has joined #openstack-infra | 14:05 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: fix classification rate https://review.openstack.org/69367 | 14:06 |
| *** morganfainberg|z is now known as morganfainberg | 14:06 | |
| ProfFalken | anteaya: awesome thanks, I'll have a read | 14:08 |
| anteaya | ProfFalken: welcome, do enjoy | 14:09 |
| *** miqui has joined #openstack-infra | 14:09 | |
| sdague | ProfFalken: the other suggestion would be to stand up the puppet repo, most of the config is captured in there | 14:09 |
| *** thuc_ has joined #openstack-infra | 14:09 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/devstack-gate: Set concurrency to num CPUs minus 2 https://review.openstack.org/69256 | 14:11 |
| *** rustlebee is now known as russellb | 14:12 | |
| *** thuc has quit IRC | 14:12 | |
| *** dims has quit IRC | 14:13 | |
| *** dims has joined #openstack-infra | 14:15 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: make job creation consistent https://review.openstack.org/60633 | 14:16 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: tests: Allow to test project parameters https://review.openstack.org/67265 | 14:16 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: project_maven: Don't require artifact-id and group-id https://review.openstack.org/66036 | 14:16 |
| openstackgerrit | afazekas proposed a change to openstack-infra/devstack-gate: tempest concurrency 3 https://review.openstack.org/69370 | 14:17 |
| *** boris-42 has quit IRC | 14:23 | |
| *** chandankumar_ has joined #openstack-infra | 14:23 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/devstack-gate: Set concurrency to num CPUs / 2 https://review.openstack.org/69256 | 14:24 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/devstack-gate: Set concurrency to num CPUs / 2 https://review.openstack.org/69256 | 14:26 |
| *** jgrimm has quit IRC | 14:26 | |
| openstackgerrit | A change was merged to openstack-dev/hacking: Trigger warnings for raw and unicode docstrings https://review.openstack.org/68774 | 14:26 |
| *** chandankumar_ has quit IRC | 14:27 | |
| *** kraman has joined #openstack-infra | 14:29 | |
| *** chandankumar_ has joined #openstack-infra | 14:29 | |
| *** thuc_ has quit IRC | 14:29 | |
| *** zhiyan_ has joined #openstack-infra | 14:30 | |
| *** eharney has joined #openstack-infra | 14:30 | |
| *** thuc has joined #openstack-infra | 14:30 | |
| *** julim has joined #openstack-infra | 14:31 | |
| *** julim has quit IRC | 14:32 | |
| *** yolanda_ has quit IRC | 14:32 | |
| *** chandankumar_ has quit IRC | 14:33 | |
| *** rcleere has joined #openstack-infra | 14:34 | |
| *** rcleere has quit IRC | 14:34 | |
| *** chandankumar_ has joined #openstack-infra | 14:34 | |
| *** habdi has joined #openstack-infra | 14:34 | |
| *** jasondotstar has quit IRC | 14:35 | |
| *** thuc has quit IRC | 14:35 | |
| *** kraman has quit IRC | 14:35 | |
| *** julim has joined #openstack-infra | 14:35 | |
| *** mriedem has joined #openstack-infra | 14:37 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Remove broken url from README https://review.openstack.org/69373 | 14:38 |
| *** dstanek has joined #openstack-infra | 14:38 | |
| *** zhiyan_ has quit IRC | 14:38 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Add support for credentials-id in git repositories. https://review.openstack.org/68734 | 14:38 |
| fungi | sdague: ttx: checking it out | 14:39 |
| *** jnoller has joined #openstack-infra | 14:41 | |
| *** yolanda_ has joined #openstack-infra | 14:43 | |
| mrmartin | re | 14:43 |
| fungi | over 80% of the nodes are assigned to jenkins02... it may be the culprit | 14:44 |
| sdague | fungi: yeh, something is very wrong | 14:44 |
| fungi | webui on it is very slow | 14:44 |
| *** ICmonitor has joined #openstack-infra | 14:44 | |
| *** chandankumar_ has quit IRC | 14:44 | |
| sdague | fungi: any chance that tripleo allocations which don't work are part of the issue? | 14:45 |
| fungi | or completely nonfunctional | 14:45 |
| sdague | yeh, it eventually gets through | 14:45 |
| fungi | sdague: maybe, but fairly unlikely. i think the problem is likely jenkins02 itself | 14:45 |
| sdague | the reason I ask is that was the only bits in the config report that changed from when this was working, to not working | 14:45 |
| sdague | fungi: ok | 14:45 |
| *** max_lobur_afk is now known as max_lobur | 14:45 | |
| *** CaptTofu has joined #openstack-infra | 14:45 | |
| fungi | jenkins sometimes just decides to fall over with excessive thread count, memory utilization and so on | 14:46 |
| sdague | fungi: gotcha | 14:46 |
| sdague | no monit in place to kill it in such situations? | 14:46 |
| mrmartin | fungi: if you have 5 minutes today, may I ask you to review my patch, why it fails on Jenkins: https://review.openstack.org/#/c/68912/ I see a nice stacktrace in the log file, but don't know the exact reason. | 14:46 |
| fungi | i'm putting jenkins02 into shutdown, then i'll delete all the nodes assigned to it, which should get activity to pick back up | 14:46 |
| sdague | fungi: cool | 14:47 |
| fungi | sdague: we'd need to research a monit equivalent for java. this is all thread count and memory consumption within the jvm | 14:47 |
| sdague | fungi: suppose you could always do an http request, and use the ability that it returns under a certain amount of time as an indicator | 14:48 |
| fungi | perhaps. also we'd need to have it try to gracefully stop jenkins in those situations (i believe its api supports that) | 14:49 |
| *** dizquierdo has quit IRC | 14:49 | |
| sdague | yeh, the service scripts do a gracefully shutdown, no? | 14:49 |
| fungi | ahh, yeah, the jenkins daemon apparently takes a --stop option, which the initscript tries | 14:51 |
| fungi | but with the long-running jobs we have, it looks like it doesn't wait nearly long enough | 14:51 |
| sdague | fungi: so I guess in those situations you'd want to retrigger those jobs on another host? | 14:52 |
| *** thuc has joined #openstack-infra | 14:52 | |
| fungi | probably, yes | 14:53 |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Update documentation https://review.openstack.org/69211 | 14:54 |
| *** prad_ has joined #openstack-infra | 14:54 | |
| fungi | however, there's another jenkins design problem which arises there... on startup it automatically onlines all offline slaves it knew about from before it was shutdown | 14:55 |
| *** che-arne has quit IRC | 14:56 | |
| *** chandankumar_ has joined #openstack-infra | 14:57 | |
| mordred | solution == finish geting rid of jenkins | 14:59 |
| *** che-arne has joined #openstack-infra | 14:59 | |
| *** sandywalsh has quit IRC | 14:59 | |
| sdague | fungi: sure | 14:59 |
| sdague | mordred: well, get on that :P | 14:59 |
| mordred | sdague: :) | 15:00 |
| sdague | I agree the future is all puppies and unicorns once all our vaporware springs into being :) | 15:01 |
| *** mgagne has quit IRC | 15:01 | |
| sdague | fungi: so how quickly should the queues pop back into functioning? | 15:02 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Unlaunchpadify projects.yaml https://review.openstack.org/62189 | 15:02 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Track direct-release projects in projects.yaml https://review.openstack.org/62190 | 15:02 |
| fungi | sdague: as quickly as these deletes get processed and replacement nodes are rebuilt and assigned to other jenkinses. maybe another 15-30 minutes we'll be back to a fairly quick pace | 15:02 |
| *** dcramer_ has joined #openstack-infra | 15:03 | |
| mordred | fungi: hey - do we have any instructions written down on making new cloud databases other than "log in to the web thing and do it by hand"? | 15:04 |
| fungi | mordred: not yet--last time it came up, troveclient was still not recommended because bugs | 15:04 |
| mordred | k | 15:05 |
| fungi | but maybe it's all better now | 15:05 |
| mordred | fungi: I'm going to create a server and a database for storyboard | 15:05 |
| fungi | excellent | 15:05 |
| *** chandankumar_ has quit IRC | 15:06 | |
| *** ativelkov has left #openstack-infra | 15:06 | |
| *** rcleere has joined #openstack-infra | 15:07 | |
| *** ryanpetrello has joined #openstack-infra | 15:09 | |
| *** lttrl has joined #openstack-infra | 15:11 | |
| *** yolanda_ has quit IRC | 15:11 | |
| *** ruhe is now known as _ruhe | 15:12 | |
| *** yolanda_ has joined #openstack-infra | 15:12 | |
| *** wenlock has joined #openstack-infra | 15:14 | |
| *** senk has joined #openstack-infra | 15:14 | |
| *** sandywalsh has joined #openstack-infra | 15:15 | |
| fungi | mrmartin: i left some comments on your 68912 change about the error you're getting | 15:15 |
| mrmartin | oh thanks | 15:16 |
| *** rwsu has joined #openstack-infra | 15:16 | |
| *** cody-somerville has joined #openstack-infra | 15:17 | |
| *** wenlock has quit IRC | 15:19 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Database fixture added https://review.openstack.org/69384 | 15:21 |
| *** thuc has quit IRC | 15:23 | |
| *** kraman has joined #openstack-infra | 15:23 | |
| *** thuc has joined #openstack-infra | 15:23 | |
| *** kraman has quit IRC | 15:24 | |
| *** sandywalsh has quit IRC | 15:25 | |
| fungi | so it looks like the actual stoppage we saw this time was nodepool launching nodes, adding them to jenkins02 as ready, for some reason jenkins02 not actually recognizing that they were registered or otherwise losing track of them, and then eventually when that count reached the minimum ready threshhold for those node types in nodepool it ceased trying to launch any more because it thought there were | 15:25 |
| fungi | enough available even though there were none | 15:25 |
| *** malini has joined #openstack-infra | 15:25 | |
| fungi | effectively the 100% of the ready count was nonexistent slaves assigned to jenkins02 | 15:25 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for bug 1273259 https://review.openstack.org/69386 | 15:26 |
| *** kraman has joined #openstack-infra | 15:26 | |
| *** boris-42 has joined #openstack-infra | 15:26 | |
| *** morganfainberg is now known as morganfainberg|z | 15:27 | |
| *** thuc has quit IRC | 15:28 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add Storyboard puppet module https://review.openstack.org/65017 | 15:28 |
| anteaya | is it just me or is Jenkins02 quirky compared to the other Jenkinses | 15:28 |
| *** habdi has quit IRC | 15:29 | |
| sdague | fungi: so is it supposed to be moving again? | 15:29 |
| fungi | anteaya: it's unique in being the only jenkins 1.543 we have on a non-performance flavor in rackspace (the other non-performance jenkins masters we have are running 1.525) | 15:29 |
| fungi | sdague: when i realized it was the ready node count which was killing us, i switched to deleting ready nodes assigned to jenkins02 while waiting for its running jobs to wind down | 15:30 |
| sdague | fungi: gotcha | 15:30 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Document the INFO+ log level query restriction https://review.openstack.org/69388 | 15:31 |
| anteaya | ah okay | 15:31 |
| fungi | though nodepool still isn't building new nodes even though the ready nodes are below the minimum threshhold now. looking to see whether i can tell why | 15:31 |
| anteaya | I wonder why that might be contributing to its unique decisions regarding nodes | 15:32 |
| fungi | anteaya: it's not making unique decisions. it's broken | 15:32 |
| *** jgrimm has joined #openstack-infra | 15:32 | |
| *** hashar has quit IRC | 15:32 | |
| anteaya | k | 15:32 |
| malini | sdague, fungi: ping | 15:32 |
| *** _ruhe is now known as ruhe | 15:33 | |
| *** markmcclain has joined #openstack-infra | 15:34 | |
| fungi | malini: hi | 15:35 |
| malini | fungi: heyy..I wanted some help with a couple of marconi patches | 15:36 |
| malini | fungi: I have two of the sitting there for a while & need them merged before I can make progress with tempest. Can I get you to review it ?https://review.openstack.org/#/c/65145/ https://review.openstack.org/#/c/65140/ | 15:36 |
| malini | ? | 15:36 |
| *** wenlock has joined #openstack-infra | 15:37 | |
| malini | fungi: they are small & as an added bonus, folks are generous with poptarts in #openstack-marconi ;) | 15:37 |
| fungi | malini: possibly later. i'm trying to fix broken infrastructure right now, which has an unfortunate tendency to leave me with very little time for code review | 15:37 |
| *** sandywalsh has joined #openstack-infra | 15:37 | |
| malini | fungi: oops :( | 15:38 |
| malini | fungi: no worries.. I am trying to get a few core folks to review it | 15:38 |
| fungi | no worries. i'll make a note to check those out later today, time permitting | 15:38 |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 15:38 | |
| *** markmcclain has quit IRC | 15:38 | |
| malini | thanks fungi!! | 15:38 |
| *** johnthetubaguy1 has joined #openstack-infra | 15:38 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add Storyboard puppet module https://review.openstack.org/65017 | 15:38 |
| mordred | ruhe: ^^ if you want to look | 15:39 |
| *** freyes has quit IRC | 15:39 | |
| *** coolsvap has joined #openstack-infra | 15:40 | |
| *** johnthetubaguy has quit IRC | 15:41 | |
| *** david-lyle has joined #openstack-infra | 15:42 | |
| fungi | this apparently happened back on friday for no particular reason... | 15:42 |
| ArxCruz | jeblair, clarkb : hey, which scp-plugin jenkins is using? | 15:42 |
| fungi | [Fri Jan 24 22:50:12 2014] Out of memory: Kill process 2524 (nodepoold) score 239 or sacrifice child | 15:42 |
| fungi | [Fri Jan 24 22:50:12 2014] Killed process 2524 (nodepoold) total-vm:2568048kB, anon-rss:17288kB, file-rss:476kB | 15:43 |
| mordred | ArxCruz: we have a patched version I believe | 15:43 |
| *** mgagne has joined #openstack-infra | 15:43 | |
| ArxCruz | mordred: where can I get this one ? | 15:43 |
| mordred | ArxCruz: I _think_ we're working on getting it released for real ... zaro ? | 15:44 |
| krtaylor | mordred, we are looking for the 1.9 version that jenkins.pp specifies | 15:44 |
| *** markmcclain has joined #openstack-infra | 15:44 | |
| fungi | oh, it's possible that oom error was related to me stopping nodepoold... that's about 30 minutes before the last time it was started | 15:44 |
| *** metabro has quit IRC | 15:44 | |
| *** habdi has joined #openstack-infra | 15:45 | |
| *** markwash has quit IRC | 15:45 | |
| mordred | fungi: any reason I don't want to add new servers to salt? | 15:46 |
| *** markvan has joined #openstack-infra | 15:47 | |
| fungi | mordred: not really, other than we've only been salting the jenkins slaves so far | 15:47 |
| fungi | but should be safe enough | 15:47 |
| *** mrmartin has quit IRC | 15:48 | |
| *** ewindisch is now known as zz_ewindisch | 15:48 | |
| krtaylor | ArxCruz, the only scp-plugin I can find in github is the 1.8 in jenkinsci | 15:49 |
| ArxCruz | krtaylor: :/ | 15:49 |
| ArxCruz | mordred: any chance we get a copy from your patched plugin ? | 15:50 |
| *** mgagne has quit IRC | 15:50 | |
| *** vkozhukalov has quit IRC | 15:50 | |
| mordred | ArxCruz: sure. you want me to just put it somewhere you can download it? | 15:51 |
| krtaylor | mordred, +1 | 15:52 |
| *** sdake has quit IRC | 15:52 | |
| ArxCruz | mordred: yes, please :) | 15:54 |
| ArxCruz | gdrive works for you mordred ? | 15:54 |
| krtaylor | mordred, I was chatting with mtreinish and he mentioned that there was someone that was interested in writing a swift plugin for jenkins | 15:55 |
| *** metabro has joined #openstack-infra | 15:56 | |
| *** virmitio has joined #openstack-infra | 15:57 | |
| sdague | fungi: so that's when we stopped it to bring in the tripleo nodes? | 15:58 |
| sdague | that times about for when it went weird | 15:58 |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 15:58 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/elastic-recheck: Add query for bug 1273292 https://review.openstack.org/69391 | 15:58 |
| fungi | sdague: that's when i restarted it, yes | 15:58 |
| *** eharney has quit IRC | 15:58 | |
| *** mgagne has joined #openstack-infra | 15:58 | |
| fungi | sdague: but i was watching it over the weekend and it seemed normal. not sure what you mean by "about when it went weird" | 15:59 |
| sdague | fungi: so does that mean that node pool was downed? | 15:59 |
| sdague | fungi: we've had this node starvation issue since at least yesterday morning | 15:59 |
| fungi | sdague: no, as i said, it looks like that oom error was about half an hour *before* i restarted it | 15:59 |
| sdague | I thought I saw some odd paterns on Sat | 15:59 |
| sdague | oh, interesting | 15:59 |
| *** zz_ewindisch is now known as ewindisch | 16:00 | |
| *** dhellmann is now known as dhellmann_ | 16:00 | |
| *** pblaho has quit IRC | 16:00 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 16:00 | |
| ArxCruz | mordred: thanks! when will you're planning to release it ? | 16:02 |
| fungi | anyway, right now it's throwing jenkinsexceptions when trying to add new nodes, which may be still due to jenkins02 timing out on api calls (so it can't check the status). i'm looking to see which changes these last couple of jobs running on jenkins02 map back to, so i can kill them and leave comments on the changes | 16:02 |
| *** markwash has joined #openstack-infra | 16:02 | |
| *** rnirmal has joined #openstack-infra | 16:02 | |
| *** eharney has joined #openstack-infra | 16:02 | |
| mordred | ArxCruz: I think we were wanting to make sure it worked before doing that ... jeblair or zaro might have more thoughts on that subject | 16:02 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/elastic-recheck: Add query for bug 1273292 https://review.openstack.org/69391 | 16:02 |
| jeblair | mordred: i think before releasing, we need to fix the backwards-compat issues; i don't think upgrading from current released version to master works | 16:04 |
| mordred | nod | 16:04 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/elastic-recheck: Add query for bug 1273292 https://review.openstack.org/69391 | 16:04 |
| ArxCruz | jeblair: how can I be aware of this? is there any announcement list or something similar? Or should I just keep bugging you? :P | 16:04 |
| jeblair | ArxCruz: if you're good with java, you could fix it and submit a pr :) | 16:05 |
| fungi | jeblair: fyi nodepoold is bound up because of jenkins02 deciding to get bogged down. you may be interested to look at the jenkinsexceptions being throws which seem to coincide with every new node build nodepoold tries (i haven't resorted to restarting nodepoold just yet) | 16:05 |
| ArxCruz | I'm not a java guy, but I can take a look | 16:05 |
| *** mrodden1 is now known as mrodden | 16:05 | |
| ArxCruz | jeblair: where do I get the source? github.com/jenkinsci/scp-plugin ? | 16:05 |
| fungi | s/throws/thrown/ | 16:05 |
| jeblair | ArxCruz: yes | 16:06 |
| jeblair | fungi: what's up with jenkins02? | 16:06 |
| *** habdi has quit IRC | 16:06 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/elastic-recheck: Add query for bug 1273283 https://review.openstack.org/69393 | 16:06 |
| fungi | jeblair: proxy timeouts in the webui, generally very slow to respond. i've put it in shutdown now and killed its remaining jobs | 16:06 |
| *** HenryG has quit IRC | 16:06 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1273259 https://review.openstack.org/69386 | 16:07 |
| russellb | according ot salv-orlando 's notes, it looks like the top gate bug (https://bugs.launchpad.net/nova/+bug/1254890) may be helped by a kernel upgrade | 16:07 |
| fungi | there are also some transaction errors in the nodepool.log now, but i think that's because i'm trying to nodepool delete the remaining jenkins02 nodes and it's taking a while to get a response from the jenkins02 api endpoint (often timing out) | 16:08 |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Groups community portal gating tasks https://review.openstack.org/68912 | 16:08 |
| jeblair | fungi: yeah, any time you do something external to nodepool you're likely to get such errors | 16:08 |
| * fungi nods | 16:08 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add Storyboard puppet module https://review.openstack.org/65017 | 16:08 |
| fungi | jeblair: however the jenkinsexcpetion tracebacks i think are not related to me deleting nodes. they don't say which jenkins they relate to though... i'm about to see if i can tell from the debug.log | 16:09 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Document the INFO+ log level query restriction https://review.openstack.org/69388 | 16:09 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add Storyboard puppet module https://review.openstack.org/65017 | 16:09 |
| *** ryanpetrello has quit IRC | 16:10 | |
| *** gyee has joined #openstack-infra | 16:10 | |
| *** ryanpetrello has joined #openstack-infra | 16:11 | |
| *** AaronGr_Zzz is now known as AaronGr | 16:11 | |
| *** gyee has quit IRC | 16:11 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1263417 https://review.openstack.org/69275 | 16:12 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add one more fingerprint for bug 1097592 https://review.openstack.org/69259 | 16:13 |
| *** nosnos_ has quit IRC | 16:14 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Remove broken url from README https://review.openstack.org/69373 | 16:14 |
| *** mfer has joined #openstack-infra | 16:14 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/elastic-recheck: Add query for bug 1273292 https://review.openstack.org/69391 | 16:14 |
| *** sdake has joined #openstack-infra | 16:15 | |
| *** sdake has quit IRC | 16:15 | |
| *** sdake has joined #openstack-infra | 16:15 | |
| *** matsuhashi has quit IRC | 16:15 | |
| *** julim has quit IRC | 16:16 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/elastic-recheck: Add query for bug 1273297 https://review.openstack.org/69398 | 16:16 |
| fungi | i'm thinking i may just need to offline the jenkins service on jenkins02 and then hand-delete the remaining nodepool nodes from its xml config | 16:17 |
| *** gyee has joined #openstack-infra | 16:18 | |
| jeblair | fungi: that's probably a good idea, but can you wait a sec while i try to load the melody page? | 16:18 |
| zaro | jeblair: i was not aware that there's a backwards compatability issue with scp plugin. what's the problem? | 16:18 |
| zaro | morning | 16:18 |
| fungi | jeblair: of course | 16:19 |
| jeblair | zaro: i think if you install the current released version, then upgrade to master, it doesn't carry over the config correctly | 16:19 |
| *** julim has joined #openstack-infra | 16:19 | |
| *** ICmonitor has quit IRC | 16:20 | |
| jeblair | fungi: none of the jenkins masters seem fast; and many of them are idle | 16:21 |
| *** ICmonitor has joined #openstack-infra | 16:21 | |
| *** gokrokve has quit IRC | 16:21 | |
| fungi | ugh | 16:21 |
| fungi | i'd been focused on jenkins02 because it had 100 nonexistent (from its perspective) ready nodes assigned to it | 16:22 |
| zaro | ArxCruz: i might have time to look scp upgrade bug jeblair pointed out tomorrow. | 16:22 |
| ArxCruz | zaro: that would be awesome :) | 16:22 |
| *** mfink has joined #openstack-infra | 16:22 | |
| *** habdi has joined #openstack-infra | 16:22 | |
| *** jergerber has joined #openstack-infra | 16:22 | |
| ArxCruz | I really need this scp plugin :) | 16:22 |
| *** dangers_away is now known as dangers | 16:23 | |
| *** VijayT has joined #openstack-infra | 16:24 | |
| *** pcrews has joined #openstack-infra | 16:24 | |
| zul | so i was just thinking trusty default python3 is python3.4 so I think we might have to be moving to python3.4 as well | 16:24 |
| *** thuc has joined #openstack-infra | 16:24 | |
| BobBall | I've got a nodepool question... Set it up with same region_name, username, project_id as using in my nova credentials for rackspace yet it still says "unable to authenticate user with credentials provided". Do I need to somehow change the auth type to rackspace as I had to do for novaclient? | 16:25 |
| *** thuc_ has joined #openstack-infra | 16:25 | |
| *** ruhe is now known as _ruhe | 16:25 | |
| fungi | BobBall: you need to use password, not api key | 16:25 |
| BobBall | oh | 16:25 |
| BobBall | that sucks :P | 16:25 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add Storyboard puppet module https://review.openstack.org/65017 | 16:25 |
| fungi | BobBall: in which case current releases of novaclient should work fine with rackspace | 16:26 |
| zaro | jog0: noticed you are active on review-dev. were you trying to do something with LP & gerrit? | 16:26 |
| fungi | BobBall: api key is a rackspace special snowflake authentication extension for novaclient | 16:26 |
| BobBall | Indeed - that works much better. | 16:26 |
| BobBall | ah understood | 16:26 |
| *** AlexF_ has joined #openstack-infra | 16:27 | |
| sdague | jeblair: yeh, well, anything that would get changes moving again would be nice :) | 16:27 |
| fungi | it's not part of novaclient at all these days. and it's not packaged in any reasonable access channel. you have to retrieve the source from rackspace's github repo for it, per their documentation, then add it to novaclient | 16:27 |
| sdague | check queue is about to cross 200 | 16:27 |
| fungi | sdague: yes, i see that | 16:27 |
| BobBall | indeed - I did that fungi- but clearly nodepool doesn't use it in the same way :) | 16:27 |
| BobBall | that's fine - I've got a "fix" for now. I think. | 16:28 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for bug 1273301 https://review.openstack.org/69400 | 16:28 |
| fungi | BobBall: it's entirely possible nodepool doesn't work with novaclient extensions. no idea | 16:28 |
| jeblair | sdague: sorry? | 16:28 |
| BobBall | password will work for now | 16:28 |
| sdague | jeblair: http://status.openstack.org/zuul/ | 16:28 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/storyboard: Don't try to install file that doesn't exist https://review.openstack.org/69402 | 16:28 |
| sdague | jeblair: basically, we're not allocating d-g nodes in any reasonable rate | 16:29 |
| jeblair | sdague: i see that. what were you trying to communicate when you said "yeh, well, anything that would get changes moving again would be nice :)" | 16:29 |
| *** thedodd has joined #openstack-infra | 16:29 | |
| sdague | jeblair: wasn't sure if you were poking on this one, sorry. | 16:29 |
| *** thuc has quit IRC | 16:29 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1273292 https://review.openstack.org/69391 | 16:30 |
| *** ICmonitor1 has joined #openstack-infra | 16:30 | |
| *** ewindisch is now known as zz_ewindisch | 16:30 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 16:30 | |
| *** ICmonitor has quit IRC | 16:30 | |
| jeblair | 2014-01-27 11:17:16,719 DEBUG nodepool.NodeLauncher: Adding node id: 1214068 to jenkins | 16:31 |
| jeblair | 2014-01-27 15:14:08,433 ERROR nodepool.NodeLauncher: Exception launching node id: 1214068: | 16:31 |
| jeblair | JenkinsException: create[bare-precise-hpcloud-az2-1214068] failed | 16:31 |
| jeblair | fungi: ^ | 16:31 |
| jeblair | fungi: jenkins02 says that it is currently handling that request | 16:31 |
| jeblair | fungi: note in particular the timestamps | 16:32 |
| fungi | ouch | 16:32 |
| mordred | wow | 16:32 |
| *** jooools has joined #openstack-infra | 16:32 | |
| jeblair | the currently running method is a regex match | 16:32 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1273283 https://review.openstack.org/69393 | 16:33 |
| jeblair | trying to get a full stacktrace | 16:33 |
| jeblair | fungi, mordred: i wonder if we shouldn't slow down the jenkins api call rate so that servers are added and removed more slowly | 16:34 |
| jeblair | it seems like we had some pretty extreme spikiness recently where nodepool wanted to add a full 100 servers to each jenkins | 16:35 |
| *** sandywalsh has quit IRC | 16:35 | |
| fungi | jeblair: it's possible we're triggering subtle races in node addition/deletion i suppose, which we would be less likely to hit that way | 16:36 |
| *** morganfainberg|z is now known as morganfainberg | 16:36 | |
| fungi | i suspect we're fairly unique in the volume and rate at which we add and remove jenkins slaves, compared to other sites running jenkins | 16:37 |
| ArxCruz | jeblair: fungi clarkb are you guys get these errors often in nodepool -> jenkins ? http://paste.openstack.org/show/61950/ | 16:37 |
| *** DinaBelova is now known as DinaBelova_ | 16:37 | |
| *** gokrokve has joined #openstack-infra | 16:38 | |
| fungi | ArxCruz: right this moment, in fact. seems to be happening to us because one of our jenkins masters is failing to respond to api calls in a timely fashion, but in your case it might be that nodepoold isn't able to reach the jenkins master's api endpoint | 16:38 |
| ArxCruz | fungi: if i call get_info manually from a script, I got no errors | 16:39 |
| *** rlandy is now known as rlandy|bbl | 16:40 | |
| *** DinaBelova_ is now known as DinaBelova | 16:41 | |
| *** UtahDave has joined #openstack-infra | 16:42 | |
| mordred | jeblair: ++ | 16:47 |
| *** emagana has joined #openstack-infra | 16:47 | |
| mordred | jeblair, fungi: you are working on more important things, but https://review.openstack.org/69402 is ready to land - I have the thigns in hiera, and I have tested it in the mordred environment on the new node that's been spun up | 16:47 |
| jeblair | mordred: why are you self-approving that? | 16:48 |
| mordred | jeblair, fungi: there are some operational issues with the codebase itself, but those are things that can be fixed by landing changes i thn the code - the puppet does the right things | 16:48 |
| mordred | jeblair: I'm not | 16:48 |
| mordred | jeblair: oh - meh. sorry - bad copy and paste above | 16:49 |
| jeblair | mordred: you left a comment that said "self-approving" | 16:49 |
| jeblair | mordred: along with an approval vote | 16:49 |
| jeblair | fungi: i have been unable to get a stacktrace with any of the ways i know how (two points in the webui, jstack, and jsadebugd) | 16:50 |
| mordred | jeblair: yup. sorry - what I meant was "https://review.openstack.org/65017 is ready to land" - the storyboard change I self-approved because it was the last blocker in testing the puppet | 16:50 |
| jeblair | fungi: i think we may need to give up on knowing where this regex call is coming from | 16:50 |
| jeblair | mordred: it is not required to _approve_ changes in order to _test_ them. | 16:50 |
| fungi | jeblair: though i did see some helpful utilization details dumped into the log (gc thread resource utilization and so on) | 16:50 |
| fungi | jeblair: okay, should i go ahead and stop jenkins on jenkins02 in that case? | 16:51 |
| jeblair | fungi: oh, wait just a sec then. | 16:51 |
| *** sandywalsh has joined #openstack-infra | 16:51 | |
| fungi | sure | 16:51 |
| jeblair | fungi: oh hey, a dump made it into the log! | 16:52 |
| fungi | okay, cool, then that *was* what i was seeing in there | 16:52 |
| fungi | i thought that's where you were looking (since the thread dumps usually end up in the jenkins log) | 16:52 |
| jeblair | fungi: only if successful; all the programs i ran told me they failed | 16:53 |
| fungi | ahh. clearly it lied | 16:53 |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 16:54 | |
| *** colinmcnamara has joined #openstack-infra | 16:54 | |
| fungi | yep, thar she be... "Full thread dump OpenJDK 64-Bit Server VM (23.7-b01 mixed mode)" | 16:54 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Make functional tests work again https://review.openstack.org/69154 | 16:56 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1262153 https://review.openstack.org/69242 | 16:58 |
| *** mfer has quit IRC | 16:58 | |
| jeblair | fungi: http://paste.openstack.org/show/61951/ | 16:58 |
| *** AlexF_ has quit IRC | 16:58 | |
| jeblair | fungi: that's the thread that was running during the stack trace; i just (hopefully) invoked another stack dump | 16:58 |
| fungi | to see if it's still there, presumably | 16:59 |
| *** markmc has quit IRC | 16:59 | |
| *** thedodd has quit IRC | 17:00 | |
| ttx | fungi: debunked bug 1097592 | 17:00 |
| ttx | fungi: cron.daily jobs running at 06:25 include apt/dpkg and when a job hits at the same time... dpkg calls fail. Fun | 17:00 |
| *** gokrokve has quit IRC | 17:01 | |
| fungi | ttx: as i suspected--in that case we should probably be disabling cron jobs like those in nodepool prep scripts | 17:01 |
| jeblair | fungi: something else is running now, it looks like it is progressing very slowly but is not stuck. | 17:01 |
| *** thedodd has joined #openstack-infra | 17:01 | |
| ttx | fungi: yep, you throw away the nodepool model daily anyway | 17:01 |
| jeblair | fungi: okay, i'm done; ready for you to force-stop/clean | 17:01 |
| fungi | jeblair: thanks! on it | 17:02 |
| ttx | fungi: you could even remove cron.daily/cron.weekly/cron.monthly jobs | 17:02 |
| ttx | since you throw away machines anyway, no point in running them accidentally | 17:03 |
| fungi | i'm curious to see whether nodepoold starts building new nodes once jenkins02's api endpoint is answering with a tcp/rst | 17:03 |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 17:03 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Don't try to install file that doesn't exist https://review.openstack.org/69402 | 17:03 |
| fungi | ttx: we could probably just stop cron entirely | 17:03 |
| fungi | ttx: there are several options. bears deeper discussion | 17:03 |
| *** dcramer_ has quit IRC | 17:04 | |
| *** reed has joined #openstack-infra | 17:05 | |
| *** Ryan_Lane has quit IRC | 17:06 | |
| jeblair | fungi: nodepool wants to launch 1100 nodes. | 17:07 |
| *** morganfainberg is now known as morganfainberg|z | 17:07 | |
| fungi | wow | 17:07 |
| mordred | jeblair: that seems like a lot of nodes | 17:08 |
| *** gokrokve has joined #openstack-infra | 17:08 | |
| jeblair | it won't quite reach that, but that's how many it would launch if it could. | 17:08 |
| fungi | mordred: there's a very many pending jobs which need them | 17:08 |
| jeblair | mordred: we need ~3000 nodes to service all of the changes currently in the queues. | 17:09 |
| jeblair | some of them have already been run, and some of them are outside the active window, thus the difference from 3000-1100 | 17:09 |
| *** sarob has joined #openstack-infra | 17:10 | |
| jeblair | jenkins01 seems so much more responsive than 3-7. :/ | 17:12 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for bug 1252947 https://review.openstack.org/69415 | 17:12 |
| jeblair | i wonder if that's just due to current circumstances, or the jenkins version, or the rackspace flavors... | 17:13 |
| fungi | i've manually washed the remaining nodepool nodes out of jenkins02's config.xml, but am still wrapping up deletion from the nodepool end before i start the service back up | 17:14 |
| *** sarob has quit IRC | 17:14 | |
| jeblair | fungi: k | 17:14 |
| *** starmer has joined #openstack-infra | 17:15 | |
| *** starmer has quit IRC | 17:15 | |
| *** sarob has joined #openstack-infra | 17:16 | |
| *** colinmcnamara has quit IRC | 17:16 | |
| *** sarob_ has joined #openstack-infra | 17:17 | |
| *** gsamfira has quit IRC | 17:17 | |
| *** gokrokve has quit IRC | 17:18 | |
| jeblair | mordred: did you say you tested https://review.openstack.org/#/c/65017/13 ? | 17:18 |
| jeblair | mordred: if so, can you leave a review and comment to that effect? | 17:18 |
| *** mfer has joined #openstack-infra | 17:18 | |
| *** gokrokve_ has joined #openstack-infra | 17:20 | |
| *** sarob has quit IRC | 17:20 | |
| mordred | jeblair: yes. I have - and yes, I wil do that | 17:20 |
| *** nicedice_ has joined #openstack-infra | 17:20 | |
| *** sarob_ has quit IRC | 17:21 | |
| *** HenryG has joined #openstack-infra | 17:24 | |
| *** gokrokve_ has quit IRC | 17:25 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 17:26 | |
| BobBall | for nodepool I want to add supporting base image IDs not just image_name. should I intelligently check if it's a uuid using a regexp or do you want another config to use base_image_id rather than base_image? | 17:30 |
| BobBall | RAX sometimes hides images that we need :) | 17:30 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for libvirt socket connection refused bug 1251521 https://review.openstack.org/69418 | 17:31 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add Storyboard puppet module https://review.openstack.org/65017 | 17:31 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Unlaunchpadify projects.yaml https://review.openstack.org/62189 | 17:31 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Track direct-release projects in projects.yaml https://review.openstack.org/62190 | 17:31 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Split config from projects list https://review.openstack.org/62187 | 17:31 |
| *** markmcclain has quit IRC | 17:31 | |
| *** freyes has joined #openstack-infra | 17:31 | |
| mordred | yay. the check-alphebetize script works properly and catches things | 17:32 |
| *** markmcclain has joined #openstack-infra | 17:32 | |
| *** markmcclain has quit IRC | 17:33 | |
| Ajaeger | mordred: yeah, it does! | 17:33 |
| *** markmcclain has joined #openstack-infra | 17:33 | |
| mriedem | is it just me or is the Related-Bug: # bot not awake today? | 17:34 |
| *** jcooley_ has joined #openstack-infra | 17:34 | |
| mriedem | fungi: ^? | 17:34 |
| *** dkliban is now known as dkliban_afk | 17:35 | |
| fungi | mriedem: it's a commit hook in gerrit. i'll have to dig into the logs in a bit to see whether it's failing. what change? | 17:35 |
| mriedem | fungi: a few, but here is the latest: https://review.openstack.org/#/c/69418/ | 17:36 |
| *** jpich has quit IRC | 17:36 | |
| fungi | mriedem: that bug doesn't have a bugtask for any project besides nova | 17:36 |
| *** CaptTofu has quit IRC | 17:37 | |
| *** Ryan_Lane has joined #openstack-infra | 17:37 | |
| mriedem | fungi: ? should it? | 17:37 |
| *** CaptTofu has joined #openstack-infra | 17:37 | |
| mriedem | it's an e-r query for a nova bug | 17:37 |
| fungi | mriedem: i'll have to look back over the update_bug.py script in jeepyb to be sure, but i think that it will only update a bug which has a bugtask for the project against which the change is proposed | 17:37 |
| *** thedodd has quit IRC | 17:38 | |
| mriedem | fungi: i've seen lots of related-bug links in e-r patches link back to non-e-r bugs | 17:38 |
| mriedem | so was working at some point | 17:38 |
| mriedem | fungi: take your time though | 17:38 |
| jeblair | fungi: i _think_ you are right; i believe we did that to avoid spamming random launchpad projects in case of a typo | 17:38 |
| fungi | mriedem: do you have an example? i've seen lots of bugs with an open bugtask for elastic-recheck | 17:38 |
| *** afazekas has quit IRC | 17:38 | |
| * mriedem digs | 17:39 | |
| *** max_lobur is now known as max_lobur_afk | 17:39 | |
| *** thedodd has joined #openstack-infra | 17:39 | |
| *** marun has joined #openstack-infra | 17:39 | |
| *** dkranz has quit IRC | 17:40 | |
| *** vkozhukalov has joined #openstack-infra | 17:40 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 17:41 | |
| *** marun has quit IRC | 17:41 | |
| fungi | mriedem: since the openstack-infra/elastic-recheck repo is mapped to the openstack-ci project in lp, i believe that's what you need to add a bugtask for on any which deserve an elastic-recheck commit... http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/review.projects.yaml.erb#n45 | 17:41 |
| *** CaptTofu has quit IRC | 17:42 | |
| sdague | jnoller: so I tried to start the PV/HVM 12.04 node this weekend, it didn't bring up network | 17:42 |
| *** ivar-lazzaro has quit IRC | 17:42 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/storyboard: Added documentation for REST API layer. https://review.openstack.org/69212 | 17:42 |
| fungi | as long as that bugtask is in place before the first patchset of the elastic-recheck change is uploaded to gerrit with a relevant bug header in its commit message, it ought to work | 17:42 |
| mriedem | fungi: ok, i could only find this one https://review.openstack.org/#/c/65489/ and like you said, openstack-ci is a task in that bug so i guess it reported it there | 17:43 |
| jeblair | fungi: i'm not sure every bug in openstack should have an openstack-ci bugtask | 17:43 |
| mriedem | in every other case i must have added them myself | 17:43 |
| sdague | reading scrollback... any postmortem yet on what happened? | 17:43 |
| fungi | though if we're going to be doing this a bunch, i think e-r needs a separate lp project | 17:43 |
| jeblair | sdague: undiagnosable problem in jenkins. | 17:43 |
| mriedem | sdague: what happened might just be me thinking things worked a certain way when they didn't | 17:43 |
| sdague | jeblair: bummer | 17:43 |
| *** CaptTofu has joined #openstack-infra | 17:43 | |
| mriedem | fungi: jeblair: yeah, don't want to add openstack-ci or elastic-recheck to every patch that gets an e-r query | 17:44 |
| fungi | jeblair: completely agree on unnecessary openstack-ci bugtasks for elastic-recheck queries | 17:44 |
| jeblair | sdague: it shouldn't have impacted nodepool that much though; if it happens again, we should put more debug time into that. | 17:44 |
| mriedem | but it's nice to know when a nova bug, for example, has an e-r query already created for it | 17:44 |
| mriedem | but...the elastic-recheck status page should show that too, so nevermind | 17:44 |
| jeblair | perhaps we should see about making jeepyb handle that case | 17:44 |
| fungi | jeblair: sdague: right. as soon as i stopped the jenkins service on jenkins02, nodepool *immediately* began launching new nodes and assigning them to the other jenkins masters | 17:45 |
| mriedem | meh, i think half the time the patches don't have Related-Bug tag in the first patch set (or any for that matter) | 17:45 |
| mriedem | jeblair: ^ | 17:45 |
| jog0 | zaro: was running some e-r tests | 17:45 |
| *** Ryan_Lane has quit IRC | 17:46 | |
| *** gokrokve has joined #openstack-infra | 17:47 | |
| sdague | fungi: so on the bug posting, I'm still under the opinion that we shouldn't do strict matching, and just have a whitelist for openstack projects and trackers | 17:47 |
| fungi | sdague: sounds good to me, though the bugtasks are sort of necessary if you expect the hook to do the right thing when it comes to assigning and changing status | 17:49 |
| jeblair | sdague: yeah, i think a rule that says any project listed in projects.yaml is fair game | 17:49 |
| jeblair | sdague: er, to finish that sentence; i think such a rule would be workable. | 17:49 |
| fungi | but for related/partial bug headers (which are a relatively recent addition) i agree that a bugtask isn't strictly necessary | 17:49 |
| fungi | since all it wants to do is add comments | 17:50 |
| sdague | fungi: yeh, I just want comments | 17:50 |
| clarkb | morning | 17:50 |
| fungi | morning clarkb | 17:50 |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/nodepool: Allow useage of server IDs as well as names. https://review.openstack.org/69424 | 17:50 |
| sdague | basically if we end up doing a devstack patch because of a neutron thing | 17:50 |
| * clarkb tries to catch up on sb. | 17:50 | |
| clarkb | if no one else is working on the dpkg lock thing, I would like to tackle that beacuse I think we need to redo how are manifests are configured to puppet slaves | 17:51 |
| jeblair | clarkb: keep in mind we're almost done with long-running slaves now | 17:52 |
| *** DinaBelova is now known as DinaBelova_ | 17:52 | |
| clarkb | jeblair: yes, about that :) I am not convinced the single use unittest slaves are using the appropriate puppet manifest | 17:52 |
| clarkb | jeblair: I was looking at this stuff in response to some questions AaronGr and others had and I think we need to redo a bit of it | 17:52 |
| jeblair | clarkb: cool | 17:53 |
| *** luqas has quit IRC | 17:53 | |
| mordred | clarkb: AaronGr actually had an idea for refactoring some of that hierarchy to make it clearer | 17:55 |
| mordred | AaronGr: ^^ if clarkb is going to dive in there, now might be a good time to suggest those | 17:55 |
| fungi | jenkins02 is back online, running non-nodepool-requiring jobs, and nodepoold is launching new nodes to add to it | 17:55 |
| *** derekh has quit IRC | 17:55 | |
| clarkb | mordred: I think for a first stab I just want to make sure that the appropriate hosts have what they need to function | 17:55 |
| *** sandywalsh has quit IRC | 17:55 | |
| mordred | yeah | 17:55 |
| mordred | just wanted to make sure that if there was more, that I nudged AaronGr to speak up on the subject appropriately | 17:56 |
| AaronGr | mordred, clarkb: i would agree with clarkb -- i have about 50 patches to land before i would feel confident about suggesting my alternatives | 17:56 |
| *** DinaBelova_ is now known as DinaBelova | 17:56 | |
| clarkb | mordred: ++ | 17:56 |
| AaronGr | right now i'm getting supporting structure wired in, it's pretty rigid atm. | 17:56 |
| zaro | jog0: ok. don't know if this affects that testing but gerrit hooks to update LP bugs isn't working on review-dev | 17:58 |
| clarkb | jeblair: do you know why for a bare slave, the nodepool scripts run puppet twice? once in prepare_node.sh and prepare_bare_node.sh? | 17:59 |
| *** fbo is now known as fbo_away | 17:59 | |
| jog0 | zaro: not a problem and I am done with review-dev for now anyway | 17:59 |
| fungi | jenkins02 has jobs running on new nodepool nodes now, so looks like everything's working | 17:59 |
| clarkb | jeblair: I am pretty sure the puppet run in prepare_bare_node.sh is not needed | 17:59 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1271664 https://review.openstack.org/69056 | 18:00 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1272468 https://review.openstack.org/68987 | 18:00 |
| jeblair | clarkb: it uses a different class | 18:00 |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/nodepool: Allow useage of server IDs as well as names. https://review.openstack.org/69424 | 18:01 |
| jeblair | clarkb: it is made a slave, then it is made a bare_slave | 18:01 |
| fungi | slave_template vs bare_slave | 18:01 |
| openstackgerrit | Aaron Greengrass proposed a change to openstack-infra/config: Enable PATH for exec functions https://review.openstack.org/69427 | 18:01 |
| *** jerryz has joined #openstack-infra | 18:02 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 18:02 | |
| *** senk has quit IRC | 18:02 | |
| clarkb | jeblair: fungi: right, but putting bare_slave on slave_template is a noop | 18:03 |
| clarkb | it doesn't remove the jenkins user, it doesn't remove the ssh key, it is a strict subset of all the work slave_template has done | 18:03 |
| *** yassine has quit IRC | 18:04 | |
| *** yassine has joined #openstack-infra | 18:04 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1101147 https://review.openstack.org/69268 | 18:05 |
| *** nati_ueno has joined #openstack-infra | 18:05 | |
| *** julim has quit IRC | 18:05 | |
| jeblair | clarkb: i believe it removes sudo | 18:05 |
| clarkb | ah | 18:05 |
| *** harlowja_away is now known as harlowja | 18:05 | |
| jeblair | clarkb: or at least, it is intended to. if it does not remove sudo, i believe it's a bug. | 18:05 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 18:05 | |
| clarkb | jeblair: ok, I will look into that, thanks | 18:05 |
| *** david-lyle has quit IRC | 18:06 | |
| *** dizquierdo has joined #openstack-infra | 18:08 | |
| fungi | clarkb: jeblair: bug confirmed. the jenkins user on a bare-precise node i just tested can sudo commands without any error | 18:08 |
| *** yassine has quit IRC | 18:08 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 18:09 | |
| *** sandywalsh has joined #openstack-infra | 18:09 | |
| jeblair | clarkb: probably missing ensure=>absent | 18:10 |
| fungi | whereas the long-running slaves don't let jenkins sudo passwordlessly | 18:10 |
| fungi | tested and confirmed | 18:10 |
| dims | test nodes graph on http://status.openstack.org/zuul/ is very pretty | 18:10 |
| *** krotscheck has joined #openstack-infra | 18:14 | |
| fungi | clarkb: jeblair: yeah, the jenkins user is ending up in the sudo group on bare-precise slaves but not on long-running slaves | 18:15 |
| clarkb | fungi: jeblair: yeah I see the problem | 18:17 |
| *** mrmartin has joined #openstack-infra | 18:18 | |
| jeblair | clarkb, fungi: help me out with https://review.openstack.org/#/c/61321/6 and https://review.openstack.org/#/c/62066/ | 18:20 |
| jeblair | clarkb, fungi: i don't understand why nodepool needs to be changed for https://review.openstack.org/#/c/61321 | 18:20 |
| *** david-lyle has joined #openstack-infra | 18:21 | |
| Ajaeger | clarkb, jeblair, fungi, mordred: could you review and approve this one, please? https://review.openstack.org/#/c/67394/ I've seen too much breakage in these repos due to the noop-gate that I'd like to have proper gating soon. | 18:21 |
| *** gokrokve has quit IRC | 18:21 | |
| *** jcooley_ has quit IRC | 18:22 | |
| jeblair | Ajaeger: i'm working on a backlog of reviews in chronological order; it may be a while but i'll get to it. | 18:22 |
| *** jcooley_ has joined #openstack-infra | 18:23 | |
| *** dhellmann_ is now known as dhellmann | 18:23 | |
| *** dhellmann is now known as dhellmann_ | 18:23 | |
| *** gokrokve_ has joined #openstack-infra | 18:24 | |
| *** senk has joined #openstack-infra | 18:24 | |
| fungi | jeblair: we wanted to be able to pass a different ssh key for jenkins-dev slaves without injecting it into the calling environment for the daemon | 18:24 |
| jeblair | fungi: i seem to be missing that for the first patch | 18:24 |
| jeblair | fungi: where does it use an alternate key? | 18:24 |
| *** mrodden has quit IRC | 18:24 | |
| mrmartin | fungi: good progress, I get a different stacktrace with community portal gating: https://review.openstack.org/#/c/68912/3 http://logs.openstack.org/12/68912/3/check/gate-config-layout/2725f66/console.html | 18:25 |
| jeblair | fungi: (i'm specifically referring to the fact that you indicated the nodepool change is a dependency of the config change) | 18:25 |
| mrmartin | fungi: jenkins_jobs.errors.JenkinsJobsException: branch-designator parameter missing to format groups-release-{branch-designator} | 18:25 |
| *** nati_ueno has quit IRC | 18:25 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1273301 https://review.openstack.org/69400 | 18:26 |
| fungi | jeblair: it's possible 61321 actually is missing the addition to the config file | 18:26 |
| *** thedodd has quit IRC | 18:26 | |
| *** nati_ueno has joined #openstack-infra | 18:26 | |
| *** boris-42 has quit IRC | 18:26 | |
| *** nati_ueno has quit IRC | 18:26 | |
| jeblair | fungi: ok. i also am not keen on the change to nodepool; it's leaking information about our particular nodepool scripts into nodepool itself. | 18:27 |
| *** dcramer_ has joined #openstack-infra | 18:27 | |
| *** nati_ueno has joined #openstack-infra | 18:27 | |
| *** dkranz has joined #openstack-infra | 18:27 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 18:27 | |
| *** senk has quit IRC | 18:28 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1252947 https://review.openstack.org/69415 | 18:28 |
| *** boris-42 has joined #openstack-infra | 18:29 | |
| fungi | i honestly need to completely reacquaint myself with those changes and the reasons we wrote them, and reassess whether they're still needed. that was long enough ago i've forgotten most of the reasoning around that | 18:29 |
| Ajaeger | jeblair: Great! | 18:30 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Use nodeenv via tox to do javascript testing https://review.openstack.org/67729 | 18:31 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for libvirt socket connection refused bug 1251521 https://review.openstack.org/69418 | 18:32 |
| *** _ruhe is now known as ruhe | 18:32 | |
| devananda | clarkb: any chance to nudge you guys on https://review.openstack.org/#/c/65845/3 ? or still slammed with fixing the gate? | 18:33 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 18:33 | |
| jeblair | devananda: i won't speak for clarkb but since you said 'guys' i'll mention that i'm working through the review backlog chronologically | 18:34 |
| devananda | jeblair: ack. thanks | 18:35 |
| *** senk has joined #openstack-infra | 18:35 | |
| *** praneshp has joined #openstack-infra | 18:36 | |
| jeblair | mordred: is the jeepyb change needed by https://review.openstack.org/#/c/62187/7 running? | 18:36 |
| *** CaptTofu has quit IRC | 18:37 | |
| *** CaptTofu has joined #openstack-infra | 18:37 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for tempest race bug 1254772 https://review.openstack.org/69441 | 18:38 |
| *** mrodden has joined #openstack-infra | 18:39 | |
| *** senk has quit IRC | 18:40 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Database fixture added https://review.openstack.org/69384 | 18:40 |
| *** jchiles has joined #openstack-infra | 18:41 | |
| *** jchiles has quit IRC | 18:41 | |
| *** rlandy|bbl is now known as rlandy | 18:42 | |
| *** mfer has quit IRC | 18:42 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1272509 https://review.openstack.org/69029 | 18:42 |
| *** CaptTofu has quit IRC | 18:42 | |
| *** CaptTofu has joined #openstack-infra | 18:45 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Use nodeenv via tox to do javascript testing https://review.openstack.org/67729 | 18:45 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Redo slave manifests for clarity and correctness. https://review.openstack.org/69442 | 18:46 |
| *** gokrokve_ has quit IRC | 18:46 | |
| clarkb | I believe ^ will fix the dpkg and sudo problems | 18:46 |
| clarkb | there is one bit where I am not sure it will do the correct thing, I will leave a note inline | 18:47 |
| *** jamesmcarthur has joined #openstack-infra | 18:47 | |
| *** DinaBelova is now known as DinaBelova_ | 18:47 | |
| jamesmcarthur | mrmartin - Stef wanted me to tell you that he is rebotting. | 18:48 |
| mrmartin | hi | 18:48 |
| jamesmcarthur | If you tried to contact him… he didn't get it :) | 18:48 |
| jamesmcarthur | Hello sir! | 18:48 |
| mrmartin | I'm in anyway | 18:48 |
| *** reed has quit IRC | 18:48 | |
| *** esp has joined #openstack-infra | 18:50 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Collect list of installed packages at end of run https://review.openstack.org/63551 | 18:51 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 18:53 | |
| *** reed has joined #openstack-infra | 18:53 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 18:53 | |
| *** SnowDust has joined #openstack-infra | 18:54 | |
| SnowDust | anyone to help me with : https://jenkins02.openstack.org/job/gate-python-troveclient-pypy/156/console | 18:54 |
| *** che-arne has quit IRC | 18:56 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Genericize javascript release artifact creation https://review.openstack.org/67731 | 18:56 |
| *** marun has joined #openstack-infra | 18:56 | |
| fungi | SnowDust: looks like discover probably encountered an import exception when it tried to enumerate your tests | 18:57 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 18:57 | |
| fungi | SnowDust: i interpret that as "import errors: troveclient.compat.tests.test_common troveclient.compat.tests.test_xml" | 18:58 |
| *** dkliban_afk is now known as dkliban | 18:59 | |
| *** thedodd has joined #openstack-infra | 19:00 | |
| SnowDust | fungi: if thats the case .. then all the rest python versions should have raised the errors : https://review.openstack.org/#/c/64439 | 19:01 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: prepare_devstack: install linux-generic-lts-saucy https://review.openstack.org/69445 | 19:01 |
| SnowDust | fungi: all other versions passed except this pypy | 19:01 |
| *** sandywalsh has quit IRC | 19:02 | |
| *** fbo_away is now known as fbo | 19:03 | |
| fungi | SnowDust: unless it's something which only breaks under the pypy interpreter | 19:03 |
| SnowDust | fungi : but said that means the trove gates may have fallen by same measure ;) | 19:04 |
| SnowDust | thats not the case i think | 19:04 |
| *** afazekas has joined #openstack-infra | 19:04 | |
| *** Ajaeger has quit IRC | 19:07 | |
| *** vkozhukalov has quit IRC | 19:08 | |
| openstackgerrit | A change was merged to openstack-infra/config: Change mysql-devel to community-mysql-devel in Fedora https://review.openstack.org/62739 | 19:09 |
| openstackgerrit | A change was merged to openstack-infra/config: Explicitly document requirements for 3rd party testing https://review.openstack.org/63478 | 19:09 |
| clarkb | SnowDust: fungi: it is an easy thing to test, get a pypy interpreter and import htat module | 19:10 |
| SnowDust | ok .. :) | 19:10 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Added artifact upload of storyboard. https://review.openstack.org/67520 | 19:11 |
| *** senk has joined #openstack-infra | 19:12 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 19:12 | |
| *** VijayT has quit IRC | 19:13 | |
| *** ruhe is now known as _ruhe | 19:15 | |
| *** senk has quit IRC | 19:17 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added no_api env https://review.openstack.org/68610 | 19:20 |
| *** afazekas is now known as afazekas_pub | 19:21 | |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/elastic-recheck: Add query for bug 1273386 https://review.openstack.org/69448 | 19:25 |
| *** esp has left #openstack-infra | 19:26 | |
| *** VijayT has joined #openstack-infra | 19:27 | |
| *** VijayT has quit IRC | 19:29 | |
| *** VijayT has joined #openstack-infra | 19:29 | |
| openstackgerrit | A change was merged to openstack-infra/publications: Add a couple services we manage and checks https://review.openstack.org/65625 | 19:30 |
| *** gokrokve has joined #openstack-infra | 19:31 | |
| *** DinaBelova_ is now known as DinaBelova | 19:33 | |
| *** coolsvap has quit IRC | 19:33 | |
| *** hogepodge has joined #openstack-infra | 19:33 | |
| *** DinaBelova is now known as DinaBelova_ | 19:34 | |
| *** jasondotstar has joined #openstack-infra | 19:35 | |
| *** gokrokve has quit IRC | 19:36 | |
| *** jooools has quit IRC | 19:36 | |
| sdague | so we are definitely cresting on our number of nodes in use, and they don't seem to be in deleting or building state? someone want to check in on nodepool and make sure it's still happy? | 19:38 |
| clarkb | sdague: 43 nodes building, 341 used, 27 delete, and 29 ready | 19:40 |
| clarkb | according to nodepool | 19:40 |
| mtreinish | clarkb: what's the best way to merge this and the corresponding e-r change? https://review.openstack.org/#/c/68741/ | 19:41 |
| sdague | right, but, we have 700 | 19:41 |
| sdague | so I'm concerned why those numbers don't add up | 19:42 |
| fungi | sdague: we have only ~480 in use looking at the graph | 19:42 |
| clarkb | sdague: where do you see we have 700? | 19:43 |
| clarkb | mordred: there is a lot of OverLimit: Quota exceeded for cores: Requested 4, but already used 28 of 28 cores (HTTP 413) in the logs. did you bump ram quota without bumping core quota in rax land? | 19:43 |
| fungi | s/in use/in existence/ | 19:43 |
| jeblair | 2014-01-27 19:42:14,757 DEBUG nodepool.NodePool: Demand from gearman: bare-precise: 28 | 19:43 |
| *** senk has joined #openstack-infra | 19:43 | |
| jeblair | 2014-01-27 19:42:14,764 DEBUG nodepool.NodePool: Deficit: bare-precise: 4 (start: 28 min: 28 ready: 24) | 19:43 |
| jeblair | 2014-01-27 19:42:14,860 DEBUG nodepool.NodePool: <AllocationRequest for 4.0 of bare-precise> | 19:43 |
| jeblair | nodepool believes that only 28 bare precise nodes are needed and it's adding the last 4 needed in order to reach that | 19:44 |
| sdague | clarkb: from 2 hrs ago, and from my understanding of the new nodes that got added | 19:44 |
| clarkb | mtreinish: we can approve 68741 first, then once that is merged approve the e-r change | 19:44 |
| sdague | jeblair: right, which is definitely not true | 19:44 |
| fungi | sdague: are you suggesting that we should be using more of our quota than we are? | 19:44 |
| sdague | because we need about 1000 right now | 19:44 |
| *** SnowDust has quit IRC | 19:44 | |
| clarkb | sdague: remember we are throttling the gate | 19:44 |
| sdague | clarkb: we need this in check | 19:44 |
| clarkb | check is mostly fullfilled | 19:45 |
| jeblair | sdague: i see a handful of bare-precise jobs not running in check | 19:45 |
| fungi | sdague: almost everything in check has nodes already assigned | 19:45 |
| sdague | oh, maybe we filled it | 19:45 |
| jeblair | 3 infra-cores agree. :) | 19:45 |
| mtreinish | clarkb: ok yeah adding things to the yaml won't break anything | 19:45 |
| sdague | jeblair: :) | 19:45 |
| *** hashar has joined #openstack-infra | 19:46 | |
| * mordred agrees with the other 3 infra-cores | 19:46 | |
| mordred | it's unanimous | 19:46 |
| fungi | it looks (from the graph) like we spiked to up around 700 until we started running out of things to assign nodes to, and now the count is steadily dropping | 19:46 |
| fungi | which matches my expectations | 19:47 |
| mordred | jeblair: re: jeepyb - I'm 99% certain it has lnaded , but I will double check just to be sure and include a note in that review | 19:47 |
| *** gokrokve has joined #openstack-infra | 19:47 | |
| sdague | so given that we have a lot of extra quota, can we bump the gate window to 20 min? Because our pass rate is actually pretty good right now, and the backup from this morning is going to take a while to get through otherwise | 19:49 |
| jeblair | sdague: ++ | 19:49 |
| clarkb | wfm | 19:49 |
| openstackgerrit | A change was merged to openstack-infra/config: Display zuul version from zuul/status https://review.openstack.org/64211 | 19:50 |
| openstackgerrit | A change was merged to openstack-infra/config: Display last reconfigured time from zuul/status https://review.openstack.org/63850 | 19:50 |
| openstackgerrit | A change was merged to openstack-infra/config: Add ironic log files to logstash indexing https://review.openstack.org/64717 | 19:51 |
| sdague | it's actually funny to see us waiting on the single use nodes now | 19:51 |
| clarkb | mtreinish: I went ahead and approved the config yaml change for e-r | 19:52 |
| mtreinish | clarkb: ok cool | 19:52 |
| mtreinish | you can +A the e-r patch too :) | 19:52 |
| mordred | jeblair: storyboard server is running via puppet from the mordred environment on those patches that havent' landed yet (yay for testing!) | 19:53 |
| clarkb | oh right. mtreinish will do that once config update is in place | 19:53 |
| openstackgerrit | A change was merged to openstack-infra/config: Simplify devstack-logs publisher https://review.openstack.org/64938 | 19:53 |
| openstackgerrit | A change was merged to openstack-infra/config: Update devstack-gate jobs for Trove tempest tests https://review.openstack.org/65065 | 19:53 |
| clarkb | mordred: I just approved the change to add php sdk, I can run manage-projects if you don't want to | 19:53 |
| openstackgerrit | A change was merged to openstack-infra/config: Add different envs for different sqlalchemy versions https://review.openstack.org/65135 | 19:53 |
| mordred | jeblair: krotscheck would like to investigate some logs at the moment - any issues with me adding his key to that server - I intend to blow it away and recreate it from scratch before we go live | 19:53 |
| mtreinish | clarkb: cool thanks | 19:54 |
| jeblair | mordred: why do you intend on blowing it away? (and if that was your intent, why did you register it in dns?) | 19:55 |
| openstackgerrit | A change was merged to openstack-infra/config: write gerrit openstack/2.8 branch activity to #openstack-infra channel https://review.openstack.org/65165 | 19:55 |
| clarkb | jeblair: sanity check on https://review.openstack.org/#/c/63551/5/functions.sh the gzip -9 will rename the .txt file then the mv will fail | 19:56 |
| clarkb | jeblair: I suppose check tests will tell us if that is the case, I will just be patient | 19:56 |
| mordred | jeblair: to make sure that in the process of iterating on making sure that things work right I didn't inadvertently do something so that it wasn't - and good point on the dns - I believe I was overeager there | 19:56 |
| *** SnowDust has joined #openstack-infra | 19:56 | |
| jeblair | clarkb: heh, whoops. | 19:56 |
| mordred | jeblair: otoh, I could skip the blow-away-and-re-build step - things seem to be solidserver-wise | 19:57 |
| sdague | so what does one do to up the min window for the gate? | 19:58 |
| jeblair | mordred: if you are going to blow it away, including cleaning up dns and everything, i'm okay with ad-hoc adding his key; otherwise please don't. | 19:58 |
| jeblair | sdague: layout.yaml change | 19:58 |
| clarkb | sdague: in the zuul layout.yaml file, there is a window-floor option under the gate pipeline. bump that number | 19:58 |
| mordred | jeblair: ++ | 19:58 |
| *** habdi has quit IRC | 19:59 | |
| mordred | jeblair: noted. I'll pick one and only one path there | 19:59 |
| openstackgerrit | A change was merged to openstack-infra/config: Add jobs for cliff integration tests https://review.openstack.org/65180 | 19:59 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove unit tests against Puppet 3.0.x https://review.openstack.org/65406 | 19:59 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: up the gate window to 20 to help get through backlog https://review.openstack.org/69452 | 20:00 |
| openstackgerrit | Malini Kamalambal proposed a change to openstack-infra/config: Add Job for marconi-tempest integration https://review.openstack.org/65140 | 20:00 |
| *** thuc_ has quit IRC | 20:00 | |
| clarkb | fungi: re https://review.openstack.org/#/c/63902/3 is that applicable after the log location changes? I want to say that is a symlink now instead of a dir | 20:00 |
| * clarkb looks at diffs | 20:00 | |
| *** thuc has joined #openstack-infra | 20:01 | |
| *** dizquierdo has quit IRC | 20:01 | |
| *** thuc_ has joined #openstack-infra | 20:01 | |
| fungi | clarkb: the setup_workspace function i believe moves that before symlinking it, so possibly okay if it's being cleared and replaced prior to the function call | 20:02 |
| clarkb | fungi: oh I see, because we need to log stuff before /opt is a thing | 20:03 |
| * fungi nods | 20:03 | |
| fungi | i suspect it's going to need a rebase first though, and the recheck (again) should show us if there's a problem. we just need to keep an eye out for missing setup logs | 20:04 |
| clarkb | I don't think it needs a rebase, it touches bits that weren't touched by the other log stuff | 20:05 |
| jnoller | damn you scroll back, who was poking me? | 20:05 |
| *** ArxCruz has quit IRC | 20:05 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add projects section to elastic recheck bot yaml https://review.openstack.org/68741 | 20:05 |
| clarkb | jnoller: sdague was, said an hvm vm didn't have networking | 20:05 |
| openstackgerrit | A change was merged to openstack-infra/config: New project request: OpenStack SDK for PHP https://review.openstack.org/62069 | 20:05 |
| jnoller | ?! | 20:05 |
| jnoller | sdague: pong | 20:05 |
| sdague | jnoller: ping | 20:05 |
| *** ArxCruz has joined #openstack-infra | 20:05 | |
| *** thuc has quit IRC | 20:05 | |
| *** rossella_s has quit IRC | 20:05 | |
| jnoller | sdague what happen | 20:05 |
| *** openstackgerrit has quit IRC | 20:06 | |
| *** openstackgerrit has joined #openstack-infra | 20:06 | |
| sdague | try to spin up a guest with that image, it doesn't respond on ssh port | 20:06 |
| *** _ruhe is now known as ruhe | 20:06 | |
| *** julim has joined #openstack-infra | 20:06 | |
| *** hogepodge has quit IRC | 20:06 | |
| *** hashar has quit IRC | 20:07 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: up the gate window to 20 to help get through backlog https://review.openstack.org/69452 | 20:08 |
| sdague | jeblair: rebase ^^^ | 20:08 |
| *** nati_uen_ has joined #openstack-infra | 20:08 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Amend the fingerprint for 1249065 https://review.openstack.org/69458 | 20:08 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/devstack-gate: Collect list of installed packages at end of run https://review.openstack.org/63551 | 20:09 |
| *** frankbutt has joined #openstack-infra | 20:09 | |
| *** frankbutt has left #openstack-infra | 20:09 | |
| *** nati_ueno has quit IRC | 20:09 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Amend the fingerprint for 1249065 https://review.openstack.org/69458 | 20:10 |
| *** ArxCruz has quit IRC | 20:11 | |
| *** yolanda_ has quit IRC | 20:11 | |
| *** habdi has joined #openstack-infra | 20:14 | |
| *** SnowDust has quit IRC | 20:16 | |
| *** sarob has joined #openstack-infra | 20:17 | |
| *** gokrokve has quit IRC | 20:19 | |
| *** ICmonitor has joined #openstack-infra | 20:20 | |
| openstackgerrit | A change was merged to openstack-infra/config: up the gate window to 20 to help get through backlog https://review.openstack.org/69452 | 20:21 |
| *** sarob has quit IRC | 20:22 | |
| *** ICmonitor1 has quit IRC | 20:22 | |
| *** jcooley_ has quit IRC | 20:22 | |
| *** sarob has joined #openstack-infra | 20:22 | |
| *** jcooley_ has joined #openstack-infra | 20:22 | |
| *** sarob has quit IRC | 20:23 | |
| *** sarob has joined #openstack-infra | 20:23 | |
| jnoller | sdague: I'll yell | 20:24 |
| jnoller | at people | 20:24 |
| sdague | jnoller: cool | 20:24 |
| dstufft | jnoller as a service | 20:25 |
| fungi | yapaas | 20:25 |
| openstackgerrit | Dan Stangel proposed a change to openstack-infra/gitdm: Add header to output files and allow date ranges https://review.openstack.org/69461 | 20:27 |
| *** ociuhandu has quit IRC | 20:28 | |
| *** jamielennox|away has quit IRC | 20:28 | |
| jnoller | sdague: I barked at the team | 20:28 |
| *** lttrl has quit IRC | 20:29 | |
| *** sarob has quit IRC | 20:29 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: prepare_devstack: install linux-generic-lts-saucy https://review.openstack.org/69445 | 20:29 |
| anteaya | jnoller: do you take requests? | 20:30 |
| jnoller | yes | 20:30 |
| anteaya | I am tired of barking | 20:30 |
| fungi | [free bird!] | 20:30 |
| anteaya | you can have my bull horn | 20:30 |
| * anteaya hands jnoller her bull horn | 20:30 | |
| *** jamielennox|away has joined #openstack-infra | 20:31 | |
| *** rfolco has quit IRC | 20:31 | |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Groups community portal gating tasks https://review.openstack.org/68912 | 20:32 |
| *** gokrokve has joined #openstack-infra | 20:35 | |
| jnoller | sdague: found the issue | 20:37 |
| jnoller | sdague: they made the image better, use image ID: b9212b28-b42c-489e-8d7f-e0f370c15f89 /cc jeblair | 20:37 |
| *** nati_uen_ has quit IRC | 20:37 | |
| *** johnthetubaguy1 has quit IRC | 20:37 | |
| *** nati_ueno has joined #openstack-infra | 20:38 | |
| *** malini is now known as malini_afk | 20:38 | |
| sdague | jnoller: trying now | 20:38 |
| jnoller | nova boot hello2 --flavor performance1-1 --image b9212b28-b42c-489e-8d7f-e0f370c15f89 | 20:38 |
| *** gokrokve has quit IRC | 20:39 | |
| reed | fungi, I'll need the first batch of ATC emails by tomorrow morning | 20:39 |
| sdague | jnoller: well I can ssh to it :) | 20:40 |
| sdague | that's much better | 20:40 |
| jnoller | :fistbump: | 20:40 |
| *** smarcet has quit IRC | 20:43 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 20:44 | |
| fungi | reed: i'll put that together tonight--thanks for the reminder | 20:44 |
| *** rfolco has joined #openstack-infra | 20:44 | |
| reed | fungi, no problem | 20:44 |
| *** smarcet has joined #openstack-infra | 20:45 | |
| *** mrda_away is now known as mrda | 20:46 | |
| *** ruhe is now known as _ruhe | 20:46 | |
| *** mrmartin has quit IRC | 20:49 | |
| * devananda is getting increasingly frustrated by trying to add tempest testing for ironic to any pipelines | 20:50 | |
| clarkb | mtreinish: sdague: the config change to make https://review.openstack.org/#/c/67540/9 is in place on status.o.o. I am happy with 67540 do I need to wait before approving? | 20:51 |
| *** ociuhandu has joined #openstack-infra | 20:51 | |
| clarkb | devananda: I thought we had that change written? | 20:51 |
| devananda | clarkb: we did. it didn' tland and jeblair just -1'd it because of the -nv jobs | 20:51 |
| mikal | Morning | 20:52 |
| *** isviridov has joined #openstack-infra | 20:52 | |
| clarkb | devananda: oh, thats simple | 20:53 |
| clarkb | devananda: the suggestion is there is no value for them in the gate | 20:53 |
| clarkb | devananda: so just remove them from the gate pipeline. They are fine in chec | 20:53 |
| devananda | oh | 20:53 |
| devananda | thanks for clarifying | 20:53 |
| clarkb | devananda: gating on non voting jobs is silly because they don't vote and the gate relies on votes | 20:54 |
| clarkb | but we can definitely run them in check | 20:54 |
| devananda | gotcha | 20:54 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Add Storyboard puppet module https://review.openstack.org/65017 | 20:54 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Unlaunchpadify projects.yaml https://review.openstack.org/62189 | 20:54 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Track direct-release projects in projects.yaml https://review.openstack.org/62190 | 20:54 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Split config from projects list https://review.openstack.org/62187 | 20:54 |
| anteaya | mikal: morning | 20:54 |
| sdague | clarkb: no, you can go whenever, I just approved 67540 | 20:56 |
| *** _ruhe is now known as ruhe | 20:56 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add multi-project irc support to the bot https://review.openstack.org/67540 | 20:58 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Add a new node type for precise with saucy kernel https://review.openstack.org/69445 | 20:59 |
| *** gokrokve has joined #openstack-infra | 21:00 | |
| *** zz_ewindisch is now known as ewindisch | 21:01 | |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/config: Enable tempest/ironic gate tests https://review.openstack.org/65845 | 21:02 |
| *** jcoufal has quit IRC | 21:03 | |
| *** rlandy has quit IRC | 21:05 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Add a new node type for precise with saucy kernel https://review.openstack.org/69445 | 21:07 |
| *** gsamfira has joined #openstack-infra | 21:07 | |
| *** jgrimm has quit IRC | 21:07 | |
| *** rfolco has quit IRC | 21:09 | |
| *** sarob has joined #openstack-infra | 21:09 | |
| clarkb | russellb if ^ works we should use that kernel on all precise test nodes imo | 21:10 |
| russellb | clarkb: yeah, has worked in some testing ... idea was to first deploy it to an experimental job as a sanity check | 21:11 |
| clarkb | ++ | 21:11 |
| *** jgrimm has joined #openstack-infra | 21:12 | |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/elastic-recheck: Add query for bug 1273386 https://review.openstack.org/69448 | 21:13 |
| *** dhellmann_ is now known as dhellmann | 21:15 | |
| *** sarob has quit IRC | 21:15 | |
| *** ok_delta has joined #openstack-infra | 21:16 | |
| *** mattoliverau has joined #openstack-infra | 21:17 | |
| *** slong has joined #openstack-infra | 21:21 | |
| mattoliverau | Good morning | 21:22 |
| clarkb | devananda: you can remove the instantionation of gate-*-nv jobs from projects.yaml and remove them from the non voting section too | 21:23 |
| clarkb | devananda: basically those jobs can be entirely removed, then keep the check-*-nv jobs as is | 21:23 |
| *** jnoller has quit IRC | 21:23 | |
| *** senk has quit IRC | 21:24 | |
| openstackgerrit | Eric Harney proposed a change to openstack-infra/reviewstats: Update Cinder core list https://review.openstack.org/69471 | 21:25 |
| *** kraman has quit IRC | 21:25 | |
| *** kraman has joined #openstack-infra | 21:26 | |
| *** mattoliverau has quit IRC | 21:26 | |
| *** hogepodge has joined #openstack-infra | 21:27 | |
| *** mattoliverau has joined #openstack-infra | 21:27 | |
| openstackgerrit | A change was merged to openstack/requirements: Mirror gear - it's needed by tripleo-ci. https://review.openstack.org/69264 | 21:28 |
| *** gokrokve has quit IRC | 21:28 | |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/config: Enable tempest/ironic gate tests https://review.openstack.org/65845 | 21:29 |
| devananda | clarkb: hm, like that ^ ? | 21:29 |
| *** jhesketh has joined #openstack-infra | 21:31 | |
| *** jhesketh__ has joined #openstack-infra | 21:31 | |
| jhesketh__ | Hey | 21:31 |
| *** slong has quit IRC | 21:31 | |
| jeblair | jhesketh__: hey! can you take a look at this change? https://review.openstack.org/#/c/64738/ | 21:32 |
| *** gokrokve_ has joined #openstack-infra | 21:32 | |
| jhesketh__ | jeblair: sure :-) | 21:32 |
| *** e0ne has joined #openstack-infra | 21:32 | |
| *** slong has joined #openstack-infra | 21:33 | |
| *** afazekas_pub has quit IRC | 21:35 | |
| *** dprince has quit IRC | 21:35 | |
| devananda | clarkb: thanks for the pointers | 21:37 |
| *** jamielennox|away has quit IRC | 21:41 | |
| *** thuc_ has quit IRC | 21:41 | |
| *** thuc has joined #openstack-infra | 21:42 | |
| *** thuc has quit IRC | 21:42 | |
| clarkb | devananda: yup | 21:43 |
| clarkb | jeblair: do you want to rereview https://review.openstack.org/#/c/65845/ before I approve? | 21:43 |
| *** thuc has joined #openstack-infra | 21:43 | |
| jeblair | clarkb: i'll take it | 21:43 |
| *** jamielennox|away has joined #openstack-infra | 21:43 | |
| openstackgerrit | A change was merged to openstack-infra/config: Adding apache redirect for the cacti url. https://review.openstack.org/66181 | 21:44 |
| clarkb | also I believe https://review.openstack.org/#/c/63551/ is ready now. The grenade failure is unrelated | 21:44 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Update Cinder core list https://review.openstack.org/69471 | 21:45 |
| jeblair | clarkb: we probably could have gone with devstack-precise-check nodes for that, but i missed that the first time through so i won't block it on that | 21:45 |
| *** prad has joined #openstack-infra | 21:45 | |
| jhesketh__ | jeblair: done | 21:46 |
| *** Ryan_Lane has joined #openstack-infra | 21:46 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add turbo-hipster to rtfd https://review.openstack.org/66769 | 21:46 |
| *** smarcet has left #openstack-infra | 21:47 | |
| *** alexpilotti has quit IRC | 21:47 | |
| clarkb | jeblair: ok | 21:47 |
| *** alexpilotti_ has joined #openstack-infra | 21:48 | |
| *** prad has quit IRC | 21:48 | |
| *** prad_ is now known as prad | 21:48 | |
| *** miqui has quit IRC | 21:48 | |
| *** ruhe is now known as _ruhe | 21:48 | |
| openstackgerrit | A change was merged to openstack-infra/config: Update the rtfd info now you don't need an id https://review.openstack.org/66770 | 21:49 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1273386 https://review.openstack.org/69448 | 21:49 |
| *** dizquierdo has joined #openstack-infra | 21:49 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for tempest race bug 1254772 https://review.openstack.org/69441 | 21:49 |
| *** Ryan_Lane has quit IRC | 21:51 | |
| *** freyes has quit IRC | 21:51 | |
| *** senk has joined #openstack-infra | 21:51 | |
| fungi | i guess we're still using devstack-precise-check for hpcloud region b | 21:51 |
| *** jcooley_ has quit IRC | 21:53 | |
| jeblair | fungi: yeah, and moreover, I think as long as we're still moving providers around, we might want to keep that ability on hand if we need it | 21:53 |
| *** jcooley_ has joined #openstack-infra | 21:53 | |
| fungi | i concur | 21:53 |
| fungi | i just had to look back through the config to remind myself whether there was anywhere we were still assigning the label | 21:53 |
| *** ianw has joined #openstack-infra | 21:55 | |
| openstackgerrit | A change was merged to openstack-infra/config: Enable tempest/ironic gate tests https://review.openstack.org/65845 | 21:55 |
| clarkb | jeblair: curious about what you think of my comment on https://review.openstack.org/#/c/63921/ | 21:55 |
| openstackgerrit | A change was merged to openstack-infra/config: Add pylint job for anvil https://review.openstack.org/67103 | 21:56 |
| *** cadenzajon has joined #openstack-infra | 21:56 | |
| jeblair | clarkb: yeah, my preferences (for all of infra) are (a) ignore or remove hacking entirely; or (b) only use it with a whitelist (that includes the python3 checks) | 21:57 |
| jeblair | clarkb: i believe i will see sergey in 3 days so i figured i could discuss it with him then | 21:57 |
| clarkb | FOSDEM? | 21:58 |
| jeblair | clarkb: i was planning on chatting with mordred at lca about it, but that didn't happen. | 21:58 |
| clarkb | ya | 21:58 |
| jeblair | clarkb: storyboard/fosdem yes | 21:58 |
| Mithrandir | it was great saying hi to you guys at lca, btw | 21:58 |
| jeblair | clarkb, fungi: oh, i'll be flying all day tomorrow; wifi state unknown. | 21:59 |
| jeblair | Mithrandir: you too! great to meet you | 21:59 |
| fungi | jeblair: thanks for the heads up. i thought that was tomorrow but was going to ask to confirm | 21:59 |
| jeblair | fungi: yeah, i was not spoiled for choice on flights; i leave here early tues morning and arrive there early wed morning. it's not pretty. | 22:00 |
| fungi | ick | 22:00 |
| clarkb | jeblair: thanks for the warning | 22:00 |
| fungi | jeblair: i just hope your health is repaired sufficiently for the journey | 22:00 |
| *** dims has quit IRC | 22:00 | |
| jeblair | fungi: i think/hope so. :) | 22:01 |
| *** dims has joined #openstack-infra | 22:02 | |
| sdague | so as soon as we merge the next 3 changes in the gate, can we get a promote on - 69476,1 ? | 22:02 |
| *** mfer has joined #openstack-infra | 22:02 | |
| devananda | \o/ thanks guys | 22:02 |
| *** markwash has quit IRC | 22:03 | |
| fungi | sdague: you bet. that stevedore release fallout is lighting the gate up like a hannukah bush | 22:03 |
| *** gokrokve_ has quit IRC | 22:03 | |
| *** jasondotstar has quit IRC | 22:04 | |
| *** markwash has joined #openstack-infra | 22:04 | |
| sdague | we should be 2 minutes from the nova job at the top from uploading logs | 22:04 |
| *** ianw has quit IRC | 22:04 | |
| *** markwash has quit IRC | 22:05 | |
| fungi | gate is now running no new jobs... good time? | 22:05 |
| sdague | yep | 22:05 |
| sdague | go for it | 22:05 |
| fungi | reset | 22:05 |
| sdague | the nova change just merged | 22:05 |
| *** rnirmal has quit IRC | 22:05 | |
| *** CaptTofu has quit IRC | 22:06 | |
| clarkb | Mithrandir: jeblair fungi zaro https://review.openstack.org/#/c/63579/ everyone ok with monkey patching writexml in that change? it was my suggestion to find a different solution to the one originaly provided and I am reasonably happy with that change | 22:07 |
| sdague | also kenichi rebased his libvirt upload patch in d-g https://review.openstack.org/#/c/61892/ | 22:07 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1273455 https://review.openstack.org/69483 | 22:08 |
| Mithrandir | clarkb: that's pretty ugly, but I don't have a better solution. | 22:08 |
| *** SumitNaiksatam has quit IRC | 22:09 | |
| clarkb | Mithrandir: ya that is how I feel about it. I think using the other xml lib is bad because python3 and so on | 22:10 |
| mfer | jeblair I take it the naming convention openstack-sdk-[lang] is ok | 22:10 |
| Mithrandir | clarkb: well, a better solution would be to say "python2.6? hahahah". | 22:10 |
| mfer | i ask because i was going to look into the existing go bindings. changeing the launchpad project | 22:10 |
| clarkb | Mithrandir: I wish | 22:10 |
| Mithrandir | but I suspect that might not fly for reasons. | 22:10 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1273455 https://review.openstack.org/69483 | 22:11 |
| fungi | mfer: the jury's still out on confirming it won't be an issue, but until we hear otherwise i think it's safe to assume that's fine | 22:11 |
| *** CaptTofu_ has joined #openstack-infra | 22:11 | |
| *** ianw has joined #openstack-infra | 22:11 | |
| fungi | mfer: also keep in mind that stackforge project renaming is non-trivial and won't be instantaneous. we'd have to schedule a gerrit outage to rename the go sdk, so it would probably take some weeks to get around to it | 22:12 |
| fungi | thus, if it's purely for cosmetic reasons, it's probably not a good idea | 22:12 |
| openstackgerrit | Guido Günther proposed a change to openstack-infra/jenkins-job-builder: Support site monitor publisher https://review.openstack.org/69290 | 22:13 |
| *** dangers is now known as dangers_away | 22:16 | |
| mfer | fungi i get that. i was only thinking the launchpad name so it would be more findable. if/when it becomes an official project i'll worry about the git repo name | 22:16 |
| openstackgerrit | Guido Günther proposed a change to openstack-infra/jenkins-job-builder: project_maven: allow to set private repository https://review.openstack.org/69484 | 22:17 |
| *** johnthetubaguy has joined #openstack-infra | 22:17 | |
| *** e0ne has quit IRC | 22:18 | |
| clarkb | woo gerrit breaking ssh cli backward compat | 22:18 |
| fungi | mfer: you might also want to strike up a mailing list thread on openstack-dev@lists.openstack.org about non-python-language sdks and try to get some community consensus around them (naming conventions, documentation, whether there is any potential for them to be "official" in the sense we currently use the term). also there's a documentation effort underway for a developer.openstack.org site to | 22:18 |
| fungi | aggregate resources for application developers targeting deployment on openstack, which might be of interest to you | 22:18 |
| *** johnthetubaguy has quit IRC | 22:19 | |
| *** habdi has quit IRC | 22:19 | |
| *** ICmonitor1 has joined #openstack-infra | 22:19 | |
| *** ICmonitor has quit IRC | 22:20 | |
| jeblair | clarkb: link? | 22:21 |
| openstackgerrit | Max Lobur proposed a change to openstack/requirements: Add futures library to global requirements https://review.openstack.org/66349 | 22:22 |
| clarkb | jeblair: zaro and I just discovered it with manage-projects/gerritlib. at least for replication https://review-dev.openstack.org/plugins/replication/Documentation/cmd-start.html | 22:22 |
| mfer | fungi i saw the developer.openstack.org stuff. i was going to jump in but then i had to travel. this week i'll have limited time. thanks for pointing it out. i'm really happy to see this stuff happening. | 22:23 |
| clarkb | I haven't heard any strong dissent about monkey patching xml stuff for python26 in JJB. I will approve https://review.openstack.org/#/c/63579/ shortly | 22:23 |
| clarkb | jeblair: my suggestion to zaro is that we update gerritlib to determine version on initial connect then use a lookup table of commands | 22:23 |
| clarkb | what is going on with developer.o.o? | 22:23 |
| fungi | clarkb: the codename for the docs project to aggregate sdk links/descriptions and related info. i am still unconvinced that they need a separate subdomain for that site | 22:24 |
| clarkb | api.openstack.org | 22:25 |
| clarkb | :) | 22:25 |
| fungi | plus, it's a potential source of confusion with that name, since docs.o.o/developer is for our api documentation | 22:25 |
| annegentle | clarkb: fungi: I brought that up at the summit session | 22:25 |
| annegentle | clarkb: fungi: and yes, that too | 22:25 |
| fungi | the argument in favor of developer.o.o as a domain is that there are people in the tech industry trained to blindly enter developer.somedomain into their browsers to find sdk download links | 22:26 |
| annegentle | clarkb: fungi: I'm happy to defend developer.openstack.org as a dev portal, with common sub-domain data at http://blog.programmableweb.com/2013/06/04/anatomy-of-a-developer-portal-url/ | 22:26 |
| annegentle | fungi: precisely, you beat me to it :) | 22:26 |
| annegentle | fungi: thunder stealer | 22:26 |
| fungi | heh, sorry ;) | 22:27 |
| clarkb | fungi: we should put lolcats at that location then | 22:27 |
| annegentle | api subdomain is a bit of a distant second the way I see it? | 22:27 |
| fungi | anyway, i say it's a lemming argument, but i will sway in whichever direction the community feels is appropriate | 22:27 |
| annegentle | but we also have the conflation problem of dev contributors vs. dev consumers | 22:27 |
| clarkb | CAN HAZ API DOCS? YOU CAN HAZ AT api.openstack.org | 22:27 |
| annegentle | clarkb: LOL CATZ | 22:27 |
| annegentle | clarkb: yeah I'm also fine with "sorry that train has left the station labeled api.openstack.org" | 22:28 |
| annegentle | clarkb: and we don't re-label trains. or something. | 22:28 |
| jeblair | i think we do all of them and put different things on all of them. | 22:28 |
| clarkb | fungi: https://review.openstack.org/#/c/64307/4/git_review/cmd.py do you know why the default is pushURL and gets url instead of being the other way around? | 22:28 |
| clarkb | fungi: I would assume pushURL should be the thing you get and default to url | 22:28 |
| jeblair | ;) | 22:28 |
| * clarkb mans git config | 22:28 | |
| annegentle | jeblair: already on that journey sir :) | 22:28 |
| clarkb | fungi: yeah I think we should flip that logic around | 22:29 |
| *** thuc has quit IRC | 22:29 | |
| *** thuc has joined #openstack-infra | 22:30 | |
| *** _ruhe is now known as ruhe | 22:30 | |
| *** ewindisch is now known as zz_ewindisch | 22:32 | |
| *** senk has quit IRC | 22:33 | |
| *** habdi has joined #openstack-infra | 22:34 | |
| openstackgerrit | Guido Günther proposed a change to openstack-infra/jenkins-job-builder: project_maven: allow to set private repository https://review.openstack.org/69484 | 22:34 |
| *** thuc has quit IRC | 22:34 | |
| *** mfink has quit IRC | 22:34 | |
| *** hogepodge_ has joined #openstack-infra | 22:35 | |
| *** hogepodge has quit IRC | 22:36 | |
| *** hogepodge_ is now known as hogepodge | 22:36 | |
| jeblair | devananda: ok with https://review.openstack.org/#/c/68092/1 ? | 22:36 |
| clarkb | I have approved JJB monkey patching | 22:37 |
| zaro | clarkb, jeblair : just noticed that there is a gerrit ssh-alias config https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#ssh-alias | 22:37 |
| *** ruhe is now known as _ruhe | 22:38 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Add Storyboard puppet module https://review.openstack.org/65017 | 22:38 |
| clarkb | zaro: jeblair: I think that solves this problem for us, but does not solve it for gerritlib where users may not have admin rights to configure the running server | 22:38 |
| openstackgerrit | A change was merged to openstack-infra/config: Add a devstack-gate core group https://review.openstack.org/64882 | 22:38 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Add tests for YamlParser and patch 2.6 minidom https://review.openstack.org/63579 | 22:38 |
| clarkb | but maybe we start there in order to get us through the migration | 22:38 |
| clarkb | russellb: do you know if the new kernel will help https://review.openstack.org/#/c/67564/ ? I seem to recall danpb saying the bug is in libvirt itself so I am not hopeful | 22:39 |
| *** dkranz has quit IRC | 22:39 | |
| zaro | we can start there. i can also patch gerritlib with an update to handle both commands as well, but maybe to apply after upgrade. | 22:39 |
| openstackgerrit | A change was merged to openstack-infra/config: Chef style testing enablement and minor speed cleanup starting with checks https://review.openstack.org/67964 | 22:40 |
| *** jamesmcarthur has quit IRC | 22:40 | |
| devananda | jeblair: yep | 22:41 |
| zaro | clarkb: i'm cool with jjb patch 63579 | 22:41 |
| openstackgerrit | A change was merged to openstack-infra/config: Pack zuul git refs daily. https://review.openstack.org/68222 | 22:41 |
| *** CaptTofu_ has quit IRC | 22:41 | |
| fungi | clarkb: agreed i think that's implemented backward in git-review | 22:43 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add bugdaystats to openstack-infra https://review.openstack.org/69489 | 22:43 |
| clarkb | jeblair: remind me, using x to advance the zuul simulation is helpful because of the remote right? | 22:44 |
| *** habdi has quit IRC | 22:44 | |
| clarkb | the remote does left and right arrows but not key click | 22:44 |
| * fungi is disappearing for a couple hours to go out for dinner, since he may be snowed in for the next couple days | 22:45 | |
| jeblair | clarkb: yep. that's totally just my remote. probably some others too. supporting left/right arrows in the simulation is _hard_. | 22:45 |
| fungi | jeblair: keep an eye on the forecast if you have a layover on the east coast. supposedly it's getting fun over here again | 22:46 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add bugdaystats to openstack-infra https://review.openstack.org/69489 | 22:46 |
| jeblair | fungi: i do connect through iad tomorrow afternoon; last i checked it was not supposed to be terrible | 22:47 |
| fungi | ahh, yeah, problems will supposedly be southeast of dulles | 22:47 |
| openstackgerrit | A change was merged to openstack-infra/publications: Allow 'x' key to advance the zuul simulation https://review.openstack.org/64975 | 22:48 |
| reed | rockstar, you there? | 22:48 |
| * fungi vanishes... bbl | 22:48 | |
| rockstar | reed, I am. | 22:49 |
| *** rnirmal has joined #openstack-infra | 22:50 | |
| *** flaper87 is now known as flaper87|afk | 22:50 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Readme enhancements https://review.openstack.org/66011 | 22:54 |
| *** ICmonitor has joined #openstack-infra | 22:55 | |
| *** ICmonitor1 has quit IRC | 22:56 | |
| *** ICmonitor1 has joined #openstack-infra | 22:59 | |
| openstackgerrit | A change was merged to openstack-infra/publications: Add sysadmin-in-git slide https://review.openstack.org/64960 | 22:59 |
| *** ArxCruz has joined #openstack-infra | 22:59 | |
| *** ICmonitor has quit IRC | 23:00 | |
| *** habdi has joined #openstack-infra | 23:00 | |
| *** dhellmann is now known as dhellmann_ | 23:01 | |
| *** thuc has joined #openstack-infra | 23:01 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/devstack-gate: Remove reference to Developer Setup in docs https://review.openstack.org/69494 | 23:04 |
| *** dizquierdo has quit IRC | 23:04 | |
| *** e0ne has joined #openstack-infra | 23:05 | |
| jeblair | notmyname: formpost builds on another piece of middleware that also does hmac validations of PUTs, right? | 23:05 |
| jeblair | notmyname: what's that piece called? | 23:05 |
| *** dcramer_ has quit IRC | 23:05 | |
| *** JpMaxMan has joined #openstack-infra | 23:06 | |
| *** jamielennox|away is now known as jamielennox | 23:06 | |
| notmyname | jeblair: you're probably thinking of tempurl. formpost uses some code from tempurl, but it doesn't depend on tempurl being enabled | 23:06 |
| jhesketh__ | jeblair: are you looking at the formpost stuff I started in zuul? | 23:07 |
| clarkb | jeblair: https://review.openstack.org/#/c/66053/ the stack there has my +2, I don't feel too strongly about squashing | 23:07 |
| jeblair | jhesketh__: yep; i was just starting to recall a conversation i had with notmyname a while ago.... | 23:07 |
| jeblair | notmyname: https://review.openstack.org/#/c/68297 (is the review under discussion) | 23:07 |
| jhesketh__ | jeblair: so the reason I used formpost there was to support multiple files (which tempurl does not) | 23:08 |
| *** kraman has quit IRC | 23:08 | |
| jeblair | jhesketh__, notmyname: where i think he said we might want to use tempurl rather than formpost since we aren't actually using a browser and can perform PUTs... | 23:08 |
| notmyname | jeblair: ah. that makes sense. | 23:08 |
| jhesketh__ | agreed, but we'd have to generated a tempurl for each file we want to push | 23:08 |
| *** thuc has quit IRC | 23:09 | |
| jeblair | jhesketh__: oh, you can't allow multiple PUTs into a directory with tempurl? | 23:09 |
| jhesketh__ | no, it has to be the final object path | 23:09 |
| jhesketh__ | not the prefix | 23:09 |
| clarkb | jeblair: I will let you approve or not approve based on squash preferences | 23:09 |
| jhesketh__ | https://review.openstack.org/#/c/68327/1/turbo_hipster/lib/utils.py -- this is an example of sending files to the formpost middleware | 23:10 |
| *** markvan has quit IRC | 23:10 | |
| jhesketh__ | quite easy with the requests lib | 23:10 |
| clarkb | hrm https://review.openstack.org/#/c/64973/ has not merged yet. wonder what is going on there | 23:10 |
| *** e0ne has quit IRC | 23:10 | |
| clarkb | http://git.openstack.org/cgit/openstack-infra/publications/log/?h=overview shows it did merge | 23:11 |
| jeblair | clarkb: yeah it sat there a while so i figure if we're okay let's just merge both changes | 23:11 |
| clarkb | gerrit is just not updating | 23:11 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack/requirements: Bump stevedore to 0.14 https://review.openstack.org/69496 | 23:11 |
| clarkb | jeblair: wfm | 23:11 |
| jeblair | jhesketh__: ah, well we definitely want that. | 23:11 |
| *** jgrimm has quit IRC | 23:12 | |
| *** masayukig has joined #openstack-infra | 23:12 | |
| jeblair | notmyname: feature request for tempurl: support allowing multiple files to be uploaded with a matching prefix like formpost. | 23:12 |
| clarkb | notmyname: fyi I just approved the change to log the swift channel | 23:12 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Use nodeenv via tox to do javascript testing https://review.openstack.org/67729 | 23:12 |
| notmyname | clarkb: thanks | 23:12 |
| jhesketh__ | jeblair: yeah, so my next step is working out how infra's log pushers can use this | 23:12 |
| jeblair | notmyname: would a bug for that be okay? | 23:12 |
| notmyname | jeblair: doesn't sound like a bug. maybe a blueprint | 23:12 |
| jeblair | jhesketh__: so i think we write a dinky program that basically takes some args of what files to push and does this https://review.openstack.org/#/c/68327/1/turbo_hipster/lib/utils.py | 23:15 |
| *** dkliban has quit IRC | 23:15 | |
| jhesketh__ | jeblair: yep, agreed.. then we just swap out the scp for it | 23:16 |
| jhesketh__ | jeblair: although with that feature request for tempurl, is that something you want to see land before zuul handles formpost? | 23:16 |
| *** jergerber has quit IRC | 23:16 | |
| jeblair | jhesketh__: or, i believe there might be swift-compatible artifact storage in jenkins... that might be worth a look | 23:16 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Genericize javascript release artifact creation https://review.openstack.org/67731 | 23:16 |
| jeblair | jhesketh__: no, it was just in the spirit of improving swift | 23:16 |
| *** sarob has joined #openstack-infra | 23:16 | |
| jeblair | jhesketh__: we can use formpost i think | 23:16 |
| notmyname | jeblair: you misspelled "making swift awesomer" | 23:17 |
| jeblair | jhesketh__: regardless of whether we have jenkins or a little python program upload the swift artifacts; we also need to do something about the console log... | 23:17 |
| *** sarob has quit IRC | 23:17 | |
| jhesketh__ | yes, that is an issue | 23:17 |
| *** ryanpetrello has quit IRC | 23:18 | |
| *** sarob has joined #openstack-infra | 23:18 | |
| clarkb | jhesketh__: jeblair: can't turbohipster spit that to swift? | 23:18 |
| *** prad has quit IRC | 23:18 | |
| jeblair | jhesketh__: i think i left that at "it's not forever, so let's just fetch it over http from jenkins then upload it to swift, even though that's not superelegant" | 23:18 |
| clarkb | oh jenkins | 23:18 |
| jeblair | clarkb: we are not running turbo hipster yet | 23:18 |
| clarkb | jeblair: for some reason I assumed using swift required getting rid of jenkins | 23:18 |
| jeblair | jhesketh__: we could have the little python program get it with pycurl | 23:18 |
| jeblair | clarkb: no, getting rid of jenkins requires swift | 23:18 |
| clarkb | if folks are brave they can update the jclouds plugin to do console.html with swift | 23:19 |
| clarkb | jeblair: why? | 23:19 |
| clarkb | jeblair: slave gearman workers could send work requests to file uploaders | 23:19 |
| jhesketh__ | jeblair: won't that fetch the console log before all the actions are completed? | 23:19 |
| jeblair | clarkb: because doing both at once is a bit beyond our capabilities at the moment, and the os-log-analyze needs to support both anyway, and that's the bulk of the work | 23:19 |
| clarkb | and use gearman to transport the data | 23:19 |
| *** _sarob has joined #openstack-infra | 23:20 | |
| *** _sarob has quit IRC | 23:20 | |
| jeblair | clarkb: does jclouds support swift formpost? | 23:21 |
| clarkb | jeblair: I am not sure | 23:21 |
| jeblair | clarkb: what are you suggesting with "jeblair: slave gearman workers could send work requests to file uploaders" ? | 23:21 |
| clarkb | jeblair: aiui the thing turbohistory (or not jenkins) would need to work around is credentials to move files from place X to Y | 23:21 |
| jeblair | clarkb: i doubt it does, so it seems to me that getting it to support both formpost and the console log is a lot of java work for something we're trying to get rid of; therefore, i think a simple python script is the way to go | 23:22 |
| clarkb | jeblair: if there are trusted machines with that info then gearman can farm work out to them | 23:22 |
| clarkb | and that should work regardless of job runner | 23:22 |
| *** sarob has quit IRC | 23:22 | |
| jeblair | clarkb: that's the easy part of this. that's what formpost is for. | 23:22 |
| *** _ruhe is now known as ruhe | 23:22 | |
| clarkb | what is the hard part? | 23:22 |
| jeblair | clarkb: we allow the machines themselves to have a limited amount of trust, therefore we don't have to funnel anything through a bottleneck | 23:23 |
| *** ok_delta has quit IRC | 23:23 | |
| clarkb | jeblair: I get that. Trying to decouple jenkins from this though | 23:23 |
| clarkb | I don't understand the strict ordering we have imposed | 23:23 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added no_api env https://review.openstack.org/68610 | 23:24 |
| *** oubiwann_ has quit IRC | 23:24 | |
| clarkb | jeblair: oh I see the confusion. I was suggesting we could continue to use scp with gearman workers that just scp. then use swift | 23:25 |
| clarkb | then console logs are not a problem. but it does mean a slower transition to swift | 23:26 |
| jeblair | clarkb: i'd like to change one thing at a time, and in a way where we can scale up or down as needed so we don't break everything at once. | 23:26 |
| clarkb | if you want to get console logs with swift formpost in jenkins I think you have to hack on java somewhere | 23:26 |
| jeblair | clarkb: yeah, and i don't want to do that because we're trying to get rid of jenkins so hacking on it when not absolutely necessary is not a good use of time | 23:27 |
| *** ArxCruz has quit IRC | 23:27 | |
| *** sdague has quit IRC | 23:27 | |
| jeblair | clarkb: instead, we can write a simple log-uplodaing script that re-uses the code from turbohipster anyway. very little time wasted on that, and it means we can shake everything out of using swift without _also_ having to worry about the entire rest of the job running framework changing at the same time | 23:28 |
| *** sdague has joined #openstack-infra | 23:28 | |
| jeblair | clarkb: the only ugly part is the console log, and i'm suggesting we just pycurl it and upload it within this log uploading script. | 23:28 |
| jeblair | jhesketh__: it will not contain the very end of the console log, but i think we can live with it. | 23:28 |
| jhesketh__ | sure, in theory the only thing it'll miss is it uploading itself | 23:29 |
| *** jpeeler has quit IRC | 23:29 | |
| jeblair | jhesketh__: especially if it's the last part | 23:29 |
| jeblair | jhesketh__: right | 23:29 |
| clarkb | jeblair: without the end of the console log you can't see success of failure in the log itself | 23:29 |
| jhesketh__ | but if the upload fails I'm not sure what that means | 23:29 |
| clarkb | which is important and why the giant hack exists in the scp plugin | 23:29 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Lump all defaults reading into one place https://review.openstack.org/69502 | 23:30 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Split the config out into two files https://review.openstack.org/69503 | 23:30 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: register failure for intermitent ipv6 fails https://review.openstack.org/69504 | 23:30 |
| *** jpeeler has joined #openstack-infra | 23:30 | |
| *** jpeeler has quit IRC | 23:30 | |
| *** jpeeler has joined #openstack-infra | 23:30 | |
| jeblair | clarkb: if we updated our jobs to output their own indication of success/failure, then the only thing we'd miss would be a failure caused by the upload step itself, which, actually we configure not to cause a failure. | 23:31 |
| mordred | clarkb, jeblair: I SO thought I'd uploaded those patches above and that they were in production already | 23:31 |
| mordred | turns out, not so much | 23:31 |
| clarkb | jeblair: we could do that | 23:31 |
| jhesketh__ | jeblair: what do you mean by having the jobs output their own indication? | 23:32 |
| jhesketh__ | you mean before they return their exit code they print something? | 23:32 |
| clarkb | I still think the potential for confusion is there if test says SUCCESS but zuul comment in gerrit says fail | 23:33 |
| *** dkliban has joined #openstack-infra | 23:33 | |
| *** dims has quit IRC | 23:33 | |
| jeblair | jhesketh__: yes | 23:34 |
| mrodden | if i a patch fails jenkins due to a job LOST, should i just do a recheck no bug on it? | 23:34 |
| mrodden | re: https://review.openstack.org/#/c/69490/ | 23:34 |
| jeblair | clarkb: i think making the log-uploading portion not be considered for job failure will greatly reduce the chance for that | 23:35 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: register failure for intermitent ipv6 fails https://review.openstack.org/69504 | 23:35 |
| clarkb | jeblair: I agree. The areas we may have trouble is ftp'ing doc jobs for example | 23:36 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Make IRC messages provide more context https://review.openstack.org/68811 | 23:36 |
| jeblair | clarkb: yep. which is really something we should try to stop doing. :) | 23:37 |
| * jhesketh__ has to run to a meeting unfortunately | 23:39 | |
| clarkb | jeblair: should I approve zuul changes that you have +2'd? | 23:40 |
| jeblair | clarkb: sure | 23:40 |
| *** david-lyle has quit IRC | 23:40 | |
| clarkb | assuming I don't find reason to -1 them. | 23:40 |
| clarkb | on that front should https://review.openstack.org/#/c/66173/3 have tests? | 23:40 |
| clarkb | jhesketh__: ^ | 23:40 |
| *** morganfainberg|z is now known as morganfainberg | 23:41 | |
| mrodden | https://review.openstack.org/#/c/69490/ << is 'LOST' bad? | 23:42 |
| mrodden | i dont think i've ever seen Jenkins lose a job before | 23:42 |
| mordred | mrodden: jenkins is a doddering old man - he loses things all the tie | 23:43 |
| *** jhesketh has quit IRC | 23:43 | |
| mrodden | mordred: touche... guess i haven't noticed until now | 23:43 |
| mrodden | is it safe to just recheck no bug that one? | 23:43 |
| jeblair | mrodden: oh, i think that job doesn't exist everywhere yet. it's a recent change that went in. yes do that. | 23:44 |
| *** rahmu_ has joined #openstack-infra | 23:44 | |
| mrodden | ok cool thanks | 23:44 |
| *** ruhe is now known as _ruhe | 23:44 | |
| *** thedodd has quit IRC | 23:45 | |
| jeblair | mrodden: you can self-debug this by checking these urls: https://jenkins01.openstack.org/job/gate-cookbook-openstack-network-chef-style/? | 23:45 |
| openstackgerrit | A change was merged to openstack-infra/config: start logging openstack-swift https://review.openstack.org/65969 | 23:45 |
| jeblair | mrodden: change jenkins01 to jenkins 01-07 | 23:45 |
| mikal | I am under the current belief that we don't want the recheck bot running. Is that correct? | 23:45 |
| *** dtroyer_zz has joined #openstack-infra | 23:45 | |
| jeblair | mrodden: and make sure it exists everywhere | 23:46 |
| *** yamahata__ has joined #openstack-infra | 23:46 | |
| mrodden | jeblair: ok, makes sense | 23:46 |
| mrodden | wasn't aware that was a new job | 23:46 |
| jeblair | mikal: we ran into a small complication with having zuul do it... | 23:46 |
| mikal | jeblair: heh, its always the way | 23:46 |
| jeblair | mikal: it turns out that gerrit does not record an updated timestamp for a vote if the vote does not change | 23:46 |
| mrodden | wonder what the diff between -lint and -style is... | 23:46 |
| *** jd__` has joined #openstack-infra | 23:47 | |
| jeblair | mikal: so determining when jenkins last left a vote on a change is complicated | 23:47 |
| mikal | jeblair: huh, but you can just hook the event stream, right? | 23:47 |
| mikal | jeblair: oh, I see | 23:47 |
| mikal | jeblair: my bot does that by doing a new gerrit query for comments, and then parsing the author of the comments | 23:47 |
| mikal | jeblair: couldn't you do that? | 23:47 |
| jeblair | mikal: what do you look for in a comment? | 23:48 |
| mikal | jeblair: I think I just check for comments authored by jenkins, but I can double check that if you'd like | 23:48 |
| mrodden | looks like the job is propagated to 01-07 now | 23:48 |
| *** rahmu has quit IRC | 23:48 | |
| *** rahmu_ is now known as rahmu | 23:48 | |
| *** dtroyer has quit IRC | 23:48 | |
| *** yamahata has quit IRC | 23:48 | |
| *** jd__ has quit IRC | 23:48 | |
| *** ianw has quit IRC | 23:48 | |
| *** clarkb has quit IRC | 23:48 | |
| *** nikhil__ has quit IRC | 23:48 | |
| *** nikhil__ has joined #openstack-infra | 23:48 | |
| *** jd__` is now known as jd__ | 23:48 | |
| *** dims has joined #openstack-infra | 23:48 | |
| *** ianw has joined #openstack-infra | 23:48 | |
| jeblair | mikal: yeah, so that won't quite work because we don't care about 'check experimental' or other such comments, we only care about voting comments from the check queue | 23:49 |
| *** clarkb has joined #openstack-infra | 23:49 | |
| *** rcleere has quit IRC | 23:49 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Add docs https://review.openstack.org/66367 | 23:49 |
| jeblair | so we could look for comments that say "Doesn't seem to work" or "Works for me" | 23:49 |
| mikal | jeblair: fair point | 23:49 |
| jeblair | and get the timestamp from those | 23:49 |
| mikal | jeblair: that would still misfire on experimental for example though | 23:49 |
| clarkb | jeblair: what was the last thing you saw from me? | 23:49 |
| *** thuc has joined #openstack-infra | 23:49 | |
| jeblair | mikal: no, they don't vote so they won't leave those phrases | 23:50 |
| mordred | mikal: experimental doesn't vote | 23:50 |
| jeblair | 23:43 < clarkb> on that front should https://review.openstack.org/#/c/66173/3 have tests? | 23:50 |
| jeblair | 23:43 < clarkb> jhesketh__: ^ | 23:50 |
| mikal | Ok, but imagine a world in which they one day do | 23:50 |
| jeblair | so i think these are our options relating to that: | 23:50 |
| mikal | Or another queue votes as well | 23:50 |
| clarkb | jeblair: I also said I approved https://review.openstack.org/#/c/66189/ without a second +2 as it seemed minor. Hopefulyl graphite.o.o doesn't disagree with me :) | 23:50 |
| mikal | Or are you locking in the assumption that only one queue will every vote? | 23:50 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Collect and report last reconfigured timestamp https://review.openstack.org/63849 | 23:50 |
| mikal | s/every/ever/ | 23:50 |
| clarkb | also yay for flapping irc | 23:50 |
| jeblair | (a) clarkb is looking into fixing gerrit so the timestamp is updated | 23:50 |
| mikal | LOL, fixing gerrit | 23:51 |
| jeblair | (b) we can have zuul clear the check vote (pro: easy; con: adds an extra entry to the gerrit comment log) | 23:51 |
| clarkb | jeblair: oh ya, zaro and I looked at it a bit today, I know how I should fix it for master now | 23:51 |
| clarkb | but, I didn't go far enough down that path to write the patch, trying to do review insteade | 23:51 |
| jeblair | (c) we search for "Works for me" and "Doesn't seem to work" comments from jenkins | 23:51 |
| jeblair | I think those are the options relating to this. | 23:52 |
| jeblair | (a) is a change to gerrit, (c) is a change to zuul, (b) is a zuul config change | 23:52 |
| jeblair | mikal: and yeah, i think we can assume that more than one zuul pipeline will not vote in the same review category | 23:52 |
| jeblair | mikal: otherwise they would fight for the vote. | 23:53 |
| mikal | So, should I fire up my temporary bot again to cover until you get something working? | 23:53 |
| *** thuc_ has joined #openstack-infra | 23:53 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Lump all defaults reading into one place https://review.openstack.org/69502 | 23:53 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Split the config out into two files https://review.openstack.org/69503 | 23:53 |
| jeblair | mikal: if we chose a or c, yes; if we chose b, no because i think we can do that quickly. | 23:53 |
| mordred | jeblair, zaro: mind if I install that ^^ on review-dev and test there? | 23:53 |
| jeblair | clarkb, mordred, fungi, sdague: ^ thoughts? | 23:54 |
| mikal | jeblair: well, let me know either way... | 23:54 |
| jeblair | mikal: wfm | 23:54 |
| mordred | jeblair: I think we should pursue a, becuase it's "right" and because we're not dying at the moment | 23:54 |
| mordred | jeblair: and b is super easy should we need it | 23:54 |
| jeblair | mordred: some of the gate reforms sdague would like to do depend on this; should we consider doing b until a is in place? or are we willing to stay in this position for, let's say 2 weeks? | 23:55 |
| jog0 | so is it just me or are these trees different: http://git.openstack.org/cgit/openstack-dev/grenade/log/ https://github.com/openstack-dev/grenade/commits/master | 23:55 |
| *** thuc has quit IRC | 23:55 | |
| mordred | jeblair: hrm. that's a good question. the cost of the extra vote is not huge, it's just distastful | 23:55 |
| clarkb | mordred: I agree we should pursue a, but it isn't going to happen this week imo | 23:56 |
| *** mrda_ has joined #openstack-infra | 23:56 | |
| clarkb | I totally expect pushback from upstream | 23:56 |
| clarkb | despite assurances in irc that it is probably a bug | 23:56 |
| mordred | jeblair: perhaps putting b in place until a gets sorted, then we can go back to unchatty | 23:56 |
| *** mrda_ has left #openstack-infra | 23:57 | |
| zaro | mordred: should be ok. | 23:57 |
| clarkb | and I expect that because looking at the code this is clearly a DB optimization they have in place | 23:57 |
| mordred | I say that - zaro, puppet hasn't been run on review-dev in quite a while ... I don't have a working key there | 23:57 |
| *** kraman has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!