| notmorgan | ugh. i somehow caused the py3 job to push to timeout (why is the timer not catching it?) | 00:00 |
|---|---|---|
| pabelanger | notmorgan: I might rework the pause / unpause command, since they are on/off. | 00:00 |
| *** bpokorny has quit IRC | 00:00 | |
| notmorgan | pabelanger: *nod* | 00:00 |
| *** dingyichen has joined #openstack-infra | 00:00 | |
| ianw | every day at 10am i seem to manage to click a "save" button on a review | 00:01 |
| ianw | right when gerrit goes off to sleep for a bit | 00:02 |
| pabelanger | ianw: BTW: http://tripleo.org/cistatus.html tripleo is off fedora22 snapshot builds. Plan on deleting it from nodepool on Monday | 00:03 |
| *** thorst has joined #openstack-infra | 00:03 | |
| ianw | very good, hopefully it lasts this time :) | 00:03 |
| jeblair | pabelanger, notmorgan: i'm not sure statsd is the best thing for 328006 | 00:03 |
| notmorgan | jeblair: it may not be. | 00:04 |
| jeblair | i think that's the sort of thing we might want to expose in status.json or something | 00:04 |
| pabelanger | jeblair: ya, I was looking for a fast, easy way to look at the state of zuul-launcher. Specifically, if in shutdown mode. | 00:05 |
| pabelanger | Ya, I was thinking status.json could be another place for it | 00:05 |
| jeblair | we could have it send updates to zuul over gearman | 00:05 |
| jeblair | then zuul would (probably) know the status (most of the time) | 00:05 |
| pabelanger | interesting, didn't think of that | 00:05 |
| *** Swami has quit IRC | 00:05 | |
| pabelanger | Right, I am sure zookeeper will expose this info better also | 00:06 |
| notmorgan | hm. | 00:06 |
| jeblair | pabelanger: yes, zk will be great for tihs | 00:06 |
| *** tqtran has quit IRC | 00:06 | |
| notmorgan | jeblair: waht is the hard-timeout on job runtime? | 00:06 |
| pabelanger | and with that said: https://review.openstack.org/#/c/324037/ :D | 00:06 |
| notmorgan | jeblair: 90m? 120m? | 00:06 |
| jeblair | part of why i didn't worry about this too much is because i kind of want zl to be as invisible to users as zm | 00:06 |
| notmorgan | (python-3 jobs specifically) | 00:07 |
| jeblair | i think (hope) we're going to be far less interested in this once they're all up and running :) | 00:07 |
| pabelanger | jeblair: right, I'd be happy to think of zl as pets over cattle | 00:08 |
| jeblair | notmorgan: looks like python34 jobs hava a 40min timeout | 00:08 |
| notmorgan | haha yeah | 00:08 |
| notmorgan | looks like :P | 00:08 |
| pabelanger | err | 00:08 |
| notmorgan | just saw | 00:08 |
| pabelanger | cattle, then pets | 00:08 |
| notmorgan | oh BAH. i see the issue. a change got dropped causing everything to fail. | 00:08 |
| *** pvaneck has quit IRC | 00:09 | |
| jeblair | pabelanger: i don't know what people mean by that anymore (cattle are the ones you spend a lot of time worrying about because they're worth $5000 each and if they die your family starves) | 00:09 |
| EmilienM | damn, I never remember how to check if JJB was applied on jenkins nodes | 00:09 |
| pabelanger | jeblair: I always imagined the cattle was health, and off to the slaughter house if they are gone | 00:10 |
| pabelanger | healthy* | 00:10 |
| EmilienM | I used the puppet dashboard before but not sure it's still here | 00:10 |
| *** xarses has joined #openstack-infra | 00:10 | |
| jeblair | pabelanger: turns out you can't sell/eat sick cattle | 00:10 |
| jeblair | so you have to spend a lot of money on a vet to make sure they get better | 00:10 |
| pabelanger | that is true | 00:11 |
| mtreinish | jeblair: well that's what the fda wants us to think... | 00:11 |
| jeblair | mtreinish will eat sick cows | 00:11 |
| pabelanger | jeblair: are you planning on bringing zl01 back online tonight? Or holding out for something related to gearman? | 00:12 |
| *** mriedem has joined #openstack-infra | 00:12 | |
| EmilienM | zl01 is down? | 00:12 |
| pabelanger | yes, but zl02 is up | 00:12 |
| EmilienM | ah | 00:12 |
| pabelanger | EmilienM: http://grafana.openstack.org/dashboard/db/nodepool | 00:12 |
| jeblair | pabelanger: was going to wait til tomorrow so we can have a chance to sync up with AJaeger about the translation jobs before we increase the chances of them failing | 00:12 |
| EmilienM | I want to check if puppet CI is grabbing logs | 00:12 |
| EmilienM | pabelanger: thanks! | 00:12 |
| pabelanger | jeblair: sure, WFM | 00:13 |
| EmilienM | oh I found http://puppetdb.openstack.org/ | 00:13 |
| EmilienM | I remember we can check if puppet ran and if JJB was applied iirc | 00:13 |
| notmorgan | hmm. jeblair, mordred, pabelanger: should i squash down some of thes changes into the bundled change to help eliminate long py34 check jobs? | 00:17 |
| *** serverascode has quit IRC | 00:17 | |
| *** flaviof has quit IRC | 00:17 | |
| *** andrewbogott has quit IRC | 00:17 | |
| *** evgenyl has quit IRC | 00:17 | |
| *** DuncanT has quit IRC | 00:17 | |
| notmorgan | jeblair, pabelanger: it just comes down to being in the middle of the change and things starting to work "better"-ish... but it gets worse before better. | 00:17 |
| *** Ng has quit IRC | 00:17 | |
| *** dirk has quit IRC | 00:17 | |
| *** warpc has quit IRC | 00:17 | |
| *** andreykurilin__ has quit IRC | 00:17 | |
| *** zaro has quit IRC | 00:17 | |
| *** aimeeu has quit IRC | 00:17 | |
| *** iremizov has quit IRC | 00:17 | |
| *** jed56 has quit IRC | 00:17 | |
| *** briancurtin has quit IRC | 00:17 | |
| *** ggherdov- has quit IRC | 00:17 | |
| *** NobodyCam has quit IRC | 00:17 | |
| *** igorbelikov has quit IRC | 00:17 | |
| *** bryan_att has quit IRC | 00:17 | |
| *** sigmavirus24 has quit IRC | 00:17 | |
| *** flaviof has joined #openstack-infra | 00:17 | |
| *** erikmwilson has quit IRC | 00:18 | |
| *** nikhil has quit IRC | 00:18 | |
| *** patricku has quit IRC | 00:18 | |
| *** yonglihe has quit IRC | 00:18 | |
| *** petems has quit IRC | 00:18 | |
| *** zxiiro has quit IRC | 00:18 | |
| *** ogelbukh has quit IRC | 00:18 | |
| *** hrybacki has quit IRC | 00:18 | |
| *** scottda has quit IRC | 00:19 | |
| *** samyewell has quit IRC | 00:19 | |
| *** patrickeast has quit IRC | 00:19 | |
| *** MarkAtwood has quit IRC | 00:19 | |
| *** kgalanov has quit IRC | 00:19 | |
| *** fdegir has quit IRC | 00:19 | |
| *** lindsayk has quit IRC | 00:19 | |
| dmsimard | Zookeper ? Dear lord why ? | 00:19 |
| *** banix has joined #openstack-infra | 00:19 | |
| *** serverascode has joined #openstack-infra | 00:19 | |
| *** andrewbogott has joined #openstack-infra | 00:19 | |
| *** Ng has joined #openstack-infra | 00:20 | |
| *** yonglihe has joined #openstack-infra | 00:20 | |
| *** igorbelikov has joined #openstack-infra | 00:20 | |
| *** jed56 has joined #openstack-infra | 00:20 | |
| *** briancurtin has joined #openstack-infra | 00:20 | |
| *** amotoki has quit IRC | 00:20 | |
| *** bryan_att has joined #openstack-infra | 00:20 | |
| *** NobodyCam has joined #openstack-infra | 00:20 | |
| fungi | dmsimard: distributed state tracking | 00:20 |
| *** zxiiro has joined #openstack-infra | 00:20 | |
| *** DuncanT has joined #openstack-infra | 00:21 | |
| *** fdegir has joined #openstack-infra | 00:21 | |
| *** ogelbukh has joined #openstack-infra | 00:21 | |
| dmsimard | Oh, wait, I was thinking of Cassandra for some reason | 00:21 |
| *** andreykurilin__ has joined #openstack-infra | 00:21 | |
| *** ggherdov- has joined #openstack-infra | 00:21 | |
| *** hrybacki has joined #openstack-infra | 00:21 | |
| *** petems has joined #openstack-infra | 00:21 | |
| *** patricku has joined #openstack-infra | 00:21 | |
| *** nikhil has joined #openstack-infra | 00:21 | |
| fungi | yeah, not that. definitely not that | 00:21 |
| *** aimeeu has joined #openstack-infra | 00:22 | |
| *** patrickeast has joined #openstack-infra | 00:22 | |
| *** kgalanov has joined #openstack-infra | 00:22 | |
| *** MarkAtwood has joined #openstack-infra | 00:22 | |
| dmsimard | :D | 00:22 |
| *** scottda has joined #openstack-infra | 00:22 | |
| *** erikmwilson has joined #openstack-infra | 00:24 | |
| mgagne | how about mongodb :D | 00:25 |
| *** samyewell has joined #openstack-infra | 00:26 | |
| *** evgenyl has joined #openstack-infra | 00:26 | |
| *** lindsayk has joined #openstack-infra | 00:26 | |
| *** iremizov has joined #openstack-infra | 00:27 | |
| *** warpc has joined #openstack-infra | 00:27 | |
| *** dirk has joined #openstack-infra | 00:27 | |
| *** zaro has joined #openstack-infra | 00:27 | |
| *** sigmavirus24 has joined #openstack-infra | 00:28 | |
| *** sarob has joined #openstack-infra | 00:29 | |
| *** amitgandhinz has joined #openstack-infra | 00:29 | |
| anteaya | zaro: so if I go to my <vm-ip>:8080 I see my gerrit however it doesn't allow me to finish configuring the h2 database | 00:32 |
| *** zul_ has joined #openstack-infra | 00:34 | |
| *** zul_ has quit IRC | 00:34 | |
| *** amitgandhinz has quit IRC | 00:34 | |
| *** zigo has quit IRC | 00:37 | |
| EmilienM | jeblair: I think my stuff works http://logs.openstack.org/07/328007/1/check/gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-xenial/7b27770/console.html#_2016-06-10_00_34_02_383 | 00:38 |
| EmilienM | and we have logs | 00:38 |
| *** Jeffrey4l_ has quit IRC | 00:39 | |
| *** kzaitsev_mb has joined #openstack-infra | 00:40 | |
| *** amotoki has joined #openstack-infra | 00:41 | |
| *** LaT0rtue has joined #openstack-infra | 00:41 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config: Don't run group-based-policy DSVM jobs on kilo https://review.openstack.org/328011 | 00:42 |
| *** Jeffrey4l_ has joined #openstack-infra | 00:43 | |
| *** kzaitsev_mb has quit IRC | 00:46 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci https://review.openstack.org/324777 | 00:48 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Remove using a swap partition https://review.openstack.org/327094 | 00:48 |
| *** akshai_ has quit IRC | 00:48 | |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Farewell oslo-incubator https://review.openstack.org/320680 | 00:48 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: Python 3 Fixes: Bundled Fixes https://review.openstack.org/325661 | 00:49 |
| *** kzaitsev_mb has joined #openstack-infra | 00:49 | |
| *** mtanino has joined #openstack-infra | 00:50 | |
| *** ddieterly has joined #openstack-infra | 00:51 | |
| *** dimtruck is now known as zz_dimtruck | 00:52 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: Python 3 Fixes: Bundled Fixes https://review.openstack.org/325661 | 00:53 |
| *** amotoki has quit IRC | 00:55 | |
| *** tphummel has quit IRC | 00:57 | |
| *** tphummel has joined #openstack-infra | 00:59 | |
| *** baoli has joined #openstack-infra | 01:00 | |
| *** zigo has joined #openstack-infra | 01:03 | |
| *** bpokorny_ has quit IRC | 01:03 | |
| fungi | turned up a handful more wiki accounts who had created spam pages but hadn't uploaded spam images. adding them to the script removed another ~150 spam pages | 01:04 |
| *** sarob_ has joined #openstack-infra | 01:04 | |
| fungi | however, i'm starting to think (at least some of) these _may_ actually be accounts with weak passwords in the launchpad sso. for example, this one had a normal history of editing the wiki, but then earlier this week started adding spam https://wiki.openstack.org/wiki/Special:Contributions/Rockyg | 01:05 |
| anteaya | zaro: I guess the gerrit jar must contain an h2 db or something since it has created db's for itsefl | 01:06 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Renaming the Non-ATC Recognition WG The Non-ATC Recognition WG has changed its name to the AUC Recognition WG. This change updates the meeting information to reflect the new name and agenda location. https://review.openstack.org/327824 | 01:06 |
| anteaya | fungi: wow they are still at it | 01:07 |
| *** ayoung has joined #openstack-infra | 01:07 | |
| anteaya | I wonder if rockyg knows | 01:07 |
| *** gongysh has joined #openstack-infra | 01:07 | |
| *** bpokorny has joined #openstack-infra | 01:07 | |
| fungi | yeah, i think when they get lucky and guess the password of of an lp sso account that already has a corresponding non-blocked account in our wiki, then they start spamming with it (now that we've got new account creation disabled) | 01:07 |
| *** kzaitsev_mb has quit IRC | 01:08 | |
| *** sarob has quit IRC | 01:08 | |
| anteaya | and I guess that answers our question on if they are still trying to spam us | 01:08 |
| anteaya | yes they are | 01:08 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: WIP: Add web-based console log streaming https://review.openstack.org/320563 | 01:08 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: Python3 Fix: tools use six.moves.urllib https://review.openstack.org/327459 | 01:08 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: Python 3 fix: GC final state inspection https://review.openstack.org/327436 | 01:08 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Move kolla meeting to 16:00 UTC https://review.openstack.org/327845 | 01:09 |
| *** tonytan4ever has joined #openstack-infra | 01:10 | |
| anteaya | fungi: on the gerrit server what is the path to the project.config file that has a heading refs/meta/config? | 01:10 |
| anteaya | I am unable to locate it thus far on my instance | 01:11 |
| fungi | anteaya: there's not a direct file path | 01:11 |
| anteaya | oh | 01:11 |
| fungi | there's a bare All-Projects.git repository | 01:11 |
| fungi | and if you clone that and check out refs/meta/config then you can find the files you're looking for | 01:12 |
| *** sputnik13 has quit IRC | 01:12 | |
| fungi | and to update that file you have to push your comits for it it back into gerrit | 01:12 |
| anteaya | I will try | 01:12 |
| *** Apoorva has quit IRC | 01:13 | |
| fungi | anteaya: http://www.lowlevelmanager.com/2012/09/modify-submit-type-for-gerrit-project.html | 01:13 |
| anteaya | I got error: Trying to write ref HEAD with nonexistent object 0000000000000000000000000000000000000000 | 01:13 |
| anteaya | fatal: Cannot update the ref 'HEAD'. | 01:13 |
| anteaya | is this expected? | 01:14 |
| anteaya | apparently not I have no git repo | 01:14 |
| *** zz_dimtruck is now known as dimtruck | 01:15 | |
| anteaya | I think I am too tired for this now and will make another attempt tomorrow | 01:15 |
| anteaya | thanks fungi | 01:16 |
| *** chlong has joined #openstack-infra | 01:16 | |
| anteaya | fungi: are there any other of our wiki users that have had their accounts compromised? | 01:16 |
| fungi | i haven't spotted others yet | 01:16 |
| *** sarob has joined #openstack-infra | 01:16 | |
| *** sarob_ has quit IRC | 01:17 | |
| *** sarob_ has joined #openstack-infra | 01:17 | |
| *** Apoorva has joined #openstack-infra | 01:17 | |
| *** sarob_ has quit IRC | 01:17 | |
| anteaya | that's good | 01:18 |
| openstackgerrit | Merged openstack/diskimage-builder: Add cinder-backup mappings https://review.openstack.org/326411 | 01:20 |
| *** bpokorny has quit IRC | 01:20 | |
| *** sarob has quit IRC | 01:20 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Pre-install pip/virtualenv packages https://review.openstack.org/327472 | 01:21 |
| *** asselin_ has quit IRC | 01:22 | |
| *** bpokorny has joined #openstack-infra | 01:22 | |
| *** ddieterly is now known as ddieterly[away] | 01:24 | |
| *** ddieterly[away] is now known as ddieterly | 01:24 | |
| *** sdake has joined #openstack-infra | 01:25 | |
| *** Apoorva has quit IRC | 01:25 | |
| greghaynes | ianw: left you a comment on ^ | 01:26 |
| ianw | greghaynes: i think that whole thing is python3 unsafe | 01:27 |
| greghaynes | hah | 01:27 |
| *** anteaya has quit IRC | 01:27 | |
| greghaynes | ianw: so, is it just going to fail at image build time? I think we currently make fedora python3 images with this in infra? | 01:27 |
| *** akshai has joined #openstack-infra | 01:28 | |
| ianw | greghaynes: well, we have python3 packages installed, but as for pip & virtualenv interactions, i think that's unexplored territory really | 01:30 |
| ianw | python3 comes with pip, so we don't need to do that | 01:30 |
| *** zul has quit IRC | 01:30 | |
| *** amitgandhinz has joined #openstack-infra | 01:30 | |
| greghaynes | ianw: ya, I am mostly worried about this running before we have a 2.7 setuptools installed so it just exit 1's because that path doesnt exit | 01:31 |
| greghaynes | which would be a regression for some images we currently support | 01:31 |
| *** claudiub has quit IRC | 01:31 | |
| greghaynes | s/doesnt exit/doesnt exist | 01:32 |
| greghaynes | actually, its rm -rf | 01:32 |
| greghaynes | so itll exit 0 regardless | 01:32 |
| *** shashank_hegde has quit IRC | 01:32 | |
| greghaynes | ianw: maybe put in a todo about this is totally py3k unsafe? | 01:33 |
| ianw | greghaynes: it installs it above explicitly, although it's a dependency of the packaged pip | 01:33 |
| greghaynes | ianw: yea, but on a recent fedora that install will be to a different path, right? | 01:33 |
| *** amotoki has joined #openstack-infra | 01:33 | |
| ianw | greghaynes: i'm not sure it's so much python3 unsafe, as just not required | 01:34 |
| greghaynes | because that install will be doing a py3k version of those packages | 01:34 |
| notmorgan | jeblair, mordred, Shrews, pabelanger: looks like zuul was teetering on the limit of log size as it was, the py3 logs are just enough more to tip it over the edge. 3.0M vs 3.1M compressed. | 01:34 |
| ianw | greghaynes: no, that would be pip3 | 01:34 |
| notmorgan | jeblair, mordred, Shrews, pabelanger: looking to see what logging i can cut out safely. | 01:34 |
| notmorgan | any advice? | 01:35 |
| greghaynes | ianw: ah, ok | 01:35 |
| *** amotoki_ has joined #openstack-infra | 01:35 | |
| *** amitgandhinz has quit IRC | 01:35 | |
| *** csomerville has joined #openstack-infra | 01:37 | |
| *** amotoki has quit IRC | 01:38 | |
| notmorgan | jeblair, mordred, Shrews, pabelanger: I'm thinking gear.server can be safely turned off | 01:39 |
| *** zul has joined #openstack-infra | 01:42 | |
| EmilienM | jeblair: also confirmed on beaker jobs, all is green in our side | 01:42 |
| EmilienM | pabelanger: fyi ^ (about logs things) | 01:42 |
| *** kengo_sakai has quit IRC | 01:43 | |
| *** gouthamr_ has joined #openstack-infra | 01:44 | |
| ianw | greghaynes: yeah, i've converted the existing stderr prints in the element_dep script to logging for better unit testing ... will try to get something out with it all rebased this afternoon my time | 01:44 |
| ianw | (re the logging config thing) | 01:45 |
| greghaynes | ianw: oh, awesome | 01:45 |
| *** yamahata has joined #openstack-infra | 01:46 | |
| *** gouthamr has quit IRC | 01:47 | |
| *** ddieterly has quit IRC | 01:50 | |
| *** akshai has quit IRC | 01:52 | |
| *** PollyZ has joined #openstack-infra | 01:54 | |
| *** kushal has joined #openstack-infra | 01:54 | |
| *** r-mibu has quit IRC | 01:57 | |
| *** yamahata has quit IRC | 01:58 | |
| *** csomerville has quit IRC | 01:59 | |
| *** amotoki_ has quit IRC | 01:59 | |
| *** amotoki has joined #openstack-infra | 02:00 | |
| *** PollyZ has quit IRC | 02:00 | |
| *** ddieterly has joined #openstack-infra | 02:01 | |
| *** PollyZ has joined #openstack-infra | 02:05 | |
| *** mtanino has quit IRC | 02:05 | |
| *** mriedem has quit IRC | 02:08 | |
| *** yamamoto_ has joined #openstack-infra | 02:10 | |
| *** mtanino has joined #openstack-infra | 02:10 | |
| *** tkelsey has joined #openstack-infra | 02:11 | |
| *** r-mibu has joined #openstack-infra | 02:12 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: WIP: Add web-based console log streaming https://review.openstack.org/320563 | 02:12 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: Reduce Log Size https://review.openstack.org/328048 | 02:12 |
| *** PollyZ has quit IRC | 02:13 | |
| *** Goneri has joined #openstack-infra | 02:14 | |
| *** tkelsey has quit IRC | 02:15 | |
| *** PollyZ has joined #openstack-infra | 02:18 | |
| *** mdrabe has joined #openstack-infra | 02:19 | |
| *** ddieterly has quit IRC | 02:19 | |
| *** jamesmcarthur has joined #openstack-infra | 02:26 | |
| *** amotoki has quit IRC | 02:27 | |
| *** gongysh has quit IRC | 02:27 | |
| *** jamesmcarthur has quit IRC | 02:31 | |
| *** akshai has joined #openstack-infra | 02:32 | |
| *** yamamoto_ has quit IRC | 02:33 | |
| *** PollyZ has quit IRC | 02:35 | |
| *** dkehn has quit IRC | 02:37 | |
| *** yamamoto_ has joined #openstack-infra | 02:38 | |
| *** nadya has joined #openstack-infra | 02:41 | |
| *** ramishra has quit IRC | 02:42 | |
| *** tphummel has quit IRC | 02:43 | |
| *** cloudtrainme has joined #openstack-infra | 02:44 | |
| *** cloudtrainme has quit IRC | 02:49 | |
| *** yamahata has joined #openstack-infra | 02:49 | |
| *** yuanying has quit IRC | 02:51 | |
| *** PollyZ has joined #openstack-infra | 02:51 | |
| *** PollyZ has quit IRC | 02:55 | |
| *** tonytan4ever has quit IRC | 02:56 | |
| *** amotoki has joined #openstack-infra | 02:57 | |
| *** akshai has quit IRC | 03:04 | |
| *** amitgandhinz has joined #openstack-infra | 03:06 | |
| *** sflanigan_ is now known as sflanigan_afk | 03:07 | |
| *** mixos has quit IRC | 03:09 | |
| *** bpokorny has quit IRC | 03:10 | |
| *** nadya has quit IRC | 03:11 | |
| *** amitgandhinz has quit IRC | 03:11 | |
| *** NM has quit IRC | 03:13 | |
| *** NM has joined #openstack-infra | 03:13 | |
| *** markvoelker has quit IRC | 03:14 | |
| *** NM has quit IRC | 03:18 | |
| fungi | okay, that's now 313 spammers identified on the wiki, and 2014 of their pages deleted. enough for tonight but i'll try to track down some more tomorrow | 03:20 |
| *** csomerville has joined #openstack-infra | 03:20 | |
| *** lvdongbing has joined #openstack-infra | 03:21 | |
| fungi | i also deleted the spam created by Rocky's account and disabled it for a week to give us time to get up with her and make sure she updates her lp sso password | 03:21 |
| fungi | night, all | 03:22 |
| *** PollyZ has joined #openstack-infra | 03:23 | |
| *** jamesmcarthur has joined #openstack-infra | 03:27 | |
| *** gouthamr_ has quit IRC | 03:28 | |
| *** sflanigan_afk is now known as sflanigan | 03:29 | |
| *** banix has quit IRC | 03:30 | |
| *** csomerville has quit IRC | 03:31 | |
| *** jamesmcarthur has quit IRC | 03:32 | |
| *** flepied has joined #openstack-infra | 03:36 | |
| *** lvdongbing has quit IRC | 03:36 | |
| *** yamahata has quit IRC | 03:37 | |
| *** PollyZ has quit IRC | 03:38 | |
| *** yamahata has joined #openstack-infra | 03:38 | |
| *** nadya has joined #openstack-infra | 03:40 | |
| *** sdake has quit IRC | 03:41 | |
| *** PollyZ has joined #openstack-infra | 03:41 | |
| *** kushal has quit IRC | 03:41 | |
| *** lvdongbing has joined #openstack-infra | 03:42 | |
| *** yamamoto_ has quit IRC | 03:42 | |
| *** yamamoto_ has joined #openstack-infra | 03:43 | |
| *** dimtruck is now known as zz_dimtruck | 03:46 | |
| *** PollyZ has quit IRC | 03:46 | |
| *** yuanying has joined #openstack-infra | 03:48 | |
| *** jaosorior has joined #openstack-infra | 03:48 | |
| *** Goneri has quit IRC | 03:49 | |
| *** asettle has joined #openstack-infra | 03:50 | |
| *** thorst has quit IRC | 03:51 | |
| *** thorst has joined #openstack-infra | 03:51 | |
| *** nadya has quit IRC | 03:53 | |
| *** asettle has quit IRC | 03:55 | |
| *** lvdongbing has quit IRC | 03:57 | |
| *** mtanino has quit IRC | 03:58 | |
| *** itisha has quit IRC | 04:00 | |
| *** thorst has quit IRC | 04:00 | |
| *** baoli has quit IRC | 04:00 | |
| *** links has joined #openstack-infra | 04:00 | |
| *** lvdongbing has joined #openstack-infra | 04:01 | |
| *** pahuang has quit IRC | 04:02 | |
| *** flepied has quit IRC | 04:04 | |
| *** amitgandhinz has joined #openstack-infra | 04:07 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/project-config: Make Zuul python 3 job voting https://review.openstack.org/328066 | 04:12 |
| *** amitgandhinz has quit IRC | 04:12 | |
| *** tkelsey has joined #openstack-infra | 04:12 | |
| *** kushal has joined #openstack-infra | 04:13 | |
| *** lvdongbing has quit IRC | 04:14 | |
| *** pahuang has joined #openstack-infra | 04:15 | |
| *** tkelsey has quit IRC | 04:17 | |
| *** psachin has joined #openstack-infra | 04:18 | |
| *** lvdongbing has joined #openstack-infra | 04:20 | |
| *** yamamoto_ has quit IRC | 04:31 | |
| *** lvdongbing has quit IRC | 04:35 | |
| openstackgerrit | Steve Kowalik proposed openstack-infra/system-config: Update puppetlabs-mysql to 3.8.0 https://review.openstack.org/328068 | 04:36 |
| *** tlian has quit IRC | 04:37 | |
| *** M-docaedo_vector has joined #openstack-infra | 04:39 | |
| openstackgerrit | Mathieu Gagné proposed openstack-infra/zuul: Add ability to configure Zuul web interface with zuul.config.js https://review.openstack.org/327892 | 04:39 |
| *** lvdongbing has joined #openstack-infra | 04:40 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Remove using a swap partition https://review.openstack.org/327094 | 04:41 |
| *** kengo_sa_ has joined #openstack-infra | 04:44 | |
| *** sdake has joined #openstack-infra | 04:45 | |
| *** mdrabe has quit IRC | 04:47 | |
| *** aeng has quit IRC | 04:47 | |
| *** kushal has quit IRC | 04:47 | |
| *** aeng has joined #openstack-infra | 04:47 | |
| *** kengo_sa_ has quit IRC | 04:48 | |
| *** jamesmcarthur has joined #openstack-infra | 04:48 | |
| *** shashank_hegde has joined #openstack-infra | 04:50 | |
| *** sree has joined #openstack-infra | 04:52 | |
| *** jamesmcarthur has quit IRC | 04:53 | |
| *** lvdongbing has quit IRC | 04:54 | |
| *** Guest99301 is now known as fkautz | 04:54 | |
| *** fkautz is now known as Guest89652 | 04:54 | |
| *** Guest89652 is now known as fkautz | 04:55 | |
| *** fkautz has joined #openstack-infra | 04:56 | |
| *** sree has quit IRC | 04:56 | |
| *** thorst has joined #openstack-infra | 04:57 | |
| *** lvdongbing has joined #openstack-infra | 05:00 | |
| *** kengo_sa_ has joined #openstack-infra | 05:02 | |
| *** thorst has quit IRC | 05:04 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Clear up "already provided" message https://review.openstack.org/290968 | 05:05 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add python logger configuration https://review.openstack.org/325409 | 05:05 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Introspect logging testing more https://review.openstack.org/328071 | 05:05 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Convert element_dependencies to logging https://review.openstack.org/328072 | 05:05 |
| ianw | greghaynes: ^ bit of a logging stack | 05:05 |
| *** winggundamth has quit IRC | 05:06 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Clear up "already provided" message https://review.openstack.org/290968 | 05:07 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Convert element_dependencies to logging https://review.openstack.org/328072 | 05:07 |
| *** amitgandhinz has joined #openstack-infra | 05:08 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Farewell oslo-incubator https://review.openstack.org/320680 | 05:12 |
| *** amitgandhinz has quit IRC | 05:12 | |
| *** NM has joined #openstack-infra | 05:14 | |
| *** sree has joined #openstack-infra | 05:14 | |
| *** markvoelker has joined #openstack-infra | 05:15 | |
| *** lvdongbing has quit IRC | 05:15 | |
| *** yamamoto has joined #openstack-infra | 05:17 | |
| openstackgerrit | Merged openstack-infra/project-config: Add the devstack-logs publisher to the ARA jobs https://review.openstack.org/327943 | 05:18 |
| *** NM has quit IRC | 05:19 | |
| *** asettle has joined #openstack-infra | 05:19 | |
| *** markvoelker has quit IRC | 05:20 | |
| *** lvdongbing has joined #openstack-infra | 05:20 | |
| openstackgerrit | Anusha Ramineni proposed openstack-infra/project-config: congress: Make new arch jobs voting https://review.openstack.org/327514 | 05:24 |
| *** asettle has quit IRC | 05:24 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Use dlrn instead of delorean for delorean-build command https://review.openstack.org/325823 | 05:26 |
| *** flepied has joined #openstack-infra | 05:26 | |
| *** mohankumar_ has joined #openstack-infra | 05:26 | |
| *** javeriak has joined #openstack-infra | 05:28 | |
| *** salv-orlando has joined #openstack-infra | 05:28 | |
| *** mhickey has joined #openstack-infra | 05:29 | |
| *** jamielennox is now known as jamielennox|away | 05:29 | |
| *** salv-orlando has quit IRC | 05:31 | |
| *** salv-orl_ has quit IRC | 05:31 | |
| *** salv-orlando has joined #openstack-infra | 05:31 | |
| *** lvdongbing has quit IRC | 05:34 | |
| *** shashank_hegde has quit IRC | 05:39 | |
| *** lvdongbing has joined #openstack-infra | 05:40 | |
| *** shashank_hegde has joined #openstack-infra | 05:41 | |
| *** gyee has quit IRC | 05:45 | |
| *** gongysh has joined #openstack-infra | 05:46 | |
| *** winggundamth has joined #openstack-infra | 05:48 | |
| *** kengo_sa_ has quit IRC | 05:51 | |
| *** dstufft has quit IRC | 05:52 | |
| *** dstufft has joined #openstack-infra | 05:53 | |
| *** lvdongbing has quit IRC | 05:55 | |
| *** javeriak has quit IRC | 05:55 | |
| *** aeng has quit IRC | 05:56 | |
| AJaeger | jroll, dtantsur|afk : Could you review https://review.openstack.org/#/c/326059/ , please? | 05:56 |
| AJaeger | jhesketh: your change https://review.openstack.org/#/c/198728/ is untouched since 6 months. The dependend job merged. Do you want to finish this off - or abandon it? | 05:58 |
| *** asettle has joined #openstack-infra | 05:59 | |
| jhesketh | AJaeger: Haven't really touched osloganalyze in a while.. can't hurt to have extra testing though so I'll fix it up | 05:59 |
| *** amitgandhinz has joined #openstack-infra | 06:00 | |
| AJaeger | thanks, jhesketh | 06:02 |
| *** thorst has joined #openstack-infra | 06:02 | |
| AJaeger | ianw, do you think persuing https://review.openstack.org/#/c/219351 is worthwhile - or should we abandon it since there does not seem interest in it? | 06:02 |
| *** chlong has quit IRC | 06:03 | |
| *** asettle has quit IRC | 06:03 | |
| *** amitgandhinz has quit IRC | 06:04 | |
| *** yolanda has joined #openstack-infra | 06:06 | |
| *** salv-orlando has quit IRC | 06:06 | |
| *** jamesmcarthur has joined #openstack-infra | 06:06 | |
| *** salv-orlando has joined #openstack-infra | 06:07 | |
| *** jamesmcarthur has quit IRC | 06:07 | |
| yolanda | good morning | 06:07 |
| *** jamesmcarthur has joined #openstack-infra | 06:08 | |
| *** thorst has quit IRC | 06:10 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Use keystoneauth.betamax for shade mocks https://review.openstack.org/298647 | 06:11 |
| *** claudiub has joined #openstack-infra | 06:11 | |
| *** jamesmcarthur has quit IRC | 06:12 | |
| AJaeger | morning, yolanda ! | 06:15 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/328091 | 06:15 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-jeepyb: Require packages before installing jeepyb https://review.openstack.org/327707 | 06:17 |
| *** lunarlamp has joined #openstack-infra | 06:17 | |
| *** zeih has joined #openstack-infra | 06:18 | |
| AJaeger | infra-root, we have UTF-8 artefacts in translation import. Is the locale not properly setup on the proposal slave? Example: https://review.openstack.org/#/c/328085/1/searchlight/locale/de/LC_MESSAGES/searchlight.po | 06:20 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Test osloganalyze in devstack https://review.openstack.org/198728 | 06:20 |
| *** chlong has joined #openstack-infra | 06:20 | |
| AJaeger | yolanda: I saw jeblair say something about locales and translations, let him handle the above ^ | 06:20 |
| yolanda | AJaeger, ok, i'm not aware of the problem | 06:20 |
| SpamapS | oy.. building a diskimage in a virtualbox VM is a major drag :-P | 06:21 |
| AJaeger | yolanda: just noticed something in backlog... | 06:21 |
| AJaeger | yolanda, jeblair : Just check https://review.openstack.org/#/c/328083/1/horizon_cisco_ui/locale/ja/LC_MESSAGES/django.po ;( | 06:22 |
| * AJaeger is going to -1 all of today's translation imports | 06:22 | |
| yolanda | AJaeger, i checked in proposal slave an our locale is set to en_US.UTF-8 | 06:24 |
| yolanda | did it start to fail today? | 06:27 |
| AJaeger | yolanda: jeblair said at 20:24 " we need to land 327921 for zlstatic01 -- the locale file on proposal.slave has a comment in it" | 06:27 |
| AJaeger | apparently it's not working as it should | 06:28 |
| ianw | AJaeger: I think that job is gone, i'd abandon | 06:28 |
| openstackgerrit | Merged openstack-infra/project-config: Farewell oslo-incubator https://review.openstack.org/320680 | 06:28 |
| AJaeger | ianw: will do -thanks | 06:29 |
| *** andreas_s has joined #openstack-infra | 06:30 | |
| AJaeger | #status log All translation imports have broken UTF-8 encoding. | 06:31 |
| openstackstatus | AJaeger: finished logging | 06:31 |
| yolanda | AJaeger, i see. Let's wait for other cores to show up. I'm not familiar with new zuul 2.5 situation | 06:31 |
| *** zeih has quit IRC | 06:31 | |
| openstackgerrit | Merged openstack-infra/project-config: Extend regexp to really include https://review.openstack.org/327942 | 06:32 |
| AJaeger | yolanda: yes, lets wait for pabelanger and jeblair - I probably should have addressed them directly instead of infra-root. | 06:32 |
| *** openstackgerrit has quit IRC | 06:32 | |
| *** openstackgerrit has joined #openstack-infra | 06:33 | |
| *** chlong has quit IRC | 06:35 | |
| AJaeger | yolanda: could you review https://review.openstack.org/#/c/325873/ as well, please? | 06:38 |
| yolanda | sure | 06:38 |
| AJaeger | thanks | 06:39 |
| *** pcaruana has joined #openstack-infra | 06:40 | |
| *** bhavik1 has joined #openstack-infra | 06:43 | |
| *** shashank_hegde has quit IRC | 06:43 | |
| openstackgerrit | Merged openstack-infra/project-config: Add stable/mitaka to tacker gerritbot https://review.openstack.org/327991 | 06:47 |
| openstackgerrit | Merged openstack-infra/project-config: Add more oslo libs to the run-tox-with-oslo-master script https://review.openstack.org/327999 | 06:48 |
| openstackgerrit | Merged openstack-infra/project-config: Don't run group-based-policy DSVM jobs on kilo https://review.openstack.org/328011 | 06:48 |
| *** javeriak has joined #openstack-infra | 06:48 | |
| openstackgerrit | Merged openstack-infra/system-config: Normalize maillist indentation https://review.openstack.org/327698 | 06:49 |
| *** salv-orlando has quit IRC | 06:52 | |
| *** salv-orlando has joined #openstack-infra | 06:53 | |
| *** flepied has quit IRC | 06:53 | |
| *** claudiub|2 has joined #openstack-infra | 06:54 | |
| *** links has quit IRC | 06:55 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/328091 | 06:56 |
| openstackgerrit | Merged openstack-infra/project-config: Clean-up kilo and juno configs for murano jobs https://review.openstack.org/327702 | 06:56 |
| openstackgerrit | Merged openstack-infra/project-config: pymod2pkg: Add testing for Python 3.x https://review.openstack.org/325873 | 06:56 |
| *** claudiub has quit IRC | 06:57 | |
| *** kengo_sa_ has joined #openstack-infra | 06:58 | |
| *** hparekh has quit IRC | 06:58 | |
| *** xarses has quit IRC | 06:59 | |
| *** xarses has joined #openstack-infra | 06:59 | |
| *** hparekh has joined #openstack-infra | 07:02 | |
| openstackgerrit | Merged openstack-infra/project-config: make all sahara-dashboard jobs voting https://review.openstack.org/326474 | 07:03 |
| openstackgerrit | Merged openstack-infra/project-config: Enable StoryBoard for the ARA project https://review.openstack.org/327693 | 07:04 |
| *** vincentll has joined #openstack-infra | 07:06 | |
| *** kengo_sa_ has quit IRC | 07:06 | |
| ttx | lifeless: am around now | 07:07 |
| *** thorst has joined #openstack-infra | 07:07 | |
| *** esikachev has joined #openstack-infra | 07:09 | |
| *** links has joined #openstack-infra | 07:10 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Test osloganalyze in devstack https://review.openstack.org/198728 | 07:10 |
| *** camunoz has quit IRC | 07:11 | |
| *** ccamacho has left #openstack-infra | 07:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Add pypi-jobs to tripleo-validations https://review.openstack.org/327611 | 07:14 |
| openstackgerrit | Merged openstack-infra/project-config: Add watcher-dashboard doc to docs.openstack.org https://review.openstack.org/327507 | 07:14 |
| openstackgerrit | Merged openstack-infra/project-config: Simplify gate tests for 'intel-nfv-ci-tests' https://review.openstack.org/326989 | 07:14 |
| openstackgerrit | Merged openstack-infra/project-config: Add Puppet Watcher repo to gerritbot https://review.openstack.org/325799 | 07:14 |
| *** tkelsey has joined #openstack-infra | 07:14 | |
| *** sree has quit IRC | 07:14 | |
| AJaeger | yolanda: project-config queue is getting shorter now, thanks also to your reviews! We're at 122 open changes now. | 07:15 |
| *** sree has joined #openstack-infra | 07:15 | |
| *** permalac has joined #openstack-infra | 07:15 | |
| *** thorst has quit IRC | 07:15 | |
| *** NM has joined #openstack-infra | 07:15 | |
| yolanda | AJaeger, nice. Let me know if there is any urgent review | 07:15 |
| AJaeger | yolanda: nothing urgent that I'm aware off ;) thanks | 07:16 |
| *** markvoelker has joined #openstack-infra | 07:16 | |
| *** rcernin has joined #openstack-infra | 07:17 | |
| *** ccamacho has joined #openstack-infra | 07:18 | |
| *** tkelsey has quit IRC | 07:19 | |
| *** NM has quit IRC | 07:20 | |
| *** markvoelker has quit IRC | 07:21 | |
| *** asettle has joined #openstack-infra | 07:21 | |
| *** amoralej|off is now known as amoralej | 07:23 | |
| *** xarses has quit IRC | 07:23 | |
| *** xarses_ has joined #openstack-infra | 07:24 | |
| *** xarses_ has quit IRC | 07:24 | |
| *** xarses_ has joined #openstack-infra | 07:24 | |
| *** dtantsur|afk is now known as dtantsur | 07:25 | |
| dtantsur | AJaeger, yeah, this patch looks incomplete to me | 07:25 |
| dtantsur | AJaeger, yolanda, also, does anything prevents you from approving https://review.openstack.org/#/c/319336/ ? | 07:26 |
| AJaeger | thanks, dtantsur for checking. | 07:26 |
| AJaeger | dtantsur: give it a +1 and I'll +A. Deal? ;) | 07:27 |
| yolanda | AJaeger, no, actually i thought i did | 07:27 |
| *** tesseract has joined #openstack-infra | 07:27 | |
| AJaeger | ;) | 07:27 |
| *** arxcruz has joined #openstack-infra | 07:28 | |
| *** gongysh has quit IRC | 07:28 | |
| *** yamada-h has joined #openstack-infra | 07:29 | |
| AJaeger | dtantsur: do you know what's going on with https://review.openstack.org/312596 - could you ping the author, please? | 07:29 |
| dtantsur | AJaeger, we definitely need this pretty soon. I'll ping Lucas when he's online (in an hour or so) | 07:30 |
| dtantsur | thanks for reminder | 07:30 |
| *** rossella_s has joined #openstack-infra | 07:32 | |
| *** jlanoux has joined #openstack-infra | 07:32 | |
| *** mhickey has quit IRC | 07:32 | |
| *** openstackgerrit has quit IRC | 07:33 | |
| AJaeger | ok | 07:33 |
| *** openstackgerrit has joined #openstack-infra | 07:33 | |
| *** asettle has quit IRC | 07:34 | |
| *** sflanigan has quit IRC | 07:34 | |
| *** bauzas is now known as bauwser | 07:35 | |
| *** jamielennox|away is now known as jamielennox | 07:36 | |
| *** zeih has joined #openstack-infra | 07:37 | |
| openstackgerrit | Merged openstack-infra/project-config: Make grenade-dsvm-ironic non voting https://review.openstack.org/319336 | 07:38 |
| *** zeih has quit IRC | 07:38 | |
| *** zeih has joined #openstack-infra | 07:38 | |
| *** hashar has joined #openstack-infra | 07:39 | |
| *** dtardivel has joined #openstack-infra | 07:40 | |
| *** tkelsey has joined #openstack-infra | 07:44 | |
| *** dizquierdo has joined #openstack-infra | 07:47 | |
| *** yamamoto has quit IRC | 07:50 | |
| *** nt has quit IRC | 07:51 | |
| AJaeger | jeblair: I propose to abandon all open translation imports. | 07:51 |
| *** nt has joined #openstack-infra | 07:54 | |
| *** ihrachys has joined #openstack-infra | 07:54 | |
| *** lvdongbing has joined #openstack-infra | 07:55 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** jtomasek_ has joined #openstack-infra | 08:03 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Set LC_ALL for translation jobs https://review.openstack.org/328145 | 08:03 |
| *** shardy has joined #openstack-infra | 08:03 | |
| *** yamamoto has joined #openstack-infra | 08:04 | |
| AJaeger | jeblair: let's make the script more robust ^ | 08:04 |
| *** e0ne has joined #openstack-infra | 08:05 | |
| openstackgerrit | YAMADA Hideki proposed openstack-infra/project-config: Add Sheepdog test job to glance_store https://review.openstack.org/328148 | 08:06 |
| *** lezbar__ has joined #openstack-infra | 08:06 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Set locale for translation jobs https://review.openstack.org/328145 | 08:07 |
| *** lvdongbing has quit IRC | 08:07 | |
| vincentll | is there a way to have more detailed ouput when a py27 checks times out? | 08:08 |
| vincentll | note: the same test on my laptop is done in < 1 s | 08:08 |
| *** lezbar has quit IRC | 08:09 | |
| AJaeger | vincentll: not more than what's in the log files | 08:10 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Ensure extracted log file are readable https://review.openstack.org/327053 | 08:10 |
| vincentll | AJaeger: ok thanks | 08:10 |
| *** bhavik1 has quit IRC | 08:11 | |
| *** e0ne has quit IRC | 08:12 | |
| *** e0ne has joined #openstack-infra | 08:12 | |
| *** thorst has joined #openstack-infra | 08:13 | |
| *** lvdongbing has joined #openstack-infra | 08:15 | |
| *** lezbar__ has quit IRC | 08:16 | |
| yamada-h | Hi, I want to add sheepdog job to glance_store, mentioned in this mail http://lists.openstack.org/pipermail/openstack-dev/2015-December/081966.html | 08:16 |
| *** markvoelker has joined #openstack-infra | 08:17 | |
| yamada-h | is this patch correct? https://review.openstack.org/328148 | 08:17 |
| *** Ravikiran_K has joined #openstack-infra | 08:18 | |
| AJaeger | yamada-h: please have patience, it will take a few days for review. If you add jobs to glance_store, ask the glance PTL to review the change as well... | 08:19 |
| *** lezbar has joined #openstack-infra | 08:19 | |
| yamada-h | AJaeger: OK, thanks! | 08:20 |
| *** jtomasek_ has quit IRC | 08:20 | |
| *** thorst has quit IRC | 08:20 | |
| AJaeger | yamada-h: I did a quick review - best discuss next with glance team... | 08:21 |
| *** markvoelker has quit IRC | 08:21 | |
| *** lvdongbing has quit IRC | 08:21 | |
| *** sdake has quit IRC | 08:21 | |
| *** sorantis has joined #openstack-infra | 08:22 | |
| *** javeriak has quit IRC | 08:22 | |
| yamada-h | Alright. | 08:22 |
| *** Hal has joined #openstack-infra | 08:24 | |
| *** javeriak has joined #openstack-infra | 08:25 | |
| *** Hal is now known as Guest39657 | 08:25 | |
| *** sorantis_ has joined #openstack-infra | 08:28 | |
| *** woodster_ has quit IRC | 08:28 | |
| AJaeger | jeblair: once again a post job failure with no logs: http://logs.openstack.org/eb/ebb418fb68b74b4b2d7b4bc5dd79a1129e33b9ab/post/tacker-docs/fde616d/ | 08:29 |
| *** jaosorior has quit IRC | 08:30 | |
| *** yamamoto has quit IRC | 08:30 | |
| *** sorantis has quit IRC | 08:30 | |
| *** sorantis_ is now known as sorantis | 08:30 | |
| *** jaosorior has joined #openstack-infra | 08:30 | |
| *** yamamoto has joined #openstack-infra | 08:31 | |
| openstackgerrit | Thomas Bechtold proposed openstack/diskimage-builder: Remove discover from test-requirements https://review.openstack.org/328170 | 08:34 |
| *** ifarkas has joined #openstack-infra | 08:37 | |
| *** javeriak has quit IRC | 08:41 | |
| *** javeriak has joined #openstack-infra | 08:41 | |
| *** dizquierdo has quit IRC | 08:42 | |
| *** _shaps_ has joined #openstack-infra | 08:42 | |
| *** javeriak has quit IRC | 08:46 | |
| vincentll | How do I specify that my package depends on nova (not novaclient) since nova doesn't seem to be on PyPI? | 08:46 |
| *** maishsk has quit IRC | 08:46 | |
| openstackgerrit | Will Foster proposed openstack-infra/project-config: recheck https://review.openstack.org/328173 | 08:46 |
| *** javeriak has joined #openstack-infra | 08:46 | |
| *** tzn has joined #openstack-infra | 08:49 | |
| *** asettle has joined #openstack-infra | 08:52 | |
| *** asettle has quit IRC | 08:52 | |
| *** asettle has joined #openstack-infra | 08:53 | |
| *** markusry has joined #openstack-infra | 08:53 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/project-config: Make zuul_swift_upload.py more accurate for mime type detection https://review.openstack.org/328178 | 08:55 |
| *** daemontool has joined #openstack-infra | 08:59 | |
| *** maishsk has joined #openstack-infra | 09:02 | |
| *** nwkarsten has joined #openstack-infra | 09:08 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: DNM timeout debugging https://review.openstack.org/328183 | 09:12 |
| *** nwkarsten has quit IRC | 09:13 | |
| *** tosky has joined #openstack-infra | 09:15 | |
| openstackgerrit | Vladyslav Drok proposed openstack-infra/project-config: Allow to test agent drivers with partition images https://review.openstack.org/326059 | 09:17 |
| *** thorst has joined #openstack-infra | 09:18 | |
| *** _degorenko|afk is now known as degorenko | 09:19 | |
| *** javeriak has quit IRC | 09:25 | |
| *** thorst has quit IRC | 09:25 | |
| *** dizquierdo has joined #openstack-infra | 09:27 | |
| AJaeger | vincentll: devstack will setup nova for you directly - and for other tests, you shouldn't need to use nova installed. | 09:27 |
| *** sree has quit IRC | 09:29 | |
| openstackgerrit | Vladyslav Drok proposed openstack-infra/devstack-gate: Set ephemeral size requested in the ironic job https://review.openstack.org/326061 | 09:29 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/ansible-role-puppet: Syslog puppet output on Ansible control machine https://review.openstack.org/327789 | 09:30 |
| vincentll | AJaeger: is devstack setup by default for gate-synergy-service-python27? | 09:30 |
| vincentll | I meant gate-*-python27 | 09:31 |
| rcarrillocruz | clarkb , mordred : still in EMEA times? | 09:32 |
| rcarrillocruz | https://review.openstack.org/#/c/327789/ | 09:32 |
| rcarrillocruz | which is about story https://storyboard.openstack.org/#!/story/2000637 | 09:32 |
| *** abregman has joined #openstack-infra | 09:35 | |
| *** abregman is now known as abregman|afk | 09:35 | |
| AJaeger | vincentll: not for python27 jobs - those jobs should not need nova at all | 09:36 |
| vincentll | ok | 09:37 |
| AJaeger | vincentll: if you python27 job needs nova, I suspect you do something wrong. Best talk on #openstack-qa how to setup tests best | 09:37 |
| *** thorst has joined #openstack-infra | 09:37 | |
| AJaeger | bbl | 09:38 |
| vincentll | AJaeger: thanks for the info | 09:42 |
| *** dtantsur is now known as dtantsur|lunch | 09:42 | |
| *** javeriak has joined #openstack-infra | 09:43 | |
| *** tzn has quit IRC | 09:44 | |
| *** sorantis has quit IRC | 09:46 | |
| *** sorantis has joined #openstack-infra | 09:47 | |
| *** sorantis has quit IRC | 09:48 | |
| mordred | rcarrillocruz: looking | 09:48 |
| *** yamada-h has quit IRC | 09:53 | |
| *** acoles has joined #openstack-infra | 09:54 | |
| *** yamahata has quit IRC | 09:56 | |
| *** thorst has quit IRC | 09:56 | |
| *** thorst has joined #openstack-infra | 09:57 | |
| *** thorst has quit IRC | 10:01 | |
| openstackgerrit | Alistair Coles proposed openstack-infra/project-config: Add feature/crypto-review events to openstack-swift https://review.openstack.org/328219 | 10:03 |
| *** lezbar has quit IRC | 10:05 | |
| *** yamamoto has quit IRC | 10:06 | |
| *** pahuang has quit IRC | 10:10 | |
| *** lezbar has joined #openstack-infra | 10:10 | |
| *** ZZelle has quit IRC | 10:11 | |
| *** ZZelle has joined #openstack-infra | 10:11 | |
| *** daemontool has quit IRC | 10:17 | |
| *** gongysh has joined #openstack-infra | 10:17 | |
| *** markvoelker has joined #openstack-infra | 10:18 | |
| openstackgerrit | Pawel Brzozowski proposed openstack-infra/project-config: Added new repos for fuel plugins https://review.openstack.org/328223 | 10:19 |
| *** HeOS has quit IRC | 10:19 | |
| *** gongysh has quit IRC | 10:22 | |
| openstackgerrit | Vladyslav Drok proposed openstack-infra/project-config: Allow to test agent drivers with partition images https://review.openstack.org/326059 | 10:22 |
| *** markvoelker has quit IRC | 10:22 | |
| *** pahuang has joined #openstack-infra | 10:22 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-openstackci: Add puppetdb class https://review.openstack.org/328228 | 10:31 |
| *** daemontool has joined #openstack-infra | 10:38 | |
| shardy | Hi all, did anything change in the last hour or so related to DEVSTACK_GATE_TIMEOUT? | 10:39 |
| shardy | I've seen some tripleo jobs run with a (wrong) timeout of 93 minutes, but then I ran another test and it's back to the (right) value of 159 minutes | 10:40 |
| shardy | ref https://bugs.launchpad.net/tripleo/+bug/1591102 | 10:40 |
| openstack | Launchpad bug 1591102 in tripleo "nonha job faiing due to wrong timeout of 20 mins" [Critical,Triaged] | 10:40 |
| * shardy couldn't see anything obvious in recent git commits | 10:41 | |
| openstackgerrit | Joe Talerico proposed openstack-infra/project-config: Adding Browbeat https://review.openstack.org/327255 | 10:41 |
| *** Guest39657 has quit IRC | 10:46 | |
| *** sorantis has joined #openstack-infra | 10:48 | |
| *** sdague has joined #openstack-infra | 10:50 | |
| *** yamada-h has joined #openstack-infra | 10:54 | |
| *** dtantsur|lunch is now known as dtantsur | 10:58 | |
| *** yamada-h has quit IRC | 10:58 | |
| *** abregman|afk has quit IRC | 10:59 | |
| openstackgerrit | Joe Talerico proposed openstack-infra/project-config: Adding Browbeat https://review.openstack.org/327255 | 11:01 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config: Run tests from sahara-tests also for tempest layer4 job https://review.openstack.org/328239 | 11:02 |
| *** kengo_sa_ has joined #openstack-infra | 11:04 | |
| *** ddieterly has joined #openstack-infra | 11:05 | |
| *** kengo_sa_ has quit IRC | 11:09 | |
| *** dmk0202 has joined #openstack-infra | 11:10 | |
| *** salv-orlando has quit IRC | 11:13 | |
| *** salv-orlando has joined #openstack-infra | 11:14 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-openstackci: Add puppetdb class https://review.openstack.org/328228 | 11:17 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Add tempest-dsvm-ironic-pxe_ipa-full-nv to Ironic https://review.openstack.org/328245 | 11:21 |
| *** rossella_s has quit IRC | 11:22 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:22 | |
| *** dmk0202 has quit IRC | 11:23 | |
| *** esikachev has quit IRC | 11:24 | |
| *** jcoufal has joined #openstack-infra | 11:26 | |
| *** HeOS has joined #openstack-infra | 11:26 | |
| *** daemontool has quit IRC | 11:28 | |
| *** deadnull_ has joined #openstack-infra | 11:28 | |
| *** daemontool has joined #openstack-infra | 11:28 | |
| *** salv-orl_ has joined #openstack-infra | 11:28 | |
| *** ccamacho is now known as ccamacho|lunch | 11:29 | |
| *** rhallisey has joined #openstack-infra | 11:29 | |
| *** salv-orlando has quit IRC | 11:31 | |
| *** javeriak has quit IRC | 11:31 | |
| *** javeriak has joined #openstack-infra | 11:32 | |
| *** andreykurilin has left #openstack-infra | 11:35 | |
| *** pahuang has quit IRC | 11:35 | |
| *** dmk0202 has joined #openstack-infra | 11:36 | |
| *** javeriak_ has joined #openstack-infra | 11:40 | |
| *** javeriak has quit IRC | 11:40 | |
| *** baoli has joined #openstack-infra | 11:41 | |
| openstackgerrit | Illia Khudoshyn proposed openstack-infra/project-config: Add Rally job running Rally against deployment with existing users https://review.openstack.org/328249 | 11:41 |
| *** baoli_ has joined #openstack-infra | 11:42 | |
| openstackgerrit | Illia Khudoshyn proposed openstack-infra/project-config: Add Rally job against deployment with existing users https://review.openstack.org/328249 | 11:44 |
| *** dprince has joined #openstack-infra | 11:44 | |
| *** baoli has quit IRC | 11:46 | |
| jroll | AJaeger: reviewed | 11:46 |
| *** fergal_ has joined #openstack-infra | 11:47 | |
| *** ddieterly has quit IRC | 11:47 | |
| jroll | clarkb: to answer your OS_TEST_TIMEOUT question, I think OS_TEST_TIMEOUT gets passed through grenade, into tempest via tox, and then either https://github.com/openstack/tempest/blob/master/.testr.conf#L4 or https://github.com/openstack/tempest/blob/master/tox.ini#L19 brings that in (maybe both?) | 11:48 |
| *** esikachev has joined #openstack-infra | 11:51 | |
| *** pcaruana has quit IRC | 11:51 | |
| *** mohankumar_ has quit IRC | 11:52 | |
| *** flepied has joined #openstack-infra | 11:55 | |
| *** rodrigods has quit IRC | 11:55 | |
| *** rodrigods has joined #openstack-infra | 11:56 | |
| *** EricGonczer_ has joined #openstack-infra | 11:57 | |
| *** maishsk has quit IRC | 11:58 | |
| *** NM has joined #openstack-infra | 12:01 | |
| *** kengo_sa_ has joined #openstack-infra | 12:05 | |
| jroll | clarkb: oh, and apologies if I missed your question, sometimes I bounce between channels too quickly and if I don't get a highlight I'm prone to missing it :/ | 12:06 |
| *** pcaruana has joined #openstack-infra | 12:07 | |
| *** kengo_sakai has joined #openstack-infra | 12:08 | |
| *** kengo_sa_ has quit IRC | 12:10 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci https://review.openstack.org/324777 | 12:10 |
| *** kengo_sakai has quit IRC | 12:12 | |
| *** ldnunes has joined #openstack-infra | 12:12 | |
| *** pcaruana has quit IRC | 12:13 | |
| *** fergal_ has quit IRC | 12:13 | |
| *** pcaruana has joined #openstack-infra | 12:13 | |
| *** aysyd has joined #openstack-infra | 12:14 | |
| *** ayoung has quit IRC | 12:15 | |
| *** aysyd has quit IRC | 12:16 | |
| *** maishsk has joined #openstack-infra | 12:17 | |
| *** vgridnev has joined #openstack-infra | 12:17 | |
| *** weshay has joined #openstack-infra | 12:18 | |
| *** markvoelker has joined #openstack-infra | 12:18 | |
| *** aysyd has joined #openstack-infra | 12:19 | |
| *** ddieterly has joined #openstack-infra | 12:20 | |
| *** nwkarsten has joined #openstack-infra | 12:21 | |
| *** markusry has quit IRC | 12:21 | |
| *** afred312 has quit IRC | 12:23 | |
| vgridnev | how I can found description of nodes that we have in infra? For example, I can see ubuntu-trusty or centos-7 in job descriptions, where them are defined? | 12:24 |
| *** dmk0202 has quit IRC | 12:25 | |
| vgridnev | and also interested in builder install-distro-packages definition | 12:25 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Allow superusers to edit other users successfully https://review.openstack.org/325196 | 12:26 |
| *** ddieterly has quit IRC | 12:27 | |
| *** xyang1 has joined #openstack-infra | 12:29 | |
| *** tpsilva has joined #openstack-infra | 12:30 | |
| openstackgerrit | Merged openstack-infra/project-config: Make grenade-dsvm-ironic -nv on grenade https://review.openstack.org/327985 | 12:30 |
| *** javeriak_ has quit IRC | 12:33 | |
| *** psilvad has joined #openstack-infra | 12:36 | |
| *** dmsimard is now known as dmsimard|afk | 12:37 | |
| *** gordc has joined #openstack-infra | 12:37 | |
| *** dizquierdo has quit IRC | 12:37 | |
| *** ddieterly has joined #openstack-infra | 12:37 | |
| *** dmk0202 has joined #openstack-infra | 12:39 | |
| *** ddieterly is now known as ddieterly[away] | 12:41 | |
| *** maishsk_ has joined #openstack-infra | 12:44 | |
| *** psilvad has quit IRC | 12:44 | |
| *** tzn has joined #openstack-infra | 12:45 | |
| *** maishsk has quit IRC | 12:45 | |
| *** maishsk_ is now known as maishsk | 12:45 | |
| *** infra-red has joined #openstack-infra | 12:45 | |
| *** EricGonczer_ has quit IRC | 12:47 | |
| *** rlandy has joined #openstack-infra | 12:48 | |
| *** alaski is now known as lascii | 12:48 | |
| *** tlian has joined #openstack-infra | 12:48 | |
| *** esikachev has quit IRC | 12:49 | |
| *** tzn has quit IRC | 12:49 | |
| *** njohnston|afk is now known as njohnston | 12:50 | |
| *** psilvad has joined #openstack-infra | 12:51 | |
| *** ccamacho|lunch is now known as ccamacho | 12:52 | |
| krotscheck | pabelanger: Did the xenial build finally work, or... | 12:53 |
| *** links has quit IRC | 12:54 | |
| *** infra-red has quit IRC | 12:54 | |
| *** matt-borland has joined #openstack-infra | 12:55 | |
| *** markusry has joined #openstack-infra | 12:55 | |
| *** qba73 has joined #openstack-infra | 12:55 | |
| *** psilvad has quit IRC | 12:55 | |
| *** esikachev has joined #openstack-infra | 12:55 | |
| *** _ari_ has joined #openstack-infra | 12:56 | |
| *** chem has quit IRC | 12:57 | |
| *** deadnull_ is now known as _deadnull | 12:57 | |
| *** dmk0202 has quit IRC | 12:58 | |
| *** dmk0202 has joined #openstack-infra | 12:59 | |
| *** psilvad has joined #openstack-infra | 12:59 | |
| *** chem has joined #openstack-infra | 13:00 | |
| *** andymaier has joined #openstack-infra | 13:01 | |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: remove unneeded builders https://review.openstack.org/328290 | 13:02 |
| *** tzn has joined #openstack-infra | 13:02 | |
| *** rodrigods has quit IRC | 13:05 | |
| *** rodrigods has joined #openstack-infra | 13:05 | |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: setup release notes job for sahara-dashboard https://review.openstack.org/327729 | 13:06 |
| *** jaosorior has quit IRC | 13:07 | |
| pabelanger | krotscheck: nodepool.o.o has the logs, but it hasn't started a new build for today yet | 13:08 |
| *** flepied has quit IRC | 13:08 | |
| *** edmondsw has joined #openstack-infra | 13:08 | |
| *** javeriak has joined #openstack-infra | 13:09 | |
| pabelanger | AJaeger: yolanda: Right, I think this is the issue jeblair exposed when we moved to zuul-launcher. I konw jeblair wants to talk with you about the job | 13:09 |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: remove unneeded builders https://review.openstack.org/328290 | 13:09 |
| *** mdrabe has joined #openstack-infra | 13:09 | |
| krotscheck | pabelanger: Kk. Horizon was wondering. | 13:10 |
| pabelanger | AJaeger: I think https://review.openstack.org/#/c/327921/ was part of the fix | 13:11 |
| *** derekh has joined #openstack-infra | 13:11 | |
| pabelanger | krotscheck: I'll manually do builds today if needed, its been long enough with stale images | 13:11 |
| *** kgiusti has joined #openstack-infra | 13:14 | |
| krotscheck | pabelanger: Excellent, let me know when xenial is up, I need to make sure the world doesn't explode. | 13:15 |
| *** krtaylor has quit IRC | 13:15 | |
| *** amotoki has quit IRC | 13:15 | |
| *** _deadnull is now known as deadnull_ | 13:16 | |
| *** ayoung has joined #openstack-infra | 13:16 | |
| *** eharney has joined #openstack-infra | 13:16 | |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: remove unneeded builders https://review.openstack.org/328290 | 13:17 |
| *** NM has quit IRC | 13:17 | |
| *** amoralej is now known as amoralej|lunch | 13:20 | |
| *** jistr is now known as jistr|mtg | 13:20 | |
| *** NM has joined #openstack-infra | 13:20 | |
| *** pcaruana has quit IRC | 13:20 | |
| *** esberglu has joined #openstack-infra | 13:20 | |
| AJaeger | pabelanger: yeah, saw it in backscroll. | 13:22 |
| * AJaeger runs some errands right now, will ask jeblair once I have some more time | 13:22 | |
| AJaeger | thanks, pabelanger | 13:22 |
| vgridnev | AJaeger, could you please take a look on https://review.openstack.org/#/c/328290/ ? | 13:23 |
| *** rbrndt has joined #openstack-infra | 13:23 | |
| AJaeger | vgridnev: why are you not needing them? | 13:24 |
| AJaeger | vgridnev: without git setup, you have no repo - how do you want t otest? | 13:25 |
| AJaeger | vgridnev: without installation of packages, you have only a very basic image - do you really want that? | 13:25 |
| SotK | does anyone have any idea what might be causing this intermittent failure we're seeing on jobs? http://logs.openstack.org/16/320516/2/gate/gate-storyboard-webclient-npm-run-test-unit/68535d3/console.html#_2016-06-10_13_18_16_738 | 13:25 |
| AJaeger | vgridnev: regarding image descriptions: check nodepool/ directory in project-config | 13:28 |
| *** mriedem has joined #openstack-infra | 13:29 | |
| *** dmk0202 has quit IRC | 13:29 | |
| *** gouthamr has joined #openstack-infra | 13:30 | |
| *** krtaylor has joined #openstack-infra | 13:31 | |
| *** nwkarsten has quit IRC | 13:31 | |
| *** nwkarsten has joined #openstack-infra | 13:32 | |
| *** sree has joined #openstack-infra | 13:33 | |
| *** mixos has joined #openstack-infra | 13:33 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Implement BUILD_TIMEOUT env variable https://review.openstack.org/328298 | 13:35 |
| *** hashar is now known as hasharAway | 13:35 | |
| *** qba73 has quit IRC | 13:35 | |
| openstackgerrit | Merged openstack-infra/project-config: Allow to test agent drivers with partition images https://review.openstack.org/326059 | 13:35 |
| *** nwkarsten has quit IRC | 13:36 | |
| vgridnev | AJaeger, for running tests the only thing we need sahara-image-elements in node, and tox Nothing else is required, and I'm not sure regarding git setup, probably this the only thing we need | 13:36 |
| *** links has joined #openstack-infra | 13:36 | |
| pabelanger | jeblair: notmorgan: I couldn't see a place where we assigned BUILD_TIMEOUT as an env var for zuul-launcher. Specifically devstack-gate references it to setup timeouts ^. And looks to be exposed by tripleo jobs | 13:36 |
| *** kushal has joined #openstack-infra | 13:36 | |
| AJaeger | vgridnev: is that for all sahara-image-elements jobs the same? | 13:37 |
| AJaeger | Then use an empty other-requirements.txt file - that way you can add packages yourself if needed. infra will not install packages for this to the base image. | 13:37 |
| *** javeriak has quit IRC | 13:37 | |
| Shrews | yolanda: hi! quick question for you about the SHADE_OS_CLOUD env variable you propose in 298647 | 13:38 |
| *** mhickey has joined #openstack-infra | 13:38 | |
| *** andrewbogott has quit IRC | 13:38 | |
| *** andrewbogott has joined #openstack-infra | 13:38 | |
| *** salv-orl_ has quit IRC | 13:39 | |
| *** fergal_ has joined #openstack-infra | 13:39 | |
| *** pcaruana has joined #openstack-infra | 13:39 | |
| *** salv-orlando has joined #openstack-infra | 13:39 | |
| *** esikachev has quit IRC | 13:39 | |
| Shrews | yolanda: you said we need that b/c then we can record against different clouds (which is great). the test clouds.yaml is currently stored within the shade repo with an exposed user/password. adding any cloud other than a devstack one would be unsafe, so how do you see us resolving that? | 13:41 |
| vgridnev | AJaeger, thanks, will try other-requirements then | 13:41 |
| *** ajo_ is now known as ajo | 13:41 | |
| *** zeih has quit IRC | 13:42 | |
| yolanda | Shrews, we could pass the password as well with an env var | 13:43 |
| yolanda | do you have another suggestion? | 13:43 |
| *** nwkarsten has joined #openstack-infra | 13:44 | |
| *** akshai has joined #openstack-infra | 13:45 | |
| Shrews | yolanda: not sure. maybe we could make the location of the clouds.yaml configurable via an env variable too? | 13:46 |
| Shrews | yolanda: not that it has to be handled in that review... i was just curious on how you saw that working. we can hash out the details later on, i think | 13:47 |
| yolanda | i exported the env var manually to test, actually, not using tox... | 13:47 |
| *** afred312 has joined #openstack-infra | 13:47 | |
| yolanda | Shrews, having a default dummy clouds.yaml and allowing to consume our own files looks good as well | 13:48 |
| yolanda | can be interesting, because endpoints may change, even on each devstack deployment, if we need to iterate over recording tests, can be a problem if we are not using the same devstack environment | 13:49 |
| *** esikachev has joined #openstack-infra | 13:50 | |
| yolanda | btw, seems i'm having disk corruption problems on my baremetal host :( | 13:54 |
| yolanda | it stopped because root partition was full, and then it cannot start due to xfs corruption! | 13:55 |
| yolanda | oh, bad channel | 13:55 |
| pabelanger | :) | 13:55 |
| *** mhickey has quit IRC | 13:55 | |
| *** Goneri has joined #openstack-infra | 13:56 | |
| *** ifarkas has quit IRC | 13:56 | |
| yolanda | that happens :) | 13:56 |
| *** yamada-h has joined #openstack-infra | 13:56 | |
| *** mriedem1 has joined #openstack-infra | 13:57 | |
| *** ramishra has joined #openstack-infra | 13:58 | |
| *** asselin_ has joined #openstack-infra | 13:58 | |
| *** mriedem has quit IRC | 13:59 | |
| *** psilvad has quit IRC | 13:59 | |
| *** yamada-h has quit IRC | 14:01 | |
| *** links has quit IRC | 14:01 | |
| *** javeriak has joined #openstack-infra | 14:02 | |
| *** inc0 has joined #openstack-infra | 14:05 | |
| *** javeriak has quit IRC | 14:05 | |
| *** maishsk has quit IRC | 14:08 | |
| *** dingyichen has quit IRC | 14:08 | |
| *** amitgandhinz has joined #openstack-infra | 14:09 | |
| *** jistr|mtg is now known as jistr | 14:09 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update JaCoCo plugin https://review.openstack.org/320758 | 14:11 |
| *** esikachev has quit IRC | 14:13 | |
| *** yamahata has joined #openstack-infra | 14:14 | |
| *** xarses_ has quit IRC | 14:14 | |
| *** zz_dimtruck is now known as dimtruck | 14:15 | |
| *** esikachev has joined #openstack-infra | 14:15 | |
| *** bauwser is now known as bauzas | 14:17 | |
| *** amoralej|lunch is now known as amoralej | 14:23 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update Cucumber Reports plugin https://review.openstack.org/321309 | 14:23 |
| *** fesp has joined #openstack-infra | 14:24 | |
| *** sorantis has quit IRC | 14:25 | |
| *** gouthamr has quit IRC | 14:25 | |
| vgridnev | AJaeger, I have questions regarding other-requirements.txt file. It's seems like install-distro-packages script just ignoring that, no idea why | 14:26 |
| vgridnev | http://logs.openstack.org/69/328269/3/check/gate-sahara-buildimages-vanilla/4106089/console.html | 14:26 |
| vgridnev | http://logs.openstack.org/69/328269/3/check/gate-sahara-buildimages-vanilla/4106089/console.html#_2016-06-10_14_12_04_036 | 14:26 |
| vgridnev | this log from commit https://review.openstack.org/#/c/328269/ | 14:26 |
| *** Jeffrey4l_ has quit IRC | 14:26 | |
| *** tonytan4ever has joined #openstack-infra | 14:27 | |
| AJaeger | vgridnev: that's really strange ;( | 14:27 |
| AJaeger | pabelanger: any ideas on what's going on with vgridnev's change? | 14:27 |
| *** fesp has quit IRC | 14:27 | |
| pabelanger | AJaeger: looking | 14:28 |
| *** fesp has joined #openstack-infra | 14:30 | |
| jeblair | mtreinish: owner_string = change.owner_name + ' <' + change.owner.email + '>' | 14:30 |
| jeblair | TypeError: coercing to Unicode: need string or buffer, NoneType found | 14:30 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Only update changed fields when editing tasks https://review.openstack.org/325247 | 14:30 |
| jeblair | mtreinish: looks like some accounts don't have email (eg proposal bot) | 14:30 |
| AJaeger | jeblair: this is the change I meant: https://review.openstack.org/328145 | 14:30 |
| AJaeger | jeblair: Or do you have a better idea? | 14:30 |
| mtreinish | jeblair: ah ok, I need to add a condition checking for the email | 14:31 |
| mtreinish | one sec I'll respin it | 14:31 |
| jeblair | AJaeger: one second, i clicked on that proposal bot change you accidentally pointed to and crashed gertty :) | 14:31 |
| jeblair | which is good! that's why i'm testing unmerged changes! | 14:31 |
| AJaeger | jeblair: Glad that I gave you a test case ;) | 14:31 |
| jeblair | AJaeger: oh, is the encoding why the job broke? | 14:31 |
| pabelanger | AJaeger: vgridnev: JJB is not setup correctly: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/sahara.yaml#n64 | 14:32 |
| *** ifarkas has joined #openstack-infra | 14:32 | |
| AJaeger | jeblair: I expect - have a look at one of the files | 14:32 |
| *** fesp has quit IRC | 14:32 | |
| pabelanger | AJaeger: vgridnev: install-distro-packages should be after gerrit-git-prep | 14:32 |
| AJaeger | jeblair: let me give you a link... | 14:32 |
| *** fesp has joined #openstack-infra | 14:32 | |
| AJaeger | jeblair: https://review.openstack.org/#/c/328164/1/oslo_db/locale/fr/LC_MESSAGES/oslo_db.po | 14:33 |
| AJaeger | All the non-ascii characters are replaced by "?" | 14:33 |
| *** xarses_ has joined #openstack-infra | 14:33 | |
| AJaeger | Better: https://review.openstack.org/#/c/328167/1/manila_ui/locale/ja/LC_MESSAGES/django.po | 14:33 |
| jeblair | AJaeger: ah, sorry, i'm caught up now | 14:34 |
| AJaeger | The changed locale is one guess, didn't test whether that helps. But that was the obvious one ;) | 14:34 |
| jeblair | pabelanger: yes, i forgot about BUILD_TIMEOUT! we need to add that to zuul_launcher | 14:34 |
| AJaeger | no worries, jeblair | 14:34 |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: put install-distro-packages after git setup https://review.openstack.org/328290 | 14:34 |
| jeblair | AJaeger: the locale thing is interesting.... here's the scoop | 14:34 |
| pabelanger | jeblair: okay, 328298 should do that | 14:34 |
| AJaeger | jeblair: So, my suggestion: Merge my change, retrigger *one* periodic job to see that it works, and then abandon all existing ones... | 14:35 |
| AJaeger | jeblair: I need to leave in 5 mins - but will listen for that time ;) | 14:35 |
| * AJaeger will be back later tonight | 14:35 | |
| *** sshnaidm has quit IRC | 14:35 | |
| vgridnev | pabelanger, this change should fix that, right https://review.openstack.org/#/c/328290/ ? | 14:35 |
| vgridnev | AJaeger, ^^ | 14:35 |
| AJaeger | yes, vgridnev | 14:35 |
| jeblair | AJaeger: jenkins was probably using whatever locale was set *on the jenkins master* to run things on the slaves | 14:35 |
| AJaeger | interesting | 14:36 |
| jeblair | AJaeger: we're trying to be better than that, and use the default locale on the slave | 14:36 |
| AJaeger | And now we use C or POSIX? | 14:36 |
| jeblair | AJaeger: it looks like the proposal slave's locale may not have been configured well | 14:36 |
| jeblair | AJaeger: at least, it behaves differently than other slaves. and of course it's very old | 14:36 |
| jeblair | AJaeger: having said that, if we know what we want, we can definitely set it in the job itself | 14:36 |
| AJaeger | UTF8 is all I want ;) | 14:37 |
| *** dizquierdo has joined #openstack-infra | 14:37 | |
| AJaeger | We should set it also by default IMHO. But making it explicit cannot hurt | 14:37 |
| notmorgan | AJaeger: ++ | 14:37 |
| jeblair | AJaeger: cool, so let's do what you suggest, and then maybe later we can fix proposal slave... may just be missing a package or something | 14:38 |
| jeblair | AJaeger: oh | 14:38 |
| jeblair | AJaeger: that's a slave script update | 14:38 |
| AJaeger | jeblair: can you retrigger one of the periodic jobs once the change is in, please? | 14:38 |
| jeblair | AJaeger: it'll take days to get that in | 14:38 |
| jeblair | AJaeger: we should do it in the job instead so it's faster | 14:38 |
| *** psachin has quit IRC | 14:38 | |
| AJaeger | why? proposal slave is updated via puppet, isn't it? | 14:38 |
| jeblair | no | 14:39 |
| jeblair | i'm wrong, you're right | 14:39 |
| jeblair | i forgot about that :) | 14:39 |
| AJaeger | ;) | 14:39 |
| jeblair | yes, i'll land it and trigger one. | 14:39 |
| openstackgerrit | Matthew Treinish proposed openstack/gertty: Add user email addresses to the change view https://review.openstack.org/327785 | 14:39 |
| mtreinish | jeblair: ^^^ that should fix it | 14:39 |
| AJaeger | But that might answer my questoin - https://review.openstack.org/326039 added a new script yesterday morning - and it still does not exist. When do you expect that one be availble on the normal slaves? | 14:40 |
| AJaeger | jeblair: thanks for landing and triggering | 14:40 |
| jeblair | AJaeger: we've had to restart nodepool a lot recently; hard to say. | 14:40 |
| jeblair | mtreinish: that's got it, thanks! | 14:41 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added js-openstack-lib to openstack-sdk's https://review.openstack.org/328326 | 14:41 |
| *** amrith has quit IRC | 14:41 | |
| *** PollyZ has joined #openstack-infra | 14:42 | |
| jeblair | notmorgan: there's a patch at the beginning of the github series to do something about test logging; i haven't looked at the latest version yet, maybe you should check it out | 14:42 |
| AJaeger | jeblair: hope we get it soon - it's part of a voting change - luckily one with a files section, so won't triggered often... | 14:42 |
| notmorgan | jeblair: yep, saw it | 14:42 |
| pabelanger | EmilienM: I think this is an error in your copy_logs.sh: http://logs.openstack.org/32/327332/2/check/gate-puppet-infracloud-puppet-beaker-rspec-ubuntu-trusty/362bfb9/console.html#_2016-06-10_14_20_53_289 | 14:42 |
| * AJaeger will be back later | 14:42 | |
| *** gouthamr has joined #openstack-infra | 14:42 | |
| EmilienM | yeah | 14:42 |
| EmilienM | I'm fixing it | 14:42 |
| *** amrith has joined #openstack-infra | 14:42 | |
| notmorgan | jeblair: and it's pretty good. I think we should be a bit more careful about dropping all / non e of the debug logs. | 14:43 |
| EmilienM | https://review.openstack.org/328303 | 14:43 |
| pabelanger | EmilienM: ack | 14:43 |
| EmilienM | sorry for that | 14:43 |
| notmorgan | jeblair: so we can still get debug but maybe not for libs we don't care about as much at that moment. in short, i think we should do both that patch and something like what i proposed. | 14:43 |
| pabelanger | EmilienM: +1, left commend | 14:44 |
| pabelanger | comment* | 14:44 |
| jeblair | notmorgan: sgtm. | 14:44 |
| *** jamesmcarthur has joined #openstack-infra | 14:44 | |
| gabriel-bezerra | hi folks, how can I change neutron port configuration for nodepool instances when they are created? I'd like to set port-security-enabled=false on them. | 14:44 |
| *** sree has quit IRC | 14:45 | |
| notmorgan | i'll see if we can roll them together and or pull the github series one off into something we can land sooner | 14:45 |
| *** psachin has joined #openstack-infra | 14:45 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update support for BuildResultTrigger plugin https://review.openstack.org/321619 | 14:45 |
| *** sree has joined #openstack-infra | 14:46 | |
| gabriel-bezerra | my instances are in an openstack kilo cloud, and setting port-security-enabled=false on the network would not let instances run on them, as nova always put a security group and the neutron extension needs that the port has no security group to work | 14:46 |
| gabriel-bezerra | the port-security extension | 14:46 |
| jeblair | notmorgan: oh i thought it was at the beginning of the series? | 14:46 |
| notmorgan | jeblair: it might be | 14:46 |
| notmorgan | i need to go look:) | 14:46 |
| gabriel-bezerra | i've looked for some hooks in nodepool configs, but could not find anything so far | 14:47 |
| jeblair | that would make that part easy | 14:47 |
| notmorgan | i didn't check how deep in it was/was not | 14:47 |
| notmorgan | yeah exactly | 14:47 |
| *** Goneri has quit IRC | 14:47 | |
| notmorgan | i also think we need to make some of zuul's debug logs "trace" level, but that can be zuulv3 | 14:47 |
| jeblair | notmorgan: there is no trace level :) | 14:47 |
| notmorgan | jeblair: we should add one! | 14:47 |
| jeblair | notmorgan: nope | 14:48 |
| jeblair | notmorgan: standard python logging config for us! | 14:48 |
| notmorgan | this is where you and i will need to disagree, i think python logging is missing that | 14:48 |
| notmorgan | seriously lacking. trace is super useful, wedging it all into debug isn;t :( | 14:48 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Implement BUILD_TIMEOUT env variable https://review.openstack.org/328298 | 14:48 |
| jeblair | notmorgan: i do agree with you on that. i'm just not interested in breaking everything around python logging to make that point. | 14:49 |
| jeblair | notmorgan: so instead, debug is trace. | 14:49 |
| *** mriedem has joined #openstack-infra | 14:49 | |
| pabelanger | gabriel-bezerra: which jobs is this? I assume you are talking about devstack? | 14:49 |
| gabriel-bezerra | yes, we are setting a 3rd party ci here | 14:49 |
| notmorgan | jeblair: it doesnt break everything though :P, it is a level below debug that is easily added and optionally turned on if/when you need it. | 14:50 |
| notmorgan | jeblair: again, i think you and i will just need to disagree here. | 14:50 |
| notmorgan | jeblair: *shrug* | 14:50 |
| jeblair | notmorgan: we have had SO MANY problems with openstack having the trace level | 14:50 |
| gabriel-bezerra | pabelanger: we need it to communicate with machines in a flat network | 14:50 |
| *** jamesmcarthur has quit IRC | 14:50 | |
| *** DevBox has joined #openstack-infra | 14:50 | |
| notmorgan | jeblair: the trace level openstack had wasn't "trace" it was "traceback" and "Error" level | 14:51 |
| *** jamesmcarthur has joined #openstack-infra | 14:51 | |
| *** mriedem1 has quit IRC | 14:51 | |
| notmorgan | jeblair: it was never a "trace" level until... mitaka? | 14:51 |
| jeblair | notmorgan: oh, yes, that's right. that was bonkers. | 14:51 |
| *** sree has quit IRC | 14:51 | |
| notmorgan | i totally agree, that was no ok | 14:51 |
| notmorgan | a proper trace level below debug is a different story :) | 14:51 |
| pabelanger | gabriel-bezerra: you'd have to see if shade exposes the ability to do that. That would be nodepools interfaces to the cloud | 14:51 |
| jeblair | notmorgan: it has still kind of put me off of wanting to add custom levels. | 14:51 |
| notmorgan | hehe sure. that i can see | 14:52 |
| jeblair | notmorgan: anyway, in zuul, the *idea* is that INFO should give an operator everything they need to know about the decisions zuul has made | 14:52 |
| *** flaper87 has quit IRC | 14:52 | |
| *** fesp is now known as flaper87 | 14:52 | |
| notmorgan | i'll remind you in zuulv3 later on | 14:52 |
| jeblair | notmorgan: like "why did zuul run this job?" | 14:52 |
| notmorgan | info absolutely should be the default and useful | 14:52 |
| *** flaper87 has quit IRC | 14:52 | |
| notmorgan | debug should be for gate/check/unit | 14:52 |
| *** flaper87 has joined #openstack-infra | 14:52 | |
| jeblair | notmorgan: and is what we would expect most people to run it at | 14:52 |
| gabriel-bezerra | pabelanger: I'll have a look | 14:52 |
| gabriel-bezerra | pabelanger: thank you | 14:52 |
| notmorgan | trace should be for "omg... WHAT did the entire data path look like" | 14:52 |
| jeblair | notmorgan: we have 2 logs -- the info log, so we can see whether we are accomplishing the goal with info (we are not, but it's low priority to fix) | 14:53 |
| pabelanger | gabriel-bezerra: mordred or clarkb might have better ideas too. I'm not to familiar with port-security-enabled | 14:53 |
| *** arxcruz has quit IRC | 14:53 | |
| jeblair | notmorgan: and the debug log, because, as the primary user, when anything goes wrong, we need to be able to completely construct the chain of events that caused the error | 14:53 |
| jeblair | notmorgan: so, for the forseeable future, openstack will be logging zuul in the most detailed log level possible in order to do that | 14:54 |
| jeblair | notmorgan: it has, so far, been extremely useful. whenever zuul has had an error in production, we've been able to recreate the exact conditions again, and put them into a regression test case, and fix the bug | 14:54 |
| jeblair | notmorgan: you'll find many test cases describe that :) | 14:55 |
| jeblair | notmorgan: so even if it were trace, that's the level we would be running at | 14:55 |
| *** jamesmcarthur has quit IRC | 14:55 | |
| jeblair | notmorgan: the same holds true for unit test failures | 14:55 |
| *** mtanino has joined #openstack-infra | 14:55 | |
| jeblair | notmorgan: given a flakey test, i can figure out what went wrong and fix it | 14:55 |
| pabelanger | jeblair: appologies, I updated the commit message on 328298 after your +3 | 14:55 |
| pabelanger | sorry about that | 14:55 |
| pabelanger | apologies* | 14:55 |
| *** flaper87 has quit IRC | 14:55 | |
| *** flaper87 has joined #openstack-infra | 14:55 | |
| jeblair | notmorgan: (which is why the missing log files are so annoying) | 14:55 |
| notmorgan | jeblair: i think some of the internal bits of house keeping really aren;'t that useful there. but generally you're right. | 14:56 |
| jeblair | notmorgan: so the thing is, if we added trace, then we woud just switch all of our logging to trace :) | 14:56 |
| jeblair | notmorgan: which is why i'm in favor of keeping debug | 14:56 |
| *** javeriak has joined #openstack-infra | 14:56 | |
| *** _tzn has joined #openstack-infra | 14:56 | |
| jeblair | notmorgan: yeah, there are some debug log lines that can go | 14:56 |
| notmorgan | jeblair: like i said, i think we can revisit this convo in v3, once we have logging cleaned up. | 14:56 |
| *** daemontool has quit IRC | 14:57 | |
| jeblair | notmorgan: okay. i'm trying to tell you it will be a hard sell. :) | 14:57 |
| notmorgan | jeblair: thats fine. | 14:57 |
| jeblair | notmorgan: i'm open to lots of things, but i'm happy with the logging system we have. it's worked for years, and it's what i want. | 14:57 |
| notmorgan | i might change my opinion when we have logging more sane | 14:57 |
| jeblair | notmorgan: i will get very grumpy the first time we "improve" it and i am not able to completely resolve a production issue after the fact | 14:58 |
| notmorgan | but today i can see a difference between debug-useful and debug-deep-dive-data-flow-trace-that-could-be-useful-conditionally | 14:58 |
| *** javeriak has quit IRC | 14:58 | |
| notmorgan | conditionally = only when debugging something that isn't related to zuul's workload itself. in this case. | 14:58 |
| notmorgan | the "debug" lines you say could go. | 14:59 |
| *** sshnaidm has joined #openstack-infra | 14:59 | |
| pabelanger | jeblair: first time I've seen this: http://paste.openstack.org/show/510563/ nodepool gave zuul-launcher a ready node, zuul-launcher failed to SSH into it. | 14:59 |
| notmorgan | but lets get over to v3 and have info logs being awesome/useful, and debug what we want. if a trace level is warranted then, i'll make the case :) | 14:59 |
| *** tzn has quit IRC | 14:59 | |
| pabelanger | jeblair: wonder if we should do something clever there to trap the ssh failure and requeue the job again | 15:00 |
| *** mdrabe has quit IRC | 15:00 | |
| pabelanger | or, likely the right move, update our result to indicate a SSH problem | 15:00 |
| *** anteaya has joined #openstack-infra | 15:00 | |
| jeblair | pabelanger: well, if we can detect that case, we could finish the job with no result which will cause zuul to reschedule it | 15:01 |
| jeblair | pabelanger: does 'exit code 3' mean that? | 15:02 |
| *** Swanson has quit IRC | 15:02 | |
| pabelanger | jeblair: my thinking too. It would only be problematic is, SSH key was wrong for ansible-playbook. Then an endless requeue | 15:02 |
| pabelanger | jeblair: not sure, googling now | 15:02 |
| jeblair | pabelanger: yeah, for that we actually should make zuul count retries and give up after, say, 5 | 15:02 |
| pabelanger | that would work | 15:02 |
| jeblair | we always meant to do that but never got around to it i think | 15:02 |
| pabelanger | AnsibleHostUnreachable == exit(3) | 15:03 |
| pabelanger | there we go | 15:03 |
| *** flepied has joined #openstack-infra | 15:03 | |
| pabelanger | AnsibleHostFailed == exit 2 if SSH failed, so that address our public key issue | 15:03 |
| pabelanger | let me get a patch up | 15:03 |
| jeblair | pabelanger: cool; thx! | 15:03 |
| *** markusry has quit IRC | 15:04 | |
| *** markusry has joined #openstack-infra | 15:04 | |
| fungi | pabelanger: what region was the node in? | 15:07 |
| pabelanger | fungi: rax-dfw | 15:08 |
| *** ihrachys has quit IRC | 15:08 | |
| fungi | not terribly surprising then. i think we still have connectivity issues between systems in dfw and between dfw and ord | 15:08 |
| fungi | which don't seem to manifest over the open internet for some reason | 15:09 |
| *** tesseract has quit IRC | 15:09 | |
| *** nwkarsten has quit IRC | 15:09 | |
| *** piet has joined #openstack-infra | 15:09 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: WIP Show a different output serialization https://review.openstack.org/328338 | 15:10 |
| mordred | Shrews, yolanda: I have written the code to do that, but not in the form it needs to be to be a serializer - mostly just wanted to put it up there to look at it | 15:10 |
| mordred | ^^ | 15:10 |
| *** pcaruana has quit IRC | 15:13 | |
| dasm | o/ infra community. over 24h ago I got +W on one of patches: https://review.openstack.org/#/c/327210/ it's not merged, but it seems to disappear from zuul. | 15:13 |
| *** mdrabe has joined #openstack-infra | 15:13 | |
| openstackgerrit | Merged openstack-infra/project-config: Set locale for translation jobs https://review.openstack.org/328145 | 15:13 |
| openstackgerrit | Merged openstack-infra/project-config: put install-distro-packages after git setup https://review.openstack.org/328290 | 15:14 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Update Team resource to add the Users subcontroller https://review.openstack.org/320516 | 15:14 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Requeue jobs if AnsibleHostUnreachable return https://review.openstack.org/328341 | 15:15 |
| pabelanger | jeblair: notmorgan: ^ requeue the job if we detect AnsibleHostUnreachable | 15:16 |
| pabelanger | jeblair: I'll see if I can figure out how to track requeue in zuul | 15:17 |
| mordred | ooo | 15:17 |
| mordred | nice | 15:17 |
| mordred | pabelanger: it's almost like we'd be resilient against nodes going away! | 15:17 |
| fungi | we got a fairly similar behavior out of zuul+jenkins if jenkins lost contact with the node | 15:19 |
| pabelanger | mordred: I know, this is starting to get legit | 15:19 |
| jeblair | yeah, and this is probably the same situation actually | 15:19 |
| fungi | it's not flawless. jobs can definitely _cause_ the node to become unreachable | 15:19 |
| *** rcernin has quit IRC | 15:19 | |
| fungi | but it's status quo for us | 15:19 |
| *** ramishra has quit IRC | 15:21 | |
| yolanda | mordred, there is a yaml serializer for betamax. But we agreed to use json, to don't add that yaml dependency to betamax | 15:22 |
| *** salv-orlando has quit IRC | 15:22 | |
| mordred | yolanda: right. the serialization code is one we can put into shade test suite itself - since shade already has a yaml depend | 15:22 |
| yolanda | mordred, a custom serializer then? | 15:23 |
| mordred | yolanda: but it's not just yaml - it's yaml with the body of the requests in well-formatted block-quoted json | 15:23 |
| mordred | yah | 15:23 |
| mordred | I'll have code for you to look at soon | 15:23 |
| *** salv-orlando has joined #openstack-infra | 15:23 | |
| mordred | I mean, I HAVE code right now, but it's not in shape for other humans to look at it | 15:23 |
| *** ramishra has joined #openstack-infra | 15:23 | |
| *** mhickey has joined #openstack-infra | 15:24 | |
| mordred | yolanda: but if you look at that file, you'll see each ['request']['body']['string'] has some nice looking json in it | 15:24 |
| *** mriedem has quit IRC | 15:24 | |
| *** mriedem has joined #openstack-infra | 15:25 | |
| yolanda | mordred, also look at that change, the hooks for the keystoneauth fixture: https://review.openstack.org/#/c/320340/. How will it affect that? | 15:25 |
| *** esikachev has quit IRC | 15:25 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Create overcloud images for liberty using EXT4 https://review.openstack.org/327773 | 15:25 |
| mordred | yolanda: I don't think it will. that said - notmorgan - does keystoneuath mind yaml depends in its test-requirements? | 15:26 |
| *** flaper87 has quit IRC | 15:27 | |
| mordred | yolanda: but just thinking out loud - I think I'll write the patch for you first just in shade, and then it'll be easier to talk about whether or not we should push it down to ksa or not | 15:27 |
| *** burgerk has joined #openstack-infra | 15:28 | |
| *** Goneri has joined #openstack-infra | 15:29 | |
| *** NM has quit IRC | 15:29 | |
| yolanda | mordred, as the betamax hooks are dealing with some json parsing to replace values, i'm worried about how this hook can interact with another custom serializer | 15:29 |
| yolanda | it may just work | 15:30 |
| mordred | yolanda: I _think_ it's going to magically just work | 15:30 |
| mordred | yolanda: but I have been horribly wrong plenty of times :) | 15:30 |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Make ironic-{pxe,agent}_ipmitool jobs voting https://review.openstack.org/312596 | 15:30 |
| yolanda | you will be right this time :) | 15:30 |
| mordred | yolanda: even a stopped clock is right twice a day | 15:31 |
| mordred | notmorgan: https://review.openstack.org/#/c/328338/1/shade/tests/unit/fixtures/test_delete_flavor.yaml is the context for the "should we put the code to produce this in ksa or leave it in shade" | 15:32 |
| fergal_ | guys does anyone know how i’d go about modifying jjb to output the steps it takes to resolve variables in the yaml files? i.e. what variables override others and where the variables come from etc | 15:32 |
| yolanda | mordred, that's a wise sentence! | 15:32 |
| mordred | yolanda: :) | 15:32 |
| mordred | yolanda: I'm about to wander out - but I'll have both some follow up patches and some reviews for you hopefully soon | 15:33 |
| anteaya | SotK: I don't know what is causing those intermittent npm failures but I noticed them as well | 15:33 |
| mordred | (this is me starting to poke at it for real) | 15:33 |
| anteaya | fungi: thanks for the work on the wiki, I see rockyg is not online, perhaps she is on holiday? | 15:34 |
| yolanda | mordred, i'll be around this weekend . Too many things going on to disconnect | 15:34 |
| *** salv-orlando has quit IRC | 15:34 | |
| *** salv-orlando has joined #openstack-infra | 15:34 | |
| fungi | anteaya: yeah, i figured the one-week disablement would be a safe but nonpermanent way to give us an opportunity to sync up with her at our convenience | 15:35 |
| anteaya | dasm: I'm looking at https://review.openstack.org/#/c/327210/ and seeing that currently zuul queue lengths are not zero so the recheck may be a queued event waiting to be processed | 15:35 |
| *** jkraj has joined #openstack-infra | 15:35 | |
| anteaya | fungi: sounds reasonable, hopefully she shows up in channel and asks | 15:35 |
| fungi | it was more that i found it interesting spammers were actively compromising accounts in the launchpad sso to be able to post more spam on our wiki | 15:36 |
| *** mhickey has quit IRC | 15:36 | |
| *** vincentll has quit IRC | 15:36 | |
| fungi | not just creating new accounts | 15:36 |
| anteaya | fungi: yes, I agree that is an interesting tactic, they are determined to compromise our wiki | 15:37 |
| nibalizer | rcarrillocruz: heya | 15:37 |
| dasm | anteaya: ok. I see ChangBo rechecked it few secs ago. I was just wondering what happened that it takes so long time to process. Didn't want to recheck without confirmation from infra that everything is OK | 15:37 |
| *** hongbin has joined #openstack-infra | 15:37 | |
| nibalizer | before we both dissapear into pumpkins, is there stuff you're needing from me? | 15:37 |
| *** chem has quit IRC | 15:38 | |
| anteaya | Zara: you write the best emails | 15:38 |
| anteaya | nibalizer: who else is about to pumpkin? | 15:38 |
| openstackgerrit | Matthew Treinish proposed openstack/os-testr: Add documentation for generate-subunit https://review.openstack.org/328352 | 15:39 |
| anteaya | dasm: well there has been a lot of activity in our tooling this week so it is possible your patch fell in a gap, thanks for checking, there is no current issue of which I am aware | 15:39 |
| notmorgan | mordred: hmm | 15:39 |
| *** NM has joined #openstack-infra | 15:40 | |
| *** andreas_s has quit IRC | 15:40 | |
| notmorgan | mordred: i don;'t mind ksa having more test-requires like yaml. | 15:40 |
| mordred | notmorgan: no need for immediate answer - just wanted to have you start pondering in the brainhole | 15:40 |
| dasm | anteaya: thanks. i see zuul is changed :) | 15:40 |
| anteaya | dasm: your patch has shown up in the check queue | 15:40 |
| *** EricGonczer_ has joined #openstack-infra | 15:40 | |
| notmorgan | mordred: though i would probably make it an extras? | 15:40 |
| mordred | notmorgan: nod. I'll have code for you to look at soonish | 15:41 |
| notmorgan | mordred: if it was required for runtime of ksa, i'd say absolutely not. | 15:41 |
| dasm | anteaya: yeah. now i see it :) thanks again | 15:41 |
| anteaya | dasm: zuul is always changing, like tina turner, she always has a brand new show you never imagined before | 15:41 |
| anteaya | dasm: welcome | 15:41 |
| dasm | anteaya: :D | 15:41 |
| *** matrohon has quit IRC | 15:42 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 15:42 |
| *** csomerville has joined #openstack-infra | 15:44 | |
| *** mixos has quit IRC | 15:46 | |
| *** bpokorny has joined #openstack-infra | 15:46 | |
| *** ramishra has quit IRC | 15:46 | |
| *** dimtruck is now known as zz_dimtruck | 15:46 | |
| jeblair | there is a job called "fuel-docs-docs" | 15:47 |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Make ironic-{pxe,agent}_ipmitool jobs voting https://review.openstack.org/312596 | 15:47 |
| openstackgerrit | Matthew Treinish proposed openstack/os-testr: Remove the TODO file https://review.openstack.org/328357 | 15:47 |
| nibalizer | anteaya: we are all pumpkin candidates | 15:48 |
| *** ramishra has joined #openstack-infra | 15:48 | |
| *** vgridnev has quit IRC | 15:48 | |
| anteaya | nibalizer: wooooo | 15:48 |
| * anteaya celebrates her pumpkin candidacy | 15:48 | |
| anteaya | jeblair: docs about docs? | 15:48 |
| *** psachin has quit IRC | 15:48 | |
| *** zz_dimtruck is now known as dimtruck | 15:51 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Build and upload instack.raw image https://review.openstack.org/315072 | 15:51 |
| jeblair | pabelanger, notmorgan: it does look like a lot (all?) of the -docs jobs are failing in post when run by zl. also releasenotes jobs. | 15:53 |
| jeblair | so i'm tailing the logs trying to catch one now | 15:53 |
| jeblair | i'll hold the node | 15:53 |
| notmorgan | jeblair: icky. | 15:53 |
| pabelanger | jeblair: ack, happy to help too | 15:53 |
| jeblair | ooh got one | 15:53 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: gerritbot: add missing stable branches for #openstack-telemetry https://review.openstack.org/328360 | 15:53 |
| notmorgan | and let me know what i can do to help out :) | 15:53 |
| jeblair | 172.99.106.130 | 15:54 |
| pabelanger | streaming | 15:54 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: gerritbot: add missing stable branches for #openstack-telemetry https://review.openstack.org/328360 | 15:55 |
| *** Kiall has quit IRC | 15:55 | |
| * notmorgan watches logs | 15:55 | |
| notmorgan | jeblair: success? | 15:56 |
| *** somerville32 has joined #openstack-infra | 15:56 | |
| jeblair | notmorgan: they fail in post after succeeding in the main job | 15:56 |
| notmorgan | ahhh | 15:56 |
| pabelanger | notmorgan: http://paste.openstack.org/show/510642/ | 15:57 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: [Manila] Add new experimental job https://review.openstack.org/328364 | 15:57 |
| pabelanger | Received abort request | 15:57 |
| pabelanger | that doesn't look right | 15:57 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Look for a passing periodic upgrades job before moving pin https://review.openstack.org/315075 | 15:57 |
| jeblair | pabelanger: every job ends with an abort :) | 15:58 |
| pabelanger | Oh | 15:58 |
| rcarrillocruz | nibalizer: I am good, pushed a preliminar fix for the ansible-puppet syslogging on puppet máster and started poking the puppetdb 3 stuff | 15:58 |
| jeblair | pabelanger: it's part of the worker shutdown | 15:58 |
| * pabelanger looks a code | 15:58 | |
| jeblair | pabelanger, notmorgan: anyway, this one succeeded (manila-docs) | 15:58 |
| jeblair | so i guess it's not every docs job | 15:58 |
| *** Kiall has joined #openstack-infra | 15:58 | |
| notmorgan | hehe | 15:58 |
| notmorgan | just "most" | 15:58 |
| pabelanger | jeblair: I see failure however | 15:58 |
| jeblair | pabelanger: where? | 15:58 |
| pabelanger | jeblair: see pastebin | 15:58 |
| *** csomerville has quit IRC | 15:58 | |
| jeblair | oh! | 15:59 |
| pabelanger | http://paste.openstack.org/show/510642/ | 15:59 |
| notmorgan | jeblair: Got ZMQ event onFinalized {"build": {"status": "{\"result\": \"FAILURE\"}" | 15:59 |
| notmorgan | .. | 15:59 |
| jeblair | this was "gate-manila-docs" not "manila-docs" | 15:59 |
| jeblair | pabelanger: oh, there was an abort request for the main job | 15:59 |
| jeblair | that was a real abort | 15:59 |
| notmorgan | pabelanger: that pastebin was also magnum? py34 | 15:59 |
| *** roxanaghe has joined #openstack-infra | 15:59 | |
| jeblair | yeah, we are also looking at 2 different jobs | 15:59 |
| jeblair | that was from a few hours ago | 16:00 |
| pabelanger | ugh | 16:00 |
| pabelanger | copy and paste fail yet again | 16:00 |
| notmorgan | hehe happens to the best of us | 16:00 |
| *** dimtruck is now known as zz_dimtruck | 16:00 | |
| jeblair | i'll include the node # next time | 16:00 |
| jeblair | ip probably got reused | 16:01 |
| pabelanger | Yup, that was the issue | 16:01 |
| *** PollyZ has quit IRC | 16:01 | |
| pabelanger | grep'd on the IP | 16:01 |
| pabelanger | 1581393 was the job I think | 16:01 |
| pabelanger | which was success | 16:01 |
| jeblair | yep | 16:01 |
| *** gongysh has joined #openstack-infra | 16:01 | |
| jeblair | we're not going to see a docs or releasenotes job for a while... | 16:03 |
| *** zz_dimtruck is now known as dimtruck | 16:03 | |
| pabelanger | okay, at least we know there is a potential issue | 16:04 |
| zaro | morning | 16:04 |
| jeblair | there are a bunch queued in the post pipeline, but we're backlogged in check | 16:04 |
| *** _shaps_ has quit IRC | 16:04 | |
| fungi | i thought the release notes jobs were in the higher priority release-post pipeline? | 16:05 |
| jeblair | there's a nova change in post with a -releasenotes job | 16:05 |
| jeblair | there's nothing in release-post | 16:06 |
| *** PollyZ has joined #openstack-infra | 16:06 | |
| jeblair | fungi: perhaps they are in both? | 16:06 |
| fungi | ahh, no i expect i was just thinking of different jobs | 16:07 |
| fungi | i definitely see them in the regular post pipeline | 16:07 |
| anteaya | morning zaro | 16:10 |
| pabelanger | jeblair: notmorgan: here is a failure from earlier today: http://paste.openstack.org/show/510644/ | 16:10 |
| pabelanger | going to peak an JJB to make sure out paths are right | 16:10 |
| *** weshay is now known as weshay_br | 16:10 | |
| *** weshay_br is now known as weshay_brb | 16:10 | |
| *** ifarkas has quit IRC | 16:10 | |
| *** PollyZ has quit IRC | 16:11 | |
| notmorgan | pabelanger: rsync issue? | 16:12 |
| pabelanger | maybe | 16:12 |
| pabelanger | is -include=\"/releasenotes/build/html/**\" valid for rsync? | 16:12 |
| notmorgan | that... makes my brain hurt | 16:12 |
| pabelanger | the forward / on releasenotes is making my spidersense tingle | 16:13 |
| notmorgan | yeah. | 16:13 |
| zaro | fergal_: i think variable substitution is done in builder.py and variables can come from a number of places, job def, macro def, and even the individual yaml for the individual plugins. i think waynr zxiiro and electrofelix can probably help more with your questions though. | 16:13 |
| notmorgan | that might need to be ./ ? | 16:13 |
| nibalizer | rcarrillocruz: fantastic | 16:13 |
| *** fergal_ has quit IRC | 16:13 | |
| openstackgerrit | Merged openstack-infra/zuul: Implement BUILD_TIMEOUT env variable https://review.openstack.org/328298 | 16:14 |
| pabelanger | notmorgan: I think so | 16:14 |
| *** fergal_ has joined #openstack-infra | 16:14 | |
| pabelanger | but checking other jobs | 16:14 |
| * notmorgan nods. | 16:14 | |
| fergal_ | sorry got disconnected | 16:15 |
| *** kzaitsev_mb has quit IRC | 16:15 | |
| jeblair | an include starting with '/' means it's anchored to the root of the transfer | 16:15 |
| notmorgan | jeblair: ah. ok | 16:15 |
| jeblair | pabelanger, notmorgan: you may want to read the "FILTER RULES" section of the rsync manual | 16:15 |
| pabelanger | jeblair: notmorgan: Ya, seems to be correct by looking at other jobs | 16:16 |
| jeblair | i've had to have it under my pillow when working on this | 16:16 |
| notmorgan | jeblair: yeah looking at FILTER RULES now | 16:16 |
| jeblair | also, there are some good comments in zuul explaining the reasoning for each rsync thing | 16:17 |
| notmorgan | ah the ** makes sense now | 16:17 |
| notmorgan | well... doesn't make sense, but "works for rsync" :P | 16:17 |
| *** jkraj has quit IRC | 16:18 | |
| *** afred312 has quit IRC | 16:18 | |
| notmorgan | /home/jenkins/workspace/tempest-releasenotes\" failed: No such file or directory | 16:18 |
| notmorgan | is ZL still using /home/jenkins/workspace? | 16:19 |
| *** dimtruck is now known as zz_dimtruck | 16:20 | |
| jeblair | yep, lots of stuff hard-coded to that | 16:20 |
| jeblair | pabelanger is working on changing it | 16:20 |
| notmorgan | ah cool. | 16:20 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: [Manila] Add new experimental job https://review.openstack.org/328364 | 16:20 |
| *** yamahata has quit IRC | 16:20 | |
| *** amotoki has joined #openstack-infra | 16:21 | |
| *** zz_dimtruck is now known as dimtruck | 16:21 | |
| *** weshay_brb is now known as weshay | 16:21 | |
| pabelanger | jeblair: notmorgan: We should be able to test our the zuul user shortly | 16:25 |
| pabelanger | out* | 16:25 |
| notmorgan | pabelanger: woo! | 16:25 |
| pabelanger | new SSH keys should be in place today | 16:25 |
| *** piet has quit IRC | 16:28 | |
| *** kzaitsev_mb has joined #openstack-infra | 16:29 | |
| jeblair | i'm retriggering the cinder propose translation update job to test AJaeger's locale fix | 16:29 |
| *** PollyZ has joined #openstack-infra | 16:33 | |
| *** ayoung has quit IRC | 16:33 | |
| *** tphummel has joined #openstack-infra | 16:34 | |
| *** jlanoux has quit IRC | 16:34 | |
| *** ccamacho is now known as ccamacho|away | 16:34 | |
| *** erlon has joined #openstack-infra | 16:35 | |
| *** asettle has quit IRC | 16:35 | |
| *** somerville32 has quit IRC | 16:35 | |
| jeblair | pabelanger: i held 1583139, 1583182 and 1583183 | 16:37 |
| *** PollyZ has quit IRC | 16:37 | |
| jeblair | and 1583290 | 16:37 |
| *** sarob has joined #openstack-infra | 16:38 | |
| *** yamahata has joined #openstack-infra | 16:38 | |
| *** somerville32 has joined #openstack-infra | 16:39 | |
| jeblair | AJaeger: does this look correct? http://logs.openstack.org/periodic/cinder-propose-translation-update/manual/console.html | 16:39 |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack-infra/project-config: Skip func/unit tests only if Nova reno changes https://review.openstack.org/328380 | 16:41 |
| *** e0ne has quit IRC | 16:41 | |
| *** chem has joined #openstack-infra | 16:42 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Use pull mode for synchronize in _makeFTPTask https://review.openstack.org/328382 | 16:43 |
| pabelanger | jeblair: notmorgan: ^ I think that is our issue | 16:43 |
| jeblair | pabelanger: wow it's right there in the log. awesome. | 16:44 |
| jeblair | rsync "/home/jenkins/workspace/neutron-docs/" "jenkins@172.99.106.185:/tmp/tmp4hVeDe/ansible/tmpR4pwNa/content" is decidedly backwards. | 16:44 |
| pabelanger | jenkins@149.202.161.180:/tmp/tmpLIgICf/ansible/tmp2_DTNi/content was what pointed me to it | 16:44 |
| pabelanger | ya | 16:44 |
| jeblair | fungi, notmorgan: https://review.openstack.org/328382 | 16:45 |
| jeblair | i will release those nodes now | 16:46 |
| notmorgan | pabelanger: oh wow yeah. | 16:46 |
| notmorgan | pabelanger: subtle but important | 16:46 |
| fungi | pabelanger: how does "pull mode" differ in behavior? | 16:46 |
| *** bhavik has joined #openstack-infra | 16:47 | |
| notmorgan | fungi: direction of syncronization | 16:47 |
| pabelanger | fungi: controls the direction of rsync: http://docs.ansible.com/ansible/synchronize_module.html | 16:47 |
| notmorgan | fungi: move from there to here, vs here to there | 16:47 |
| notmorgan | fungi: remote host is the source wuth mode=pull | 16:48 |
| fungi | oh, so we were trying to push (nonexistent) files from the launcher to the node? | 16:48 |
| jeblair | yep | 16:48 |
| fungi | wow | 16:48 |
| pabelanger | right | 16:48 |
| fungi | yeah, that's a subtle yet important distinction | 16:48 |
| jeblair | and we all stared at that log line and only pabelanger saw it :) | 16:48 |
| slagle | clarkb: hi, interested in your thoughts on this: https://review.openstack.org/#/c/326095/ | 16:48 |
| pabelanger | to be fair, I did diff it between the scp task | 16:49 |
| fungi | pabelanger: nice eyesight | 16:49 |
| jeblair | "rsync...something something...jenkins...something...something...tmpdir" | 16:49 |
| notmorgan | jeblair: yep. | 16:49 |
| pabelanger | I still get cross eye'd looking at our local_action rsync command | 16:50 |
| *** sdake has joined #openstack-infra | 16:50 | |
| notmorgan | jeblair: so the log reduction for the github integration sets the default logging for gate/check to INFO, that doesn't seem like what we want. | 16:50 |
| notmorgan | jeblair: but the ability to change the levels in the tests is good. | 16:50 |
| jeblair | notmorgan: hrm. | 16:50 |
| jeblair | notmorgan: i was really hoping for "discard logs on success, include logs on failure." | 16:51 |
| notmorgan | jeblair: that is a bit harder to do in unit tests. | 16:51 |
| jeblair | true, but all the logging goes through a special test fixture.. | 16:51 |
| notmorgan | jeblair: yeah. we'd need to change things at the testr level/subunit level i think | 16:52 |
| *** vhosakot has joined #openstack-infra | 16:52 | |
| jeblair | it's all magic to me. i have no idea how this stuff works. | 16:52 |
| jeblair | all i can do at this point is say what i need as a user of it :( | 16:52 |
| *** dtantsur is now known as dtantsur|afk | 16:52 | |
| notmorgan | jeblair: a short-term fix is dropping gear.Server (and maybe gear.Client) logs to info | 16:52 |
| jeblair | notmorgan: yes, i think that's reasonable | 16:53 |
| notmorgan | and i can work on something to drop logs for successful tests. | 16:53 |
| jeblair | it's been a long time since we've needed those | 16:53 |
| *** vgridnev has joined #openstack-infra | 16:53 | |
| notmorgan | i'm trying to think if there is an easy way i can make an ENV var to create multiple fake loggers so we can turn it back on easily if needed | 16:53 |
| jeblair | notmorgan: server and client at info should be find | 16:54 |
| jeblair | fine | 16:54 |
| notmorgan | vs. hard-codding or having 500000 different OS_LOGGING_<blah> vars for each lib | 16:54 |
| clarkb | slagle: I have two issues with it. the first is going to 3 nodes before 2. I woukd start with two and iron out issues first. its just easier. the other is please whatever you do dont reinvent our networking things | 16:54 |
| notmorgan | i'll re-spin my log reduction thing, i think i have a way. and i'll explore the drop logs on success option but i think i'm going to have to invoke mtreinish for help on that :) | 16:54 |
| clarkb | slagle: start with primary node undercloud deploying all in one subnode | 16:55 |
| openstackgerrit | Merged openstack-infra/zuul: Requeue jobs if AnsibleHostUnreachable return https://review.openstack.org/328341 | 16:55 |
| slagle | clarkb: we can't yet do that in tripleo, but we might get there this cycle | 16:56 |
| *** amotoki has quit IRC | 16:56 | |
| slagle | we're working on adding flexibility in the roles that get deployed | 16:56 |
| slagle | should allow us to deploy an all-in-one | 16:56 |
| clarkb | that is where I wpuld start | 16:57 |
| clarkb | otherwise you will spend more time debugging cloud things than your software | 16:57 |
| slagle | just b/c of the networking stuff mostly? | 16:58 |
| *** sree has joined #openstack-infra | 16:58 | |
| clarkb | networking is the big one yes | 16:58 |
| slagle | is that mostly what's setup by setup_multinode_connectivity in devstack-vm-gate.sh? | 16:59 |
| clarkb | yes | 16:59 |
| *** PollyZ has joined #openstack-infra | 16:59 | |
| slagle | i was looking at that, and not sure this would work for tripleo, given we expect to setup and configure the bridges as part of the deployment | 17:00 |
| clarkb | we have to create our own overlays to make sure nested VM connectivity works | 17:00 |
| clarkb | slagle: you are confusing the separation here | 17:00 |
| clarkb | slagle: this is your hardware switch | 17:00 |
| clarkb | which is distinct from what ooemstack sets up | 17:00 |
| *** DevBox has quit IRC | 17:01 | |
| slagle | i see, i'll take another look at it | 17:01 |
| slagle | that could work if we pre-configure it using this same code | 17:01 |
| *** sree has quit IRC | 17:03 | |
| *** kzaitsev_mb has quit IRC | 17:03 | |
| *** NM has quit IRC | 17:03 | |
| slagle | initially i've been using openvpn to create the private subnets we'd use | 17:03 |
| slagle | but i get your point about reusing what is already there | 17:03 |
| clarkb | and fuel uses tinc so I am sort of trying to get us tobuse the wheel that is already there and known to work on all our clouds | 17:05 |
| slagle | yea, definitely | 17:05 |
| *** NM has joined #openstack-infra | 17:05 | |
| jeblair | clarkb: the other day, you were talking to someone about getting secrets onto nodes... and something you said suggested the idea that we could replace our static workers with dynamic ones by putting the secrets onto nodepool, and then having special node types with dib elements that install the secrets onto those images. | 17:05 |
| jeblair | clarkb: does that sound plausible? | 17:06 |
| pabelanger | krotscheck: DIBs were successful, uploads running now | 17:06 |
| clarkb | jeblair: ya I think either dib or nodepool.via ready script could do that | 17:06 |
| krotscheck | pabelanger: \o/ | 17:06 |
| krotscheck | matt-borland, robcresswell: ^^ | 17:06 |
| pabelanger | krotscheck: should take about 4-5 hours. | 17:07 |
| pabelanger | for uploads to finish | 17:07 |
| jeblair | clarkb: ready script would take some extra work i think (i don't think we have a good way of passing secret per-image-type info to a ready script) | 17:07 |
| jeblair | clarkb: but just a "install pypi credentials" element would be pretty easy, yeah? | 17:08 |
| pabelanger | fungi: clarkb: jeblair: What do you think about removing vexxhost-ca-ymq-1 from nodepool? We still do image uploads each day but unsure when it will be coming back online | 17:08 |
| fungi | pabelanger: i thought the upshot of our discussion was that we were going to stop using it for nodepool and start putting control plane servers there | 17:08 |
| clarkb | jeblair: ya we just write it out to the elements path with secrets interpolated from hiera or whereveer | 17:09 |
| matt-borland | krotscheck, cool | 17:09 |
| *** somerville32 has quit IRC | 17:09 | |
| pabelanger | fungi: Oh, maybe. I didn't sit in our that discussion in Austin | 17:09 |
| jeblair | clarkb: neat. i'd love to get our special workers running more standard images... | 17:09 |
| pabelanger | fungi: I have heard talk of control plan server however | 17:10 |
| *** johnny_ has joined #openstack-infra | 17:11 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Remove vexxhost-ca-ymq-1 from nodepool https://review.openstack.org/328395 | 17:12 |
| pabelanger | fungi: ^ | 17:12 |
| jeblair | clarkb, fungi: the static slaves have LANG="en_US.UTF-8" in /etc/default/locale, but language-pack-en is not installed on them which results in this error: http://logs.openstack.org/16/1665ad79cc479f0c9d744076cc7ad4abe2eca808/release/python-tackerclient-pypi-both-upload/f902134/console.html | 17:13 |
| clarkb | jeblair: we also need to becareful with image build logging there | 17:13 |
| jeblair | clarkb, fungi: our dynamic slaves have LANG=C.UTF-8, and also don't have language-pack-en installed | 17:13 |
| jeblair | clarkb: good point | 17:13 |
| *** yamamoto has joined #openstack-infra | 17:14 | |
| jeblair | apparently cloud-init set the locale on the static slaves | 17:14 |
| *** derekh has quit IRC | 17:14 | |
| fungi | jeblair: unsurprising, and unfortunate | 17:14 |
| *** ayoung has joined #openstack-infra | 17:14 | |
| jeblair | i'm confused on a number of points... 1) i thought jenkins was setting the locale to en_us.utf-8 as well, so i don't understand why we would get that error when running under zuul. 2) if that *is* the default locale, then i don't understand why https://review.openstack.org/328145 was needed | 17:15 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: Reduce Log Size https://review.openstack.org/328048 | 17:16 |
| *** pfallenop has quit IRC | 17:16 | |
| notmorgan | pabelanger, jeblair: ^ logging reduction with knobs/configurability | 17:16 |
| notmorgan | (for testing purposes) | 17:17 |
| *** pfallenop has joined #openstack-infra | 17:17 | |
| clarkb | jeblair: iirc when we switched to *-minimal based images the locale defaulted to C | 17:17 |
| clarkb | and caused some things to fail | 17:17 |
| fungi | yeah, i'm exploring now | 17:18 |
| jeblair | what is even printing that warning? | 17:18 |
| *** deadnull_ has quit IRC | 17:19 | |
| fungi | note that ssh will try to set the remote encoding to match your local encoding, for starters. is it possible this is being conveyed from the launcher? | 17:19 |
| *** Swami has joined #openstack-infra | 17:19 | |
| notmorgan | jeblair: ssh does that iirc. | 17:19 |
| notmorgan | i've run into this even with language packs installed - it makes a number of things cranky. | 17:19 |
| jeblair | notmorgan: that warning is emitted by ssh? | 17:19 |
| jeblair | (we're not running ssh there, so that's weird) | 17:20 |
| notmorgan | jeblair: well no, not ssh, sorry, the shell initialization | 17:20 |
| notmorgan | i just tend to see it on sshing into a system | 17:20 |
| notmorgan | or mosh or anything remote like that | 17:20 |
| pabelanger | diskimage-builder doesn't install language-pack-en by default, we have bindep do that | 17:20 |
| jeblair | notmorgan, fungi: zuul_runner starts with an empty environment, then sets locale things based on /etc/default/locale | 17:21 |
| pabelanger | so, we should likely update infra-package-needs to include it | 17:21 |
| notmorgan | but this is ubuntu being friendly when starting a shell up. | 17:21 |
| jeblair | so it should not be tainted by the incoming ssh connection | 17:21 |
| notmorgan | ssh sets LC_* | 17:21 |
| *** hasharAway has quit IRC | 17:21 | |
| fungi | trying to figure out what actually emits that error. if i interactively ssh into the proposal.slave node i don't see it | 17:21 |
| notmorgan | and so the locale is requested as UTF-8 | 17:21 |
| notmorgan | even though it isn't installed | 17:21 |
| notmorgan | so the shell complains and says "uh. you asked for something i can't provide" | 17:21 |
| notmorgan | you could prevent LC_* from being set and relying strictly on /etc/locale post-fact | 17:22 |
| jeblair | fungi: me neither, nor do i see it when running 'bash -l' | 17:22 |
| jeblair | notmorgan: that what i just said we do | 17:22 |
| notmorgan | it's an sshd_config option | 17:22 |
| *** sdake has quit IRC | 17:22 | |
| fungi | cloud-init is adding that warning, based on my minimal spelunking | 17:23 |
| jeblair | fungi: me neither, nor do i see it when running 'bash -l', which is what zuul_runner does. what's weird, is that if it were shell initialization, we should see it in subsequent shell segments | 17:23 |
| jeblair | fungi: is it a one-time warning? | 17:23 |
| notmorgan | fungi: ah cloud-init? blagh. | 17:23 |
| notmorgan | maybe we should make dib do "/var/lib/cloud/instance/locale-check.skip" | 17:23 |
| jeblair | no no no | 17:23 |
| notmorgan | :P | 17:23 |
| jeblair | please, this is very confusing, can we try to stay focused? | 17:24 |
| notmorgan | jeblair: no i am being serious. if it is cloud init and it's a safe warning | 17:24 |
| notmorgan | to ignore. ignore it. | 17:24 |
| jeblair | notmorgan: i am being serious too. i need to understand this. | 17:24 |
| jeblair | there are SEVERAL things that do not make sense that i outlined above | 17:24 |
| fungi | it's emitted by /etc/profile.d/Z99-cloud-locale-test.sh | 17:24 |
| *** yamada-h has joined #openstack-infra | 17:24 | |
| *** afred312 has joined #openstack-infra | 17:25 | |
| pabelanger | I would be in favor of moving http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/data/bindep-fallback.txt#n19 into http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/infra-package-needs/package-installs.yaml | 17:25 |
| fungi | well, the start of the discussion is that it doesn't make sense that we have a default locale set which is not installed (or may not be installed, i'm still verifying) | 17:25 |
| notmorgan | fungi: and this is occuring when we ssh into the system, if i understand correctly? | 17:25 |
| jeblair | yes, that i'm willing to write off as cloud-init being weird. what i don't understand is why we didn't see this under jenkins | 17:25 |
| pabelanger | fungi: right, not install by default. bindep does it | 17:25 |
| jeblair | notmorgan: no, we're seeing it in this job: http://logs.openstack.org/16/1665ad79cc479f0c9d744076cc7ad4abe2eca808/release/python-tackerclient-pypi-both-upload/f902134/console.html | 17:26 |
| pabelanger | jeblair: good point | 17:26 |
| notmorgan | jeblair: ok looking | 17:26 |
| *** kzaitsev_mb has joined #openstack-infra | 17:26 | |
| jeblair | and i don't understand why we needed https://review.openstack.org/328145 | 17:26 |
| fungi | pabelanger: no bindep on long-lived worker jobs (no sudo to root at all) | 17:26 |
| jeblair | i have said both of those things already | 17:26 |
| jeblair | and am annoyed at repeating myself | 17:26 |
| pabelanger | fungi: okay, now I understand. This is long-lived | 17:26 |
| notmorgan | jeblair: sorry i'm trying to play a little catchup. | 17:26 |
| fungi | i also don't (yet) undedstand why we need 328145 | 17:26 |
| jeblair | /home/jenkins/.cloud-locale-test.skip exists on release | 17:27 |
| jeblair | /etc/profile.d/Z99-cloud-locale-test.sh checks that before running | 17:27 |
| jeblair | # only show the message once | 17:27 |
| jeblair | : > ~/.cloud-locale-test.skip 2>/dev/null || : | 17:27 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: [Manila] Add new experimental job https://review.openstack.org/328364 | 17:27 |
| jeblair | so that's one mystery -- | 17:28 |
| jeblair | it *was* emitted by the bash login shell that began at the top of http://logs.openstack.org/16/1665ad79cc479f0c9d744076cc7ad4abe2eca808/release/python-tackerclient-pypi-both-upload/f902134/console.html | 17:28 |
| *** salv-orl_ has joined #openstack-infra | 17:28 | |
| jeblair | but then that file was written, and that's why we don't see it again later on | 17:28 |
| jeblair | (even though there are further bash login shells in there) | 17:28 |
| notmorgan | right | 17:29 |
| jeblair | we may never have seen it in jenkins because jenkins may not run a login shell | 17:29 |
| jeblair | so that may have literally been the first jenkins-user login shell on that host ever | 17:29 |
| *** yamada-h has quit IRC | 17:29 | |
| *** bhavik has quit IRC | 17:29 | |
| pabelanger | right, that make sense | 17:29 |
| notmorgan | possibly. | 17:29 |
| jeblair | that lets us comfortably reassure ourselves that we may in fact be replicating the same environment as jenkins | 17:29 |
| *** e0ne has joined #openstack-infra | 17:30 | |
| jeblair | right up until we ask why https://review.openstack.org/328145 was needed :( | 17:30 |
| jeblair | i think i want to have jenkins.o.o run a job on the proposal slave that print env | 17:31 |
| jeblair | and triple confirm that en_us.utf-8 is really what it was using | 17:31 |
| fungi | on proposal.slave.o.o, `locale -a` output includes en_US.utf8 | 17:31 |
| *** salv-orlando has quit IRC | 17:32 | |
| jeblair | fungi: but not en_US.UTF-8 | 17:32 |
| jeblair | could case be significant there? | 17:32 |
| fungi | i'm still hunting for whether LANG="en_US.UTF-8" will alias to it | 17:32 |
| *** degorenko is now known as _degorenko|afk | 17:32 | |
| fungi | it's not listed in /etc/locale.alias anyway | 17:32 |
| fungi | though my local system also uses en_US.UTF-8 instead of en_US.utf8 and seems to be working | 17:33 |
| *** sputnik13_ has joined #openstack-infra | 17:33 | |
| *** hashar has joined #openstack-infra | 17:34 | |
| *** dprince has quit IRC | 17:34 | |
| notmorgan | fungi: same, all my systems use en_US.UTF-8 and it aliases/works | 17:34 |
| notmorgan | fungi: just checked ubuntu 14.04, 16.04 and tumbleweed. | 17:34 |
| pabelanger | jeblair: what creates .cloud-locale-test.skip in the homedir? It's was last updated yesterday | 17:35 |
| *** dprince has joined #openstack-infra | 17:35 | |
| jeblair | pabelanger: /etc/profile.d/Z99-cloud-locale-test.sh | 17:35 |
| notmorgan | pabelanger: the z99-cloud-locale-test.sh | 17:35 |
| *** salv-orl_ has quit IRC | 17:35 | |
| pabelanger | thanks | 17:35 |
| fungi | locale -a also mentions C.UTF-8 as supported on that node | 17:36 |
| *** salv-orlando has joined #openstack-infra | 17:36 | |
| fungi | i'm checkign to see whether release.slave is consistent (that's where the pypi upload job from that log ran) | 17:36 |
| fungi | seems consistent with proposal.slave | 17:37 |
| *** xarses_ is now known as xarses | 17:37 | |
| *** Hunner has quit IRC | 17:37 | |
| *** bmjen has quit IRC | 17:38 | |
| fungi | aha, locale -v -a | 17:38 |
| jeblair | here is the output of env on release.slave via jenkins: http://paste.openstack.org/show/510659/ | 17:38 |
| jeblair | LANG=en_US.UTF-8 | 17:39 |
| notmorgan | jeblair: looks correct to me. | 17:39 |
| openstackgerrit | Min Pae proposed openstack-infra/project-config: Add cue periodic jobs with oslo master https://review.openstack.org/325032 | 17:39 |
| fungi | note that the error message includes quoted quotes. could that be throwing off the check at login? | 17:40 |
| fungi | testing now | 17:40 |
| jeblair | oh! | 17:40 |
| jeblair | zuul_runner is probably setting LANG to a *quoted* string | 17:40 |
| notmorgan | OH. | 17:40 |
| pabelanger | fungi: Good thinking | 17:40 |
| jeblair | its parsing of that file is very naive | 17:41 |
| *** yamamoto has quit IRC | 17:41 | |
| fungi | yeah, the ideal parser for that file is a shell. though that may not be a safe/clean parsing solution | 17:41 |
| jeblair | and that would explain why https://review.openstack.org/328145 was necessary | 17:41 |
| jeblair | fungi: yeah... i'm open to ideas on how to implement that | 17:41 |
| fungi | i'd be inclined to go the ugly route with line.split('=')[1].strip('"') | 17:42 |
| jeblair | maybe we have zuul runner start with a clean env, then run/source all the files we know about, then print 'env', then parse the output from that? | 17:42 |
| *** tkelsey has quit IRC | 17:42 | |
| jeblair | fungi: yeah, all we're missing from that is the strip('"') | 17:43 |
| jeblair | the rest is basically what we're doing | 17:43 |
| jeblair | (we ignore comments now! also, we ignore the latter part of strings with "#" in them! :) | 17:43 |
| *** gyee has joined #openstack-infra | 17:44 | |
| *** mtanino has quit IRC | 17:44 | |
| *** cody-somerville has joined #openstack-infra | 17:44 | |
| pabelanger | \o/ we figured it out | 17:44 |
| clarkb | what about strippibg '? | 17:44 |
| jeblair | could be handy | 17:44 |
| fungi | clarkb: i was pondering that as well. dunno if we'll hit that case but can't hurt and it's not much more code | 17:45 |
| *** kushal has quit IRC | 17:45 | |
| *** kzaitsev_mb has quit IRC | 17:45 | |
| jeblair | options on the table: a) hack more on our hacky parser; b) run bash+env to try to get the 'right' answer | 17:45 |
| *** nadya has joined #openstack-infra | 17:45 | |
| jeblair | anything else? any preferences? | 17:45 |
| *** stajkowski has quit IRC | 17:46 | |
| fungi | if this is going to live on in zuul v3, then i vote for option b | 17:46 |
| jeblair | i honestly have no idea at this point | 17:46 |
| fungi | if this is just to limp us along until zuul v3, option a | 17:46 |
| fungi | long term, i agree that having our runner be able to build an arbitrary calling environment from a pile of files (system-supplied or otherwise) is useful | 17:47 |
| jeblair | how to run things and get the "console logs" that we know and love is one of the remaining unanswered questions of v3 | 17:47 |
| pabelanger | what about updating our shell builder to prepend the 'source envfile' command | 17:48 |
| pabelanger | for the shell script | 17:48 |
| *** jbernard has quit IRC | 17:48 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT MERGE TESTING https://review.openstack.org/316436 | 17:48 |
| notmorgan | jeblair: i would prefer bash+env | 17:49 |
| jeblair | pabelanger: so "bash -l -c 'source /etc/default/locale; /tmp/shell_command_from_job'" ? | 17:49 |
| notmorgan | fungi: ++ | 17:49 |
| *** _tzn has quit IRC | 17:49 | |
| pabelanger | jeblair: I was thinking adding it right to /tmp/shell_command_from_job if possible. | 17:49 |
| pabelanger | just looking at code now | 17:49 |
| fungi | pabelanger: or is there an ansible option to tell it to load some environment file(s)? | 17:49 |
| pabelanger | fungi: Oh, maybe | 17:49 |
| fungi | but yeah, that would probably work fine | 17:49 |
| jeblair | pabelanger: we can not be certain that is actually a bash shell script | 17:49 |
| *** mriedem1 has joined #openstack-infra | 17:49 | |
| *** e0ne has quit IRC | 17:50 | |
| jeblair | it could actually be #!/usr/bin/env python | 17:50 |
| notmorgan | fungi: something like ~/profile.d/<env files> ? | 17:50 |
| jeblair | (though no one has ever done that :) | 17:50 |
| pabelanger | jeblair: right, thats what I am thinking too | 17:50 |
| *** mriedem has quit IRC | 17:51 | |
| jeblair | well, for the record, what we're actually doing here is recreating what the pam_env module does | 17:51 |
| notmorgan | jeblair: true | 17:51 |
| pabelanger | ansible does have environment for playbooks, but don't think you can reference a file | 17:51 |
| *** markusry has quit IRC | 17:52 | |
| pabelanger | you'd have to list out your vars | 17:52 |
| jeblair | that's what actually sets the locale, and is why we're not automatically getting the benefit of this | 17:52 |
| fungi | jeblair: good point, perhaps we can just leverage pam? | 17:52 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet/beaker: add xenial-nv jobs to all modules https://review.openstack.org/328433 | 17:52 |
| jeblair | (because the closest pam_env invocation comes with all the ansible baggage) | 17:52 |
| EmilienM | pabelanger: got beaker/xenial jobs working on puppet-keystone ! | 17:52 |
| *** mriedem1 is now known as mriedem | 17:52 | |
| EmilienM | pabelanger: moving forward with other modules now: https://review.openstack.org/328433 -- the next and last step will be to gate on it once jobs are stable | 17:52 |
| jeblair | fungi: i thought about that, but the only way i could think to do that was using the test-pam-module program which needs another system package... seemed a bit much dependency-wise | 17:53 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet/beaker: add xenial-nv jobs to all modules https://review.openstack.org/328433 | 17:53 |
| *** jianghuaw has joined #openstack-infra | 17:53 | |
| fungi | fair enough | 17:53 |
| jeblair | fungi: or, i mean, maybe actually linking in pam | 17:53 |
| jeblair | which seemed even much more muchness | 17:53 |
| fungi | i agree that does seem a little heavyweight | 17:53 |
| *** permalac has quit IRC | 17:54 | |
| jeblair | maybe we could invoke 'login' or something | 17:54 |
| *** e0ne has joined #openstack-infra | 17:54 | |
| jeblair | oh, that needs root, that's the problem with that | 17:54 |
| *** cody-somerville has quit IRC | 17:54 | |
| fungi | or needs a password | 17:55 |
| fungi | right | 17:55 |
| pabelanger | jeblair: I wonder if we could use INI file lookup from ansible some how in zuul_launcher: http://docs.ansible.com/ansible/playbooks_lookups.html#the-ini-file-lookup | 17:55 |
| *** lezbar has quit IRC | 17:55 | |
| pabelanger | maybe not | 17:55 |
| jeblair | pabelanger: well, we're past ansible at this point | 17:55 |
| *** bpokorny_ has joined #openstack-infra | 17:56 | |
| pabelanger | Ya, I was looking in ansiblelauncher | 17:56 |
| *** shardy has quit IRC | 17:56 | |
| *** vgridnev has quit IRC | 17:56 | |
| *** Sukhdev has joined #openstack-infra | 17:56 | |
| *** vgridnev_ has joined #openstack-infra | 17:56 | |
| *** dtardivel has quit IRC | 17:57 | |
| jeblair | i'll see how hard bash+env is to implement | 17:57 |
| *** vgridnev_ has quit IRC | 17:57 | |
| *** jbernard has joined #openstack-infra | 17:57 | |
| *** bpokorny_ has quit IRC | 17:57 | |
| jeblair | and if it takes more than a few minutes, i'll hack in some quote stripping :) | 17:57 |
| *** tzn has joined #openstack-infra | 17:58 | |
| *** asettle has joined #openstack-infra | 17:58 | |
| notmorgan | jeblair: wfm. | 17:58 |
| *** EricGonczer_ has quit IRC | 17:58 | |
| fungi | sounds good | 17:58 |
| *** bpokorny_ has joined #openstack-infra | 17:58 | |
| *** salv-orlando has quit IRC | 17:58 | |
| jeblair | fungi, pabelanger, notmorgan: thanks for your help, this was super-confusing :) | 17:59 |
| *** Ravikiran_K has quit IRC | 17:59 | |
| *** jbernard has quit IRC | 17:59 | |
| notmorgan | jeblair: happy to toss brain in on things like this. no guarantees my brain has good answers though :P so if it helped even a little. | 17:59 |
| *** EricGonczer_ has joined #openstack-infra | 17:59 | |
| *** salv-orlando has joined #openstack-infra | 17:59 | |
| *** bpokorny has quit IRC | 17:59 | |
| fungi | jeblair: i'll cop to still being mildly confused, but mostly grokking the situation now | 18:00 |
| *** javeriak has joined #openstack-infra | 18:00 | |
| *** asettle has quit IRC | 18:00 | |
| *** EricGonc_ has joined #openstack-infra | 18:00 | |
| *** gyee has quit IRC | 18:00 | |
| openstackgerrit | Matthew Treinish proposed openstack/os-testr: Add documentation for generate-subunit https://review.openstack.org/328352 | 18:00 |
| jeblair | fungi: i think the facts are: * jenkins set the locale to en_us.utf8. * zuul set the locale to "en_us.utf8" (with quotes!). * that broke the translation proposal jobs. * that also triggered the warning in the release job. * we only saw it once because it writes that file in ~. * when we parse the file correctly, zuul will set the locale to en_us.utf8, and everything will be okay. :) | 18:04 |
| *** EricGonczer_ has quit IRC | 18:04 | |
| jeblair | i will probably pick this back up after lunch | 18:05 |
| fungi | that matches my understanding | 18:05 |
| *** acoles is now known as acoles_ | 18:06 | |
| fungi | also .utf8 vs .UTF-8 seems to be handled automagically as best i can tell | 18:06 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-openstackci: Move subunit_worker.pp from system-config https://review.openstack.org/328438 | 18:06 |
| *** mtanino has joined #openstack-infra | 18:06 | |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/system-config: Pull out config_file parameter from subunit worker class https://review.openstack.org/328439 | 18:06 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/system-config: Move subunit_worker.pp into puppet-openstackci https://review.openstack.org/328440 | 18:06 |
| pabelanger | EmilienM: good work | 18:07 |
| EmilienM | pabelanger: yeah we're close to use 100% xenial on master and well keep trusty jobs for stable/liberty & mitaka jobs | 18:07 |
| jeblair | pabelanger: oh, one reason not to put the source commands in our invocation -- i do want zuul/users to be able to override these (and not just by putting them in the job). so if someone passes a LANG arg to a job, i want it to have an effect | 18:08 |
| jeblair | pabelanger: so starting with system env vars, then layering zuul env vars, then running the content is the way i think | 18:08 |
| pabelanger | jeblair: Yup, that makes sense | 18:08 |
| *** EricGonc_ has quit IRC | 18:09 | |
| fungi | huh, unit tests for 328382 failed once it was in the gate? | 18:09 |
| pabelanger | okay, have we address our current issues with zuulv25? I'm not working on anything ATM | 18:10 |
| pabelanger | fungi: ya, I think we have a race condition is one of our tests | 18:13 |
| *** EricGonczer_ has joined #openstack-infra | 18:13 | |
| fungi | trying to decipher the log there, looks like issues connecting to gearman? | 18:13 |
| *** yamamoto has joined #openstack-infra | 18:13 | |
| pabelanger | I think notmorgan said he was looking into it? | 18:13 |
| *** sdague has quit IRC | 18:13 | |
| *** adreznec has quit IRC | 18:15 | |
| *** jianghuaw has quit IRC | 18:15 | |
| openstackgerrit | Henry Gessau proposed openstack-infra/project-config: Add non-voting coverage check job for neutron-lib https://review.openstack.org/328445 | 18:15 |
| *** gongysh has quit IRC | 18:15 | |
| *** dimtruck has quit IRC | 18:15 | |
| *** dimtruck has joined #openstack-infra | 18:16 | |
| *** adreznec has joined #openstack-infra | 18:17 | |
| *** nadya has quit IRC | 18:17 | |
| *** fergal_ has quit IRC | 18:20 | |
| *** jbernard has joined #openstack-infra | 18:20 | |
| *** fergal_ has joined #openstack-infra | 18:21 | |
| *** roxanaghe has quit IRC | 18:21 | |
| *** fergal_ has quit IRC | 18:21 | |
| *** lezbar has joined #openstack-infra | 18:22 | |
| *** e0ne has quit IRC | 18:22 | |
| *** rbrndt has quit IRC | 18:23 | |
| *** e0ne has joined #openstack-infra | 18:23 | |
| notmorgan | fungi: hmm? | 18:24 |
| *** vhosakot has quit IRC | 18:25 | |
| notmorgan | pabelanger, fungi: I've been tyring to dig into this but the lack of logs makes it hard. bascially it's dieing somewhere in a bad way | 18:26 |
| notmorgan | i've been trying to figure out the special method to get it to spew some logs for us when it fails. | 18:26 |
| fungi | notmorgan: "lack of logs" isn't something i would normally accuse zuul's unit tests of | 18:26 |
| fungi | but i suppose not the right logs | 18:26 |
| notmorgan | in these failure cases, i can't duplicate locally and i can't get logs from the gate :P | 18:27 |
| *** gyee has joined #openstack-infra | 18:27 | |
| *** tzn has quit IRC | 18:27 | |
| notmorgan | i ran the tests for ~3hrs in a loop, zero failures (load, no load, excessive io load, etc) | 18:27 |
| *** amoralej is now known as amoralej|off | 18:28 | |
| notmorgan | fungi: these test failures net zero logs because of the way the timer fixture kills it. | 18:28 |
| fungi | ahh, got it | 18:28 |
| fungi | can we trigger a thread dump before killing the process? | 18:29 |
| fungi | would that help? | 18:29 |
| pabelanger | I don't how to control it, but it would be cool to add some logic into zuul if we new a job was randomly failing to check something first (yaml or something) and auto flag the node as hold on failure | 18:29 |
| notmorgan | fungi: that is my next test. going to subclass the timer | 18:29 |
| pabelanger | rather then having somebody race to manually hold it | 18:31 |
| notmorgan | fungi: and try and make it exlpode in a way that dumps sys._current_frames | 18:31 |
| fungi | pabelanger: yes, now that jenkins is out of the picture that becomes more possible | 18:31 |
| notmorgan | pabelanger: that would be cool too. | 18:31 |
| *** ilyashakhat has joined #openstack-infra | 18:31 | |
| *** tzn has joined #openstack-infra | 18:31 | |
| fungi | "hold the next node matching this job name and status code" | 18:31 |
| fungi | if zmq were sticking around in zuul/nodepool v3 we could add some details to the oncomplete event | 18:31 |
| notmorgan | fungi: even without zmq, totally doable | 18:32 |
| notmorgan | but it'll be a different mechanism / more work i'm sure. | 18:32 |
| fungi | notmorgan: right, was just explaining how i'd expect it to work in a v2.x landscape | 18:33 |
| * notmorgan needs to spend some time on this stupid zuul gate-failure | 18:33 | |
| fungi | presumably nodepool will be able to find that info in zookeeper or however that end sup working | 18:33 |
| fungi | since it's more of a convenience/wishlist thing i wouldn't want anyone to work on it until v3 is baked | 18:34 |
| notmorgan | hmm. maybe i can just have someone give me a node that is *like* one that failed and i can just run the tox commend in a loop in the gate | 18:34 |
| AJaeger | jeblair: the cinder proposal job looks fine - it gives the same content as yesterday (no changed strings) and that is just perfect. | 18:34 |
| *** ilyashakhat has quit IRC | 18:34 | |
| AJaeger | jeblair: Can you abandon *all* open translation jobs? https://review.openstack.org/#/q/status:open+topic:zanata/translations,n,z | 18:34 |
| notmorgan | since it only ever seems to fail on our gate nodes. | 18:35 |
| fungi | notmorgan: i can totally do that. maybe hold the next one that runs that job and then add your keys after it finishes regardless of result? | 18:35 |
| notmorgan | fungi: let me get some caffiene | 18:36 |
| AJaeger | jeblair: problem is: If there is no change - like with cinder in your retrigger - then nothing will be proposed and those 40+ jobs will be there for quite some time and might confuse people. If we abandan them, the next run will create new ones | 18:36 |
| notmorgan | i think i need that first. :) i might ask for that on monday instead. | 18:36 |
| notmorgan | it's coming up on the weekend and i dunno if i want to commit to poking at this on a friday afternoon ;) | 18:36 |
| fungi | notmorgan: totally sensible position' | 18:36 |
| *** hongbin has quit IRC | 18:36 | |
| fungi | notmorgan: fwiw, my recheck of 328382 failed that job again | 18:37 |
| notmorgan | yup | 18:37 |
| notmorgan | i saw | 18:37 |
| fungi | it just hasn't reported back into gerrit yet | 18:37 |
| notmorgan | i've been battling this a bunch the last few days. | 18:37 |
| *** claudiub|2 has quit IRC | 18:37 | |
| notmorgan | it's my "next thing to fix with zuul" now that i have py3 things that can pass gate and a log-reduction patch posted | 18:37 |
| jeblair | AJaeger: i think we should do that, but maybe some other infra-root could? i will have to wait until after lunch | 18:37 |
| notmorgan | also because i noticed how bad it was this week doing this work | 18:37 |
| fungi | jeblair: AJaeger: to can absolutely abandon those changes | 18:38 |
| *** sree has joined #openstack-infra | 18:38 | |
| jeblair | notmorgan: well, i'm hoping for a systemic solution to this problem | 18:38 |
| AJaeger | that would be great fungi! Take the query and just abandon everything, please | 18:38 |
| notmorgan | jeblair: totally. | 18:38 |
| jeblair | notmorgan: like, no matter what happens, in the future, if a unit test exceeds the timeout, we need to get logs | 18:39 |
| jeblair | holding a node is not something we should ever have to do | 18:39 |
| notmorgan | jeblair: but if we can at least identify the current issue and handle it - lets do that -- and work on the bigger fix as we go | 18:39 |
| jeblair | no one will work on the bigger fix | 18:39 |
| notmorgan | jeblair: or at the very least get an interim stack-dump-of-threads in zuul | 18:39 |
| jeblair | this *will* happen again | 18:39 |
| jeblair | and we will repeat the same process | 18:39 |
| notmorgan | jeblair: don't underestimate my "i'm just annoyed at this here is fixes for all the things" drive | 18:40 |
| notmorgan | jeblair: why do you think i took over ldappool :P | 18:40 |
| jeblair | notmorgan: i'm excited by it :) | 18:40 |
| notmorgan | jeblair: fixing testr to not break between python versions (seriously .testrespository SHOULD be python-version specific) is also on my short list | 18:40 |
| AJaeger | regarding utf-8 vs utf8: glibc does some normalization AFAIR - and that would explain the behaviour you see | 18:41 |
| *** dimtruck has quit IRC | 18:41 | |
| *** cody-somerville has joined #openstack-infra | 18:41 | |
| *** cody-somerville has joined #openstack-infra | 18:41 | |
| notmorgan | jeblair: but i see "fixing zuul so we can land code" as somewhat more important today -- you know... for $reasons$ | 18:41 |
| notmorgan | :P | 18:41 |
| notmorgan | even if it's short-term | 18:41 |
| *** dizquierdo has quit IRC | 18:42 | |
| *** cody-somerville has quit IRC | 18:42 | |
| jeblair | notmorgan: on a related issue -- about the log size -- maybe we could post-process the subunit log | 18:42 |
| jeblair | notmorgan: after everything is done, strip the logs from the sucessful tests | 18:42 |
| notmorgan | jeblair: oh strip out the su.. yeah | 18:42 |
| notmorgan | that was where i was headed since we run --subunit to extrac the data | 18:43 |
| *** dimtruck has joined #openstack-infra | 18:43 | |
| jeblair | AJaeger: we think we understand what caused the utf8 issues, and we'll have a fix for it in zuul soon, though i think we should just leave the LANG setting in that script. | 18:43 |
| jeblair | AJaeger: we also think we know what's going on with the -docs jobs failures; that fix has landed but isn't running yet (in a few hours hopefully) | 18:44 |
| AJaeger | jeblair: I skimmed backlog, thanks for summarizing! | 18:44 |
| jeblair | cool | 18:44 |
| jeblair | back after lunch | 18:45 |
| AJaeger | jeblair: enjoy lunch and thanks for solving all these riddles! | 18:46 |
| *** EricGonc_ has joined #openstack-infra | 18:46 | |
| *** julim has quit IRC | 18:46 | |
| *** jcoufal has quit IRC | 18:46 | |
| * AJaeger sees lots of abandoned changes showing up in his inbox - thanks fungi! | 18:47 | |
| fungi | still quite a few to go. should be done in a few minutes | 18:47 |
| *** roxanagh_ has joined #openstack-infra | 18:47 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet/logs: use mktemp to store puppet-openstack-integration https://review.openstack.org/328453 | 18:48 |
| EmilienM | pabelanger: would something like this work? ^ | 18:48 |
| AJaeger | fungi: A lot to -1 this morning - I didn't want anybody to merge them quickly ;) | 18:48 |
| *** jcoufal has joined #openstack-infra | 18:48 | |
| jeblair | notmorgan, fungi: wait, what do you want to get from a held zuul unit test node? | 18:49 |
| *** EricGonczer_ has quit IRC | 18:49 | |
| jeblair | notmorgan, fungi: the problem is that the hard timeout is killing the test process (including zuul), which causes it to lose the accumulated logs | 18:50 |
| *** EricGonc_ has quit IRC | 18:50 | |
| *** mhickey has joined #openstack-infra | 18:50 | |
| jeblair | by the time it fails, there will be nothing running | 18:50 |
| fungi | jeblair: increased likelihood of being able to reproduce the failure condition i guess? though yeah testr is going to have killed it even if you preproduce, and we already collect all relevant artifacts | 18:50 |
| pabelanger | EmilienM: no, because dest needs to be . for install-distro-packages to work. install-distro-package.sh excepts other-requirement.txt to exist in the local path | 18:50 |
| pabelanger | EmilienM: you've just checked the clone from /home/jenkins/workspace/openstack/puppet-openstack-integration to /home/jenkins/workspace/tmp23as3 | 18:51 |
| pabelanger | it needs to be /home/jenkins/workspace | 18:51 |
| fungi | all i can figure is maybe he wanted to try bumping the testr timeout waaaay high and then manually strace the process once it's running too long | 18:51 |
| *** EricGonczer_ has joined #openstack-infra | 18:52 | |
| *** andymaier has quit IRC | 18:52 | |
| fungi | not much else being able to reproduce locally will be likely to reveal | 18:52 |
| *** tzn has quit IRC | 18:52 | |
| *** ihrachys has joined #openstack-infra | 18:52 | |
| notmorgan | jeblair: basically my thought is i can't get it to fail here, it fails in test nodes in gate | 18:52 |
| pabelanger | EmilienM: The proper fix would be to update install-distro-package.sh, adding an args to override the default location of other-requirements.txt | 18:52 |
| EmilienM | pabelanger: I don't want to revert really, I would prefer fixing it. So far it only affects our stable/liberty gate and it can wait a bit. We don't have much backports lately. | 18:52 |
| EmilienM | pabelanger: ok | 18:53 |
| notmorgan | jeblair: i'd figure out a node *like* the node it failed and run the tox stuff with a long timeout and poking to see what has happened | 18:53 |
| *** yolanda has quit IRC | 18:53 | |
| notmorgan | jeblair: but after hours of looking at tests running in a loop and zero failures here, i just can't determine what is actually different between local (VM, non-vm, etc) and the gate nodes. | 18:54 |
| EmilienM | pabelanger: ok but what path would you give it to? | 18:54 |
| jroll | sounds like zuul is unhappy? is that what I'm seeing here? http://logs.openstack.org/83/327083/4/check/gate-tempest-dsvm-ironic-agent_ssh/f6da2bd/console.html | 18:54 |
| notmorgan | jeblair: i'm going to try and get at least an active frame dump from the gate on failure so we can see where it's hung (subclassed timer) when it happens. | 18:54 |
| *** vhosakot has joined #openstack-infra | 18:54 | |
| pabelanger | EmilienM: path to your cloned repo | 18:54 |
| pabelanger | by default we expect . | 18:55 |
| EmilienM | ok | 18:55 |
| *** yamamoto has quit IRC | 18:55 | |
| EmilienM | pabelanger: because now, where does it look for? | 18:55 |
| *** rbrndt has joined #openstack-infra | 18:55 | |
| * notmorgan has a second idea how to extract the data as well. | 18:55 | |
| pabelanger | EmilienM: the current directory | 18:56 |
| *** lezbar has quit IRC | 18:56 | |
| *** edmondsw has quit IRC | 18:56 | |
| EmilienM | pabelanger: ok so `cd puppet-openstack-integration`, right? | 18:56 |
| pabelanger | no | 18:56 |
| pabelanger | I can get a patch up | 18:56 |
| EmilienM | thanks | 18:56 |
| *** ihrachys has quit IRC | 18:57 | |
| *** bpokorny_ has quit IRC | 18:57 | |
| *** bpokorny has joined #openstack-infra | 18:57 | |
| *** e0ne has quit IRC | 18:59 | |
| *** bpokorny has quit IRC | 18:59 | |
| fungi | AJaeger: all abandoned now | 18:59 |
| AJaeger | thanks a lot, fungi! | 19:00 |
| AJaeger | that's the first time I remember that this query is empty ;) | 19:00 |
| *** hongbin has joined #openstack-infra | 19:01 | |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/infra-specs: Add wiki modernization spec https://review.openstack.org/328455 | 19:03 |
| *** yolanda has joined #openstack-infra | 19:03 | |
| pleia2 | pabelanger, fungi ^ a day late, but here we are, comment away! | 19:03 |
| *** amrith is now known as _amrith_ | 19:06 | |
| AJaeger | pleia2: you need to add it to index file | 19:06 |
| fungi | in related news, since some of the wikispammers i found weren't blocked yet, i added account blocking to my cleanup script http://paste.openstack.org/show/510666 | 19:07 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update FSTrigger plugin https://review.openstack.org/328456 | 19:07 |
| pleia2 | AJaeger: thanks | 19:07 |
| *** mhickey has quit IRC | 19:08 | |
| fungi | i'm gonna go grab a bite to eat, will review wiki spec as soon as i get back from that | 19:08 |
| fungi | bbiab | 19:08 |
| *** sputnik13 has joined #openstack-infra | 19:09 | |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/infra-specs: Add wiki modernization spec https://review.openstack.org/328455 | 19:09 |
| *** sputnik13 has quit IRC | 19:10 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Use custom bindep isntaller for puppet jobs https://review.openstack.org/328457 | 19:10 |
| pabelanger | EmilienM: that should work ^ | 19:11 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Use custom bindep installer for puppet jobs https://review.openstack.org/328457 | 19:11 |
| EmilienM | pabelanger: what about beaker jobs? | 19:11 |
| pabelanger | EmilienM: ya, we'll need to add it there too | 19:12 |
| pabelanger | EmilienM: which job is failing? | 19:12 |
| EmilienM | pabelanger: puppet-openstack-integration on stable/liberty | 19:13 |
| *** sambetts is now known as sambetts|afk | 19:13 | |
| * AJaeger wishes everybody a great weekend | 19:14 | |
| pabelanger | EmilienM: I am not sure which job that is | 19:15 |
| pabelanger | EmilienM: can you post a log of it failing | 19:15 |
| EmilienM | pabelanger: https://review.openstack.org/328015 | 19:15 |
| EmilienM | sure logs are coming: | 19:15 |
| anteaya | AJaeger: thanks you too | 19:15 |
| EmilienM | http://logs.openstack.org/15/328015/2/check/gate-puppet-openstack-integration-3-scenario001-tempest-centos-7/c35e1fc/console.html#_2016-06-10_02_44_29_617 | 19:15 |
| pabelanger | EmilienM: the review I linked should fix that | 19:16 |
| EmilienM | pabelanger: excellent | 19:16 |
| EmilienM | pabelanger: thanks :) | 19:17 |
| pabelanger | it _should_ fix it, I haven't tested it | 19:17 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update support for BuildResultTrigger plugin https://review.openstack.org/321619 | 19:20 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/infra-specs: Add wiki modernization spec https://review.openstack.org/328455 | 19:20 |
| EmilienM | pabelanger, AJaeger, can we get https://review.openstack.org/#/c/328433/ before the week end please? | 19:23 |
| *** sarob has quit IRC | 19:23 | |
| pabelanger | EmilienM: working over the weekend? | 19:25 |
| *** ddieterly has joined #openstack-infra | 19:25 | |
| EmilienM | pabelanger: enjoying free CI :-) | 19:25 |
| *** sdague has joined #openstack-infra | 19:26 | |
| *** lezbar has joined #openstack-infra | 19:26 | |
| *** sputnik13_ has quit IRC | 19:26 | |
| pabelanger | gate is moving fast today | 19:26 |
| *** ilyashakhat has joined #openstack-infra | 19:27 | |
| *** PollyZ has quit IRC | 19:28 | |
| *** sarob has joined #openstack-infra | 19:28 | |
| EmilienM | people are watching european football maybe | 19:28 |
| EmilienM | go France! :-) (they are playing now) | 19:28 |
| mordred | EmilienM: they are indeed! | 19:29 |
| mordred | EmilienM: I recommend watching the game in a beer cave in prague, btw | 19:29 |
| *** PollyZ has joined #openstack-infra | 19:30 | |
| EmilienM | lol | 19:31 |
| EmilienM | mordred: what happens in Prague... | 19:31 |
| *** lezbar has quit IRC | 19:31 | |
| mordred | EmilienM: ... probably involves beer? :) | 19:31 |
| *** sarob has quit IRC | 19:33 | |
| *** tzn has joined #openstack-infra | 19:36 | |
| *** tzn has quit IRC | 19:36 | |
| mtreinish | jeblair: is build_newrev one of things the new zuul stuff is adding: http://status.openstack.org/openstack-health/#/?groupKey=build_newrev | 19:36 |
| *** cody-somerville has joined #openstack-infra | 19:37 | |
| dougwig | hello infra. when someone has time, can you make myself or njohnston an op in #openstack-fwaas, so we can setup the acls and such for openstack bot access? (or just do that directly, whichever is the least work for you folks.) ? | 19:38 |
| pleia2 | ah, it's not registered so I think jeblair will need to take care of that | 19:39 |
| *** csomerville has joined #openstack-infra | 19:39 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-openstack_health: Add build_newrev to the list of ignored keys https://review.openstack.org/328467 | 19:39 |
| mtreinish | pleia2: ^^^ that should be a fast review | 19:39 |
| pleia2 | mtreinish: thumbs up | 19:41 |
| *** PollyZ has quit IRC | 19:41 | |
| anteaya | dougwig: hey there, do you know who created the -fwaas channel at all? | 19:41 |
| njohnston | dougwig: Thanks! | 19:41 |
| dougwig | anteaya: no, i don't. the only thing i know is that infra can reclaim #openstack-. i don't know if that's too much work, though. | 19:41 |
| anteaya | dougwig: yes it is possible for jeblair to do that, we just like to ensure the easy stuff has been done first | 19:42 |
| *** cody-somerville has quit IRC | 19:42 | |
| njohnston | anteaya: sc68cal might know (about #openstack-fwaas) | 19:42 |
| anteaya | dougwig: for instance if you knew who it was and they included you, one less thing on jeblair's plate | 19:43 |
| pleia2 | unless everyone can be convinced to leave the channel, since it's unregistered only one who can do anything is jeblair | 19:43 |
| anteaya | pleia2: ah thank you | 19:43 |
| notmorgan | mordred: ++ can we just say it all involves beer? not just what is in prague? | 19:43 |
| dougwig | it's usually impossible to get everyone to leave, because of auto-login bots. | 19:43 |
| pleia2 | dougwig: yep | 19:43 |
| njohnston | yep | 19:43 |
| *** aviau has quit IRC | 19:44 | |
| *** aviau has joined #openstack-infra | 19:44 | |
| * notmorgan needs to figure out the weechat color that got assigned to dougwig's nick and remove it from his config... i want to see when dougwig is talking :) | 19:44 | |
| dougwig | ooh, so i can say anything about notmorgan right now. :) | 19:44 |
| *** Sukhdev has quit IRC | 19:45 | |
| notmorgan | dougwig: as long as you don't say my name, i can't read the nick talking. | 19:45 |
| notmorgan | dougwig: but i cna see what was said. | 19:45 |
| anteaya | the silent colour | 19:46 |
| anteaya | wish I could assign that at my discretion | 19:46 |
| njohnston | <blink>dougwig</blink> | 19:46 |
| anteaya | ha ha ha | 19:46 |
| notmorgan | anteaya: lol | 19:48 |
| notmorgan | dougwig: mind saying something now? | 19:48 |
| notmorgan | oh nvm i see your color! | 19:48 |
| notmorgan | turns out 073 (blue) is REALLY hard to read on a semi-transparent-black/grey background | 19:49 |
| anteaya | nice | 19:49 |
| anteaya | good ole 073 | 19:49 |
| notmorgan | but anteaya's name is no longer a lovely cyan but is a green. while also good, just not as good. | 19:49 |
| anteaya | that is my alternate name for you dougwig | 19:49 |
| anteaya | ah, glad I was cyan for a bit | 19:49 |
| anteaya | and glad I am a good green now too | 19:49 |
| dougwig | ooh, my nick is rgb that looks like octal. | 19:49 |
| anteaya | I was green in my morning meditation, so that fits too | 19:50 |
| notmorgan | anteaya: for the last few months actually :P i try not to change nick color list cause it shuffles everyone's colors.... and sadly, i map nick+color not just nick as who i'm talking to in irc | 19:50 |
| dougwig | i've been varying shades of red all week, so i'll be happy to be blue. | 19:50 |
| *** bpokorny has joined #openstack-infra | 19:50 | |
| notmorgan | dougwig: you're a yellow-ish color now in my weechat. | 19:50 |
| notmorgan | or yellow-green-ish | 19:50 |
| dougwig | like baby vomit? | 19:51 |
| dougwig | :) | 19:51 |
| notmorgan | nah, thats more pea-green-pureee color | 19:51 |
| notmorgan | or pea+squash green | 19:51 |
| anteaya | dougwig: ha ha ha | 19:51 |
| notmorgan | this is more... yellow+green in 256colors | 19:51 |
| *** sdake has joined #openstack-infra | 19:52 | |
| anteaya | notmorgan: I understand | 19:52 |
| dougwig | we clearly need a pasted screenshot. | 19:52 |
| anteaya | looking forward to getting my weechat server up again | 19:52 |
| anteaya | clearly | 19:52 |
| *** zul has quit IRC | 19:53 | |
| *** zul has joined #openstack-infra | 19:53 | |
| *** ilyashakhat has quit IRC | 19:54 | |
| *** jcoufal has quit IRC | 19:54 | |
| *** PollyZ has joined #openstack-infra | 19:55 | |
| mgagne | is there a way to tell Gerrit to force replicate a project? | 19:56 |
| anteaya | mgagne: can you share why you want that? | 19:58 |
| mgagne | anteaya: private setup with borked replication =) | 19:58 |
| anteaya | ah | 19:59 |
| clarkb | mgagne: you can there us a replicate command that can do all projects or specific projects | 19:59 |
| clarkb | or push a new change to the project | 19:59 |
| anteaya | dhellmann: can I trouble you to comment in reply to Zara's last comment on https://storyboard.openstack.org/#!/story/2000610 with regards to storyboard and the conversation I had with you, I believe last week | 20:00 |
| *** hongbin has quit IRC | 20:00 | |
| mgagne | can't find the replicate command o_O, does the change need to be merged? | 20:00 |
| clarkb | mgagne: no doesnt need to be merged. its in a plugin now so not part of the gerrit docs | 20:00 |
| EmilienM | anteaya: if you have some time, can you review https://review.openstack.org/#/c/328433/ please? | 20:00 |
| clarkb | we have it documented somehwere | 20:01 |
| dhellmann | anteaya : looking | 20:01 |
| anteaya | dhellmann: thank you | 20:01 |
| mgagne | clarkb: yea, was trying to find it in infra docs, couldn't find it | 20:01 |
| mgagne | unless I checked in wrong section | 20:01 |
| pleia2 | anteaya: thanks for reviewing the wiki spec so quickly | 20:01 |
| anteaya | pleia2: thanks for writing it! | 20:02 |
| dhellmann | anteaya : I don't know anyone named SotK and I don't remember any conversation about search other than the one you and I had. | 20:02 |
| anteaya | dhellmann: thanks, could you say as much in a reply to Zara's comment? | 20:03 |
| anteaya | dhellmann: thanks for looking | 20:03 |
| dhellmann | ok | 20:03 |
| *** hongbin has joined #openstack-infra | 20:03 | |
| clarkb | mgagne: https://review.openstack.org/plugins/replication/Documentation/cmd-start.html | 20:03 |
| anteaya | dhellmann: thank you | 20:04 |
| anteaya | EmilienM: can I get a link to a puppet-keystone change with this job running on it? | 20:04 |
| mgagne | clarkb: yea, found that do. I think I don't have the right permission then, replication isn't found | 20:04 |
| EmilienM | anteaya: https://review.openstack.org/#/c/320554/ | 20:04 |
| anteaya | EmilienM: thank you | 20:05 |
| EmilienM | anteaya: gate-puppet-keystone-puppet-beaker-rspec-ubuntu-xenial-nv SUCCESS | 20:05 |
| *** kzaitsev_mb has joined #openstack-infra | 20:05 | |
| *** rhallisey has quit IRC | 20:05 | |
| *** baoli_ has quit IRC | 20:06 | |
| notmorgan | anteaya, dougwig: http://i.imgur.com/p66eqCn.png | 20:06 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: WIP: Add DB API method to get run wall times by run_metadata https://review.openstack.org/328478 | 20:06 |
| *** julim has joined #openstack-infra | 20:06 | |
| jeblair | mtreinish: no, newrev is something that is present only on jobs triggered by ref-updated events | 20:07 |
| jeblair | mtreinish: (as is oldrev, and i think, refname) | 20:07 |
| mtreinish | jeblair: hmm, I wonder why we hadn't seen that before then | 20:07 |
| anteaya | EmilienM: so here's the thing I am seeing the puppet-keystone job is experimental on puppet-keystone right? but is not in the experimental pipeline in this huge change that adds it to everything else | 20:07 |
| anteaya | EmilienM: am I seeing the change correctly? | 20:08 |
| EmilienM | anteaya: we already have the puppet-keystone job | 20:08 |
| EmilienM | we want it everywhere else | 20:08 |
| *** roxanagh_ has quit IRC | 20:08 | |
| pabelanger | jeblair: mind walking me through the restart process on zl02.o.o? Is it a matter of running zuul graceful now (not that I am actually going to run the command) | 20:09 |
| anteaya | EmilienM: what I am saying is your test case put the job in the experimental pipeline but now propigates it out to other jobs not in the experimental pipeline | 20:09 |
| jeblair | pabelanger: yeah, i think graceful landed, so you can just issue 'zuul-launcher graceful' and it will eventually stop. | 20:09 |
| EmilienM | anteaya: everything stays in experimental pipeline | 20:10 |
| jeblair | pabelanger: that's not reversible though, so make sure that's what you want to do | 20:10 |
| EmilienM | anteaya: for now | 20:10 |
| mgagne | clarkb: will force merge of a change and see if the situation fixes itself | 20:10 |
| EmilienM | anteaya: my patch just creates jenkins jobs and run them on all puppet-* thanks to zuul layout | 20:10 |
| mtreinish | jeblair: oh, it's when AJaeger and I got the translation jobs in the db: http://paste.openstack.org/show/510671/ | 20:10 |
| jeblair | pabelanger: the reversible equivalent is 'zuul-launcher pause'; 'zuul-launcher release' | 20:10 |
| jeblair | mtreinish: yeah, those are in post, so are ref-update triggered | 20:10 |
| notmorgan | jeblair: i like graceful | 20:10 |
| anteaya | EmilienM: right, that is the part I can't see yet, I can't see that this patch adding the new job everywhere adds them to the experimental pipeline | 20:10 |
| pabelanger | jeblair: okay, let me wait until 328382 lands then | 20:10 |
| jeblair | pabelanger: then when it's idle, you can 'zuul-launcher stop' | 20:11 |
| pabelanger | that will pick up of FTPTask fix | 20:11 |
| anteaya | EmilienM: perhaps I am looking in the wrong place | 20:11 |
| clarkb | mgagne: you may not have the plugin installed | 20:11 |
| EmilienM | anteaya: jobs are already defined, in zuul/layout.yaml | 20:11 |
| jeblair | pabelanger: i'd like to see about fixing the env thing, then starting up zl01 with that, then gracefuling zl02 | 20:11 |
| EmilienM | anteaya: I don't add them they are here already!! | 20:11 |
| mtreinish | jeblair: https://review.openstack.org/328467 should fix it | 20:11 |
| anteaya | EmilienM: okay I'll stop now | 20:11 |
| EmilienM | thanks :) | 20:11 |
| pabelanger | jeblair: okay, I don't mind holding up. | 20:11 |
| anteaya | sorry we can't seem to communicate today | 20:11 |
| jeblair | pabelanger: cool. i'm going to go write that now | 20:12 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: WIP: Add web-based console log streaming https://review.openstack.org/320563 | 20:12 |
| anteaya | notmorgan: those are lovely colours | 20:12 |
| notmorgan | anteaya: :) you see why i map name+color then right? :) | 20:13 |
| EmilienM | anteaya: look https://review.openstack.org/#/c/328433/2/zuul/layout.yaml | 20:13 |
| EmilienM | anteaya: puppet-beaker-jobs-nv already exists | 20:13 |
| EmilienM | anteaya: before my patch | 20:13 |
| *** denisra has quit IRC | 20:13 | |
| EmilienM | anteaya: it contains experimental jobs | 20:13 |
| EmilienM | anteaya: we use them already in puppet-keystone. My patch adds the jobs for other modules too | 20:13 |
| anteaya | notmorgan: I do | 20:13 |
| EmilienM | anteaya: because it works on puppet-keystone | 20:13 |
| *** denisra has joined #openstack-infra | 20:14 | |
| EmilienM | anteaya: https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L700-L702 | 20:14 |
| *** gouthamr has quit IRC | 20:14 | |
| anteaya | http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n700 | 20:15 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/project-config: Make Zuul python 3 job voting https://review.openstack.org/328066 | 20:15 |
| anteaya | EmilienM: thanks that was the bit I was missing :) | 20:15 |
| EmilienM | anteaya: same link.. | 20:15 |
| EmilienM | anteaya: thanks for your review! | 20:16 |
| *** csomerville has quit IRC | 20:17 | |
| *** jamesmcarthur has joined #openstack-infra | 20:18 | |
| *** _ari_ has quit IRC | 20:18 | |
| *** vhosakot has quit IRC | 20:18 | |
| *** csomerville has joined #openstack-infra | 20:18 | |
| mgagne | clarkb: is the plugin required to control replication? I mean, replication is working atm, is this plugin only for additional admin tools? | 20:18 |
| mriedem | what is the point in the pypy jobs? | 20:19 |
| mriedem | because gate-python-novaclient-pypy never passes | 20:19 |
| anteaya | EmilienM: thank you for your patience | 20:19 |
| EmilienM | always :) | 20:19 |
| anteaya | EmilienM: :) | 20:20 |
| anteaya | mriedem: /me shrugs | 20:20 |
| anteaya | mriedem: would you like to remove it from novaclient? | 20:20 |
| mriedem | anteaya: yeah | 20:21 |
| mriedem | it's non-voting but it never passes | 20:21 |
| mriedem | so might as well not even run it | 20:21 |
| *** sree has quit IRC | 20:21 | |
| anteaya | mriedem: no argument here | 20:21 |
| mtreinish | mriedem: you don't want to ensure you can use novaclient in a pypy env? :p | 20:21 |
| anteaya | noone that I know of works on fixing pypy job failures | 20:22 |
| mriedem | mtreinish: i give 0 ___ | 20:22 |
| *** jamesmcarthur has quit IRC | 20:22 | |
| *** ilyashakhat has joined #openstack-infra | 20:23 | |
| * mordred agrees with mriedem | 20:23 | |
| clarkb | mgagne: yes it is required for all replication things | 20:23 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/system-config: define a release-job-failures mailing list https://review.openstack.org/328480 | 20:25 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Remove pypy-jobs from python-novaclient https://review.openstack.org/328481 | 20:26 |
| mriedem | anteaya: ^ | 20:26 |
| anteaya | mriedem: wonderful commit message | 20:26 |
| *** ilyashakhat has quit IRC | 20:26 | |
| mriedem | "nobody (except mtreinish) ... " | 20:27 |
| anteaya | ha ha ha | 20:27 |
| *** wajdi has joined #openstack-infra | 20:27 | |
| wajdi | good day all :) | 20:27 |
| anteaya | does he care or was he trolling | 20:27 |
| anteaya | I have my money on trolling myself | 20:27 |
| mriedem | the latter of course | 20:27 |
| mtreinish | mriedem: heh, I don't actually care | 20:27 |
| mordred | anteaya: I am trying to think of a time when mtreinish was not trolling | 20:28 |
| anteaya | ding ding ding | 20:28 |
| anteaya | mordred: ha ha ha | 20:28 |
| mgagne | clarkb: I think I'm missing some sort of permissions. plugin is installed. | 20:28 |
| anteaya | I miss jogo for that | 20:28 |
| anteaya | the two of them together would tag team | 20:28 |
| anteaya | hello wajdi | 20:28 |
| wajdi | hi anteaya :) | 20:28 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: add email notification of release job failures https://review.openstack.org/328486 | 20:29 |
| anteaya | did you need some help or were you just saying hello? | 20:29 |
| openstackgerrit | Merged openstack-infra/project-config: Skip func/unit tests only if Nova reno changes https://review.openstack.org/328380 | 20:29 |
| wajdi | I do need help, actually. | 20:29 |
| * anteaya is shocked | 20:29 | |
| wajdi | :) | 20:29 |
| *** dprince has quit IRC | 20:30 | |
| * mordred hands anteaya an extra peacock he found walking around | 20:30 | |
| *** gouthamr has joined #openstack-infra | 20:31 | |
| *** dprince has joined #openstack-infra | 20:31 | |
| EmilienM | anteaya: also https://review.openstack.org/#/c/328457/ if you can :) | 20:31 |
| openstackgerrit | Merged openstack-infra/project-config: puppet/beaker: add xenial-nv jobs to all modules https://review.openstack.org/328433 | 20:32 |
| anteaya | mordred: oh I like them, thank you | 20:32 |
| EmilienM | anteaya: I swear I'm done after :P | 20:32 |
| mgagne | ok, merging a change fixed the replication. thanks! | 20:32 |
| anteaya | mordred: India? Australia? | 20:32 |
| *** ilyashakhat has joined #openstack-infra | 20:32 | |
| mgagne | wajdi: ask away | 20:32 |
| anteaya | mordred: where are you with peacocks and without your house cat | 20:32 |
| anteaya | mgagne: yay | 20:32 |
| wajdi | I'm trying to determine how to properly get auto-merging jobs configured in Zuul. | 20:33 |
| openstackgerrit | Merged openstack-infra/project-config: Remove keystone-propose-config-updates https://review.openstack.org/315130 | 20:33 |
| wajdi | Trying to determine how in the check pipeline, to see if there is a way to allow an automatic workflow +1 based on specific criteria | 20:33 |
| wajdi | namely, jenkins user, for example | 20:34 |
| wajdi | all while making sure that any other user is held to the standard rules of not having any automatic workflow being triggered | 20:34 |
| anteaya | EmilienM: you are never done, it is one of the things I admire about you :) | 20:34 |
| EmilienM | ahah | 20:35 |
| anteaya | :) | 20:35 |
| EmilienM | I don't give up easily :P | 20:35 |
| anteaya | EmilienM: I admire that about you | 20:35 |
| anteaya | wajdi: well we don't do that ourselves | 20:35 |
| anteaya | we ensure that patches require a human reviewer to merge | 20:36 |
| anteaya | as auto merging patches could be used as a malicious vector | 20:36 |
| anteaya | the criteria for merging is set in gerrit and zuul reports back to gerri t based on test outcomes | 20:37 |
| anteaya | but zuul doesn't make any decisions about merging patches | 20:37 |
| *** ilyashakhat has quit IRC | 20:37 | |
| anteaya | it does make decisions about queuing patches for testing | 20:37 |
| *** eharney has quit IRC | 20:38 | |
| mgagne | anteaya: I work with wajdi. Our setup isn't public and submitted changes come from a trusted source. So we aren't worried about malicious intents. | 20:38 |
| anteaya | mgagne: that is fine | 20:38 |
| anteaya | but in terms of following a workflow we impliment we don't impliement that workflow | 20:38 |
| pabelanger | jeblair: zuul-launcher graceful blocks until zuul-launcher has stopped right? | 20:38 |
| openstackgerrit | Merged openstack-infra/zuul: Use pull mode for synchronize in _makeFTPTask https://review.openstack.org/328382 | 20:39 |
| mgagne | anteaya: so I guess this "feature" isn't really tested nor implemented since there is no use case for the openstack project | 20:39 |
| anteaya | mgagne: right | 20:39 |
| anteaya | I'm not saying it is not possible to do, it might very well be | 20:39 |
| *** tkelsey has joined #openstack-infra | 20:40 | |
| wajdi | which in turn, would give us the indication that if we wanted to implement this, we are better off configuring a separate pipeline with its own set of rules that runs independent | 20:40 |
| anteaya | but we haven't tried it ourselves nor are we likely to | 20:40 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-zuul: Add graceful-stop to zuul launcher init script https://review.openstack.org/328490 | 20:40 |
| anteaya | wajdi: I sure can't speak for infra about giving indications about what you should do | 20:40 |
| *** Goneri has quit IRC | 20:41 | |
| anteaya | but this isn't something we want to do ourselves, and at last summit we specifically discussed that we don't want to do this | 20:41 |
| pabelanger | jeblair: ^ I ask because I'm going to add an ansible-playbook we can use to graceful-stop and start zuul-launcher | 20:41 |
| anteaya | the malicious vector argument being compelling in the discussion | 20:41 |
| jeblair | pabelanger: it does not block | 20:43 |
| pabelanger | okay | 20:43 |
| wajdi | Thanks anteaya. I was more talking out loud on what approach I would probably take to figure this out. Thanks for the input. Appreciated. | 20:43 |
| pabelanger | Will add some ansible magic to check | 20:43 |
| anteaya | wajdi: ah okay, welcome | 20:44 |
| jeblair | pabelanger: or we could add a 'zuul-launcher wait' command | 20:44 |
| *** tkelsey has quit IRC | 20:44 | |
| jeblair | pabelanger: it could probably just try opening the socket every second until that no longer works | 20:44 |
| jeblair | pabelanger: the thing to know is that once you issue 'graceful' zuul-launcher itself will process no more commands from the socket | 20:44 |
| pabelanger | jeblair: okay, let me see how to add that | 20:45 |
| * jogo waves at anteaya | 20:47 | |
| anteaya | jogo: hello, I miss you! | 20:47 |
| openstackgerrit | Merged openstack-infra/project-config: Use custom bindep installer for puppet jobs https://review.openstack.org/328457 | 20:47 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-zuul: Add graceful-stop to zuul launcher init script https://review.openstack.org/328490 | 20:47 |
| anteaya | jogo: and I miss you and mtreinish together | 20:48 |
| notmorgan | OH look it's a jogo | 20:48 |
| jogo | :) | 20:48 |
| *** lezbar has joined #openstack-infra | 20:48 | |
| anteaya | jogo: I hope the pinny people are still treating you well | 20:48 |
| jogo | notmorgan: I am even upgrading my IRC ssh server soon (raspberry pi 3) | 20:48 |
| *** mriedem has quit IRC | 20:48 | |
| anteaya | fancy | 20:48 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add DB API method to get run wall times by run_metadata https://review.openstack.org/328478 | 20:48 |
| mtreinish | mordred: ^^^ do you know any magic sql tricks to make that a single query | 20:49 |
| anteaya | notmorgan: what colour is jogo? | 20:49 |
| jogo | anteaya: they are very fashionable her .. I don't fit in | 20:49 |
| anteaya | jogo: ha ha ha, come back here, we've kept you a spot | 20:49 |
| mtreinish | because as a loop in python it's just awful | 20:49 |
| *** ayoung has quit IRC | 20:49 | |
| *** mriedem has joined #openstack-infra | 20:50 | |
| anteaya | jogo: we were talking irc colours in notmorgan's weechat client earlier: http://i.imgur.com/p66eqCn.png | 20:50 |
| notmorgan | anteaya: jogo is purple | 20:51 |
| anteaya | notmorgan: oh nice | 20:51 |
| anteaya | notmorgan: a pretty purple? | 20:51 |
| *** dprince has quit IRC | 20:51 | |
| * anteaya needs another screenshot | 20:51 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: fix timeout var https://review.openstack.org/328492 | 20:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: get initial env from bash https://review.openstack.org/328493 | 20:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: be smarter about env quotes https://review.openstack.org/328494 | 20:51 |
| jogo | ohh purple | 20:52 |
| openstackgerrit | Merged openstack-infra/project-config: Remove pypy-jobs from python-novaclient https://review.openstack.org/328481 | 20:52 |
| jeblair | fungi, pabelanger, notmorgan: i took a look at pam_env... it also does not do full bash parsing. here's the relevant function: https://github.com/aosm/pam/blob/master/pam/modules/pam_env/pam_env.c#L203 | 20:52 |
| anteaya | jogo: you can't do better than that | 20:52 |
| *** lezbar has quit IRC | 20:52 | |
| notmorgan | jeblair: ahh. | 20:52 |
| jeblair | fungi, pabelanger, notmorgan: so i actually don't think that 'make our hacky parser a little better' is entirely the wrong approach | 20:52 |
| notmorgan | jeblair: fair enough | 20:52 |
| *** tonytan_brb has joined #openstack-infra | 20:53 | |
| jeblair | fungi, pabelanger, notmorgan: nonetheless, i did it both ways. so 328493 and 328494 are mutually exclusive options. | 20:53 |
| jeblair | you can take a look at both and see which you like better | 20:53 |
| jeblair | pabelanger: also, fixed an oops in the timeout var thing; see 328492 | 20:53 |
| *** tonytan4ever has quit IRC | 20:54 | |
| *** tonytan_brb is now known as tonytan4ever | 20:54 | |
| notmorgan | jeblair: so i functionally like 328493 more, but i think 328494 is just so much easier to look at. | 20:54 |
| notmorgan | jeblair: and ease of understanding does count for something. | 20:55 |
| jeblair | notmorgan: yeah, i was down with 493 right up until i wrote the ignore line. also, there's a little voice in my head saying "what if bash puts quotes around those?" :) | 20:55 |
| *** roxanaghe has joined #openstack-infra | 20:55 | |
| pabelanger | jeblair: Doh, ya. Didn't notice that. Thansk | 20:55 |
| jeblair | i mean, i've tested both, and they both work with our current expected input sets | 20:55 |
| jeblair | but you know, future and all | 20:56 |
| notmorgan | jeblair: i'm inclined to take 494 and revisit 493 with zuulv3 if 494 isn't sufficient | 20:56 |
| jeblair | notmorgan: it is part of the public record now, we can always dig it up again :) | 20:56 |
| notmorgan | jeblair: exactlu | 20:56 |
| jeblair | oh, i'll leave comments cross-referencing them | 20:56 |
| * notmorgan has a syntetic test failing the same way the gate is failing and is working to extract data from the stream objects when the test fails. | 20:57 | |
| jeblair | ooooh | 20:57 |
| notmorgan | the hard part is we patch the low level sockets for stream capture | 20:58 |
| EmilienM | mordred: 2:1 !!!!! yeah | 20:58 |
| notmorgan | jeblair: but i think i'm there just need to make sure my signal handler is called even if gentle=false | 20:59 |
| *** edmondsw has joined #openstack-infra | 21:00 | |
| pabelanger | jeblair: Agreed, I think 494 get us working. But 493 might be a revisit down the road | 21:00 |
| *** hongbin has quit IRC | 21:01 | |
| *** roxanaghe has quit IRC | 21:02 | |
| *** roxanaghe has joined #openstack-infra | 21:03 | |
| *** dimtruck is now known as zz_dimtruck | 21:03 | |
| *** mriedem has quit IRC | 21:05 | |
| *** sarob has joined #openstack-infra | 21:06 | |
| *** mriedem has joined #openstack-infra | 21:06 | |
| jeblair | okay, i'll push 494 through and abandon 493 for now | 21:09 |
| *** sarob has quit IRC | 21:10 | |
| *** mriedem has quit IRC | 21:10 | |
| pabelanger | I think 493 is a clever solutions however. | 21:11 |
| *** csomerville has quit IRC | 21:11 | |
| *** mriedem has joined #openstack-infra | 21:11 | |
| *** zz_dimtruck is now known as dimtruck | 21:11 | |
| *** aysyd has quit IRC | 21:12 | |
| *** csomerville has joined #openstack-infra | 21:13 | |
| *** NM has quit IRC | 21:15 | |
| openstackgerrit | Merged openstack-infra/zuul: Ansible launcher: fix timeout var https://review.openstack.org/328492 | 21:16 |
| krotscheck | pabelanger: It's been 5 hours, I'm going to try rechecking my xenial test build. | 21:17 |
| *** ldnunes has quit IRC | 21:18 | |
| pabelanger | krotscheck: sure, we are still uploading, to if you hit ovh-bhs1 or rax-iad you are using the old image | 21:18 |
| pabelanger | checking other providers to make sure they are good too | 21:18 |
| krotscheck | pabelanger: Alright, will recheck if that happens. | 21:18 |
| *** akshai has quit IRC | 21:18 | |
| *** dizquierdo has joined #openstack-infra | 21:19 | |
| jeblair | notmorgan: this may not be relevant now, but when you're out of the fog of dealing with reproducing that failure, you may be interested in setting ZUUL_TEST_ROOT to point to a location on a tmpfs. it will make tests run *way* faster and also not thrash your disk (or prematurely wear out your ssd) | 21:19 |
| jeblair | notmorgan: i set that in my .bashrc | 21:19 |
| *** rbrndt has quit IRC | 21:19 | |
| *** kzaitsev_mb has quit IRC | 21:19 | |
| pabelanger | krotscheck: Ya, need to re upload for rax-ord too | 21:19 |
| pabelanger | it failed | 21:19 |
| notmorgan | jeblair: I have a *hacky* as hell fix but i get log output | 21:20 |
| notmorgan | jeblair: about to propose it. | 21:20 |
| notmorgan | and i am working on a little more cleanup. but i have it. | 21:21 |
| notmorgan | jeblair: ^_^ | 21:21 |
| jeblair | yay! | 21:21 |
| notmorgan | oh yeah. this is a hacky thing, but supplying our own timeout fixture makes this possible. but ugh. | 21:22 |
| notmorgan | hm. now i just need to figure out how to call the underlying alarm handler directly | 21:23 |
| openstackgerrit | Merged openstack-infra/zuul: Ansible launcher: be smarter about env quotes https://review.openstack.org/328494 | 21:25 |
| *** dizquierdo has quit IRC | 21:25 | |
| *** roxanaghe has quit IRC | 21:25 | |
| *** javeriak has quit IRC | 21:25 | |
| *** matt-borland has quit IRC | 21:25 | |
| notmorgan | jeblair: oh interesting. | 21:28 |
| jeblair | pabelanger: in a few minutes, once ansible updates zuul, we can start the restart | 21:28 |
| notmorgan | jeblair: so, it looks like gentle is exiting things fine. we might be ok to use | 21:28 |
| *** _alastor_ has joined #openstack-infra | 21:28 | |
| jeblair | notmorgan: cool, maybe we cleaned up the edge cases there over time | 21:29 |
| notmorgan | jeblair: to be super-duper safe, i can just reset the handler and sigalarm(1) and still hard exit | 21:29 |
| _alastor_ | Hey folks, Is there anything else needed for a "Needs Workflow" review that has a +2 from two different cores for the project? | 21:29 |
| notmorgan | jeblair: which i'm inclined to do, so we'll stream the log out to stderr then if hard-bail. and i'll keep poking to make sure it's not an indefinite hang | 21:29 |
| notmorgan | s/if/do | 21:30 |
| *** ffrank has joined #openstack-infra | 21:30 | |
| *** darvon has quit IRC | 21:30 | |
| clarkb | _alastor_: a +1 workflow vote | 21:30 |
| jeblair | notmorgan: cool | 21:30 |
| *** pfallenop has quit IRC | 21:30 | |
| _alastor_ | clarkb: and only a core can give that? | 21:30 |
| *** sputnik13 has joined #openstack-infra | 21:31 | |
| *** inc0 has quit IRC | 21:31 | |
| mriedem | jeblair: fyi, i still can't rebase that nova series w/o everything past the first 2 patches failing with a merge conflict | 21:31 |
| mriedem | https://review.openstack.org/#/c/297906/ | 21:31 |
| clarkb | _alastor_: correct | 21:32 |
| _alastor_ | clarkb: ok, thanks. I'll go one of them about it :) | 21:32 |
| *** pfallenop has joined #openstack-infra | 21:33 | |
| *** darvon has joined #openstack-infra | 21:33 | |
| *** hongbin has joined #openstack-infra | 21:33 | |
| fungi | that's some mighty scrollback | 21:34 |
| anteaya | fungi: we worked hard | 21:34 |
| anteaya | fungi: we didn't want you to think we were slacking | 21:34 |
| *** ddieterly is now known as ddieterly[away] | 21:35 | |
| *** sdake has quit IRC | 21:36 | |
| *** amitgandhinz has quit IRC | 21:37 | |
| jeblair | mriedem: maybe we ought to bump up the delay then... :| | 21:37 |
| jeblair | and maybe we can revisit it once nova is gc'd | 21:38 |
| jeblair | (i mean that literally with respect to the git repo, not, like metaphorically garbage collecting the project or anything) | 21:38 |
| *** esberglu has quit IRC | 21:39 | |
| *** pradk has quit IRC | 21:39 | |
| *** gouthamr has quit IRC | 21:39 | |
| *** rbrndt has joined #openstack-infra | 21:40 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Increase the Gerrit trailing delay https://review.openstack.org/328504 | 21:40 |
| jeblair | mriedem: ^ | 21:40 |
| pabelanger | jeblair: great | 21:41 |
| *** sdake has joined #openstack-infra | 21:41 | |
| fungi | jeblair: notmorgan: pabelanger: all arguments considered, 328494 is my pick for now as well | 21:41 |
| jeblair | pabelanger: ansible is still running on jenkins02; probably a slow jjb update, so zuul hasn't updated yet | 21:42 |
| mriedem | jeblair: thanks a ton | 21:42 |
| jeblair | mriedem: thanks for the report :) | 21:42 |
| mriedem | jeblair: that big series is extra sucky because it's always requiring a rebase | 21:42 |
| mriedem | the other nova cores need to just approve it all asap and not worry about it's correctness | 21:43 |
| pabelanger | jeblair: Ya, will be happy once jenkins02 goes into shutdown mode, our slowest master | 21:43 |
| jeblair | mriedem: that'll need a zuul restart to take effect, btw, so won't be instantaneous. but i have other issues pressing for a restart too, so hopefully can go in together | 21:43 |
| jeblair | pabelanger: yep | 21:43 |
| *** sdake_ has joined #openstack-infra | 21:43 | |
| *** Sukhdev has joined #openstack-infra | 21:43 | |
| fungi | indeed, especially glad to see jenkins01 and 02 out of the picture. they were... "venerable" | 21:44 |
| jeblair | mriedem: there's also the "can't be worse than what we got now, right?" approach. :) | 21:44 |
| jeblair | i use that one a lot | 21:44 |
| jeblair | pabelanger: i'll manually install updated zuul on those hosts | 21:45 |
| mriedem | jeblair: well, users wanting to create servers on the latest microversion might care | 21:45 |
| mriedem | but meh | 21:45 |
| pabelanger | jeblair: ack | 21:45 |
| mriedem | users | 21:45 |
| mriedem | psh | 21:45 |
| *** burgerk has quit IRC | 21:45 | |
| *** sdake has quit IRC | 21:45 | |
| pabelanger | jeblair: wheel moving again | 21:45 |
| zaro | fungi: i just noticed we have a duplicate project we want to remove? | 21:46 |
| jeblair | pabelanger: good, though i think we'd still be waiting for 2200 | 21:46 |
| zaro | fungi: i don’t think you need to reindex for that. | 21:46 |
| jeblair | pabelanger: okay, latest zuul is installed on all launchers | 21:47 |
| fungi | zaro: yes, i screwed up and reenabled puppet without giving the rename patches plenty of extra time to merge | 21:47 |
| jeblair | pabelanger: do you want to start zl01 and graceful zl02? | 21:47 |
| jeblair | pabelanger: (i'll take zlstatic01) | 21:47 |
| fungi | zaro: makes sense, i suspected it wasn't necessary since there are no associated changes | 21:47 |
| pabelanger | jeblair: sure | 21:47 |
| pabelanger | zl01 starting | 21:48 |
| zaro | fungi: you can delete with delete-project plugin without having to run reindex so you can do anytime | 21:48 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 21:49 | |
| fungi | zaro: well, there also shouldn't be any database references to it, so the only actual persistent data for it is on disk | 21:49 |
| pabelanger | zuul-launcher graceful on zl02 | 21:49 |
| zaro | fungi: does it appear on the gerrit UI projects list? | 21:50 |
| jeblair | wow, the wheel mirrors are running *legitimately* very long jobs :| | 21:50 |
| *** Hunner has joined #openstack-infra | 21:51 | |
| *** Hunner has quit IRC | 21:51 | |
| *** Hunner has joined #openstack-infra | 21:51 | |
| pabelanger | still waiting for zl01.o.o to get a ready node | 21:51 |
| fungi | zaro: yes, because there is a git repo for it in ~gerrit2/review_site/git/openstack-infra/ansible-puppet.git | 21:51 |
| fungi | zaro: but no references in the database | 21:51 |
| pabelanger | jeblair: I think the python3 wheels are the issue. I have a patch up to disable them for now. fungi pointed it out last week | 21:51 |
| jeblair | oh | 21:52 |
| *** lascii is now known as alaski | 21:52 | |
| fungi | yeah, when the ubuntu wheel build job expanded to also do a pass building python 3.4 wheels, that at least doubled its runtime | 21:52 |
| pabelanger | 325955 is the review to disable them | 21:52 |
| jeblair | i think one of them will time out right around 2200 | 21:52 |
| *** mriedem has quit IRC | 21:53 | |
| jeblair | the other one maybe 40 minutes later | 21:53 |
| *** edmondsw has quit IRC | 21:53 | |
| pabelanger | there we go, zl01 picked up its first node | 21:54 |
| jeblair | w00t | 21:54 |
| *** kzaitsev_mb has joined #openstack-infra | 21:54 | |
| anteaya | zaro: when I try to start gerrit it tells me it doesn't like line 22 in my gerrit.config: http://paste.openstack.org/show/510677/ | 21:55 |
| pabelanger | totally except rolling restart to be way easier for zuul-launcher and ansible | 21:55 |
| zaro | fungi: ohh, i thought an entry in the db was what makes it appear in the list. but if not then i guess you may not even need the delete-project plugin. | 21:55 |
| fungi | zaro: no, as far as i can tell the projects list is determined entirely by looking in review_site/git | 21:56 |
| pabelanger | jeblair: zl02 is still accepting jobs. I wouldn't have expected that | 21:57 |
| jeblair | hrm | 21:57 |
| jeblair | pabelanger: oh, i wonder if zl02 doesn't have the graceful command | 21:58 |
| jeblair | i mean the running server | 21:58 |
| pabelanger | 2016-06-10 21:49:24,226 DEBUG zuul.CommandSocket: Received graceful from socket | 21:58 |
| pabelanger | from the logs | 21:58 |
| jeblair | yeah, it received it, but then didn't do anything | 21:58 |
| pabelanger | right | 21:58 |
| pabelanger | okay, so let me do pause / stop? | 21:58 |
| *** roxanaghe has joined #openstack-infra | 21:58 | |
| jeblair | pabelanger: pause, then release | 21:58 |
| *** EricGonc_ has joined #openstack-infra | 21:58 | |
| pabelanger | jeblair: ack | 21:59 |
| *** flepied1 has joined #openstack-infra | 21:59 | |
| jeblair | pabelanger: don't stop until it's finished | 21:59 |
| *** sarob has joined #openstack-infra | 21:59 | |
| *** flepied has quit IRC | 21:59 | |
| jeblair | pabelanger: (i just use ps to look for ansible processes running to know) | 21:59 |
| *** slaweq_ has joined #openstack-infra | 21:59 | |
| *** EricGonczer_ has quit IRC | 22:00 | |
| pabelanger | jeblair: okay, when all the ansible-playbook process are gone, it is safe to release | 22:00 |
| jeblair | pabelanger: it's safe to release after pausing | 22:00 |
| pabelanger | Oh | 22:00 |
| pabelanger | okay, right after each other | 22:00 |
| jeblair | pabelanger: release == send idle nodes back to nodepool | 22:00 |
| jeblair | yep | 22:00 |
| *** ddieterly[away] is now known as ddieterly | 22:00 | |
| jeblair | (this is a thing we did not have with jenkins :) | 22:00 |
| pabelanger | gotcha | 22:00 |
| pabelanger | okay, both commands run | 22:01 |
| jeblair | oh, it does look like zl02 is running the broken release code too, so we probably didn't actually do that this time. that's fixed in tree though. | 22:01 |
| *** csomerville has quit IRC | 22:02 | |
| jeblair | oy the wheel logs are huge | 22:02 |
| *** sarob has quit IRC | 22:03 | |
| jeblair | okay, the wheel jobs are done | 22:04 |
| zaro | anteaya: are you able to retrieve a story from SB from your gerrit server using the SB rest api? | 22:04 |
| jeblair | one was killed, the other finished | 22:04 |
| zaro | anteaya: doing something like this? curl -v https://storyboard-dev.o.o/api/v1/stories/1 | 22:05 |
| jeblair | restarting zlstatic01 | 22:05 |
| anteaya | zaro: well I just started the vm now and gerrit was stopped, so I tried to start gerrit and it said it didn't like my gerrit.config | 22:05 |
| anteaya | zaro: which is here http://paste.openstack.org/show/510677/ | 22:05 |
| anteaya | zaro: so I'm guessing in the documentation you don't actually want me to use [@PLUGIN@] | 22:06 |
| zaro | the storyboard plugin uses the SB rest api to retrieve story/task info so that needs to work at least | 22:06 |
| pabelanger | jeblair: when we do the release, it doesn't appear ready nodes in zuul-launcher are affected. They still appear assigned to zl02 | 22:06 |
| anteaya | zaro: yeah I got that, I'm trying to start gerrit, and it doesn't like my current config: http://paste.openstack.org/show/510677/ | 22:06 |
| zaro | correct, @PLUGIN@ is a variable for the plugin name i.e. "it-storyboard" | 22:06 |
| anteaya | okay thank you | 22:07 |
| anteaya | I'll make that change | 22:07 |
| openstackgerrit | Merged openstack-infra/project-config: Stop building python3 wheels https://review.openstack.org/325955 | 22:07 |
| pabelanger | jeblair: or is that misreporting my statsd | 22:07 |
| zaro | it should be something like this.. http://paste.openstack.org/show/510678/ | 22:07 |
| pabelanger | I'm looking here: http://grafana.openstack.org/dashboard/db/nodepool | 22:07 |
| pabelanger | confirmed, nodepool still lists them as ready | 22:08 |
| jeblair | pabelanger: that's the bug i meant | 22:08 |
| pabelanger | Ah | 22:08 |
| pabelanger | cool | 22:09 |
| jeblair | pabelanger: you'll see errors in the log about ansible_proc not being defined or something. that broke the release process and they're still idle | 22:09 |
| anteaya | zaro: thank you for the paste, that helps | 22:09 |
| *** slaweq_ has quit IRC | 22:09 | |
| jeblair | pabelanger: shouldn't be a problem next time | 22:09 |
| pabelanger | jeblair: Ah, okay. I remember | 22:09 |
| pabelanger | I'll delete them on nodepool.o.o | 22:09 |
| jeblair | should be fine. zuul-launcher won't know they've been deleted, but we're about to shut it down, so. | 22:10 |
| anteaya | zaro: gerrit is started, last night I got as far as: https://gerrit.googlesource.com/plugins/its-storyboard/+/master/src/main/resources/Documentation/quick-install-guide.md#its_enable_its_enableEnable-the-Plugin | 22:10 |
| zaro | fungi: anyways, i’ve puppeted the delete-project plugin for installation in case you want to actually use it. https://review.openstack.org/#/c/319400/ | 22:10 |
| anteaya | zaro: and I couldn't clone the All-Projects repo | 22:10 |
| *** Sukhdev has quit IRC | 22:11 | |
| *** sarob has joined #openstack-infra | 22:12 | |
| anteaya | I tried this on my terminal, not my vm, fungi how can we get curl to get past the self signed cert and get to the storyboard-dev api? http://paste.openstack.org/show/510679/ | 22:12 |
| zaro | anteaya: command is: git clone ssh://user@localhost29418/All-Projects && cd All-Projects && git fetch origin refs/meta/config && git checkout FETCH_HEAD | 22:12 |
| anteaya | zaro: thank you I will try that | 22:12 |
| fungi | anteaya: curl --insecure (or -k) should take care of it | 22:14 |
| *** csomerville has joined #openstack-infra | 22:14 | |
| anteaya | fungi: thank you | 22:14 |
| *** vhosakot has joined #openstack-infra | 22:15 | |
| *** sarob has quit IRC | 22:15 | |
| *** kzaitsev_mb has quit IRC | 22:15 | |
| *** sarob has joined #openstack-infra | 22:15 | |
| pabelanger | jeblair: okay, looks like we have a few devstack jobs on zl02. Going to step away for a bit, I'll check back in a bit. I assume we'll be bringing zl02 back online once everything has finished? | 22:15 |
| anteaya | zaro: I'm obviously not the right user or something, do I have to set myself up as a user? http://paste.openstack.org/show/510680/ | 22:16 |
| jeblair | pabelanger: we could probably do that at this point -- i think we've covered all the known bugs so far and could increase our zuul/jenkins ratio | 22:17 |
| *** kzaitsev_mb has joined #openstack-infra | 22:17 | |
| jeblair | pabelanger: i will go ahead and put jenkins02 in prepare for shutdown mode | 22:17 |
| zaro | anteaya: yes, replace ‘user’ with your username or whatever account you have setup in gerrit | 22:17 |
| jeblair | traffic is slow enough at this point we don't need all of the jenkinseses | 22:17 |
| pabelanger | jeblair: Okay, agreed. I think we're in good shape for 2 zuul-launchers | 22:17 |
| pabelanger | feel free to start zl02 if I'm not back in time. Otherwise, I can do it when back | 22:18 |
| jeblair | pabelanger: ack | 22:18 |
| pabelanger | jeblair: thanks | 22:18 |
| *** sarob has quit IRC | 22:18 | |
| *** PollyZ has quit IRC | 22:19 | |
| *** sarob has joined #openstack-infra | 22:19 | |
| anteaya | zaro: I had not set up any user in gerrit, I guess that is the initial login step in the instructions | 22:19 |
| jeblair | #status log jenkins02 is in prepare for shutdown in preparation for decomissioning | 22:19 |
| openstackstatus | jeblair: finished logging | 22:19 |
| anteaya | zaro: and yes I can see the storyboard-dev api from my gerrit server: http://paste.openstack.org/show/510681/ | 22:19 |
| jeblair | huh, weird that jenkins doesn't have a 'prepare for decomissioning' button | 22:19 |
| anteaya | is there a jenkins dry-dock or something | 22:20 |
| anteaya | for decommissioned jenkinsii? | 22:20 |
| *** sarob has quit IRC | 22:20 | |
| *** sarob has joined #openstack-infra | 22:20 | |
| jeblair | oh, the epa made us shut it down, so we just send them offshore | 22:21 |
| jeblair | speaking of which ... i have noticed the ghost fleet getting much smaller | 22:21 |
| jeblair | https://en.wikipedia.org/wiki/Suisun_Bay_Reserve_Fleet | 22:22 |
| notmorgan | jeblair: so interesting side effect of those SIGALRM failures, we're splatting everything in the test runner. | 22:23 |
| *** mdrabe has quit IRC | 22:23 | |
| notmorgan | jeblair: we really do need to use gentle unless we use a subinterpreter (for example) per test - i think i see how i'd like to try and fix this in testr long term | 22:24 |
| jeblair | notmorgan: yeah, i like gentle, we just weren't able to use it in the past | 22:25 |
| notmorgan | jeblair: the simplest thing is I can get the output to the console for a failed test right now - but it wont be in the .testrepository files | 22:25 |
| jeblair | anything will do :) | 22:25 |
| notmorgan | ok i'll just stream it directly to stderr which is console. | 22:25 |
| notmorgan | and hard fail with a subsequent alarm, then we cna evaluate gentle | 22:25 |
| notmorgan | and i have ideas on what we should be doing in testrepository to help with this, but that is going to be a big discussion and likely revolve around me chatting a bunch with lifeless to make sure I have my head wrapped around it. | 22:26 |
| anteaya | zaro: what do I put in the open id sign in to create an account for myself on my gerrit? | 22:26 |
| *** yamahata has quit IRC | 22:27 | |
| anteaya | zaro: when I put in my email for infra's gerrit I get: Provider is not supported, or was incorrectly entered. | 22:28 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: Capture Logs when Timeout is hit https://review.openstack.org/328516 | 22:28 |
| zaro | ohh i remembered that since gerrit runs with java jvm you may need to setup the certs for the JVM so that it can talk to SB. | 22:29 |
| *** yamada-h has joined #openstack-infra | 22:29 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: DNM: Syntetic Test to show TimeoutFixture failure https://review.openstack.org/328517 | 22:29 |
| zaro | anteaya: setup is something like this.. http://paste.openstack.org/show/510690/ | 22:29 |
| anteaya | zaro: okay thanks, right now I am stuck on creating a user for myself in my gerrit | 22:29 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: Capture Logs when Timeout is hit https://review.openstack.org/328516 | 22:30 |
| anteaya | zaro: whether I register or sign in I can't seem to enter what the open id page is looking for | 22:30 |
| zaro | ohh don’t use open id, replace it with ‘type=OPENID’ with ‘type = DEVELOPMENT_BECOME_ANY_ACCOUNT’ in gerrit.config and restart | 22:30 |
| anteaya | zaro: thanks | 22:31 |
| *** rlandy has quit IRC | 22:31 | |
| *** csomerville has quit IRC | 22:31 | |
| *** csomerville has joined #openstack-infra | 22:32 | |
| *** yamada-h has quit IRC | 22:33 | |
| notmorgan | jeblair: http://logs.openstack.org/17/328517/1/check/gate-zuul-python27-db/b782d91/console.html#_2016-06-10_22_32_38_066 example with the synthetic failure | 22:34 |
| notmorgan | jeblair: (that was a time.sleep() to force the timeout) | 22:34 |
| jeblair | notmorgan: cool -- though one thing i just thought of -- if we have more than one test hit that, we could end up with confusing output | 22:35 |
| anteaya | zaro: Invalid value: auth.type='DEVELOPMENT_BECOME_ANY_ACCOUNT' | 22:35 |
| jeblair | notmorgan: but hopefully, if it's a fallback to a fallback, maybe it's good enough :) | 22:35 |
| notmorgan | jeblair: yeah that is a bit confusing, let me add a start entry point to console and a prefix so we can grep per-test | 22:36 |
| *** johnny_ has quit IRC | 22:36 | |
| notmorgan | hhmm or at least a start w/ a test name. | 22:36 |
| *** vhosakot has quit IRC | 22:36 | |
| notmorgan | but hey *some* logs > no logs :) | 22:36 |
| jeblair | yep | 22:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Remove jenkins02 from nodepool https://review.openstack.org/328521 | 22:37 |
| *** sputnik13 has quit IRC | 22:37 | |
| anteaya | zaro: it didn't like the quotes, not quotes seems to be okay | 22:37 |
| *** tpsilva has quit IRC | 22:37 | |
| *** amitgandhinz has joined #openstack-infra | 22:38 | |
| notmorgan | jeblair: fwiw, i already have more info in what is causing the spurrious timeouts ;) | 22:38 |
| notmorgan | jeblair: cause it failed... with one of them on check! | 22:38 |
| jeblair | notmorgan: rock on! let me know if you want me to pore over the logs | 22:38 |
| notmorgan | jeblair: let me repropose and extract the actual failing test.. then we can play games of "what causes this" | 22:38 |
| notmorgan | cause it's one of 4 tests.. | 22:39 |
| notmorgan | :P | 22:39 |
| *** julim has quit IRC | 22:39 | |
| notmorgan | though i *think* i know which one it is. | 22:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Decomission jenkins02 https://review.openstack.org/328523 | 22:39 |
| *** yamahata has joined #openstack-infra | 22:41 | |
| *** vgridnev_ has joined #openstack-infra | 22:41 | |
| *** tkelsey has joined #openstack-infra | 22:42 | |
| *** amitgandhinz has quit IRC | 22:42 | |
| *** mtanino__ has joined #openstack-infra | 22:43 | |
| *** mtanino has quit IRC | 22:45 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: Capture Logs when Timeout is hit https://review.openstack.org/328516 | 22:46 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: DNM: Syntetic Test to show TimeoutFixture failure https://review.openstack.org/328517 | 22:46 |
| *** tkelsey has quit IRC | 22:46 | |
| notmorgan | jeblair: lets see if we timeout now (again), hopefully i'll have a real example of this failure in a few rechecks at most | 22:47 |
| *** wajdi has quit IRC | 22:47 | |
| anteaya | fungi: zaro so I was able to git clone from All Projects on my local gerrit however it ended with status fatal saying it could not remote ref refs/meta/config: http://paste.openstack.org/show/510691/ | 22:48 |
| anteaya | clearly I have not done something it wanted me to do | 22:48 |
| anteaya | it could not find remote ref refs/meta/config | 22:49 |
| *** kzaitsev_mb has quit IRC | 22:49 | |
| *** sarob has quit IRC | 22:50 | |
| *** sarob has joined #openstack-infra | 22:51 | |
| *** kzaitsev_mb has joined #openstack-infra | 22:51 | |
| *** hongbin has quit IRC | 22:52 | |
| anteaya | zaro: fungi here is the access I currently have on All Projects: http://imgur.com/MM7c8HY | 22:52 |
| *** rbrndt has quit IRC | 22:53 | |
| *** Deathfean has joined #openstack-infra | 22:53 | |
| *** yamahata has quit IRC | 22:53 | |
| *** yamahata has joined #openstack-infra | 22:54 | |
| *** mtanino has joined #openstack-infra | 22:54 | |
| *** xyang1 has quit IRC | 22:55 | |
| *** mtanino__ has quit IRC | 22:56 | |
| anteaya | zaro: as the first signed in user I'm the admin user, right? so setting access in the gui should be accessible to me and not greyed out, correct? | 22:59 |
| *** sarob has quit IRC | 22:59 | |
| zaro | yes, you can do that | 22:59 |
| *** sarob has joined #openstack-infra | 23:00 | |
| anteaya | zaro: right but I am saying my gui is greyed out | 23:00 |
| anteaya | I'm the first signed in user and the buttons are greyed out | 23:00 |
| *** ddieterly has quit IRC | 23:00 | |
| zaro | what version did you install? | 23:00 |
| anteaya | the version I believed we have on -dev: tarballs.openstack.org/ci/test/gerrit-v2.11.4.13.cb9800e.war | 23:01 |
| zaro | ok. then i’m guessing your not the first user. | 23:01 |
| *** tosky has quit IRC | 23:01 | |
| anteaya | how can I be not the first user | 23:01 |
| *** Jeffrey4l_ has joined #openstack-infra | 23:01 | |
| zaro | i think only way is to look at the db | 23:02 |
| anteaya | sigh | 23:02 |
| zaro | but i would just try to create a new site with same gerrit.config settings and see if you can edit that one | 23:02 |
| zaro | i would create new site (do not start it), copy gerrit.config from existing site to new site, then start it. then create the 1st user and add your ssh key | 23:04 |
| mgagne | can't you see if you are in Admin groups in Settings -> Groups ? | 23:04 |
| clarkb | ni | 23:04 |
| jeblair | clarkb: augh! | 23:04 |
| clarkb | Norah says hello | 23:04 |
| zaro | anteaya: mgagne is a smart guy | 23:04 |
| anteaya | zaro: I'm glad for that | 23:05 |
| * jeblair gives Norah a shrubbery | 23:05 | |
| clarkb | haha | 23:05 |
| anteaya | I don't know why but somehow I got myself as the second user | 23:05 |
| anteaya | so I am created a new user and trying to get in via the new user | 23:06 |
| mgagne | I'm not sure if admin has the right SSH key for that but you can add yourself from CLI (on Gerrit server itself): sudo -u gerrit2 ssh -p 29418 admin@localhost gerrit set-members --add username@example.org "'Administrators'" | 23:06 |
| *** jerryz has joined #openstack-infra | 23:06 | |
| sc68cal | njohnston: anteaya - we tried a year ago to get everyone to leave. Of course there was one person who didn't leave - logan- so we weren't able to reclaim it and register it | 23:06 |
| fungi | anteaya: mgagne: you can also make most of those api calls as the "magic" Gerrit Code Review user (using the ssh api host key as a user authentication key) | 23:07 |
| mgagne | this assumes the first user is "admin" and has the public key found for in gerrit2 home directory authorized for said user | 23:07 |
| mgagne | fungi: very good to know, didn't know about that | 23:07 |
| mgagne | so you can "sudo" as anybody? | 23:08 |
| mgagne | what would be the username? admin? | 23:08 |
| anteaya | zaro: mgagne now the new user is the admin | 23:08 |
| fungi | ssh -i review_site/etc/ssh_host_rsa_key -p 29418 'Gerrit Code Review'@127.0.0.1 gerrit set-members --add username@example.org "'Administrators'" | 23:08 |
| anteaya | sc68cal: thank you, at least you tried | 23:08 |
| mgagne | awesome, adding to our doc then | 23:09 |
| fungi | mgagne: the username is actually "Gerrit Code Review" (as in my example above) | 23:09 |
| anteaya | mgagne: thank you | 23:09 |
| fungi | that's one of those backdoors that's been in gerrit for as long as i've been administering gerrit | 23:09 |
| fungi | and presumably much longer | 23:09 |
| anteaya | mgagne: right that assumption about the first user being admin, that was the part I didn't get right | 23:09 |
| mgagne | mine is protected with a passphrase :O | 23:10 |
| mgagne | anteaya: me neither :-/ | 23:10 |
| fungi | mgagne: that's weird. if your review_site/etc/ssh_host_rsa_key file is passphrase-encrypted then i wonder how gerrit is accepting ssh connections | 23:11 |
| mgagne | I'm using an SQL script to create the admin user anyway =) | 23:11 |
| *** Deathfean has quit IRC | 23:11 | |
| *** hashar has quit IRC | 23:11 | |
| mgagne | ¯\_(ツ)_/¯ | 23:12 |
| *** dimtruck is now known as zz_dimtruck | 23:13 | |
| anteaya | I've allowed read and push on refs/meta/config for Registered Users | 23:13 |
| anteaya | so I'll try to clone again | 23:13 |
| mgagne | found the script here: https://gist.github.com/mindjiver/81d299b974c01aebc872 | 23:13 |
| mgagne | as Jinja2 template for Ansible: https://gist.github.com/mgagne/0df3bb707176f2401fdd0b99915c5fdd | 23:14 |
| *** sarob has quit IRC | 23:15 | |
| fungi | here's an example of calling an admin-only api method as the Gerrit Code Review user on our 2.11 deployment http://paste.openstack.org/show/510692 | 23:15 |
| *** gouthamr has joined #openstack-infra | 23:16 | |
| notmorgan | jeblair: http://logs.openstack.org/16/328516/3/check/dev-zuul-coverage/4796cb4/console.html#_2016-06-10_23_12_50_922 looks like it is in-fact TestScheduler.test_client_enqueue_negative | 23:16 |
| anteaya | apparently git checkout refs/meta/config is incorrect: http://paste.openstack.org/show/510694/ | 23:17 |
| jeblair | notmorgan: i like the ascii art | 23:17 |
| notmorgan | jeblair: hehe nmakes it easy to see start/stop :) since well yeah | 23:17 |
| notmorgan | also http://logs.openstack.org/16/328516/3/check/gate-zuul-python27-db/298e3fd/console.html#_2016-06-10_23_14_23_651740 same recheck but the actual py27 run | 23:17 |
| notmorgan | jeblair: ooh well that is harder to read =/ silly threads. | 23:18 |
| zaro | anteaya: try this way http://blog.bruin.sg/2013/04/how-to-edit-the-project-config-for-all-projects-in-gerrit/ | 23:19 |
| anteaya | zaro: thank you | 23:19 |
| notmorgan | oh interesting, looks like it trapped the same log data (possibly) for all the failures.. weird. | 23:19 |
| *** sdague has quit IRC | 23:19 | |
| anteaya | I've worked so hard to get to get access to project.config I can't remember what I wanted to do once I got here | 23:21 |
| anteaya | I think I will get food and then come back | 23:21 |
| anteaya | thanks all for the help | 23:21 |
| notmorgan | jeblair: it may be missing some relevant data from the stderr capture that didn't hit the log | 23:21 |
| jeblair | notmorgan: ok | 23:22 |
| *** kzaitsev_mb has quit IRC | 23:22 | |
| *** gordc has quit IRC | 23:22 | |
| notmorgan | jeblair: oh look this might be the issue: http://logs.openstack.org/16/328516/3/check/dev-zuul-coverage/4796cb4/console.html#_2016-06-10_23_12_50_975 | 23:23 |
| notmorgan | jeblair: at least one of them, just spinning on connect/error connectind/disconnect | 23:23 |
| jeblair | notmorgan: agreed... and that seems to be the wrong port number | 23:23 |
| *** mixos has joined #openstack-infra | 23:24 | |
| jeblair | notmorgan: it looks like the gear server is actually at 43535? | 23:24 |
| notmorgan | jeblair: oh wow. yeah | 23:24 |
| notmorgan | and that failed after the invalid project check. | 23:25 |
| notmorgan | are we clobbering the fixture somehow? | 23:25 |
| jeblair | notmorgan: the actual hang is that the test is sitting waiting for the invalid job gearman function to return | 23:25 |
| notmorgan | ah | 23:25 |
| notmorgan | yeah | 23:25 |
| jeblair | notmorgan: presumably it's the zuul rpclistener that is in the reconnect loop, since that's what would be running the job | 23:26 |
| notmorgan | right. that makes sense | 23:26 |
| *** sshnaidm has quit IRC | 23:26 | |
| jeblair | notmorgan: so yeah, fixture clobbering seems likely.... maybe we couldn't repro this because you need lots of simultaneous test runners or something | 23:26 |
| jeblair | notmorgan: maybe also related to funny business going on with sockets... | 23:26 |
| notmorgan | yeah i can't really repro more than ~4 runners without major pain | 23:27 |
| notmorgan | it might need more like 8 | 23:27 |
| *** tonytan4ever has quit IRC | 23:27 | |
| *** salv-orl_ has joined #openstack-infra | 23:28 | |
| *** mixos has quit IRC | 23:28 | |
| *** mriedem has joined #openstack-infra | 23:29 | |
| *** weshay has quit IRC | 23:29 | |
| *** sputnik13 has joined #openstack-infra | 23:29 | |
| notmorgan | jeblair: i think we have 2 issues - some fixture clobbering, and the timeout. | 23:30 |
| *** elo has quit IRC | 23:31 | |
| jeblair | 2016-06-10 23:12:50.972 | 2016-06-10 23:12:22,428 zuul.RPCClient DEBUG Connecting to gearman at 127.0.0.1:43535 | 23:31 |
| jeblair | that looks right though | 23:31 |
| notmorgan | yeah | 23:31 |
| *** salv-orlando has quit IRC | 23:31 | |
| notmorgan | looking at the py27 one | 23:31 |
| notmorgan | and it looks like it's just hanging waiting for a complete. | 23:31 |
| notmorgan | that isn't happening | 23:31 |
| jeblair | notmorgan: link? | 23:32 |
| notmorgan | http://logs.openstack.org/16/328516/3/check/gate-zuul-python27-db/298e3fd/console.html#_2016-06-10_23_14_31_002818 | 23:33 |
| *** elo has joined #openstack-infra | 23:33 | |
| jeblair | i'm so glad that's orange | 23:33 |
| notmorgan | i know, makes it *SO* easy to read | 23:33 |
| jeblair | okay, so that's the same hangup in the test, but without the weird gear reconnection stuff | 23:34 |
| notmorgan | yeah | 23:34 |
| notmorgan | so i'm going to go on a limb and say *some* fixture clobbering, but that isn't the core issue | 23:34 |
| notmorgan | and likely is incidental | 23:34 |
| jeblair | notmorgan: ah, and that has 2 failures | 23:35 |
| jeblair | in that run | 23:35 |
| notmorgan | that is odd too | 23:35 |
| notmorgan | but its 2 failures for the same test... which... wut? | 23:35 |
| jeblair | notmorgan: different tests | 23:36 |
| jeblair | 2016-06-10 23:15:13.871034 tests.test_scheduler.TestScheduler.test_client_enqueue_ref | 23:36 |
| jeblair | 2016-06-10 23:15:13.871063 tests.test_scheduler.TestScheduler.test_client_enqueue_negative | 23:36 |
| notmorgan | ooooh | 23:36 |
| notmorgan | yeah hey | 23:36 |
| jeblair | very similar test construct though | 23:36 |
| notmorgan | woot | 23:36 |
| jeblair | and i seem to be able to distinguish the output from the 2, so that's nice | 23:36 |
| notmorgan | yeah i id the sys.stderr.write() as a single tx | 23:37 |
| notmorgan | so it should be atomic | 23:37 |
| notmorgan | not a per-line write | 23:37 |
| jeblair | cool | 23:37 |
| notmorgan | the ascii art was extra lines but meh | 23:37 |
| anteaya | zaro: in this paste, what is the meaning behind line 39? http://paste.openstack.org/show/510678/ | 23:37 |
| anteaya | zaro: should I omit line 39 in my project.config? | 23:38 |
| zaro | yes omit | 23:38 |
| *** amitgandhinz has joined #openstack-infra | 23:38 | |
| pabelanger | jeblair: looks like zl02.o.o is done now. Ready for it to come back online? | 23:40 |
| jeblair | pabelanger: yep | 23:40 |
| pabelanger | okay, stoppped and started | 23:41 |
| *** pfallenop has quit IRC | 23:41 | |
| jeblair | notmorgan: i think this is the problem: http://paste.openstack.org/show/510695/ | 23:41 |
| *** pfallenop has joined #openstack-infra | 23:42 | |
| notmorgan | jeblair: that makes sense. | 23:42 |
| jeblair | notmorgan: it looks like we are trying to grab a job and go into pre_sleep *before* we have registered all of our functions | 23:42 |
| anteaya | zaro: thank you | 23:42 |
| notmorgan | jeblair: i could see that | 23:43 |
| notmorgan | jeblair: so we're racing our setup. | 23:43 |
| *** amitgandhinz has quit IRC | 23:43 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Register functions in the RPC listener before running https://review.openstack.org/328566 | 23:44 |
| notmorgan | so we're hanging in submitjob.in the time.sleep(0.1) | 23:44 |
| jeblair | notmorgan: ^ | 23:45 |
| notmorgan | yah | 23:45 |
| notmorgan | aha | 23:45 |
| notmorgan | want me to rebase my logscraper on that patch? | 23:45 |
| jeblair | MANY BEERS FOR NOTMORGAN! | 23:45 |
| notmorgan | (or a copy of it even w/ DNM) | 23:45 |
| jeblair | notmorgan: yeah, how about a dnm copy? | 23:46 |
| notmorgan | that is an annoying bug... i'm guessing cloud instances are under JUST enough contention for resources you get weird timing issues. | 23:46 |
| notmorgan | jeblair: give me a moment will have it posted soon. | 23:46 |
| jeblair | prolly so | 23:46 |
| pabelanger | notmorgan: Excellent work! | 23:46 |
| *** jerryz has quit IRC | 23:47 | |
| jeblair | we fixed so many bugs today! | 23:47 |
| jeblair | we may have even fixed more than we created! (so rare!) | 23:47 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: DNM: TESTING FIX: Capture Logs when Timeout is hit https://review.openstack.org/328568 | 23:47 |
| notmorgan | ^ | 23:47 |
| pabelanger | I'm happy how well we are doing with zuul-launcher, awesome work to all | 23:47 |
| jeblair | hopefully we will get nothing useful out of either of those changes! | 23:48 |
| notmorgan | jeblair: that bug has been *VERY* annoying trying to write the py3 things and other things. | 23:48 |
| notmorgan | like... i just didn't want to look at gerrit zuul because "OMG TIMEOUT ISSUE" | 23:48 |
| *** xarses has quit IRC | 23:48 | |
| notmorgan | s/zuul/zuul patches | 23:48 |
| jeblair | yup. it has been very painful. | 23:49 |
| notmorgan | i am going to see if we can do subinterpreters for testr and capture the stderr, so ALRM can exit safely without burning the house down | 23:49 |
| pabelanger | zl02.o.o now processing jobs | 23:49 |
| notmorgan | also we now have the log capture thing as a tool if we need to pull it in. | 23:50 |
| notmorgan | for more of these bugs. [please lets not repeat this:P] | 23:50 |
| notmorgan | jeblair: on the plus side, your "fix" passed ;) | 23:50 |
| jeblair | notmorgan: do you think we can switch to gentle now? | 23:50 |
| jeblair | jenkins02 has run its last job. it was a failure. too bad. | 23:51 |
| pabelanger | notmorgan: a few rechecks for 328566 in order? | 23:51 |
| jeblair | i admit to being slightly vague on the long term plan here. | 23:51 |
| notmorgan | pabelanger: we want to recheck 328568 | 23:51 |
| notmorgan | pabelanger: because that has the log capture in it | 23:51 |
| pabelanger | okay | 23:52 |
| notmorgan | jeblair: so - i'm holding off on saying gentle is safe | 23:52 |
| jeblair | https://review.openstack.org/328521 is ready for some +2s, though i'm about to afk so don't +3 unless you want to babysit nodepool (i don't know which zmq changes cause it to fail -- i think only additions, but maybe removals too) | 23:52 |
| notmorgan | jeblair: i *think* so, but... to be fair, i am still guessing | 23:52 |
| jeblair | while you're at it, https://review.openstack.org/328523 could use some +2 love as well | 23:52 |
| anteaya | zaro: at the top of your paste you reference the storyboard.jar file: http://paste.openstack.org/show/510678/ but I can't see a storyboard.jar file: gerrit2@gerrit-nopuppet:~$ ls its-storyboard/src/main/java/com/googlesource/gerrit/plugins/its/storyboard/Storyboard | 23:52 |
| anteaya | StoryboardClient.java StoryboardItsFacade.java StoryboardModule.java | 23:52 |
| notmorgan | jeblair: and i'd really not wedge things for 40+m at a time if we are wrong. | 23:52 |
| jeblair | clarkb, notmorgan, fungi, mordred, pabelanger: also take a look at this pls: https://review.openstack.org/328504 | 23:54 |
| *** rwsu has quit IRC | 23:54 | |
| jeblair | i'm going to self-approve https://review.openstack.org/327768 with notmorgan's +1 it's pretty trivial | 23:54 |
| notmorgan | jeblair: yeah 504 lgtm | 23:55 |
| notmorgan | jeblair: also... yeesh when did we get close to 0.3MM patches? ;) | 23:56 |
| notmorgan | erm changesets. | 23:56 |
| lifeless | ttx: nvm all sorted | 23:56 |
| pabelanger | jeblair: any impact on 504 for 3rd party CI users? | 23:56 |
| lifeless | notmorgan: o/ | 23:56 |
| notmorgan | lifeless: oh hi! i want to talk testr and improving it some :) | 23:56 |
| notmorgan | lifeless: but... next week. | 23:57 |
| lifeless | kk | 23:57 |
| notmorgan | lifeless: cause it's 5pm on friday :) | 23:57 |
| *** kzaitsev_mb has joined #openstack-infra | 23:57 | |
| notmorgan | lifeless: and i need to drink some cider and eat food after the zuul debugging :) | 23:57 |
| jeblair | notmorgan: we do want https://review.openstack.org/328516 in though, right? | 23:57 |
| notmorgan | jeblair: that is your call. i am ok with holding it for a bit and seeing if we need it | 23:58 |
| notmorgan | i'd feel better not having that kind of hack in., i'm also ok with landing it because we shouldn't hit it :P | 23:58 |
| jeblair | notmorgan: is there a downside? it still only fires in the case of a hard timeout, which is currently very bad for us anyway... | 23:58 |
| notmorgan | jeblair: no downside afaict because some icky hack code | 23:58 |
| notmorgan | but it shoudn't need to live too terribly long | 23:59 |
| notmorgan | so yeha lets get that in | 23:59 |
| jeblair | pabelanger: yes, it will make everyone trail gerrit by 10 seconds instead of 5. that will be *good* for openstack third-party ci's. it will be unecessary for unrelated zuul users, but hopefully they won't mind. | 23:59 |
| jeblair | probably we should make it configurable | 23:59 |
| jeblair | s/probably// | 23:59 |
| notmorgan | jeblair: if 5 seconds matter to the un-related zuul users, i think they might be doing something wrong tbh | 23:59 |
| pabelanger | jeblair: ya, was thinking configurable too. | 23:59 |
| notmorgan | and ++ to configurable | 23:59 |
| pabelanger | jeblair: we can do a follow up patch however | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
