Friday, 2016-02-12

« Thursday, 2016-02-11 Index Saturday, 2016-02-13 »
clarkbShrews: are you using gmail?00:00
*** stevebaker has quit IRC00:00
Shrewsclarkb: yes sir00:00
nibalizerfungi: urgh00:01
*** stevebaker has joined #openstack-infra00:01
Shrewshrm, openstack-dev emails seem MIA too00:02
*** amotoki has quit IRC00:02
fungiShrews: i see in review's exim log successful deliveries to your gmail.com address00:02
fungigmail is not rejecting these00:02
fungiC="250 2.0.0 OK 1455231015 np10si12159991oeb.30 - gsmtp"00:03
Shrewsfungi: oh, gmail has decided all openstack email is junk00:03
openstackgerritlifeless proposed openstack/requirements: Try to avoid future attempts to 'fix' usage of mock  https://review.openstack.org/27932900:03
openstackgerritSpencer Krum proposed openstack-infra/system-config: Add omfracloud to nodepool  https://review.openstack.org/27548500:03
openstackgerritSpencer Krum proposed openstack-infra/system-config: Add OmfraCloud to puppetmaster_clouds  https://review.openstack.org/27547700:03
fungiShrews: sometimes i'm temped to agree00:03
fungitempted00:03
Shrewsfungi: can't say i disagree with it, but this is terribly inconvenient00:03
*** lucas-dinner has quit IRC00:03
*** hashar has quit IRC00:03
fungialso we have, like, zero control over gmail's spam marking policies00:04
fungiafaik00:04
anteayasometimes gmail makes a wise assessment00:04
openstackgerritMadhusudhan Kandadai proposed openstack-infra/project-config: Add rally gate for neutron-vpnaas project  https://review.openstack.org/27933100:05
Shrews*sigh*00:05
*** baoli has quit IRC00:05
*** wolverineav has joined #openstack-infra00:06
*** wolverin_ has joined #openstack-infra00:06
*** rbrndt has quit IRC00:06
*** lucasagomes has joined #openstack-infra00:06
clarkbdougwig: preallocate swapfile merged almost 20 minutes ago so any jobs started from this point forward should hopeflly break properly if they need all the disk and memory00:07
dougwigclarkb: thanks, i'll keep an eye on the rally jobs for a bit.00:07
pleia2anteaya: haha00:09
clarkbShrews: in the spirit of broken email I just asked fastmail to delete 125k messages00:09
clarkbShrews: it got very angry with me00:09
anteaya:(00:09
Shrewsclarkb: i have had EVERY email classified as junk by gmail. i am not pleased00:10
*** jamielennox is now known as jamielennox|away00:10
jeblairjhesketh: not sure if you saw, but i think we have another regression that snuck in with connections:  https://review.openstack.org/27639500:10
clarkbthe folder completely disappeared after it spat out an error00:10
Shrewsno wonder i was getting so much work done00:10
clarkbShrews: ouch00:10
*** doug-fis_ has joined #openstack-infra00:10
anteayaShrews: haha haha00:10
clarkbI am worried I wil lhave to do the support ticket of shame "hey I tried to delete 125k messages because I saw I was using 50% of my quota and well the whole thing exploded on me"00:11
jeblairclarkb: i actually think they will think that's awesome00:12
anteayathe support ticket of shame00:12
anteayaI would think that is awesome00:12
clarkboh hey the emails are back and they are all marked read00:12
jeblairclarkb: i think bron is really proud of how fast they made cyrus00:12
clarkbwell :( I have to be smarter tha nthe computers00:12
prometheanfireianw: I don't understand :(00:13
*** zul__ has joined #openstack-infra00:13
clarkbit would be really neat to attach a retention policy to a folder00:13
jeblairclarkb: you could probably just post a bug report to the cyrus list :)00:13
jeblairclarkb: cyrus supports that00:13
jeblairclarkb: see mailbox annotations00:14
jeblairnot sure if fastmail does00:14
clarkbI hvaen't seen fastmail expose it00:14
clarkbbut I can look harder00:14
openstackgerritlifeless proposed openstack/requirements: Try to avoid future attempts to 'fix' usage of mock  https://review.openstack.org/27932900:14
*** daemontool_ has joined #openstack-infra00:14
ianwprometheanfire: why did you move it from 99 to 95?00:14
clarkboh hey there is a mass delete thing hidden under advanced settings00:14
anteayawooooo00:15
*** doug-fish has quit IRC00:15
*** zul_ has quit IRC00:15
*** daemontool has quit IRC00:15
*** mtanino__ has quit IRC00:15
jeblairclarkb: we used annotations to do that at cal; so you could say "delete anything in this folder older than 7 days"00:15
anteayapleia2: thanks00:15
clarkbjeblair: yup just found that on fastmail00:15
clarkbjeblair: this is what I needed it is just hidden away00:16
jeblairclarkb: cool00:16
*** mtanino__ has joined #openstack-infra00:16
pleia2anteaya: np00:16
prometheanfireianw: because other actions need to be taken after package-uninstalls runs00:17
*** Sukhdev has quit IRC00:17
*** moravec has quit IRC00:17
*** thorst has joined #openstack-infra00:17
nibalizerinfra-root way done with the lock, thanks00:18
jeblairclarkb: i want to say fastmail implemented a lot of that :)00:18
prometheanfireianw: trying not to take out my frustration on you, had a bad week with others jerking me around with reviews00:18
prometheanfireianw: you are not quite that bad :P00:18
*** Sukhdev has joined #openstack-infra00:19
*** Jeffrey4l has joined #openstack-infra00:20
clarkbjeblair: I have to say I have been quite pleased with their service. The recent outage was annoying but otherwise things have owrked well00:21
*** thorst has quit IRC00:21
*** wolverin_ has quit IRC00:22
*** wolverineav has joined #openstack-infra00:22
*** thorst has joined #openstack-infra00:22
anteayaclarkb: nice to hear00:23
ajmillerHi.  I've seen a couple of strange check/gate failures within the past half-hour or so.  zuul-cloner times out.  An example is https://jenkins05.openstack.org/job/gate-neutron-vpnaas-dsvm-functional/289/console00:25
nibalizerclarkb: okay so finally the auth stuff has been fixed, so we can mirrors tomorrow I guess?00:25
nibalizerassuming that picks up reviews in the night00:25
prometheanfireianw: behaviour or behavior :P00:25
*** salv-orlando has quit IRC00:25
anteayaajmiller: stderr: 'fatal: unable to connect to git.openstack.org:00:26
anteayaI see that as a connection error rather than a zuul cloner error00:26
*** austin81 has left #openstack-infra00:27
clarkbanteaya: ajmiller that is the thing I was collecting info  on00:27
clarkbhappens only to ovh, I have collected info on 22 instances and forwarded it to ovh00:27
clarkbtheory is something on ovh's side with networking00:27
clarkbjust started today00:28
ianwprometheanfire: here's what i would write : http://paste.openstack.org/show/486783/00:28
clarkbnibalizer: kk00:28
anteayaclarkb: awesome00:28
anteayaclarkb: so what is the signature, can't connect to git.o.o.?00:28
ajmillerclarb: thanks.  Another example is https://jenkins03.openstack.org/job/gate-neutron-vpnaas-dsvm-functional/242/00:29
nibalizerfungi: confirmed:00:29
nibalizernibz@compute004:~$ cat /etc/hostname00:29
nibalizercompute00400:29
clarkbanteaya: yes00:30
anteayaclarkb: great, do you care about more examples? or just, this is a known issue, waiting for reply from ovh00:30
prometheanfireianw: thanks00:30
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Reorder the package-uninstall action  https://review.openstack.org/27821100:30
*** thorst has quit IRC00:31
prometheanfireianw: I added the dep to the other one as well00:31
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Remove eclean-dist as it's not available by default  https://review.openstack.org/27658100:31
clarkbthey are trivially founf by logstash00:31
*** moravec has joined #openstack-infra00:31
anteayaso I hear no, you aren't interested in more examples00:32
anteayacool00:32
*** sridhar_ram has joined #openstack-infra00:33
clarkbinstances outside of ovh may be useful to refute the premise that only ovh is affected00:33
anteayaokay if I see it happen on any instances outside of ovh I'll speak up00:34
clarkbthe relative occurence according to health dashboard is low00:36
anteayagood00:36
clarkbso I dont want to stop using ovh00:36
*** erlon has quit IRC00:36
boris-42dougwig: clarkb so yep we can use nano flavor in gates and save some resources00:37
boris-42dougwig: clarkb for most of scenarios (everything except VMTask.*)00:37
anteayaclarkb: right, that is fair00:38
*** ashleighfarnham has quit IRC00:38
clarkbwhat does VMTask do?00:38
nibalizerfungi: so do you think i just ansible hostname assertively aacross the compute hosts?00:39
nibalizerthe controller seems to know its own hostname00:39
openstackgerritMerged openstack-infra/subunit2sql: Fix brackets on get_all_runs_time_series_by_key  https://review.openstack.org/27447000:40
crinklenibalizer: iirc mordred did something to its /etc/hosts that probably hasn't been done on the compute nodes00:42
*** moravec has quit IRC00:43
sdake_hey quick question clarkb00:44
clarkbsdake_: sure00:44
sdake_what is the endpoint for the authentication service at openstack00:44
sdake_.org00:44
clarkbsdake_: https://openstackid.org00:44
anteayabtw jenkins01 and 06 appear to be not doing their optimum at jenkining: http://grafana.openstack.org/dashboard/db/nodepool00:45
sdake_someone at cisco is having trouble submitting a gerrit patch but  their info looks correct on my screen00:45
*** annegentle_ has joined #openstack-infra00:45
clarkbsdake_: is it failing on the you need to submit contact info step?00:45
mordredclarkb: what did I do?00:45
mordredcah00:45
clarkbanteaya: 01 like 05 was not running the srevice00:46
*** devananda has joined #openstack-infra00:46
clarkbI will be so happy if that leak gets fixed but also if we just stop jenkinsing00:46
anteayafair enough00:46
clarkbanteaya: jenkins06 has a jenkins running but it is unhappy00:47
anteaya:(00:47
clarkbif I can shutdown mode it I will and if it is already shutdown mode I can clean it up and restart00:47
anteayaawesome thank you00:47
sdake_clarkb it fails with ssh://.... "did not work"00:47
*** arxcruz has quit IRC00:47
clarkbsdake_: well need more info than that :) can you paste the output of git review -v ?00:47
sdake_clarkb roger00:48
sdake_he does have his contact inf o set00:48
clarkbgit review -v should help quite a bit to narrow this down00:48
mtreinishclarkb: by health dashboard you mean openstack-health? if so that probably won't count git issues. Unless they occur during something that generates subunit like devstack00:48
clarkbmtreinish: it will record them as failed jobs won't it?00:49
sdake_clarkb can't paste it because it prompts for a user name00:49
clarkbmtreinish: on the whole we have had like 40 failures in ovh today and run many more tests than that there00:49
mtreinishclarkb: not if there is no subunit stream00:49
clarkbmtreinish: gotcha, well just with my napkin math it is very small percentage00:49
clarkbit still urts though00:49
*** annegentle_ has quit IRC00:49
clarkbsdake_: not sure I follow00:50
mtreinishsure00:50
sdake_paste coming up clarkb00:50
sdake_thanks for your help ;)00:50
mtreinishclarkb: fwiw, we can easily subunitize d-g that'll probably catch these issues00:50
clarkb40 cases in the last 24 hours00:51
mtreinishand also let us time d-g00:51
clarkbstill none outside of ovh00:51
sdake_clarkb paste.openstack.org/show/48678600:51
mtreinishactually I think i had a patch for that at some point00:52
clarkbsdake_: is the username used there the one that shows up at https://review.openstack.org/#/settings/ ?00:52
*** arthus has joined #openstack-infra00:52
clarkbsdake_: if not then you want to use the username that shows up at ^00:53
fungisdake_: so unfortunately, i think git-review has grown more opaque since it started test-pushing to try to confirm the account is valid. it doesn't know the difference between incorrect credentials and a rejection because of insufficient cla/contact in gerrit00:53
clarkbif it is the same username then check the ssh keys00:53
*** SumitNaiksatam has quit IRC00:53
fungiclarkb: the username and keys can be entirely correct and git-review will do that00:54
fungia better test is to try a non-cla-enforced repo like openstack-dev/sandbox first00:54
sdake_clarkb username = "naren.narendra"00:54
sdake_he just set his username about 10 mins ago00:55
sdake_is there a propogation delay possibly?00:55
clarkbno shouldn't be a delay on that00:55
fungisdake_: nothing in gerrit to propagate no00:55
clarkbI would check the ssh keys00:55
clarkbmake sure the lcaol keypair matches a public key in https://review.openstack.org/#/settings/ssh-keys00:56
clarkbalso note ecdsa keys own't work iirc00:56
clarkbI am correcting jenkins06 now00:56
funginaren.narendra is successfully logging into gerrit according to its logs00:56
anteayaclarkb: thank you00:56
fungi[2016-02-12 00:48:43,053 +0000] 58d2b6f4 naren.narendra a/20530 git-receive-pack./openstack/kolla.git 1ms 1ms 100:57
jlvillalfungi: I proposed this awhile back: https://review.openstack.org/#/c/271526/  Reverting the block on pip > 8.00:58
jlvillalfungi: Is that something that is good to keep alive or should I abandon it?00:58
fungisdake_: not sure what you were saying about "contact information filed" earlier but gerrit's database entirely disagrees that contact information has been submitted through the gerrit webui00:58
*** dangers is now known as dangers_away00:58
*** ajmiller has quit IRC00:58
sdake_fungi thanks00:59
sdake_thats it00:59
fungisdake_: was http://docs.openstack.org/infra/manual/developers.html#account-setup followed accurately?00:59
sdake_fungi he filled out openstackid contact info00:59
sdake_but not gerrit contact info00:59
clarkbanteaya: jenkins06 is coming back up again00:59
anteayawhen did we start using openstackid?00:59
fungisdake_: oh, yeah entirely unrelated00:59
*** baoli has joined #openstack-infra00:59
anteayaclarkb: woot!01:00
openstackgerritJesse Keating proposed openstack/requirements: Put an upper bound on eventlet  https://review.openstack.org/27934001:00
fungianteaya: openstackid is used to log into www.openstack.org to set up your foundation member profile01:00
anteayaright01:00
*** pvaneck has quit IRC01:00
anteayabut has nothing to do with gerrit at this point, that I know of01:00
fungii assume that's the "contact information" he was originally mistaking for gerrit's contact info api01:00
anteayaah01:00
jheskethfungi, clarkb: I have a quick patch if you two have a moment to look at https://review.openstack.org/#/c/278967/ that'd be handy please :-)01:00
*** moravec has joined #openstack-infra01:01
*** eil397 has quit IRC01:01
fungijlvillal: that looks like something we still want to do. thanks for tracking it01:02
jlvillalfungi: Okay.  I just wasn't sure :)  Thanks.01:02
*** ayoung has joined #openstack-infra01:02
clarkb Ineed a firefox plugin that treats my open tabs as a ring buffer unless I pin the tab01:02
clarkbthen maybe I won't have 400 open tabs01:02
fungijhesketh: ahh, yep i guess the old rule we had there from when the zuul v2 feature branch was in play got cleaned up at some point01:03
pleia2jhesketh: there you go01:03
clarkbanteaya: and 06 is running jobs again01:04
jheskethfungi: did we have a zuulv2 feature branch? I thought we just got fair enough along and tagged it (maybe it was before my time)01:04
jheskethpleia2, fungi: thanks heaps :-)01:04
fungijhesketh: i remember a feature branch01:04
clarkbgearman01:04
jheskethokay, I probably just nevern oticed01:04
fungibut that was many beers ago, so maybe i misremember01:04
fungiyeah, gearman implementation was zuul v2 right?01:04
sdake_folks fungi thanks for the pointer to that doc01:05
anteayaclarkb: yay!01:05
anteayathank you01:05
jheskethright I think when I started working on it it was already gearman01:05
fungisdake_: you're welcome. we try to make that document the only document on setting up an account, because what we have at the moment is fiddly and any time someone else tries to "summarize" it somewhere else they end up omitting necessary pieces01:06
anteayaclarkb: how can you work without 400 open tabs?01:06
fungisdake_: so if you/they were following other random instructions from somewhere, we'd appreciate a pointer so that we can try to get it removed/redirected to the currently maintained documentation01:06
clarkbanteaya: I periodically close all of them01:07
anteayaclarkb: no01:07
clarkbanteaya: but 95% of my RAM use is firefox01:07
anteayaI don't believe it01:07
anteayaha ha ha ha01:07
anteayaI do believe that01:07
clarkbfirefox has 1.6GB of resident memory01:07
anteayaha h aha01:07
clarkbout of 3.4GB used01:07
clarkbso not quite 95%01:07
anteayanice01:07
clarkbcloser to 50%01:07
anteayaa good 50%01:07
*** shashank_hegde has quit IRC01:08
fungiclarkb: i've found that firefox's delayed tab loading on start helps with a lot of that. i have a ton of tabs open but if i haven't actually viewed them since the last time i started the browser it doesn't seem to load them until i do01:08
anteayaI like to pretend like yesterday never happened01:08
fungiso periodically restarting the browser avoids me having to close older inactive tabs if i'm in a pinch for ram01:08
anteayabecause by the time I finish reading backscroll I usually have a page of tabs open01:09
clarkbmostly I am bad at closing tabs when I am done with them01:09
anteayafungi: ha ha ha01:09
fungianteaya: yeah, a "good" day is one where i end up with no more open browser tabs than i started it. most days are not good days01:09
anteayaclarkb: you are the person who leaves doors open beside them and the jam jar open on the kitchen counter01:10
openstackgerritMatthew Treinish proposed openstack-infra/devstack-gate: WIP: Subunitize devstack-gate  https://review.openstack.org/27934301:10
anteayafungi: ha ha ha01:10
anteayaI used to work retail01:10
anteayacity folks would open the door and never close it01:10
fungia "really good" day is one where i have time to look at tabs i opened on a previous day for reasons not having to do with something happening on the current day01:10
*** thorst has joined #openstack-infra01:10
anteayaused to spend half my time walking behind them as they stand in the open doorway to close the door01:10
anteayain winter they just stand there letting the heat out01:11
anteayafungi: wow, how mahy really good days have you had lately?01:11
*** thorst has quit IRC01:11
openstackgerritMerged openstack-infra/project-config: Allow zuul-release to push merge commits  https://review.openstack.org/27896701:11
anteayaseems 'closing the door behind you' is a rural concept01:12
fungianteaya: one a week on average, if i get lucky01:12
anteayanice01:12
*** thorst has joined #openstack-infra01:12
anteayas/you are/are you01:13
clarkbfungi: one thing I realized is that many/most/all of our post jobs run on bare-*01:13
anteayathat was a question, not an assertion01:13
clarkbthis is making the queueing there even worse because we have fewer bare-*01:13
clarkbanteaya: eh we do it in the city too otherwise someone might go thorugh my trash for cans01:13
*** moravec has quit IRC01:14
anteayagood point01:14
fungiclarkb: yeah, i have a change up to broaden the experimental bindep jobs to a bunch of projects01:15
anteayaI have to go outside and view the beautiful moon and feed the fire01:15
fungiunfortunately it's hard to really test most of the post jobs in the experimental pipeline01:15
anteayathen I'm going to do some not online things and go to bed01:15
anteayag'night01:15
fungii added coverage, but it's sort of a bad post example anyway01:15
fungi'night anteaya01:16
fungiclarkb: something like 6 jobs on 14 projects (some servers, some clients)01:16
fungiclarkb: so i can push some empty dnm changes up for them and see what the results of running on basically branch tip are01:17
*** alex_xu_ has quit IRC01:17
*** eliqiao_ has quit IRC01:17
*** dolpher has quit IRC01:17
fungiin theory this should be a seamless transition but the experimental runs will give us some assurance of that01:17
*** shashank_hegde has joined #openstack-infra01:18
clarkbcool01:18
*** gildub has quit IRC01:18
clarkband the ubuntu-trusty images are relatively recent in all lcouds01:18
*** gildub has joined #openstack-infra01:18
*** gildub has quit IRC01:19
*** thorst has quit IRC01:20
*** gyee has joined #openstack-infra01:21
*** mtanino has quit IRC01:22
*** moravec has joined #openstack-infra01:22
*** eliqiao_ has joined #openstack-infra01:22
*** dolpher has joined #openstack-infra01:23
*** jpr has joined #openstack-infra01:24
*** sridhar_ram has quit IRC01:25
*** alex_xu has joined #openstack-infra01:25
*** slogan621 has joined #openstack-infra01:27
slogan621Is it possible to have a jobs: section without content, or perhaps a job that is a no-op. This is for an initial commit and we want to defer turning on things like python-jobs until a later time. https://review.openstack.org/#/c/275873/6/jenkins/jobs/projects.yaml01:28
clarkbslogan621: yes, but you don't need anything in jenkins/jobs, you just add the noop jobs to your project(s) in zuul/layout.yaml01:28
jeblairslogan621: yes; add nothing to that file, and only add the "noop" job to zuul01:28
slogan621I had one person say we could submit with no jobs yet removing the section seems to fail the parser01:29
slogan621oh, so, don't put anything in that file?01:29
slogan621ok :-)01:29
jeblairslogan621: right, then use the noop-jobs template in zuul/layout.yaml01:30
slogan621is publishing to pypi dependent on any of the other jobs? I'd like to see us in pypi if possible, now that I have actually created pypi projects for it01:30
*** baoli has quit IRC01:30
jeblairslogan621: make your project in zuul/layout.yaml look like openstack-dev/cookiecutter to get the noop behavior01:30
slogan621jeblair: ok01:31
*** wolverin_ has joined #openstack-infra01:31
*** wolverineav has quit IRC01:31
*** wolverin_ has joined #openstack-infra01:31
jeblairslogan621: to publish to pypi, you'll want 'python-jobs' in jenkins/jobs/projects.yaml and the 'publish-to-pypi' jobs in zuul01:32
*** SumitNaiksatam has joined #openstack-infra01:32
slogan621ok, we'll just wait on that then01:32
slogan621learned so much in this process, thanks everyone01:33
clarkbok, jenkinses should be relatively happy. d-g uses preallocated swap file now, rally is moving to use nano flavor, ovh emailed about the git.o.o network fails, anything else we need to do to make tests more reliable?01:33
jeblairslogan621: also i think 'pypi-jobs' in jenkins/jobs01:33
clarkbslogan621: also note that with python jobs you can have a trivial assertTrue(True) test to start01:33
clarkbmay make transitioning to real tests easier if you do that work upfront01:33
jeblairgood point01:33
jeblairslogan621: also note that the pep8 job has a side effect of testing your tarball builds01:34
jeblair(that's a long story)01:34
*** jamielennox|away is now known as jamielennox01:34
*** moravec has quit IRC01:35
slogan621jeblair: we'll just stay out of there for now. The absolute simple case is what works best for us. I would have turned on pep8, python27 but got errors from tox because it could not find a git repo (which makes sense)01:35
slogan621we'll just get it later01:36
*** tzn has quit IRC01:39
clarkbjeblair: krotscheck the old pypi mirrors are running some /bin/bash /etc/wheel_mirror/rebuild_wheel_afs_index.sh /srv/static/mirror/web/wheel and cronspamming because find: `/srv/static/mirror/web/wheel/*': No such file or directory01:40
jeblairclarkb: weren't they deleted?01:40
clarkbjeblair: not according to my email from 9 minutes ago01:40
* clarkb tests with ssh01:40
jeblairfungi: do i remember correctly you were on that ^?01:40
clarkbpypi.gra1.openstack.org is a thing01:41
*** arthus has quit IRC01:41
clarkbthat is the vast majority of my recent cronspam01:42
jeblairat any rate, i'm ready for them to go01:42
*** FallenPegasus has joined #openstack-infra01:42
*** Gordoni has quit IRC01:42
clarkbkk01:42
fungii can delete them now, sure01:43
fungiand the old pbx.o.o that i shutdown last week too01:43
*** yamamoto_ has joined #openstack-infra01:44
*** bpokorny has quit IRC01:47
*** sdake_ is now known as sdake01:47
fungiwe don't have a credential file for ovh-bhs1 on puppetmaster?01:48
fungifor the openstackci tenant i mean01:49
clarkbat the very least should be in clouds.yaml01:49
*** nwkarsten has joined #openstack-infra01:51
fungiodd, pypi.region-b.geo-1.openstack.org is still in our ansible inventory01:51
*** Daisy has joined #openstack-infra01:52
clarkbzombie01:54
*** thorst has joined #openstack-infra01:54
*** thorst has quit IRC01:55
*** harlowja has quit IRC01:55
*** ddieterly has joined #openstack-infra01:55
*** thorst has joined #openstack-infra01:56
fungiadded bhs101:56
*** kzaitsev_mb has quit IRC01:56
*** yamahata has quit IRC01:56
fungiand deleted all the old pypi mirror hosts now01:56
*** eliqiao_ has quit IRC01:56
*** Daisy has quit IRC01:56
fungiand the old pbx.o.o that was shutdown01:57
*** edmondsw has quit IRC01:57
*** eliqiao_ has joined #openstack-infra01:58
*** amotoki has joined #openstack-infra01:58
openstackgerritMadhusudhan Kandadai proposed openstack-infra/project-config: Add rally gate job for neutron-vpnaas project  https://review.openstack.org/27933101:58
*** ybathia has quit IRC02:01
*** amotoki has quit IRC02:02
*** rossella_s has quit IRC02:03
*** rossella_s has joined #openstack-infra02:03
*** tiswanso has joined #openstack-infra02:03
*** yamamoto_ has quit IRC02:04
*** asahlin has quit IRC02:04
*** thorst has quit IRC02:04
openstackgerritMerged openstack-infra/shade: Use release version of Ansible for testing  https://review.openstack.org/27905702:05
*** asahlin has joined #openstack-infra02:05
*** asahlin_ has joined #openstack-infra02:07
*** akshai has quit IRC02:07
*** akshai has joined #openstack-infra02:08
*** nwkarsten has quit IRC02:08
*** nwkarsten has joined #openstack-infra02:09
*** akshai has quit IRC02:10
*** asahlin has quit IRC02:10
*** nwkarsten has quit IRC02:14
*** Sukhdev has quit IRC02:15
*** shashank_hegde has quit IRC02:21
openstackgerritgreghaynes proposed openstack/diskimage-builder: Install docker for tests  https://review.openstack.org/27802102:21
openstackgerritgreghaynes proposed openstack/diskimage-builder: Fail functests if refusing to run tests  https://review.openstack.org/17700202:21
*** tzn has joined #openstack-infra02:23
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Merge master into feature/zuulv3  https://review.openstack.org/27936402:23
*** arthus has joined #openstack-infra02:26
greghaynesianw: left you a reply on https://review.openstack.org/#/c/276563/102:28
*** ggillies has joined #openstack-infra02:29
*** shashank_hegde has joined #openstack-infra02:29
greghaynesclarkb: https://review.openstack.org/#/c/268241/ Is waiting for some feedback on you, FYI02:31
*** r-daneel has quit IRC02:32
clarkbit can be abandoned02:33
greghaynesok02:33
ianwgreghaynes: i don't much mind.  we can also just specify the tests we want to run in the tox.ini ... which is probably a better idea as it's pretty clear then what is running and what isn't02:34
greghaynesianw: I just dont want to be specifying which tests to run at all, but I am totally on board with a skip list02:34
*** ccrouch has quit IRC02:34
ianwgreghaynes: i dunno, i'm sort of liking the idea of that you run tox, and the test list is specified in tox.ini.  if you run by hand, it can just go ahead and run everything (or, you can specify an argument when you're just working on your one test)02:36
ianwit sort of keeps the list of things close to where its' being run, if that makes sense02:37
greghaynesianw: Where that list exists seems unrelated to whether its a opt-in or out list. I am fine with it being in either location02:37
ianwgreghayens: but an opt-in list also functions as a list of what will be run, as opposed to a list of what is *not* run, which to figure out what is run you then need to invert manually02:38
ianwand, hopefully, we have many more tests than we hvae time to run on every change :)02:39
greghaynesianw: Sure, I don't get the real value of having that list, if we really want it then we can add an argument to the test runner to generate the list for us (taking out the skip list). I am a lot more concerned that we will silently be skipping tests without realizing it if its opt-in02:40
*** sdake has quit IRC02:40
greghaynesOr said differently, we should do the safe thing by default (which is to run a test) and require extra input to do the unsafe thing02:41
*** zz_dimtruck is now known as dimtruck02:43
*** sdake has joined #openstack-infra02:44
*** baoli has joined #openstack-infra02:44
*** thorst has joined #openstack-infra02:46
prometheanfireianw: thanks02:48
prometheanfiregreghaynes: if you are still up, mind looking at https://review.openstack.org/276581 https://review.openstack.org/27821102:48
greghaynesif those are dib things im working up the stack right now02:48
*** yamahata has joined #openstack-infra02:48
*** moravec has joined #openstack-infra02:49
prometheanfiresure02:50
*** yamamoto_ has joined #openstack-infra02:51
*** kzaitsev_mb has joined #openstack-infra02:52
*** salv-orlando has joined #openstack-infra02:53
*** salv-orlando has quit IRC02:55
*** annegentle_ has joined #openstack-infra03:00
openstackgerritIan Wienand proposed openstack/diskimage-builder: Rework functional test runner  https://review.openstack.org/27656303:01
ianwgreghaynes: ^ that has a skip list03:01
greghaynesianw: awesome, ty03:02
openstackgerritMatthew Treinish proposed openstack-infra/devstack-gate: WIP: Subunitize devstack-gate  https://review.openstack.org/27934303:02
*** amrith is now known as _amrith_03:02
*** thorst has quit IRC03:02
*** thorst has joined #openstack-infra03:03
*** tphummel has quit IRC03:06
*** ajmiller has joined #openstack-infra03:06
*** shashank_hegde has quit IRC03:08
*** ajmiller_ has joined #openstack-infra03:08
*** gyee has quit IRC03:09
greghaynesdid we have some git.o.o explosions today? I just saw a test fail due to a failure to clone a repo03:10
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Merge master into feature/zuulv3  https://review.openstack.org/27936403:10
*** dims has quit IRC03:11
*** dims has joined #openstack-infra03:11
*** thorst has quit IRC03:12
*** annegentle_ has quit IRC03:12
*** ajmiller has quit IRC03:12
ajmiller_greghaynes: Yes, apparently there has been an issue with one of the nodepool providers03:12
*** ajmiller_ is now known as ajmiller03:13
ajmillerhttp://status.openstack.org//elastic-recheck/#129800603:13
*** sabeen3 has quit IRC03:13
greghaynesah, fun03:13
greghaynesty03:13
ajmillerYeah, its really messing with some patches I'm trying to get merged.03:14
ianwgreghaynes: don't know if you want to weigh in on -> https://review.openstack.org/#/q/topic:nodepool-dib-git  (topic is fairly self explanatory)03:14
greghaynesThe clouds do not dissapoint ;)03:14
ianw(otherwise, a dib release after those f23 fixups merge would be great :)03:14
greghaynesianw: ah, ok. I think really its the infra rooters who should weigh in on that since they are the ones who would be dealing with the effects03:15
greghaynesI really have no strong opinions on that03:15
ianwyeah, i may have to turn it into a spec if we can't get some consensus03:15
mgagnelooks like ZMQ plugin for Jenkins is missing from update center. I found this bug: https://issues.jenkins-ci.org/browse/INFRA-55103:17
mgagnecan someone validate? zmq plugin missing from Jenkins update center clarkb ^03:18
*** doug-fis_ has quit IRC03:19
*** wolverineav has joined #openstack-infra03:19
*** doug-fish has joined #openstack-infra03:21
*** doug-fish has quit IRC03:21
*** akshai has joined #openstack-infra03:22
*** baoli has quit IRC03:22
*** wolverin_ has quit IRC03:22
*** kzaitsev_mb has quit IRC03:23
*** claudiub has quit IRC03:28
*** moravec has quit IRC03:30
cody-somervilleFascinating. The new Lumberyard 3D game engine that Amazon just released explicitly prohibits running  Lumberyard on any "Alternate Web Service" and games you develop with it can not read or write data to any "Alternate Web Service". Alternate Web Service meaning any non-AWS compute, database, storage, or container service that is similar to or can act as a replacement for AWS.03:30
*** apoorvad has quit IRC03:30
ianwthere's no such thing as a free lunch03:31
*** dims has quit IRC03:31
*** sridhar_ram has joined #openstack-infra03:31
*** links has joined #openstack-infra03:32
*** wolverineav has quit IRC03:32
prometheanfiregreghaynes: thankyou sir03:33
greghaynesprometheanfire: thank you :)03:33
prometheanfirehopefully that's it for the initial add and I can focus on adding support to specific elements so that gentoo can be tested against03:36
*** sdake has quit IRC03:37
*** achanda has joined #openstack-infra03:39
*** gildub has joined #openstack-infra03:40
*** arthus has quit IRC03:40
*** achanda has quit IRC03:41
*** ddieterly has quit IRC03:41
*** hdd has joined #openstack-infra03:43
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: create_service() should normalize return value  https://review.openstack.org/27409103:43
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Add docs tox target  https://review.openstack.org/27687003:43
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Clarify Munch object usage in documentation  https://review.openstack.org/27687803:44
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Add enabled flag to keystone service data  https://review.openstack.org/27411603:44
*** sdake has joined #openstack-infra03:45
*** achanda has joined #openstack-infra03:46
*** woodster_ has quit IRC03:46
*** rguillebert has quit IRC03:47
*** akshai_ has joined #openstack-infra03:49
*** akshai has quit IRC03:53
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: Move gate-networking-midonet-v2-dsvm-rally-nv out of experimental  https://review.openstack.org/27937303:53
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: Move gate-networking-midonet-v2-dsvm-rally-nv out of experimental  https://review.openstack.org/27937303:54
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Update SSL configuration  https://review.openstack.org/27928603:55
*** rakhmerov has quit IRC03:57
*** rakhmerov has joined #openstack-infra03:58
*** sdake has quit IRC04:03
*** hamzy has quit IRC04:03
openstackgerritMerged openstack/diskimage-builder: Fix startup race with growroot/systemd  https://review.openstack.org/27818704:03
openstackgerritMerged openstack/diskimage-builder: Do not remove python-dev from ironic-agent image  https://review.openstack.org/27903104:04
openstackgerritMerged openstack/diskimage-builder: Reorder the package-uninstall action  https://review.openstack.org/27821104:04
openstackgerritColleen Murphy proposed openstack-infra/system-config: Update InfraCloud SSL configuration  https://review.openstack.org/27937504:05
*** thorst has joined #openstack-infra04:10
*** annegentle_ has joined #openstack-infra04:12
*** thorst has quit IRC04:16
*** yamahata has quit IRC04:17
*** annegentle_ has quit IRC04:17
*** kzaitsev_mb has joined #openstack-infra04:19
*** hamzy has joined #openstack-infra04:20
*** slogan621 has quit IRC04:21
*** kzaitsev_mb has quit IRC04:24
*** arthus has joined #openstack-infra04:26
*** asahlin_ has quit IRC04:33
*** salv-orlando has joined #openstack-infra04:37
*** fedexo has joined #openstack-infra04:40
*** amotoki has joined #openstack-infra04:41
*** ddieterly has joined #openstack-infra04:42
*** salv-orlando has quit IRC04:42
*** sridhar_ram has quit IRC04:44
*** ddieterly has quit IRC04:47
*** shashank_hegde has joined #openstack-infra04:49
*** tiswanso has quit IRC04:53
*** kushal has joined #openstack-infra04:58
openstackgerritSachi King proposed openstack-infra/project-config: Update stable constraints url  https://review.openstack.org/26794105:01
*** FallenPegasus has quit IRC05:08
*** FallenPegasus has joined #openstack-infra05:09
*** bdemers_ has joined #openstack-infra05:09
*** charz_ has quit IRC05:09
*** dtroyer has quit IRC05:09
*** vmbrasseur has quit IRC05:10
*** dimtruck has quit IRC05:10
*** comstud has quit IRC05:10
*** pc_m has quit IRC05:10
*** mgagne has quit IRC05:10
*** hockeynut has quit IRC05:10
*** pedroalvarez has quit IRC05:10
*** arthus has quit IRC05:10
*** pedroalvarez has joined #openstack-infra05:10
*** changbl has quit IRC05:11
*** camunoz has quit IRC05:11
*** camunoz has joined #openstack-infra05:11
*** dtroyer has joined #openstack-infra05:11
*** rm_work has quit IRC05:11
*** stevemar has quit IRC05:11
*** hughsaunders has quit IRC05:11
*** mcarden has quit IRC05:11
*** vmbrasseur has joined #openstack-infra05:11
*** hockeynut has joined #openstack-infra05:12
*** pc_m has joined #openstack-infra05:12
*** bdemers has quit IRC05:12
*** dimtruck has joined #openstack-infra05:12
*** mcarden has joined #openstack-infra05:12
*** charz has joined #openstack-infra05:12
*** mgagne has joined #openstack-infra05:12
*** mgagne is now known as Guest368705:12
*** stevemar has joined #openstack-infra05:13
*** changbl has joined #openstack-infra05:13
*** arthus has joined #openstack-infra05:14
*** tphummel has joined #openstack-infra05:15
*** thorst has joined #openstack-infra05:15
*** hughsaunders has joined #openstack-infra05:17
*** rm_work has joined #openstack-infra05:18
*** hichihara has joined #openstack-infra05:18
*** comstud has joined #openstack-infra05:18
*** ajmiller_ has joined #openstack-infra05:18
*** Jeffrey4l has quit IRC05:19
*** chlong has quit IRC05:19
*** jaosorior has joined #openstack-infra05:20
*** ajmiller has quit IRC05:21
*** thorst has quit IRC05:21
*** ajmiller_ has quit IRC05:21
*** boris-42 has quit IRC05:24
*** vgridnev has joined #openstack-infra05:24
*** chlong has joined #openstack-infra05:33
*** achanda has quit IRC05:38
*** Jeffrey4l has joined #openstack-infra05:41
*** chlong has quit IRC05:42
*** tzn has quit IRC05:44
*** ddieterly has joined #openstack-infra05:44
*** ddieterly has quit IRC05:48
*** salv-orlando has joined #openstack-infra05:54
*** chlong has joined #openstack-infra05:55
*** mattoliverau has quit IRC05:55
*** salv-orlando has quit IRC05:56
*** achanda has joined #openstack-infra05:57
*** wolverineav has joined #openstack-infra06:01
*** rossella_s has quit IRC06:03
*** rossella_s has joined #openstack-infra06:03
*** YorikSar_ has quit IRC06:05
*** wolverineav has quit IRC06:05
*** jaosorior has quit IRC06:06
*** jaosorior has joined #openstack-infra06:06
*** jaosorior has quit IRC06:06
*** YorikSar has joined #openstack-infra06:07
*** sc68cal has quit IRC06:07
*** achanda has quit IRC06:09
*** jaosorior has joined #openstack-infra06:11
*** achanda has joined #openstack-infra06:11
*** chlong has quit IRC06:11
*** wolverineav has joined #openstack-infra06:18
*** thorst has joined #openstack-infra06:20
*** achanda has quit IRC06:20
*** kzaitsev_mb has joined #openstack-infra06:21
*** achanda has joined #openstack-infra06:21
*** hdd has quit IRC06:23
*** FallenPegasus has quit IRC06:25
*** thorst has quit IRC06:26
*** kzaitsev_mb has quit IRC06:26
*** achanda has quit IRC06:28
*** med_ has quit IRC06:28
*** FallenPegasus has joined #openstack-infra06:32
*** chlong has joined #openstack-infra06:35
*** kaisers has left #openstack-infra06:37
*** arthus has quit IRC06:40
*** tzn has joined #openstack-infra06:40
*** arthus has joined #openstack-infra06:41
*** med_ has joined #openstack-infra06:43
*** med_ is now known as Guest7650706:43
*** ddieterly has joined #openstack-infra06:45
*** smanjara has joined #openstack-infra06:46
*** ddieterly has quit IRC06:50
*** kushal has quit IRC06:53
*** armax has quit IRC06:54
*** gildub has quit IRC06:55
*** camunoz has quit IRC06:57
*** [1]Thelo has joined #openstack-infra06:57
openstackgerritKanagaraj Manickam proposed openstack-infra/project-config: Adds new project oslo.namos  https://review.openstack.org/27939206:58
*** kushal has joined #openstack-infra06:59
openstackgerritMasayuki Igawa proposed openstack-infra/project-config: Add python3-jobs to openstack-health  https://review.openstack.org/27939307:00
*** Thelo has quit IRC07:00
*** [1]Thelo is now known as Thelo07:00
*** matt6434 has joined #openstack-infra07:02
*** boris-42 has joined #openstack-infra07:02
*** chlong has quit IRC07:03
*** matt6434 is now known as mattoliverau07:04
*** k4n0 has joined #openstack-infra07:05
*** _nadya_ has joined #openstack-infra07:09
*** wolverineav has quit IRC07:10
*** salv-orlando has joined #openstack-infra07:12
*** dimtruck is now known as zz_dimtruck07:13
*** tzn has quit IRC07:14
*** wolverineav has joined #openstack-infra07:15
*** KanagarajM has joined #openstack-infra07:16
*** tphummel has quit IRC07:16
*** amotoki has quit IRC07:16
openstackgerritKanagaraj Manickam proposed openstack-infra/project-config: Python-namosclient  https://review.openstack.org/27939907:18
openstackgerritKanagaraj Manickam proposed openstack-infra/project-config: Adds new project oslo.namos  https://review.openstack.org/27939207:19
*** k4n0 has quit IRC07:21
*** kzaitsev_mb has joined #openstack-infra07:22
*** salv-orlando has quit IRC07:23
*** hichihara has quit IRC07:24
*** amotoki has joined #openstack-infra07:24
*** thorst has joined #openstack-infra07:24
*** rubasov has quit IRC07:26
*** kzaitsev_mb has quit IRC07:27
*** lmiccini|away is now known as lmiccini07:27
*** zeih_ has joined #openstack-infra07:29
*** ianw has quit IRC07:30
*** ianw has joined #openstack-infra07:31
*** thorst has quit IRC07:31
*** amotoki has quit IRC07:35
*** amotoki has joined #openstack-infra07:39
*** amotoki has quit IRC07:43
*** scheuran has joined #openstack-infra07:45
*** ddieterly has joined #openstack-infra07:45
*** Sukhdev has joined #openstack-infra07:49
*** ddieterly has quit IRC07:51
*** Vivek has quit IRC07:51
*** kushal has quit IRC07:53
*** sflanigan has quit IRC07:55
*** amotoki has joined #openstack-infra07:55
*** vgridnev has quit IRC07:58
*** sputnik13 has joined #openstack-infra07:58
*** achanda has joined #openstack-infra07:58
*** Vivek has joined #openstack-infra08:02
*** flepied has quit IRC08:02
*** fedexo has quit IRC08:02
*** wolverineav has quit IRC08:07
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix tag publishing of server projects  https://review.openstack.org/27923808:07
*** YorikSar has quit IRC08:09
*** YorikSar has joined #openstack-infra08:10
*** dtardivel has joined #openstack-infra08:10
*** shashank_hegde has quit IRC08:10
*** tzn has joined #openstack-infra08:11
*** zz_ja has quit IRC08:11
yolandagood morning08:12
*** zz_ja has joined #openstack-infra08:13
*** kzaitsev_mb has joined #openstack-infra08:23
*** jlanoux has joined #openstack-infra08:24
*** dizquierdo has joined #openstack-infra08:24
*** esker has joined #openstack-infra08:24
*** rubasov has joined #openstack-infra08:26
openstackgerritMerged openstack/diskimage-builder: Increase interface has link retries to 20  https://review.openstack.org/27717708:26
*** dingyichen has quit IRC08:26
*** harlowja_at_home has quit IRC08:27
*** mikelk has joined #openstack-infra08:28
*** salv-orlando has joined #openstack-infra08:28
*** kzaitsev_mb has quit IRC08:28
*** markusz is now known as markus_z08:28
*** esker has quit IRC08:29
*** kushal has joined #openstack-infra08:30
*** thorst has joined #openstack-infra08:30
*** Daisy has joined #openstack-infra08:30
*** _nadya_ has quit IRC08:31
*** arxcruz has joined #openstack-infra08:33
*** salv-orlando has quit IRC08:35
*** |-paul-| has joined #openstack-infra08:35
*** thorst has quit IRC08:36
*** kozhukalov has joined #openstack-infra08:37
*** flepied has joined #openstack-infra08:37
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Add class to deal with cacert on infra cloud  https://review.openstack.org/27836208:38
*** arthus has quit IRC08:39
*** zz_ja has quit IRC08:39
*** zz_ja has joined #openstack-infra08:40
*** slogan621 has joined #openstack-infra08:41
*** ifarkas has joined #openstack-infra08:42
*** FallenPegasus has quit IRC08:44
*** tzn has quit IRC08:44
*** esker has joined #openstack-infra08:45
*** arthus has joined #openstack-infra08:45
*** ddieterly has joined #openstack-infra08:46
*** Daisy has quit IRC08:47
*** Daisy has joined #openstack-infra08:47
KanagarajMAJaeger,  hi,  arround? like to get help on https://review.openstack.org/#/c/279392/208:48
*** isaacb has joined #openstack-infra08:48
*** esker has quit IRC08:49
*** e0ne has joined #openstack-infra08:49
*** vgridnev has joined #openstack-infra08:50
*** ddieterly has quit IRC08:50
*** Daisy has quit IRC08:52
*** abregman has joined #openstack-infra08:54
*** abregman has quit IRC08:55
*** abregman has joined #openstack-infra08:55
*** isaacb has quit IRC08:58
*** dizquierdo has quit IRC09:01
openstackgerritKanagaraj Manickam proposed openstack-infra/project-config: Adds new project oslo.namos  https://review.openstack.org/27939209:01
AJaegerKanagarajM: what kind of help do you need?09:01
KanagarajMAJaeger, for the governence, should i add the patch now?09:02
AJaegeryes, please do.09:03
openstackgerritMerged openstack-infra/project-config: Add distil jobs back  https://review.openstack.org/27882709:03
KanagarajMAJaeger, its at very early stage.09:03
AJaegerand then have the oslo PTL +1 all changes09:03
AJaegerKanagarajM: that's not a problem.09:03
AJaegergovernance does not imply stability, just team assocication.09:03
openstackgerritMerged openstack-infra/project-config: Remove neutron grenade from lbaas jobs lists  https://review.openstack.org/27926609:04
KanagarajMAJaeger, sure. kindly help me on how to apply for governence?09:04
AJaegerKanagarajM: Did you read the infra manual on setting up the change?09:04
*** kushal has quit IRC09:04
AJaegerKanagarajM: If your project is part of oslo, adding it to governance repo is just house keeping.09:05
KanagarajMAJaeger, reffered  http://docs.openstack.org/infra/manual/creators.html09:05
AJaegerYes, that one contains a section on what to do09:05
*** salv-orlando has joined #openstack-infra09:05
KanagarajMAJaeger, http://docs.openstack.org/infra/manual/creators.html#add-new-repository-to-the-governance-repository this one, the next step :)09:06
*** esker has joined #openstack-infra09:06
KanagarajMAJaeger, for this one https://review.openstack.org/279399  is governanace required ?09:07
AJaegeryes, that'S the next step09:07
*** wolverineav has joined #openstack-infra09:08
AJaegerKanagarajM: I suggest you discuss with oslo PTL first - and you explain in your commit messages a bit how everything fits together. Right now I don't know what 279399 does (what's the server?) and why it's needed09:08
AJaegerAh, there's openstack/namos - then state it!09:09
*** sputnik13 has quit IRC09:09
openstackgerritMerged openstack-infra/project-config: Add job gate-swift-tox-func-in-process-fast-post  https://review.openstack.org/27682309:09
KanagarajMAJaeger, sure. I will update the commit message. thanks.09:09
AJaegerSo, no governance needed for python-namosclient at this stage09:09
KanagarajMAJaeger, sure.09:10
AJaegerBut for anything in the olso namespace, talk with oslo team first...09:10
KanagarajMAJaeger, sure.09:11
*** esker has quit IRC09:11
*** ihrachys has joined #openstack-infra09:12
*** jtomasek has joined #openstack-infra09:13
*** wolverineav has quit IRC09:13
*** zeih_ has quit IRC09:14
*** flepied has quit IRC09:15
openstackgerritMerged openstack-infra/project-config: Add a tinyIPA non-voting test to Ironic  https://review.openstack.org/26968409:16
*** flepied has joined #openstack-infra09:17
*** zeih_ has joined #openstack-infra09:17
openstackgerritMerged openstack-infra/project-config: import release-test repository  https://review.openstack.org/27849509:18
*** placko_ has joined #openstack-infra09:20
*** bauzas is now known as bauwser09:21
*** zeih_ has quit IRC09:22
*** kzaitsev_mb has joined #openstack-infra09:24
openstackgerritKanagaraj Manickam proposed openstack-infra/project-config: Adds new project Python-namosclient  https://review.openstack.org/27939909:26
*** rvba has quit IRC09:26
*** rvba has joined #openstack-infra09:27
*** rvba has quit IRC09:27
*** rvba has joined #openstack-infra09:27
*** achanda has quit IRC09:27
*** dtantsur|afk is now known as dtantsur09:27
*** esker has joined #openstack-infra09:27
*** Jeffrey4l has quit IRC09:28
*** kzaitsev_mb has quit IRC09:29
*** esker has quit IRC09:32
*** ociuhandu has quit IRC09:32
*** abregman is now known as abregman|mtg09:33
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: remove large-ops testing  https://review.openstack.org/27897809:34
*** amotoki has quit IRC09:34
*** thorst has joined #openstack-infra09:34
*** kushal has joined #openstack-infra09:37
*** _nadya_ has joined #openstack-infra09:37
*** Sukhdev has quit IRC09:37
openstackgerritKanagaraj Manickam proposed openstack-infra/project-config: Adds new project oslo.namos  https://review.openstack.org/27939209:41
*** thorst has quit IRC09:41
*** smanjara has quit IRC09:43
*** [HeOS] is now known as HeOS09:44
*** ddieterly has joined #openstack-infra09:47
*** amotoki has joined #openstack-infra09:48
*** kushal has quit IRC09:48
*** esker has joined #openstack-infra09:48
*** ddieterly has quit IRC09:51
*** salv-orlando has quit IRC09:52
*** ociuhandu has joined #openstack-infra09:52
*** esker has quit IRC09:53
*** amotoki has quit IRC09:53
*** dizquierdo has joined #openstack-infra09:54
*** amotoki has joined #openstack-infra09:56
openstackgerritVolodymyr Samotiy proposed openstack-infra/project-config: Add openstack/broadview-lib project  https://review.openstack.org/27587309:56
*** kzaitsev_mb has joined #openstack-infra09:57
*** sdake has joined #openstack-infra09:57
*** smanjara has joined #openstack-infra09:57
*** kozhukalov has quit IRC09:57
*** sdake_ has joined #openstack-infra10:00
*** amotoki has quit IRC10:01
*** kzaitsev_mb has quit IRC10:01
*** rhallisey has quit IRC10:01
*** kushal has joined #openstack-infra10:02
*** sdake has quit IRC10:03
*** salv-orlando has joined #openstack-infra10:03
*** placko_ has quit IRC10:06
*** esker has joined #openstack-infra10:09
*** amotoki has joined #openstack-infra10:10
*** ildikov has quit IRC10:11
*** amotoki has quit IRC10:12
*** amotoki has joined #openstack-infra10:13
*** placko_ has joined #openstack-infra10:13
*** esker has quit IRC10:14
*** rguillebert has joined #openstack-infra10:16
*** zeih_ has joined #openstack-infra10:19
*** sdake_ is now known as sdake10:25
openstackgerritTimur Sufiev proposed openstack-infra/project-config: Source exports from horizon repo for horizon dsvm-job  https://review.openstack.org/26957810:28
*** daemontool has joined #openstack-infra10:28
*** zeih_ has quit IRC10:29
*** ldnunes has joined #openstack-infra10:30
*** daemontool_ has quit IRC10:31
*** openstackgerrit has quit IRC10:32
*** openstackgerrit has joined #openstack-infra10:32
*** tzn has joined #openstack-infra10:34
*** _amrith_ is now known as amrith10:35
*** ldnunes has quit IRC10:35
*** ldnunes has joined #openstack-infra10:35
*** thorst has joined #openstack-infra10:40
*** notnownikki has joined #openstack-infra10:41
*** salv-orl_ has joined #openstack-infra10:41
*** dims has joined #openstack-infra10:41
*** hashar has joined #openstack-infra10:42
*** kushal has quit IRC10:44
*** kushal has joined #openstack-infra10:45
*** salv-orlando has quit IRC10:45
*** smanjara has quit IRC10:45
*** thorst has quit IRC10:47
*** esikachev has joined #openstack-infra10:47
*** ddieterly has joined #openstack-infra10:48
openstackgerritMoshe Levi proposed openstack/diskimage-builder: Adding InfiniBand Support  https://review.openstack.org/27478110:49
*** ddieterly has quit IRC10:52
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Drop test libs from dsvm+oslo.* master job  https://review.openstack.org/27946710:52
*** Daisy has joined #openstack-infra10:54
*** abregman|mtg is now known as abregman10:57
*** smanjara has joined #openstack-infra10:58
*** kzaitsev_mb has joined #openstack-infra11:02
*** zz_ja has quit IRC11:02
openstackgerritEvgeny Sikachev proposed openstack-infra/project-config: Update timeout for sahara-dashboard gate  https://review.openstack.org/27947411:03
*** zz_ja has joined #openstack-infra11:03
*** aspiers has quit IRC11:04
*** arthus has quit IRC11:07
*** aysyd has joined #openstack-infra11:11
openstackgerritEvgeny Sikachev proposed openstack-infra/project-config: Update timeout for sahara-dashboard gate  https://review.openstack.org/27947411:12
*** kzaitsev_mb has quit IRC11:12
esikachevAJaeger: patch is updated11:14
*** aysyd has quit IRC11:17
*** aysyd has joined #openstack-infra11:17
*** kozhukalov has joined #openstack-infra11:20
*** sileht has quit IRC11:20
*** alex_xu has quit IRC11:27
*** alex_xu has joined #openstack-infra11:28
*** achanda has joined #openstack-infra11:28
*** amotoki has quit IRC11:30
*** ldnunes has quit IRC11:32
*** ldnunes has joined #openstack-infra11:32
*** rcarrillocruz has joined #openstack-infra11:33
*** achanda has quit IRC11:34
*** amrith is now known as _amrith_11:35
*** kozhukalov has quit IRC11:35
*** marcusvrn_ has joined #openstack-infra11:36
*** sileht has joined #openstack-infra11:38
*** smanjara has quit IRC11:38
*** tpsilva has joined #openstack-infra11:40
AJaegersdague: could you review https://review.openstack.org/#/c/278051 https://review.openstack.org/#/c/278061 and https://review.openstack.org/#/c/278271/ , please?11:41
AJaegeryolanda: do you want to re-approve after a rebase this change, please? https://review.openstack.org/27897811:44
yolandasure11:44
*** thorst has joined #openstack-infra11:45
*** kzaitsev_mb has joined #openstack-infra11:48
yolandadegorenko so that was not the issue, when running from command line is discovered real one11:48
yolandaSSLError: [Errno 336265225] _ssl.c:355: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib11:48
yolandamore issues with certs11:48
yolandaoh, bad channel11:48
*** ddieterly has joined #openstack-infra11:49
openstackgerritAlexander Tivelkov proposed openstack/requirements: Bump yaql version to 1.1.0  https://review.openstack.org/27898311:51
*** thorst has quit IRC11:51
openstackgerritTomi Juvonen proposed openstack-infra/infra-manual: "Closes-bug" instead of "Fixes bug"  https://review.openstack.org/27949511:52
*** ddieterly has quit IRC11:53
*** rfolco has quit IRC11:54
*** rfolco has joined #openstack-infra11:54
openstackgerritMerged openstack-infra/project-config: remove large-ops testing  https://review.openstack.org/27897811:54
*** tojuvone has joined #openstack-infra11:55
*** zz_ja has quit IRC11:55
*** placko_ has quit IRC11:56
*** scheuran has quit IRC11:57
*** FallenPegasus has joined #openstack-infra11:57
*** Daisy has quit IRC11:57
*** zz_ja has joined #openstack-infra11:59
*** _nadya_ has quit IRC12:02
*** ganso has joined #openstack-infra12:04
*** _nadya_ has joined #openstack-infra12:05
*** hdd has joined #openstack-infra12:06
*** weshay has joined #openstack-infra12:06
*** _nadya_ has quit IRC12:06
*** links has quit IRC12:10
*** yamamoto_ has quit IRC12:12
*** aspiers has joined #openstack-infra12:13
KanagarajMAJaeger, oslo.namos patches will take more time, as i need to get approval from oslo ptl, and i have strong development dependency on the https://review.openstack.org/279399 , could you please have a look into it?12:15
openstackgerritMerged openstack-infra/infra-ansible: Remove shade-ansible from pip requirements  https://review.openstack.org/26799712:16
*** FallenPegasus has quit IRC12:16
openstackgerritMerged openstack-infra/infra-ansible: Remove unused docker-py dependency  https://review.openstack.org/26799812:16
openstackgerritMerged openstack-infra/infra-ansible: Use virtualenv python binary  https://review.openstack.org/26800112:16
openstackgerritDmitriy Ukhlov proposed openstack-infra/project-config: Adds neutron tempest tests for oslo.messaging drivers  https://review.openstack.org/27921512:17
*** FallenPegasus has joined #openstack-infra12:18
*** jlanoux has quit IRC12:21
*** zz_ja has quit IRC12:21
*** Jeffrey4l has joined #openstack-infra12:22
*** vgridnev has quit IRC12:25
*** baoli has joined #openstack-infra12:26
openstackgerritAndrey Pavlov proposed openstack-infra/puppet-refstack: use specific tag version of refstack for installation.  https://review.openstack.org/27951112:29
*** lucasagomes is now known as lucas-hungry12:30
*** zz_ja has joined #openstack-infra12:30
*** baoli_ has joined #openstack-infra12:33
*** vgridnev has joined #openstack-infra12:36
*** baoli has quit IRC12:36
*** daemontool has quit IRC12:36
openstackgerritsebastian marcet proposed openstack-infra/system-config: OpenStackId Production Release  https://review.openstack.org/27951312:36
*** |-paul-| has quit IRC12:38
openstackgerritMerged openstack/requirements: Update Oslo VersionedObjects to version 1.5.0  https://review.openstack.org/27900312:39
*** edmondsw has joined #openstack-infra12:40
*** smarcet has joined #openstack-infra12:40
*** jlk has quit IRC12:41
*** vgridnev has quit IRC12:41
*** rhallisey has joined #openstack-infra12:41
*** baoli has joined #openstack-infra12:42
*** Daisy has joined #openstack-infra12:44
*** thorst has joined #openstack-infra12:45
*** jlk has joined #openstack-infra12:45
*** baoli_ has quit IRC12:45
*** baoli_ has joined #openstack-infra12:46
*** jlk has joined #openstack-infra12:46
*** jlk has quit IRC12:46
*** vgridnev has joined #openstack-infra12:47
*** Daisy has quit IRC12:49
*** baoli has quit IRC12:49
*** ddieterly has joined #openstack-infra12:49
*** matrohon has joined #openstack-infra12:49
*** jaypipes is now known as leakypipes12:50
*** daemontool has joined #openstack-infra12:50
*** ddieterly has quit IRC12:54
*** ildikov has joined #openstack-infra12:55
*** aysyd has quit IRC12:56
pc_mI had a question on requriements.txt files. If a project does not call out a pacakage in requirements.txt is the global-requirements value used for a package?12:56
*** zeih_ has joined #openstack-infra12:57
*** salv-orl_ has quit IRC12:58
*** _amrith_ is now known as amrith12:58
*** ociuhandu has quit IRC12:59
*** ddieterly has joined #openstack-infra12:59
*** placko_ has joined #openstack-infra13:01
*** zz_ja has quit IRC13:03
*** jordanP has joined #openstack-infra13:03
*** jlk has joined #openstack-infra13:03
*** jlk has quit IRC13:04
*** jlk has joined #openstack-infra13:04
*** jordanP has quit IRC13:04
*** dprince has joined #openstack-infra13:04
*** jordanP has joined #openstack-infra13:04
*** zz_ja has joined #openstack-infra13:06
openstackgerritMax Lobur proposed openstack-infra/project-config: Create openstack/bareon-func-test repository  https://review.openstack.org/27912013:07
*** ociuhandu has joined #openstack-infra13:07
*** wolverineav has joined #openstack-infra13:09
*** placko_ has quit IRC13:10
*** hashar has quit IRC13:11
*** ociuhandu has quit IRC13:12
*** ociuhandu has joined #openstack-infra13:13
*** jlanoux has joined #openstack-infra13:13
*** ddieterl_ has joined #openstack-infra13:13
openstackgerritBulat Gaifullin proposed openstack-infra/project-config: Create openstack/packetary-specs repository  https://review.openstack.org/27952613:15
*** ddieterly has quit IRC13:15
*** wolverineav has quit IRC13:15
*** aysyd has joined #openstack-infra13:16
*** hdd has quit IRC13:19
openstackgerritKanagaraj Manickam proposed openstack-infra/project-config: Adds new project Python-namosclient  https://review.openstack.org/27939913:20
*** zz_ja has quit IRC13:20
*** zz_ja has joined #openstack-infra13:22
*** alivigni has joined #openstack-infra13:23
*** alex_xu has quit IRC13:28
*** scheuran has joined #openstack-infra13:29
*** alex_xu has joined #openstack-infra13:30
*** sambetts has joined #openstack-infra13:31
*** lucas-hungry is now known as lucasagomes13:31
openstackgerritSean Dague proposed openstack-infra/project-config: create openstack/service-types-authority  https://review.openstack.org/27861213:32
HeOSHello! I want to ask about your gerrit's related manifests: http://tinyurl.com/zsq5ld6. Have you used 'exec' to start the service because puppet is working on your servers by cron?13:32
sambettsHi, what shell is used in the gate for running all the scripts?13:32
*** achanda has joined #openstack-infra13:32
tznhi folk, I know it's kind of hot time, but can we get +workflow on https://review.openstack.org/#/c/267453/13:33
*** amotoki has joined #openstack-infra13:33
tznit's sitting there for 2 weeks since last change13:33
*** dkranz has joined #openstack-infra13:33
*** nwkarsten has joined #openstack-infra13:33
*** tzn has quit IRC13:33
*** _tzn has joined #openstack-infra13:35
*** achanda has quit IRC13:38
*** kozhukalov has joined #openstack-infra13:39
openstackgerritVolodymyr Samotiy proposed openstack-infra/project-config: Add openstack/broadview-lib project  https://review.openstack.org/27587313:39
openstackgerritAndrey Pavlov proposed openstack-infra/project-config: Add job with functional tests for refstack  https://review.openstack.org/27953613:41
openstackgerritBulat Gaifullin proposed openstack-infra/project-config: Create openstack/packetary-specs repository  https://review.openstack.org/27952613:43
*** amotoki has quit IRC13:43
*** placko_ has joined #openstack-infra13:43
*** xyang1 has joined #openstack-infra13:46
AJaegersambetts: which scripts do you mean? For most we set bash explicitly13:47
sambettsdevstack mainly13:47
openstackgerritPaul Belanger proposed openstack-infra/project-config: Move bindep-fallback to non-voting  https://review.openstack.org/27916113:48
AJaegersambetts: Which file exactly? I expect that all scripts use an explicity shebang in the first line that shows which shell it is13:48
sambettsAJaeger, http://logs.openstack.org/08/277408/8/check/gate-tempest-dsvm-ironic-pxe_ipa-tinyipa-nv/5ebc499/logs/devstacklog.txt.gz13:48
sambettsthis is the log, the build-tinyipa.sh script has a shebang to force it to sh, locally even when forcing it to run under sh I can recreate the error13:49
sambettscan´t recreate*13:50
VivekAJaeger: Hi, I just went through the OpenStack Infra Documentation and got familiar with the flow.13:50
*** annegentle_ has joined #openstack-infra13:50
AJaegersambetts: do you mean http://logs.openstack.org/08/277408/8/check/gate-tempest-dsvm-ironic-pxe_ipa-tinyipa-nv/5ebc499/logs/devstacklog.txt.gz#_2016-02-12_12_30_00_289 ?13:50
VivekWanted to contribute to the Puppet,Ansible part.13:50
*** amotoki has joined #openstack-infra13:51
sambettsAJaeger, yes13:51
AJaegersambetts: Where does build-tinyipa.sh life?13:51
*** vgridnev has quit IRC13:51
AJaegerVivek: welcome. Best talk with pabelanger13:51
VivekAJaeger: sure13:52
sambettsAJaeger, http://git.openstack.org/cgit/openstack/ironic-python-agent/tree/imagebuild/tinyipa13:52
*** vgridnev has joined #openstack-infra13:52
*** amotoki has quit IRC13:52
sambettsAJaeger, http://git.openstack.org/cgit/openstack/ironic-python-agent/tree/imagebuild/tinyipa13:53
sambettssorry didn´t mean to send that twice...13:53
*** edmondsw has quit IRC13:53
AJaegersambetts: we use for most scripts bash - and I guess that /bin/sh is an alias on the Ubuntu system to it (cannot verify).13:54
*** edmondsw has joined #openstack-infra13:54
AJaegerAnd we use "#!/bin/bash -xe" - that way you see all the expansion and execution.13:54
openstackgerritMerged openstack-infra/shade: create_service() should normalize return value  https://review.openstack.org/27409113:54
openstackgerritMerged openstack-infra/shade: Add docs tox target  https://review.openstack.org/27687013:54
AJaegerI suggest you change your script for that and see what exactly happens13:54
*** aspiers has quit IRC13:54
openstackgerritMerged openstack-infra/shade: Clarify Munch object usage in documentation  https://review.openstack.org/27687813:55
pabelangerAJaeger: Vivek: sure, I mostly just review the open puppet issues on review.o.o for infra and start there.13:55
openstackgerritMerged openstack-infra/shade: Add enabled flag to keystone service data  https://review.openstack.org/27411613:55
AJaegerah, it does "set -ex" - is that available as part of /bin/sh ? S13:55
pabelangerVivek: you can see a list of the current puppet modules -infra uses: http://git.openstack.org/cgit/openstack-infra/13:55
sambettsAJaeger, the whole script works perfectly fine locally under both sh and bash13:56
pabelangerVivek: that will provide a good start to search in http://review.openstack.org for patchsets to review and test13:56
VivekOk.13:56
VivekThanks for the pointers13:57
AJaegersambetts: sorry, can't help debugging right now, please ask others13:57
sambettsAJaeger, thanks for your help :)13:57
*** amotoki has joined #openstack-infra13:58
*** nwkarsten has quit IRC13:58
*** dizquierdo has quit IRC13:58
*** nwkarsten has joined #openstack-infra13:59
*** ddieterl_ has quit IRC14:00
*** clayton has quit IRC14:00
AJaegerpabelanger: just reviewing your update: Keep in mind that templates are just syntactic sugar, so you cannot make a template non-voting. You always operate on jobs. So, your updated change is fine...14:01
pabelangerAJaeger: Ya, I should have ran tox local.  What I really should have done was add bindep-fallback and bindep-fallback-nv, with the JJB too. But, I rushed and did it at the end of the day.14:02
pabelangerAJaeger: thanks for the tips14:02
*** rossella_s has quit IRC14:03
AJaegeryou're welcome, pabelanger14:03
*** sfinucan has joined #openstack-infra14:03
*** _nadya_ has joined #openstack-infra14:03
*** rossella_s has joined #openstack-infra14:03
*** julim has joined #openstack-infra14:06
*** jaosorior has quit IRC14:06
*** jaosorior has joined #openstack-infra14:06
*** amrith is now known as _amrith_14:07
pabelangeryolanda: did you get curl working on trusty with the SSL cert?14:07
yolandahi pabelanger, yes, it worked finally14:07
yolandait was a problem with my cert14:07
*** KanagarajM has quit IRC14:07
*** kgiusti has joined #openstack-infra14:08
EmilienMtrying do save nodes consumption for puppet CI, can someone look at https://review.openstack.org/#/c/278072/ please ?14:08
pabelangeryolanda: Odd that RHEL worked with it. Guess ubuntu more strict or different defaults?14:09
yolandapabelanger, the problem i found is that i had ca:false set on my openssl.cnf14:09
*** tiswanso has joined #openstack-infra14:10
pabelangeryolanda: good to know14:10
yolandapabelanger, so crinkle was refactoring the puppet-infracloud modules to work with self signed certs, and i've been trying today. So far it starts to work14:10
pabelangeryolanda: nice. looking forward to lending a hand in the coming week14:11
*** ajmiller has joined #openstack-infra14:11
*** jsavak has joined #openstack-infra14:12
*** clayton- has joined #openstack-infra14:12
yolandapabelanger thanks, i hope we had something working from scratch in midcycle14:12
*** moravec has joined #openstack-infra14:12
openstackgerritAndrey Pavlov proposed openstack-infra/project-config: Add job with functional tests for refstack  https://review.openstack.org/27953614:13
*** placko_ has quit IRC14:13
*** salv-orlando has joined #openstack-infra14:13
*** daemontool has quit IRC14:16
*** Daisy has joined #openstack-infra14:16
*** sabeen1 has joined #openstack-infra14:21
*** amitgandhinz has joined #openstack-infra14:21
*** Daisy has quit IRC14:22
*** Daisy has joined #openstack-infra14:23
*** matrohon has quit IRC14:23
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate: multinode: on error, collect worlddump for all nodes  https://review.openstack.org/27956014:23
*** alaski is now known as lascii14:25
*** cgalan_ has joined #openstack-infra14:25
*** sabeen3 has joined #openstack-infra14:25
*** dmsimard has quit IRC14:25
*** mriedem has joined #openstack-infra14:26
*** abregman is now known as abregman|afk14:27
fungiVivek: welcome!14:27
*** Daisy has quit IRC14:27
*** sabeen1 has quit IRC14:27
*** dansmith is now known as superdan14:28
*** matrohon has joined #openstack-infra14:28
*** hashar has joined #openstack-infra14:31
AJaegerfungi, could you give me a quick +2 on https://review.openstack.org/#/c/278051, please ? It removes non-existing entries from a job-group. Then I can rebase the rest of the stack more easily...14:31
AJaegerthanks!14:32
*** abregman|afk is now known as abregman14:33
*** abregman is now known as abregman|afk14:33
VivekThanks fungi14:34
VivekI have free time to contribute and would like to make best use of time helping the community.14:34
_tznguys, any chance for merging https://review.openstack.org/#/c/267453/ It's just new repo for already existing project14:35
*** hdd has joined #openstack-infra14:35
*** daemontool has joined #openstack-infra14:40
openstackgerritMerged openstack-infra/project-config: Remove non-existing jobs from devstack-jobs  https://review.openstack.org/27805114:41
*** cfarquhar has quit IRC14:43
*** aspiers has joined #openstack-infra14:43
*** ajmiller has quit IRC14:44
*** cfarquhar has joined #openstack-infra14:45
*** cfarquhar has quit IRC14:45
*** cfarquhar has joined #openstack-infra14:45
*** amotoki has quit IRC14:45
*** _tzn has quit IRC14:46
openstackgerritZara proposed openstack-infra/storyboard-webclient: Move search icon  https://review.openstack.org/27920414:47
*** amotoki has joined #openstack-infra14:48
*** pradk has joined #openstack-infra14:52
*** vikram has joined #openstack-infra14:53
vikramHi All14:53
vikramI can find jerkin jobs are not running for my PS "https://review.openstack.org/#/c/205756/"14:53
vikramcan't find what's wrong ;(14:53
vikramcan someone please help14:53
*** amotoki has quit IRC14:53
boris-42AJaeger: hi14:54
*** clayton- is now known as clayton14:55
boris-42AJaeger: I see something strange in rally gates ( 278744,3 24 hr 2 min ) one of jobs stays in queue forever14:55
AJaegerboris-42: you need infra-root help for that, I'm the wrong person to ping for that.14:56
boris-42AJaeger: ok14:56
ansiwenmordred, lifeless: I'm not 100% sure, but I think it's wrong, that pbr just copies all git files into the build directory. I guess it should exclude .po files and copy .mo files instead. because setup.py compiles the .mo files in the src tree, not in the build directory.14:58
*** piet has joined #openstack-infra14:59
*** piet has quit IRC14:59
*** doug-fish has joined #openstack-infra15:01
*** rbrndt has joined #openstack-infra15:01
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for o.m/n-net/rabbit socket closed bug 1545002  https://review.openstack.org/27957915:02
openstackbug 1545002 in oslo.messaging "Grenade Failure - MessagingTimeout on floating IP remove" [High,Confirmed] https://launchpad.net/bugs/154500215:02
*** changbl has quit IRC15:03
*** julim has quit IRC15:03
openstackgerritJames E. Blair proposed openstack/gertty: Do not use GitPython 1.0.2  https://review.openstack.org/27958215:05
*** julim has joined #openstack-infra15:05
*** kozhukal` has joined #openstack-infra15:06
fungivikram: given the litany of depends-on headers in 205756 some of which are going to very long patch chains, i'm going to guess there's a circular dependency somewhere in the mix. i'll check zuul's logs to figure it out15:07
openstackgerritMerged openstack/gertty: Fix unicode regression regressions  https://review.openstack.org/27847215:07
*** kozhukalov has quit IRC15:07
openstackgerritMerged openstack/gertty: Improve handling of abandoned related changes  https://review.openstack.org/27699715:08
openstackgerritMerged openstack/gertty: Add support for conflicts-with  https://review.openstack.org/27699815:08
openstackgerritMerged openstack/gertty: Add debug-sync option  https://review.openstack.org/27747315:08
openstackgerritMerged openstack/gertty: Add support for last_seen  https://review.openstack.org/27747415:08
openstackgerritMerged openstack/gertty: Add support for customized dashboard sorting  https://review.openstack.org/27747515:08
openstackgerritMerged openstack/gertty: Make sorting commands use two two keys  https://review.openstack.org/27747615:08
openstackgerritMerged openstack/gertty: Fix return values in handleCommands in project_list  https://review.openstack.org/27747715:08
openstackgerritMerged openstack/gertty: Add another utf8 test change comment  https://review.openstack.org/27848015:08
openstackgerritMerged openstack/gertty: Do not use GitPython 1.0.2  https://review.openstack.org/27958215:09
boris-42fungi: hi15:09
*** amotoki has joined #openstack-infra15:09
vikramfungi:  thanks15:09
*** yamamot__ has joined #openstack-infra15:09
fungiboris-42: yep, saw your question in scrollback too. will take a look once i get vikram's question investigated15:09
boris-42fungi: thanks15:09
fungiboris-42: though i'm betting that's the issue clarkb and dougwig were looking into with rally jobs for neutron exhausting worker memory, crashing the machine and causing zuul to think the job was a victim of random cloud instability and restarting it (indefinitely)15:10
fungiwhich is where the nano flavor recommendation came from15:11
boris-42fungi: but this is not related to that topic15:11
boris-42fungi: I agree that we should reduce the flavors in gates (and we are going to fix that soon)15:11
fungiboris-42: neat, well, i'll have a look shortly15:11
AJaegerfungi: looks like "gate-rally-install-bare-precise: queue"15:11
*** jpr has quit IRC15:12
boris-42AJaeger: yep, and it seems like there is no available host for it15:12
*** zeih_ has quit IRC15:12
boris-42fungi: ^15:12
AJaegerfungi: there'S also a system-config change wtih "gate-infra-puppet-apply-bare-precise: queue"15:12
AJaegerThat'S tin the gate queue, waiting since 14 hours15:12
*** gordc has joined #openstack-infra15:13
odyssey4meodd, it seems that one of the nodpool VM's is hanging: https://jenkins04.openstack.org/job/gate-openstack-ansible-openstack_hosts-dsvm-ansible-functional-trusty/5/15:13
jrollrandom question: what is /bin/sh in gate runs? a script run with ./build-tinyipa.sh fails with "Bad substitution" on a line with only echo "$0"15:13
*** erlon has joined #openstack-infra15:13
*** jaosorior has quit IRC15:13
*** jaosorior has joined #openstack-infra15:14
*** abregman|afk is now known as abregman15:15
*** tzn has joined #openstack-infra15:15
openstackgerritMerged openstack-infra/elastic-recheck: Add query for o.m/n-net/rabbit socket closed bug 1545002  https://review.openstack.org/27957915:16
openstackbug 1545002 in oslo.messaging "Grenade Failure - MessagingTimeout on floating IP remove" [High,Confirmed] https://launchpad.net/bugs/154500215:16
*** samuelBartel has joined #openstack-infra15:16
bknudson_I still have figure out my stable/kilo patches are failing jenkins ... other changes are working.15:16
bknudson_It looks like the failure is E: You must put some 'source' URIs in your sources.list -- http://logs.openstack.org/72/278772/1/check/gate-requirements-integration-dsvm/5043a8e/console.html#_2016-02-12_00_43_43_64515:17
*** ikhudoshyn has joined #openstack-infra15:17
fungiboris-42: AJaeger: i agree, at first glance it looks like we probably aren't building enough bare-precise workers15:17
bknudson_I don't find "+ sudo apt-get build-dep -y --force-yes python-numpy"  in the successful run -- http://logs.openstack.org/40/279340/1/check/gate-requirements-integration-dsvm/a752b16/console.html#_2016-02-12_01_48_04_79115:17
*** nelsnelson has joined #openstack-infra15:18
fungibknudson_: whatever is calling apt-get build-dep needs to be switched to the equivalent apt-get install of a list of packages15:19
fungiour new apt mirrors don't include source packages, so apt-get build-dep won't work15:19
fungiit was not obvious based on cursory testing that anyone was relying on source packages (though arguably build-dep just uses the sources index and not any of the source packages themselves)15:20
*** zz_dimtruck is now known as dimtruck15:20
bknudson_it's really weird that sometimes the script is doing apt-get build-dep and sometimes it doesn't... must be a conditional15:21
*** nelsnels_ has quit IRC15:21
*** doug-fish has quit IRC15:21
jrollfungi: you know offhand what /bin/sh is in the gate images?15:22
*** cgalan_ has quit IRC15:22
*** sdake_ has joined #openstack-infra15:22
*** doug-fish has joined #openstack-infra15:22
fungivikram: can you explain what made you think jobs aren't running for 205756? at http://status.openstack.org/zuul/ i see it in the check pipeline. it's been there for 26 minutes, most of its jobs are completed, the remaining few are slated to complete in another 41 minutes15:22
*** jcoufal has joined #openstack-infra15:23
*** cgalan_ has joined #openstack-infra15:24
*** sdake has quit IRC15:24
*** matrohon has quit IRC15:24
vikramfungi: I did a recheck .. I think it might have triggered15:25
fungiboris-42: AJaeger: we're definitely building bare-precise nodes to run those jobs (9 are in a building state right now) but it's possible they're meeting with an untimely end before they become ready. i'll track one in the logs to find out15:25
boris-42fungi: thank you15:26
vikramfungi: After rebase, I coundn't find the jerkin result for almost 7hrs15:27
*** yamahata has joined #openstack-infra15:27
vikramfungi: Same i noticed couple of days back15:27
*** aspiers has quit IRC15:27
*** ajmiller has joined #openstack-infra15:28
fungivikram: that's more useful information. we have a known regression very recently in zuul where merge failures are not being reported into changes. the fix ( https://review.openstack.org/277958 ) merged a few days ago but won't be live until we restart zuul during our gerrit maintenance window today15:28
*** nwkarsten has quit IRC15:29
openstackgerritMatt Riedemann proposed openstack/requirements: Require kombu>=3.0.25  https://review.openstack.org/27960415:29
mriedemsileht: ^15:29
openstackgerritBrant Knudson proposed openstack/requirements: Add checks for setup.cfg existence in project dictionary  https://review.openstack.org/27877215:29
fungivikram: so mystery solved on that15:29
*** mtanino has joined #openstack-infra15:29
vikramfungi: Got it.. Thanks for your help15:29
*** ccrouch has joined #openstack-infra15:30
bknudson_I'm going to try rebasing this stable/kilo change that's failing, looks like it's maybe using an old version of the tools/integration.sh script15:30
fungivikram: you're welcome!15:30
fungibknudson_: oh! if you're talking about requirements integration, yes that's where we noticed it. i believe it got fixed in master but don't know whether it's been backported to any stable branches of openstack/requirements15:30
fungibknudson_: also, i doubt a rebase will help. zuul merges your patch with the branch tip prior to testing. if the results on it were from a day or two ago, just try a recheck (but first check that the fix has been backported and merged in that branch)15:31
bknudson_fungi: I noticed my local clone has "./tools/integration.sh:sudo apt-get build-dep -y --force-yes python-numpy" , but http://git.openstack.org/cgit/openstack/requirements/tree/tools/integration.sh?h=stable%2Fkilo doesn't have that line.15:31
*** jtomasek has quit IRC15:31
*** jtomasek_ has joined #openstack-infra15:32
*** vikram has quit IRC15:32
fungifor your local tree, sure, you may need to rebase15:32
*** changbl has joined #openstack-infra15:32
fungithough our ci will end up testing it with the branch tip instead15:32
bknudson_ok, so I don't know why the build system didn't have it. I did recheck yesterday when I saw another change had passed.15:33
fungibknudson_: the backport may not have merged at that point15:33
fungiwould need to look at the timeline in git15:33
*** zz_ja has quit IRC15:35
fungiboris-42: AJaeger: mordred: mystery solved. we have a bug with the configure_mirror.sh nodepool ready script on ubuntu 12.04 lts (precise). we should only try to disable multi-arch on 14.04 lts. precise throws "dpkg: error: unknown option --remove-architecture"15:35
*** dprince has quit IRC15:36
*** akshai_ has quit IRC15:36
boris-42fungi: thank you so much15:36
*** achanda has joined #openstack-infra15:36
fungii'll delete the latest nodepool bare-precise images to try and roll this back while i work on a patch15:36
mordredfungi: oh! I'm sorry - I did not realize somehow in my brainhole that we still had precise nodes15:37
mordredfungi: want me to make a patch?15:37
fungimordred: half our infra is still running on precise :/15:37
mordredyah - I just forgot we still had precise nodes in nodepool15:37
*** changbl has quit IRC15:37
fungimordred: please! i'll get these images deleted and them move on to researching jroll's question15:37
fungiand help you fast-track the precise fix15:38
*** rhallisey has quit IRC15:38
jrollfungi: if you point me to where I can research I can do it myself :)15:38
*** rhallisey has joined #openstack-infra15:38
jrolljust added an `echo $SHELL` to my debugging patch15:38
*** changbl has joined #openstack-infra15:38
jrollwhich may just give me my answer15:38
*** abregman is now known as abregman|afk15:39
*** achanda has quit IRC15:41
fungijroll: well, also your question is vague. any particular image? we have rather a few different platforms in use now, but /bin/sh is defaulted to whatever those distros choose upstream15:42
openstackgerritMonty Taylor proposed openstack-infra/project-config: Do not disable multi-arch on precise  https://review.openstack.org/27961015:42
mordredfungi: ^^15:42
jrollfungi: looking at this failure specifically, I assume ubuntu as it's a dsvm job http://logs.openstack.org/69/279569/1/check/gate-tempest-dsvm-ironic-pxe_ipa-tinyipa-nv/b00facc/logs/devstacklog.txt.gz#_2016-02-12_15_09_04_33015:42
mordredjroll: it almost certainly is dash15:43
*** scheuran has quit IRC15:43
mordredjroll: ubuntu switch to dash for /bin/sh a few years ago15:43
jrollfungi: the failing line is "echo $0", in a script run as "./build-tinyipa.sh", from make15:43
*** dmsimard has joined #openstack-infra15:43
jrollmordred: I agree, however this script runs fine locally in devstack as opposed to failing oddly in CI :)15:43
mordredjroll: if you have a script that wants bash-isms, it's important to explicitly set /bin/bash in the shebang line15:43
mordredneat!15:43
*** _amrith_ is now known as amrith15:44
*** nwkarsten has joined #openstack-infra15:45
mordredjroll: hrm.15:45
jrollmordred: running again with an echo $SHELL to (try to?) be sure15:46
jrollbut it's a doozy, maybe best to just hold back a node15:46
jeblairthat works with dash15:46
mordredjroll: only think I can think of is maybe make is doing someting weird since this is launching the shell script from a makefile15:47
fungiboris-42: AJaeger: mordred: the current bare-precise images are all 2-3 days old now so should hopefully build nodes successfully15:47
mordredbut I cannot imagine in what way that would break anything15:47
jrollmordred: yeah, though make runs fine locally as well15:47
mordredyah15:47
*** burgerk has joined #openstack-infra15:48
jrollmust be something up the stack, but I've no clue what it could be15:48
jrollactually, mind holding this node for me?15:48
*** sigmavirus24_awa is now known as sigmavirus2415:48
mordredjroll: which node?15:48
jrollmordred: dang, still queued15:48
fungiboris-42: AJaeger: mordred: i'll check back in on these in another hour or so because we'll likely end up with more broken images to replace them soon (nodepool was already busy building new ones carrying over the bug from yesterday's images)15:49
*** KanagarajM has joined #openstack-infra15:49
jrollmordred: I'll poke you when it's running I guess?15:49
mordredjroll: cool15:49
jeblairjroll: if that is set -x, why aren't we seeing any output from it?15:49
jrollthanks15:49
jeblairjroll: could it actually be the invocation of the script that's at issue?15:49
jrolljeblair: because it fails while parsing the first line, best I can tell15:49
jeblairjroll: gotcha15:50
*** nwkarsten has quit IRC15:50
jrolljeblair: associated code is https://review.openstack.org/#/c/279569/1/imagebuild/tinyipa/build-tinyipa.sh15:50
jrollwas also failing before that change15:50
jrollI broke it out to see where15:50
*** abregman|afk is now known as abregman15:51
*** zz_ja has joined #openstack-infra15:51
*** jcoufal has quit IRC15:52
mordredjroll: yah - I cannot make any of the lines at the top of the file break in any meaningful way15:53
mordredusing any combinatoin of dash, bash or bash --posix15:53
*** akshai has joined #openstack-infra15:53
jrollyep, agree15:53
* mordred will be fascinated by the results here15:53
SwansonHaving trouble with cloud on cloud action for my CI.  I can create a devstack under my devstack.  Networking looks right.  I can create and attach volumes to the jr devstack but cannot attach to the vms created on it.  Thots where to look?15:53
jeblairjroll, mordred: nor can i do that when invoking the script from make :)15:54
jrollindeed15:54
jrollmordred: 2016-02-12 15:54:26.699 | Building remotely on devstack-trusty-ovh-bhs1-7935570 (devstack-trusty) in workspace /home/jenkins/workspace/gate-tempest-dsvm-ironic-pxe_ipa-tinyipa-nv15:54
jrollis that what you need?15:54
jeblairjroll: yes15:55
jeblairmordred, jroll: held15:55
fungijroll: keep in mind that $SHELL is going to be the calling environment's user's default shell typically, not the shell running the script, afaik15:55
jrollthanks15:55
jrollfungi: yeah, was wondering about that15:55
mordredI think jroll is going to win today's prize for weirdest issue15:56
mordredcongratulations jroll15:56
fungils -l /bin/sh might be more interesting15:56
jeblairjroll: can you link me to a copy of your ssh pubkey?15:56
jroll\o/15:56
jrolljeblair: http://paste.openstack.org/show/tTM8BOX1Kbqg2dlSlPPS/15:56
fungijroll: for example, if i create a /bin/sh script that echoes $SHELL, it says /bin/bash (because that's my login shell) even though ls -l /bin/sh reports /bin/sh -> dash15:57
mordredyah - so $0 is also likely more userful15:57
jeblairjroll: you should be able to log in as jenkins@158.69.87.21715:57
jroll$0 is what breaks :P15:57
jrolljeblair: thank you sir15:57
jrollI am in15:57
*** tzn has quit IRC15:58
*** sdake_ has quit IRC15:58
jeblairlrwxrwxrwx 1 root root 4 Feb 19  2014 /bin/sh -> dash15:58
jrollyep15:59
jroll./build-tinyipa.sh works, make works15:59
* jroll x.x15:59
jeblairjroll: try rewriting it in go?  :)15:59
jrollon it!16:00
*** amrith is now known as _amrith_16:00
*** salv-orlando has quit IRC16:01
odyssey4meis there some sort of default timeout for a jenkins job? this one hung before it even kicked off the actual check: https://jenkins04.openstack.org/job/gate-openstack-ansible-openstack_hosts-dsvm-ansible-functional-trusty/5/16:02
odyssey4meit's been at the same point for over 90 mins now16:02
jeblairjroll: i think there might be a perms issue...16:02
jeblairjroll: devstack runs as the stack user16:02
mordredodyssey4me: clarkb has been investigating some weird hangs recently16:03
jrolljeblair: -rwxrwxr-x 1 stack stack 3301 Feb 12 15:58 build-tinyipa.sh16:03
mordredodyssey4me: I dont know if this is related16:03
jrolljeblair: things run fine as stack user as well16:03
*** jpr has joined #openstack-infra16:04
*** tzn has joined #openstack-infra16:04
ansiwenmordred: saw my message above?16:04
*** zz_ja has quit IRC16:07
mordredansiwen: yes - haven't gotten to pbr thinking in my day yet16:07
mordredinfra-root: bluebox had leaked a ton of floatingips again - I am deleting them16:07
*** zz_ja has joined #openstack-infra16:08
*** KanagarajM has quit IRC16:08
*** tzn has quit IRC16:10
*** pblaho has quit IRC16:11
*** pblaho has joined #openstack-infra16:11
*** zeih_ has joined #openstack-infra16:12
*** ifarkas has quit IRC16:13
openstackgerritMerged openstack-infra/project-config: Do not disable multi-arch on precise  https://review.openstack.org/27961016:13
boris-42fungi: seems like it works, thanks!16:13
openstackgerritCarl Baldwin proposed openstack-infra/irc-meetings: A meeting for discussing and coordinating routed networks  https://review.openstack.org/27811416:14
*** austin81 has joined #openstack-infra16:14
*** ccrouch1 has joined #openstack-infra16:16
*** nwkarsten has joined #openstack-infra16:16
*** nwkarsten has quit IRC16:16
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add DB API method to get failure rate for jobs by run_metadata  https://review.openstack.org/26550716:17
jrollmordred: ummm, this is totally working this time? wat16:17
* jroll uber confused now16:17
*** nwkarsten has joined #openstack-infra16:17
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add DB API method to get failure rate for jobs by run_metadata  https://review.openstack.org/26550716:17
openstackgerritMerged openstack-infra/project-config: Create new project solar-resources  https://review.openstack.org/26745316:17
*** zeih_ has quit IRC16:18
*** zz_ja has quit IRC16:18
*** ccrouch has quit IRC16:18
*** sc68cal has joined #openstack-infra16:19
*** pblaho has quit IRC16:20
*** jordanP has quit IRC16:20
*** pblaho has joined #openstack-infra16:20
*** HeOS has quit IRC16:20
*** bpokorny has joined #openstack-infra16:21
fungimordred: is this something bluebox knows about, i guess?16:21
*** pblaho has quit IRC16:22
*** pblaho has joined #openstack-infra16:23
*** links has joined #openstack-infra16:23
*** pblaho has quit IRC16:24
*** e0ne has quit IRC16:24
*** mriedem is now known as mriedem_meeting16:24
*** pblaho has joined #openstack-infra16:25
jrollmordred: one thing I did notice is the failures I have seem to be on rax16:25
jrollwhere iirc the images are different?16:25
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Add test option to use Ansible source repo  https://review.openstack.org/27964616:26
mordredtimburke, clarkb: https://review.openstack.org/#/c/278797/ updated - thanks16:26
mordredjroll: ah - good point16:27
*** nwkarsten has quit IRC16:27
mordredjroll: we are not fully on dib-created uploaded images on rax yet16:27
fungiinfra-root: (and everyone else interested) ovh did respond to us about the random git connectivity issues from yesterday. they apparently were struggling with a (unspecified but massive) network outage16:27
*** cgalan_ has quit IRC16:27
jrollmordred: so those use the rackspace base images?16:27
mordredjroll: yes16:27
*** nwkarsten has joined #openstack-infra16:27
mordredjroll: I believe we'll be finally off of those soon16:27
jrollmordred: fwiw my "local" tests have been on rackspace so idk16:27
mordredhrm16:27
*** jpr has quit IRC16:28
jrollbut again not behind devstack(-gate)16:28
*** cgalan_ has joined #openstack-infra16:28
mordredjroll: I think it would be fantastic if we could reproduce this16:28
openstackgerritMerged openstack-infra/storyboard: Fix Bugs in Timeline Events  https://review.openstack.org/27738616:29
jrollmordred: I agree16:29
jeblairjroll: should i release the node now?16:29
jrolljeblair: yeah, go ahead16:29
jeblairwe can try again for a rax node if you want16:29
*** zz_ja has joined #openstack-infra16:30
openstackgerritMerged openstack/diskimage-builder: Remove eclean-dist as it's not available by default  https://review.openstack.org/27658116:31
*** armax has joined #openstack-infra16:32
*** nwkarsten has quit IRC16:32
*** tzn has joined #openstack-infra16:32
jrolljeblair: yeah, I'll ping, there's an ironic guy with software to set up a gate-like environment that I could use instead16:32
*** ikhudoshyn has left #openstack-infra16:34
*** pblaho has quit IRC16:36
*** samuelBartel has quit IRC16:37
*** zz_ja has quit IRC16:38
openstackgerritMerged openstack-infra/storyboard-webclient: Rename board and worklist controllers  https://review.openstack.org/27221116:38
*** cgalan_ has quit IRC16:38
*** _nadya_ has quit IRC16:39
*** woodster_ has joined #openstack-infra16:39
*** smarcet has quit IRC16:39
*** abregman is now known as abregman|afk16:40
openstackgerritAntoine Musso proposed openstack-infra/zuul: Update merge status after merge:merge is submitted  https://review.openstack.org/12892116:40
*** zz_ja has joined #openstack-infra16:40
*** esikachev has quit IRC16:40
openstackgerritBrant Knudson proposed openstack/requirements: Add location to requirement.Requirement  https://review.openstack.org/27877316:41
openstackgerritBrant Knudson proposed openstack/requirements: Fix editing constraints files with urls in them.  https://review.openstack.org/27877416:41
openstackgerritBrant Knudson proposed openstack/requirements: Deal with underscores in names in edit-constraints  https://review.openstack.org/27877516:41
pleia2good morning16:41
openstackgerritBrant Knudson proposed openstack/requirements: Expose extras information in requirements  https://review.openstack.org/27877616:41
*** dizquierdo has joined #openstack-infra16:42
*** esikachev has joined #openstack-infra16:44
funginibalizer: crinkle: heads up, a lot of the infra-cloud controllers have been cronspamming about /var/log/exim4/paniclog content this week. sysadmin addresses for nibalizer, jeblair, clarkb and mordred are also bouncing these because of failing sender address verification (due i think to the lack of fqdn) so the rest of us are getting those ndr bounces in addition to the cronspam16:44
*** mikelk has quit IRC16:46
rcarrillocruzeugh16:47
mordredfungi: awesome. so - we need to update the exim config on those boxes and also clear out panic log?16:47
fungiyeah, nibalizer was going to work on fixing /etc/mailname on them to use an fqdn, i think16:48
*** aspiers has joined #openstack-infra16:48
openstackgerritAustin Clark proposed openstack-infra/project-config: Create stackviz element and script in nodepool (WIP)  https://review.openstack.org/27931716:48
*** jaosorior has quit IRC16:48
fungithe actual paniclog content seems to be primarily lines of "User 0 set for local_delivery transport is on the never_users list"16:49
*** claudiub has joined #openstack-infra16:49
fungiwhich implies some other misconfiguration16:49
phschwartzhas anyone actually tested https://review.openstack.org/#/c/277099/ ?16:50
phschwartzI just pulled the latest module down and the syntax is reported invalid for escaping from puppet16:50
phschwartzWarning: Unrecognised escape sequence '\(' in file /etc/puppet/modules/nodepool/manifests/init.pp at line 31416:50
fungiphschwartz: what version of puppet are you using? our jobs should have parsed that and failed if it were invalid16:51
fungithough i guess if it's just a warning it's possible that doesn't break anything16:51
phschwartzfungi: 3.8.6 that was installed with install_puppet.sh16:52
*** esikachev has quit IRC16:52
*** arxcruz has quit IRC16:52
fungiphschwartz: yeah, our puppet apply jobs logged that warning but did not exit nonzero over them16:53
phschwartzfungi: what version are you guys running for puppet?16:53
fungii expect it's benign, because puppet shouldn't be trying to parse those as an escape sequence anyway16:53
phschwartzfungi: I can always rerun install_puppet and see if it pulls a newer version16:53
*** sridhar_ram has joined #openstack-infra16:54
fungigood question on the puppet version. latest 3.x i think? i don't immediately see where it's logged in the job16:54
*** jpr has joined #openstack-infra16:54
fungiphschwartz: as for the nodepool crontab, it looks like it updated on our server with the content we wanted from that change16:56
fungithe \ is preserved in \( and \)16:56
jeblairfungi: the exim panic is because it's trying to deliver that bounce mail to root, and it's been configured not to deliver mail to root16:56
jeblairfungi: fixing the sender should fix that too16:56
fungijeblair: cool, i wondered whether that was likely a circular error16:56
jeblairyep16:56
*** vgridnev has quit IRC16:57
*** dprince has joined #openstack-infra16:57
*** [1]Thelo has joined #openstack-infra16:57
*** shashank_hegde has joined #openstack-infra16:58
*** yamamot__ has quit IRC16:59
*** yamamoto_ has joined #openstack-infra17:00
*** Thelo has quit IRC17:00
*** [1]Thelo is now known as Thelo17:00
*** jlanoux has quit IRC17:01
mmedvedephschwartz: my bad on that. I actually did test original string, that is why there is double escape before ;, but did not test puppet apply on the addition. I'll push a quick fix, less warnings is good17:01
BobBallLooks like node creation fails ATM because solar-resources is an empty git repository, so cache_git_repos fails?  Is that right, any workaround?17:01
yolandabtw, fungi, crinkle, certs in east worked fine now, and compute + controller nodes can run puppet successfully. I'm pushing an image to glance so i can start testing real stuff17:03
*** jaosorior has joined #openstack-infra17:04
*** arxcruz has joined #openstack-infra17:04
openstackgerritJohn-Paul Robinson proposed openstack-infra/nodepool: Allow jenkins user and apikey to be optional for unauthenticated jenkins as documented  https://review.openstack.org/27967117:05
openstackgerritMikhail S Medvedev proposed openstack-infra/puppet-nodepool: Fix unrecognized escape puppet warning  https://review.openstack.org/27967217:05
fungiyolanda: that's great news!17:05
mmedvedefungi: phschwartz: submitted the fix ^^17:06
jeblairyolanda: yay!17:07
yolandafungi, one limit that we have in east now is the lack of ips. We have been given only a /24 range, shared with one controller + 90 computes. So that leaves room for few floating ips, much less than the capacity we can have. I hope that this can be fixed when we move17:08
*** max_lobur has quit IRC17:09
mordredyolanda: s/floating ips/public ips/17:09
mordredyolanda: there should be no floating ips at all in either of these clouds17:09
*** smarcet has joined #openstack-infra17:10
*** yamahata has quit IRC17:10
yolandaah bad words, i meant public ips17:10
mordred:)17:10
*** nwkarsten has joined #openstack-infra17:10
*** kozhukal` has quit IRC17:10
mordredyolanda: I figured - just thought I'd double check17:10
*** slogan621 has quit IRC17:11
*** wolverineav has joined #openstack-infra17:12
*** jpr has quit IRC17:12
*** zeih has joined #openstack-infra17:13
*** tzn has quit IRC17:14
*** gyee has joined #openstack-infra17:14
BobBallAny reason why cache_git_repos forcibly checks that HEAD exists?  If we have a new repo (such as solar-resources here) then it will fail until the first checkin.17:14
*** Jeffrey4l has quit IRC17:15
fungiBobBall: it's so that we avoid making an image with a broken git repo cached locally17:16
fungiBobBall: and also avoid making an image which has failed to cache a repo (skipping caching nova, for example, results in a massive self-inflicted ddos against our git farm)17:16
*** openstackgerrit has quit IRC17:17
*** openstackgerrit has joined #openstack-infra17:17
BobBallSo new repos will prevent image creation because they actually don't have a HEAD yet17:17
fungiBobBall: new repos should always have head17:17
*** shashank_hegde has quit IRC17:18
fungiBobBall: if they don't, something has broken during repo creation/import17:18
jeblairBobBall: there is a race condition which we occasionally hit.  in those rare cases, it's better for us to abort the image build17:18
*** ociuhandu has quit IRC17:19
*** nwkarsten has quit IRC17:19
jeblair(where gerrit has created the repo, but jeepyb has not pushed content to it yet; or the same similar situation with replication to the git mirrors)17:19
*** zeih has quit IRC17:19
BobBallAh.17:19
jeblairgenerally that's a window of a few seconds, unless as fungi says, something has gone very wrong17:20
fungithere are also rarely failures occurring when running the manage-projects script to create a new repo, where it ends up aborted midway before an initial commit or repo import has happened17:20
BobBallIn which case something may have gone very wrong with solar-resources because it's been empty for a good 20 minutes+ at least17:20
dimsanteaya : around? easy one hopefully, please take a look - https://review.openstack.org/#/c/279467/17:21
openstackgerritK Jonathan Harker proposed openstack-infra/puppet-elasticsearch: Make network.publish_host configurable  https://review.openstack.org/27969017:22
*** esikachev has joined #openstack-infra17:22
*** asselin_ has quit IRC17:24
openstackgerritgreghaynes proposed openstack/diskimage-builder: Allow setting per-release mirrors for tests  https://review.openstack.org/27969117:24
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove {name} jobs from devstack-jobs  https://review.openstack.org/27807317:25
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove unused templates from devstack-gate.yaml  https://review.openstack.org/27806117:25
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Sort devstack-jobs list  https://review.openstack.org/27808417:25
*** smarcet has quit IRC17:25
*** tzn has joined #openstack-infra17:25
*** tzn has quit IRC17:26
*** sridhar_ram has quit IRC17:27
*** annegentle_ has quit IRC17:27
*** dtantsur is now known as dtantsur|afk17:27
*** tphummel has joined #openstack-infra17:28
clarkbheard back from ovh, apparently they had a network outage which should be better now?17:29
*** sfinucan has quit IRC17:29
clarkbsdague: ^ the oddity you saw was likely that17:29
clarkbthe time frame roughly matches up17:29
AJaegerfungi, thanks for solving the mystery about the precise images17:29
*** yamahata has joined #openstack-infra17:30
openstackgerritMerged openstack-infra/project-config: Drop test libs from dsvm+oslo.* master job  https://review.openstack.org/27946717:31
*** amotoki has quit IRC17:31
fungiBobBall: openstack/solar-resources should be fixed now (i performed a manual run of manage-projects for it, which did not error). i'm still trying to track down why/how that repo import broke17:31
BobBallAwesome - thanks fungi!17:31
dimsthanks anteaya17:32
fungiunfortunately the amount of debug logging from manage-projects via puppet into syslog is not great17:32
fungioh, wait, we redirect it to a separate file now. yay!17:32
*** esikachev has quit IRC17:32
*** dprince has quit IRC17:33
*** fedexo has joined #openstack-infra17:34
Clintre: http://logs.openstack.org/periodic/propose-devstack-plugins-list/249d045/console.html , is force-setting ZUUL_REF the right fix or is there a better way?17:34
*** asselin has joined #openstack-infra17:34
*** esikachev has joined #openstack-infra17:34
*** dprince has joined #openstack-infra17:34
fungiand looks like for some reason it had not run yet, but the empty repos on the git farm had already been created. i wonder if we have a problem with the order of operations for our git and gerrit ansible roles17:34
clarkbI have confirmed that that logstash query doesn't return any recent results17:34
clarkbso agree with OVH that this particular issue has gone away17:34
*** jsavak has quit IRC17:34
*** Guest68220 has quit IRC17:35
*** jsavak has joined #openstack-infra17:35
*** kzaitsev_mb has quit IRC17:35
*** ociuhandu has joined #openstack-infra17:35
fungiClint: what's the trigger for that job? is it periodic? or release-driven? or something else?17:36
Clintfungi: periodic, yes17:36
jeblairClint: no, that means that job is not designed to be run in the periodic pipeline.17:36
Clintah, i think i understand17:36
clarkbfungi: was there a verdict on the bare-precise situation?17:38
* clarkb is slowly catching up on scrollback and email17:38
*** wolverineav has quit IRC17:38
*** achanda has joined #openstack-infra17:38
jeblairclarkb: yes: https://review.openstack.org/27961017:39
*** achanda has quit IRC17:39
*** achanda has joined #openstack-infra17:40
clarkbdo we need to rebuild bare-precise? I can do that if no one else has started it17:40
*** flepied has quit IRC17:40
jeblairi don't have an answer to that one; fungi, mordred: ?17:40
*** sambetts is now known as sambetts|afk17:41
ajmillerclarkb: regarding ovh, anecdotally, I haven't seen the kinds of failures this morning that I saw last night.17:41
*** achanda has quit IRC17:41
clarkbajmiller: ya I checked iwth logstash to be double sure and it looks better17:42
ajmillercool.17:42
*** hashar has quit IRC17:42
*** esikachev has quit IRC17:42
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Import puppet-pacemaker from github/redhat-openstack  https://review.openstack.org/27969817:42
*** achanda has joined #openstack-infra17:42
clarkbjeblair: fungi the bare-precise images are just over 2 hours old which is older than that patch merging, I will kick off some builds17:42
*** achanda has quit IRC17:43
jeblairclarkb: i thought fungi said he was going to delete the most recent ones?17:43
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Import puppet-pacemaker from github/redhat-openstack  https://review.openstack.org/27969817:43
jeblairclarkb: perhaps that didn't work as expected17:43
jeblairor possibly just got lost (busy morning)17:44
*** sdake has joined #openstack-infra17:44
clarkbthere is a day old image and a 2 hour old image in each of the regions, rebuilds are quick ish and have been started17:44
*** jaosorior has quit IRC17:44
clarkbdo we know if the day old image is old enough to not have the broken config mirror script on it? if so I can also delete the 2 hour old images17:44
jeblairclarkb: i don't know17:45
pabelangeryolanda: do you have a moment to help review 279101, 279161 and 27930817:45
AJaegerteam, I have a review question: EmilienM's change https://review.openstack.org/278072 wants to blacklist a few jobs from a template. The change does an unconditional skip-if for the jobs. Is that ok to do - or should the template usage replaced by the jobs?17:46
EmilienMyeah I was wondering too17:46
EmilienMbut I looked at zuul code and I thought it would work17:46
AJaegerpabelanger: are those new projects? yolanda, for those please wait with approval until clarkb has rebuild the nodes...17:46
clarkbthe change that added it merged feb 9 at 7:25 pst17:47
clarkbwhich is >1 day old so I will just wait for new images to be built17:47
*** apoorvad has joined #openstack-infra17:47
*** sdake_ has joined #openstack-infra17:48
pabelangerAJaeger: Ah, they are. Thanks for the heads up17:49
fungiclarkb: jeblair: i did delete the most recent ones, i'm checking now to see if nodepool has uploaded even newer ones17:49
*** burgerk has quit IRC17:49
openstackgerritgreghaynes proposed openstack/diskimage-builder: Allow setting per-release mirrors for tests  https://review.openstack.org/27969117:49
AJaegerproject-config cores, could I get a review for 278641 and 278627, please? Those rename linter to pep8 usage and have already a +2 by yolanda17:49
fungiand indeed it has done one in rax-ord so i'll delete that now17:49
*** sdake has quit IRC17:50
*** sridhar_ram has joined #openstack-infra17:50
fungiunless we think there's a chance that image is not broken17:50
clarkbfungi: it doesn't matter17:50
clarkbfungi: the ready images are all broken17:50
clarkbat least if the timestamps are all correct17:50
fungioh, right, fix hasn't merged yet17:50
fungiwe yeah, it has17:50
clarkbit merged shortly after those builds completed17:50
clarkbI am running new builds17:51
fungiwell, only one build has completed17:51
fungi355281 in rax-iad, which i'll delete now17:51
*** kushal has quit IRC17:51
*** tiswanso has quit IRC17:52
clarkbeverythin should be working again in about 25 minutes17:52
fungithat wasn't a day-old image, it was an hour-old image17:52
fungii deleted all the day-old images a few hours ago17:52
clarkboh17:52
clarkbI read the timestamp wrong17:52
clarkbbut even the 2 day old is too new17:53
jeblairit has a lot of numbers and colons17:53
fungithe 2-day-old images also had that error?17:53
*** bmwiedemann has joined #openstack-infra17:53
clarkbfungi: https://review.openstack.org/#/c/277114/ is the change that broke precise builds and it merged 3 days ago17:53
*** mriedem_meeting is now known as mriedem17:53
clarkb3 days and ~2.5 hours ago17:53
*** achanda has joined #openstack-infra17:54
fungiit looked from the zuul status like stuff had only been hung for ~24 hours, but that may be because day before yesterday's rax-ord build failed17:54
fungiso maybe best if i delete the 2-day old images as well and we can coast on rax-ord while new images are being created17:54
clarkbthat seems reasonable17:56
*** ihrachys has quit IRC17:56
fungiwatching a new node booting in rax-ord to see if it succeeds17:56
*** ihrachys has joined #openstack-infra17:57
*** ihrachys has quit IRC17:57
clarkbnext thing I need to catch up o nis whether or not those neutron rally jobs and magnum jobs and ceph jobs are either A) passing and not crashing or B) crashing with real failures and not hosing their hosts17:58
*** wolverineav has joined #openstack-infra17:58
*** links has quit IRC17:58
fungi7939176 (bare-precise in rax-ord) transitioned to ready and then used17:59
*** wolverineav has quit IRC18:00
*** wolverin_ has joined #openstack-infra18:00
fungithat's from a 3-day-old image18:00
clarkbdougwig: the oldest neutron change I see is 1 hour and 50 minutes old in check. That implie sto me that the rally job is functioning now18:00
clarkbdougwig: do you have any more information we can use to confirm or reject this?18:00
*** wolverin_ has quit IRC18:00
clarkbfungi: it must've just missed the merge of the original breaking change18:01
dougwigclarkb: i opened everything that was in progress last night and watched all the jenkins console, and they all made it to the end. i think you might've nailed it.18:01
fungi7939318 (bare-precise in rax-iad) is booting from a 2-day-old image right now, so if it fails i'll delete the 2-day-old bare-precise images and let nodepool start building replacement images for those18:01
*** wolverineav has joined #openstack-infra18:01
AJaegerpabelanger: here's a grafana change, do you want to review it? https://review.openstack.org/27883218:01
dougwigconsoles, even.18:01
*** annegentle_ has joined #openstack-infra18:01
clarkbdougwig: woot I was really worried I would have to continue to debug that problem18:01
dougwigclarkb: i'll keep an eye on it, but i think you got it.  thank you.  :)18:02
AJaegerEmilienM: no need to add me to reviews, I'll go through them when I have time...18:02
EmilienMAJaeger: ok18:02
*** annegentle_ has quit IRC18:03
fungiclarkb: dougwig: interesting though that the job would have been using so much disk the kernel couldn't allocate blocks to the sparse swapfile, but preallocating the same size swapfile doesn't cause the job to run out of disk?18:03
greghaynescrinkle: yolanda In infra-cloud west, is the neutron network flat with no segmentation id and we gave neutron a phy which was already tagged? (I think thats what I remember, but trying to make sure)18:03
*** annegentle_ has joined #openstack-infra18:03
fungiclarkb: dougwig: or was the effective fix switching to nano flavor for most tests?18:03
yolandacrinkle , do you have details of neutron net-show handy?18:04
dougwigfungi: the nano switch hasn't happened yet, watching the bug activity.18:04
fungialso the boot from 2-day-old image broke, so deleting those now18:04
crinkleyolanda: greghaynes sure18:04
openstackgerritMerged openstack/requirements: Require kombu>=3.0.25  https://review.openstack.org/27960418:04
clarkbfungi: it is also possible that swap via loopback is just broken on that newer kernel18:04
*** dimtruck is now known as zz_dimtruck18:05
clarkbubuntu pushed a new kernel roughly around when we started noticing the failures18:05
fungioh, that makes a bit more sense18:05
fungias a possible cause18:05
crinkleyolanda: greghaynes http://paste.openstack.org/show/486861/18:05
*** jsavak has quit IRC18:06
yolandanet looks similar18:06
greghaynescrinkle: yolanda cool, so ya - we must have given neutron a tagged phy which makes sense18:06
openstackgerritMerged openstack-infra/system-config: Add #openstack-third-party-ci to channel logging  https://review.openstack.org/27857418:06
dougwigfungi, clarkb: could also be a major difference in perf between a fragmented and linear swapfile, if the underlying storage is spinny.18:06
fungialso possible, sure18:06
*** achanda has quit IRC18:06
greghaynesyolanda: so your neutron network is setup correctly, you are just  not giving neutron a correctly setup phy18:06
*** jsavak has joined #openstack-infra18:07
greghaynesyolanda: this is the deal where we made a veth pair off a bridge off our tagged interface18:07
fungiwaiting for the last of the nodes booting fro the 2-day-old images to error out and watching the nodepool log for any new hits on "remove-architecture" hoping they subside in the next few minutes18:07
yolandagreghaynes, i did the same, creating a veth pair from my bridge18:07
greghaynesyolanda: what interface is added to that bridge?18:07
yolandathe difference i see, is that you point your gateway to the baremetal, and run a double dnsmasq server there, and i point to my original gateway18:08
crinklegreghaynes: yolanda this might help too http://paste.openstack.org/show/486862/18:08
yolandagreghaynes, eth0.159818:08
greghaynescrinkle: awesome, yep - veth1 is added to br-vlan2518:09
*** claudiub has quit IRC18:09
AJaegerdougwig: here's a networking change for review if you have time: https://review.openstack.org/#/c/279373/18:09
greghaynesyolanda: is that eth0 with vlan tag 1598?18:09
yolandagreghaynes, br-vlan15988000.1e27b23c052enoeth0.159818:09
clarkbcrinkle: brctl show ? I think is another command that shows oyu how the interfaces connect18:09
greghaynesclarkb: yea, that ip link show has all the bridge info18:10
*** esker has joined #openstack-infra18:10
greghayneswell, where the tubes connect18:10
crinklebrctl show is a little less cluttered http://paste.openstack.org/show/486865/18:10
openstackgerritMerged openstack-infra/system-config: Configure out puppet runs to log to syslog  https://review.openstack.org/27108718:10
*** piet has joined #openstack-infra18:10
*** eil397 has joined #openstack-infra18:10
greghaynesyolanda: can you paste an ip link show18:11
yolandasure18:11
dougwigAJaeger: done18:11
yolandahttp://paste.openstack.org/show/486866/18:11
greghaynesyolanda: ok, so if vlan 1598 is the correct vlan for external access then that looks correct18:12
clarkbcrinkle: ya so you have eth2 - eth2.25 - br-vlan25 - veth1 - veth2 - neutron stuff18:12
greghaynesyolanda: and it looks like neutron made its bridge off the veth pair so that is also correct18:12
yolandayep, we have our baremetal, controller, computes with that, and have external access18:12
AJaegerthanks, dougwig18:12
yolandagreghaynes yes, so i'm lost ... not having connectivity18:12
greghaynesyolanda: Where are you testing from?18:13
*** kzaitsev_mb has joined #openstack-infra18:13
yolandagreghaynes, oh... took long , very long to boot, telnet on that port works now18:13
greghaynesyolanda: it was probably resizing the rootfs18:14
yolandait was showing as active, but i could not reach it18:14
pabelangerAJaeger: reviewed18:14
* yolanda dances18:14
*** terryw is now known as otherwiseguy18:14
yolandaso seems we are good18:14
greghaynesawesome18:14
pabelangerAJaeger: jeblair: maybe we can land the nodepool templating dashboard? https://review.openstack.org/#/c/264994/18:14
greghaynescongrats18:14
*** amotoki has joined #openstack-infra18:14
AJaegerpabelanger: one more change for you, please: https://review.openstack.org/#/c/279326/1/grafana/tempest.yaml18:14
clarkbyolanda: greghaynes if it conintued to be a problem checking that eth0 is the right interface to tag that vlan on is where I would've gone next18:14
yolandait took more than 10 minutes to boot since i launched :(18:14
crinkleyolanda: you can tail the console log on the hypervisor to see what it's doing18:15
greghaynesclarkb: yea, these boxes should only have one interface actually hooked up, but it dances around sometimes which is :(18:15
clarkbgreghaynes: you can udev it into staticness18:15
*** madhu_ak has joined #openstack-infra18:15
greghayneswhere is biosdevname when you actually want it18:15
clarkbbut it is a pita18:15
yolandacrinkle, yep... i found that just hanged after configuring the network, for a long long time18:15
yolandagreghaynes, clarkb, that is script i created to bring up the bridge from configdrive https://review.openstack.org/27629218:16
*** salv-orlando has joined #openstack-infra18:16
*** fedexo has quit IRC18:16
greghaynesyolanda: awesome18:16
clarkbwe should probably explicitly write down the "eth2 - eth2.25 - br-vlan25 - veth1 - veth2 - neutron stuff" setup in the infra cloud spec18:17
openstackgerritMerged openstack-infra/project-config: tempest/experimental: run puppet scenarios on centos7  https://review.openstack.org/27808018:17
*** esikachev has joined #openstack-infra18:17
greghaynesclarkb: hrm, I thought that got documented somewhere...18:18
clarkbso that it can be explained in words and not just code18:18
clarkbgreghaynes: maybe it did18:18
yolandaclarkb, if we are ok with the element i created for that, i can document it18:18
yolandagreghaynes, i saw a task in storyboard, but not documentation18:18
greghaynesyolanda: ah18:18
yolandaif that idea of configuration is ok, i can just write18:18
fungiokay, i see 6 bare-precise nodes in use in rax-ord now and changes which have been waiting on workers are starting to move in the zuul status, so i think we're in the clear for now18:18
greghaynesyolanda: yep, it looks correct18:18
clarkbyolanda: my one concern with that is can't glean express that itself?18:19
yolandaclarkb, so glean cannot configure bridges, and it's not on the scope18:19
clarkbI guess we would have to inject config drive info that isn't part of the cloud18:19
greghaynesclarkb: not the bridge and veth pair bits18:19
yolandaclarkb, the config drive info is there, but glean cannot manage it18:19
AJaegerdougwig: is https://review.openstack.org/#/c/279178/ ok? Then I can +A it...18:19
clarkbhrm18:19
*** gordc has quit IRC18:19
yolandaclarkb, i'm configuring the bridge reusing all items that we write in configdrive18:20
clarkby athats why I immedaitely figured glean could do it18:20
*** e0ne has joined #openstack-infra18:20
yolandawe've discussed that with mordred, greghaynes, and the outcome was that glean should not create the bridges, as it's a particular implementation18:20
clarkbok18:21
*** jsavak has quit IRC18:22
greghaynesYep, and the only reason were doing this is because were doing the thing you shouldn't do of having only one interface18:22
annegentle_clarkb: or fungi: could one of you call me with the ftp password for the developer.openstack.org site? My new computer doesn't have it... sorry.18:22
annegentle_or jeblair18:22
*** tiswanso has joined #openstack-infra18:22
*** BobBall is now known as BobBall_AWOL18:22
*** flepied has joined #openstack-infra18:23
*** rbrndt has quit IRC18:23
clarkbyolanda: greghaynes I don't see where that makes the veth pair18:23
yolandaclarkb, that's on puppet18:23
yolandapuppet-infracloud18:23
clarkbgotcha18:23
clarkbthen ya that looks reasonable, let me actually properly review this and I can vote18:23
yolandathanks18:23
*** andrey-mp has joined #openstack-infra18:24
openstackgerritMadhusudhan Kandadai proposed openstack-infra/project-config: Add rally gate job for neutron-vpnaas project  https://review.openstack.org/27933118:24
*** openstack has joined #openstack-infra18:25
*** shashank_hegde has joined #openstack-infra18:25
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: rename dist to prepare for release  https://review.openstack.org/27971518:26
*** harlowja_at_home has joined #openstack-infra18:26
clarkbyolanda: is the ip route necessary? are we doing like ovh here and using a gateway that isn't within our netmask?18:27
yolandaclarkb i thought that bridges needed to have a default route set18:27
mordrednotmyname: release to pypi only gets triggered by semver versions18:27
yolandai can try remove18:27
mordrednotmyname: so 1.1 would not trigger it - you'd need 1.1.018:28
mordrednotmyname: also, it sohuld be signed if it wasnt already18:28
funginotmyname: mordred: also https://git.openstack.org/cgit/openstack/swift-bench/tree/setup.cfg has np [metadata] section18:28
*** ociuhandu has quit IRC18:28
notmynamemordred: thanks. it was signed18:28
fungier, has no18:28
notmynamebut just 1.1 not 1.1.018:28
clarkbyolanda: a bridge is an l2 construct it shouldn't care. We do attach an ip addr to it but if it knows its gateway it should just use it18:28
fungiwhich is where the script actually died18:28
fungihttp://logs.openstack.org/ae/ae2fec19e1ff7fed9f1ff320c67d7f1dd88ee432/release/swift-bench-pypi-both-upload/dc93a2c/console.html18:28
mordrednotmyname: yah - zuul will ignore a tag named 1.118:28
mordredfungi: oh - did it trigger?18:28
fungimordred: it won't ignore a tag named 1.1. i proposed that strawman patch like a year ago but it never reached consensus18:29
greghaynesclarkb: where is the ip route?18:29
notmynamethis is the first time in 2 years we've done a release for this repo18:29
mordredfungi: ah. nevermind18:29
fungimordred: instead we just break if you upload a non-pbr-normalized tag18:29
greghaynesoh, post-up18:29
clarkbgreghaynes: https://review.openstack.org/#/c/276292/14/files/elements/infra-cloud-bridge/static/opt/create_bridge.py line 5118:29
mordrednotmyname: the releasing system assumes pbr-based repos18:29
openstackgerritMerged openstack-infra/project-config: Adding new channel #openstack-networking-cisco to bot lists  https://review.openstack.org/27917818:29
yolandaclarkb, if i remove route, it loses connectivity network18:29
greghaynesyea, it has a gateway too, so that should be adding a default route18:29
fungimordred: which this would have done too, except it never got far enough to break on that because setup.cfg is incomplete for that repo18:29
mordredyah18:30
clarkbyolanda: what is your route table without that?18:30
yolanda10.23.212.0     0.0.0.0         255.255.255.128 U     0      0        0 eth218:30
yolanda10.23.212.0     0.0.0.0         255.255.255.128 U     0      0        0 eth318:30
yolanda15.126.48.0     0.0.0.0         255.255.255.0   U     0      0        0 br-vlan159818:30
*** achanda has joined #openstack-infra18:30
greghaynesyolanda: I wonder if there is already a default route and ubuntu is being smart and not setting it18:30
greghaynesyep18:30
greghaynesthats it18:30
*** tiswanso has quit IRC18:30
crinklein my experience sometimes the private interface comes up first and sets itself as the default route18:30
fungiannegentle_: i'll look up the password. want to /msg me a phone number? or would you rather have a pgp-encrypted e-mail?18:30
notmynamefungi: thanks for the info18:30
greghaynesyolanda: clarkb so on the 10-net you should not be specifying a gateway from dnsmasq18:31
*** tiswanso has joined #openstack-infra18:31
yolandagreghaynes, that's glean configuring that with dhcp automatically18:31
andrey-mphello all! Can anyone help us with with this https://review.openstack.org/#/c/279511/  we are faced with problem when something deploys refstack website after every merge...18:31
greghaynesyolanda: crinkle I have no idea why you have two interfaces with default routes though, only one should be plugged in18:31
mordrednotmyname: this is what's breaking, btw: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/pypi-extract-name.py18:31
greghaynesyolanda: the dhcp server should not be sending a gateway18:31
yolandagreghayes , i tweaked config to don't send a default route18:32
yolandabut can i tweak to don't even send a gateway?18:32
greghaynesyolanda: ok, so those might be left over18:32
notmynamemordred: I wonder if simply adding that section and value would be sufficient18:32
greghaynesyolanda: on the untagged interface I would18:32
openstackgerritMerged openstack-infra/tripleo-ci: Add dstat  https://review.openstack.org/27121818:33
yolandagreghaynes, untagged is set as manual18:33
yolandahttp://paste.openstack.org/show/486870/18:33
funginotmyname: mordred: actually since that repo doesn't use pbr, i think that job may work with a non-pbr-normalized (e.g. non-semver+pep-440) tag, as pbr won't end up renaming the sdist and wheel to normalized version numbers18:33
clarkbyolanda: I think greghaynes means for the 10.23.212.0/25 network18:33
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Rework devstack-jobs templating  https://review.openstack.org/27803418:33
greghaynesyep, that is tagged18:33
*** jpr has joined #openstack-infra18:34
openstackgerritAustin Clark proposed openstack-infra/project-config: Create stackviz element and script in nodepool (WIP)  https://review.openstack.org/27931718:34
yolandagreghaynes, i need to investigate about that option for dnsmasq and will need to update bifrost18:34
yolandai was able to remove default route with dhcp-option=318:35
funginotmyname: mordred: but yeah, we need some static means of extracting information from the sdist without executing arbitrary python (i.e. setup.py) since this worker has sensitive pypi upload credentials on it18:35
greghaynesyolanda: its probably also worth figuring out what west does18:35
yolandabut i don't know about how to don't send a gateway18:35
yolandagreghaynes, that's hardcoded on dib18:35
clarkbyolanda: I think you just don't configure the gateway in dnsmasq18:35
clarkbif the option isn't set dhcp shouldn't infer it18:35
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Rework devstack-jobs templating  https://review.openstack.org/27803418:35
crinklethis is what we have in west https://review.openstack.org/#/c/231757/34/templates/bifrost/interfaces.d/br-vlan.tag.cfg.erb18:36
greghaynesyolanda: ah, right18:36
clarkbok that adds a default route too18:36
greghaynesthat is the tagged if18:36
yolandayes, element on west adds the same18:36
greghaynesI think we hard bake in the untagged config in dib?18:36
greghaynescool18:36
crinkleyep18:37
crinklewithout the post-up rules the routes come up inconsistently18:37
clarkbI guess I can live with that but that basically says we have tools competing with each other18:37
clarkbwhich is a bug imo18:37
greghaynesclarkb: yep, that was a temporary workaround18:37
yolandayes,glean cannot guarantee an order of booting18:37
greghayneswhen we were in "just get cloud clouding" mode18:37
clarkbyolanda: you don't need that if there are no other gateways18:38
mordrednotmyname: it should be sufficient18:38
clarkbbasically if there is only one gateway that gateway should win18:38
yolandayes but there are18:38
clarkbyolanda: for what?18:38
*** krtaylor has quit IRC18:38
yolandaglean does18:38
*** julim has quit IRC18:38
clarkbthe 10 net is routabel?18:38
*** annegentle_ has quit IRC18:38
*** kzaitsev_mb has quit IRC18:38
yolandaif no config is set, glean configures all attached nics with dhcp18:38
mordrednotmyname: although in general it's worth noting that we make assumptions in tooling about pbr-ification of all of the python projects flowing through the system -so I cannot promise that you will not break in the future18:38
greghaynesyolanda: that should be fine, dhcp should not add a default route if no gateway is sent from the dhcp server18:38
yolandagreghaynes, is there a way to just limit glean to configure the network we want?18:39
notmynamemordred: noted :-)18:39
*** annegentle_ has joined #openstack-infra18:39
greghaynesyolanda: we want glean to run dhcp18:39
funginotmyname: mordred: actually, i take that back, we need a static way of looking up what the package name will be from the source, so that we can figure out what the sdist and wheel are actually named, so that we can download them from the tarballs.o.o site18:39
mordredfungi: yes18:39
yolandagreghaynes, so dhcp is not assigning default routes with my option. When it did, i could not bring up my bridge, because it was complaining about a default route already being set18:39
yolandanow i can consistently create a default route18:39
mordrednotmyname, fungi: that is correct - and I believe in this specific instance adding [metadata]\nname=swift-bench to the setup.cfg should fix this symptom and also not hurt anything else18:40
greghaynesyolanda: well, *something* is assigning a default route...18:40
yolandait's not sending the default route, but is sending a ip + gateway18:40
greghaynesyolanda: yes, so dont send a gateway at all on the untagged interface18:40
greghaynesyolanda: gateway implies a default route18:40
*** pvaneck has joined #openstack-infra18:41
yolandagreghaynes, and i wonder how can we do not send a gateway in dnsmasq18:41
*** andrey-mp has quit IRC18:41
*** kzaitsev_mb has joined #openstack-infra18:41
*** andrey-mp has joined #openstack-infra18:41
yolandagreghaynes ok, dnsmasq was sending gateway + also creating a static default route prior to my changes18:41
greghaynesyolanda: just don't specify that config option? I don't know the specifics of that bt18:41
yolandanow it only sends the gateway18:41
*** andrey-mp has quit IRC18:41
clarkbyes typically you just don't configure it18:41
yolandamm, no18:41
*** lucasagomes is now known as lucas-afk18:41
clarkban option that isn't set won't be sent18:41
*** andrey-mp has joined #openstack-infra18:41
fungimordred: notmyname: also i'm still not convinced this will work for swift-bench since the sdist is generated as swift-bench-1.1.tar.gz but the wheel is swift_bench-1.1-py2-none-any.whl18:42
yolandaif you don't send the option, dnsmasq behaves sending the ip of the dnsmasq controller as gateway18:42
fungimordred: notmyname: (- vs _)18:42
fungibut i could be wrong18:42
greghaynesyolanda: wow18:42
greghaynesyolanda: thats really annoying18:42
yolandai discovered that this week18:42
yolandayep18:42
greghaynesyolanda: well, we either need to figure out how to make it not send a gateway, or maybe we need to switch to isc-dhcp-server18:43
fungimordred: notmyname: tl;dr, standardizing python packaging is necessary for uploading, and python packaging is hard to keep consistent, which is why we have pbr18:43
fungier, necessary for automating18:43
*** annegentle_ has quit IRC18:43
*** e0ne has quit IRC18:44
*** smarcet has joined #openstack-infra18:44
fungiprojects can "wing it" and roll their own packaging, but will likely put themselves in a position where they can't take advantage of much of our automation and end up doing it manually instead18:44
*** annegentle_ has joined #openstack-infra18:46
nibalizerfungi: ohai18:46
*** openstackgerrit has quit IRC18:47
fungi'sup nibalizer?18:47
clarkbyolanda: options 3?18:47
*** openstackgerrit has joined #openstack-infra18:47
*** mrmartin has joined #openstack-infra18:47
*** SumitNaiksatam has quit IRC18:47
yolandaclarkb, we have set like that, that prevents on creating the default route18:47
annegentle_fungi: pgp-encrypted email is great18:47
greghaynesyolanda: that sounds like exactly what we want to happen18:48
yolandaclarkb, to verify, i just updated the interfaces.d file, commenting the default route, and rebooting18:48
nibalizerfungi: so for fixing mail18:48
yolandagreghaynes, that 's what is happening then, but it is not routing properly18:48
nibalizerI need to assert hostname and /etc/mailname?18:48
nibalizerfor infracloud?18:48
*** dizquierdo has quit IRC18:48
fungiannegentle_: to your justwriteclick.com address?18:48
annegentle_fungi: yes, please18:49
funginibalizer: probably, yes18:49
annegentle_fungi: there's an Anne B at the Foundation now :)18:49
fungiannegentle_: perhaps unsurprisingly, i knew this ;)18:50
*** esker has quit IRC18:50
fungishe's also awesome, if you haven't met her yet hopefully you will in a few months18:50
clarkbjust don't race bicycles18:51
fungisrsly18:51
mordredfungi: agree18:51
clarkbnew bare-precise images are all built18:52
fungiannegentle_: sent18:54
fungishould hopefully receive it momentarily18:54
yolandaclarkb, greghaynes, so when i rebooted it worked ok. Issue may be also with a race, because at first boot, glean configures eth0.1598 as a tagged vlan, without adding the bridge18:55
yolandathen i come, and i configure bridge on top, and start it18:55
yolandanow on a reboot, with just eth0.1598 set as manual, it creates the gateway properly 0.0.0.0         15.126.48.1     0.0.0.0         UG    0      0        0 br-vlan159818:56
greghaynesyolanda: if the only thing adding a gateway is your script to add the bridge then there shouldnt be any race18:56
greghaynesyolanda: thats good it works, though18:56
yolandagreghaynes no, because glean configures eth0.1598 first18:56
yolandait creates a gateway there as well18:56
*** kzaitsev_mb has quit IRC18:56
greghaynesyolanda: ah!18:56
yolandainitially, network is configured without bridge, just the tagged vlan, and has the gateway that reads from config-drive18:57
asselinany bandersnatch experts around? I'm trying to figure out why bandersnatch mirror doesn't acutally add any missing packages18:57
greghaynesyolanda: ok, so for that your script should remove that route18:57
clarkbyolanda: greghaynes is there some way to mark an interface as should be ignored by glean?18:57
*** piet has quit IRC18:57
greghaynesclarkb: you can bake a config into the image18:58
greghayneswhich is not a good thing IMO18:58
clarkbasselin: when you run it what does the log say?18:58
yolandagreghayes, it will be a bit tricky to test because the problem will happen only on first boot. When glean is adding eth0.1598 . After reboots, as eht0.1598 is configured with bridge, it comes up correctly19:00
yolandabut i can give a try , so we simplify the networking even more19:00
*** esikachev has quit IRC19:00
asselinclarkb, there's a few errors, but none related to the missing packages. But the missing packages are there at all. e.g. https://pypi.python.org/pypi/oslotest19:00
asselin2.1.0 (2016-02-09)19:01
clarkbasselin: the way bandersnatch works is there is a journal of changes and it basically consumes the journal from beginning to end applying the changes19:01
clarkbasselin: the contents of the log will tell you where it is at on the stream, if it is up to date then your mirror likely missed or failed on some chnages19:02
clarkbasselin: if it thinks it is up to date yo ucan delete the status? file I think it is called and rerun and it will start from the beginning19:02
clarkbmaybe it is called state19:02
asselinclarkb, yes i did that feb 9th and now it's still missing files since then19:03
clarkbasselin: and you have run bandersnatch since then?19:04
asselinclarkb, also, I remember deleting that state file but can't seem to find it today.19:04
asselinclarkb, it runs as a cron every 5 minutes. i also ran it manually. no updates19:04
clarkbok then I would chekc the log to see where it thinks it is on the journal19:04
*** notnownikki has quit IRC19:05
greghaynesyolanda: yep, I can see where there would be a race - if the dhcp request for eth0.1598 is async and the bridge is configured after eth0.x is up but not before dhcp returns19:05
greghaynesyolanda: and I have no good idea for preventin gthat19:05
clarkbhttps://jenkins06.openstack.org/job/gate-fuel-agent-pep8/141/console is a fun one19:06
clarkbit hung trying to upload to swift19:06
greghaynesyolanda: it actually shouldnt be an issue I think - as long as the bridge is configured if the dhcp tries again later and fails that is totally fine19:06
clarkband I guess the job timeout doens't apply there because it is in the post build steps19:06
clarkbI am going to stop the job manually19:06
*** tiswanso has quit IRC19:07
*** kzaitsev_mb has joined #openstack-infra19:08
*** zz_dimtruck is now known as dimtruck19:10
*** dmsimard has quit IRC19:10
clarkbI rechecked it and left a note about why it failed19:10
*** rbrndt has joined #openstack-infra19:12
*** Sukhdev has joined #openstack-infra19:12
asselinclarkb, http://paste.openstack.org/show/486875/ Not sure why it says "Resuming interrupted sync from local todo list."19:14
clarkbasselin: that current serial = 0 log message is probably realted to not having a state file19:15
clarkbcheck file perms?19:15
clarkbasselin: is it running on a timeout?19:16
clarkbyou may need to run it manually to fully sync up19:16
clarkbthen let it run in cron19:16
asselindrwxr-xr-x 21 bandersnatch bandersnatch  4096 Feb 12 17:55 web19:16
crinkleyolanda: the latest cert you gave me still doesn't work for hostname controller00.hpuswest.ic.openstack.org, but it does if i muck with /etc/hosts to point localhost at controller00.hpuseast.ic.openstack.org, so i think the wildcard thing still isn't working19:16
asselinclarkb, yeah I did that a few days ago. I guess I deleted the todo file and had it run for about 5 hours19:17
clarkb2200 for gerrit work right?19:17
clarkbnibalizer: you want to try spin up a mirror first?19:17
clarkbanyone know why the proposal slave is offline?19:18
clarkbthat is part of the post queue backup19:18
clarkbhrm offline due to gearman request19:19
clarkbI am going to put it online but we have a job that needs to not be single using19:19
nibalizerclarkb: sure19:19
nibalizeri'm not sure how big of a deal it is that our mail keeps getting rejected19:19
clarkbnibalizer: well I think it is annoying to the people getting the bounces :)19:19
nibalizerhuh19:20
clarkbpropose-devstack-plugins is the job that took us offline19:20
nibalizerthe bounces don't seem to be reaching me19:20
nibalizeroh wait there it is19:20
nibalizerit goes into my homedir so19:20
clarkbhttps://jenkins.openstack.org/job/propose-devstack-plugins-list/19:21
nibalizerclarkb: need https://review.openstack.org/#/c/275477/ for mirrors19:25
nibalizerer for infracloud, for mirrors19:25
*** eil397 has quit IRC19:25
*** ccrouch1 has quit IRC19:25
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/27973519:25
clarkbwill review as soon as I fix the devstack plugins job19:25
*** dmsimard has joined #openstack-infra19:26
mordredclarkb: don't forget https://review.openstack.org/#/c/278797/19:26
mordredclarkb: it's calling your name :)19:26
*** kzaitsev_mb has quit IRC19:27
mordredclarkb: oh - nevermind - timburke has another good suggestion19:27
*** vgridnev has joined #openstack-infra19:27
*** bpokorny_ has joined #openstack-infra19:27
*** dtardivel has quit IRC19:28
*** ccrouch has joined #openstack-infra19:28
*** slogan621 has joined #openstack-infra19:28
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: rename dist to prepare for release  https://review.openstack.org/27971519:29
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: handle repositories in parts of the tree other than openstack/  https://review.openstack.org/27973919:29
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: expand the path to the repository directory  https://review.openstack.org/27974019:29
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: avoid some false positives finding bug urls  https://review.openstack.org/27974119:29
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: make temporary directories stand out  https://review.openstack.org/27974219:29
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: handle first releases better in the release notes  https://review.openstack.org/27974319:29
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/27974419:29
*** bpokorny_ has quit IRC19:29
*** mrmartin has quit IRC19:29
openstackgerritgreghaynes proposed openstack/diskimage-builder: Allow setting per-release mirrors for tests  https://review.openstack.org/27969119:30
*** bpokorny_ has joined #openstack-infra19:30
*** jsavak has quit IRC19:30
openstackgerritgreghaynes proposed openstack/diskimage-builder: Allow setting per-release mirrors for tests  https://review.openstack.org/27969119:30
*** bpokorny has quit IRC19:31
*** sdake_ has quit IRC19:31
*** sputnik13 has joined #openstack-infra19:31
openstackgerritClark Boylan proposed openstack-infra/project-config: Don't offline proposal when jobs run on it  https://review.openstack.org/27974519:32
clarkbI serached on gerrit to see if anyone else had written ^ first but couldn't find it19:33
clarkbmerging that would be good19:33
*** esp has joined #openstack-infra19:33
*** yamamoto_ has quit IRC19:34
*** dmsimard has quit IRC19:36
*** NobodyCa1 has joined #openstack-infra19:37
yolandacrinkle that's bad. I created that with alt names for *.hpusesat and west, and i verified that dns entries were shown19:37
*** sputnik13 has quit IRC19:37
*** krtaylor has joined #openstack-infra19:38
*** annegentle_ has quit IRC19:39
*** kzaitsev_mb has joined #openstack-infra19:39
*** annegentle_ has joined #openstack-infra19:40
*** esp has quit IRC19:40
*** jsavak has joined #openstack-infra19:40
*** andrey-mp has left #openstack-infra19:41
*** NobodyCa1 has quit IRC19:42
*** sdague has quit IRC19:42
*** austin81 has left #openstack-infra19:42
*** julim has joined #openstack-infra19:43
clarkbalso fwiw nodepool image builds and uploads are all happening19:43
nibalizerclarkb: are there osic paatches I should be reviewing19:43
clarkbI need to restat it to patch in the fix for failed builders19:43
clarkbwill likely do that later today19:43
clarkbnibalizer: not yet, this week turned into firefighting19:43
nibalizerok19:44
clarkbnibalizer: but I figured I could help you get your things through then osic19:44
nibalizerjust waanted to confirm19:44
nibalizerideally19:44
nibalizer:)19:44
slogan621Looking for one more +2 from a core for a new project. We've gone through about 7 changesets, the change is about as simple as it gets now :-). https://review.openstack.org/#/c/27587319:44
slogan621thanks!19:44
*** annegentle_ has quit IRC19:44
clarkbnibalizer: after the conversation yesterday is https://review.openstack.org/#/c/275477/4/modules/openstack_project/files/puppetmaster/infra_cacert.pem what we want?19:44
*** vponomaryov has left #openstack-infra19:44
nibalizerwhat was the conversation yesterday? I was in and out19:45
nibalizerwas it related to the 'run our own CA' spec yolanda put up?19:45
clarkbnibalizer: yes and we seemed to settle on lets just self sign and not run a CA19:45
slogan621AJaeger: thanks for your earlier +2, really appreciate it.19:46
*** edmondsw has quit IRC19:46
nibalizerclarkb: https://review.openstack.org/#/c/278989/19:46
*** bpokorny_ has quit IRC19:46
onovyanyone for simple final +A? :) https://review.openstack.org/#/c/278945/ // thanks19:46
nibalizerso the infracloud ca file that we're vendoring in there is basically that ya?19:47
*** bpokorny has joined #openstack-infra19:47
openstackgerritMatthew Treinish proposed openstack-infra/puppet-subunit2sql: Fix the expiration cron job  https://review.openstack.org/27929719:47
clarkbnibalizer: so thats a proper self signed cert ?19:47
mtreinishjeblair, pleia2: ^^^ I had to respin because of puppet-lint19:47
clarkbnibalizer: I am just checking because patch was pushed before discusion19:48
nibalizerclarkb: no thats the CA and CA chain file that were used to sign the certs in infracloud19:48
clarkbright which is what we don't want19:48
nibalizer'we wont use a CA, we'll use self-signed certs' isn't a thing?19:48
nibalizeryou always have a CA19:48
nibalizerthe difference is how much you trust the CA19:48
nibalizerin this case we write down the infra-root CA certs and point oscc at them19:49
clarkbnibalizer: we explicitly said that we would prefer to have each cert sign itself19:49
smarcetfungi: clarkb: afternoon, could u review https://review.openstack.org/#/c/279513/?19:49
clarkbthat was the discussion yesterday19:49
nibalizeris that a real thing?19:49
clarkbnibalizer: yes19:49
smarcetits to deploy openstackid to production19:49
mordredclarkb: were we able to get self-signed certs to work with intra-service communication?19:49
*** sabeen1 has joined #openstack-infra19:49
pleia2mtreinish: oh fun, lgtm19:49
mordredclarkb: I thought we determined that the CA was required for that19:49
clarkbmordred: yes it was working fine19:49
mordredok. great19:50
openstackgerritAliaksandr Buhayeu proposed openstack/python-jenkins: Adds method to retrieve all jobs from the specific view  https://review.openstack.org/27975019:50
clarkbthere was pebkac or something but I believe pabelanger and crinkle got it working19:50
mordredsweet19:50
*** kzaitsev_mb has quit IRC19:50
*** bpokorny has quit IRC19:50
clarkbpabelanger in the general case an crinkle in the make cloud work case19:50
*** HeOS has joined #openstack-infra19:50
mordredso then we'll just used "verify=false" in clouds.yaml for nodepool19:50
nibalizercrinkle: ?19:50
pabelangerYup, yolanda provided me a cert and got it working under fedora / centos using curl19:50
*** bpokorny has joined #openstack-infra19:51
mtreinishpleia2: thanks19:51
nibalizerand it works through puppet and all that?19:51
*** krtaylor has quit IRC19:51
mordredcurl isn't the problem19:51
clarkbmordred: right but curl wasn't working either19:51
nibalizerso I didn't know you could sign a cert with itself19:51
*** sdague has joined #openstack-infra19:51
mordreddid we get nova talking to glance with the slef-signed cert19:51
clarkbso it was solved in stages19:51
nibalizercan someone link me some docs on that19:51
mordredawesome19:51
mtreinishpleia2: it's those pesky {}19:51
clarkbmordred: that is what I believe crinkle did19:51
*** alex_xu has quit IRC19:51
clarkbshe posted an etherpad on how to do it19:51
mordredclarkb: sweet. I'm excited about that19:51
clarkbnibalizer: let me find the therpad in scrollback19:51
yolandanibalizer, so i got it working in puppet as well19:51
pabelangerYa, I didn't test with requests.  But could if needed19:51
yolandabut seems the alt names theory is not working good19:51
*** bpokorny has quit IRC19:52
yolandaand i cannot do just a wildcard because we are at two levels of depth19:52
yolandaso *.ic.openstack.org is not matching controllerxx.hpxxx.ic.openstack.org19:52
clarkbhttps://etherpad.openstack.org/p/debug-infracloud-ssl19:52
*** sabeen3 has joined #openstack-infra19:52
yolandapabelanger, worked fine finally with requests as well, as long as you add the cert to global trust19:52
pabelangeryolanda: right, I would expect that too19:53
*** bpokorny has joined #openstack-infra19:53
nibalizerso why is this better than a CA?19:53
nibalizerthis seems worse19:53
clarkbnibalizer: because its simple and doesn't require anything "extra"19:53
clarkbsee fungi's comment(s)19:53
*** alex_xu has joined #openstack-infra19:54
clarkband it is just as secure19:54
nibalizerI mean if I have to add stuff to my local certificate store on every unix box then yes, it does require extra19:54
*** sabeen1 has quit IRC19:54
nibalizerpoint is moving around and pointing to a CAfile is really easy to do with git19:54
clarkbyou have to do that with the CA too19:54
clarkbits the same effort19:54
nibalizerI don't have to no, I have OS_CACERT or similar19:54
crinklenibalizer: clarkb is https://review.openstack.org/#/c/279286/ what you're talking about?19:54
clarkbnibalizer: you can do that iwth self signed cert too...19:54
mordrednibalizer: you do not have OS_CACERT for service-to-server commuication19:55
openstackgerritZara proposed openstack-infra/storyboard-webclient: Make header 'quick navigate' and 'search' clearer  https://review.openstack.org/27898719:55
openstackgerritZara proposed openstack-infra/storyboard-webclient: WIP for boards page  https://review.openstack.org/27975319:55
mordrednibalizer: that's the difference - you must add it to the global trust either way19:55
clarkbcrinkle: no https://review.openstack.org/#/c/275477/4 is the change19:56
openstackgerritLucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: builders: update cmake to match CMake plugin versions 2.x  https://review.openstack.org/27975419:56
crinkleclarkb: ah, well the change i linked is what sets it up with just the self-signed certs19:57
nibalizerso its an issue for the compute hosts19:57
nibalizer?19:57
nibalizeralso how is that the solution, if the services all use requests which uses its own store?19:57
clarkbnibalizer: there are a few layers of problems, but using a self managed CA or using a wildcard self signed cert are roughly equivalent for management19:57
crinklenibalizer: we think that's not a problem because the distros package it to behave normally19:58
clarkbnibalizer: the deployed cloud using distro requests which will use the distro store. Our clients can be told to use OS_CACERT or don't verify, but that is the same issue in both cases19:58
fungiyeah, pypi-distributed requests bundles its own certs. distro-packaged python-requests gets that stripped out and pointed at the system-provided cert store19:58
nibalizerfun19:59
fungialso if we set up a loosely-secured "infra ca" there's a slippery slope where we might slowly start to depend on it for more secure systems19:59
*** alivigni has quit IRC20:00
nibalizerfungi: i mean easy_rsa in /opt/ca on puppetmaster wouldn't be too much of a management burden20:00
*** _nadya_ has joined #openstack-infra20:00
fungiit's not a management burden at all20:00
fungiit's an attractive nuisance vulnerability20:00
openstackgerritLucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: builders: update cmake to match CMake plugin versions 2.x  https://review.openstack.org/27975420:01
nibalizerI think I see what you're saying20:01
fungiat least with self-signed certs it's blatantly obvious we're not trusting them for security-sensitive things20:01
yolandacrinkle, i'll take a moment this weekend to look at proper way to generate cert with the alt names, i see that added on the csr i created but i'm missing something for sure20:02
fungiand if we get to the point later where we want to be able to trust infra-cloud for more secure things, we'll buy certs from a well-known ca for that20:02
crinkleyolanda: we could also not worry about it and use two different certs20:02
*** tiswanso has joined #openstack-infra20:02
yolandawell, i wanted to make it as simple as possible, but we can go at fqdn level and setup different ones, yes20:02
yolandanot an issue really, and we may end being at same region so the naming can change20:03
fungii have to disappear soon for an accountant appointment, but plan to be back prior to the beginning of our gerrit maintenance window20:04
fungit-minus two hours to maintenance, btw20:04
*** achanda has quit IRC20:04
clarkbcrinkle: yolanda or collapse the dns hierarchy?20:04
clarkband do something like controller01-hpwest20:05
fungis/\./-/20:05
fungiyeah20:05
crinkleyeah we could do that20:05
*** kzaitsev_mb has joined #openstack-infra20:05
*** sridhar_ram1 has joined #openstack-infra20:05
nibalizerclarkb: so thaat patch would end up looking different?20:06
nibalizerwe'd have the ssl cert itself in the patch instead of a CA file?20:06
nibalizerbut still point to it with OS_CACERT (or oscc equiv?)20:06
*** jtomasek_ has quit IRC20:06
clarkbnibalizer: ya20:07
*** sridhar_ram has quit IRC20:07
yolandaat these point what i know is that east is going to be moved to west20:08
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Rework devstack-jobs templating  https://review.openstack.org/27803420:08
yolandaso we need to take some decisions when they are there, if we want to unify both, continue with two regions...20:08
yolandabut we don't have the full details yet20:08
nibalizerok20:09
nibalizerso I wold like infracloud to boot nodes20:09
nibalizerso if we need to redo the ssl to do that lets do it20:09
clarkb++20:09
nibalizercrinkle: do you know how to do this, I do not20:09
*** achanda has joined #openstack-infra20:10
*** changbl has quit IRC20:10
crinklenibalizer: i need to fix https://review.openstack.org/#/c/279286/20:10
crinklethen we generate self-signed certs like normal20:10
openstackgerritZara proposed openstack-infra/storyboard-webclient: Make header 'quick navigate' and 'search' clearer  https://review.openstack.org/27898720:10
crinkleand put them in hiera20:11
*** kzaitsev_mb has quit IRC20:12
*** mriedem has quit IRC20:12
openstackgerritZara proposed openstack-infra/storyboard-webclient: Make header 'quick navigate' and 'search' clearer  https://review.openstack.org/27898720:12
fungiokay, heading out. be back in hopefully ~90 minutes20:13
*** mriedem has joined #openstack-infra20:13
*** ldnunes has quit IRC20:14
clarkbproposal slave is steadily chewing through the post queue20:16
clarkbI should get lunch before the fun starts20:16
nibalizerme too20:18
openstackgerritMerged openstack-infra/project-config: Use pep8 instead of linters for bandit  https://review.openstack.org/27862720:20
openstackgerritMerged openstack-infra/project-config: Use pep8 instead of linters for ekko  https://review.openstack.org/27864120:20
mordredclarkb: https://review.openstack.org/27879720:20
mordredclarkb: new plan - we'll now ALWAYS pass both a token and a username/password in20:21
mordredthat way we're doing the same token workflow as before20:21
mordredbut now adding to the params to swift all info we know about usernames and passwords20:21
mordredin case swift needs it20:21
*** edmondsw has joined #openstack-infra20:21
openstackgerritMerged openstack-infra/project-config: Remove magnum bandit gate  https://review.openstack.org/27827120:22
*** esikachev has joined #openstack-infra20:23
*** krtaylor has joined #openstack-infra20:26
openstackgerritMerged openstack/diskimage-builder: Handle install with pip -e  https://review.openstack.org/21956220:28
sdagueI would be happy if https://review.openstack.org/#/c/278612/ could become a thing before the weekend, so we can start pushing forward next week20:29
openstackgerritgreghaynes proposed openstack/diskimage-builder: Allow setting per-release mirrors for tests  https://review.openstack.org/27969120:31
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Update SSL configuration  https://review.openstack.org/27928620:32
AJaegersdague: +2. What about governance change?20:33
*** daemontool has quit IRC20:33
*** daemontool has joined #openstack-infra20:33
*** FallenPegasus has quit IRC20:34
*** daemontool has quit IRC20:34
Guest3687zaro: looks like ZMQ plugin for Jenkins is missing from update center. I found this bug: https://issues.jenkins-ci.org/browse/INFRA-55120:34
*** Guest3687 is now known as mgagne20:34
*** mgagne has quit IRC20:35
*** mgagne has joined #openstack-infra20:35
*** daemontool has joined #openstack-infra20:35
mgagnezaro: was me ^20:35
*** aysyd has quit IRC20:35
sdagueAJaeger: it's up20:35
clarkbmgagne: I ant to say someone deleted it20:35
clarkband it wasnt us20:35
sdagueunless I forgot to push20:35
clarkbso I asked zaro to make sure things are going to work if we uplaod20:35
sdaguewhich I did20:35
sdagueAJaeger: https://review.openstack.org/#/c/279780/20:36
AJaegersdague: let's add the link as comment -I'll do.20:36
AJaegersdague: thansk20:36
mgagneclarkb: bug mentions lack of wiki page.20:36
clarkbwow20:36
clarkbso did they delete our stuff rather than just add a wiki page?20:36
clarkbthats nice20:36
AJaegersdague: could you check 278091, please? I'm trying to reduce number of unused jobs...20:37
dimsfungi : looks like mtreinish 's subunit2sql puppet change is green, please peek when you get a chance https://review.openstack.org/#/c/279297/20:37
mgagneclarkb: seems to be something they planned to do for months20:37
anteayaso renaming happening at 22:00 utc today and here is the etherpad if anyone involved wants to get prepared: https://etherpad.openstack.org/p/repo-renaming-2016-02-1220:39
nibalizerwoot20:39
nibalizerty anteaya20:39
anteayaSergeyLukjanov: did you ever get a chance to look at that etherpad?20:39
clarkbmgagne: they didn't notify me20:39
anteayanibalizer: welcome, thank you20:39
mgagne=(20:39
clarkbmgagne: in any case we should host the builds on tarballs.o.o20:39
clarkbmgagne: let me find a link20:39
mgagnethat's a bit inconvenient for people relying on update center to install plugins20:40
SergeyLukjanovanteaya I've just dropped all meetings and trying to hide in the meeting room :) starting looking on the etherpad20:40
clarkbmgagne: http://tarballs.openstack.org/ci/zmq-event-publisher/20:40
*** hashar has joined #openstack-infra20:40
*** annegentle_ has joined #openstack-infra20:40
clarkbmgagne: eh its the same amount of work, you go in the gui and instead of selecting version from lits put in the url20:40
mgagneyou know what I meant =)20:40
SergeyLukjanovI can rebase patches if it's not yet done20:40
clarkbmgagne: and I can control tarballs.o.o and won't delete those on your20:41
anteayaSergeyLukjanov: okay thanks20:41
clarkbmgagne: upstream has clearly shown they don't care20:41
anteayaSergeyLukjanov: well look over the ether pad first20:41
SergeyLukjanovanteaya sure20:41
anteayaSergeyLukjanov: ask any questions you need to20:41
AJaegerSergeyLukjanov: are any patches conflicting? In that case, you want to base them on top of each other20:41
*** sdague has quit IRC20:41
anteayaSergeyLukjanov: thanks20:41
anteayaAJaeger: none were conflicting yesterday20:41
SergeyLukjanovAJaeger shouldn't be conflicting20:41
AJaegergreat20:42
AJaegeranteaya, SergeyLukjanov : I'd like to remove my -2 from https://review.openstack.org/#/c/269815/, any of you wants to block it?20:42
anteayaSergeyLukjanov: why don't you go ahead20:42
anteayaSergeyLukjanov: and -2 https://review.openstack.org/#/c/248897/ while you are at it as well and I'll remove my -220:43
SergeyLukjanovsure, lemme put my -2 on them to secure20:43
anteayathanks20:43
* AJaeger won't be around at 22:00...20:43
anteayaAJaeger: understood, thanks for your help20:43
SergeyLukjanovall -2'd20:43
SergeyLukjanovAJaeger thx20:43
anteayaSergeyLukjanov: thank you20:43
anteayathis patch needs to be merged before the downtime: https://review.openstack.org/#/c/248126/20:45
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove linters from jenkins/jobs/python-jobs  https://review.openstack.org/27978120:45
anteayaSergeyLukjanov nibalizer if you could be so kind as to review 24812620:45
*** hashar has quit IRC20:45
*** hashar has joined #openstack-infra20:46
*** _nadya_ has quit IRC20:46
anteayapleia2: this script of zaro's you +A'd last night failed in the gate: https://review.openstack.org/#/c/270537/20:46
anteayawe can still use the script but might be good to know why it failed20:46
*** hashar has quit IRC20:46
SergeyLukjanovanteaya 248126 approved20:47
*** hashar has joined #openstack-infra20:47
anteayaSergeyLukjanov: thank you20:47
*** hashar_ has joined #openstack-infra20:47
openstackgerritMerged openstack-infra/jeepyb: Don't create new groups for system ones  https://review.openstack.org/24812620:47
anteayaSergeyLukjanov: zaro left new suggestions for reindexing in step 7, if you could review them and ask any questions, that would be great20:47
AJaegeranteaya: 270537 has no logs, looks like it run into timeout. Let's recheck it...20:48
*** yamamoto_ has joined #openstack-infra20:48
SergeyLukjanovanteaya ack20:48
anteayaAJaeger: thanks for looking20:48
SergeyLukjanovI've review the etherpad and it seems ok for me, looking on reindexing doc now20:48
anteayaSergeyLukjanov: thank you20:48
SergeyLukjanovso, we have an hour before the maintenance widnow, I prefer to rebase renaming patches one on top of another to have new tests results and ensure no conflicts20:49
SergeyLukjanovanteaya are ok with it?20:49
anteayaI am fine with that20:51
anteayathanks SergeyLukjanov20:51
SergeyLukjanovIMO we can try to enable more therads for reindex20:51
pleia2anteaya: it timed out on the precise job, probably needs a recheck20:51
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/27973520:51
pleia2ah, AJaeger is ahead of me :)20:51
anteayapleia2: I think AJaeger took care of it20:51
anteayapleia2: he's like that :)20:51
SergeyLukjanovand see how long it'll be with let's say 8 threads or even more -- it'll be useful for future20:52
pabelangerclarkb: mgagne: they might just be hiding the directory: https://updates.jenkins-ci.org/download/plugins/zmq-event-publisher/0.0.3/zmq-event-publisher.hpi still works for me20:52
SergeyLukjanovotherwise I'm afraid we'll have extra load on gerrit after start20:52
anteayayes, tracking our reindexing time will be helpful20:52
*** yamamoto_ has quit IRC20:52
pabelangerclarkb: mgagne: but good to know, going to update playbooks20:52
*** smarcet has quit IRC20:52
*** apoorvad has quit IRC20:52
anteayaSergeyLukjanov: extra load on gerrit after start, how do you mean?20:52
openstackgerritSergey Lukjanov proposed openstack/requirements: Apply sahara-scenario -> sahara-tests rename  https://review.openstack.org/27978420:53
SergeyLukjanovanteaya I mean that if we'll skip reindex during the rename maintenance window20:54
anteayaokay well unfortunately zaro is not here20:54
anteayaand I don't know all the ramifications of reindexing after gerrit is started20:55
anteayapleia2 nibalizer have you any thoughts about step 7?20:55
SergeyLukjanovAFAIR we were always reindexing while renaming maintenance20:55
openstackgerritMadhusudhan Kandadai proposed openstack-infra/project-config: Add a new CI to run neutron_vpnaas tempest tests  https://review.openstack.org/27978520:55
*** hashar_ has quit IRC20:55
SergeyLukjanovand I'm proposing add more threads (default is only one) and record needed time20:55
SergeyLukjanovnext time we could do it after gerrit started20:56
SergeyLukjanovand compare the time20:56
SergeyLukjanovand how gerrit will be affected by old index20:56
*** zeih has joined #openstack-infra20:56
pleia2SergeyLukjanov: we always wait for the reindexing to complete20:57
anteayaSergeyLukjanov: you are driving this maintenance, I'm fine with that as an approach20:57
pleia2it's a decent chunk of time spent doing nothing20:57
clarkbpleia2: anteaya fungi SergeyLukjanov https://review.openstack.org/#/c/279745/1 would be good to get in asap to avoid offlining the propsal slave again20:57
pleia2is there any reason we *shouldn't* do online reindexing now that it's supported?20:57
anteayapleia2: zaro offered new instructions since reindexing has changed for 2.1120:57
pleia2clarkb: lgtm20:57
anteayaunfortunately zaro isn't here to help us understand and guide this decision20:57
SergeyLukjanovclarkb looking on it20:58
pleia2anteaya: ok, maybe it's best to go with SergeyLukjanov's plan this time around, we'll get some data and can work with zaro to do online reindexing next time20:58
anteayapleia2: I support that20:58
*** esikachev has quit IRC20:58
SergeyLukjanovin etherpad it's said -- "If we want to minimize downtime" so, I prefer to consult with zaro first on this question20:58
* pleia2 prefixes reindex command with `time`20:58
anteayaI wouldn't ask someone driving a maintenance to take any steps they felt uncomfortable with20:59
SergeyLukjanovmy experience says that old cache is very bad :)20:59
anteayapleia2: thank you20:59
anteayaSergeyLukjanov: fair enough20:59
anteayaSergeyLukjanov: we have an hour for this maintenance if we need it20:59
SergeyLukjanovit should be more than enough :)21:00
anteayawe aren't schduled to have gerrit back online until 23:00 utc21:00
anteayagreat21:00
anteayaSergeyLukjanov: were you going to specify threads to be used in the reindex command?21:01
*** ericksonsantos has quit IRC21:01
SergeyLukjanovdo we need https://review.openstack.org/#/c/279745/1 merged before rename?21:01
anteayaSergeyLukjanov: if so, would you like to update the etherpad with the command you would like to run?21:01
* nibalizer is setup for downtime21:01
SergeyLukjanovanteaya yup, I'm currently googling for potential issues with it21:01
anteayanibalizer: thank you21:02
AJaegerpleia2: I have a translation change up, could I bother you with putting it on your review queue for later, please? https://review.openstack.org/#/c/279255/21:02
anteayaclarkb: does 279745 need to be merged before the maintenance window?21:02
anteayaSergeyLukjanov: awesome thank you21:02
clarkbanteaya: no, just before we all go off to weekend things and forget and the slave goes offline again21:03
anteayaclarkb: very good thank you21:03
anteayaSergeyLukjanov: ^^21:03
*** salv-orlando has quit IRC21:03
pleia2AJaeger: I'll add it to my list21:04
anteaya8 threads for reindexing, anyone have any feedback on SergeyLukjanov's decision?21:04
AJaegerthank you, pleia221:04
*** changbl has joined #openstack-infra21:05
clarkbanteaya: iirc we found it didn't speed things up much but it didn't hurt either so giving it a go should be fine21:05
nibalizeryep21:05
nibalizeri want to say I ran it with 4 last time21:05
SergeyLukjanovanteaya I'm not yet sure, reading about how it's working, but I expect to set threads number equal to CPU cores21:05
openstackgerritMadhusudhan Kandadai proposed openstack-infra/project-config: Add a new CI to run neutron_vpnaas tempest tests  https://review.openstack.org/27978521:05
*** piet has joined #openstack-infra21:05
*** rlandy has quit IRC21:06
anteayaokay great21:06
AJaegerthanks, SergeyLukjanov !21:07
clarkbslogan621: re https://review.openstack.org/#/c/275873/8 I can approve it after the gerrit downtime. There is no existing repo to import?21:07
openstackgerritMerged openstack-infra/project-config: Don't offline proposal when jobs run on it  https://review.openstack.org/27974521:08
openstackgerritSergey Lukjanov proposed openstack-infra/project-config: Rename fuel-plugin-astra to fuel-plugin-astara  https://review.openstack.org/27913821:09
openstackgerritSergey Lukjanov proposed openstack-infra/project-config: rename ceilometer-specs to telemetry-specs  https://review.openstack.org/24889721:09
openstackgerritSergey Lukjanov proposed openstack-infra/project-config: Rename sahara-scenario -> sahara-tests  https://review.openstack.org/26981521:09
*** weshay has quit IRC21:09
*** kzaitsev_mb has joined #openstack-infra21:09
AJaegeranteaya: Seems you had the same allergic reaction that I had reaction the commit message ;)21:09
anteayaAJaeger: everything is now a CI21:09
anteayaI'm not a fan of the current fashion trend21:10
AJaegeranteaya: including you and me? ;)21:10
notmynamefungi: mordred: sweet. got updated this time. thanks https://pypi.python.org/pypi/swift-bench21:10
*** |-paul-| has joined #openstack-infra21:10
openstackgerritColleen Murphy proposed openstack-infra/system-config: Use SSL for rabbitmq  https://review.openstack.org/27760521:10
*** weshay has joined #openstack-infra21:11
pabelangerclarkb: so, reading more about the removal of zmq-event-publisher. I guess it is because the code is not hosted under the jenkins namespace?21:11
clarkbpabelanger: I have no idea21:11
odyssey4meis there a way to kill this job - it's been hanging for nearly 7 hours now: https://jenkins04.openstack.org/job/gate-openstack-ansible-openstack_hosts-dsvm-ansible-functional-trusty/5/21:11
clarkbpabelanger: I am just frustrated that they said nothing to me/us, removed it, and now I am expected to fix it21:12
*** weshay is now known as msg21:12
clarkbpabelanger: when I already host a useable alternative21:12
*** msg is now known as weshay21:12
pabelangerthis is what I see: Upload your plugin code to a repository on GitHub21:12
pabelangerThis will be forked into the Jenkins organisation, which will become the canonical source repo21:12
pabelangerguess that is the issue now21:12
anteayado we feel like we should post a downtime reminder to the mailing list?21:12
clarkbodyssey4me: yes, also thats the second one that hung on swift21:12
clarkbpabelanger: ok I don't want them to be upstream either21:12
clarkbodyssey4me: one sec21:12
pabelangerclarkb: right, that's what I expect too21:13
odyssey4methanks clarkb21:13
pabelangerreading from https://wiki.jenkins-ci.org/display/JENKINS/Hosting+Plugins21:13
*** sc68cal has quit IRC21:13
clarkbodyssey4me: and done21:13
clarkbI mean I suppose nothing prevents someone else from forkingand making that the upstream21:14
clarkbbut I am not personally interested in doing that21:14
openstackgerritMadhusudhan Kandadai proposed openstack-infra/project-config: Add a new job to run neutron_vpnaas tempest tests  https://review.openstack.org/27978521:14
odyssey4methanks clarkb - recheck in progress21:14
clarkbpabelanger: they once force pushed an old code state over all o ftheir plugins21:15
clarkbpabelanger: I don't find that to be a very good place to host code I am responsible for21:15
pabelangerclarkb: agreed.21:16
clarkbbeause they hand out admin rights to just about anyone21:16
pabelangerespecially since it is just a mirror of tarballs.o.o21:16
*** zeih has quit IRC21:16
nibalizerclarkb: what happened with the plugin?21:16
clarkbnibalizer: apparnetly because it isn't hosted in their github or we got removed from the maven repo21:17
madhu_akAJaeger, anteaya thanks for clearing the confusion.. ha ha :)21:17
clarkbnibalizer: which is fine I guess, except that we applied for an account and everything to push there ...21:17
clarkbnibalizer: and used the process that was documented. Then they changed it without notifying us21:17
*** tmcpeak has joined #openstack-infra21:18
*** yamamoto_ has joined #openstack-infra21:18
tmcpeakhey all, Happy Friday21:18
nibalizerugh, did they send us an email?21:18
tmcpeakany idea why Zuul isn't posting status on my review here? https://review.openstack.org/#/c/279767/21:18
nibalizerI can poke rtyler and ask him wtf is going on21:18
anteayamadhu_ak: thanks for listening21:18
clarkbnibalizer: I didn't receive anything21:18
clarkbbut maybe it get spam assassined21:18
AJaegertmcpeak: it's "recheck", "reverify" is obsolete...21:19
anteayatmcpeak: I see a status update from 10 minutes ago: https://review.openstack.org/#/c/279767/21:19
tmcpeakahh ok21:19
slogan621clarkb: thanks! we have source code for the repo that conforms to the openstack cookiecutter template that will be initially committed once the repo exists21:19
anteayatmcpeak: what were you looking for?21:19
clarkbslogan621: ok, that works to21:19
clarkb*too21:19
pabelangerhere are the 2 commits to the wiki page of interest21:20
pabelangerhttps://wiki.jenkins-ci.org/pages/diffpages.action?originalId=79495453&pageId=7949545521:20
AJaegertmcpeak: and I see it running at http://status.openstack.org/zuul/21:20
*** abregman|afk is now known as abregman21:20
pabelangerand https://wiki.jenkins-ci.org/pages/diffpages.action?originalId=73533956&pageId=7441752421:20
tmcpeakanteaya: doesn't the status of the gates typically get posted on the review itself in line with the patch upload status etc?21:20
pabelangerwhich changes the hosting requirements21:20
AJaegertmcpeak: so, drink a beer and wait ;)21:20
tmcpeakAJaeger: ^21:20
AJaegertmcpeak: of gates yes. But it's in check queue21:20
tmcpeakalthough that idea sounds reasonable too21:20
tmcpeakahhh21:20
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Import puppet-pacemaker from github/redhat-openstack  https://review.openstack.org/27969821:20
AJaegertmcpeak: and it will not go in the gate queue without a +A ;) Note that even if it's in the gate and fails there, a recheck will put it back in the check queue...21:21
*** smarcet has joined #openstack-infra21:21
*** yamamoto_ has quit IRC21:22
tmcpeakAJaeger: ahh ok, I could be remembering wrong but I thought it normally posts check status there too21:22
AJaegertmcpeak: no, it does not21:22
AJaegertmcpeak: only once it's finished...21:23
tmcpeakAJaeger: you're right21:23
tmcpeak:D21:23
tmcpeakmaybe I should follow your advice, drink that beer and embrace Friday21:23
AJaegerteam, I'll have to leave now and wish you a smooth rename21:23
pleia2AJaeger: have a nice weekend21:23
AJaegerI have a few cleanup changes, would be glad if some could put them on their review queue, please. I suggest not to approve before the rename. Changes are: 278073, 278061, 278084 (those are stacked) and 27978121:24
openstackgerritMerged openstack-infra/system-config: Add helper script to delete Gerrit groups  https://review.openstack.org/27053721:24
clarkbI could go for a Helles21:24
tmcpeakcool thanks AJaeger, anteaya sorry for false alarm :P21:24
* AJaeger wishes everybody a great weekend21:24
clarkbAJaeger: enjoy21:24
pleia2AJaeger: btw, I'm out monday re: US holiday, so see you tuesday!21:24
daemontoolAJaeger, likewise21:24
slogan621clarkb: one we have done that, we will be creating two, possibly more projects using what we learned doing this one (and I am thankful we decided to just try doing one of them first, we had a lot to learn!). Together, these will support Monasca and eventually ceilometer. We have a talk proposed on all of this along with the Monasca PTL for Austin.21:24
slogan621https://www.openstack.org/summit/austin-2016/vote-for-speakers/Presentation/853621:24
anteayatmcpeak: no worries21:24
AJaegerpleia2: ah, that US holiday... Everybody in the US: Enjoy a *long* weekend21:24
anteayaAJaeger: thank you21:24
*** tmcpeak has left #openstack-infra21:24
clarkbah yes the good ol' make a bunch of new projects before the usmmit rush :)21:24
anteayaAJaeger: enjoy the weekend21:24
pleia2clarkb: must.resist.grumpy.21:25
slogan621clarkb: heh, yeah, I can imagine21:25
anteayaI get a lot out of a good round of grumpy when it comes to using things and people to obtain a summit talk slot21:25
clarkbpleia2: time for a helles21:26
* slogan621 waves21:26
anteayalooking forward to ttx's proposal for spliting them up21:26
pleia2clarkb: ++21:26
clarkbI actually don't have any helles but maybe I can get some21:26
pleia2yeah, you live in portland :)21:26
pleia2and it's almost helles season21:26
*** erlon has quit IRC21:26
nibalizerhuh21:26
nibalizerso the doc says in one place 'We encourage plugin developers to host the plugins in the Jenkins community repositories on Github.' and in another place 'you must'21:27
anteayaat one point does the gracefully shut down zuul part need to begin?21:28
clarkbanteaya: recently we have just been dumping the queues and reenqueing after, so probably when we start21:28
anteayaif I remember correctly that takes a while, yes, and is usually started prior to the gerrit downtime so zuul is offline by the time we begin?21:28
anteayaclarkb: okay thanks21:29
*** matrohon has joined #openstack-infra21:29
SergeyLukjanovack21:29
pabelangeranteaya: then you'll love: https://www.openstack.org/summit/austin-2016/vote-for-speakers/presentation/7337 \o/21:30
anteayaif I remember correctly Monasca is one of the repos that still lacks installation docs, if I recall my recent conversation with thingee21:30
anteayapabelanger: I doubt I want to click on that link21:30
anteayaI highly doubt it21:31
clarkbits the "how to work with upstream" talk21:31
nibalizerdoes rally still have a curl| sudo bash as the install docs?21:31
anteayatoo bad Monasca can't seem to attract some devs to help with installation talks21:31
anteayapabelanger clarkb oh okay yeah, so perhaps a short bit in there about not createing repos to game the summit talk selection committee21:32
anteayaif you feel so inclined21:32
anteayas/installation talks/installation docs21:32
*** kzaitsev_mb has quit IRC21:33
anteayatalking about it apparently isn't the same as documenting it21:33
cp16netAJaeger: i updated this patch with a better commit message like you requested. https://review.openstack.org/#/c/278095/21:33
*** kgiusti has left #openstack-infra21:33
nibalizerum21:34
nibalizerhttps://review.openstack.org/#/c/279138/1 i can't voet on that for whatever reason21:34
*** jsavak has quit IRC21:34
anteayaI see a vote from you21:34
clarkbnibalizer: you have to change the patchset to 221:34
clarkbnibalizer: its the new ui being helpful21:35
*** sdake has joined #openstack-infra21:35
*** changbl has quit IRC21:35
anteayayou gave a +2 on patchset 221:35
anteayaI see it21:35
anteayaat :33 after the hour21:35
*** jsavak has joined #openstack-infra21:35
nibalizerya21:35
nibalizerit got a new patchset between me opening and finishing reviewing i guesss21:36
anteayaah21:36
nibalizeror21:36
nibalizeri guess i voted21:36
nibalizerman this new thing is really bad21:36
anteayaI see a vote from you21:36
anteayaI also find the new gui tricky21:36
*** tpsilva has quit IRC21:36
*** marcusvrn_ has quit IRC21:37
jeblairgertty master is in an excellent state if anyone feels like trying out conflicts-with and recently-seen changes :)21:37
anteaya:)21:37
nibalizerjeblair: gertty works great for me sitting down to review21:38
jeblairalso project topics are way cool21:38
nibalizerbut its hard for me to take a link in chat or etherpad and put that in gertty21:38
*** esikachev has joined #openstack-infra21:38
nibalizeri guess topics help21:38
jeblairnibalizer: oh!  did you see the thing we did with clickable links?21:38
nibalizerI think I heard something something urxvt21:38
nibalizerhaven't followed up though21:38
jeblairnibalizer: http://gertty.readthedocs.org/en/master/usage.html#terminal-integration21:39
jeblairyeah, tbh, it's way more useful than i thought it would be.21:39
anteayanice21:39
*** piet has quit IRC21:40
jeblairmaybe other terminals support that kind of thing too, i dunno.  will accept documentation patches if they do.  :)21:40
SergeyLukjanovanteaya I've found on the rename doc that we have rename-projects playbook21:40
SergeyLukjanovhttps://github.com/openstack-infra/system-config/blob/master/playbooks/rename_repos.yaml21:40
anteayayeah I don't believe that works21:40
SergeyLukjanovoh, okay21:40
anteayalast i heard that was mordred's playbook for making the big stackforge rename less painful21:40
jeblairwe're at t-20, right?21:40
pleia2jeblair: yep21:41
anteayathat wasn't ready in time21:41
jeblairshall we status notice?21:41
anteayaSergeyLukjanov: but thanks for mentioning it21:41
anteayajeblair: good idea21:41
*** bauwser is now known as bauzas21:41
jeblairwho wants to do that? :)21:42
nibalizeri can21:42
anteayanibalizer: thank you21:42
jeblairis there an etherpad?21:42
nibalizeryes21:42
SergeyLukjanovhttps://etherpad.openstack.org/p/repo-renaming-2016-02-1221:42
nibalizerhttps://etherpad.openstack.org/p/repo-renaming-2016-02-1221:42
* anteaya refrains from also posting the link21:43
jeblairi think during the downtime, i'm going to delete the repos on the zuul mergers21:43
anteayaumm, I don't have context for why that is helpful21:43
nibalizersomething like "the infrastructure team is taking gerrit offline for maintenance this afternoon. We should have it back online around 23:00 utc. http://lists.openstack.org/pipermail/openstack-dev/2016-February/086195.html"21:43
fungiokay, back now21:43
anteayais that a step that needs to be added to our rename workflow?21:43
anteayafungi: welcome back21:43
clarkbnibalizer: beginning at 2200UTC ?21:44
jeblairSergeyLukjanov: 3 changes, right?21:44
jeblairi have updated their topics to "project-rename"21:44
anteayanibalizer: yeah add what clarkb said and sounds good21:44
anteayajeblair: thank you21:44
jeblairso the "topic:project-rename" returns all of them21:44
nibalizerok21:44
SergeyLukjanovjeblair 4, +1 for requirements/projects.txt21:44
nibalizer#status The infrastructure team is taking gerrit offline for maintenance this afternoon, beginning at 22:00 utc. We should have it back online around 23:00 utc. http://lists.openstack.org/pipermail/openstack-dev/2016-February/086195.html21:44
openstackstatusnibalizer: unknown command21:44
nibalizer!status The infrastructure team is taking gerrit offline for maintenance this afternoon, beginning at 22:00 utc. We should have it back online around 23:00 utc. http://lists.openstack.org/pipermail/openstack-dev/2016-February/086195.html21:44
openstacknibalizer: Error: "status" is not a valid command.21:44
jeblairnibalizer: status notice21:44
*** sabeen3 has quit IRC21:45
nibalizer#status notice The infrastructure team is taking gerrit offline for maintenance this afternoon, beginning at 22:00 utc. We should have it back online around 23:00 utc. http://lists.openstack.org/pipermail/openstack-dev/2016-February/086195.html21:45
openstackstatusnibalizer: sending notice21:45
SergeyLukjanovjeblair I've set topic to project-rename for it as well21:45
jeblairnibalizer: (and then, at 2200, we usually follow it up with a "status alert" (not notice) saying "gerrit is down...")21:45
anteayajeblair: including the requirements patch21:45
jeblairnibalizer: notice is an announcement, alert is a topic change that must be cleared with 'status ok'21:45
*** yolanda has quit IRC21:45
anteayajeblair: so thank you21:45
*** ajmiller has quit IRC21:45
nibalizerjeblair: rgr21:45
openstackgerritMerged openstack-infra/project-config: Make propose_translation_update more robust  https://review.openstack.org/27925521:46
-openstackstatus- NOTICE: The infrastructure team is taking gerrit offline for maintenance this afternoon, beginning at 22:00 utc. We should have it back online around 23:00 utc. http://lists.openstack.org/pipermail/openstack-dev/2016-February/086195.html21:46
*** yolanda has joined #openstack-infra21:46
*** sridhar_ram1 is now known as sridhar_ram21:46
*** achanda has quit IRC21:46
jeblairi'm going to claim the zuul steps21:47
openstackstatusnibalizer: finished sending notice21:47
jeblairoh, we're not doing online reindexing?21:48
*** yamamoto_ has joined #openstack-infra21:48
*** nikhil has quit IRC21:48
*** ajmiller has joined #openstack-infra21:48
anteayajeblair: SergeyLukjanov has questions about it, zaro isn't here to answer them21:48
SergeyLukjanovI'm going to run 6-821:48
pleia2welp, I guess I'll be the guinea pig for that new script in step 4.521:49
anteayapleia2: go you!21:49
SergeyLukjanovjeblair AFAIR it wasn't too long to run it, but what we'll happen when users will start using gerrit with outdated cache? is it safe enough?21:49
clarkbI am uh around and willing to be told what to do21:49
anteayaclarkb: see anything you like in the etherpad?21:49
clarkbI can github21:49
clarkbsince no one has taken those yet21:49
anteayaclarkb: trooper, thank you21:50
anteayaI know you don't like that job21:50
openstackgerritJulia Kreger proposed openstack-infra/glean: Revise systemd determination to verify systemctl presence  https://review.openstack.org/27981821:50
jeblairSergeyLukjanov: yeah, i expect that it would just mean search results are inaccurate for a bit, but i understand if you don't want any extra surprises for the first rename on this version21:50
clarkbeh it need sto be done :)21:50
SergeyLukjanovI'm putting myseld on puppetmaster ON/OFF21:50
SergeyLukjanovjeblair exactly :)21:50
anteayaclarkb: that it does, thank you21:50
openstackgerritEvgeny Sikachev proposed openstack/requirements: Rename sahara-scenario->sahara-tests  https://review.openstack.org/27981921:50
*** weshay has quit IRC21:51
anteayastep 16 hasn't been claimed yet21:51
anteayawere we saving something for fungi?21:52
pleia2added the actual command for 4.5 if anyone wants to confirm21:52
clarkbanteaya: it doesn't apply beause it isn't an org move21:52
*** yamamoto_ has quit IRC21:52
pabelangerpleia2: mind linking the steps you are following along on?21:52
fungidon't feel obligated to save any steps for me21:52
anteayaclarkb: shall we strikethrough step 16 then?21:52
fungijust coming up to speed and trying to catch up on any urgent things which transpired over the last 90 minutes, sorry21:52
clarkbanteaya: yup21:52
anteayathanks21:53
*** apoorvad has joined #openstack-infra21:53
pleia2pabelanger: https://etherpad.openstack.org/p/repo-renaming-2016-02-1221:53
anteayafungi: no worries, looks like we have a good crowd today21:53
pabelangerpleia2: danke21:53
fungiyeah, we have lots of people, so i'm happy to let others get more experience wit this process and just jump in if things go sideways because gerrit 2.11 something something21:54
pleia2also, I am trusting zaro's step 4.5 though I'm not sure what the 'Registered Users' group is or why it's errant :)21:54
anteayafungi: thanks21:54
pabelangerfungi: have we ever tested the ansible playbook that mordred created?21:55
anteayapleia2: https://review.openstack.org/#/admin/groups/1230,members21:55
fungiwe have a comfortable hour budgeted for this so hopefully nobody feels rushed21:55
pabelangerfungi: for renames...21:55
openstackgerritJulia Kreger proposed openstack-infra/glean: Revise systemd determination to verify systemctl presence  https://review.openstack.org/27981821:55
anteayapleia2: that group should not have been allowed to be created21:55
pleia2anteaya: aha, just happened to find it hanging out there?21:55
fungipabelanger: nope, it's documented (unfortunately the documentation for that merged and replaced the process we'd been using before it was completely tested) but we should plan for that next time21:55
pleia2anteaya: I see :)21:55
anteayapleia2: Registered Users is a system-group21:55
SergeyLukjanov-5 mins21:56
anteayapleia2: yet 2.11 allows ssh group creation of system-groups21:56
anteayapleia2: zaro filed a bug and fixed it upstream21:56
fungi"internal" groups i think is the term21:56
openstackgerritMerged openstack-infra/puppet-subunit2sql: Fix the expiration cron job  https://review.openstack.org/27929721:56
anteayafungi: thanks, internal groups21:56
fungibut yeah, more accurately, the ssh api allows you to create groups with the same names as internal groups21:56
anteayapleia2: but a backport was too messy21:56
fungiand then ends up using those instead when mentioned in acls21:56
*** achanda has joined #openstack-infra21:57
fungiregardless, jeepyb is now fixed to avoid doing that21:57
anteayapleia2: right so 4.5 cleans up the current mess21:57
fungithe gerrit upstream patch was more as a courtesy to others21:57
anteayapleia2: make sense yet?21:57
anteayayeah, and so we don't get bitten again when we upgrade21:58
*** daemontool has quit IRC21:58
*** apoorvad has quit IRC21:58
pleia2yep, makes sense21:58
fungiso the main thing 4.5 is doing is removing the group declaration for "registered users" in the database, because it's not supposed to be there21:58
anteayawonderful, thanks for asking about it21:58
pleia2looks like I'll need to run this with sudo21:58
anteayaah21:59
pleia2root or gerrit2 user?21:59
pleia2this is the script: https://review.openstack.org/#/c/270537/5/tools/gerrit_delete_group.sh21:59
pleia2(merged now, but this link was handy)21:59
fungiif you do sudo -H then root should work and not prompt for credentials21:59
fungiit's only altering the db21:59
fungiso root is fine for that21:59
pleia2fungi: wfm21:59
anteayago22:00
fungialternatively you could do it with the gerrit db credentials22:00
jeblairplease wait22:00
SergeyLukjanovdoes somebody writing the alert?22:00
fungisudo -H is really more just a shortcut because we have a ~root/my.cnf with credentials already in it22:00
anteayawaiting22:00
pleia2fungi: yep, right22:00
nibalizerSergeyLukjanov: i can ya22:00
anteayanibalizer: waiting for jeblair22:00
SergeyLukjanovnibalizer yeah, plz22:00
*** thorst has quit IRC22:01
jeblairi'm wondering if we should wait just a minute for https://jenkins06.openstack.org/job/gate-tempest-dsvm-neutron-full/21588/console to finish22:01
anteayaI'm fine waiting for a minute22:01
*** dprince has quit IRC22:01
jeblairi'm not sure how much time it has left22:01
anteayaany objections to waiting?22:01
SergeyLukjanovjeblair I think we can stop puppetmaster already?22:01
fungiyeah, stopping puppet is safe22:02
jeblairSergeyLukjanov: oh, er, that needs to have happened already actually22:02
nibalizer#status alert Gerrit is offline for maintenacne until 23:00 utc22:02
openstackstatusnibalizer: sending alert22:02
fungiusually i do that 15-30 minutes before the maintenance22:02
nibalizeroh man i can totally spell22:02
pleia2nibalizer: it's a tricky one22:02
anteayasorry I asked, and I didn't remember that step22:02
fungiwhat a terrible case of mainten acne22:02
jeblairput some cream on it22:02
SergeyLukjanovI've commented puppet runs on master, but it seems like ansible playbooks is now running22:03
*** rossella_s has quit IRC22:03
nibalizerdo i have to wait for it to propogate to change?22:03
jeblairSergeyLukjanov: yeah, we'll need to wait for it to finish22:03
fungiSergeyLukjanov: yeah, that's why i usually do it early, so it has time to complete if it's running22:03
anteayagood thing we have an hour22:03
jeblairnibalizer: yes, it sets topics as it makes the announcements22:03
*** rossella_s has joined #openstack-infra22:03
*** smarcet has quit IRC22:04
openstackgerritAntoine Musso proposed openstack/diskimage-builder: debian-minimal: configurable debootstrap components  https://review.openstack.org/27982122:04
SergeyLukjanovsorry folks, I forgot about it, heh22:04
hasharoh diskimage-builder spams this place ;)22:04
-openstackstatus- NOTICE: Gerrit is offline for maintenacne until 23:00 utc22:04
*** ChanServ changes topic to "Gerrit is offline for maintenacne until 23:00 utc"22:04
anteayaSergeyLukjanov: well I didn't know either, now we both know22:04
jeblairmy guess is 5-10 minutes on that jenkins job22:05
fungithat may be enough time for the ansible run to complete too if we're lucky22:05
anteayawell we have to wait for puppetmaster anyway22:05
anteayayup22:05
jeblairright, otherwise, i would say don't wait for it22:05
*** abregman has quit IRC22:05
SergeyLukjanovyup22:05
anteayawonderful22:05
anteayait's all working out22:05
openstackgerritgreghaynes proposed openstack/diskimage-builder: Allow setting per-release mirrors for tests  https://review.openstack.org/27969122:06
jeblairansipuppet is running cacti22:06
jeblairthat's usually the last (it's very slow)22:07
anteayapabelanger: to answer your question about mordred's rename ansible playbook, it isn't ready22:07
nibalizerSergeyLukjanov: fwiw we only needed to comment out the last cron not all of them22:07
*** thorst has joined #openstack-infra22:07
anteayapabelanger: it was to be ready for the large stackforge rename and it wasn't so now I don't worry about it22:07
nibalizerbut its not hurting anything22:07
openstackstatusnibalizer: finished sending alert22:07
pabelangeranteaya: right, I was curious if it was ever tested. Sounds like not yet22:07
*** ccrouch has quit IRC22:08
SergeyLukjanovnibalizer good point22:08
nibalizerwe could also change tto at to touching the lockfile22:08
anteayapabelanger: I have not seen or heard of it being tested22:08
*** matrohon has quit IRC22:09
jeblairexpect cacti to finish at 22:1122:09
*** mriedem has quit IRC22:09
* anteaya will expect that22:09
jeblairi think at this point in the process it would probably be safe for us to proceed22:09
*** amitgandhinz has quit IRC22:09
jeblairwe have passed all of the gerrit/git hosts22:09
anteayashall we proceed?22:10
jeblairi'll let someone else make the call on going or waiting22:10
clarkbjeblair: just be careful puppet doesn't restart zuul on you22:10
nibalizeragree22:10
*** jsavak has quit IRC22:10
nibalizerit shouldn't22:10
jeblairclarkb: i think we're past zuul too (and hopefully it doesn't enforce running)22:10
fungiyeah, zuul service is not enable=>running22:10
nibalizeryep22:10
*** bkero has quit IRC22:11
nibalizerwe've been over /that/ a few times :)22:11
fungior wasn't, hopefully that hasn't somehow gotten "fixed"22:11
jeblairoh good, job is finishing, let's let it upload logs and report22:11
anteayaI think we are safe to take gerrit offline?22:11
*** thorst has quit IRC22:11
anteayaand for pleia2 to run step 4.5?22:11
fungionce the job reports22:11
anteayavery good22:12
fungiand after zuul's queues are dumped and it's stopped22:12
*** esikachev has quit IRC22:12
anteayaokay sorry for jumping ahead22:12
jeblairansible is finished22:12
anteayayay!22:12
jeblairwaiting on zuul22:13
SergeyLukjanovand job is finished22:13
*** thorst has joined #openstack-infra22:13
SergeyLukjanov+ CR merged22:13
anteayaSergeyLukjanov: which CR merged?22:13
SergeyLukjanovack ansible finished22:13
SergeyLukjanovanteaya the one we were waiting for (neutron job)22:13
anteayaoh okay thanks22:14
SergeyLukjanovseems like we could start now, any objections?22:14
anteayawaiting on zuul22:14
jeblairi'll stop zuul now22:14
SergeyLukjanovcool22:14
jeblairzuul is stopped22:14
anteayawonderful22:14
pleia2ok, stopping gerrit22:14
*** _nadya_ has joined #openstack-infra22:15
*** annegentle_ has quit IRC22:15
pleia2ok, gerrit is stopped, and confirmed stopped22:16
* clarkb does github now22:16
anteayawonderful22:16
pleia2going to run the script to get rid of that group22:16
*** pradk has quit IRC22:16
SergeyLukjanovI'm going to run #6 & #822:16
*** zeih has joined #openstack-infra22:16
*** thorst has quit IRC22:17
pleia2done, moving to step 522:17
clarkbgithub things are done22:18
* slogan621 cheers22:18
anteayaclarkb: awesome22:18
*** jpr has quit IRC22:18
pleia2having some mysql syntax problems, working to clean up22:18
anteayapleia2: do you want to paste?22:18
anteayawere the commands wrong?22:19
*** thorst has joined #openstack-infra22:19
*** angdraug has joined #openstack-infra22:20
pleia2anteaya: stray spaces are bad :)22:20
anteayathere were stray spaces in the commands, I'm sorry :(22:20
*** _nadya_ has quit IRC22:20
pleia25 is done22:20
*** redixin has joined #openstack-infra22:20
SergeyLukjanov#6 is done22:21
SergeyLukjanovre-indexing22:21
pleia2anteaya: I reviewed it too and didn't catch them, we're in this together22:21
SergeyLukjanov#8 done22:21
*** vgridnev has quit IRC22:21
SergeyLukjanovheh, we already have 910 projects :)22:21
SergeyLukjanovI think, when I've joined there were less then 10022:22
anteayapleia2: tell me where they were after we're done, so I can review better next time22:22
*** ccrouch has joined #openstack-infra22:22
*** zeih has quit IRC22:22
SergeyLukjanovanteaya +1, I've been lookoing on it as well and missed22:22
pleia2anteaya: I don't remember :\ I just eyeballed the etherpad and cleaned up where I saw 2 spaces rather than 122:22
nibalizerwell gerrit is down, or I would run ls-projects :P22:22
anteayapleia2: okay, sorry about that22:22
anteayaSergeyLukjanov: do share the reindex update progress bar status22:23
jeblairthe mergers do not want to restart (i can't blame them, they had been up for many, many months).22:23
fungitwo spaces between terms in an sql query is a syntax error?22:23
jeblairIOError: [Errno 2] No such file or directory: '/usr/lib/python2.7/dist-packages/lockfile-0.8.egg-info'22:23
* fungi did not know this22:23
anteayajeblair: :(22:24
*** thorst has quit IRC22:24
pleia2fungi: http://paste.openstack.org/show/486897/22:24
pleia22nd one worked22:24
*** thorst has joined #openstack-infra22:24
nibalizerfungi: huh22:24
fungiwow22:24
fungiwhat a crazy strict tokenizer22:24
pleia2indeed22:25
anteayabetween where and project-name22:25
pleia2a space, what do I do |o|22:25
*** mtanino has quit IRC22:25
fungithis must be why i never wanted to become a dba22:25
pleia2hehe22:25
*** annegentle_ has joined #openstack-infra22:25
fungijeblair: all the zuul mergers are doing that?22:25
jeblairfungi: yeah22:25
SergeyLukjanov7%22:25
jeblairis this a pip8 thing or something?22:25
*** mtanino has joined #openstack-infra22:25
jeblairpip says 7.1.222:26
*** eliqiao_ has quit IRC22:26
nibalizeryoure issuing a service zuul-merger start ?22:26
anteayaSergeyLukjanov: thanks22:26
fungii'm guessing pylockfile is pbrized?22:26
SergeyLukjanov>> reindex progress: 10%22:26
jeblairnibalizer: yeah.  also pip install -U . in /opt/zuul similarly complains.22:27
*** eliqiao_ has joined #openstack-infra22:27
fungihuh. there is no 0.8 tag at http://git.openstack.org/cgit/openstack/pylockfile so must be prior to our takeover22:27
SergeyLukjanovoh22:27
anteayaSergeyLukjanov: ?22:28
jeblairokay, i will rm the relevant dead symlinks, and pip install -U /opt/zuul22:28
SergeyLukjanovanteaya surprised re tag :)22:28
anteayaSergeyLukjanov: ah okay22:28
SergeyLukjanov>> reindex progress: 17%22:28
fungithe pylockfile got log is amusing. there's a commit with subject line "*argh*"22:28
fungithey also had commits with a commit message of "."22:29
fungi(git says we have to provide a commit message!)22:29
anteayamust be from the castle argh22:29
*** thorst has quit IRC22:29
*** krtaylor has quit IRC22:29
jeblairlockfile-0.8.egg-info paramiko-1.7.7.1.egg-info pycrypto-2.4.1.egg-info PyYAML-3.10.egg-info22:29
jeblairare the affected packages22:29
slogan621could be they meant **argv and just typed it wrong22:29
SergeyLukjanovdue to the impl of reindex it seems like to use the number of threads equal to nu,ber of cores, but I think we could try using [1.5*cores] for it safely22:29
jeblairslogan621: perhaps they were dictating?22:29
* slogan621 posts bad C humor22:29
fungiheh22:30
*** akshai has quit IRC22:30
fungipython humor meets python humor22:30
*** annegentle_ has quit IRC22:30
SergeyLukjanov>> reindex progress: 22% (63002/277491)22:30
nibalizerfungi: a line with just '.' followed by a newline, right :)22:30
SergeyLukjanov:)22:30
nibalizerthe move from rcs -> git is a rough one22:30
*** rhallisey has quit IRC22:30
funginibalizer: yeah, commits like http://git.openstack.org/cgit/openstack/pylockfile/commit/?id=d4f139ce62f39244f899deb6db8ae65d25cfe8fb22:31
nibalizerhahahahaha22:31
anteayanice22:31
anteayavery helpful22:31
fungiso many, many months seems like we're probably not going to find logs still remaining with the pip failures from whenever this broke22:31
nibalizerjeblair: worked?22:31
jeblairnibalizer: yep.  mergers are all repaired and running now22:31
slogan621well, to be fair, the change is highly readable :-|22:32
nibalizerjeblair: excellent22:32
SergeyLukjanov>> reindex progress: 30%22:32
slogan621if they space indented the '.' characters, doing a git log | xargs grep ^commit could be quite entertaining22:32
jeblairfungi: yeah.  i want to say i vaguely remember this from zuul/nodepool actual.  hopefully we fixed it there. :)22:32
SergeyLukjanovthere are two exception while running reindex, both because of MissingObjectException: Missing blob e1b60551d607142580d962122399613060afde6922:33
anteayaSergeyLukjanov: can you paste?22:33
pleia2pretty sure that's normal22:33
SergeyLukjanovhttp://paste.openstack.org/show/486898/22:34
anteayathanks22:34
SergeyLukjanovyeah, it should be okay22:34
SergeyLukjanovI will check this ref when we'll enable back gerrit22:34
SergeyLukjanov>> reindex progress: 36%22:34
fungiyeah, i want to say we saw it on previous reindexes as well22:35
SergeyLukjanovyeah, I'm 99.999% sure I saw it last time participated renaming22:35
fungii suppose next rename we can try out the online reindex22:35
notmorganclose 222:35
SergeyLukjanov(heh, SLA five nines :) )22:35
zaroThat's a known problem. Just ignore it22:35
*** Sukhdev has quit IRC22:35
anteayafungi: yeah we didn't this time because zaro wasn't here at decision time22:35
SergeyLukjanovfungi yeah +1, we'll have the exec time for it22:35
anteayahey zaro22:35
jeblairfungi: yeah, SergeyLukjanov wanted to reduce chances of surprise22:36
anteayayeah that22:36
fungino argument from me22:36
fungii'm a fan of boring maintenance windows22:36
anteaya++22:36
jeblairbut we definitely should online-reindex next time22:36
fungiand/or try out the ansible playbook for project renames22:36
anteayais it ready?22:37
SergeyLukjanovfolks, I'll be happy to help with such stuff like project renames, sorry for being non-active last time... ping me if I can help with such kind of stuff...22:37
jeblairokay, nodepool list should _definitely_ not connect to zuul.22:37
anteayaI have been operating from the belief that it is not22:37
fungianteaya: our documentation claims it's our process, which i take exception to but at least it's documented22:37
anteayaSergeyLukjanov: always nice to have your help :)22:37
jeblairis anyone working on removing the extraneous network connections from nodepool commands?22:37
SergeyLukjanov>> reindex progress: 41%22:37
anteayafungi: interesting22:37
zaroSerge:  you can find that exception in gerrit ml.22:38
anteayajeblair: I am not22:38
SergeyLukjanovzaro ack, thx22:38
jeblairbecause they've just gone from being minorly annoying to a regression in our ability to admin nodepool22:38
fungijeblair: i thought that had already been cleaned up but i guess not. greghaynes may have been working on that since it was related to the image builder workers?22:38
*** jcoufal has joined #openstack-infra22:38
*** DericHorn-HP has joined #openstack-infra22:38
greghayneshrm?22:39
greghaynesI don't know of any nodepool work on the commands, no22:39
jeblairgreghaynes: 'nodepool list' and 'nodepool delete' now connect to gearman22:39
fungipresumably all nodepool cli subcommands do whether they need to introspect the gearman queue or not22:39
greghaynesI don't know what would have caused that behavior to change22:40
anteaya20 minutes remaining in our scheduled maintenance window22:40
anteayaSergeyLukjanov: any thoughts if we will make it in time?22:40
clarkbshould be trivial to not gearman on those22:40
jeblairgreghaynes: i expect it's due to some of the recent config/setup moves22:40
fungiright, we likely just need a list of which subcommands need info from gearman and then switch on that22:40
SergeyLukjanov>> reindex progress: 50%22:41
anteayaSergeyLukjanov: that's promising22:41
SergeyLukjanovI'm trying to calc :_22:41
greghaynesjeblair: Yep - there is an explicit call to reconfigureGearmanClients in main() now22:41
anteayaSergeyLukjanov: well the last 1% takes the longest22:41
SergeyLukjanovso, it should take ~20 mins to complete22:41
* greghaynes is just rereading the code22:41
*** achanda has quit IRC22:41
anteayaSergeyLukjanov: we'll be close22:41
greghaynesjeblair: That should be easy enough to check against which command is being run22:42
fungiSergeyLukjanov: the percentages progress at a nondeterministic rate, so i wouldn't put much stock in it22:42
SergeyLukjanovfungi it's mostly about the CRs count and the number is big enough, I think it's more or less deterministic22:42
*** thorst has joined #openstack-infra22:42
anteayaSergeyLukjanov: no it isn't, it is a known issue22:43
*** edmondsw has quit IRC22:43
anteayathe biggest project takes the longest and is considered 1%22:43
fungiyeah, zaro had some fun anecdotes about the progress reindex counter22:43
fungier, reindex progress counter22:43
anteayaso you will hang at 99% until nova finishes reindexing22:43
*** changbl has joined #openstack-infra22:44
* anteaya preheats the oven for dinner22:45
jeblairso i'll just do 'update node set state=4'22:45
greghaynesI have a patch to do that once gerrit is back (dont connect to gearman unless doing builder commands)22:45
jeblairor maybe ' where state>2'22:45
jeblairmysql> update node set state=4 where state>2;22:45
jeblairQuery OK, 162 rows affected (0.00 sec)22:45
jeblairoh wow the subnode table is a mess22:46
anteaya:(22:46
jeblairi think something is leaking there22:46
jeblairlots of stuff in building state22:46
jeblairmysql> update subnode set state=4 where state=1;22:46
jeblairQuery OK, 3505 rows affected (0.01 sec)22:46
*** thorst has quit IRC22:47
jeblairthat's going to be some extra api calls.22:47
SergeyLukjanovI'm posting the CRs persentage, not projects, so, I think it's more cleat (maybe I still wrong :) )22:47
jeblairanyway, nodepool can work on those while we wait22:47
SergeyLukjanov>> reindex progress: 60%22:47
*** yamamoto_ has joined #openstack-infra22:48
SergeyLukjanovyeah, seems like I'm wrong, it's progressing slowly right now22:48
anteayalooks like we won't exactly make 23:00 utc22:48
SergeyLukjanovprobably it's time to evaluate could we start gerrit while reindexing or add up to 30 mins to the mainteinance window22:49
anteayaupdate the topic or no?22:49
pleia2it'll be close enough22:49
pleia2computers, eh? :)22:49
anteayazaro: any input on starting gerrit while reindexing?22:49
jeblairi do not think we should do that22:50
fungii believe the online reindex is a different procedure?22:50
jeblairthe online reindexing is an internal gerrit command22:50
fungibetter to just let the window drag out a little and apologize later22:50
jeblairso it's different than just running a reindex while gerrit is running22:50
SergeyLukjanovagree with extending window22:50
anteayaokay22:50
* mordred returns to find massive fun has occurred22:51
anteayahey there22:51
SergeyLukjanovgerrit index start <- for online re-index22:51
jeblairSergeyLukjanov: yep22:51
SergeyLukjanov>> reindex progress: 64%22:51
*** _amrith_ is now known as amrith22:52
SergeyLukjanovsounds like we definitely need to switch to online re-indexing22:52
pleia2yeah, we knew that, one of the exciting 2.11 things we were looking forward to22:52
*** tzn has joined #openstack-infra22:52
*** yamamoto_ has quit IRC22:52
SergeyLukjanovI think we can try running online re-index overnight on the weekend (sure, it's a bad idea if it'll go down)22:53
SergeyLukjanov67%22:53
*** baoli_ has quit IRC22:54
fungioffline reindexing has gotten a lot slower in 2.11 than it was in 2.822:54
*** apoorvad has joined #openstack-infra22:54
fungibefore the upgrade it was taking ~15 minutes22:54
*** baoli has joined #openstack-infra22:54
SergeyLukjanovyeah, I remember it22:55
SergeyLukjanovit's already working for ~35 mins22:55
fungion track for being 3-4x slower22:56
anteayamore is better, right?22:56
pleia2more slower?22:56
anteayamore anything22:56
anteayamore22:56
pleia2hehe22:56
anteayathat's what I hear22:56
pleia2it's less fast :(22:56
*** tzn has quit IRC22:57
anteayait is22:57
fungiit's better at being slower22:58
*** apoorvad has quit IRC22:59
anteayamuch better22:59
SergeyLukjanov73%23:00
nibalizeryep23:00
nibalizerupdate the topics you think?23:00
pleia2I used to call things slow as a platypus, but it turns out they are very fast swimmers, maybe on land they are slow, but I have not done a platypus land experiment23:00
SergeyLukjanovyeah, I think we should update topic23:01
nibalizerpleia2: did you get to see one in AU?23:01
pleia2next LCA23:01
pleia2nibalizer: I did!23:01
nibalizerawww23:01
pleia2at the zoo, it was swimming :)23:01
nibalizerso cool23:01
jeblairnibalizer, SergeyLukjanov: you can issue another "#status alert" and it will dtrt23:01
fungioh, did that get fixed?23:02
SergeyLukjanovjeblair I'll do23:02
jeblairfungi: when was it broken?23:02
SergeyLukjanovlet's say +30 mins23:02
fungithough the last time we tried that the #status ok reverted back to the first alert rather than the original topics23:02
anteayaSergeyLukjanov: ++23:02
SergeyLukjanovjeblair I was thinking it's not possible too :)23:02
fungis/though/thought/23:02
fungibut i can't remember how long ago that was now23:02
nibalizerok23:03
jeblairfungi: that would happen if it restarted23:03
SergeyLukjanov#status alert Gerrit is offline for maintenance, ETA updated to 23:30 utc23:03
openstackstatusSergeyLukjanov: sending alert23:03
fungii guess we'll find out!23:03
SergeyLukjanov79%23:03
*** baoli has quit IRC23:03
SergeyLukjanov80%23:03
jeblairfungi: oh -- the other thing is don't do it while it's sending a topic update23:04
jeblairthat might be the breakage you're thinking of23:04
fungijeblair: oh! found the details. it was still in the process of sending an alert when we told it to send an updated one23:04
fungiyep, that was it23:05
jeblairso if you typo -- live with it :)23:05
fungifor example, if you have terrible mainten acne23:05
-openstackstatus- NOTICE: Gerrit is offline for maintenance, ETA updated to 23:30 utc23:05
*** ChanServ changes topic to "Gerrit is offline for maintenance, ETA updated to 23:30 utc"23:05
nibalizerits a common openstack issue23:05
anteayawe lived with it, we survived23:05
pleia2good thing for us, the non-native speakers know how to spell23:05
pleia2(thanks SergeyLukjanov!)23:06
nibalizerindeed23:06
*** HeOS has quit IRC23:07
SergeyLukjanovI've checked 3 times23:07
SergeyLukjanov88%23:07
*** lucas-afk is now known as lucasagomes23:08
openstackstatusSergeyLukjanov: finished sending alert23:08
SergeyLukjanov90%23:08
*** andymaier has joined #openstack-infra23:09
SergeyLukjanov93%23:10
EmilienM /pup23:10
EmilienM(sorry) and hello23:11
pleia2EmilienM is trying to run puppet on us23:11
anteayaI don't puppet well23:11
anteayasorry23:11
EmilienMdamn weechat alias I still don't masterize after 6 months23:11
anteayaand hello EmilienM23:11
pleia2hehe23:11
* EmilienM watching infra folks with pop corn23:11
anteayaI'm about to have pizza23:11
anteayagoes well with popcorn23:11
*** jcoufal has quit IRC23:12
*** nikhil has joined #openstack-infra23:12
*** Sukhdev has joined #openstack-infra23:12
SergeyLukjanov94%23:13
*** nikhil has quit IRC23:13
fungiif you like to watch paint dry, you may also enjoy gerrit offline reindexing23:14
jeblairi have some paint that takes 2 months to dry.23:15
*** nikhil has joined #openstack-infra23:15
*** piet_ has joined #openstack-infra23:16
*** _nadya_ has joined #openstack-infra23:16
fungicasein paint?23:17
fungioh, oils23:17
SergeyLukjanov95%23:17
*** baoli has joined #openstack-infra23:18
nibalizergerrit is offline, but the jokes are online23:18
EmilienMgerrit is down, but level of joke is high23:18
*** zeih has joined #openstack-infra23:19
jeblairfungi: yeh, though, actually waterborne alkyds23:19
jeblairso 'oil' without the oil23:19
fungioh, neat!23:20
*** _nadya_ has quit IRC23:21
sigmavirus24lifeless: I can't send a review because gerrit is offline, but if you could look at https://github.com/sigmavirus24/pbr/commit/8e996fbd2a80bae4ed15e5a82ad80f9ee460f8cd I'd appreciate it. I know it needs tests, but I want to know if you're okay with where I'm shimming that functionality in23:21
*** krtaylor has joined #openstack-infra23:21
SergeyLukjanov96%23:21
*** sigmavirus24 is now known as sigmavirus24_awa23:22
*** baoli has quit IRC23:23
*** DericHorn-HP has quit IRC23:23
*** zeih has quit IRC23:23
*** leakypipes has quit IRC23:23
clarkbjeblair: greghaynes gave me a patch diff to fix the nodepool gearman thing, I will push that up once gerrit is back (he had to run)23:28
jeblairclarkb, greghaynes: cool thanks!23:29
SergeyLukjanovstill 96%23:31
*** yamamoto_ has joined #openstack-infra23:33
*** baoli has joined #openstack-infra23:33
*** achanda has joined #openstack-infra23:36
SergeyLukjanov97%23:36
mordredsigmavirus24_awa: I do not have strong feelings on the location - but I support the goal of your patch23:37
*** yamamoto_ has quit IRC23:37
*** piet_ has quit IRC23:37
*** lucasagomes is now known as lucas-afk23:37
*** roxanaghe has joined #openstack-infra23:37
SergeyLukjanov#status alert Gerrit is offline for maintenance, ETA updated to 23:59 utc23:38
openstackstatusSergeyLukjanov: sending alert23:38
*** baoli has quit IRC23:38
anteayain the area of good news, I was able to eat my dinner uninterrupted!23:40
anteayayay23:40
-openstackstatus- NOTICE: Gerrit is offline for maintenance, ETA updated to 23:59 utc23:40
*** ChanServ changes topic to "Gerrit is offline for maintenance, ETA updated to 23:59 utc"23:40
mordredanteaya: it's not the same thing as uninterrupted dinner, but I can report that the cat it merrily frolicking in the back yard right now23:41
clarkbI can report that baby is currently undergoing physical therapy and protesting every minute23:41
*** markvoelker has quit IRC23:41
fungii was watching the neighbor's puppy running around crazily in the snow from my office window... at least until the sun finally went down23:41
mordredfungi: oh - you have snow23:42
fungia few inches so far23:42
mordredfungi: I have 21 degrees, sun and am sitting by the pool23:42
* slogan621 reports it is 91 degrees where he is23:42
slogan621snow?23:42
fungii'll get some photos of snow on the dock tomorrow23:42
*** EricGonczer_ has joined #openstack-infra23:42
anteayamordred: awesome23:43
anteayaclarkb: :(23:43
jeblairi suspect you don't all have the same degrees23:43
fungiwe've also got a nice accumulation of brackish ice sickles on the pilings23:43
mordredjeblair: I report degrees in units that make logical sense23:43
SergeyLukjanov21 degrees (C)23:43
openstackstatusSergeyLukjanov: finished sending alert23:43
anteayafungi: yay puppies23:43
slogan62191 (F)23:43
mordredSergeyLukjanov: ++23:43
mordredslogan621: your degrees represent a temperature state I would prefer to mine23:44
fungidifferent degrees latitude too, no doubt23:44
mordredbut I prefer both of them to the temperature where fungi is, whatever it happens to be23:44
*** salv-orlando has joined #openstack-infra23:44
jeblair22C here23:44
* slogan621 is in San Diego, CA23:44
mordred33.0383° N23:44
fungi~0c right now, but dropping quickly since the sun's been down for a little while23:44
*** armax has quit IRC23:44
mordredjeblair: you're beating me by a degree23:44
*** armax has joined #openstack-infra23:44
SergeyLukjanovI've been skiing on Tahoe last weekend and it was 10-14 degrees Celsius, super sunny and tons of snow23:45
SergeyLukjanov:)23:45
anteayaSergeyLukjanov: nice23:45
fungi36.0347° N here23:45
anteayaha ha ha23:45
slogan621SergeyLukjanov: what drought, right?23:45
SergeyLukjanovbtw, I've been looking on our cloud DBs on RAX and gerrit DB is 8 out 10 GBs23:45
*** tmcpeak has joined #openstack-infra23:45
mordredSergeyLukjanov: mmmm23:46
SergeyLukjanovslogan621, not really23:46
*** tmcpeak has left #openstack-infra23:46
SergeyLukjanovhttps://www.instagram.com/p/BBc-6RmNUbP/ :)23:46
fungiluckily trove databases are easy to resize. we should probably plan to grow it during the next mainten acne (if trove online resize isn't a thing)23:46
slogan621nice23:47
SergeyLukjanovfungi etherpad db is now showing as resizing ;)23:47
mordredfungi: I'm actually going to write up some thoughts on maybe doing something different with those23:47
fungiSergeyLukjanov: that's an "old" etherpad database instance which is stuck for many months now23:47
fungiit's not in use23:47
mordredfungi: our db dumps are starting to provide noticable service interruptions23:47
SergeyLukjanovfungi good ;)23:48
EmilienMfungi: we use trove for mysql databases? nice. How is it deployed?23:48
clarkbEmilienM: by rackspace23:48
mordredfungi: so I'm thinking MAYBE it's time to take advantage of having some legit mysql ops exerpience around here to do something23:48
SergeyLukjanovbtw, still 97%, that's really sad23:48
anteayaSergeyLukjanov: yup23:48
anteayaSergeyLukjanov: sad but true23:48
fungimordred: oh, yes please let's consider replication and backing up a quiescent replica or something23:48
*** achanda has quit IRC23:48
mordredyup23:48
EmilienMclarkb: ok thanks23:48
jeblair++23:49
fungii've happily used that model in $oldjob to great effect on much larger, much more active databases. though thankfully i did not have to configure them myself23:49
*** kzaitsev_mb has joined #openstack-infra23:50
anteayamordred: so you have a backyard23:50
jeblairfungi: if we play our cards right, we still may not have to!23:50
fungiha!23:50
fungitouché23:50
fungialso i'm starting to remember now that the offline lucene reindex after our upgrade to gerrit 2.11 took on the order of two hours. wishing i had remembered that when we were planning this window :/23:51
*** gangil has joined #openstack-infra23:51
mordredanteaya: oh yes23:52
SergeyLukjanov7k CRs left to re-index23:52
anteayamordred: tell me about it23:52
anteayamordred: we have a few minutes23:52
anteayafungi: :(23:53
*** salv-orlando has quit IRC23:53
mordredanteaya: it's a lovely backyard with a bunch of trees that serve as a sort of tall-ish privacy hedge23:54
anteayanice23:54
mordredanteaya: there is also a covered patio with ceiling fan and a pool - although it is not warm enough for me to have entered the pool yet23:54
anteayayou have privacy?23:54
mordredanteaya: yup23:54
anteayawow23:54
anteayamust be a big change from NY23:55
*** apoorvad has joined #openstack-infra23:55
mordredanteaya: there are many things about this place that are a big change from NY23:55
*** annegentle_ has joined #openstack-infra23:55
anteayalike space I hear23:55
mordredyup. plenty of that23:55
mordredthere are also more airplanes23:56
anteayaokay that would make sense23:56
anteayacan you see them from your house?23:56
mordredI mean, we're near the DFW airport23:56
mordredthere are a non-zero number of times during the day when one can see airplanes overhead23:57
anteayafun23:57
anteayaespecially if you like airplanes23:57
mordredyes23:57
anteayaI understand you like airplanes23:57
anteayaand DFW is actually a really good airport23:57
mordredour friend ben lives in this neighborhood, which is why we chose it23:57
mordredand his partner likes airplanes a LOT23:58
mordredas in he enjoys going to airports to take pictures of planes taking off and landing23:58
fungii like the bbq joint in the airport23:58
mordredso the plane traffic to him was a big selling point on their house23:58
* NobodyCam really likes to watch the planes take off and land at PSP23:58
anteayawow23:58
anteayaha ha ha23:58
fungiunfortunately, the inside of that airport is the _only_ think i've seen in texas so far23:58
mordredNobodyCam: I have never been to PSP23:58
fungier, thing23:58
anteayajust as a data point, plane traffic would not be a selling point for me purchasing property23:59
NobodyCamoh it is a awesome (yet small) airport23:59
mordredfungi: we can have a meeting in my back yard one of these days23:59
anteayain case anyone was unclear on that23:59
mordredanteaya: I would have been shocked if it was23:59
anteayamordred: good, you have the correct impression23:59
mordredit's also not a selling point for me, but the amount is within my tolerance23:59
fungimordred: count me in23:59
anteayamordred: glad to hear it23:59
clarkbboeing field and everett are good for plane watching23:59
*** apoorvad has quit IRC23:59
clarkbpaine field?23:59
mordredclarkb: ++23:59
« Thursday, 2016-02-11 Index Saturday, 2016-02-13 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!