Thursday, 2022-04-28

*** amoralej|off is now known as amoralej06:34
elodillesgood morning o/ fyi, i see lot of tox-docs job failure in zuul (nova + in several other projects) which seems to be a result of the latest oslo.policy release from yesterday. I'll try to look into it, but first I've pinged the oslo team on #openstack-oslo06:42
gibi^^ fix is up and I checked locally it works for the nova jobs https://review.opendev.org/c/openstack/oslo.policy/+/83971109:51
bauzasthanks gibi10:02
bauzas(sorry was on and off)10:03
sean-k-mooneygibi: when you get a chance can you look at the comments i left on https://review.opendev.org/c/openstack/nova/+/83855510:08
gibisean-k-mooney: looking...10:41
gibiand pulling the patch down...10:46
gibisean-k-mooney: hm, the test passing locally for me even after a rebase10:50
opendevreviewBalazs Gibizer proposed openstack/nova stable/train: Remove unavailable but not reported PCI devices at startup  https://review.opendev.org/c/openstack/nova/+/83971710:50
opendevreviewBalazs Gibizer proposed openstack/nova stable/train: Simulate bug 1969496  https://review.opendev.org/c/openstack/nova/+/83971810:51
opendevreviewBalazs Gibizer proposed openstack/nova stable/train: Allow claiming PCI PF if child VF is unavailable  https://review.opendev.org/c/openstack/nova/+/83971910:51
gibilet see if after the rebase it will be clean10:51
gibiif not then something is interfeering between test cases10:51
gibihm10:56
gibinow gerrit says it is in merge conflict10:56
gibibah I pushed it against stein :D10:57
gibi*train10:57
sean-k-mooneyhehe10:57
opendevreviewBalazs Gibizer proposed openstack/nova master: Remove unavailable but not reported PCI devices at startup  https://review.opendev.org/c/openstack/nova/+/83855310:58
opendevreviewBalazs Gibizer proposed openstack/nova master: Simulate bug 1969496  https://review.opendev.org/c/openstack/nova/+/83855410:58
opendevreviewBalazs Gibizer proposed openstack/nova master: Allow claiming PCI PF if child VF is unavailable  https://review.opendev.org/c/openstack/nova/+/83855510:58
gibiI blame it for the lack of coffein10:58
sean-k-mooneygibi: sorry was pinned for something else runing it locally but ill readd my +2 and review the final patch when it completes11:39
gibino worries, we have a block in the docs job until https://review.opendev.org/c/openstack/oslo.policy/+/839711 merged and released11:40
* sean-k-mooney needs to install openssl dev packages11:40
sean-k-mooneyi breifly looked at that but did not get the full context11:41
sean-k-mooneyso oslo-config-generator chagne the kw paramater to a flag?11:42
sean-k-mooneyremoving the boolean paramater 11:42
sean-k-mooneyand this is adapting to that backwards incompatible change correct11:42
sean-k-mooneythat shoudl have been a major version bump11:44
sean-k-mooneyas in oslo.policy=4.0.0 not 3.12.011:44
sean-k-mooneygibi: nova.tests.unit.pci.test_manager.PciDevTrackerTestCase.test_set_hvdevs_unavailable_vf_removed is failing for me locally11:45
sean-k-mooney File "/home/sean/repos/openstack/nova-3/nova/objects/pci_device.py", line 238, in _from_db_object11:45
sean-k-mooney    setattr(pci_device, key, db_dev[key])11:45
sean-k-mooney    KeyError: 'id'11:45
sean-k-mooneyits form the create call11:46
sean-k-mooney  File "/home/sean/repos/openstack/nova-3/nova/tests/unit/pci/test_manager.py", line 413, in test_set_hvdevs_unavailable_vf_removed11:46
sean-k-mooney    self._create_tracker([fake_db_dev_3, fake_db_dev_4, fake_db_dev_5])11:46
gibilooking11:47
gibistrange, it is passing for me11:49
gibido you have commit hash f395e71168 ?11:49
sean-k-mooneyf395e71168d843f06c8de7b04874c29f1e10e5a811:58
sean-k-mooneyill run it again11:58
sean-k-mooneywith -r11:58
sean-k-mooneyi think it was a clean venv but we will see if it repoduces11:59
sean-k-mooneyoh odd12:00
sean-k-mooneyhttps://zuul.opendev.org/t/openstack/build/f877f8e9afa04b5bb993d6615b8bd55812:00
sean-k-mooneyit failed on the 3.8 arm job12:00
sean-k-mooneybut passed on 3.912:00
gibihm I use 3.9 locally12:00
sean-k-mooneylet me check which version im using i have 3.8-3.10 locally12:01
sean-k-mooneyim proably useing 3.812:01
gibiok it fails iwith 3.8 locally for me too12:01
sean-k-mooneythat is super weird12:01
gibiyes12:02
sean-k-mooneythis is not in code you are changin i think it appear to be in the fixture/test setup code12:02
sean-k-mooneyis it this https://review.opendev.org/c/openstack/nova/+/838553/3/nova/tests/unit/pci/test_manager.py#14712:03
gibiyes but now I can trace and compare12:03
sean-k-mooneyid is technically a reserved keywrod for the id function12:03
sean-k-mooneybut when you asign to it as a kwarg it shoudl intoduce it also as a variable12:04
sean-k-mooneyits disucuraged but legal12:04
sean-k-mooneyhum12:05
sean-k-mooneyi wonder if this is realted to not cloning the fake object or soemthing like that12:06
sean-k-mooneyi didnt get the failure this time12:07
sean-k-mooneyoh this is failing on the first patch12:09
sean-k-mooneyoh is it12:09
sean-k-mooneyno third i just have the wrong tab open12:09
sean-k-mooneythe first patch also hass the same issue12:10
gibihm, you have a point about cloning12:10
gibiif the id is dropped by our db code then that is now a global change on the db dict12:11
sean-k-mooneyyep12:11
sean-k-mooneynova.tests.unit.pci.test_manager.PciDevTrackerTestCase.test_set_hvdevs_unavailable_pf_removed is failing in the first patch12:11
sean-k-mooneyyou need to do 12:12
sean-k-mooney        fake_pci_devs = [copy.deepcopy(fake_pci), copy.deepcopy(fake_pci_2),12:12
sean-k-mooney                         copy.deepcopy(fake_pci_3)]12:12
gibibut non of the test does it so probably the rest of test can be broken too12:14
gibiI can add the deepcopy to _fake_get_pci_devices to fix them all12:14
sean-k-mooneysome do12:14
sean-k-mooneylike test_set_hvdev_changed_stal12:14
sean-k-mooneymany do all_devs = fake_db_devs_tree[:]12:15
gibithat is a shallow copy12:15
sean-k-mooneyself._create_tracker(all_devs)12:15
gibionly duplicate the list12:15
sean-k-mooneyit is yes12:15
sean-k-mooneybut i guess they dont currently modify thigns i guess.12:16
gibibut that does not duplicat the dict having the id field12:16
sean-k-mooneywe likely do need to fix other test but i think we have just got lucky 12:16
sean-k-mooneythre are certenly sevel test that explcitly do a deepcopy12:17
sean-k-mooneyi dont think it s the _create_tracker that copl;es the global state12:19
sean-k-mooneyi think its self.tracker._set_hvdevs([fake_db_dev_3, fake_db_dev_4])12:19
sean-k-mooney_create_tracker does not actully pass the fake devs to the tracker12:20
sean-k-mooneyhttps://github.com/openstack/nova/blob/master/nova/tests/unit/pci/test_manager.py#L144-L147=12:21
gibiyeah you are right12:22
gibithe set_hvdev does the coupling12:23
sean-k-mooneyyep so you are not actully initalisign the tracker with the devices you created before using them12:23
*** amoralej is now known as amoralej|lunch12:25
gibiI will push a fix12:26
gibifor the new test cases12:26
gibiand a separate commit for the existing ones not having deepcopy before _set_hvdevs12:27
sean-k-mooneyso i tihnk it was "working" becaue of this stub https://github.com/openstack/nova/blob/5f5551448dcfcde26095963e223f973b90e6f637/nova/tests/unit/pci/test_manager.py#L153-L15412:27
sean-k-mooneyi was just trying to fiture out how the the tracker had any devices when it was not called12:28
sean-k-mooneythat just returns self.fake_devs https://github.com/openstack/nova/blob/5f5551448dcfcde26095963e223f973b90e6f637/nova/tests/unit/pci/test_manager.py#L153-L15412:28
sean-k-mooneyso ya anything not doign a deep copy was sharing state12:29
tridentWe have had a few policy overrides to allow regular users to create flavors. At some point they seem to have stopped working (probably upgrade to Victoria) - Neither a policy.json with the old format or a policy.yaml seem to help.12:32
tridentAny known reasons why a policy override like "os_compute_api:os-flavor-manage:create": "rule:admin_or_owner" (and the same for update and delete) would not work nowadays?12:33
tridentVersion 22.3.212:34
sean-k-mooneyit should be possibel to configure that today12:41
sean-k-mooneynot advised but possible12:42
tridentWeird. I still get a 403 Policy doesn't allow os_compute_api:os-flavor-manage:create to be performed back...12:45
tridentsean-k-mooney: You say "not advised" - what would be the advised way to accomplish that?12:48
sean-k-mooneyyou said it was victoria12:49
sean-k-mooney22.3.212:49
sean-k-mooney22.3.2 does not exist in git12:49
sean-k-mooneyon 2212:49
sean-k-mooneyaslo no does not exist12:50
sean-k-mooneyhttps://github.com/openstack/nova/tree/22.3.2/nova12:50
sean-k-mooneyhttps://github.com/openstack/nova/tree/22.3.0/nova is the most recent release i see12:51
sean-k-mooneysame via opendev https://opendev.org/openstack/nova/src/tag/22.3.012:51
sean-k-mooneyhttps://opendev.org/openstack/nova/src/tag/22.3.0/nova/policies/flavor_manage.py#L22-L5912:52
sean-k-mooneythis looks correct12:52
sean-k-mooneyam you have not turned on scope enforement have you12:53
sean-k-mooneythis requires a system scope token12:53
sean-k-mooneytrident: you cannot use a project scope token if you have scope enfroment enabled12:53
sean-k-mooneyso normal project_member tokens would not work even i fyou udated teh check_str unless you also added 'project' to the scope_types12:54
sean-k-mooneytrident: by the way im not sure admin_or_ower shoudl work in this context12:56
sean-k-mooneytrident: the resouce does not exsit yet so you cant be an ower of it12:56
sean-k-mooneyyou woudl want something likse we use for server create 12:57
sean-k-mooneyhttps://opendev.org/openstack/nova/src/tag/22.3.0/nova/policies/servers.py#L166-L17512:57
sean-k-mooneywhich is just project_member12:57
sean-k-mooneyactully no12:58
sean-k-mooneyhttps://opendev.org/openstack/nova/src/tag/22.3.0/nova/policies/create_backup.py#L24-L3612:58
sean-k-mooneycreat backup is better12:58
sean-k-mooneytrident: you woudl want PROJECT_MEMBER_OR_SYSTEM_ADMIN12:58
sean-k-mooneywith both scope types12:58
sean-k-mooneyagain we dont really recommend doing this but if i was to do that personally i woudl create role and then in the custom polciy string allow peopel with admin or the "create_flavor" role to create flavors12:59
sean-k-mooneythat way you can atleast limit it to a subset of peopel but admin_or_owner shoudl not work for create13:00
tridentHm, yeah, that makes sense - as you say, it doesn't exist, so owner doesn't make sense at the time of creation. I'm however pretty sure those rules have worked previously. Probably at least on train. 13:01
sean-k-mooneyadmin_or_ower was defiend as 'is_admin:True or project_id:%(project_id)s'13:01
sean-k-mooneybut flavors are not part of projects13:01
sean-k-mooneyso that woudl not work in anycase13:01
tridentThanks for the advise! I'll do some testing and see what I end up with. :)13:02
sean-k-mooneyon train it was more or less the same https://opendev.org/openstack/nova/src/tag/train-em/nova/policies/base.py#L27-L3013:02
*** dasm|off is now known as dasm13:10
*** amoralej|lunch is now known as amoralej13:11
gibisean-k-mooney: I can reproduce the keyerror on master too with a bit of change and I think I see the issue.  _create_tracker should take dict that come from the db with id field but _set_hvdevs should take dict that come from the hypervisor (no id field). My new tests (and one existing test) passed db dict to _set_hvdevs causing the db dict to get corrupted13:13
gibihttps://github.com/openstack/nova/blob/028b3bca16c750f6c7edf1b389ed6c79a2c9843d/nova/tests/unit/pci/test_manager.py#L36113:13
sean-k-mooneyya so that can work if its a copy13:13
sean-k-mooneysince it wont affect the gloabl sate but ya13:14
gibiso we need the copy in L361 as that already corrupts the global state13:14
gibieven on master13:14
sean-k-mooney yep13:14
sean-k-mooneywell13:15
sean-k-mooneywe shoudl do the copy on line 34713:15
sean-k-mooneyor we shoudl hvae _create_tracker do the copy internally13:15
sean-k-mooneyand then work on self.fake devs if need13:15
sean-k-mooneybut ya in anycase i wonder why we are not seeign this fail in general13:16
sean-k-mooneytheses test are not flaky in my experince13:16
gibiwe only have one test on master that does the mistake to pass a db dict with id to _sethvdevs. That corrupts the global state but no other test depends on that. Then I added another test that did this mistake and blow if the two test run in the same executor13:18
gibiyou can reproduce the issue by duplicating test_set_hvdev_remove_tree_maintained_with_allocations on master13:18
sean-k-mooneyack ok 13:18
gibiI will try to clean this up13:18
sean-k-mooneyi wonder if we shoudl jsut have _create_tracker deep copy in genreal13:19
sean-k-mooneywe can still explictly do it but that would remove the need to do it in the default case13:20
sean-k-mooneyonly if you call set_hvdevs13:20
sean-k-mooneygibi: glad we caught that before it was merged13:21
sean-k-mooneythat kind of think is a pain to debug in the gate when it only failes ocationally13:21
gibiyes, it was a good catch13:21
gibinobody likes interfeering test cases :)13:21
opendevreviewBalazs Gibizer proposed openstack/nova master: Isolate PCI tracker unit tests  https://review.opendev.org/c/openstack/nova/+/83976613:56
gibisean-k-mooney: ^^ I will base the current series on top of this13:56
opendevreviewBalazs Gibizer proposed openstack/nova master: Isolate PCI tracker unit tests  https://review.opendev.org/c/openstack/nova/+/83976614:02
opendevreviewBalazs Gibizer proposed openstack/nova master: Remove unavailable but not reported PCI devices at startup  https://review.opendev.org/c/openstack/nova/+/83855314:02
opendevreviewBalazs Gibizer proposed openstack/nova master: Simulate bug 1969496  https://review.opendev.org/c/openstack/nova/+/83855414:02
opendevreviewBalazs Gibizer proposed openstack/nova master: Allow claiming PCI PF if child VF is unavailable  https://review.opendev.org/c/openstack/nova/+/83855514:02
sean-k-mooneygibi: ack ok that makes sesne14:13
opendevreviewElod Illes proposed openstack/nova master: [CI] Install dependencies for docs target  https://review.opendev.org/c/openstack/nova/+/83978115:21
elodillesgibi bauzas : another stable gate fix to review ^^^ when you have time o:)15:29
elodilles(today's broken oslo.policy release showed an error in our tox docs target as the job is failing for stable branches due to a release on zed)15:30
*** amoralej is now known as amoralej|off15:36
clarkbelodilles: gibi  bauzas email was sent about that problem a few weeks ago http://lists.openstack.org/pipermail/openstack-discuss/2022-April/028016.html there is a good chance that much of openstack needs that sort of update15:46
gibielodilles, clarkb: thanks I'm +2 on it15:47
sean-k-mooneywhen did that get remvoed15:49
sean-k-mooneywe used to install requiremets.txt15:49
clarkbsean-k-mooney: a while back there was a big push to switch to trimming the doc requirements down so you didn't have to install everything. What that missed was that the doc builds depended on the projects to collect cli command output and such. Basically I think it was docs having their own requirements that introduced the bug15:50
clarkbthe intent was good, but no one realized that this flaw existed15:50
sean-k-mooneyah so it was applied genericly 15:51
sean-k-mooneyi just did not recally this patch going in15:51
sean-k-mooneywe might also need test-requiremetns in some cases but in generaly not15:51
elodillesclarkb: thanks, i'll try to check other projects as i've seen +24 broken stable-periodic tox-docs job today (neutron has already a similar patch on the gate right now)15:53
elodilles(this one: https://review.opendev.org/c/openstack/neutron/+/839777 )15:54
sean-k-mooneyclarkb: so it would b enice to have included the change id of the change that remvoed it but i dont think we shoudl hold this up for that so ill review it now 15:59
clarkbI mean its not my change. I just helped debug a similar problem a few weeks ago and we told everyone about it hoping they would audit and fix their repos15:59
clarkbseems that didn't happen hence the current situation15:59
sean-k-mooneyi never new this happend i must have missed the mail15:59
clarkbit was a huge cross openstack effort to change the doc build system16:02
clarkbit was a while ago so I don't remember the details just that it happened and a lot of stuff got updates16:02
opendevreviewMerged openstack/nova master: [CI] Install dependencies for docs target  https://review.opendev.org/c/openstack/nova/+/83978117:57
opendevreviewBalazs Gibizer proposed openstack/nova stable/ussuri: Reproduce bug 1953359  https://review.opendev.org/c/openstack/nova/+/82204718:01
opendevreviewBalazs Gibizer proposed openstack/nova stable/ussuri: Extend the reproducer for 1953359 and 1952915  https://review.opendev.org/c/openstack/nova/+/82204818:01
opendevreviewBalazs Gibizer proposed openstack/nova stable/ussuri: [rt] Apply migration context for incoming migrations  https://review.opendev.org/c/openstack/nova/+/82205018:01
opendevreviewElod Illes proposed openstack/nova stable/yoga: [CI] Install dependencies for docs target  https://review.opendev.org/c/openstack/nova/+/83980918:17
opendevreviewElod Illes proposed openstack/nova stable/xena: [CI] Install dependencies for docs target  https://review.opendev.org/c/openstack/nova/+/83981018:22
opendevreviewElod Illes proposed openstack/nova stable/wallaby: [CI] Install dependencies for docs target  https://review.opendev.org/c/openstack/nova/+/83981118:23
opendevreviewElod Illes proposed openstack/nova stable/victoria: [CI] Install dependencies for docs target  https://review.opendev.org/c/openstack/nova/+/83981218:25
opendevreviewElod Illes proposed openstack/nova stable/ussuri: [CI] Install dependencies for docs target  https://review.opendev.org/c/openstack/nova/+/83981318:26
melwittdansmith: yoga fix for docs job is ready https://review.opendev.org/c/openstack/nova/+/83980919:06
dansmithmelwitt: I'm going to go out on a limb and say it'd be okay for you to slam those mofos in :)19:07
dansmithyou know, IMHO :D19:07
sean-k-mooneyi certenly would not object19:08
melwitthaha ok19:11
*** artom__ is now known as artom19:32
*** dasm is now known as dasm|off20:32
opendevreviewDan Smith proposed openstack/nova master: DNM: Run against performance.json patch  https://review.opendev.org/c/openstack/nova/+/83893421:37
dansmithclarkb: around?22:11
clarkbdansmith: hi22:15
dansmithnot even a full devstack run and 57k queries to the keystone db.. seems high, no?22:15
clarkbdansmith: that does seem high. But openstackclient does have to get a new token for everything since there is no token caching22:16
clarkbperhaps related to that?22:16
dansmithstill, 57k22:16
dansmithalso, they're almost all select22:16
dansmithdon't we have to insert when we create a token?22:16
clarkbyes I think so22:16
dansmithhttps://termbin.com/s2xj22:17
clarkbI wonder if we need to instrument keystoen directly to try and identify that?22:17
dansmithI think I'd like to know, cause that seems like some n^2 stuff to me22:17
clarkball of the other services look pretty reasonable22:17
dansmithyes22:17
dansmithneutron is pretty high,22:17
dansmithand I see the number climb pretty fast when it's creating our network and subnet, which seems weird,22:18
dansmithbut it's still not 57k-level concerning22:18
clarkbthe token issuance would've been my first guess but I agree that those should be writes not reads22:18
dansmithI would think22:18
clarkband even then we don't do 57k osc commands22:18
clarkblike maybe 1k22:18
dansmithright22:18
clarkbI guess every other api request wiht a keystone token may have to validate with keystone?22:19
clarkbbut napkin math adding everything else together there doens't come close to 57k22:19
dansmithyes, I expect validates to turn into selects, but still seems crazy high22:20
dansmith113 inserts, so maybe say 100 of those are tokens 22:20
dansmiththat is 570 validates for each one22:21
dansmithI guess catalog lookups maybe22:21
dansmithbut still, production systems must be getting _hammered)22:21
clarkbit definitely seems like identifying the source of those and either reducingthem or making them more performant would be a worthwhile exercise22:21
dansmithyeah curious to see if the keystone people think that's crazy or not22:22
dansmithclarkb: check this: https://zuul.opendev.org/t/openstack/build/f31b8439b6dc47a19f9c99bbe3653d74/log/controller/logs/devstacklog.txt#2015822:24
dansmith82k by the end of the devstack run22:24
dansmithno wonder my 100k limit was rolling over on a full tempest run22:25
clarkbwow and that is before tempest runs22:25
dansmithyeah22:26
dansmithlike maybe some ORM usage is causing a bunch of lookups each time we pull a token or something22:27
dansmithlike 20 queries to pull the catalog entries or something22:27
dansmithdmendiza[m]: I dunno what tz you're in, but are you around?22:28
clarkbdansmith: if you want we can hold a node then you can check the count and check it again 5 minute slater and see if background tasks have a big impact. You can also do things like catalog list and check the delta etc22:28
clarkbthough I seem to recall you are good about running local devstack too22:29
dansmithclarkb: it repros locally just fine, but thanks :)22:29
dansmithyeah22:29
dansmithmy local run crashed in the middle, which is why I only got to 57k apparently22:30
dansmithit might be cool to have a way to run tempest single-threaded and capture the after-before numbers for each test to see which operations inflate numbers like these22:31
dansmithclarkb: one test into tempest locally and keystone selects increase by 250023:12
clarkbdansmith: once you track this down every database behind openstack will owe you beers :)23:14
dansmithI'm just surprised, this might be common knowledge, I dunno23:15
dansmithand I'm just telling you because, I dunno, I need a buddy to be surprised with23:15
clarkbI mean I'm surprised too23:17
clarkbthat seems pretty excessive to do a lgocial unit of work in an openstack cloud23:17
sean-k-mooney[m]we are using fernet tokens now instead of uuid tokens right?23:17
dansmithsean-k-mooney[m]: are we? I thought that meant we didn't have to actually store them in order to validate23:18
dansmithprovider = fernet23:18
sean-k-mooney[m]so i tought we made the switch by default a while ago23:18
dansmithyep ^23:18
sean-k-mooney[m]have you tried deploying just keystone and then makeing a singel token issue request to see what that results in from a db point of view23:20
dansmithno, I'm just watching the numbers during runs and was surprised.. wasn't looking for anything, was just like "this number has too many zeroes"23:20
clarkbeven then why do fernet tokens make you think thousnads of db queries per logical cloud action?23:21
dansmithneutron queries are super high too,23:21
dansmithI'm running tempest mostly doing compute tests right now and neutron and keystone are both ~25k selects23:21
dansmithand nova-api is ~2500 for reference23:22
sean-k-mooney[m]clarkb they done i was wonderign if we were sitll using uuid tokens by mistake or something like that23:22
dansmithsean-k-mooney[m]: ah, good thought then.. but config mentions fernet, so I assume..23:22
sean-k-mooney[m]im kind of surpiseed how low placement its in those results23:24
sean-k-mooney[m]for a service that is basicaly a restapi bolted on top a db it does not do much in the db during that run23:25
dansmithsean-k-mooney[m]: that is before a tempest run, so not much for placement to have done yet23:25
sean-k-mooney[m]i assume that keystone is somewhat inflated as presumable devstack is not caching the tokens so every osc call is a new token ectra23:26
dansmithit is, but still, how can it be 80k?23:26
sean-k-mooney[m]it might be interseting to just add a count of osc calls to devstack but ya its very high23:27
dansmithso in my ongoing tempest run, we're testing compute, nova-api has made 3700 select calls, keystone has made 42k so far23:28
dansmithand tempest doesn't get a token for every operation, AFAIK23:28
dansmith(neutron has made 47k btw)23:29
sean-k-mooney[m]that is nuts considering that neutron with ovn which moves some of the state to the ovn db23:37
sean-k-mooney[m]granted ovn is mainly ment to reduce rpc load not db load23:37
dansmith...and considering we're not testing neutron yet, just using it through nova23:37
dansmithI'm up to 52k for keystone, 80k for neutron23:37
sean-k-mooney[m]there are quite a few warning in the keystone log23:38
sean-k-mooney[m]https://zuul.opendev.org/t/openstack/build/f31b8439b6dc47a19f9c99bbe3653d74/log/controller/logs/screen-keystone.txt?severity=323:38
dansmith6k for nova-api23:38
sean-k-mooney[m]looks the initalial account creation23:38
sean-k-mooney[m]but not sure why the default domain and roles cant be found23:39
dansmithbefore bootstrap is run maybe?23:39
sean-k-mooney[m]maybe but i would have expected us to bring up keystone pretty early in devstack23:40
dansmithit's mid-early23:40
dansmithor, early-mod23:40
dansmithmid23:40
sean-k-mooney[m]anyway my tablet is about to die so ill check back tomorrow23:41
dansmitho/23:41
dansmithI'm about to break 100k on neutron, I should crack a beer23:41
clarkbits over 9 thouasand23:46

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!