Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"NotUniqueError: Tried to save duplicate unique keys" on st2 register content #3933

Closed
arm4b opened this issue Jan 8, 2018 · 7 comments
Closed
Assignees
Labels
Milestone

Comments

@arm4b
Copy link
Member

arm4b commented Jan 8, 2018

Getting the following error on Ubuntu 16 latest stable st2 packages during the st2ctl --register-all (first & clean run):

    Registering content...[flags = --config-file /etc/st2/st2.conf --register-all]
    2017-11-10 16:04:14,401 INFO [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
    2017-11-10 16:04:20,355 ERROR [-] Conflict while trying to save in DB.
    Traceback (most recent call last):
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/base.py", line 173, in add_or_update
        model_object = cls._get_impl().add_or_update(model_object)
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 336, in add_or_update
        instance.save()
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 364, in save
        raise NotUniqueError(message % six.text_type(err))
    NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_type_d_b index: uid_1 dup key: { : "trigger_type:core:st2.key_value_pair.update" })
    2017-11-10 16:04:20,432 ERROR [-] Conflict while trying to save in DB.
    Traceback (most recent call last):
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/base.py", line 173, in add_or_update
        model_object = cls._get_impl().add_or_update(model_object)
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 336, in add_or_update
        instance.save()
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 364, in save
        raise NotUniqueError(message % six.text_type(err))
    NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_d_b index: uid_1 dup key: { : "trigger:core:st2.key_value_pair.value_change:99914b932bd37a50b983c5e7c90ae93b" })
    2017-11-10 16:04:20,451 ERROR [-] Conflict while trying to save in DB.
    Traceback (most recent call last):
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/base.py", line 173, in add_or_update
        model_object = cls._get_impl().add_or_update(model_object)
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 336, in add_or_update
        instance.save()
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 364, in save
        raise NotUniqueError(message % six.text_type(err))
    NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_type_d_b index: uid_1 dup key: { : "trigger_type:core:st2.key_value_pair.delete" })
    2017-11-10 16:04:20,567 INFO [-] =========================================================
    2017-11-10 16:04:20,568 INFO [-] ############## Registering triggers #####################
    2017-11-10 16:04:20,568 INFO [-] =========================================================
    2017-11-10 16:04:20,673 INFO [-] Registered 0 triggers.

Happens randomly, probably some race and I couldn't reproduce it consistently so far.
Hence, still good to leave it here until we or someone else finds more info.

@arm4b arm4b added the bug label Jan 8, 2018
@arm4b
Copy link
Member Author

arm4b commented Mar 6, 2018

Seeing it again on StackStorm first install.
screenshot-from-2018-03-06-15-32-37

The worst thing is that the error doesn't result in st2ctl reload script failure and so installation goes further as nothing wrong happened.

I think it should fail on failure like this, so we stop the provision on that error instead of going further. @Kami do you think that's something at minimum we can fix?

@arm4b
Copy link
Member Author

arm4b commented Mar 20, 2018

More of this:

    virtualbox-iso: 20180320T114415+0000 Registering content...[flags = --config-file /etc/st2/st2.conf --register-all]
    virtualbox-iso: 20180320T114418+0000 2018-03-20 11:44:18,975 INFO [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
    virtualbox-iso: 20180320T114421+0000 2018-03-20 11:44:21,985 ERROR [-] Conflict while trying to save in DB.
    virtualbox-iso: 20180320T114421+0000 Traceback (most recent call last):
    virtualbox-iso: 20180320T114421+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/base.py", line 173, in add_or_update
    virtualbox-iso: 20180320T114421+0000     model_object = cls._get_impl().add_or_update(model_object)
    virtualbox-iso: 20180320T114421+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 388, in add_or_update
    virtualbox-iso: 20180320T114421+0000     instance.save()
    virtualbox-iso: 20180320T114421+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 364, in save
    virtualbox-iso: 20180320T114422+0000     raise NotUniqueError(message % six.text_type(err))
    virtualbox-iso: 20180320T114422+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_type_d_b index: uid_1 dup key: { : "trigger_type:core:st2.generic.actiontrigger" })
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,039 ERROR [-] Conflict while trying to save in DB.
    virtualbox-iso: 20180320T114422+0000 Traceback (most recent call last):
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/base.py", line 173, in add_or_update
    virtualbox-iso: 20180320T114422+0000     model_object = cls._get_impl().add_or_update(model_object)
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 388, in add_or_update
    virtualbox-iso: 20180320T114422+0000     instance.save()
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 364, in save
    virtualbox-iso: 20180320T114422+0000     raise NotUniqueError(message % six.text_type(err))
    virtualbox-iso: 20180320T114422+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_d_b index: uid_1 dup key: { : "trigger:core:st2.generic.actiontrigger:99914b932bd37a50b983c5e7c90ae93b" })
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,146 ERROR [-] Conflict while trying to save in DB.
    virtualbox-iso: 20180320T114422+0000 Traceback (most recent call last):
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/base.py", line 173, in add_or_update
    virtualbox-iso: 20180320T114422+0000     model_object = cls._get_impl().add_or_update(model_object)
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 388, in add_or_update
    virtualbox-iso: 20180320T114422+0000     instance.save()
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 364, in save
    virtualbox-iso: 20180320T114422+0000     raise NotUniqueError(message % six.text_type(err))
    virtualbox-iso: 20180320T114422+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_type_d_b index: uid_1 dup key: { : "trigger_type:core:st2.generic.notifytrigger" })
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,547 ERROR [-] Conflict while trying to save in DB.
    virtualbox-iso: 20180320T114422+0000 Traceback (most recent call last):
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/base.py", line 173, in add_or_update
    virtualbox-iso: 20180320T114422+0000     model_object = cls._get_impl().add_or_update(model_object)
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 388, in add_or_update
    virtualbox-iso: 20180320T114422+0000     instance.save()
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 364, in save
    virtualbox-iso: 20180320T114422+0000     raise NotUniqueError(message % six.text_type(err))
    virtualbox-iso: 20180320T114422+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_d_b index: uid_1 dup key: { : "trigger:core:st2.key_value_pair.value_change:99914b932bd37a50b983c5e7c90ae93b" })
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,586 ERROR [-] Conflict while trying to save in DB.
    virtualbox-iso: 20180320T114422+0000 Traceback (most recent call last):
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/base.py", line 173, in add_or_update
    virtualbox-iso: 20180320T114422+0000     model_object = cls._get_impl().add_or_update(model_object)
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 388, in add_or_update
    virtualbox-iso: 20180320T114422+0000     instance.save()
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 364, in save
    virtualbox-iso: 20180320T114422+0000     raise NotUniqueError(message % six.text_type(err))
    virtualbox-iso: 20180320T114422+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_type_d_b index: uid_1 dup key: { : "trigger_type:core:st2.key_value_pair.delete" })
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,762 INFO [-] =========================================================
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,762 INFO [-] ############## Registering triggers #####################
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,763 INFO [-] =========================================================
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,887 INFO [-] Registered 0 triggers.
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,887 INFO [-] =========================================================
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,887 INFO [-] ############## Registering sensors ######################
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,887 INFO [-] =========================================================
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,934 INFO [-] Registered 1 sensors.
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,934 INFO [-] =========================================================
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,934 INFO [-] ############## Registering runners ######################
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,934 INFO [-] =========================================================
    virtualbox-iso: 20180320T114423+0000 2018-03-20 11:44:23,537 INFO [-] Registered 14 runners.
    virtualbox-iso: 20180320T114423+0000 2018-03-20 11:44:23,537 INFO [-] =========================================================
    virtualbox-iso: 20180320T114423+0000 2018-03-20 11:44:23,537 INFO [-] ############## Registering actions ######################
    virtualbox-iso: 20180320T114423+0000 2018-03-20 11:44:23,537 INFO [-] =========================================================
    virtualbox-iso: 20180320T114424+0000 2018-03-20 11:44:24,921 INFO [-] Registered 51 actions.
    virtualbox-iso: 20180320T114424+0000 2018-03-20 11:44:24,921 INFO [-] =========================================================
    virtualbox-iso: 20180320T114424+0000 2018-03-20 11:44:24,921 INFO [-] ############## Registering rules ########################
    virtualbox-iso: 20180320T114424+0000 2018-03-20 11:44:24,921 INFO [-] =========================================================
    virtualbox-iso: 20180320T114424+0000 2018-03-20 11:44:24,958 INFO [-] Registered 1 rules.
    virtualbox-iso: 20180320T114424+0000 2018-03-20 11:44:24,958 INFO [-] =========================================================
    virtualbox-iso: 20180320T114424+0000 2018-03-20 11:44:24,958 INFO [-] ############## Registering aliases ######################
    virtualbox-iso: 20180320T114424+0000 2018-03-20 11:44:24,958 INFO [-] =========================================================
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:24,999 INFO [-] Registered 4 aliases.
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,000 INFO [-] =========================================================
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,000 INFO [-] ############## Registering policy types #################
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,000 INFO [-] =========================================================
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,023 INFO [-] Registered 3 policy types.
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,023 INFO [-] =========================================================
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,023 INFO [-] ############## Registering policies #####################
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,023 INFO [-] =========================================================
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,024 INFO [-] Registered 0 policies.
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,024 INFO [-] =========================================================
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,024 INFO [-] ############## Registering configs ######################
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,024 INFO [-] =========================================================
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,026 INFO [-] Registered 0 configs.

@arun2803
Copy link

arun2803 commented Apr 5, 2019

I am also facing same issue

20190405T104945+0000 Registering content...[flags = --config-file /etc/st2/st2.conf --register-all]
20190405T104949+0000 2019-04-05 10:49:49,621 INFO [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
20190405T104949+0000 2019-04-05 10:49:49,626 INFO [-] Successfully connected to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
20190405T104958+0000 2019-04-05 10:49:58,815 ERROR [-] Conflict while trying to save in DB.
20190405T104958+0000 Traceback (most recent call last):
20190405T104958+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/persistence/base.py", line 174, in add_or_update
20190405T104959+0000     model_object = cls._get_impl().add_or_update(model_object, validate=True)
20190405T104959+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 452, in add_or_update
20190405T104959+0000     instance.save(validate=validate)
20190405T104959+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/mongoengine/document.py", line 403, in save
20190405T104959+0000     raise NotUniqueError(message % six.text_type(err))
20190405T104959+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_d_b index: uid_1 dup key: { : "trigger:core:st2.key_value_pair.update:99914b932bd37a50b983c5e7c90ae93b" })
20190405T104959+0000 2019-04-05 10:49:58,853 ERROR [-] Conflict while trying to save in DB.
20190405T104959+0000 Traceback (most recent call last):
20190405T104959+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/persistence/base.py", line 174, in add_or_update
20190405T104959+0000     model_object = cls._get_impl().add_or_update(model_object, validate=True)
20190405T104959+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 452, in add_or_update
20190405T104959+0000     instance.save(validate=validate)
20190405T104959+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/mongoengine/document.py", line 403, in save
20190405T104959+0000     raise NotUniqueError(message % six.text_type(err))
20190405T104959+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_type_d_b index: uid_1 dup key: { : "trigger_type:core:st2.key_value_pair.value_change" })
20190405T104959+0000 2019-04-05 10:49:58,910 ERROR [-] Conflict while trying to save in DB.
20190405T105000+0000 Traceback (most recent call last):
20190405T105000+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/persistence/base.py", line 174, in add_or_update
20190405T105000+0000     model_object = cls._get_impl().add_or_update(model_object, validate=True)
20190405T105000+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 452, in add_or_update
20190405T105000+0000     instance.save(validate=validate)
20190405T105000+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/mongoengine/document.py", line 403, in save
20190405T105000+0000     raise NotUniqueError(message % six.text_type(err))
20190405T105000+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_type_d_b index: uid_1 dup key: { : "trigger_type:core:st2.key_value_pair.delete" })
20190405T105000+0000 2019-04-05 10:49:59,090 ERROR [-] Conflict while trying to save in DB.
20190405T105000+0000 Traceback (most recent call last):
20190405T105000+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/persistence/base.py", line 174, in add_or_update
20190405T105000+0000     model_object = cls._get_impl().add_or_update(model_object, validate=True)
20190405T105000+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 452, in add_or_update
20190405T105000+0000     instance.save(validate=validate)
20190405T105000+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/mongoengine/document.py", line 403, in save
20190405T105000+0000     raise NotUniqueError(message % six.text_type(err))
20190405T105000+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_d_b index: uid_1 dup key: { : "trigger:core:st2.sensor.process_spawn:99914b932bd37a50b983c5e7c90ae93b" })
20190405T105001+0000 2019-04-05 10:49:59,133 ERROR [-] Conflict while trying to save in DB.
20190405T105001+0000 Traceback (most recent call last):
20190405T105001+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/persistence/base.py", line 174, in add_or_update
20190405T105001+0000     model_object = cls._get_impl().add_or_update(model_object, validate=True)
20190405T105001+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 452, in add_or_update
20190405T105001+0000     instance.save(validate=validate)
20190405T105001+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/mongoengine/document.py", line 403, in save
20190405T105001+0000     raise NotUniqueError(message % six.text_type(err))
20190405T105001+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_type_d_b index: uid_1 dup key: { : "trigger_type:core:st2.sensor.process_exit" })
20190405T105001+0000 2019-04-05 10:49:59,199 ERROR [-] Conflict while trying to save in DB.
20190405T105001+0000 Traceback (most recent call last):
20190405T105001+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/persistence/base.py", line 174, in add_or_update
20190405T105001+0000     model_object = cls._get_impl().add_or_update(model_object, validate=True)
20190405T105001+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 452, in add_or_update
20190405T105001+0000     instance.save(validate=validate)
20190405T105002+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/mongoengine/document.py", line 403, in save
20190405T105002+0000     raise NotUniqueError(message % six.text_type(err))
20190405T105002+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_d_b index: uid_1 dup key: { : "trigger:core:st2.sensor.process_exit:99914b932bd37a50b983c5e7c90ae93b" })

@arun2803
Copy link

arun2803 commented Apr 5, 2019

I fixed it by

  1. uninstalling st2* , postgres* and mongo* packages
  2. Delete all data directory for postgres and mongo
  3. Delete the configs related with these packages
  4. Reinstall packages

Also st2chatops 2.10 is dependent on nodejs > nodejs-8.0, which doesn't get install with the single command installation process, so I installed it from https://rpm.nodesource.com/pub_8.x/el/7/x86_64/nodejs-8.1.0-1nodesource.el7.centos.x86_64.rpm

@nmaludy
Copy link
Member

nmaludy commented Apr 9, 2019

I can confirm i'm seeing this too, as reported in #4626.

Here's a gist of my install log: https://gist.github.com/nmaludy/875acebf3772d10384df5eccfc2605e9

I noticed that this only happened when installing on CentOS 7, i couldn't reproduce using Ubuntu 16.04.

Here are my steps to reproduce:

git clone https://github.com/StackStorm/st2vagrant
cd st2vagrant
BOX=centos/7 vagrant up

@Kami
Copy link
Member

Kami commented Apr 9, 2019

I will try to have a look this week.

From a quick glance - some of those errors (trigger type already exists) are not fatal.

IIRC, we already made some changes in the past to log those errors under DEBUG instead of ERROR so it doesn't confuse the user, but it may need more work (and there might be another face hiding there which is fatal).

@Kami Kami self-assigned this Apr 9, 2019
@Kami Kami added this to the 3.0.0 milestone Apr 10, 2019
@Kami
Copy link
Member

Kami commented Apr 10, 2019

I was able to reproduce this issue today.

It's indeed related to a non-fatal race - we try to register internal trigger types on each service startup and during register content script run.

Operation itself is idempotent so failure isn't fatal (it simply means internal trigger has already been registered by another service). Only problem is with logging - we log error as fatal, even though it's not.

I will open a PR which fixes the logging and logs this error under DEBUG instead of ERROR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants