Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FreeIPA: on a replica, ipa ca-del subca fails with @pki/master repo #4669

Closed
flo-renaud opened this issue Feb 15, 2024 · 5 comments · Fixed by #4673
Closed

FreeIPA: on a replica, ipa ca-del subca fails with @pki/master repo #4669

flo-renaud opened this issue Feb 15, 2024 · 5 comments · Fixed by #4673

Comments

@flo-renaud
Copy link

The following issue happens when installing pki from @pki/master repository.
Scenario: install a server + replica with CA service, create a subca on the server, disable the subca on the replica and delete the subca on the replica. The command ipa ca-del fails with ipa: ERROR: Request failed with status 500: Non-2xx response from CA REST API: 500.

Reproducer:
On the server:

dnf copr enable -y @pki/master
dnf copr enable -y @freeipa/freeipa-master-nightly
dnf install -y freeipa-server-dns
ipa-server-install --domain ipa.test --realm IPA.TEST --setup-dns --auto-forwarders -a Secret123 -p Secret123 -U

On the replica:

dnf copr enable -y @pki/master
dnf copr enable -y @freeipa/freeipa-master-nightly
dnf install -y freeipa-server-dns
ipa-replica-install --domain ipa.test --realm IPA.TEST --setup-dns --auto-forwarders --setup-ca --principal admin --password Secret123 --server server.ipa.test -U

Create a sub ca on the server:

echo Secret123 | kinit admin
ipa ca-add subca --subject cn=subca,O=IPA.TEST

Disable and delete the sub ca on the replica:

echo Secret123 | kinit admin
ipa ca-disable subca
ipa ca-del subca

The last command fails.

The replica logs show that PKI is trying to find a subca with a dn cn=null,ou=certificateRepository,ou=ca,o=ipaca:

2024-02-15 07:47:06 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-5] INFO: AuthorityService: deleting authority 826c3884-348f-4fde-86aa-5aa07c8ed878
2024-02-15 07:47:06 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-5] INFO: LDAPSession: Retrieving cn=null,ou=certificateRepository, ou=ca,o=ipaca
2024-02-15 07:47:06 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-5] SEVERE: Error modifying authority: Record not found
Record not found
        at com.netscape.cmscore.dbs.LDAPSession.read(LDAPSession.java:200)
        at com.netscape.cmscore.dbs.LDAPSession.read(LDAPSession.java:149)
        at com.netscape.cmscore.dbs.CertificateRepository.readCertificateRecord(CertificateRepository.java:819)
        at org.dogtagpki.server.ca.CAEngine.revokeAuthority(CAEngine.java:2184)
        at org.dogtagpki.server.ca.CAEngine.deleteAuthority(CAEngine.java:2256)
        at org.dogtagpki.server.ca.rest.AuthorityService.deleteCA(AuthorityService.java:424)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:236)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:406)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:213)
        at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228)
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
        at jdk.internal.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:222)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:584)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:250)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:142)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:207)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:137)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:136)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
        at jdk.internal.reflect.GeneratedMethodAccessor43.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:222)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:584)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:250)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:202)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:176)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:137)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:136)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:596)
        at com.netscape.cms.tomcat.ExternalAuthenticationValve.invoke(ExternalAuthenticationValve.java:83)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
        at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:545)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:670)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:424)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: netscape.ldap.LDAPException: No such object (32); matchedDN = ou=certificateRepository,ou=ca,o=ipaca
        at netscape.ldap.LDAPConnection.checkMsg(LDAPConnection.java:4933)
        at netscape.ldap.LDAPConnection.checkSearchMsg(LDAPConnection.java:2686)
        at netscape.ldap.LDAPConnection.search(LDAPConnection.java:2658)
        at netscape.ldap.LDAPConnection.search(LDAPConnection.java:2484)
        at com.netscape.cmscore.dbs.LDAPSession.read(LDAPSession.java:178)
        ... 63 more

2024-02-15 07:47:06 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-5] INFO: PKIExceptionMapper: Returning PKIException
2024-02-15 07:47:06 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-5] INFO: PKIExceptionMapper: XML exception:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<PKIException>
    <ClassName>com.netscape.certsrv.base.PKIException</ClassName>
    <Attributes/>
    <Code>500</Code>
    <Message>Error modifying authority: Record not found</Message>
</PKIException>

Version: dogtag-pki-ca-11.5.0-0.4.alpha7.20240214145232UTC.a064ebe7.fc39.noarch

@fmarco76
Copy link
Member

@flo-renaud I am trying to reproduce this error using latest version from copr and two containers but I get no error.
For the configuration I am using this scenario: https://github.com/dogtagpki/pki/blob/master/.github/workflows/ipa-clone-test.yml. After set the primary ipa instance and the clone in the secondary container I get:

fmarco76@fedora:~/Projects/upstream/pki$ docker exec primary ipa ca-add subca --subject cn=subca,O=EXAMPLE.COM
------------------
Created CA "subca"
------------------
  Name: subca
  Authority ID: 94041ba8-0451-4d05-8bb3-8ece448c1150
  Subject DN: CN=subca,O=EXAMPLE.COM
  Issuer DN: CN=Certificate Authority,O=EXAMPLE.COM
  Certificate: MIIEezCCAuOgAwIBAgIBDTANBgkqhkiG9w0BAQsFADA2MRQwEgYDVQQKDAtFWEFNUExFLkNPTTEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4XDTI0MDIxOTEyMjI0N1oXDTQ0MDIxOTEyMTEzMlowJjEUMBIGA1UECgwLRVhBTVBMRS5DT00xDjAMBgNVBAMMBXN1YmNhMIIBojANBgkqhkiG9w0BAQEFAAOCAY8AMIIBigKCAYEAwFzbkSBM9xyc6Dh/UzR7E5VK0/niq/beAIDruC6dgntPlSzZSRIBDc6uj55/9aZ+4Y9ADd4qKA1YA31OeZB+oTtN6V4FV9ner/z8CNedFHecyAhroZ+kojwWPI+DDeWoTvng/AzQe5DVBIKPvM7791z+hBpVpOfpsouj/EFZ7C+9ImlFsWEYLrebC8iHzr98jCN8LUx3uyAIN/Ma04SM/BzSNP1M7ybA8+IgPeDm8KxkuDUBfquD4ywfe47pZBJnb1O/awoO3lS3xyQiGesSsqXTVFJmSt0+W1qe4eF7Fcl/alQe+qA7o1MhL9uRILG8qC9WhAaS66oW13Rp/3yp+D6etLv/dUFc7Das7maV6vEP2iYOJwnZW1al/K8+UAxQV2qULfQ3pXsHQkEi/zJOEDUrNsfWqyRMh3lYLWTai8L9NXi6yiETarL6GuCwX/YOEUDgo2Z8UHAhJm1IXaqvWC3lMQ271JIzVRXfWkpxByNlj0vZhSDT46NS/oUBtkUnAgMBAAGjgaMwgaAwHQYDVR0OBBYEFKxaEdUvqHFEj+rC4e4Ah6cLc0SaMB8GA1UdIwQYMBaAFJqg+3asCRoMmAiNLGJOU0z+5Xz4MA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgHGMD0GCCsGAQUFBwEBBDEwLzAtBggrBgEFBQcwAYYhaHR0cDovL2lwYS1jYS5leGFtcGxlLmNvbS9jYS9vY3NwMA0GCSqGSIb3DQEBCwUAA4IBgQAX5hJXrucQBn+2+844Gpuk2lEb4iu4zfQ2WDW9lvExe3kevWJB2ruFcy5zD4AEkttsR2qwFsYyFoRUfQ+ktwAEVJjrMNLzPaJ99qFLUaIV4077oKlL6x/3Ve0eN0yMPEubm7zCz2udDitYgoslc8MWEIj9SuKoTT3acIRNeFuFIrObIxjgQ1mGZhtmGv29eeSIspOL9E4vHsechaSV4t0rr6rrhfhEEbwo7OXCMALNJR2oVY5+ewQ9EIm2/KjtxRLJmB8YOnLs8tr07HJE4wJ+9fG1G6N3ewl+rQhL+/uoMYJ3jPh1THgi7I/9FkLh4qDh8Lms2/Mzdu9bcoPmw+kxyOP1hlo5GAMh6w65O7d/WQA40QyDIhCxR/D7UwgOltp4Dk/Y5OLX8uc+AL+bySYR0aKS9jBsmO2OIXTxPMo0LelO8xlGaz1SWqATd9jWOnuXfr7ENCDQDFh7iJ8keVun+wwtYFUvuEOOoGpRlvrhSRCaPEn6zgjrRZzARp0srGE=
fmarco76@fedora:~/Projects/upstream/pki$ docker exec secondary ipa ca-find
-------------
2 CAs matched
-------------
  Name: ipa
  Description: IPA CA
  Authority ID: af582b4a-c6e8-4950-84e2-13e295adfb72
  Subject DN: CN=Certificate Authority,O=EXAMPLE.COM
  Issuer DN: CN=Certificate Authority,O=EXAMPLE.COM
  RSN Version: 0

  Name: subca
  Authority ID: 94041ba8-0451-4d05-8bb3-8ece448c1150
  Subject DN: CN=subca,O=EXAMPLE.COM
  Issuer DN: CN=Certificate Authority,O=EXAMPLE.COM
----------------------------
Number of entries returned 2
----------------------------
fmarco76@fedora:~/Projects/upstream/pki$ docker exec secondary ipa ca-disable subca
-------------------
Disabled CA "subca"
-------------------
fmarco76@fedora:~/Projects/upstream/pki$ docker exec secondary ipa ca-del subca
------------------
Deleted CA "subca"
------------------
fmarco76@fedora:~/Projects/upstream/pki$ docker exec secondary ipa ca-find
------------
1 CA matched
------------
  Name: ipa
  Description: IPA CA
  Authority ID: af582b4a-c6e8-4950-84e2-13e295adfb72
  Subject DN: CN=Certificate Authority,O=EXAMPLE.COM
  Issuer DN: CN=Certificate Authority,O=EXAMPLE.COM
  RSN Version: 0
----------------------------
Number of entries returned 1
----------------------------

The setup is very similar but the dns is not configure. I tried to add dns configuration but get some problems with docker. Could you verify if the problem is still present and/or the dns is relevant to get the error? Thanks

@flo-renaud
Copy link
Author

@fmarco76 yes the problem is still present. I did a setup similar to yours (no dns server setup) and the issue is still here.
version: dogtag-pki-ca-11.5.0-0.4.alpha7.20240219105515UTC.2a24da92.fc39.noarch

Did you also setup the CA role on the clone?

@fmarco76
Copy link
Member

Did you also setup the CA role on the clone?

I tried with and without the ca role with similar results. I'll do a new setup and playing with the parameter to get the error. I would be sure the error is still present before performing additional tests.

@fmarco76
Copy link
Member

OK @flo-renaud , after several tests I have found how to reproduce the error. The replica is generated differently in the clone workflow. I'll work on this. Thanks

fmarco76 added a commit to fmarco76/pki that referenced this issue Feb 21, 2024
The tracking update allows to avoid reloading an authority if no changes
have been done to the record. To verify the changes it is used the
LDAP attribute `entryUSN` which is updated by DS server on any change.

The tracker update mechanism has a race condition when an entry is
modified by the `CAEngine`. When the method `modifyAuthorityEntry()` is
invoked, this will update the tracker and no CA are reloaded by the
`AuthorityMonitor` thread because the tracker is already to the newest
value. However, in case of CA clones, when a sub CA is created in the
primary CA, the clone will get the record but there is no serial.
The CA is registered and when the related keys are retrieved the
record is update with the serial. The update is done by the
`modifyAuthorityEntry()` which will update the trackers. As a result the
`AuthorityMonitor` will not update the CA object and when used
it will miss the serial so some operations will fails.

Since, the `trackerUpdate` method has not other impact has been removed so all
trackers are managed by the `AuthorityMonitor`.

Fix dogtagpki#4669
fmarco76 added a commit that referenced this issue Feb 21, 2024
The tracking update allows to avoid reloading an authority if no changes
have been done to the record. To verify the changes it is used the
LDAP attribute `entryUSN` which is updated by DS server on any change.

The tracker update mechanism has a race condition when an entry is
modified by the `CAEngine`. When the method `modifyAuthorityEntry()` is
invoked, this will update the tracker and no CA are reloaded by the
`AuthorityMonitor` thread because the tracker is already to the newest
value. However, in case of CA clones, when a sub CA is created in the
primary CA, the clone will get the record but there is no serial.
The CA is registered and when the related keys are retrieved the
record is update with the serial. The update is done by the
`modifyAuthorityEntry()` which will update the trackers. As a result the
`AuthorityMonitor` will not update the CA object and when used
it will miss the serial so some operations will fails.

Since, the `trackerUpdate` method has not other impact has been removed so all
trackers are managed by the `AuthorityMonitor`.

Fix #4669
@flo-renaud
Copy link
Author

Hi @fmarco76
I manually tested with dogtag-pki-ca-11.5.0-1.20240222102149UTC.d8df8dab.fc39.noarch and the issue doesn't happen any more. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants