Hi all As at 2023-05-29 10:45 PM we have identified an interaction in our DNSSEC signing of the .nz second level domains which depending on recursive resolver cache state may invalidate second level zones. If you encounter issues on recursive resolvers such as SERVFAIL relating to any of the .nz second levels, flushing the resolver cache for the respective zones should allow DNS services to resume functioning. Flushing . net.nz first is recommended. https://status.internetnz.nz/incidents/gq1c6slz3198 will contain updates. Regards Josh -- *Josh Simpson* .nz Operations Manager *InternetNZ | Ipurangi Aotearoa* Helping New Zealanders harness the power of the Internet www.internetnz.nz *M* +64 21 783 399 *O* +64 4 555 0124 *GPG:* 542B 26CA 6CF6 00B1 6E6B 6D84 058C 8FE5 CEEF 5FAB [image: InternetNZ]
Hi all,
Afternoon everyone, InternetNZ has just posted an update on our status page
of a maintenance window tonight
https://status.internetnz.nz/incidents/81fyr308hsw5 this is in responce to
the issues earlier this week.
Maintenance Window Notification - 3:30pm to 5:30pm, 1 June NZST.
Zone generation maintenance
Following the recent DNSSEC chain validation incident and to ensure the
integrity of our final DNSSEC updates to complete the DNSSEC key rollover
in the .nz zone, we've decided to manually validate them.
This validation requires the temporary postponement of zone pushes during a
window between 3:30pm to 5:30pm tonight, 1 June NZST. For reference, this
maintenance should have no impact on recursive name server operations.
After the incident, we enacted several preventive measures. The DNSSEC key
rollover process was temporarily halted, which allowed us to carefully
analyse the situation and allocate sufficient time for the expiry of any
mismatched records.
While these measures were necessary, they have increased the size of our
DNS responses for a sustained period, creating a higher likelihood of
technical issues. To reduce this risk, we're now initiating the completion
of the key rollover process, which includes the removal of outdated keys.
We understand that this may impact your operations, and we appreciate your
understanding as we take these steps to maintain the stability and security
of the system.
Regards
Josh
On Tue, 30 May 2023 at 09:43, Josh Simpson
Hi all
As at 2023-05-29 10:45 PM we have identified an interaction in our DNSSEC signing of the .nz second level domains which depending on recursive resolver cache state may invalidate second level zones.
If you encounter issues on recursive resolvers such as SERVFAIL relating to any of the .nz second levels, flushing the resolver cache for the respective zones should allow DNS services to resume functioning. Flushing . net.nz first is recommended.
https://status.internetnz.nz/incidents/gq1c6slz3198 will contain updates.
Regards Josh
--
*Josh Simpson* .nz Operations Manager *InternetNZ | Ipurangi Aotearoa*
Helping New Zealanders harness the power of the Internet www.internetnz.nz
*M* +64 21 783 399 *O* +64 4 555 0124 *GPG:* 542B 26CA 6CF6 00B1 6E6B 6D84 058C 8FE5 CEEF 5FAB [image: InternetNZ]
-- *Josh Simpson* .nz Operations Manager *InternetNZ | Ipurangi Aotearoa* Helping New Zealanders harness the power of the Internet www.internetnz.nz *M* +64 21 783 399 *O* +64 4 555 0124 *GPG:* 542B 26CA 6CF6 00B1 6E6B 6D84 058C 8FE5 CEEF 5FAB [image: InternetNZ]
Kia ora! Following the DNSSEC chain validation issue for .nz domains on 29-31 May 2023, InternetNZ has published a technical incident report. https://internetnz.nz/news-and-articles/dnssec-chain-validation-issue-techni... In the report, we identified the cause of the issue, which is a misalignment between our existing DNSSEC signing configuration policies and the resulting Delegation Signer record Time To Live from the InternetNZ Registry System zone build process. The report also considers other contributing factors and describes the scale and timing of the incident, as well as our learnings and next steps. In addition to the internal review, InternetNZ Council is also commissioning an independent review to examine the events leading up to the incident and our response, and to make recommendations to prevent similar failures in the future. We’ll inform you as soon as the completion date for this review is known, and the resulting report will be published. Please reach out to registry(a)internetnz.net.nz of you have any questions regarding the technical incident report. Kind regards, -- Josh Simpson .nz Operations Manager InternetNZ | Ipurangi Aotearoa M +64 21 783 399 O +64 4 555 0124 GPG: 542B 26CA 6CF6 00B1 6E6B 6D84 058C 8FE5 CEEF 5FAB
Kia ora! Following the DNSSEC chain validation issue for .nz domains on 29-31 May 2023, InternetNZ has initiated an independent review of the incident. The review Terms of reference document is now available on our website at https://internetnz.nz/assets/Archives/RFC_ToR-DNSSEC_FINAL-v2.pdf The purpose of the review is to examine the events leading up to the incident, our response, and to make recommendations for improvement to prevent similar failures in future. The review will consider the technical, and non-technical (e.g. business process, human) factors that led to the incident. We expect to receive the report in August 2023. The report and recommendations will be made public. Please reach out to registry(a)internetnz.net.nz of you have any questions regarding the independent review ToR. Kind regards, -- Josh Simpson .nz Operations Manager InternetNZ | Ipurangi Aotearoa Helping New Zealanders harness the power of the Internet www.internetnz.nz M +64 21 783 399 O +64 4 555 0124 GPG: 542B 26CA 6CF6 00B1 6E6B 6D84 058C 8FE5 CEEF 5FAB
participants (1)
-
Josh Simpson