When Everything Broke... and the Cook'n Family Came Through
For the first time in nearly 20 years, the Cook'n Newsletter went silent.
After sending this newsletter faithfully every week for two decades, we missed more than a month. Several of you wrote to ask what happened, so I thought I would tell you the whole story.
It began on the Saturday before Easter.
That afternoon, we started receiving reports that Cook'n for iPhone was failing to launch. At first, we were baffled. But before long, we discovered the cause.
An important domain name used by Cook'n Cloud services had expired.
Unfortunately, since it was a highly trafficked domain, the moment it expired, it was purchased by someone else. We could not simply renew it and move on. We had to reconfigure Cook'n to use a different domain.
This was a much bigger problem than it may sound.
Over the years, Cook'n has grown into a sophisticated ecosystem that includes desktop apps for Windows and Mac, mobile apps for iPhone, iPad, Android, Kindle Fire, the Cook'n Web App, browser Capture extensions, the Alexa Skill, the Apple Watch app, Cloud sync services, billing systems, and many other behind-the-scenes tools.
All of them depended, in one way or another, on that domain.
Since Easter Sunday was just hours away and tens of thousands of families would be using Cook'n to prepare Easter dinner, it was an all-hands-on-deck emergency.
And the Cook'n Elves were magnificent.
The entire team dropped whatever they were doing and immediately went to work.
Within hours:
We also prepared an urgent iPhone update, but when we tried to submit it to Apple, their developer website was malfunctioning.
We stayed up until 1:30 a.m. trying every possible workaround. One suggestion was to simply wait 1 to 3 hours to see if their developer console would refresh. So, we went to bed and woke up again at 3:00 am. Still no luck. But, we persisted.
At one point, ChatGPT gave me a piece of advice that was both amusing and surprisingly insightful:
I had been clicking the same series of buttons and links in different sequences over and over, hoping for a different result.
Fortunately, Apple resolved their issue first thing Monday morning. We submitted the update, requested an expedited review, and Apple approved it the same day.
But the surprises were not over.
That very same weekend, for the first time ever, our email account was suspended due to a separate DNS and DKIM problem involving our dvo.com domain. So even though I wanted to send everyone an email explaining what was happening, I couldn't.
I was completely cut off.
To make matters even more interesting, I discovered at 4:30 a.m. on Easter morning that the Cook'n Capture Plugin had stopped working as well.
One by one, we found and fixed every issue.
In my 34 years of developing Cook'n, this was the largest technical crisis we have ever faced.
For most users, the disruption was relatively minor. For me, however, it was all-consuming.
This experience reminded me of several important things.
First, I was struck by how deeply I care about the Cook'n Family. I was surprised at how much my peace of mind and mental and emotional well-being is wrapped up in making sure Cook'n works perfectly. And, if there is something out of order with Cook'n, my whole universe is disrupted! When something prevents you from having the amazing Cook'n experience I want you to have, I truly cannot rest until it is fixed.
Second, I was amazed by the complexity of what the Cook'n Elves have built. There are countless moving parts, and it is remarkable how seamlessly they normally work together.
Third, I was deeply touched by the way our team responded. No one complained. No one pointed fingers. Everyone simply did whatever was necessary to make things right.
And finally, I was overwhelmed by the kindness of the Cook'n Family.
Longtime subscriber Hilary Gillen, known to many of you as DeliBaker, sent the entire team a care package along with this note:
The gifts were wonderful, but the sentiment behind them meant even more.
At a moment when I felt personally responsible for causing a major disruption, your kindness and encouragement lifted our spirits.
I say this often, but I mean it with all my heart:
There is no better customer community in the world than the Cook'n Family.
Thank you for your patience.
Thank you for your support.
Thank you for your forgiveness.
And thank you for allowing us to be part of your kitchens and your lives.
I thank God every day for the Cook'n Elves and for each and every one of you.
The good news is that everything is now working properly again.
And, even while resolving this issue, the Cook'n Elves continued making exciting progress. We recently added an AMAZING new feature to Cook'n for iPhone (click here to learn more about Magic Scan for iOS and be sure to give this a try!). And, we completed a major desktop database upgrade that makes Cook'n faster and more resilient than ever.
We have so many cool new things coming up that I can't wait to tell you about. This domain issue was definitely a hiccup that delayed our progress a bit. But, we're through it now and we're on to bigger and better. Keep your eyes peeled for fun new things to come!
blog comments powered by Disqus
After sending this newsletter faithfully every week for two decades, we missed more than a month. Several of you wrote to ask what happened, so I thought I would tell you the whole story.
It began on the Saturday before Easter.
That afternoon, we started receiving reports that Cook'n for iPhone was failing to launch. At first, we were baffled. But before long, we discovered the cause.
An important domain name used by Cook'n Cloud services had expired.
Unfortunately, since it was a highly trafficked domain, the moment it expired, it was purchased by someone else. We could not simply renew it and move on. We had to reconfigure Cook'n to use a different domain.
This was a much bigger problem than it may sound.
Over the years, Cook'n has grown into a sophisticated ecosystem that includes desktop apps for Windows and Mac, mobile apps for iPhone, iPad, Android, Kindle Fire, the Cook'n Web App, browser Capture extensions, the Alexa Skill, the Apple Watch app, Cloud sync services, billing systems, and many other behind-the-scenes tools.
All of them depended, in one way or another, on that domain.
Since Easter Sunday was just hours away and tens of thousands of families would be using Cook'n to prepare Easter dinner, it was an all-hands-on-deck emergency.
And the Cook'n Elves were magnificent.
The entire team dropped whatever they were doing and immediately went to work.
Within hours:
- The Cook'n Desktop App was updated.
- The Cook'n Web App was updated.
- Android and Kindle Fire updates were submitted.
- The Alexa Skill and Apple Watch app were modified.
- Browser capture extensions were repaired.
- Our website and support pages were updated continuously.
- Our support staff monitored customer emails around the clock.
We also prepared an urgent iPhone update, but when we tried to submit it to Apple, their developer website was malfunctioning.
We stayed up until 1:30 a.m. trying every possible workaround. One suggestion was to simply wait 1 to 3 hours to see if their developer console would refresh. So, we went to bed and woke up again at 3:00 am. Still no luck. But, we persisted.
At one point, ChatGPT gave me a piece of advice that was both amusing and surprisingly insightful:
“No amount of clicking will fix this.”It was absolutely right.
I had been clicking the same series of buttons and links in different sequences over and over, hoping for a different result.
Fortunately, Apple resolved their issue first thing Monday morning. We submitted the update, requested an expedited review, and Apple approved it the same day.
But the surprises were not over.
That very same weekend, for the first time ever, our email account was suspended due to a separate DNS and DKIM problem involving our dvo.com domain. So even though I wanted to send everyone an email explaining what was happening, I couldn't.
I was completely cut off.
To make matters even more interesting, I discovered at 4:30 a.m. on Easter morning that the Cook'n Capture Plugin had stopped working as well.
One by one, we found and fixed every issue.
In my 34 years of developing Cook'n, this was the largest technical crisis we have ever faced.
For most users, the disruption was relatively minor. For me, however, it was all-consuming.
This experience reminded me of several important things.
First, I was struck by how deeply I care about the Cook'n Family. I was surprised at how much my peace of mind and mental and emotional well-being is wrapped up in making sure Cook'n works perfectly. And, if there is something out of order with Cook'n, my whole universe is disrupted! When something prevents you from having the amazing Cook'n experience I want you to have, I truly cannot rest until it is fixed.
Second, I was amazed by the complexity of what the Cook'n Elves have built. There are countless moving parts, and it is remarkable how seamlessly they normally work together.
Third, I was deeply touched by the way our team responded. No one complained. No one pointed fingers. Everyone simply did whatever was necessary to make things right.
And finally, I was overwhelmed by the kindness of the Cook'n Family.
Longtime subscriber Hilary Gillen, known to many of you as DeliBaker, sent the entire team a care package along with this note:
Hi Dan,Inside the package were gourmet peanut brittle, boxes of KIND Minis, Twinings Black Tea, and Don Francisco's Hawaiian Coffee Blend.
You and your Elves put in a Herculean effort over this past holiday weekend, foregoing family and friends to help all of us Cook'n fanatics! That dedication deserves recognition, so I've sent another “care package” for the entire team.
Thanks again, and I'm grateful for all that you, the tech team, and the support staff do for us every day.
Regards,
Hilary J. Gillen
a.k.a. DeliBaker
The gifts were wonderful, but the sentiment behind them meant even more.
At a moment when I felt personally responsible for causing a major disruption, your kindness and encouragement lifted our spirits.
I say this often, but I mean it with all my heart:
There is no better customer community in the world than the Cook'n Family.
Thank you for your patience.
Thank you for your support.
Thank you for your forgiveness.
And thank you for allowing us to be part of your kitchens and your lives.
I thank God every day for the Cook'n Elves and for each and every one of you.
The good news is that everything is now working properly again.
And, even while resolving this issue, the Cook'n Elves continued making exciting progress. We recently added an AMAZING new feature to Cook'n for iPhone (click here to learn more about Magic Scan for iOS and be sure to give this a try!). And, we completed a major desktop database upgrade that makes Cook'n faster and more resilient than ever.
We have so many cool new things coming up that I can't wait to tell you about. This domain issue was definitely a hiccup that delayed our progress a bit. But, we're through it now and we're on to bigger and better. Keep your eyes peeled for fun new things to come!
Dan Oaks
Founder of DVO Enterprises
Creator of Cook'n
Husband of 1. Father of 5. Grandpa of 5.
Monthly Newsletter Contributor since 2024
Email the author! dan@dvo.com