______________________________________________________________________
2021.04.19 Monday 1135a
I am getting a number of questions about data from Friday, so I will
describe it again in different terms that may be more helpful:
1) The server was restored from a Thursday 10pm backup.
2) Therefore, any data that was new or changed AFTER
Thursday 10pm is NOT on the server.
3) There are two possible ways that YOU can use to get
some Friday data anyway:
a) You can go to
SpamTitan and re-send incoming email from external sources. See below for
more details.
b) If you use
Outlook from a real PC (not a MAC), in many cases Outlook will back-fill
some data back to the server.
If a) or b) do not help, there is no
other way to retrieve the data.
The above refers to EVERYTHING in outlook: emails, calendar, contacts, tasks, sub-folders – the whole nine yards.
______________________________________________________________________
2021.04.19 Monday 1118a
Exception to the
below trick in spamtitan to release: emails INTERNAL to your
company will not be there. They are lost forever..
______________________________________________________________________
2021.04.19 Monday 1031a
The server is
completely back in service in all respects as on Sunday 930am.
The email server was rendered inoperative by an attempted Chinese
attack.
No data was compromised.
If you are missing emails from Friday that you know had been
delivered:
1) You can ask the Spam Titan to re-send emails to you.
Logon to the
spamtitan at https://spamtitan.rainmakernetworks.com
Find an email
that you want under REPORTING then HISTORY.
Use “options”
then “release”.
No, there is
no way to tell it to release in bulk.
2) We are sending a list of all received email for your entire
company to the principal contact at your business. If you do not
know who that is, please ask a co-worker.
Notable items:
Many of you realized over the weekend that we would probably
have a status page on our website. That is always a good place to go
to check for information. We updated it several times over the 48h
period.
A few of you texted instead…..you got an autoreply pointing you to
that website.
A few of you called and left several voice mails…..please understand
that we have over 2000 users…..in a crisis moment it is impossible to
answer or return calls.
And last but not least…..a very small few of you emailed us using your
company email to tell us that you were having problems with your
company email. I will just let that sink in for a moment…..we did not
receive such emails until Sunday morning after everything was fixed.
______________________________________________________________________
2021.04.18 Sunday 1215p
The exchange
server was recovered from the whole-machine backup that was made at
10pm THU night 2021.04.15.
Some email in the interim period while repairs were being made was
already in your Outlook session, if you use Outlook and it was up and
running. Any such email will be sync'd back to the server when you
open your Outlook sesison again. Phones and Webclient do NOT have this
sync-back capability.
Any email that you do not see after opening your Outlook and letting
it sync for a few minutes will need to be re-sent to you. We do not
have any way to retrieve it.
______________________________________________________________________
2021.04.18 Sunday
930am
Exchange is up and
fully operational.
______________________________________________________________________
2021.04.18 Sunday
500am
The server is up and running to teh extenbt that Outlook clients connect,
Outlook Web Access works, email sends out correctly, and we have full
control.
STILL WORKING ONE FUNCTION: the server is actively refusing to accept
incoming email. Working on that now.
2021.04.17 Saturday
1000pm
I am liking China less and less.
The server functionality has been restored.
Then I immediately intentionally took it back down to apply Microsoft's
security fixes from this week.
It should be a fairly quick install - two parts; one about an hour and one
abut 10 minutes.
______________________________________________________________________
2021.04.17 Saturday
200pm
The Rainmaker Networks
Hosted Exchange server rebuild is almost complete. testing will commence
in a few minutes.
If it goes as expected, we will be able to restore services within the
next two hours, and it will have data all in place.
2021.04.17 Saturday
825am
The Rainmaker Networks
Hosted Exchange server fell victim to a variant of the China attack.
This most recent version can be read about in further detail in this
link from Microsoft's Exchange team on Tuesday this week:
Released: April 2021 Exchange Server Security Updates - Microsoft Tech Community
We of course
immediately applied the updates provided.
Note that the updates are generally preventative, not corrective,
so one cannot determine if the horse is already out of the barn.
That is, one never knows if the updates from Microsoft are too late for
a particular installation.
Our hosted exchange server was in fact too late.
We are building a fresh server that should be in service at some point
mid-morning Saturday 2021.04.17.
Historical data will fill in form the back side.
Nothing will be lost, including emails sent to you in the interim.
Check back here periodically for updates.
Thank you for understanding that we are not able to respond to the
innumerable phone calls and texts from every individual user asking for
updates.
Future:
Given that Exchange will continue to be an inviting target for China and
other folks of bad intent, we have a number of changes in store for this
service to diversify its installation to minimize the risk that Microsoft
doesnt seme to be able to do themselves.
We will describe that in some detail via email in the coming days after we
are past the immediate recovery.
Ed Goodwin