Updates on investigation on r1 SaaS API keys

18:00 PST 2024-07-05

This update provides a non-technical overview of what we have determined is the impact of the recent security event. In the coming weeks, we will be performing an in-depth post-mortem on this event. The output of that post-mortem will include an engineering blog post detailing what we’ve learned, additional details on how our systems work, and a review of the technical controls we have put in place as a result of this event.

1. We have evidence that an employee leaked a copy of confidential internal code to a self-described “hacktivist” group. The employee has been terminated and remains under investigation.
2. The leaked code included several API keys. However, we were already in the process of migrating secrets out of code and into AWS Secrets Manager (a tool for storing secrets, like API keys, securely). We prioritized this process for all keys with access to customer personal information and continued to migrate additional keys over time.
3. On 25 June 2024, we were made aware of that same hacktivist group claiming to have access to working API keys for our 3rd party services. We immediately began a forced rotation of all known secrets.
4. We have reviewed our logs and believe the only abuse of those keys was to send defamatory emails to rabbit employees, a small number of journalists who encourage the work of hacktivists, and other members of the hacktivist group.
5. We have reviewed the potential impact of these keys, and although no customer data was exposed in this event, consistent with our goals of remaining as transparent as possible, we describe below the roles and functions these keys have and the reason we have applied additional security controls to them:
   a. ElevenLabs
   i.

   This key is used to provide text-to-speech services for the r1 device, and has access to bulk pseudo-anonymized data.

   Example: When a user says, “What’s the weather in San Francisco,” we create a text response, “It’s 72 degrees and sunny in San Francisco.” That text is sent to ElevenLabs, and they return voice data that the r1 device will play.

   Someone with our ElevenLabs key could download “It’s 72 degrees and sunny in San Francisco,” but they would not know:

   1. Who that response was for
   2. What request was made to generate that response
   ii.

   This key could be used to modify global r1 voice settings, resulting in a temporary outage of voice responses. It is important to note that this would not “brick” the r1 device, which was mistakenly claimed in some third party online reports. While having access to this key might allow bad actors to disrupt the device as described above, it would not disable the functions of a user’s r1 or rabbithole account.

   iii.

   As previously mentioned, this key has been rotated, and a new key is now stored in AWS Secrets Manager.

   iv.

   In the coming weeks, we intend to disable history logging with ElevenLabs completely, so that this pseudo-anonymized data is no longer available.

   b. SendGrid
   i.

   This key is used to send emails from “rabbit@r1.rabbit.tech” to our customers as part of the spreadsheet feature. Its ability to send email is limited to <anything>@r1.rabbit.tech.

   ii.

   The leaked key did NOT have access to historical email data.

   iii.

   As previously mentioned, we have confirmed that no user data was accessed. However, we note that, because this key does have the ability to update the receiver of “spreadsheet revisions,” if attackers were to gain access to this key, they could potentially break the spreadsheet revision feature and gain access to the email address of the customer requesting a revision and the prompt used for the revision.

   For example, if someone uses their r1 to generate a spreadsheet, they will receive an email from rabbit@r1.rabbit.tech containing the spreadsheet. If they reply to the email, “switch column A and column B,” our service will do that. An attacker would NOT have access to the spreadsheet, but they would know that “myemail@gmail.com” has a spreadsheet, and they would know that the user wants to “switch column A and column B,” even though the attacker will not have access to the content of those columns.

6. We’ve received feedback that our 180 day timeline on our Vulnerability Disclosure Program (VDP) is longer than security researchers like. Most Series A startups don’t have a VDP at all, but we believe that building a relationship with good-faith security researchers is a key path to protecting our customers. We’ve heard the feedback, and have reduced this timeline to 90 days.
7. In addition to all the previously mentioned security measures, we’re hiring a 3rd party security firm to perform an independent audit of our codebase to verify that all historical secrets have been revoked. We expect that audit to be completed by the end of August, and plan to share those learnings as part of an engineering blog post.
8. Thank you again for allowing us to be part of your digital life. We are very grateful for the trust you’ve shown us. We hope that the transparency we’ve provided through these updates is helpful. If you have any additional questions, please feel free to reach out to support@rabbit.tech and we will do our best to address them.

18:00 PST 2024-06-27

On 25 June 2024, a hacktivist group published an article claiming that they had access to rabbit’s internal source code which included several API keys used to provide services to r1 customers. Upon discovery of the article, our team began the investigation and response process.

• We performed an inventory of all secrets currently in use.
• We began immediately revoking and rotating those secrets.

As part of the inventory process, we identified additional secrets that were not properly stored in AWS Secrets Manager.

As part of the rotation process, the team updated relevant portions of the codebase to ensure that all secrets were properly stored.

Actions in Progress

• We’re reviewing historical code versions for any additional secrets stored in code. Those secrets will be revoked as we identify them.
• We’re implementing automated code review checks that will prevent developers from committing secrets to our codebase.
• We are reviewing audit logs of our SaaS platforms to check for any theft of customer data. As of the publishing of this update, we have not found there to be any compromise of our critical systems or of the safety of customer data.

We will continue to use this page to provide updates as they become available.

13:00 PST 2024-06-26

On June 25, 2024, we were notified that a third-party may have had access to working API keys for multiple SaaS providers used by rabbit to provide services to our customers. Based on this notice, the rabbit security team rotated the keys to those APIs, which caused a brief downtime on the devices. Our team is continuing our investigation. As of today, we have not found there to be any compromise of our critical systems or of the safety of customer data.

We will use this page to provide updates as they become available. We’re grateful for the trust our customers have placed in us. Our top priority is maintaining that trust by protecting customer data and by providing transparency while we investigate.