Dec 2024 Update Page
Firstly, we received an email last month that someone was unhappy that we referred to the Chinese team as “the Chinese” or “the Chinese team” and they called us racist. We want to be very clear that we refer to our teams by their locations simply as a way to distinguish them to you all geographically, such as “the Chinese team” for the team we hired from China and “the Ukrainian team” for the team we hired from Ukraine. This is not meant to encapsulate the entirety of China, Ukraine, or their cultures, and we are not racist.
In our last update, we shared the progress we’ve made and explained the only major setback we encountered was that the Over-the-Air (OTA) updates—used to remotely update the safes—were unexpectedly resetting the safes’ unique serial numbers. We suspect when the Chinese team handed over the files, perhaps they deliberately sabotaged the system to create unnecessary challenges for us. Who knows. Regardless, while this is deeply frustrating, we’ve been working tirelessly to identify and resolve the problem. Since our Ukrainian team didn’t handle the firmware from the start, this has taken longer than anticipated. We sincerely apologize for the delay and appreciate your patience.
In Ukraine, we have separate engineers working on (A) the firmware and (B) the PCBA (Printed Circuit Board Assembly) development. As you read through this update, please note that we don’t always specify which engineering team is involved. Keep in mind that we have two different Ukrainian teams, each focused on their respective areas of expertise.
Our firmware engineers have been trying many different approaches, and to be honest, we had expected to find a solution by now. Unfortunately, we’re not there yet. So far, our attempts have been only 67% successful when upgrading the system. The other 33% of firmware updates fail unpredictably during installation, even though the OTA downloads the update successfully on the safe. This randomness makes it incredibly challenging to identify and fix the root cause. When these failures occur, the system enters recovery mode, effectively bricking the safe. At that point, the safe becomes completely non-functional, and the only way to restore it is through a specialized process that requires our team’s expertise. This involves reformatting the system, reinstalling an older firmware version, and trying a different approach.
Why is it important we resolve before MP / shipping Trail Product? If we release an OTA update with new features or bug fixes, we cannot risk 33% of safes becoming non-functional every time there’s an update. We know this delay is frustrating for everyone, but we are working tirelessly to resolve the issue. To expedite the process, we’ve even brought back a consultant who helped us last year. Rest assured, many brilliant minds are collaborating to solve this problem.
Another issue that’s come to light is that our Ukrainian PCBA engineers have identified a hardware problem with the data/address line matching on the eMMC. This issue could theoretically be contributing to data inconsistencies and may even explain why updates are failing 33% of the time. Unfortunately, this hardware problem stems from the last PCBA design developed by the Chinese team in 2023. Our Ukrainian PCBA engineers are working on resolving this in the next PCBA version. Addressing this issue and trying to figure out if this is the cause for the failures is complex and a time-consuming process.
For the next monthly update, our engineers will work together to determine if the data/address line issue is affecting the OTA functionality and overall system performance. If we confirm this is the cause, we will first complete the new PCBA redesign and conduct the proper testing (something obviously the Chinese did not do). Only then can we produce new PCBAs for the trial production safes before finally moving forward with their shipping.
Just a heads up for our next monthly update:
We don’t anticipate having much new information, as most of our engineers will be unavailable over the next two weeks due to the holidays.
Anything else we should be aware of?
- Yes, on a positive note, we are still in the process of the main PCBA board development. Until this is complete, we cannot proceed to mass production. Thus far the development has been going really well, and our Ukrainian team has been far more communicative and responsive compared to when we worked with the Chinese team.
- Yes. Several months ago, when we received the files from the Chinese team, our files contained a total of seven PCBAs. Aside from the main PCBA board, the files for the six PCBAs were all incomplete, which would require weeks of effort to recreate the missing pieces for those boards (something we've previously mentioned). Unfortunately, we only realized last month that one additional PCBA was entirely missing — the files for the camera IR LEDs. This brings the total to eight PCBA boards needed for our device, but we had only received seven.
What are the camera IR LEDs? These are responsible for activating invisible white lights when the room is dark, allowing the camera to see. In a previous monthly update, we mentioned discovering a single IR LED in the camera that the Chinese team had failed to inform us about. However, the missing PCBA file relates to a different design containing six IR LEDs, which is distinct from the single IR LED in the camera (you can see the image below for reference).
What does this mean, and how are we resolving it? For the nearly the past 4 weeks, we’ve been reaching out to the Chinese team daily. Most of the time, there has been no response. On the rare occasion they do reply, they’ve been unhelpful, claiming to have “deleted our files” from their systems and backups — a claim we don’t find credible. And we are hoping for better news with them.
While we’ll continue reaching out to the Chinese in hopes of recovering the missing files, our Ukrainian team will move forward with redesigning the IR LED PCBA from scratch. Fortunately, this PCBA is not overly complex, and we estimate the redesign will take about 6-8 weeks from start to finish. To save both time and money, we will continue pursuing the Chinese team for these files. However, based on our experience with them, we do not expect to receive the files and have accounted for this additional time in our internal projected completion date.
We kindly request your cooperation in refraining from sending emails requesting delivery dates and any negative messages, as we no longer plan to respond to those individual inquiries. Rest assured, we're actively working to expedite the process and will provide updates as soon as they become available. Your understanding and patience during this time is greatly appreciated.
Sorry for the delay. Thank you for your unwavering support, your understanding and patience during this time are greatly appreciated.
We want to remind everyone that crowdfunding is a unique opportunity to help fund ideas that you care about, and be a part of bringing projects to life. We understand that it can be discouraging when a campaign you have funded experiences delays; such as ours. However, as with any early-stage project, crowdfunding campaigns may take unexpected turns, which can include changes, delays or unforeseen challenges; which you have all seen from our updates over the last three years. While your contribution helps to bring something new closer to reality, we ask that you continue to have patience until our project is complete.
TLDR - SUMMARY
-
Keeping serial numbers when upgrading is still not resolved (when attempting both manually and OTA).
-
The Chinese didn’t provide us with the PCBA for the IR LED, and it will need to be redrawn from scratch (up to 8 weeks).
-
The new PCBA is currently being worked on 🎉.
-
If all goes well, we estimate the trial production will ship out in 1-2 months.
-
Mass production won’t start to ship before July 2025.
Thanks for your support!