Problem
The email service is tightly coupled with the production hyper.media web service, and it has no reason to be.
This causes troubles when we need a hands-on approach to debug the email service. Our main bug is that the email service frequently fails and requires a restart before it will resume sending emails. This is probably due to an edge case that can be fixed, but the deep integration with the central server slows that down and causes us to put our production web service at risk, just to debug the email system.
Solution
Move the Email system into an external service hosted on Render. For the moment I prefer render over the self-administered docker workflow because it has a faster deployment. As a bonus there is a nice GUI to view the logs.
To keep the setup and deployment simple, we can copy the configuration and environment of the web app to the new notify app. This includes Remix but Alex Burdiyan and horacio have expressed concerns about using Remix again in this service.
Open Questions
Use gRPC of the daemon directly, or use HTTP wrappers?
If we use the gRPC directly, we should find a secure networking solution that does not publicly expose the production gRPC endpoint.
HTTP wrappers would be effective but they are a bit fragile because the services may need to be versioned together.
Possibly we should use the docker container workflow for the email service, instead of Render.
.