Account deletion @ bkmrkd.it

I’ve been struggling with the account deletion feature at bkmrkd.it. Specifically if users have large amounts of bookmarks. The Lambda proces time limit make it a bit trickier than just a normal program. You have to limit the amount of CPU per call. So the simple method first tried was to paginate (like browsing through your bookmarks) thourg the list of bookmarks and deleting them in batches using SQS for triggering each batch. This triggered another “hidden” feature I didn’t know about while testing this: Recursive loop detection and auto remediation.

At first I thought I missed something in my code as the deletion process didn’t finish as expected but stopped halfway. It all became clear the next day when I received an email notification from AWS:

Hello,

AWS Lambda has detected that one or more Lambda functions in your AWS Account: ########## are being invoked in a recursive loop with other AWS resources. To prevent unexpected charges from being billed to your AWS account, Lambda has stopped the recursive invocations listed in the ‘Affected resources’ tab of your AWS Health Dashboard.

This scared me at first but my trusty AI companion walked me though some of the options. The difficulty being that unlike relational databases there is no possibility do a “DELETE FROM TABLE WHERE COLUM = X”, you have to do a scan or query and then iterate trough the result set which is in itself timeconsuming on large datasets. As I stored 2 records for each bookmark, one for quick lookup and finding and one with all the details this made it increasingly difficult. So I did a rethink on the datamodel, which resulted in a single record per bookmark.

The records now all have the same PK which makes deleting everything from the same account easier. Another design choice was to move the bulk deletion out of Lambda and proces it locally. I now only delete the account details directly and leave the bulk deletion for a later moment outside of Lamdba.

By the way, the export functionality was quite the opposite and easy without significant problems. It is basically the same idea because of the time limit. I chunk through all the bookmarks, outside the main program and build a set of JSON files which I zip together at the end and send an email with a download link. Works like a charm.