Should you Delete or Soft Delete?

[@zimcoder]

Sometimes a user requests that all the data be deleted permanently, you can't just make another entry elsewhere that they requested to be deleted and capture data on that event because you will be violating their right to be forgotten. Sometime a delete is just a delete.

[@amlyo6722]

right-to-be-forgotten requests can be incorporated into the design of ledger based systems with appropriate encryption and a forgettable key per customer.

[@zimcoder]

@@amlyo6722 that simply means you have just hidden their data and not forgotten it... A sophisticated soft delete

[@amlyo6722]

@@zimcoder If the person who does your backups ever tells you this, it's time to start panicking

[@Andrej.]

🤣

[@kanadaj3275]

@@amlyo6722 I mean, he isn't exactly wrong. Forgetting an encryption key simply means it's "forgotten for the foreseeable future", not actually "forgotten". Someone 10 years from now could obtain a leaked dump of your database and decrypt it using, say, a quantum computer. It's not exactly a clear-cut issue.

[@everydreamai]

A lot of your videos hit on a very important topic without actually explicitly getting into it. It's very important to have a competent product owner who can focus on delivering their team with meaningful acceptance criteria for actual business cases instead of things that look like CRUD operations. This is often a challenge with businesses that do a poor job aligning teams with business functionality and not technical considerations.

[@CodeOpinion]

You got the point of the video!

[@TheRicherthanyouguy]

I can 100% support this mind set. I’ve worked in large scale and super small projects and in my experience I’ve started evolving away from a soft delete an instead use some kind of enumeration or status code id usually stored in a database of some sort to define that current status of data. I think a lot of the frustration caused here is generally because these type of thinking does require user interfaces and queries to be a bit smarter than we might like but I’ve found it’s usually worth it.

[@ddruganov]

A great example of domain specific language usage!

[@sethdodson5475]

One notable exception: if your business wants to comply with European privacy laws and receives a request to delete PII for an individual, then deletion truly is in line with a business process.

[@hemslingo7219]

Very easy situation to get into. The most annoying repercussion I've come across is having to omit IsDeleted entities throughout the entire application, including navigation properties. If you're selecting from the db at the root entity level, it's very difficult to deal with in a large app.

[@cdnkarasu]

Soft delete is useful for adding a versioning mechanism. Also an alternative is to use temporal tables.

[@grumpydeveloper69]

Even when from a business standpoint you don't think of hard deletes, regulations like the GDPR and right to be forgotten laws would still requires systems to be able to hard delete. How do you handle such regulatory requirements?

[@CodeOpinion]

Delete if you need to delete.

[@sangmin7648]

Thanks for the video! Choosing between event sourcing and crud seems to be a difficult job. How should I know when to use which? And later down the line, when I realize that I should’ve gone with the other method, how should I make the change? Changing from event sourcing to crud seems pretty simple, as I already have all the data. But how should I change from crud to event sourcing? Or should I just event source everything to be safe?

[@CodeOpinion]

It's not about CRUD vs Event Sourcing. You could be recording current state in the way I showed it with a document and having a collection that recorded when they were hired and terminated.

[@Thorarin]

And then GDPR became a thing for many of us, so we cannot just keep data around indefinitely. Eventually you will have to hard delete if it applies to your business.

[@fabiogusmaoribeiro]

Expanding on the subject of soft-delete, do you believe that adding "created by", "created at", "updated by", "updated at", "deleted by" and "deleted at" columns to your database is redundant when you can grab that information from the event store?

[@CodeOpinion]

Depends how those columns are being used, I suspect just for information/query/view purposes. If you're in CRUD mode, then yes. If you're not then you probably don't have those columns but more-so like a CancelledDate, EmploymentTerminatedDate, etc.

[@chadgregory9037]

about 15 minutes of real, actual work.

[@CodeOpinion]

ha!

[@MiningForPies]

Soft deletes are the bane of my existence. Got a system I’ve inherited where the soft delete flag means something different depending on the entity it’s on. And query/report performance is crap because it’s got to be included in every one

[@CodeOpinion]

Yup! It's not really "deleted" it has some explicit meaning that is different in a different entity/context but it's implied with isDeleted. Not fun. Make the implicit explicit.

[@br3nto]

It would be helpful if you describe what the database schema and database changes would be for the evented version. It’s easy to see how soft delete works because you showed the schema with the IsDeleted column and how/when it is updated. But it’s not clear on what the db schema is and how data is managed in the evented version.

[@CodeOpinion]

If your were event sourcing, the individual events define their own schema. All the events for a particular aggregate are persisted in a stream. This video might be helpful if you haven't watched already. kzbin.info/www/bejne/d4bNZYBjqNlmn8U

[@AaronInCincy]

How do you handle mis-keyed data in a task based system? Let's say an employee moved, and you go through the workflow of entering a new address. You click save, the system kicks off an employee moved event and everything is great. Then you noticed you miskeyed their street name. Do you have another employee moved event? They didn't really move twice. Do you have an AddressCorrected event? At that point do you have to support task based AND crud based events for every piece of data?

[@CodeOpinion]

Task based doesn't imply you're publishing events. It sure helps to do so, but performing a command doesn't mean it has to publish an event.

[@rauldeandrade]

The employee didn't move twice, but they also didn't move to the first address. I'd rather just create a second move and maybe add a field "reason" for moving where I can say "previous move error" under EmployeeAddedAddress event. Although I'm not against deleting an address, I must confess I'd stress out about the address use dependencies elsewhere. I'm just thinking off the top of my head. I don't really have an implementation example come to mind where I had to address this before

[@FISS007]

I totally agree with you. There are even some regulations and laws that forbid data deletion and enforce keeping data history.

[@ramonennik2536]

That's was the same question I had! For example we do have accounts where all personal data has to be wiped or scrambled when they want to delete their account because of the GDPR. (So the other way around, we must delete the data by law)

[@CodeOpinion]

The intent of the video is to illustrate that often there are business concepts to capture that soft deleting doesn't provide. Capture business concepts if applicable. If you need to delete data, delete it.

[@ronifintech9434]

Looks like this is a good approach for nosql? how do you implement something like this for SQL?

[@marna_li]

"Why store all these events and data taking up space?" - Because of auditing. "But I don't need that" - So be it, this is just a concern that might become important to your business later down the road. You need to know that this is a way of thinking about your system. And you don't have to go all in on event sourcing for this either.

[@CodeOpinion]

Yes, capturing business intent has many benefits, one being auditing.

[@Oswee]

Nah... you should cover GDPR where users has rights to wipe out their data. So you should wipe out all sensitive marked data. You can't keep it in original form.

[@MiningForPies]

Only if the data is covered by gdpr

[@CodeOpinion]

Correct. This video has nothing to do with GDPR. There are many ways of handling, one of which is deleting data. If you need to delete data because of GDPR, sure. You could also encrypt data and lose the key. The point of the video is to illustrate to capture business concepts explicitly.

[@Oswee]

Then i didn't got a message of this video correctly. My point was that there are more than Delete vs Soft delete.

[@ColinWhittingham]

In that case could you just anonymize any identifiable fields in existing data (name, address, email etc.)? Would this satisfy GDPR whilst preserving most of the events?

[@Oswee]

@@ColinWhittingham This is how this is typically done. But you must do the homework to avoid the hunting for data traces.

[@ndev92]

What if you Aggregate Root has a collection of items, and then somewhere in your code you write AggregateRoot.Collection.Remove(item)? In my case the collection is lets say an Assessment and the collection is Reviewers. But I want to remove one of the Reviewers of the Assessment. Lets say I am using EF and SQL database, What would be the best way to handle that?

[@markovcd]

What about GDPR?

[@CodeOpinion]

What about it? If you need to delete data, delete it. The point of the video is to capture business concepts if applicable. If you need to delete data, go ahead.

[@chengchen9032]

yean totally agree that events are more informative than crud, but I feel like the storage of events could be challenging. do you have any advice on that? let's say one entity is a row in the sql db, if its crud, no matter how many events happened in the domain, cuuud ends up with d, one row less for the table, however, it would be five events to stored if we do event sourcing.

[@CodeOpinion]

You don't need to do event sourcing to capture the business concept. As I illustrated with the document example that's recording current state (not event sourcing) I had a collection that recorded the hiring and termination. If you were using tables, you'd probably have table with columns for HireDate, TerminationDate, etc. With a single record for their employment.

[@chengchen9032]

@@CodeOpinion makes total sense, I got it, so a row with termination date being empty/default value when created. Then update to fulfill that information when the corresponding event happened.

[@everydreamai]

This specific example is unlikely to scale as I think a business is unlikely to hire and fire one person thousands of times, so just shoving the data in "EmployeeHistory" would probably suffice. That would apply whether you are using events/pub/sub or just an API/ORM, and for RDBMS or a NoSQL store. It is a concern if you are talking about something with a much higher cardinality. Say like clock in/out events for laborers. You might have several per day (breaks, etc) per employee, seeral thousands per year, and then it grows forever as long as someone is employed and working. You have to question then what the business case to read the data back out later is and how you will access it. It may make sense that data is stored by PayPeriod + EmployeeId, which roughly fixes your cardinality to maybe a few dozen or couple hundred no matter how many employees you have and how long they work at the company. To some extent, you could say Derek seems to be putting the cart in front of the horse, he's coming as a developer in a video presumably focused on providing information to other developers, but business should be the main driver to store this extra tabular dimension (history) instead of just a scalar or two (i.e. current employment status or hire/fire dates) It's a good thing to bring up from the dev side, though. There are a lot of really good talking points here, and knowing the difference between the scalar and tabular or event aggregation is good knowledge to have kicking around in your head. It's this sort of knowledge that separates someone who blindly completes a user story and someone who can help steer their business away from information black holes.

[@mahmutjomaa6345]

So you would add data with more information (i.e. EmployeeTermination and EmployeeHiring) instead of toggling an isDeleted flag because the facts of what happened might be important.

[@CodeOpinion]

Correct. It's about capturing the business concept (where appropriate).

[@ujwaldhakal6004]

Do you delete user related data from even store to comply with GDPR?

[@CodeOpinion]

You could delete relevant streams or rebuild them with the relevant user related data omitted.

[@sodreigor]

I can think of several occasions where you should actually hard delete things. The first one that comes to mind is: If I click "Terminate My Account" on facebook for instance, Facebook should hard delete all of the media that I uploaded there. Now the question is: Will facebook actually delete it? I don't know. But should facebook hard delete it? Yes.

[@sodreigor]

Just as a complement. I understand that it could exists some information that should be anonymized but still kept for integrity of references to some other data. But the majority of personal data should be hard deleted.

[@testuser429]

If you want to be GDPR compliant, you should be able to erase all user's personal data.

[@wboumans]

Derek do you use C4 diagrams? If so would you ever do a primer on them?

[@CodeOpinion]

I don't but I should get maybe Simon on here to do a quick one about it.

[@void_star_void]

What about the right to be forgotten? instances when you ask the company to delete your data

[@CodeOpinion]

Then delete the data. The point of the video really is to capture business concepts explicitly when traditionally you would think about deleting or soft deleting data.

[@dasfahrer8187]

I'm pretty sure if I'm Peter I want to hard delete all TPS reports.

[@CodeOpinion]

Yeah. The coversheet. I know, I know. Uh, Bill talked to me about it.

[@dasfahrer8187]

@@CodeOpinion Time to go fishing.

[@roko567]

Soft deleting user info is illegal in Europe

[@FrantisekSidak]

You can anonymize, and if I know, in special cases, you have obligation to keep data anyway (accounting). So the concept still makes sense.

[@mybackhurts247]

Peter Gibbons: what would you say you do here? do you deal with the gd customers so the engineers don't have to? do you have people skills?

[@CodeOpinion]

🤣

[@JackiePrime]

Doesn't this make your life as a developer much more difficult? For example we use soft delete, and in our EF database context we filter out based on IsDeleted == false as the default. Only when we explicitly requested deleted data is it returned from the API. This means we don't have to think about deleted data. If we went with your approach, we would have to have seperate logic for filtering out deleted records for every entity.

[@CodeOpinion]

Absolutely would make your life more difficult if your focusing on crud. If your viewing things based on business concepts it's not difficult at all since the concept of deleting likely doesn't exist.

[@brandonpearman9218]

so what you are saying hireEvent = add to db, and terminateEvent = update db... so your answer is softdelete.

[@CodeOpinion]

Well the point I'm trying to make is the word "delete" makes no sense. You're not deleting anything, you're capturing data related to a business concept (hiring/terminating).

[@TheAceInfinity]

Answering the question of whether to hard delete vs soft delete with the the notion of "adding" things doesn't make any sense. It still doesn't answer the question on whether you should use one or the other because the it's irrelevant to the question, with no alternative provided, and assumes extremely contextual business application requirements that do not apply universally! There's lots of cases where recording a chronological order of events is not the right solution. Not a great answer to the question IMHO.

[@CodeOpinion]

You answered it yourself, its "extremely contextual". There's no single answer to the question. The point with the adding is there are more options beyond the soft/hard delete.

[@TheAceInfinity]

@@CodeOpinion That was my point, although I was thinking this video would be about hard and soft delete exclusively and when to choose between one or the other based on the title.

[@remigiuszkowalski7405]

​ @CodeOpinion I've read the comments :D Please stop explaining purpose of this video to everyone. It's like riding a bicycle with inverted handlebars - your brain needs to adopt first.

[@CodeOpinion]

Well I clearly didn't do a good enough job in the video or didn't anticipate the common reaction.

[@remigiuszkowalski7405]

​@@CodeOpinion I disagree. You did perfect job! I've been watching your previouse videos when I was learning DDD approach and I had same problems - i was asking question from my limited context. It is so clear for me now and even funny :D