When you delete someone, none of their posts are deleted, including posts to any teams that they were in. All of the tasks, files, notes, etc. that they posted are still also accessible via search and the various apps.
You won't be able to directly get to your one-on-one conversation with that person any more (i.e., they won't be in the People section on the left pane or on the People page itself). However, you can find content post by that person, and even get back to the conversation, via search. For example, if there was a file posted in that conversation and you search for it, clicking on that search result will take you back into the conversation with that person. You can find content posted by that person by browsing one of the apps. For example, you could go to the Files app to find files posted by the person.