When you're writing a book that is going to be a narrative with characters and events, you're walking very close to fiction, since you're using some of the methods of fiction writing. You're lying, but some of the details may well come from your general recollection rather than from the particular scene. In the end it comes down to the readers. If they believe you, you're OK. A memoirist is really like any other con man; if he's convincing, he's home. If he isn't, it doesn't really matter whether it happened, he hasn't succeeded in making it feel convincing.