Prince Harry is reflecting on a major turning point in his relationship with brother Prince William. In the second instalment of their Netflix docuseries, Harry & Meghan-which dropped Dec. 15-Harry and his wife Meghan Markle recalled the royal family’s reaction to their decision to step back from their duties in 2020. As the Duke of Sussex explained, a meeting with his family took place at the Sandringham House shortly after the announcement, which he says invoked a notable response. “It was terrifying to have my brother scream and shout at me and my father say things that simply weren’t true and my grandmother quietly sit there and take it all in,” Harry said. Referring to his grandmother, he added, “But you have to understand that from the family’s perspective, especially from hers, there are ways of doing things, her ultimate mission goal slash responsibility is the institution.” The Duke of Sussex also reflected on the impact he feels it had on his relationship with his older sibling. “The saddest part of it was this wedge created between myself and my brother so that he’s now on the institution’s side,” Harry said. “Part of that I get, I understand, right, that’s his inheritance. So, to some extent, it’s already ingrained in him that part of his responsibility is the survivability and continuation of this institution.” Elsewhere, Harry and Meghan also recalled the late Queen inviting them to the stay the night shortly after news broke of their decision, but that the invitation was quickly forfeited. “This urgent message comes through to H saying, ‘You are not allowed to go and see Her Majesty,'” Meghan said. “She is busy; she has plans all week.” As for Harry, once he got in touch with his grandmother directly, the situation became clear. “She said, ‘Yes I didn’t know that I’m busy, I’ve been told that I’m busy all week,'” he recalled. “I was like, ‘Wow.'” Harry also slammed the notion that the couple “blindsided” the Queen with their move to North America. “This would never happen,” he shared. “I have so much respect for her.” E! News has reached out to Buckingham Palace for comment and has not heard back. All episodes of Harry & Meghan-produced in part by the couple’s Archewell Productions-are available for streaming on Netflix.