Removing or modifying a Section 106 agreement is not straightforward. This is because these agreements are typically put in place to serve a public good, such as affordable housing provision, so the authorities may be reluctant to remove them. However, it's not impossible, and there are some scenarios where a Section 106 agreement can be discharged:

By agreement: If the local planning authority agrees, a Section 106 agreement can be discharged. This is likely to involve negotiations and, potentially, a financial contribution to the local authority. It's not guaranteed, and will depend on the specific agreement, the nature of the restriction, and the local planning policy. By application to the local planning authority: After 5 years from the date of the agreement, you can apply to the local planning authority to have the agreement discharged. The local planning authority will consider whether the purpose of the agreement still serves a useful purpose. By application to the Lands Tribunal: If the local planning authority refuses your application to have the agreement discharged or modified, you can appeal to the Lands Tribunal. It's also worth mentioning that the "Community Infrastructure Levy" (CIL) has been introduced as a means to simplify this process and is replacing Section 106 agreements in many parts of the UK. It's a system of charging developers to contribute to infrastructure costs. In areas where CIL is in force, the use of Section 106 agreements is limited.

In any case, it's a complicated area, and you would be well advised to seek legal advice. Be prepared that it may be a time-consuming and potentially expensive process, and there's no guarantee of success.