I think you can speak with people who know English, are not English native speaker and do not speak Mandarin.
This way, you must speak English with them because it is the only common language and you don't have to be afraid of making mistakes because they are not speaking flawless English.
It should be a step-by-step approach which caters to different people's needs.
If you are "有点点放不开", then speak with a non-native so that you can gain confidence in speaking in English.
When you have enough confidence and are feeling comfortable, start to speak with the native.