Carer's right to have interests considered
In your case I think you need to make a firm stand. If you are being asked to do something that you feel is too much, say so - you have the backing of the doctor. I think the social worker needs reminding that no family member is under any obligation to do anything whereas social services are. Moreover, remind them that you are acting as primary carer - your not being a close relative is irrelevent - and that if your own needs aren't given consideration, then your own health will likely fail and/or you won't be able to cope...in which case you won't be able to do the caring and they will have to!
I believe that Social Services always work under the assumption that carers are "coping" unless they are told loud, clear and often that we aren't.
It is by no means in any way "selfish" - you have a right to a life of your own, your own happiness and fulfillment, you are not simply a drudge who exists only to service the needs of a relative. If this doesn't happen, then your own health will suffer, and what will happen then?
For myself I'd pluck up courage and say to the Social Worker "What would happen if I just walked away? Because that's exactly what I feel like doing!". You could also try and enlist the support of your own GP. If you were employed in a paid job and under this level of stress you would be probably be signed off sick - and likely told to resign before you have a nervous breakdown. I sometimes think that advantage is taken over some carers because they know we can't or rather won't "resign".
Social Services almost certainly don't want you to walk away - that would saddle them with the problem - and it could be a very expensive one if they have to provide paid carers to do what you are currently doing.
I agree, probably the first thing to do is to write to the director of the local social services department and express your concerns.