The integration of Smart Grid (SG) with cloud computing promises to develop an improved energy management system for utilities and consumers. New applications and services are developed which create large amount of data to be processed on cloud. Fog computing as an extension of cloud computing which helps to mitigate load on cloud data centers. In this paper, a three layered model based on cloud and fog framework is proposed to reduce load of consumers and power generation system. End user layer contains clusters of buildings which are connected to fog server layer. Fog layer is an intermediate layer which connects the end user layer to cloud layer. Three load balancing algorithms Round Robin (RR), throttled and proposed Particle Swarm Optimization with Simulated Annealing (PSOSA) are used for resource allocation. The service broker policy considered in this paper is optimized response time. The findings demonstrate that PSOSA performs better than RR and throttled in order to alleviate response time, processing time and cost of virtual machine, microgrid and data transfer.