Optimizing HVAC&R System Efficiency and Comfort Levels Using Machine Learning-Based Control Methods
Article Sidebar
Main Article Content
Abstract
The Heating, Ventilation, Air Conditioning, and Refrigeration (HVAC&R) system is a complex, nonlinear behavior with a high uncertainty control system that equips the thermal comfort desired but consumes significant electrical energy and costs in different types of buildings, such as residential, commercial, and industrial. This paper introduces a new approach for online controlling of HVAC&R systems using model-based reinforcement learning (MB-RL) style to diminish energy usage and energy cost, maintain the occupants’ comfort levels by controlling the buildings' indoor temperature, and maintain the desired carbon dioxide levels simultaneously. For this purpose, a new model based on energy and mass conservation laws is presented to model the dynamic variations of temperature and CO2 concentration levels. The HVAC&R system control trouble is defined as a specific Markov Decision Processes (MDPs) model. The reward function balances the ability to increase energy conservation while preserving the interior comfort requirements of occupants. Employing the deterministic policy algorithm (DP), the proposed methodology can manage the dimensionality curse problem due to increased state-action space. Then, it overcomes the nonlinearity and the control system uncertainty. The MB-RL algorithm, which uses a unique DP called DP-MB-RL, can select the best decisions instead of stochastic policy to reduce the calculation time. A real case, a building in Basra City, Iraq, is simulated using MATLAB software. Devoting the MB-RL and DP-MB-RL techniques to online control of an HVAC&R system, the simulation results for both methods are provided. For instance, the parameters, like electrical power, internal comfort levels, energy consumed, and energy cost at different pricing schemes, such as fixed pricing (FP), time-of-use (TOU), and real-time pricing (RTP), are assessed. The results indicated that the suggested DP-MB-RL methodology had better indoor thermal and air quality satisfaction levels, energy-saving (more than 15%), and reduced the cost of electricity by more than 15%, 13%, and 10% for FP, TOU, and RTP pricing schemes, respectively, compared to the benchmark MB-RL style controller. The DP-MB-RL controller also performed better than the Takagi-Sugeno Fuzzy (TSF) controller for the same building, saving more than 21% energy.
Metrics
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
THIS IS AN OPEN ACCESS ARTICLE UNDER THE CC BY LICENSE http://creativecommons.org/licenses/by/4.0/
Plaudit
References
Chen Y, Norford LK, Samuelson HW, Malkawi A. Optimal Control of HVAC and Window Systems for Natural Ventilation Through Reinforcement Learning. Energy and Buildings 2018; 169:195-205. DOI: https://doi.org/10.1016/j.enbuild.2018.03.051
Homod RZ, Togun H, Abd HJ, Sahari KSM. A Novel Hybrid Modelling Structure Fabricated by Using Takagi-Sugeno Fuzzy to Forecast HVAC Systems Energy Demand in Real-Time for Basra City. Sustainable Cities and Society 2020; 56(June 2019):102091. DOI: https://doi.org/10.1016/j.scs.2020.102091
Du Y, Zandi H, Kotevska O, Kurte K, Munk J, Kadir A, Evan M, Fangxing L. Intelligent Multi-Zone Residential HVAC Control Strategy Based on Deep Reinforcement Learning. Applied Energy 2021; 281(November 2020):116117. DOI: https://doi.org/10.1016/j.apenergy.2020.116117
Homod RZ, Almusaed A, Almssad A, Jaafar MK, Goodarzi M, Sahari KS. Effect of Different Building Envelope Materials on Thermal Comfort and Air-Conditioning Energy Savings: A Case Study in Basra City, Iraq. Energy Storage 2021; 34:101975. DOI: https://doi.org/10.1016/j.est.2020.101975
Homod RZ, Gaeid KS, Dawood SM, Hatami A, Sahari KS. Evaluation of Energy-Saving Potential for Optimal Time Response of HVAC Control System in Smart Buildings. Applied Energy 2020; 271(August):115255. DOI: https://doi.org/10.1016/j.apenergy.2020.115255
Zhao H, Zhao J, Shu T, Pan Z. Hybrid-Model-Based Deep Reinforcement Learning for Heating, Ventilation, and Air-Conditioning Control. Frontiers in Energy Research 2021; 8:412. DOI: https://doi.org/10.3389/fenrg.2020.610518
Kim NK, Shim MH, Won D. Building Energy Management Strategy Using an HVAC System and Energy Storage System. Energies 2018; 11(10):2690. DOI: https://doi.org/10.3390/en11102690
Zhang Z, Chong A, Pan Y, Zhang C, Lam KP. Whole Building Energy Model for HVAC Optimal Control: A Practical Framework Based on Deep Reinforcement Learning. Energy and Buildings 2019; 199:472-490. DOI: https://doi.org/10.1016/j.enbuild.2019.07.029
Kurte K, Munk J, Kotevska O, Amasyali K, Smith R, Mckee E, et al. Evaluating the Adaptability of Reinforcement Learning Based HVAC Control for Residential Houses. Sustainability 2020; 12(18):1-38. DOI: https://doi.org/10.3390/su12187727
Vázquez-canteli J, Ulyanin S, Kämpf J, Nagy Z. Fusing TensorFlow with Building Energy Simulation for Intelligent Energy Management in Smart Cities. Sustainable Cities and Society 2018; 45:243-257. DOI: https://doi.org/10.1016/j.scs.2018.11.021
Azuatalam D, Lee W, Nijs F De, Liebman A. Reinforcement Learning for Whole-Building HVAC Control and Demand Response. Energy and AI 2020; 2:100020. DOI: https://doi.org/10.1016/j.egyai.2020.100020
Bragagnolo SN, Schierloh RM, Vega JR, Vaschetti JC. Demand Response Strategy Applied to Planning the Operation of an Air Conditioning System. Application to a Medical Center. Journal of Building Engineering 2022; 57:104927. DOI: https://doi.org/10.1016/j.jobe.2022.104927
Kang J, Weng S, Li Y, Ma T. Study of Building Demand Response Method Based on Indoor Temperature Setpoint Control of VRV Air Conditioning. Buildings 2022; 12(4):415. DOI: https://doi.org/10.3390/buildings12040415
Ahmad MW, Mourshed M, Yuce B, Rezgui Y, Chen Y, Norford LK, et al. Computational Intelligence Techniques for HVAC Systems: A Review. Building Simulation 2016; 9(4):359-398. DOI: https://doi.org/10.1007/s12273-016-0285-4
Seyedzadeh S, Rahimian FP, Glesk I, Roper M. Machine Learning for Estimation of Building Energy Consumption and Performance: A Review. Visualization in Engineering 2018; 6(1):1-20. DOI: https://doi.org/10.1186/s40327-018-0064-7
Moubayed A, Injadat M, Nassif AB, Lutfiyya H, Shami A. E-Learning: Challenges and Research Opportunities Using Machine Learning Data Analytics. IEEE Access 2018; 6:39117-39138. DOI: https://doi.org/10.1109/ACCESS.2018.2851790
Zhang C, Kuppannagari SR, Kannan R, Prasanna VK. Building HVAC Scheduling Using Reinforcement Learning via Neural Network-Based Model Approximation. The 6th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, New York, USA, 2019; 287-296. DOI: https://doi.org/10.1145/3360322.3360861
Ding X, Du W, Cerpa AE. MB2C: Model-Based Deep Reinforcement Learning for Multi-Zone Building Control. The 7th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, New York, USA, 2020; 50-59. DOI: https://doi.org/10.1145/3408308.3427986
Ahn KU, Park CS. Application of Deep Q-Networks for Model-Free Optimal Control Balancing Between Different HVAC Systems. Science and Technology for the Built Environment 2020; 26(1):61-74. DOI: https://doi.org/10.1080/23744731.2019.1680234
Yuan X, Pan Y, Yang J, Wang W, Huang Z. Study on the Application of Reinforcement Learning in the Operation Optimization of HVAC System. Building Simulation 2020; 14:75-87. DOI: https://doi.org/10.1007/s12273-020-0602-9
Qiu S, Li Z, Li Z, Zhang X. Model-Free Optimal Chiller Loading Method Based on Q-Learning. Science and Technology for the Built Environment 2020; 26(8):1100-1116. DOI: https://doi.org/10.1080/23744731.2020.1757328
Dalamagkidis K, Kolokotsa D, Kalaitzakis K, Stavrakakis GS. Reinforcement Learning for Energy Conservation and Comfort in Buildings. Building and Environment 2007; 42(7):2686-2698. DOI: https://doi.org/10.1016/j.buildenv.2006.07.010
Fazenda P, Veeramachaneni K, Lima P, Reilly UO. Using Reinforcement Learning to Optimize Occupant Comfort and Energy Usage in HVAC Systems. Ambient Intelligence and Smart Environments 2014; 6(6):675-690. DOI: https://doi.org/10.3233/AIS-140288
Ruelens F, Claessens BJ, Vandael S, Schutter B De, Member S, Babuška R, Belmans R. Residential Demand Response of Thermostatically Controlled Loads Using Batch Reinforcement Learning. IEEE Transactions on Smart Grid 2016; 8(5):2149-2159. DOI: https://doi.org/10.1109/TSG.2016.2517211
Ruelens F, Iacovella S, Claessens BJ, Belmans R. Learning Agent for a Heat-Pump Thermostat with a Set-Back Strategy Using Model-Free Reinforcement Learning. Energies 2015; 8(8):8300-8318. DOI: https://doi.org/10.3390/en8088300
Vázquez-Canteli J, Kämpf J, Nagy Z. Balancing Comfort and Energy Consumption of a Heat Pump Using Batch Reinforcement Learning with Fitted Q-Iteration. Energy Procedia 2017; 122:415-420. DOI: https://doi.org/10.1016/j.egypro.2017.07.429
Chen B, Cai Z, Bergés M. Gnu-RL: A Precocial Reinforcement Learning Solution for Building HVAC Control Using a Differentiable MPC Policy. The 6th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, New York, USA, 2019; 316-325. DOI: https://doi.org/10.1145/3360322.3360849
Wang Y, Velswamy K, Huang B. A Long-Short Term Memory Recurrent Neural Network-Based Reinforcement Learning Controller for Office Heating Ventilation and Air Conditioning Systems. Processes 2017; 5(3):46. DOI: https://doi.org/10.3390/pr5030046
Zhang Z, Chong A, Pan Y, Zhang C, Lu S, Lam KP. A Deep Reinforcement Learning Approach to Using Whole Building Energy Model for HVAC Optimal Control. Building Performance Analysis Conference and Simbuild, ASHRAE and IBPSA-USA, Chicago, 2018; 22-23.
Polydoros A, Nalpantidis L. Survey of Model-Based Reinforcement Learning: Applications on Robotics. Intelligent & Robotic Systems 2017; 86(2):153-173. DOI: https://doi.org/10.1007/s10846-017-0468-y
Forootan MM, Larki I, Zahedi R, Ahmadi A. Machine Learning and Deep Learning in Energy Systems: A Review. Sustainability 2022; 14(8):4832. DOI: https://doi.org/10.3390/su14084832
Heidari A, Mar F, Khovalyg D. Reinforcement Learning for Occupant-Centric Operation of Residential Energy System: Evaluating the Adaptation Potential to the Unusual Occupant's Behavior During COVID-19 Pandemic. CLIMA 2022 Conference The 14th REHVA HVAC World Congress, Rotterdam, 2022; 1-7.
Ardabili S, Abdolalizadeh L, Mako C, Torok B, Mosavi A. Systematic Review of Deep Learning and Machine Learning for Building Energy. arXiv preprint arXiv:2202.12269 2022; 10(March):1-19. DOI: https://doi.org/10.3389/fenrg.2022.786027
Biemann M, Scheller F, Liu X, Huang L. Experimental Evaluation of Model-Free Reinforcement Learning Algorithms for Continuous HVAC Control. Applied Energy 2021; 298(May):117164. DOI: https://doi.org/10.1016/j.apenergy.2021.117164
Hussein LA, Ateeq AA, Homod RZ. Energy Saving by Reinforcement Learning for Multi-Chillers of HVAC Systems. 2nd International Multi-Disciplinary Conference Theme: Integrated Sciences and Technologies, IMDC-IST 2021, Sakarya, Turkey, 2021; 118.
Dawood SM, Hatami A, Homod RZ. Trade-Off Decisions in a Novel Deep Reinforcement Learning for Energy Savings in HVAC Systems. Journal of Building Performance Simulation 2022; 15(6):809-831. DOI: https://doi.org/10.1080/19401493.2022.2099465
Gao G, Li J, Wen Y. Energy-Efficient Thermal Comfort Control in Smart Buildings via Deep Reinforcement Learning. arXiv preprint arXiv:1901.04693 2019; 1-11.
Jiang Z, Risbeck MJ, Ramamurti V, Murugesan S, Amores J, Zhang C, et al. Building HVAC Control with Reinforcement Learning for Reduction of Energy Cost and Demand Charge. Energy and Buildings 2021; 239:110833. DOI: https://doi.org/10.1016/j.enbuild.2021.110833
Abdulgader M, Lashhab F. Energy-Efficient Thermal Comfort Control in Smart Buildings. 2021 IEEE 11th Annual Computing and Communication Workshop and Conference (CCWC), NV, USA, 2021; 0022-0026. DOI: https://doi.org/10.1109/CCWC51732.2021.9376175
Jabari F, Mohammadi-ivatloo B. Short-Term Co-Optimization of Multi-Chiller Plants and Ice Storage System. 2018 Smart Grid Conference (SGC), 2018; 1-6. DOI: https://doi.org/10.1109/SGC.2018.8777869
Ma G, Wang Z, Yuan X, Zhou F. Improving Model-Based Deep Reinforcement Learning with Learning Degree Networks and Its Application in Robot Control. Journal of Robotics 2022; 2022(1):7169594. DOI: https://doi.org/10.1155/2022/7169594
Dazeley R, Vamplew P, Cruz F. Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey. Neural Computing and Applications 2023; 35(23):16893-16916. DOI: https://doi.org/10.1007/s00521-023-08423-1
Homod RZ, Sahari KSM, Almurib HAF, Nagi FH. Double Cooling Coil Model for Non-Linear HVAC System Using RLF Method. Energy and Buildings 2011; 43(9):2043-2054. DOI: https://doi.org/10.1016/j.enbuild.2011.03.023
Homod RZ, Mahlia TMI, Mohamed HAF. PID-Cascade for HVAC System Control. The Second International Conference on Control, Instrumentation and Mechatronic Engineering (CIM09), Malacca, Malaysia, 2009; 598-603.
Chiang ML, Li-Chen F. Hybrid System Based Adaptive Control for the Nonlinear HVAC System. Proceeding of the Conference on American Control, Minneapolis, MN, USA, 2006; 5324-5329.
Chiang ML, Li-Chen F. Adaptive Control of Switched Systems with Application to HVAC System. IEEE International Conference on Control Applications, Singapore, 2007; 367-372. DOI: https://doi.org/10.1109/CCA.2007.4389258
Dawood SM, Hatami A, Homod RZ. HVAC System Modeling and Control Methods: A Review and Case Study. Journal of Energy Management and Technology 2022; 6(4):217-231.
ASHRAE. Ventilation for Acceptable Indoor Air Quality: ASHRAE Standard 62. Atlanta, USA: American Society of Heating, Refrigerating and Air-Conditioning Engineers; 2001. Available from: https://www.ashrae.org/technical-resources/bookstore/standards-62-1-62-2
Homod RZ, Salleh K, Sahari M, Almurib HAF. Energy Saving by Integrated Control of Natural Ventilation and HVAC Systems Using Model Guide for Comparison. Renewable Energy 2014; 71:639-650. DOI: https://doi.org/10.1016/j.renene.2014.06.015
Amouei A, Aghalari Z, Zarei A. Evaluating the Relationships Between Air Pollution and Environmental Parameters with Sick Building Syndrome in Schools of Northern Iran. Indoor and Built Environment 2019; 28(10):1422-1430. DOI: https://doi.org/10.1177/1420326X19842302
Wang H, Xie L, Liu S. A Model-Based Control of CO2 Concentration in Multi-Zone ACB Air-Conditioning Systems. 2016 12th IEEE International Conference on Control and Automation (ICCA), Kathmandu, Nepal, 2016; 467-472. DOI: https://doi.org/10.1109/ICCA.2016.7505321
Baghaee S, Ulusoy I. User Comfort and Energy Efficiency in HVAC Systems by Q-Learning. 2018 26th Signal Processing and Communications Applications Conference (SIU), Izmir, Turkey, 2018; 1-4. DOI: https://doi.org/10.1109/SIU.2018.8404287
Chapra SC, Canale RP. Numerical Methods for Engineers. 6th ed. New York, USA: McGraw-Hill; 2011.
Ayoub A, Jia Z, Szepesv C, Lin W. Model-Based Reinforcement Learning with Value-Targeted Regression. 37th International Conference on Machine Learning, 2020; 463-474.
Silver D, Lever G, Heess N, Degris T, Wierstra D, Riedmiller M. Deterministic Policy Gradient Algorithms. International Conference on Machine Learning, Beijing, China, 2014; 387-395.
Rijal HB, Humphreys MA, Nicol JF. Development of a Window Opening Algorithm Based on Adaptive Thermal Comfort to Predict Occupant Behavior in Japanese Dwellings. Japan Architectural Review 2018; 1(3):310-321. DOI: https://doi.org/10.1002/2475-8876.12043
Noel MM, Pandian BJ. Control of a Nonlinear Liquid Level System Using a New Artificial Neural Network-Based Reinforcement Learning Approach. Applied Soft Computing Journal 2014; 23(October) :444-451. DOI: https://doi.org/10.1016/j.asoc.2014.06.037
Wang L, Wang Z, Yang R. Intelligent Multiagent Control System for Energy and Comfort Management in Smart and Sustainable Buildings. IEEE Transactions on Smart Grid 2012; 3(2):605-617. DOI: https://doi.org/10.1109/TSG.2011.2178044
Yuan Z, Huang Y, Lu X, Huang J, Liu Q, Qi G, Cao Z. Measurement of CO2 by Wavelength Modulated Reinjection Off-Axis Integrated Cavity Output Spectroscopy at 2 μm. Atmosphere (Basel) 2021; 12(10):1247. DOI: https://doi.org/10.3390/atmos12101247
Talebi A, Hatami A. Online Fuzzy Control of HVAC Systems Considering Demand Response and Users' Comfort. Energy Sources, Part B: Economics, Planning, and Policy 2020; 15(7-9):403-422. DOI: https://doi.org/10.1080/15567249.2020.1825557
Turner SC, et al. ANSI/ASHRAE Standard 55-2010, Thermal Environmental Conditions for Human Occupancy. Atlanta, GA: American Society of Heating, Refrigerating and Air-Conditioning Engineers; 2011. Available from: www.ashrae.org
ASHRAE. Standard 55-2004. Thermal Environmental Conditions for Human Occupancy. Atlanta, USA: American Society of Heating, Refrigerating and Air-Conditioning Engineers; 2004. Available from: https://webstore.ansi.org/standards/ashrae/ansiashrae552004