因磁阵hang住导致集群业务卡住的问题

问题现象

资源池化性能机异常，集群业务卡住。

定位方法

1. 查看pg_stat_activity视图，查看业务卡住的基本信息，如线程号、SQL、查询开始时间等。

1. openGauss=# select * from pg_stat_activity; 2. -[ RECORD 1 ]----+------------------------------------------------------------------------------------------------------------------------------------ 3. datid | 15199 4. datname | postgres 5. pid | 281450315045296 6. sessionid | 281450315045296 7. usesysid | 10 8. usename | carrot 9. application_name | gsql 10. client_addr | 11. client_hostname | 12. client_port | -1 13. backend_start | 2024-10-12 10:17:01.195964+08 14. xact_start | 2024-10-12 10:17:02.421981+08 15. query_start | 2024-10-12 10:17:02.421981+08 16. state_change | 2024-10-12 10:17:02.421988+08 17. waiting | f 18. enqueue | 19. state | active 20. resource_pool | default_pool 21. query_id | 562949953434164 22. query | insert into a select * from a; 23. connection_info | {"driver_name":"libpq","driver_version":"(openGauss 6.0.0 build d1f02e70) compiled at 2024-10-11 10:39:30 commit 0 last mr debug"} 24. unique_sql_id | 3750366782 25. .... 2. 查看pg_thread_wait_status，查看卡死的业务的等待信息，均在等待IO。

1. openGauss=# select * from pg_thread_wait_status; 2. node_name | db_name | thread_name | query_id | tid | sessionid | lwtid | psessionid | tlevel | smpid | wait_status | wait_event | locktag | lockmode | block_sessionid | 3. global_sessionid 4. -----------+----------+------------------------+-----------------+-----------------+-----------------+---------+------------+--------+-------+-------------+---------------+---------+----------+-----------------+ 5. ------------------ 6. dn1 | | PageWriter | 0 | 281451104491952 | 281451104491952 | 2956573 | | 0 | 0 | wait io | DataFileWrite | | | | 7. 0:0#0 8. dn1 | | PageWriter | 0 | 281451123431856 | 281451123431856 | 2956572 | | 0 | 0 | wait io | DataFileWrite | | | | 9. 0:0#0 10. dn1 | | PageWriter | 0 | 281451140274608 | 281451140274608 | 2956571 | | 0 | 0 | wait io | DataFileWrite | | | | 11. 0:0#0 12. dn1 | | PageWriter | 0 | 281451177892272 | 281451177892272 | 2956569 | | 0 | 0 | wait io | DataFileWrite | | | | 13. 0:0#0 14. dn1 | | PageWriter | 0 | 281451161049520 | 281451161049520 | 2956570 | | 0 | 0 | wait io | DataFileWrite | | | | 15. 0:0#0 16. dn1 | | undo recycler | 0 | 281450350696880 | 281450350696880 | 2980833 | | 0 | 0 | none | none | | | | 17. 0:0#0 18. dn1 | | LWLock Monitor | 0 | 281450779564464 | 281450779564464 | 2956800 | | 0 | 0 | none | none | | | | 19. 0:0#0 20. dn1 | | TwoPhase Cleaner | 0 | 281450800470448 | 281450800470448 | 2956799 | | 0 | 0 | none | none | | | | 21. 0:0#0 22. dn1 | | Wal Writer Auxiliary | 0 | 281450963589552 | 281450963589552 | 2956790 | | 0 | 0 | none | none | | | | 23. 0:0#0 24. dn1 | | Wal Writer | 0 | 281451266628016 | 281451266628016 | 2956789 | | 0 | 0 | wait io | WalWrite | | | | 25. 0:0#0 26. dn1 | | CheckPointer | 0 | 281451211577776 | 281451211577776 | 2956567 | | 0 | 0 | none | none | | | | 27. 0:0#0 28. dn1 | | InvalidBufferBgWriter | 0 | 281451194735024 | 281451194735024 | 2956568 | | 0 | 0 | none | none | | | | 29. 0:0#0 30. dn1 | postgres | gsql | 562949953434190 | 281450315045296 | 281450315045296 | 680205 | | 0 | 0 | wait io | DataFileWrite | | | | 31. 0:0#0 32. dn1 | postgres | gsql | 562949953434195 | 281450080098736 | 281450080098736 | 671968 | | 0 | 0 | wait io | DataFileWrite | | | | 33. 0:0#0 3. 已确认为存储系统问题，登陆磁阵平台，查看磁阵网络状态、IO状态、日志等，进一步确认磁阵故障。

问题根因

磁阵网络异常或功能异常。

解决方案

若磁阵网络异常，则恢复网络。

若磁阵功能异常，则需要按照磁阵故障手册继续排查。