different hotspot on libssl/libcrypto from sysbench 1.0 to 1.1

[Nor7th]

I'm running sysbench 1.0.20 / 1.1.0 on an ARM server, but observed different hotspots on the mysqld side:

(mysql version: 8.0.30)

The v1.0.20 case:

The v1.1.0 case:

The first case obviously has a nonnegligible burden on libssl/libcrypt. Here is the expanded result:

+   99.99%    64.96%  mysqld                                                                                                                                                                                                                                          
+   99.99%     1.32%  libpthread-2.28.so                                                                                                                                                                                                                              
+   99.98%     5.34%  libc-2.28.so                                                                                                                                                                                                                                    
-   26.58%     1.90%  libssl.so.1.1.1k                                                                                                                                                                                                                                
   - 20.32% SSL_write                                                                                                                                                                                                                                                 
      - 0xffff886a598c                                                                                                                                                                                                                                                
         - 20.30% 0xffff886946b8                                                                                                                                                                                                                                      
            - 17.37% 0xffff886944a0                                                                                                                                                                                                                                   
               - 17.35% 0xffff88693750                                                                                                                                                                                                                                
                  - 17.35% BIO_write                                                                                                                                                                                                                                  
                     - 17.14% 0xffff88451f64                                                                                                                                                                                                                          
                        - 17.11% 0xffff88452f30                                                                                                                                                                                                                       
                           - 17.05% 0xffff88458fe0                                                                                                                                                                                                                    
                              - 17.04% __libc_write                                                                                                                                                                                                                   
                                 - 16.91% el0_sync                                                                                                                                                                                                                    
                                    - el0_sync_handler                                                                                                                                                                                                                
                                       - 16.88% do_el0_svc                                                                                                                                                                                                            
                                          - 16.64% __arm64_sys_write                                                                                                                                                                                                  
                                             - 16.63% ksys_write                                                                                                                                                                                                      
                                                - 16.46% vfs_write                                                                                                                                                                                                    
                                                   - 16.35% __vfs_write                                                                                                                                                                                               
                                                      - 16.31% new_sync_write                                                                                                                                                                                         
                                                         - 16.24% sock_write_iter                                                                                                                                                                                     
                                                            - 16.18% sock_sendmsg                                                                                                                                                                                     
                                                               - 16.06% inet6_sendmsg                                                                                                                                                                                 
                                                                  - 15.96% tcp_sendmsg                                                                                                                                                                                
                                                                     - 15.70% tcp_sendmsg_locked                                                                                                                                                                      
                                                                        - 13.82% tcp_push                                                                                                                                                                             
                                                                           - 13.79% __tcp_push_pending_frames                                                                                                                                                         
                                                                              - 13.71% tcp_write_xmit                                                                                                                                                                 
                                                                                 - 13.11% __tcp_transmit_skb                                                                                                                                                          
                                                                                    - 12.46% ip_queue_xmit                                                                                                                                                            
                                                                                       - 12.43% __ip_queue_xmit                                                                                                                                                       
                                                                                          - 12.29% ip_local_out                                                                                                                                                       
                                                                                             - 9.44% ip_output                                                                                                                                                        
                                                                                                - 9.11% ip_finish_output                                                                                                                                              
                                                                                                   - 9.09% __ip_finish_output                                                                                                                                         
                                                                                                      - 9.01% ip_finish_output2                                                                                                                                       
                                                                                                         - 7.72% __local_bh_enable_ip                                                                                                                                 
                                                                                                            - do_softirq.part.1                                                                                                                                       
                                                                                                               - 7.65% __softirqentry_text_start                                                                                                                      
                                                                                                                  - 7.53% net_rx_action                                                                                                                               
                                                                                                                     - 7.43% __napi_poll                                                                                                                              
                                                                                                                        - 7.41% process_backlog                                                                                                                       
                                                                                                                           - 7.30% __netif_receive_skb                                                                                                                
                                                                                                                              - 7.29% __netif_receive_skb_core                                                                                                        
                                                                                                                                 - 7.08% ip_rcv                                                                                                                       
                                                                                                                                    - 5.76% ip_rcv_finish                                                                                                             
                                                                                                                                       - 5.64% ip_local_deliver                                                                                                       
                                                                                                                                          - 5.01% ip_local_deliver_finish                                                                                             
                                                                                                                                             - 4.94% ip_protocol_deliver_rcu                                                                                          
                                                                                                                                                - 4.80% tcp_v4_rcv                                                                                                    
                                                                                                                                                   - 4.09% tcp_v4_do_rcv                                                                                              
                                                                                                                                                      - 3.99% tcp_rcv_established                                                                                     
                                                                                                                                                         - 1.66% tcp_ack                                                                                              
                                                                                                                                                              1.09% tcp_clean_rtx_queue                                                                               
                                                                                                                                                         - 1.56% tcp_data_ready                                                                                       
                                                                                                                                                            - sock_def_readable                                                                                       
                                                                                                                                                               - 1.47% __wake_up_sync_key                                                                             
                                                                                                                                                                    __wake_up_common_lock                                                                             
                                                                                                                                            0.54% nf_hook_slow
                                                                                                                                    - 1.16% nf_hook_slow                                                                                                              
                                                                                                                                       - 0.95% nft_do_chain_inet                                                                                                      
                                                                                                                                            0.94% nft_do_chain                                                                                                        
                                                                                                         - 1.13% dev_queue_xmit                                                                                                                                       
                                                                                                            - 1.12% __dev_queue_xmit                                                                                                                                  
                                                                                                               - 0.63% dev_hard_start_xmit                                                                                                                            
                                                                                                                    0.51% loopback_xmit                                                                                                                               
                                                                                             - 2.82% __ip_local_out                                                                                                                                                   
                                                                                                - 2.59% nf_hook_slow                                                                                                                                                  
                                                                                                   - 1.46% ipv4_conntrack_local                                                                                                                                       
                                                                                                      - 1.37% nf_conntrack_in                                                                                                                                         
                                                                                                           0.68% nf_conntrack_tcp_packet                                                                                                                              
                                                                        - 0.70% sk_stream_alloc_skb                                                                                                                                                                   
                                                                             0.59% __alloc_skb                                                                                                                                                                        
            - 2.46% 0xffff88693e4c                                                                                                                                                                                                                                    
               - 1.80% 0xffff88698824                                                                                                                                                                                                                                 
                  - 1.79% 0xffff884f8de0                                                                                                                                                                                                                              
                     - 1.78% 0xffff884ef2a0                                                                                                                                                                                                                           
                        - CRYPTO_gcm128_encrypt_ctr32                                                                                                                                                                                                                 
                             1.11% CRYPTO_gcm128_encrypt_ctr32                                                                                                                                                                                                        
   - 4.08% SSL_read                                                                                                                                                                                                                                                   
      - 0xffff886a56e0                                                                                                                                                                                                                                                
         - 4.05% 0xffff8869b600                                                                                                                                                                                                                                       
            - 4.03% 0xffff88694a60                                                                                                                                                                                                                                    
               - 2.24% 0xffff88696e8c                                                                                                                                                                                                                                 
                  - 2.22% 0xffff88693454                                                                                                                                                                                                                              
                     - BIO_read                                                                                                                                                                                                                                       
                        - 1.95% 0xffff88451dd4                                                                                                                                                                                                                        
                           - 1.93% 0xffff88452f80                                                                                                                                                                                                                     
                              - 1.85% 0xffff88458f44                                                                                                                                                                                                                  
                                 - 1.84% __libc_read                                                                                                                                                                                                                  
                                    - 1.67% el0_sync                                                                                                                                                                                                                  
                                       - el0_sync_handler                                                                                                                                                                                                             
                                          - 1.66% do_el0_svc                                                                                                                                                                                                          
                                             - 1.38% __arm64_sys_read                                                                                                                                                                                                 
                                                - 1.37% ksys_read                                                                                                                                                                                                     
                                                   - 1.22% vfs_read                                                                                                                                                                                                   
                                                      - 1.09% __vfs_read                                                                                                                                                                                              
                                                         - 1.04% new_sync_read                                                                                                                                                                                        
                                                            - 0.97% sock_read_iter                                                                                                                                                                                    
                                                               - 0.91% sock_recvmsg                                                                                                                                                                                   
                                                                  - 0.86% inet6_recvmsg                                                                                                                                                                               
                                                                       0.82% tcp_recvmsg                                                                                                                                                                              
                 0.94% 0xffff886971d0                                                                                                                                                                                                                                 
               - 0.83% 0xffff88697478                                                                                                                                                                                                                                 
                  - 0.83% 0xffff88693454                                                                                                                                                                                                                              
                     - BIO_read                                                                                                                                                                                                                                       
                        - 0.78% 0xffff88451dd4                                                                                                                                                                                                                        
                           - 0.78% 0xffff88452f80                                                                                                                                                                                                                     
                              - 0.77% 0xffff88458f44                                                                                                                                                                                                                  
                                 - 0.77% __libc_read                                                                                                                                                                                                                  
                                    - 0.69% el0_sync                                                                                                                                                                                                                  
                                       - el0_sync_handler                                                                                                                                                                                                             
                                          - 0.69% do_el0_svc                                                                                                                                                                                                          
                                             - 0.59% __arm64_sys_read                                                                                                                                                                                                
                                                - ksys_read                                                                                                                                                                                                           
                                                   - 0.54% vfs_read                                                                                                                                                                                                   
                                                      - 0.51% __vfs_read                                                                                                                                                                                              
                                                           0.50% new_sync_read
   - 1.90% thread_start                                                                                                                                                                                                                                               
        start_thread                                                                                                                                                                                                                                                  
        pfs_spawn_thread                                                                                                                                                                                                                                              
        handle_connection                                                                                                                                                                                                                                             
      - do_command                                                                                                                                                                                                                                                    
         - 0.95% dispatch_command                                                                                                                                                                                                                                     
            - THD::send_statement_status                                                                                                                                                                                                                              
               - 0.85% Protocol_classic::send_eof                                                                                                                                                                                                                     
                    net_send_ok                                                                                                                                                                                                                                       
                    net_flush                                                                                                                                                                                                                                         
                    net_write_packet                                                                                                                                                                                                                                  
                    vio_ssl_write                                                                                                                                                                                                                                     
                  - SSL_write                                                                                                                                                                                                                                         
                     - 0.75% 0xffff886a598c                                                                                                                                                                                                                           
                          0.63% 0xffff886946b8                                                                                                                                                                                                                        
         - 0.95% Protocol_classic::get_command                                                                                                                                                                                                                        
              my_net_read                                                                                                                                                                                                                                             
              net_read_packet                                                                                                                                                                                                                                         
              net_read_raw_loop                                                                                                                                                                                                                                       
            - vio_ssl_read                                                                                                                                                                                                                                            
               - 0.88% SSL_read                                                                                                                                                                                                                                       
                  - 0.75% 0xffff886a56e0                                                                                                                                                                                                                              
                     - 0.69% 0xffff8869b600                                                                                                                                                                                                                           
                          0.53% 0xffff88694a60                                                                                                                                                                                                                        
+   24.24%     3.77%  libcrypto.so.1.1.1k                                                                                                                                                                                                                             
+   22.19%    22.19%  [kernel.kallsyms]                                                                                                                                                                                                                               
     0.34%     0.34%  [vdso]                                                                                                                                                                                                                                          
     0.28%     0.16%  libstdc++.so.6.0.30                                                                                                                                                                                                                             
     0.01%     0.01%  libm-2.28.so

There is a CRYPTO_gcm128_encrypt_ctr32() function above, seems sysbench v1.0 is communicating with mysqld through AES_128_GCM, I'm not sure about the correct ssl cipher name. Anyway, this plays a part of mysqld hotspots.

After changing to sysbench v1.1 (current master branch) without changing any other configuration like mysql cnf or any environment variables or any sysbench parameters, just change the sysbench binary, the above hotspots on libssl/libcrypto are all gone.

Does anyone knows the change/background on SSL cipher mechnism from sysbench v1.0 to v1.1 ???